Detect accidental answer-key changes with a sentinel

Define the local evaluation task

A synthetic evaluation build has fixed sentinel case IDs and literal answer strings stored in a baseline. Compare every baseline sentinel with the proposed answer key; missing keys and changed strings are integrity failures before candidates are scored. Extra non-sentinel cases may be added without altering sentinel expectations.

Retain fixed sentinel baseline
Compare baseline keys and literal answers
Hold changed or missing definitions before scoring

An authorised rubric change needs a new explicit baseline and retained old evidence.

Use the filled cases

Original synthetic input and literal answer key
Case	Supplied input	Expected structured output	Decision and reason
Sentinel answer altered	{"baseline":{"S1":"R"},"proposed":{"S1":"S"}}	{"missing":[],"changed":["S1"],"intact":false}	R became S without an accepted baseline revision.
Only a new case added	{"baseline":{"S1":"R"},"proposed":{"S1":"R","C2":"A"}}	{"missing":[],"changed":[],"intact":true}	The new non-sentinel case leaves the fixed sentinel unchanged.
Sentinel omitted	{"baseline":{"S1":"R","S2":"A"},"proposed":{"S1":"R"}}	{"missing":["S2"],"changed":[],"intact":false}	Missing S2 is an integrity gap, not a model answer failure.

Download editable CSV ↓

Work the sentinel-missing case. Which output is required, and why?

Show the worked answer

{"missing":["S2"],"changed":[],"intact":false} — Missing S2 is an integrity gap, not a model answer failure.

Run the self-contained evaluator

Save as .mjs and run with Node.js; supplied fixtures only

import assert from 'node:assert/strict';

function evaluate(x){
 const missing=[],changed=[];for(const id of Object.keys(x.baseline).sort()){if(!Object.hasOwn(x.proposed,id))missing.push(id);else if(x.proposed[id]!==x.baseline[id])changed.push(id);}
 return {missing,changed,intact:missing.length+changed.length===0};
}

const cases = [
  {
    "id": "sentinel-mutated",
    "input": {
      "baseline": {
        "S1": "R"
      },
      "proposed": {
        "S1": "S"
      }
    },
    "expected": {
      "missing": [],
      "changed": [
        "S1"
      ],
      "intact": false
    }
  },
  {
    "id": "new-case-only",
    "input": {
      "baseline": {
        "S1": "R"
      },
      "proposed": {
        "S1": "R",
        "C2": "A"
      }
    },
    "expected": {
      "missing": [],
      "changed": [],
      "intact": true
    }
  },
  {
    "id": "sentinel-missing",
    "input": {
      "baseline": {
        "S1": "R",
        "S2": "A"
      },
      "proposed": {
        "S1": "R"
      }
    },
    "expected": {
      "missing": [
        "S2"
      ],
      "changed": [],
      "intact": false
    }
  }
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));

Download runnable example ↓

Use a known sentinel to detect accidental key edits. This check does not turn the sentinel into an unseen test and does not prevent every other dataset error.

Define the local evaluation task

Use the filled cases

Run the self-contained evaluator

Related resources