Keep disputed labels out of automatic candidate scoring

Define the local evaluation task

A synthetic answer-key review marks cases agreed or disputed. Only agreed cases with a Boolean expected label may be scored against a Boolean candidate label; disputed cases remain held regardless of the candidate. Report scored results and held IDs separately. An invalid agreed label blocks that case as invalid rather than choosing the first reviewer.

Read adjudication state before candidate
Score only valid agreed labels
Retain disputed and invalid cases separately

Held labels need a recorded adjudication; no majority or first-label fallback is supplied.

Use the filled cases

Original synthetic input and literal answer key
Case	Supplied input	Expected structured output	Decision and reason
One label disputed	{"cases":[{"id":"C1","state":"agreed","expected":true,"answer":true},{"id":"C2","state":"disputed","expected":null,"answer":true}]}	{"scored":[{"id":"C1","pass":true}],"held":["C2"],"invalid":[]}	Only C1 is scored; C2 does not enter a pass denominator.
Agreed key, wrong candidate	{"cases":[{"id":"C1","state":"agreed","expected":false,"answer":true}]}	{"scored":[{"id":"C1","pass":false}],"held":[],"invalid":[]}	The candidate differs from the agreed Boolean key.
Agreed but malformed key	{"cases":[{"id":"C3","state":"agreed","expected":"pass","answer":true}]}	{"scored":[],"held":[],"invalid":["C3"]}	A string key cannot be scored under the Boolean rubric.

Download editable CSV ↓

Work the invalid-agreed-key case. Which output is required, and why?

Show the worked answer

{"scored":[],"held":[],"invalid":["C3"]} — A string key cannot be scored under the Boolean rubric.

Run the self-contained evaluator

Save as .mjs and run with Node.js; supplied fixtures only

import assert from 'node:assert/strict';

function evaluate(x){
 const scored=[],held=[],invalid=[];for(const c of x.cases){if(c.state==='disputed')held.push(c.id);else if(c.state!=='agreed'||typeof c.expected!=='boolean'||typeof c.answer!=='boolean')invalid.push(c.id);else scored.push({id:c.id,pass:c.answer===c.expected});}
 return {scored,held,invalid};
}

const cases = [
  {
    "id": "one-held-label",
    "input": {
      "cases": [
        {
          "id": "C1",
          "state": "agreed",
          "expected": true,
          "answer": true
        },
        {
          "id": "C2",
          "state": "disputed",
          "expected": null,
          "answer": true
        }
      ]
    },
    "expected": {
      "scored": [
        {
          "id": "C1",
          "pass": true
        }
      ],
      "held": [
        "C2"
      ],
      "invalid": []
    }
  },
  {
    "id": "agreed-candidate-fail",
    "input": {
      "cases": [
        {
          "id": "C1",
          "state": "agreed",
          "expected": false,
          "answer": true
        }
      ]
    },
    "expected": {
      "scored": [
        {
          "id": "C1",
          "pass": false
        }
      ],
      "held": [],
      "invalid": []
    }
  },
  {
    "id": "invalid-agreed-key",
    "input": {
      "cases": [
        {
          "id": "C3",
          "state": "agreed",
          "expected": "pass",
          "answer": true
        }
      ]
    },
    "expected": {
      "scored": [],
      "held": [],
      "invalid": [
        "C3"
      ]
    }
  }
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));

Download runnable example ↓

Keep candidate outputs even while the key is disputed. Scoring fewer cases changes coverage; a clean scored subset must not be presented as the complete evaluation.

Define the local evaluation task

Use the filled cases

Run the self-contained evaluator

Related resources