Grade unordered outputs without ignoring duplicates

Define the local evaluation task

A synthetic generated list is graded as an unordered set, but its output contract allows each member once only. Expected IDs are distinct. Report missing, extra and repeated candidate IDs separately. Harmless ordering changes pass; converting the answer to a set must not hide duplicates.

Record candidate multiplicities
Compare distinct members both ways
Grade order separately from duplicates

The once-only policy must be changed explicitly if repeated occurrences are meaningful.

Use the filled cases

Original synthetic input and literal answer key
Case	Supplied input	Expected structured output	Decision and reason
Same set in another order	{"expected":["A","B"],"answer":["B","A"]}	{"missing":[],"extra":[],"duplicates":[],"pass":true}	Unordered membership is satisfied without repeated entries.
Repeated value would be hidden by a set	{"expected":["A","B"],"answer":["A","A","B"]}	{"missing":[],"extra":[],"duplicates":["A"],"pass":false}	Membership is correct but repeated A fails the once-only requirement.
Expected definition repeats an ID	{"expected":["A","A"],"answer":["A"]}	{"error":"expected-set-definition"}	A duplicated expected ID invalidates the set definition instead of granting a candidate pass.
One missing and one extra	{"expected":["A","B"],"answer":["A","C"]}	{"missing":["B"],"extra":["C"],"duplicates":[],"pass":false}	C does not replace required B; both defects remain visible.

Download editable CSV ↓

Work the missing-extra case. Which output is required, and why?

Show the worked answer

{"missing":["B"],"extra":["C"],"duplicates":[],"pass":false} — C does not replace required B; both defects remain visible.

Run the self-contained evaluator

Save as .mjs and run with Node.js; supplied fixtures only

import assert from 'node:assert/strict';

function evaluate(x){
 if(new Set(x.expected).size!==x.expected.length)return {error:'expected-set-definition'};
 const seen=[...new Set(x.answer)].sort(),duplicates=seen.filter(k=>x.answer.filter(v=>v===k).length>1),missing=x.expected.filter(k=>!seen.includes(k)).sort(),extra=seen.filter(k=>!x.expected.includes(k));
 return {missing,extra,duplicates,pass:missing.length+extra.length+duplicates.length===0};
}

const cases = [
  {
    "id": "permuted-set",
    "input": {
      "expected": [
        "A",
        "B"
      ],
      "answer": [
        "B",
        "A"
      ]
    },
    "expected": {
      "missing": [],
      "extra": [],
      "duplicates": [],
      "pass": true
    }
  },
  {
    "id": "repeat-hidden-by-set",
    "input": {
      "expected": [
        "A",
        "B"
      ],
      "answer": [
        "A",
        "A",
        "B"
      ]
    },
    "expected": {
      "missing": [],
      "extra": [],
      "duplicates": [
        "A"
      ],
      "pass": false
    }
  },
  {
    "id": "duplicate-expected-id",
    "input": {
      "expected": [
        "A",
        "A"
      ],
      "answer": [
        "A"
      ]
    },
    "expected": {
      "error": "expected-set-definition"
    }
  },
  {
    "id": "missing-extra",
    "input": {
      "expected": [
        "A",
        "B"
      ],
      "answer": [
        "A",
        "C"
      ]
    },
    "expected": {
      "missing": [
        "B"
      ],
      "extra": [
        "C"
      ],
      "duplicates": [],
      "pass": false
    }
  }
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));

Download runnable example ↓

Use this evaluator when a model returns category or procedure IDs with no ranking. A set criterion is unsuitable for a ranked answer or ordered action trace.

Define the local evaluation task

Use the filled cases

Run the self-contained evaluator

Related resources