Define the local evaluation task
A synthetic extraction request asks for a unique-ID list in its exact source order. Grade membership and order separately. A reordered list can retain every ID yet fail the order criterion; repeated source or candidate IDs are invalid under this unique-list contract. No automatic sorting is applied to the accepted output.
- Check unique-list contract
- Compare membership separately
- Compare original positions without sorting
Repeated IDs need an occurrence-aware dataset rather than this unique-list rule.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Sorted without request | {"source":["C","A","B"],"answer":["A","B","C"]} | {"membership":true,"order":false} | Every ID remains, but the requested source order C,A,B is lost. |
| Source order retained | {"source":["C","A","B"],"answer":["C","A","B"]} | {"membership":true,"order":true} | Both membership and ordered positions match. |
| Answer repeats an ID | {"source":["C","A","B"],"answer":["C","A","A","B"]} | {"error":"duplicate-id"} | Duplicate A is rejected before order grading; sorting cannot repair it. |
Work the duplicate-answer case. Which output is required, and why?
Show the worked answer
{"error":"duplicate-id"} — Duplicate A is rejected before order grading; sorting cannot repair it.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(new Set(x.source).size!==x.source.length||new Set(x.answer).size!==x.answer.length)return {error:'duplicate-id'};
const membership=x.source.length===x.answer.length&&x.source.every(id=>x.answer.includes(id));
return {membership,order:membership&&x.source.every((id,i)=>x.answer[i]===id)};
}
const cases = [
{
"id": "unrequested-sort",
"input": {
"source": [
"C",
"A",
"B"
],
"answer": [
"A",
"B",
"C"
]
},
"expected": {
"membership": true,
"order": false
}
},
{
"id": "source-order-retained",
"input": {
"source": [
"C",
"A",
"B"
],
"answer": [
"C",
"A",
"B"
]
},
"expected": {
"membership": true,
"order": true
}
},
{
"id": "duplicate-answer",
"input": {
"source": [
"C",
"A",
"B"
],
"answer": [
"C",
"A",
"A",
"B"
]
},
"expected": {
"error": "duplicate-id"
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep the source sequence in the answer key. A requested ordering must survive generation even if an alphabetic list looks tidier; a membership pass should not be reported as an order pass.