Define the local evaluation task
A synthetic search rewrite is reviewed against an explicit list of comparison targets. Targets are canonical IDs supplied by staff, not inferred from arbitrary prose. The rewritten target list must contain every original target and no added target. Report missing and added IDs separately; the input parser is outside this checker.
- Retain original canonical target list
- Compare rewritten target set in both directions
- Repair missing or added scope
Unresolved target extraction needs staff labels before this check.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| One comparison target lost | {"original":["induction","refresher"],"rewritten":["induction"]} | {"missing":["refresher"],"added":[],"preserved":false} | The rewrite no longer asks about the refresher procedure. |
| Same targets in another order | {"original":["induction","refresher"],"rewritten":["refresher","induction"]} | {"missing":[],"added":[],"preserved":true} | This local target set is unordered; both required objects remain. |
| An unrelated target added | {"original":["induction"],"rewritten":["induction","payroll"]} | {"missing":[],"added":["payroll"],"preserved":false} | Even with induction retained, the extra payroll target changes retrieval scope. |
Work the new-target case. Which output is required, and why?
Show the worked answer
{"missing":[],"added":["payroll"],"preserved":false} — Even with induction retained, the extra payroll target changes retrieval scope.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
const a=[...new Set(x.original)].sort(),b=[...new Set(x.rewritten)].sort();
const missing=a.filter(id=>!b.includes(id)),added=b.filter(id=>!a.includes(id));return {missing,added,preserved:missing.length+added.length===0};
}
const cases = [
{
"id": "lost-comparison",
"input": {
"original": [
"induction",
"refresher"
],
"rewritten": [
"induction"
]
},
"expected": {
"missing": [
"refresher"
],
"added": [],
"preserved": false
}
},
{
"id": "reordered-targets",
"input": {
"original": [
"induction",
"refresher"
],
"rewritten": [
"refresher",
"induction"
]
},
"expected": {
"missing": [],
"added": [],
"preserved": true
}
},
{
"id": "new-target",
"input": {
"original": [
"induction"
],
"rewritten": [
"induction",
"payroll"
]
},
"expected": {
"missing": [],
"added": [
"payroll"
],
"preserved": false
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep the original wording and the labelled comparison set with each rewrite. A set pass only confirms these declared targets survived; it does not validate the rest of a natural-language query.