Define the local evaluation task
A synthetic staff knowledge answer needs every document in a declared required bundle. Count distinct required documents present in the supplied retrieval list; unrelated hits and repeated chunks earn no coverage. Return missing IDs and the exact numerator/denominator. An empty required bundle is unresolved rather than automatically successful.
- Deduplicate required document IDs
- Intersect required IDs with supplied hits
- Return missing IDs and bundle completeness
An empty required list needs an answer-bundle definition.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Missing independent part | {"required":["D1","D2"],"hits":["D1","X","Y"]} | {"found":["D1"],"missing":["D2"],"numerator":1,"denominator":2,"complete":false} | D1 does not substitute for D2; a relevant hit alone is insufficient. |
| Complete with repeated hit | {"required":["D1","D2"],"hits":["D2","D1","D1"]} | {"found":["D1","D2"],"missing":[],"numerator":2,"denominator":2,"complete":true} | Both required IDs occur; the repeated D1 does not change the denominator. |
| Required bundle not declared | {"required":[],"hits":["D1"]} | {"found":[],"missing":[],"numerator":0,"denominator":0,"complete":null} | No required bundle was supplied, so completeness is unknown rather than 0/0 success. |
Work the undeclared-bundle case. Which output is required, and why?
Show the worked answer
{"found":[],"missing":[],"numerator":0,"denominator":0,"complete":null} — No required bundle was supplied, so completeness is unknown rather than 0/0 success.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
const required=[...new Set(x.required)].sort(),hits=new Set(x.hits);
const found=required.filter(id=>hits.has(id)),missing=required.filter(id=>!hits.has(id));
return {found,missing,numerator:found.length,denominator:required.length,complete:required.length?missing.length===0:null};
}
const cases = [
{
"id": "missing-piece",
"input": {
"required": [
"D1",
"D2"
],
"hits": [
"D1",
"X",
"Y"
]
},
"expected": {
"found": [
"D1"
],
"missing": [
"D2"
],
"numerator": 1,
"denominator": 2,
"complete": false
}
},
{
"id": "duplicate-hit",
"input": {
"required": [
"D1",
"D2"
],
"hits": [
"D2",
"D1",
"D1"
]
},
"expected": {
"found": [
"D1",
"D2"
],
"missing": [],
"numerator": 2,
"denominator": 2,
"complete": true
}
},
{
"id": "undeclared-bundle",
"input": {
"required": [],
"hits": [
"D1"
]
},
"expected": {
"found": [],
"missing": [],
"numerator": 0,
"denominator": 0,
"complete": null
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep one required-bundle row per staff question beside the retrieved IDs. This checker diagnoses supplied retrieval material; it neither generates an answer nor establishes that a passage supports a claim.