Define the local evaluation task
A synthetic diagnostic run supplies a nonempty required document set, the actual supplied context IDs and a staff-labelled answer-correct flag. Attribute an incomplete context to retrieval. Only when the context is complete may an incorrect answer be attributed to generation. Correct answers with incomplete context remain retrieval-evidence gaps, not proven grounded successes.
- Check the required context first
- Inspect answer label only after context completeness
- Record the diagnosed stage and missing IDs
No required set or no staff answer label leaves the diagnosis unprepared.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Required evidence absent | {"required":["D1"],"context":["X"],"correct":false} | {"stage":"retrieval","missing":["D1"]} | D1 never reached the supplied context; answer repair alone cannot establish retrieval. |
| Evidence present but answer wrong | {"required":["D1"],"context":["D1"],"correct":false} | {"stage":"generation","missing":[]} | The declared context requirement is met; the supplied wrong-answer label points downstream. |
| Answer label not supplied | {"required":["D1"],"context":["D1"]} | {"error":"answer-label"} | Without a Boolean staff label, the evaluator cannot call the answer correct or incorrect. |
| Correct answer with evidence gap | {"required":["D1","D2"],"context":["D1"],"correct":true} | {"stage":"retrieval","missing":["D2"]} | A correct label cannot erase missing D2 from the input trace. |
Work the lucky-answer-gap case. Which output is required, and why?
Show the worked answer
{"stage":"retrieval","missing":["D2"]} — A correct label cannot erase missing D2 from the input trace.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(!x.required.length)return {stage:'unresolved'};
if(typeof x.correct!=='boolean')return {error:'answer-label'};
const missing=[...new Set(x.required)].filter(id=>!x.context.includes(id)).sort();
return {stage:missing.length?'retrieval':x.correct?'pass':'generation',missing};
}
const cases = [
{
"id": "missing-context",
"input": {
"required": [
"D1"
],
"context": [
"X"
],
"correct": false
},
"expected": {
"stage": "retrieval",
"missing": [
"D1"
]
}
},
{
"id": "wrong-generation",
"input": {
"required": [
"D1"
],
"context": [
"D1"
],
"correct": false
},
"expected": {
"stage": "generation",
"missing": []
}
},
{
"id": "missing-answer-label",
"input": {
"required": [
"D1"
],
"context": [
"D1"
]
},
"expected": {
"error": "answer-label"
}
},
{
"id": "lucky-answer-gap",
"input": {
"required": [
"D1",
"D2"
],
"context": [
"D1"
],
"correct": true
},
"expected": {
"stage": "retrieval",
"missing": [
"D2"
]
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Retain the same source/question when isolating stages. This checker uses supplied labels and does not claim to infer a model’s internal cause or independently prove semantic answer correctness.