Define the local evaluation task
A synthetic extractive answer must be one exact contiguous source substring. Case and punctuation are significant, and an empty candidate is invalid. Report its first zero-based start/end positions when found. A plausible combination of separated words is not a source extract under this local rule.
- Keep exact source and candidate
- Find an unchanged contiguous span
- Record span or non-extractive result
A non-extractive answer may be a paraphrase, but it fails this extractive criterion.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Contiguous exact text | {"source":"Collect after review, never before review.","answer":"after review"} | {"extractive":true,"start":8,"end":20} | Characters eight through nineteen form the exact candidate. |
| Separated words joined | {"source":"Collect after review, never before review.","answer":"Collect before review"} | {"extractive":false,"start":null,"end":null} | Those words do not occur as one contiguous source span. |
| Empty candidate | {"source":"Collect after review.","answer":""} | {"extractive":false,"start":null,"end":null} | An empty substring is rejected by the explicit answer contract. |
Work the empty-extract case. Which output is required, and why?
Show the worked answer
{"extractive":false,"start":null,"end":null} — An empty substring is rejected by the explicit answer contract.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(!x.answer.length)return {extractive:false,start:null,end:null};const start=x.source.indexOf(x.answer);
return {extractive:start>=0,start:start>=0?start:null,end:start>=0?start+x.answer.length:null};
}
const cases = [
{
"id": "exact-span",
"input": {
"source": "Collect after review, never before review.",
"answer": "after review"
},
"expected": {
"extractive": true,
"start": 8,
"end": 20
}
},
{
"id": "mosaic-claim",
"input": {
"source": "Collect after review, never before review.",
"answer": "Collect before review"
},
"expected": {
"extractive": false,
"start": null,
"end": null
}
},
{
"id": "empty-extract",
"input": {
"source": "Collect after review.",
"answer": ""
},
"expected": {
"extractive": false,
"start": null,
"end": null
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Use this check only when an exact extract was requested. Its span result does not judge whether the extract answers the question, is current or may be disclosed.