Define the local evaluation task
A synthetic Boolean rubric requires title and source fields. Declared structured reason codes are missing-title, missing-source or all-required-met. Derive the bounded missing-field code set from supplied field Booleans. A recorded pass/fail result and reason set must both match; unknown or repeated reason codes are invalid. Free prose is not interpreted or graded.
- Derive missing fields from supplied Booleans
- Build the exact permitted reason set
- Check result and reasons independently
Unknown codes or free prose need a different rubric; this checker cannot interpret them.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Result and reason contradict | {"fields":{"title":false,"source":true},"pass":false,"reasons":["all-required-met"]} | {"expectedPass":false,"expectedReasons":["missing-title"],"coherent":false} | Failure is correct but its all-required-met code contradicts the missing title. |
| Two missing-field reasons | {"fields":{"title":false,"source":false},"pass":false,"reasons":["missing-source","missing-title"]} | {"expectedPass":false,"expectedReasons":["missing-source","missing-title"],"coherent":true} | Both declared missing-field codes match the bounded field rubric. |
| Unregistered reason code | {"fields":{"title":true,"source":true},"pass":true,"reasons":["looks-good"]} | {"status":"invalid-reason-input"} | Looks-good is free wording outside the declared structured code set. |
Work the unknown-reason case. Which output is required, and why?
Show the worked answer
{"status":"invalid-reason-input"} — Looks-good is free wording outside the declared structured code set.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
const allowed=['missing-title','missing-source','all-required-met'];if(typeof x.fields.title!=='boolean'||typeof x.fields.source!=='boolean'||typeof x.pass!=='boolean'||x.reasons.some(r=>!allowed.includes(r))||new Set(x.reasons).size!==x.reasons.length)return {status:'invalid-reason-input'};
const expected=[];if(!x.fields.title)expected.push('missing-title');if(!x.fields.source)expected.push('missing-source');const expectedPass=expected.length===0;if(expectedPass)expected.push('all-required-met');expected.sort();
return {expectedPass,expectedReasons:expected,coherent:x.pass===expectedPass&&JSON.stringify([...x.reasons].sort())===JSON.stringify(expected)};
}
const cases = [
{
"id": "contradictory-reason",
"input": {
"fields": {
"title": false,
"source": true
},
"pass": false,
"reasons": [
"all-required-met"
]
},
"expected": {
"expectedPass": false,
"expectedReasons": [
"missing-title"
],
"coherent": false
}
},
{
"id": "two-missing-reasons",
"input": {
"fields": {
"title": false,
"source": false
},
"pass": false,
"reasons": [
"missing-source",
"missing-title"
]
},
"expected": {
"expectedPass": false,
"expectedReasons": [
"missing-source",
"missing-title"
],
"coherent": true
}
},
{
"id": "unknown-reason",
"input": {
"fields": {
"title": true,
"source": true
},
"pass": true,
"reasons": [
"looks-good"
]
},
"expected": {
"status": "invalid-reason-input"
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep structured reasons and field evidence with each grading decision. A coherent reason can only support this bounded rubric; it does not prove arbitrary explanatory prose is true.