Define the local evaluation task
A synthetic ordered evaluation stops at the first failed case. Later cases must carry null, meaning not run. Observed results form one contiguous prefix: no Boolean result appears after the first failure or after a not-run null begins the suffix. Report planned coverage and success among observed cases separately; do not call unattempted cases passed or failed.
- Validate the ordered stop trace
- Count observed and unattempted cases
- Report success with its observed denominator
A result after the stopping point requires a corrected run definition or ledger.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Second case fails and stops | {"results":[true,false,null,null,null]} | {"planned":5,"observed":2,"passed":1,"notRun":3} | Observed success is 1/2 and planned coverage 2/5; three cases remain not run. |
| All planned cases pass | {"results":[true,true,true]} | {"planned":3,"observed":3,"passed":3,"notRun":0} | No failure triggers early stop, so all three observations are retained. |
| Observation after an unattempted middle case | {"results":[true,null,false]} | {"error":"stop-ledger"} | Once the not-run suffix begins, a later observation violates the contiguous ordered trace. |
| Another result after stopping | {"results":[false,true,null]} | {"error":"stop-ledger"} | The recorded trace violates the supplied stop-on-first-failure rule. |
Work the result-after-stop case. Which output is required, and why?
Show the worked answer
{"error":"stop-ledger"} — The recorded trace violates the supplied stop-on-first-failure rule.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
let stopped=false;for(const v of x.results){if(stopped&&v!==null)return {error:'stop-ledger'};if(v===false||v===null)stopped=true;if(v!==null&&typeof v!=='boolean')return {error:'label'};}
const observed=x.results.filter(v=>v!==null),passed=observed.filter(Boolean).length;return {planned:x.results.length,observed:observed.length,passed,notRun:x.results.length-observed.length};
}
const cases = [
{
"id": "stop-on-second",
"input": {
"results": [
true,
false,
null,
null,
null
]
},
"expected": {
"planned": 5,
"observed": 2,
"passed": 1,
"notRun": 3
}
},
{
"id": "all-completed",
"input": {
"results": [
true,
true,
true
]
},
"expected": {
"planned": 3,
"observed": 3,
"passed": 3,
"notRun": 0
}
},
{
"id": "observation-after-not-run",
"input": {
"results": [
true,
null,
false
]
},
"expected": {
"error": "stop-ledger"
}
},
{
"id": "result-after-stop",
"input": {
"results": [
false,
true,
null
]
},
"expected": {
"error": "stop-ledger"
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep planned IDs beside null dispositions. This finite ledger cannot supply outcomes for skipped cases or a population accuracy estimate.