Define the local evaluation task
A synthetic observation ledger provides Boolean pass labels for repeated attempts of each case under one fixed revision. Each unique nonempty text case ID has exactly one grouped row and at least one attempt; duplicate groups are rejected before consistency counts. Report ever-pass cases, always-pass cases and unstable case IDs separately. Consistently failed cases are not unstable; these are invented labels, not model runs.
- Validate at least one Boolean attempt per case
- Separate any-pass from all-pass
- List mixed-label cases
Missing or non-Boolean attempts require ledger correction.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| One case is unstable | {"cases":[{"id":"C1","passes":[true,true,true]},{"id":"C2","passes":[false,true,false]}]} | {"cases":2,"everPass":2,"alwaysPass":1,"unstable":["C2"]} | Both cases pass once, but only C1 passes every supplied attempt. |
| Consistently failed | {"cases":[{"id":"C3","passes":[false,false]}]} | {"cases":1,"everPass":0,"alwaysPass":0,"unstable":[]} | C3 is consistently failed, distinct from mixed labels. |
| Same case split across two groups | {"cases":[{"id":"C1","passes":[true]},{"id":"C1","passes":[false]}]} | {"error":"duplicate-case-id"} | Two groups cannot hide C1 instability or count it as two distinct cases. |
| Case has no attempt | {"cases":[{"id":"C4","passes":[]}]} | {"error":"attempt-label"} | An empty attempt list cannot establish consistent success or failure. |
Work the unattempted-case case. Which output is required, and why?
Show the worked answer
{"error":"attempt-label"} — An empty attempt list cannot establish consistent success or failure.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(x.cases.some(c=>typeof c.id!=='string'||!c.id.trim()))return {error:'case-id'};
if(new Set(x.cases.map(c=>c.id)).size!==x.cases.length)return {error:'duplicate-case-id'};
if(x.cases.some(c=>!c.passes.length||c.passes.some(p=>typeof p!=='boolean')))return {error:'attempt-label'};
return {cases:x.cases.length,everPass:x.cases.filter(c=>c.passes.some(Boolean)).length,alwaysPass:x.cases.filter(c=>c.passes.every(Boolean)).length,unstable:x.cases.filter(c=>c.passes.some(Boolean)&&!c.passes.every(Boolean)).map(c=>c.id).sort()};
}
const cases = [
{
"id": "one-unstable-case",
"input": {
"cases": [
{
"id": "C1",
"passes": [
true,
true,
true
]
},
{
"id": "C2",
"passes": [
false,
true,
false
]
}
]
},
"expected": {
"cases": 2,
"everPass": 2,
"alwaysPass": 1,
"unstable": [
"C2"
]
}
},
{
"id": "consistent-failure",
"input": {
"cases": [
{
"id": "C3",
"passes": [
false,
false
]
}
]
},
"expected": {
"cases": 1,
"everPass": 0,
"alwaysPass": 0,
"unstable": []
}
},
{
"id": "split-duplicate-groups",
"input": {
"cases": [
{
"id": "C1",
"passes": [
true
]
},
{
"id": "C1",
"passes": [
false
]
}
]
},
"expected": {
"error": "duplicate-case-id"
}
},
{
"id": "unattempted-case",
"input": {
"cases": [
{
"id": "C4",
"passes": []
}
]
},
"expected": {
"error": "attempt-label"
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Retain every attempt with the fixed revision identifier. These finite observed labels describe fixture stability only and cannot establish a general probability of model success.