Define the local evaluation task
Two synthetic staff graders provide one Boolean judgement per unique nonblank text case ID, compared exactly. Missing/nontext IDs or duplicate case rows are held before denominator counts; empty batches remain valid as empty observation ledgers. Build the full two-by-two agreement matrix: both pass, both fail, A-only pass and B-only pass. Report agreement numerator and total case denominator. Do not resolve disagreements or treat agreement as proof the labels are correct.
- Validate both staff labels per case
- Populate the four agreement cells
- Retain disagreement IDs for separate review
Missing labels remain an evidence gap, not an automatic disagreement or agreement.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Four-case agreement table | {"rows":[{"id":"1","a":true,"b":true},{"id":"2","a":true,"b":true},{"id":"3","a":false,"b":false},{"id":"4","a":true,"b":false}]} | {"matrix":{"bothPass":2,"bothFail":1,"aOnly":1,"bOnly":0},"agreement":3,"total":4,"disagreements":["4"]} | Three of four agree; case 4 remains a labelled disagreement. |
| Disagreement in both directions | {"rows":[{"id":"A","a":true,"b":false},{"id":"B","a":false,"b":true}]} | {"matrix":{"bothPass":0,"bothFail":0,"aOnly":1,"bOnly":1},"agreement":0,"total":2,"disagreements":["A","B"]} | Equal pass counts conceal opposite case decisions. |
| Same case judged twice in the ledger | {"rows":[{"id":"A","a":true,"b":false},{"id":"A","a":false,"b":true}]} | {"error":"review-case-identity"} | Two rows for A cannot count as two independent cases or conceal contradictory judgements. |
| String instead of Boolean | {"rows":[{"id":"A","a":"pass","b":true}]} | {"error":"review-label"} | The grader requires Boolean labels; no implicit string conversion is allowed. |
Work the invalid-label case. Which output is required, and why?
Show the worked answer
{"error":"review-label"} — The grader requires Boolean labels; no implicit string conversion is allowed.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(x.rows.some(r=>typeof r.id!=='string'||!r.id.trim())||new Set(x.rows.map(r=>r.id)).size!==x.rows.length)return {error:'review-case-identity'};
if(x.rows.some(r=>typeof r.a!=='boolean'||typeof r.b!=='boolean'))return {error:'review-label'};
const matrix={bothPass:0,bothFail:0,aOnly:0,bOnly:0},disagreements=[];for(const r of x.rows){const key=r.a?(r.b?'bothPass':'aOnly'):(r.b?'bOnly':'bothFail');matrix[key]++;if(r.a!==r.b)disagreements.push(r.id);}
return {matrix,agreement:matrix.bothPass+matrix.bothFail,total:x.rows.length,disagreements};
}
const cases = [
{
"id": "four-case-matrix",
"input": {
"rows": [
{
"id": "1",
"a": true,
"b": true
},
{
"id": "2",
"a": true,
"b": true
},
{
"id": "3",
"a": false,
"b": false
},
{
"id": "4",
"a": true,
"b": false
}
]
},
"expected": {
"matrix": {
"bothPass": 2,
"bothFail": 1,
"aOnly": 1,
"bOnly": 0
},
"agreement": 3,
"total": 4,
"disagreements": [
"4"
]
}
},
{
"id": "opposite-disagreements",
"input": {
"rows": [
{
"id": "A",
"a": true,
"b": false
},
{
"id": "B",
"a": false,
"b": true
}
]
},
"expected": {
"matrix": {
"bothPass": 0,
"bothFail": 0,
"aOnly": 1,
"bOnly": 1
},
"agreement": 0,
"total": 2,
"disagreements": [
"A",
"B"
]
}
},
{
"id": "duplicate-reviewed-case",
"input": {
"rows": [
{
"id": "A",
"a": true,
"b": false
},
{
"id": "A",
"a": false,
"b": true
}
]
},
"expected": {
"error": "review-case-identity"
}
},
{
"id": "invalid-label",
"input": {
"rows": [
{
"id": "A",
"a": "pass",
"b": true
}
]
},
"expected": {
"error": "review-label"
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Use the matrix to locate a rubric disagreement before adjudication. No chance-corrected statistic or general inter-rater reliability claim is implied by this local fraction.