Skip to content
Astera Technology

Define evaluator results for an empty candidate file

An empty output yields a vacuous all-passed result from an empty loop.

Local grader correctness →

Download complete case notes and answer keys ↓

Define the local evaluation task

A synthetic batch grader requires exactly one candidate for every declared case ID. Candidate values are not graded until missing and extra case IDs are checked. An empty expected set is an invalid evaluation definition; an empty candidate list against three expected cases is incomplete, not a vacuous pass.

  1. Validate a nonempty expected set
  2. Find missing and extra candidate IDs
  3. Only then permit content grading

Coverage failures remain incomplete; do not synthesize outputs for missing cases.

Use the filled cases

Original synthetic input and literal answer key
CaseSupplied inputExpected structured outputDecision and reason
Candidate file empty{"expected":["C1","C2","C3"],"candidates":[]}{"status":"incomplete","missing":["C1","C2","C3"],"extra":[]}All three candidates are missing, so no content pass rate is established.
All expected IDs present{"expected":["C1","C2"],"candidates":["C2","C1"]}{"status":"ready-to-grade","missing":[],"extra":[]}Coverage is complete; content grading is a subsequent step.
Expected case list empty{"expected":[],"candidates":[]}{"status":"invalid-definition"}An empty test definition cannot establish complete evaluated coverage.
Download editable CSV ↓

Work the no-evaluation-definition case. Which output is required, and why?

Show the worked answer

{"status":"invalid-definition"} — An empty test definition cannot establish complete evaluated coverage.

Run the self-contained evaluator

Save as .mjs and run with Node.js; supplied fixtures only
import assert from 'node:assert/strict';

function evaluate(x){
 if(!x.expected.length)return {status:'invalid-definition'};const missing=x.expected.filter(id=>!x.candidates.includes(id)).sort(),extra=x.candidates.filter(id=>!x.expected.includes(id)).sort();
 return {status:missing.length+extra.length?'incomplete':'ready-to-grade',missing,extra};
}

const cases = [
  {
    "id": "empty-candidates",
    "input": {
      "expected": [
        "C1",
        "C2",
        "C3"
      ],
      "candidates": []
    },
    "expected": {
      "status": "incomplete",
      "missing": [
        "C1",
        "C2",
        "C3"
      ],
      "extra": []
    }
  },
  {
    "id": "full-candidates",
    "input": {
      "expected": [
        "C1",
        "C2"
      ],
      "candidates": [
        "C2",
        "C1"
      ]
    },
    "expected": {
      "status": "ready-to-grade",
      "missing": [],
      "extra": []
    }
  },
  {
    "id": "no-evaluation-definition",
    "input": {
      "expected": [],
      "candidates": []
    },
    "expected": {
      "status": "invalid-definition"
    }
  }
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Download runnable example ↓

Use a separate duplicate-key check before this coverage step. A ready-to-grade disposition is not a pass on any candidate’s factual content.

Related resources