Local grader correctness
Test duplicates, exact tolerances, parsed equality, citation coordinates and candidate coverage before scoring.
Grade unordered outputs without ignoring duplicates
Sorting hides duplicate model outputs as well as harmless order changes.
Apply a supplied absolute tolerance once in an evaluator
Relative tolerance or display rounding is silently substituted for a declared absolute bound.
Compare JSON objects without accepting different array order
Raw JSON string equality fails harmless key order; broad sorting changes meaningful arrays.
Check citation offsets after a Unicode representation change
Byte offsets are reused as code-point offsets and select a different substring.
Define evaluator results for an empty candidate file
An empty output yields a vacuous all-passed result from an empty loop.
Reject duplicate candidate case keys before grading
A map assignment overwrites a failed candidate with a later passing candidate.