Structured staff grading
Retain reviewer matrices, disputed keys, required dimensions, order checks and bounded reason codes.
Measure agreement between two staff graders
A list of final decisions omits systematic disagreement between reviewers.
Keep disputed labels out of automatic candidate scoring
A provisional majority or first label is used as final expected truth.
Retain independent grading dimensions without averaging a blocker
Good language quality compensates for a blocking task criterion.
Compare grading with candidate order reversed
A staff decision changes with presentation order while case content remains fixed.
Check that a grader’s reason matches its recorded result
Check declared structured reason codes against a bounded Boolean field rubric.