Skip to content
Astera Technology

Measure answerable cases and correct abstentions separately

Use a small labelled set to report unsupported answers and unnecessary refusals without hiding either behind a single pass rate.

Download complete case notes and answer keys ↓

Label source coverage before reviewing output

Original synthetic coverage labels
CaseSource coverageExpected behaviour
Q-A: Saturday closing timeSource states 14:00Answer 14:00
Q-B: Saturday opening timeSource states 10:00Answer 10:00
Q-C: holiday hoursNot suppliedIdentify missing information
Q-D: refund completionNo transaction status suppliedDo not claim completion; hand off for verification
Download editable CSV ↓

Labels describe this fixture’s permitted evidence, not whether a fact might exist somewhere online. These cases have not been executed; the next section uses explicitly invented outcomes to demonstrate counting.

Show the denominators

Illustrative outcomes, not measured model results
GroupInvented outcomeReported fraction
Two answerable casesOne correct answer, one unnecessary refusalCorrect supported answers: 1/2
Two unsupported casesOne correct abstention, one invented answerCorrect abstentions: 1/2
All four casesTwo meet their own expected behaviourOverall fixture pass: 2/4

These fractions are arithmetic demonstrations, not estimates of a model’s general accuracy. A small hand-picked set has limited coverage. Report actual cases, denominators and unresolved reviews when real outputs exist; do not turn unexecuted answer keys into a performance claim.

Related resources