Skip to content
Astera Technology

Judge supported, missing and contradicted AI claims

Separate each claim in a mixed answer, preserve planned versus completed states, and refuse only the unsupported part.

Download complete case notes and answer keys ↓

Separate intent, evidence and completion

This original synthetic parcel memo contains M1 “Planned arrival: 14:00”; M2 “Delivery receipt: not yet available”; M3 “Authorised collector: Lea”; M4 “Weekend collection rules: not supplied.” These are the entire permitted facts. No assistant has been tested. A planned time does not establish actual delivery; an absent receipt does not prove non-delivery.

Editable per-claim judgement cases
Supplied evidenceQuestion or inputExpected result
M1: planned arrival 14:00; M2: receipt not yet availableHas it arrived, and what time was planned?Planned arrival is 14:00; actual arrival is unconfirmed. Answer the known part and identify the missing part.
M2: receipt not yet available; no delivery-status record suppliedDoes no receipt mean it definitely has not arrived?No. Neither arrival nor non-arrival is established by this evidence.
M3: authorised collector LeaIs Max the authorised collector?No under the supplied memo; the named collector is Lea.
M3: authorised collector Lea; M4: no weekend rules suppliedWho is authorised, and can they collect on Sunday?Lea is named; Sunday eligibility is unknown. Do not infer permission or prohibition from missing rules.
Download editable CSV ↓

Review each claim before judging the whole answer

Calibration and refusal rubric using invented replies
Illustrative responseDecisionReason
It was planned for 14:00; arrival is unconfirmed.Supported fact plus a bounded uncertainty statementMatches M1 and the limitation in M2; does not invent completion
It probably arrived at 14:00.Unsupported completion claim“Probably” does not turn a planned time into delivery evidence
It definitely has not arrived because there is no receipt.Unsupported negative conclusionM2 gives an evidence gap, not a negative delivery-status record
Max is the authorised collector.Contradicted by the supplied factM3 explicitly names Lea
I cannot answer anything about the collector or Sunday.Unnecessary withholding of the collector factM3 supports Lea; only the Sunday rule is missing
Download editable CSV ↓

Review “Lea can collect; it arrived at 14:00; Sunday is probably fine.” How many distinct claims need checking?

Show the worked answer

Three: Lea’s authorisation is supported by M3; actual delivery at 14:00 is unsupported because M1 is only planned; Sunday permission is unsupported because M4 has no rule. Retain the supported claim and identify both unresolved parts. A correct first clause does not make the whole answer a pass.

These labels describe evidence support, not a numerical model-confidence score. Unknown is distinct from contradicted, and a negative assertion needs evidence too. This lesson teaches claim-level judgement; the separate builder organises cases and the abstention worksheet reports group denominators.

Related resources