Track exposure at question level
| Question | Source snapshot | Expected answer and boundary | Use | Reason |
|---|---|---|---|---|
| D-1: “What are Saturday hours?” | Synthetic sheet: Saturday 10:00–14:00; no admission or booking policy | Saturday 10:00–14:00; no admission or booking guarantee | Development example | Used while editing the instructions |
| R-1: “Is 13:45 on Saturday within the stated opening interval?” | Synthetic sheet: Saturday 10:00–14:00; comparison includes opening and excludes closing; admission/booking rules not supplied | Yes: 10:00 ≤ 13:45 < 14:00. This establishes time within the interval only, not admission permission or an available booking. | Reserved review question | Interval comparison rather than reciting hours; not used for tuning |
| D-2: “Give Saturday opening times.” | Synthetic sheet: Saturday 10:00–14:00; no admission or booking policy | Saturday 10:00–14:00; no admission or booking guarantee | Development family with D-1 | Near paraphrase; do not present it as a wholly new behaviour |
If the actual question is “May I arrive at 13:45?”, the expected answer must separate known hours from unknown admission permission: 13:45 is within the interval, but the sheet does not establish permission, required visit length or booking availability. A test for admission needs a separate supplied admission policy.
The split is a workflow for these original examples, not a statistical guarantee. A reserved question can use known source facts without being used to tune the answer wording. Keep question exposure separate from whether the underlying source document is available.
Move an exposed review case into the development record
R-1 fails, and the team adds its exact question and answer to the assistant’s instructions. Can a repeat of R-1 still be called an unseen review?
Show the worked answer
No. Keep R-1 as a regression case and record its new development exposure. Reserve another genuinely different question for the later review; do not erase the original failure.
A differently worded synonym is useful for language variation but is not automatically a new failure mode. Describe the split, question families and exposure dates when reporting an executed evaluation. None of the questions here has been run against a model.