Define the local evaluation task
A synthetic robustness pair declares exactly which semantic fields may differ between baseline and variant. Compare the union of their own keys; adding/removing an unapproved field is a change too. Keep permitted and unexpected changes separately. This includes evidence-preservation tests: a wording-only variant must keep its required evidence ID set.
- Declare permitted changed fields
- Compare all own keys and canonical fixture values
- Retain unexpected changes before model testing
Unexpected changes invalidate the test pair, not the model candidate.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Wording changed, facts retained | {"base":{"wording":"Read code K1","code":"K1","quantity":3},"variant":{"wording":"Read code K1","code":"K1","quantity":3},"allowed":["wording"]} | {"allowedChanges":["wording"],"unexpected":[],"valid":true} | Only the declared wording factor changed. |
| Quantity changed as well | {"base":{"wording":"Read K1","code":"K1","quantity":3},"variant":{"wording":"Read K1","code":"K1","quantity":4},"allowed":["wording"]} | {"allowedChanges":["wording"],"unexpected":["quantity"],"valid":false} | Quantity changed the expected business answer, invalidating the controlled pair. |
| Required evidence removed | {"base":{"wording":"Read K1","evidence":["D1"]},"variant":{"wording":"Please read K1","evidence":[]},"allowed":["wording"]} | {"allowedChanges":["wording"],"unexpected":["evidence"],"valid":false} | A wording test that removes D1 also changes source availability and must be rejected as a matched pair. |
Work the required-evidence-removed case. Which output is required, and why?
Show the worked answer
{"allowedChanges":["wording"],"unexpected":["evidence"],"valid":false} — A wording test that removes D1 also changes source availability and must be rejected as a matched pair.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
const keys=[...new Set([...Object.keys(x.base),...Object.keys(x.variant)])].sort();const changed=keys.filter(k=>Object.hasOwn(x.base,k)!==Object.hasOwn(x.variant,k)||JSON.stringify(x.base[k])!==JSON.stringify(x.variant[k]));
return {allowedChanges:changed.filter(k=>x.allowed.includes(k)),unexpected:changed.filter(k=>!x.allowed.includes(k)),valid:changed.every(k=>x.allowed.includes(k))};
}
const cases = [
{
"id": "spacing-only",
"input": {
"base": {
"wording": "Read code K1",
"code": "K1",
"quantity": 3
},
"variant": {
"wording": "Read code K1",
"code": "K1",
"quantity": 3
},
"allowed": [
"wording"
]
},
"expected": {
"allowedChanges": [
"wording"
],
"unexpected": [],
"valid": true
}
},
{
"id": "quantity-confound",
"input": {
"base": {
"wording": "Read K1",
"code": "K1",
"quantity": 3
},
"variant": {
"wording": "Read K1",
"code": "K1",
"quantity": 4
},
"allowed": [
"wording"
]
},
"expected": {
"allowedChanges": [
"wording"
],
"unexpected": [
"quantity"
],
"valid": false
}
},
{
"id": "required-evidence-removed",
"input": {
"base": {
"wording": "Read K1",
"evidence": [
"D1"
]
},
"variant": {
"wording": "Please read K1",
"evidence": []
},
"allowed": [
"wording"
]
},
"expected": {
"allowedChanges": [
"wording"
],
"unexpected": [
"evidence"
],
"valid": false
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
This fixture uses canonical JSON values in a fixed key/array order. If equivalent serialisations are allowed, replace the comparator explicitly; keep evidence availability fixed when testing wording alone.