Define the local evaluation task
A synthetic drafting task requires exactly one occurrence of a supplied literal placeholder string. Count exact case-sensitive occurrences in the invented candidate draft; escaped HTML text and a fabricated resolved ID are not the same literal. The checker makes no substitutions.
- Keep the literal target unchanged
- Count exact non-overlapping occurrences
- Require exactly one occurrence
An empty target is an invalid test definition.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Required token remains once | {"literal":"{CASE_ID}","draft":"Open case {CASE_ID} after review."} | {"count":1,"retainedOnce":true} | The draft teaches a placeholder rather than inventing a case value. |
| Placeholder replaced by invented ID | {"literal":"{CASE_ID}","draft":"Open case C99 after review."} | {"count":0,"retainedOnce":false} | C99 does not preserve the requested literal. |
| Placeholder repeated twice | {"literal":"{CASE_ID}","draft":"{CASE_ID} follows {CASE_ID}."} | {"count":2,"retainedOnce":false} | Literal preservation alone is insufficient: this contract requires exactly one occurrence. |
Work the repeated-placeholder case. Which output is required, and why?
Show the worked answer
{"count":2,"retainedOnce":false} — Literal preservation alone is insufficient: this contract requires exactly one occurrence.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
if(!x.literal.length)return {error:'empty-literal'};let count=0,pos=0;
while((pos=x.draft.indexOf(x.literal,pos))!==-1){count++;pos+=x.literal.length;}
return {count,retainedOnce:count===1};
}
const cases = [
{
"id": "literal-once",
"input": {
"literal": "{CASE_ID}",
"draft": "Open case {CASE_ID} after review."
},
"expected": {
"count": 1,
"retainedOnce": true
}
},
{
"id": "invented-value",
"input": {
"literal": "{CASE_ID}",
"draft": "Open case C99 after review."
},
"expected": {
"count": 0,
"retainedOnce": false
}
},
{
"id": "repeated-placeholder",
"input": {
"literal": "{CASE_ID}",
"draft": "{CASE_ID} follows {CASE_ID}."
},
"expected": {
"count": 2,
"retainedOnce": false
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep draft, literal target and occurrence requirement in the dataset. Literal matching does not grade the rest of the instruction’s meaning or make it ready to send.