Define the local evaluation task
A synthetic assistant may propose only exact tool names from a supplied local registry. The example checks plan names and returns unknown names; it executes no tool. Names are case-sensitive and inherited object names are not registered tools. A near spelling must not be silently corrected.
- Read proposed operation names
- Compare exact own registry entries
- Retain unknown names without execution
Unknown names stop name validation; no replacement tool is inferred.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Near name | {"registry":{"read_case":"read-only"},"plan":["read_cases"]} | {"unknown":["read_cases"],"valid":false} | The extra s identifies an unavailable tool, not a registered alias. |
| Exact registered tool | {"registry":{"read_case":"read-only"},"plan":["read_case"]} | {"unknown":[],"valid":true} | Exact registry membership passes name resolution only. |
| Inherited object name | {"registry":{"read_case":"read-only"},"plan":["toString"]} | {"unknown":["toString"],"valid":false} | A JavaScript inherited property is not an own registry entry. |
Work the inherited-name case. Which output is required, and why?
Show the worked answer
{"unknown":["toString"],"valid":false} — A JavaScript inherited property is not an own registry entry.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){const unknown=x.plan.filter(name=>!Object.hasOwn(x.registry,name));return {unknown,valid:unknown.length===0};}
const cases = [
{
"id": "near-spelling",
"input": {
"registry": {
"read_case": "read-only"
},
"plan": [
"read_cases"
]
},
"expected": {
"unknown": [
"read_cases"
],
"valid": false
}
},
{
"id": "exact-tool",
"input": {
"registry": {
"read_case": "read-only"
},
"plan": [
"read_case"
]
},
"expected": {
"unknown": [],
"valid": true
}
},
{
"id": "inherited-name",
"input": {
"registry": {
"read_case": "read-only"
},
"plan": [
"toString"
]
},
"expected": {
"unknown": [
"toString"
],
"valid": false
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Keep registry revision and proposed call names beside the fixture. Name validity does not establish valid arguments, read permission or successful execution.