Define the local evaluation task
A synthetic generated plan is an ordered list of parallel batches. A call may consume results only from completed earlier batches; another call in its own batch is unavailable even if listed first. Call IDs are unique. Dependencies absent from all earlier batches remain unavailable. Check the plan without executing any calls.
- Track only completed earlier batches
- Check each call’s result dependencies
- Add current IDs after the batch check
Unavailable references or repeated call IDs require plan repair.
Use the filled cases
| Case | Supplied input | Expected structured output | Decision and reason |
|---|---|---|---|
| Dependency in same batch | {"batches":[[{"id":"S1","needs":[]},{"id":"S2","needs":["S1"]}]]} | {"errors":[{"call":"S2","error":"unavailable","dependency":"S1"}],"valid":false} | S1 has not finished merely because it is listed first in a parallel batch. |
| Result from earlier batch | {"batches":[[{"id":"S1","needs":[]}],[{"id":"S2","needs":["S1"]}]]} | {"errors":[],"valid":true} | S1 is available before the second batch starts. |
| Dependency never produced | {"batches":[[{"id":"S2","needs":["S9"]}]]} | {"errors":[{"call":"S2","error":"unavailable","dependency":"S9"}],"valid":false} | No preceding batch produces S9; do not invent its output. |
Work the unknown-dependency case. Which output is required, and why?
Show the worked answer
{"errors":[{"call":"S2","error":"unavailable","dependency":"S9"}],"valid":false} — No preceding batch produces S9; do not invent its output.
Run the self-contained evaluator
import assert from 'node:assert/strict';
function evaluate(x){
const done=new Set(),all=new Set(),errors=[];
for(const batch of x.batches){for(const c of batch){if(all.has(c.id))errors.push({call:c.id,error:'duplicate'});all.add(c.id);for(const d of c.needs)if(!done.has(d))errors.push({call:c.id,error:'unavailable',dependency:d});}for(const c of batch)done.add(c.id);}
return {errors,valid:errors.length===0};
}
const cases = [
{
"id": "same-batch-dependency",
"input": {
"batches": [
[
{
"id": "S1",
"needs": []
},
{
"id": "S2",
"needs": [
"S1"
]
}
]
]
},
"expected": {
"errors": [
{
"call": "S2",
"error": "unavailable",
"dependency": "S1"
}
],
"valid": false
}
},
{
"id": "earlier-batch",
"input": {
"batches": [
[
{
"id": "S1",
"needs": []
}
],
[
{
"id": "S2",
"needs": [
"S1"
]
}
]
]
},
"expected": {
"errors": [],
"valid": true
}
},
{
"id": "unknown-dependency",
"input": {
"batches": [
[
{
"id": "S2",
"needs": [
"S9"
]
}
]
]
},
"expected": {
"errors": [
{
"call": "S2",
"error": "unavailable",
"dependency": "S9"
}
],
"valid": false
}
}
];
for (const c of cases) assert.deepEqual(evaluate(c.input), c.expected, c.id);
console.log(JSON.stringify({checked:cases.length,caseIds:cases.map(c=>c.id)}));
Export each rejected call/dependency pair for plan review. Valid batch ordering says nothing about successful execution, elapsed duration or permission to run a tool.