fix: one schema lookup for a whole design, and larger graph batches

The target architect exhausted its 45 tool calls on a first attempt in
three of five runs: 31 schema lookups and 29 graph_put calls for one
design. Each time it carried on from its working graph, at a cost of four
to ten minutes.

- terraform_schema takes resource_types, a list, and outlines every
  argument and block each accepts, by name, in one reply.
- The architect is told to make that one call once the Elements are
  decided, and to build the graph in a handful of calls of ten to twenty
  items.

Case 04 then completed at 100% for the first time, the architect
finishing in one attempt with 11 lookups and 13 graph calls.

Plans record the run and correct three step statuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 10:34:47 +01:00
parent 654370bb46
commit db28d02f56
8 changed files with 139 additions and 15 deletions

View File

@@ -18,3 +18,11 @@ Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `test
After the run of 2026-10-07 16:59 UTC: the target architect made 39 lookups for 8 distinct resource types and exhausted its budget. `find` now takes several names separated by commas, the tool's description says the plain lookup lists every argument and block, and `agents/target-architect.md` asks for one lookup per distinct resource type (budget 45). An outline longer than 8,000 characters is cut with a note to use `block` or `find`; 7 of the provider's 1,370 resource types are that long, `google_container_cluster` the longest.
After the runs of 2026-10-08: the target architect still exhausted its 45 tool calls on its first attempt in three of five runs. In the last it made 31 schema lookups and 29 `graph_put` calls for one design. Each exhausted attempt cost four to ten minutes before the next one carried on from the working graph. Changed on 2026-10-09:
- `terraform_schema` takes `resource_types`, a list (or several names in `resource_type`), and returns an outline of each: every argument and block it accepts, by name, required ones starred, with no descriptions. Up to 20 types a call, 4,000 characters a type. The 11 resource types of that design come back in one reply of 14,600 characters.
- `agents/target-architect.md` asks for that one call once the Elements are decided, and for `graph_put` in a handful of calls of ten to twenty items each, where it said "a few items per call".
- The schema review at submission (step 24) already checks every setting a Check names, so a lookup the architect skips is still caught.
Run on 2026-10-09 (case 04, `eval/results/20261009T083440Z/`): the target architect finished its first attempt in 314 seconds without exhausting its tool calls, making 11 schema lookups and 13 `graph_put` calls over the whole run, against 31 and 29 the run before. It still looked types up more than once, so the instruction is only partly followed.