fix: one schema lookup for a whole design, and larger graph batches
The target architect exhausted its 45 tool calls on a first attempt in three of five runs: 31 schema lookups and 29 graph_put calls for one design. Each time it carried on from its working graph, at a cost of four to ten minutes. - terraform_schema takes resource_types, a list, and outlines every argument and block each accepts, by name, in one reply. - The architect is told to make that one call once the Elements are decided, and to build the graph in a handful of calls of ten to twenty items. Case 04 then completed at 100% for the first time, the architect finishing in one attempt with 11 lookups and 13 graph calls. Plans record the run and correct three step statuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -33,17 +33,17 @@
|
||||
| 14 | Provider-owned contracts | C | M | planned |
|
||||
| 15 | Group Executions by Customer case | added | S | complete |
|
||||
| 16 | Cache Terraform providers between checks | added | S | complete |
|
||||
| 17 | A working copy for files and for the graph | added | M | built; case 01 ran in 18 minutes against 60, with no model timeout; the architect's lookups then tuned, unrun |
|
||||
| 17 | A working copy for files and for the graph | added | M | complete; in every run since 2026-10-07 |
|
||||
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | complete; the description was kept on Check-only Revisions in runs of 2026-10-07 to 09 |
|
||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
|
||||
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
|
||||
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
|
||||
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
|
||||
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
|
||||
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; a hole found by a run and closed; not rerun |
|
||||
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
@@ -18,3 +18,11 @@ Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `test
|
||||
|
||||
After the run of 2026-10-07 16:59 UTC: the target architect made 39 lookups for 8 distinct resource types and exhausted its budget. `find` now takes several names separated by commas, the tool's description says the plain lookup lists every argument and block, and `agents/target-architect.md` asks for one lookup per distinct resource type (budget 45). An outline longer than 8,000 characters is cut with a note to use `block` or `find`; 7 of the provider's 1,370 resource types are that long, `google_container_cluster` the longest.
|
||||
|
||||
After the runs of 2026-10-08: the target architect still exhausted its 45 tool calls on its first attempt in three of five runs. In the last it made 31 schema lookups and 29 `graph_put` calls for one design. Each exhausted attempt cost four to ten minutes before the next one carried on from the working graph. Changed on 2026-10-09:
|
||||
|
||||
- `terraform_schema` takes `resource_types`, a list (or several names in `resource_type`), and returns an outline of each: every argument and block it accepts, by name, required ones starred, with no descriptions. Up to 20 types a call, 4,000 characters a type. The 11 resource types of that design come back in one reply of 14,600 characters.
|
||||
- `agents/target-architect.md` asks for that one call once the Elements are decided, and for `graph_put` in a handful of calls of ten to twenty items each, where it said "a few items per call".
|
||||
- The schema review at submission (step 24) already checks every setting a Check names, so a lookup the architect skips is still caught.
|
||||
|
||||
Run on 2026-10-09 (case 04, `eval/results/20261009T083440Z/`): the target architect finished its first attempt in 314 seconds without exhausting its tool calls, making 11 schema lookups and 13 `graph_put` calls over the whole run, against 31 and 29 the run before. It still looked types up more than once, so the instruction is only partly followed.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user