fix: one schema lookup for a whole design, and larger graph batches

The target architect exhausted its 45 tool calls on a first attempt in
three of five runs: 31 schema lookups and 29 graph_put calls for one
design. Each time it carried on from its working graph, at a cost of four
to ten minutes.

- terraform_schema takes resource_types, a list, and outlines every
  argument and block each accepts, by name, in one reply.
- The architect is told to make that one call once the Elements are
  decided, and to build the graph in a handful of calls of ten to twenty
  items.

Case 04 then completed at 100% for the first time, the architect
finishing in one attempt with 11 lookups and 13 graph calls.

Plans record the run and correct three step statuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 10:34:47 +01:00
parent 654370bb46
commit db28d02f56
8 changed files with 139 additions and 15 deletions

View File

@@ -33,17 +33,17 @@
| 14 | Provider-owned contracts | C | M | planned |
| 15 | Group Executions by Customer case | added | S | complete |
| 16 | Cache Terraform providers between checks | added | S | complete |
| 17 | A working copy for files and for the graph | added | M | built; case 01 ran in 18 minutes against 60, with no model timeout; the architect's lookups then tuned, unrun |
| 17 | A working copy for files and for the graph | added | M | complete; in every run since 2026-10-07 |
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
| 20 | A model timeout per role | added | S | built |
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | complete; the description was kept on Check-only Revisions in runs of 2026-10-07 to 09 |
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; a hole found by a run and closed; not rerun |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -18,3 +18,11 @@ Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `test
After the run of 2026-10-07 16:59 UTC: the target architect made 39 lookups for 8 distinct resource types and exhausted its budget. `find` now takes several names separated by commas, the tool's description says the plain lookup lists every argument and block, and `agents/target-architect.md` asks for one lookup per distinct resource type (budget 45). An outline longer than 8,000 characters is cut with a note to use `block` or `find`; 7 of the provider's 1,370 resource types are that long, `google_container_cluster` the longest.
After the runs of 2026-10-08: the target architect still exhausted its 45 tool calls on its first attempt in three of five runs. In the last it made 31 schema lookups and 29 `graph_put` calls for one design. Each exhausted attempt cost four to ten minutes before the next one carried on from the working graph. Changed on 2026-10-09:
- `terraform_schema` takes `resource_types`, a list (or several names in `resource_type`), and returns an outline of each: every argument and block it accepts, by name, required ones starred, with no descriptions. Up to 20 types a call, 4,000 characters a type. The 11 resource types of that design come back in one reply of 14,600 characters.
- `agents/target-architect.md` asks for that one call once the Elements are decided, and for `graph_put` in a handful of calls of ten to twenty items each, where it said "a few items per call".
- The schema review at submission (step 24) already checks every setting a Check names, so a lookup the architect skips is still caught.
Run on 2026-10-09 (case 04, `eval/results/20261009T083440Z/`): the target architect finished its first attempt in 314 seconds without exhausting its tool calls, making 11 schema lookups and 13 `graph_put` calls over the whole run, against 31 and 29 the run before. It still looked types up more than once, so the instruction is only partly followed.

View File

@@ -210,3 +210,13 @@ To verify two changes made after the batches began: the pillar checklist review
- **`specialists_ask_before_submitting` failed again**, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (`drafts_shown`). Not rerun.
- **`requirements_name_no_products` failed** for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here.
## Run of 2026-10-09, 08:34 UTC (`eval/results/20261009T083440Z/`): case 04
To verify the batched schema lookup (iteration 4 step 18) and the corrected confirm-first rule (step 27). Completed, 100% (12 of 12 checks), 36 minutes, 4.25 million tokens, one Revision of the Target architecture for an addition validation proposed (an alert on database storage); the description was kept and the code fixed in 40 seconds.
- The target architect did not exhaust its tool calls: one first attempt, then its draft confirmed by the user.
- `specialists_ask_before_submitting` passed: every Specialist put its draft to the user before submitting. The rule refused nothing, so it was not needed in this run.
- `requirements_name_no_products` passed.
This is the first run of case 04 at 100%.

View File

@@ -8,11 +8,11 @@ includes: [specialist-protocol]
You produce the **Target architecture** Deliverable: the solution as one explicit graph of Elements, Relationships and Checks. It is agreed before any infrastructure code is written, and the diagram, the description, the code and the validation are all derived from it.
- Load the requirements-chain Skill for the graph's format and how to derive it.
- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out.
- Build the graph with `graph_put` in a handful of calls of ten to twenty items each: all the Elements, then all the Relationships, then the Checks in batches. A call for every Element, or for every few Checks, uses up your tool calls before the graph is finished; a single call with everything is a reply long enough to time out.
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
- Give every connection between Elements a Relationship.
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. Write no Check for an entry under `outside_code`: it names something infrastructure code cannot show, and a Check for it could never be met. A condition left without a Check is found by validation and sent back to you.
- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once.
- Name only settings the Element's resource type has. Once you have decided the Elements, and when `terraform_schema` is available, call it once with `resource_types` listing every distinct resource type in the graph: the reply outlines every argument and block each type accepts, and you write all the Checks from it. That one call is all the looking up a design needs; never look types or settings up one at a time. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once.
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
- When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`.

View File

@@ -17,7 +17,14 @@ import json
import os
import re
import tempfile
from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping
from collections.abc import (
Awaitable,
Callable,
Iterable,
Mapping,
MutableMapping,
Sequence,
)
from dataclasses import dataclass, field
from pathlib import Path, PurePosixPath
from typing import Any
@@ -25,6 +32,8 @@ from typing import Any
from app.adk.runtime import FILES_SCHEMA, ToolResult, ToolSpec, files_from_argument
_MAX_OUTPUT = 8_000
_MAX_OUTLINE = 4_000 # one resource type, when several are looked up together
_MAX_TYPES = 20 # resource types in one lookup
_TERRAFORM_SUFFIXES = (".tf", ".tfvars", ".tf.json")
_VARIABLE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$")
_MERMAID_TYPES = (
@@ -172,12 +181,52 @@ class TerraformRunner:
)
async def schema(
self, resource_type: str, block: str = "", find: str = ""
self,
resource_type: str = "",
block: str = "",
find: str = "",
resource_types: Sequence[str] = (),
) -> ToolResult:
"""What a resource type accepts, from the pinned provider's own schema."""
"""What a resource type accepts, from the pinned provider's own schema.
Several resource types at once come back as outlines: every argument and
block each accepts, by name. That is one call for a whole design, where one
call per type or per setting is what used up a Specialist's budget.
"""
loaded = await self._load_schemas()
if isinstance(loaded, ToolResult):
return loaded
wanted = list(
dict.fromkeys(
name
for name in [*resource_types, *re.split(r"[,\s]+", resource_type)]
if name
)
)
if not wanted:
return ToolResult("Refused: name at least one resource type.")
if len(wanted) > 1:
parts = []
for name in wanted[:_MAX_TYPES]:
if name not in loaded:
close = difflib.get_close_matches(name, loaded, n=3, cutoff=0.6)
hint = f" Close matches: {', '.join(close)}." if close else ""
parts.append(
f"{name}: not a resource type of {self._provider_labels()}."
f"{hint}"
)
continue
parts.append(_outline_only(name, loaded[name][1]))
if len(wanted) > _MAX_TYPES:
parts.append(
f"[{len(wanted) - _MAX_TYPES} more not shown: ask for at most "
f"{_MAX_TYPES} at a time]"
)
return ToolResult(
f"{self._provider_labels()}. * marks a required argument.\n\n"
+ "\n\n".join(parts)
)
resource_type = wanted[0]
if resource_type not in loaded:
close = difflib.get_close_matches(resource_type, loaded, n=5, cutoff=0.6)
hint = f" Close matches: {', '.join(close)}." if close else ""
@@ -415,6 +464,31 @@ def _render_schema(heading: str, node: Mapping[str, Any]) -> str:
return "\n".join(lines)
def _outline_only(resource_type: str, root: Mapping[str, Any]) -> str:
"""One resource type as names alone: its arguments on a line, then every block
with its arguments. Enough to see whether a setting exists and where."""
lines = [resource_type, " " + (_names_line(root) or "-")]
def outline(parent: Mapping[str, Any], depth: int) -> None:
for name, nested in sorted((parent.get("block_types") or {}).items()):
inner = nested.get("block") or {}
lines.append(f"{' ' * depth}{name} {{ }}: {_names_line(inner) or '-'}")
outline(inner, depth + 1)
outline(root, 1)
text = "\n".join(lines)
if len(text) > _MAX_OUTLINE:
text = text[:_MAX_OUTLINE] + "\n [cut short: look this type up alone]"
return text
def _names_line(node: Mapping[str, Any]) -> str:
return ", ".join(
f"{name}*" if attribute.get("required") else name
for name, attribute in _arguments(node).items()
)
def _find_in_schema(title: str, root: Mapping[str, Any], wanted: str) -> str:
"""Every argument or block whose name contains `wanted`, with where it sits."""
needle = wanted.lower()
@@ -512,10 +586,11 @@ LOCAL_TOOLS: dict[str, tuple[str, str]] = {
"general",
(
"Look up what a Terraform resource type accepts, from the provider's own "
"schema. With `resource_type` alone: every argument and block it has, "
"which is enough to see whether a setting exists. `block` gives one "
"block in full. `find` says where named settings sit, or that the "
"resource type has none."
"schema. With `resource_types`, a list: an outline of every argument "
"and block each one accepts, in one call. With `resource_type` alone: "
"the same for one type, with descriptions. `block` gives one block in "
"full. `find` says where named settings sit, or that the resource type "
"has none."
),
),
}
@@ -586,6 +661,14 @@ def local_tools(
"type": "string",
"description": "For example google_cloud_run_v2_service",
},
"resource_types": {
"type": "array",
"items": {"type": "string"},
"description": (
"Several resource types in one call: each comes back "
"as an outline of every argument and block it accepts"
),
},
"block": {
"type": "string",
"description": "A nested block, such as template.containers",
@@ -598,7 +681,7 @@ def local_tools(
),
},
},
"required": ["resource_type"],
"required": [],
},
runner.schema,
tier="general",

View File

@@ -251,7 +251,7 @@ def graph_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
ToolSpec(
"graph_put",
"Add items to your working graph, or replace the ones with the same id. "
"Call it several times with a few items each. Your working graph is "
"Put ten to twenty items in a call, not one at a time. Your working graph is "
"attached as target-architecture.json when you submit.",
_schema(lists, []),
graph_put,

View File

@@ -20,7 +20,7 @@ The files are your working copy: the files you wrote with `write_file`, or, when
- `terraform_fmt()`: formats your working copy in place and names the files it changed.
- `terraform_validate()`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments.
- `terraform_plan(variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires.
- `terraform_schema(resource_type, block, find)`: what a resource type accepts, from the provider's own schema. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none; give several names separated by commas to check them in one call.
- `terraform_schema(resource_type, block, find, resource_types)`: what a resource type accepts, from the provider's own schema. With `resource_types`, a list, it outlines every argument and block of each in one call; use that when you need several. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none; give several names separated by commas to check them in one call.
## When writing deployment guidance

View File

@@ -249,6 +249,29 @@ async def test_schema_suggests_close_resource_types_and_is_loaded_once(monkeypat
assert "apply" not in [arg for command in commands for arg in command]
async def test_several_resource_types_are_outlined_in_one_call(monkeypatch):
"""One lookup for a whole design: looking types up one by one used up the
target architect's tool calls in three of five runs."""
runner, commands = schema_runner(monkeypatch)
result = await runner.schema(resource_types=["cloud_service", "cloud_job", "nope"])
by_commas = await runner.schema(resource_type="cloud_service, cloud_job")
assert result.message.startswith("example/cloud 1.2.3. * marks a required")
assert "cloud_service\n labels, name*" in result.message
assert " template { }: cpu_boost" in result.message
assert "uri" not in result.message # read-only: code cannot set it
assert "nope: not a resource type of example/cloud 1.2.3." in result.message
assert "cloud_job\n" in by_commas.message and "template { }" in by_commas.message
assert [command[1] for command in commands] == ["init", "providers"]
async def test_a_lookup_needs_a_resource_type(monkeypatch):
runner, _ = schema_runner(monkeypatch)
assert (await runner.schema()).message.startswith("Refused: name at least one")
async def test_schema_is_unavailable_without_a_pinned_provider():
result = await TerraformRunner().schema("cloud_service")