diff --git a/.agents/plans/iter-04-learning-and-skills/MAP.md b/.agents/plans/iter-04-learning-and-skills/MAP.md index c9ec9f7..17ed329 100644 --- a/.agents/plans/iter-04-learning-and-skills/MAP.md +++ b/.agents/plans/iter-04-learning-and-skills/MAP.md @@ -33,17 +33,17 @@ | 14 | Provider-owned contracts | C | M | planned | | 15 | Group Executions by Customer case | added | S | complete | | 16 | Cache Terraform providers between checks | added | S | complete | -| 17 | A working copy for files and for the graph | added | M | built; case 01 ran in 18 minutes against 60, with no model timeout; the architect's lookups then tuned, unrun | +| 17 | A working copy for files and for the graph | added | M | complete; in every run since 2026-10-07 | | 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider | | 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built | | 20 | A model timeout per role | added | S | built | -| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun | +| 21 | A Deliverable stands while what it was written from is unchanged | added | S | complete; the description was kept on Check-only Revisions in runs of 2026-10-07 to 09 | | 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision | | 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout | | 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check | | 25 | An Execution that ends with warnings keeps its files | added | S | complete | | 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar | -| 27 | A user sees a Deliverable before it is first submitted | added | S | built; a hole found by a run and closed; not rerun | +| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse | Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability. diff --git a/.agents/plans/iter-04-learning-and-skills/steps/step-18.md b/.agents/plans/iter-04-learning-and-skills/steps/step-18.md index c1a5971..9591a15 100644 --- a/.agents/plans/iter-04-learning-and-skills/steps/step-18.md +++ b/.agents/plans/iter-04-learning-and-skills/steps/step-18.md @@ -18,3 +18,11 @@ Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `test After the run of 2026-10-07 16:59 UTC: the target architect made 39 lookups for 8 distinct resource types and exhausted its budget. `find` now takes several names separated by commas, the tool's description says the plain lookup lists every argument and block, and `agents/target-architect.md` asks for one lookup per distinct resource type (budget 45). An outline longer than 8,000 characters is cut with a note to use `block` or `find`; 7 of the provider's 1,370 resource types are that long, `google_container_cluster` the longest. +After the runs of 2026-10-08: the target architect still exhausted its 45 tool calls on its first attempt in three of five runs. In the last it made 31 schema lookups and 29 `graph_put` calls for one design. Each exhausted attempt cost four to ten minutes before the next one carried on from the working graph. Changed on 2026-10-09: + +- `terraform_schema` takes `resource_types`, a list (or several names in `resource_type`), and returns an outline of each: every argument and block it accepts, by name, required ones starred, with no descriptions. Up to 20 types a call, 4,000 characters a type. The 11 resource types of that design come back in one reply of 14,600 characters. +- `agents/target-architect.md` asks for that one call once the Elements are decided, and for `graph_put` in a handful of calls of ten to twenty items each, where it said "a few items per call". +- The schema review at submission (step 24) already checks every setting a Check names, so a lookup the architect skips is still caught. + +Run on 2026-10-09 (case 04, `eval/results/20261009T083440Z/`): the target architect finished its first attempt in 314 seconds without exhausting its tool calls, making 11 schema lookups and 13 `graph_put` calls over the whole run, against 31 and 29 the run before. It still looked types up more than once, so the instruction is only partly followed. + diff --git a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md index c8f78fd..35b607a 100644 --- a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md +++ b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md @@ -210,3 +210,13 @@ To verify two changes made after the batches began: the pillar checklist review - **`specialists_ask_before_submitting` failed again**, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (`drafts_shown`). Not rerun. - **`requirements_name_no_products` failed** for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here. +## Run of 2026-10-09, 08:34 UTC (`eval/results/20261009T083440Z/`): case 04 + +To verify the batched schema lookup (iteration 4 step 18) and the corrected confirm-first rule (step 27). Completed, 100% (12 of 12 checks), 36 minutes, 4.25 million tokens, one Revision of the Target architecture for an addition validation proposed (an alert on database storage); the description was kept and the code fixed in 40 seconds. + +- The target architect did not exhaust its tool calls: one first attempt, then its draft confirmed by the user. +- `specialists_ask_before_submitting` passed: every Specialist put its draft to the user before submitting. The rule refused nothing, so it was not needed in this run. +- `requirements_name_no_products` passed. + +This is the first run of case 04 at 100%. + diff --git a/agents/target-architect.md b/agents/target-architect.md index 6f34cd7..4a097c7 100644 --- a/agents/target-architect.md +++ b/agents/target-architect.md @@ -8,11 +8,11 @@ includes: [specialist-protocol] You produce the **Target architecture** Deliverable: the solution as one explicit graph of Elements, Relationships and Checks. It is agreed before any infrastructure code is written, and the diagram, the description, the code and the validation are all derived from it. - Load the requirements-chain Skill for the graph's format and how to derive it. -- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out. +- Build the graph with `graph_put` in a handful of calls of ten to twenty items each: all the Elements, then all the Relationships, then the Checks in batches. A call for every Element, or for every few Checks, uses up your tool calls before the graph is finished; a single call with everything is a reply long enough to time out. - Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks. - Give every connection between Elements a Relationship. - Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. Write no Check for an entry under `outside_code`: it names something infrastructure code cannot show, and a Check for it could never be met. A condition left without a Check is found by validation and sent back to you. -- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once. +- Name only settings the Element's resource type has. Once you have decided the Elements, and when `terraform_schema` is available, call it once with `resource_types` listing every distinct resource type in the graph: the reply outlines every argument and block each type accepts, and you write all the Checks from it. That one call is all the looking up a design needs; never look types or settings up one at a time. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once. - Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job. - When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design. - When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`. diff --git a/app/tools/local.py b/app/tools/local.py index cdd4f46..b064b03 100644 --- a/app/tools/local.py +++ b/app/tools/local.py @@ -17,7 +17,14 @@ import json import os import re import tempfile -from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping +from collections.abc import ( + Awaitable, + Callable, + Iterable, + Mapping, + MutableMapping, + Sequence, +) from dataclasses import dataclass, field from pathlib import Path, PurePosixPath from typing import Any @@ -25,6 +32,8 @@ from typing import Any from app.adk.runtime import FILES_SCHEMA, ToolResult, ToolSpec, files_from_argument _MAX_OUTPUT = 8_000 +_MAX_OUTLINE = 4_000 # one resource type, when several are looked up together +_MAX_TYPES = 20 # resource types in one lookup _TERRAFORM_SUFFIXES = (".tf", ".tfvars", ".tf.json") _VARIABLE_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$") _MERMAID_TYPES = ( @@ -172,12 +181,52 @@ class TerraformRunner: ) async def schema( - self, resource_type: str, block: str = "", find: str = "" + self, + resource_type: str = "", + block: str = "", + find: str = "", + resource_types: Sequence[str] = (), ) -> ToolResult: - """What a resource type accepts, from the pinned provider's own schema.""" + """What a resource type accepts, from the pinned provider's own schema. + + Several resource types at once come back as outlines: every argument and + block each accepts, by name. That is one call for a whole design, where one + call per type or per setting is what used up a Specialist's budget. + """ loaded = await self._load_schemas() if isinstance(loaded, ToolResult): return loaded + wanted = list( + dict.fromkeys( + name + for name in [*resource_types, *re.split(r"[,\s]+", resource_type)] + if name + ) + ) + if not wanted: + return ToolResult("Refused: name at least one resource type.") + if len(wanted) > 1: + parts = [] + for name in wanted[:_MAX_TYPES]: + if name not in loaded: + close = difflib.get_close_matches(name, loaded, n=3, cutoff=0.6) + hint = f" Close matches: {', '.join(close)}." if close else "" + parts.append( + f"{name}: not a resource type of {self._provider_labels()}." + f"{hint}" + ) + continue + parts.append(_outline_only(name, loaded[name][1])) + if len(wanted) > _MAX_TYPES: + parts.append( + f"[{len(wanted) - _MAX_TYPES} more not shown: ask for at most " + f"{_MAX_TYPES} at a time]" + ) + return ToolResult( + f"{self._provider_labels()}. * marks a required argument.\n\n" + + "\n\n".join(parts) + ) + resource_type = wanted[0] if resource_type not in loaded: close = difflib.get_close_matches(resource_type, loaded, n=5, cutoff=0.6) hint = f" Close matches: {', '.join(close)}." if close else "" @@ -415,6 +464,31 @@ def _render_schema(heading: str, node: Mapping[str, Any]) -> str: return "\n".join(lines) +def _outline_only(resource_type: str, root: Mapping[str, Any]) -> str: + """One resource type as names alone: its arguments on a line, then every block + with its arguments. Enough to see whether a setting exists and where.""" + lines = [resource_type, " " + (_names_line(root) or "-")] + + def outline(parent: Mapping[str, Any], depth: int) -> None: + for name, nested in sorted((parent.get("block_types") or {}).items()): + inner = nested.get("block") or {} + lines.append(f"{' ' * depth}{name} {{ }}: {_names_line(inner) or '-'}") + outline(inner, depth + 1) + + outline(root, 1) + text = "\n".join(lines) + if len(text) > _MAX_OUTLINE: + text = text[:_MAX_OUTLINE] + "\n [cut short: look this type up alone]" + return text + + +def _names_line(node: Mapping[str, Any]) -> str: + return ", ".join( + f"{name}*" if attribute.get("required") else name + for name, attribute in _arguments(node).items() + ) + + def _find_in_schema(title: str, root: Mapping[str, Any], wanted: str) -> str: """Every argument or block whose name contains `wanted`, with where it sits.""" needle = wanted.lower() @@ -512,10 +586,11 @@ LOCAL_TOOLS: dict[str, tuple[str, str]] = { "general", ( "Look up what a Terraform resource type accepts, from the provider's own " - "schema. With `resource_type` alone: every argument and block it has, " - "which is enough to see whether a setting exists. `block` gives one " - "block in full. `find` says where named settings sit, or that the " - "resource type has none." + "schema. With `resource_types`, a list: an outline of every argument " + "and block each one accepts, in one call. With `resource_type` alone: " + "the same for one type, with descriptions. `block` gives one block in " + "full. `find` says where named settings sit, or that the resource type " + "has none." ), ), } @@ -586,6 +661,14 @@ def local_tools( "type": "string", "description": "For example google_cloud_run_v2_service", }, + "resource_types": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Several resource types in one call: each comes back " + "as an outline of every argument and block it accepts" + ), + }, "block": { "type": "string", "description": "A nested block, such as template.containers", @@ -598,7 +681,7 @@ def local_tools( ), }, }, - "required": ["resource_type"], + "required": [], }, runner.schema, tier="general", diff --git a/app/tools/workspace.py b/app/tools/workspace.py index 55794ee..dccf95b 100644 --- a/app/tools/workspace.py +++ b/app/tools/workspace.py @@ -251,7 +251,7 @@ def graph_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]: ToolSpec( "graph_put", "Add items to your working graph, or replace the ones with the same id. " - "Call it several times with a few items each. Your working graph is " + "Put ten to twenty items in a call, not one at a time. Your working graph is " "attached as target-architecture.json when you submit.", _schema(lists, []), graph_put, diff --git a/skills/iac-validation/SKILL.md b/skills/iac-validation/SKILL.md index 119da15..8c33cda 100644 --- a/skills/iac-validation/SKILL.md +++ b/skills/iac-validation/SKILL.md @@ -20,7 +20,7 @@ The files are your working copy: the files you wrote with `write_file`, or, when - `terraform_fmt()`: formats your working copy in place and names the files it changed. - `terraform_validate()`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments. - `terraform_plan(variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires. -- `terraform_schema(resource_type, block, find)`: what a resource type accepts, from the provider's own schema. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none; give several names separated by commas to check them in one call. +- `terraform_schema(resource_type, block, find, resource_types)`: what a resource type accepts, from the provider's own schema. With `resource_types`, a list, it outlines every argument and block of each in one call; use that when you need several. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none; give several names separated by commas to check them in one call. ## When writing deployment guidance diff --git a/tests/tools/test_local.py b/tests/tools/test_local.py index eab6c72..ffb0db5 100644 --- a/tests/tools/test_local.py +++ b/tests/tools/test_local.py @@ -249,6 +249,29 @@ async def test_schema_suggests_close_resource_types_and_is_loaded_once(monkeypat assert "apply" not in [arg for command in commands for arg in command] +async def test_several_resource_types_are_outlined_in_one_call(monkeypatch): + """One lookup for a whole design: looking types up one by one used up the + target architect's tool calls in three of five runs.""" + runner, commands = schema_runner(monkeypatch) + + result = await runner.schema(resource_types=["cloud_service", "cloud_job", "nope"]) + by_commas = await runner.schema(resource_type="cloud_service, cloud_job") + + assert result.message.startswith("example/cloud 1.2.3. * marks a required") + assert "cloud_service\n labels, name*" in result.message + assert " template { }: cpu_boost" in result.message + assert "uri" not in result.message # read-only: code cannot set it + assert "nope: not a resource type of example/cloud 1.2.3." in result.message + assert "cloud_job\n" in by_commas.message and "template { }" in by_commas.message + assert [command[1] for command in commands] == ["init", "providers"] + + +async def test_a_lookup_needs_a_resource_type(monkeypatch): + runner, _ = schema_runner(monkeypatch) + + assert (await runner.schema()).message.startswith("Refused: name at least one") + + async def test_schema_is_unavailable_without_a_pinned_provider(): result = await TerraformRunner().schema("cloud_service")