fixed multile issues
Some checks failed
validation / verify (push) Failing after 8s

This commit is contained in:
2026-10-07 17:33:36 +01:00
parent 4c4d44ef87
commit 5bb22d984e
76 changed files with 5799 additions and 179 deletions

View File

@@ -137,3 +137,35 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Rationale:** Each check runs `terraform init` in a new scratch directory, so every call downloaded its providers again. The setting `TF_PLUGIN_CACHE_DIR` existed and was unset by default.
- **Where:** `TerraformRunner` in `app/tools/local.py`, `app/config.py`, `.env.example`.
- **Acceptance criteria:** The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.
## Step 17: A working copy for files and for the graph
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
- **Capability:** A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
- **Rationale:** In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every `terraform_fmt` and `terraform_validate` call carried every file, `terraform_fmt` returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
- **Where:** `app/tools/workspace.py` (new), `specialist_tools` in `app/tools/execution.py`, `app/tools/local.py`, `app/tools/toolkit.py`, `DeliverableSpec.working_copy` in `app/workflows/deliverables.py`, `render_tables` in `app/workflows/target_architecture.py`, `agents/target-architect.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
- **Acceptance criteria:** The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.
## Step 18: Provider schema lookup and one pinned provider version
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** An agent can ask what a resource type accepts, and get the provider's own answer.
- **Rationale:** All 13 Terraform refusals in the 2026-10-07 run were one mistake: `startup_cpu_boost` and `cpu_idle` in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
- **Where:** `TerraformRunner.schema` in `app/tools/local.py`, `terraform.required_providers` in `providers/gcp.yaml`, `app/providers.py`, the role files of the target architect, the IaC author and the validator.
- **Acceptance criteria:** `terraform_schema` lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.
## Step 19: A Check the code cannot meet goes back to the Target architecture
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
- **Rationale:** Check K-30 of the 2026-10-07 run asked for `startup_cpu_boost` on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
- **Where:** `check_implementation` and `unimplementable_checks` in `app/workflows/chain.py`, the playbook in `app/nodes/router.py`, `_orchestrator_context` in `app/nodes/execution_nodes.py`, `agents/orchestrator.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** `implementation.json` may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.
## Step 20: A model timeout per role
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A role whose replies are long is given longer for one model request than the rest.
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.

View File

@@ -2,9 +2,9 @@
**Iteration:** 4, learning and Skills
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
**Status:** steps 1 to 8 and 15 built; no evaluation run has completed on them (see `steps/step-8.md`); step 9 drafted and awaiting the owner; steps 10 to 14 planned. Steps 1 to 6 are in commit `7ee49ab`; 7, 8 and 15 are uncommitted.
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; no evaluation run has completed on any of them (see `steps/step-8.md`), so every "tokens fall" or "Revisions fall" criterion is unmeasured; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
**This directory covers:** steps 1 to 16
**This directory covers:** steps 1 to 20
## Sources of truth
@@ -27,12 +27,16 @@
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
| 9 | ADR: the lessons loop and overlay Skills | B | S | drafted as `docs/adr/0004-lessons-and-overlays-arrive-as-skills.md`, status proposed; awaits the owner |
| 10 | Lessons Skill and lesson curator | B | L | planned |
| 11 | IaC patterns Skill | B | M | planned |
| 11 | IaC patterns Skill | B | M | built; every pattern validates; effect on Revisions and tokens unmeasured |
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
| 13 | Trim Specialist context | C | M | planned |
| 13 | Trim Specialist context | C | M | built; effect on tokens unmeasured |
| 14 | Provider-owned contracts | C | M | planned |
| 15 | Group Executions by Customer case | added | S | complete |
| 16 | Cache Terraform providers between checks | added | S | complete |
| 17 | A working copy for files and for the graph | added | M | built; effect on time and tokens unmeasured |
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
| 20 | A model timeout per role | added | S | built |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
@@ -49,5 +53,7 @@ Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that
- Step 14: depends on step 4
- Step 15: no dependencies (added on 2026-10-06 at the owner's request)
- Step 16: no dependencies (added on 2026-10-07 at the owner's request)
- Steps 17 to 20: no dependencies (added on 2026-10-07 at the owner's request, from the causes of the 60 minute run recorded in `../iter-06-target-architecture-graph/steps/`)
- Step 11 uses step 18's pinned provider version; step 19 uses step 18's lookup to find the Checks it reports
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.

View File

@@ -0,0 +1,13 @@
# Step 11 — Implementation Detail
Built on 2026-10-07 as `skills/google-cloud-iac-patterns/`. The name carries the Cloud provider because the content does; the provider bundle assigns it to the IaC author (`role_skills.iac-author`), so another provider brings its own patterns Skill with no code change.
1. `SKILL.md` says how to use the patterns and lists one reference per product. `assets/versions.tf` pins `hashicorp/google` 8.6.0, the version `providers/gcp.yaml` pins (step 18).
2. Thirteen references, each a short list of what goes wrong and one self-contained pattern: enabling APIs, IAM, Cloud Run service, Cloud Run job, Pub/Sub, Cloud Storage, Secret Manager, Cloud KMS, Cloud SQL, VPC networking, Cloud Scheduler, Artifact Registry, Monitoring and Logging.
3. Seeded from the 2026-10-07 run's lesson candidates. All 13 Terraform refusals were `startup_cpu_boost` or `cpu_idle` in the wrong block or on a Cloud Run job. The service reference says which block each setting sits in; the job reference lists what a job does not have.
Differs from the plan: the plan's acceptance is a fall in IaC author Revisions or tokens on the three cases. **No evaluation has run, so that is unmeasured.** What is verified is that the patterns are correct.
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.

View File

@@ -0,0 +1,14 @@
# Step 13 — Implementation Detail
Built on 2026-10-07.
1. `DeliverableSpec.reads_in_full` names the earlier Deliverables a Specialist reads whole. Left unset it is the Deliverables it depends on.
2. `dependencies_for(kind, deliverables)` gives every other input as its data file alone: `requirements.json`, `components.json` and so on. Those ids are all the Requirements chain checks need, so submission checks are unchanged.
3. Two overrides. The IaC author reads nothing whole: the code is written from `target-architecture.json`. The packager reads everything whole, since the guide is assembled from every Deliverable's prose.
4. A Deliverable given as data can be read whole with `read_deliverable(kind)`, offered only when something was trimmed.
Differs from the plan: the plan said "summaries". The data file is used instead. It already exists, is exact, and costs no model call to produce.
**Unmeasured:** the plan's acceptance is that describer tokens fall against the step 8 baseline. No evaluation has run.
Tests: `tests/execution/test_working_copy.py`, "each Specialist reads what it works from".

View File

@@ -0,0 +1,21 @@
# Step 17 — Implementation Detail
Built on 2026-10-07. `app/tools/workspace.py` keeps a working copy in the Deliverable's working memory, which the Execution already checkpoints and returns on every attempt.
Files (`working["files"]`), for the Deliverable whose `working_copy` is its own kind (deployment guidance):
1. `write_file`, `edit_file` (replaces the one place `old` matches, and refuses when it matches none or several), `read_file` (one file, or the list) and `delete_file`.
2. `terraform_fmt`, `terraform_validate` and `terraform_plan` run on the Terraform files in the working copy and take no `files` argument. `terraform_fmt` formats in place and returns the names it changed. Files passed anyway are written to the working copy first, so a model that ignores the instruction still works.
3. `submit_deliverable` attaches the working copy; a file passed as an attachment replaces its copy. The submitted files become the working copy, so a Revision starts from them, and the task context lists them under `working_copy`.
The validator's `working_copy` is deployment guidance: its tools are given a fresh copy of the submitted files on every attempt, it has no write tools, and nothing from the copy is attached to its Deliverable.
Graph (`working["graph"]`), for the Target architecture:
4. `graph_put` adds items or replaces those with the same id, `graph_remove` removes by id, `graph_read` returns the graph. The submitted graph becomes the working graph.
5. `submit_deliverable` attaches the working graph when no `target-architecture.json` is passed.
6. `render_tables` writes the Elements, Relationships, Checks and not-applicable tables from the graph, and the engine appends them to the content on submission. The contract's sections are now `Design rationale` alone. This closes the duplicate output iteration 6 left: the architect no longer writes the graph twice.
**Unmeasured:** no evaluation has run, so the fall in requests, tokens and time is expected, not shown. The mechanism is what is tested.
Tests: `tests/execution/test_working_copy.py` (21), including an Execution in which the validator validates the submitted code with no files passed.

View File

@@ -0,0 +1,17 @@
# Step 18 — Implementation Detail
Built on 2026-10-07.
1. `providers/gcp.yaml` pins `terraform.required_providers.google` to `hashicorp/google` 8.6.0. `load_provider` requires a `source` and an exact `version`.
2. `TerraformRunner.schema(resource_type, block, find)` writes a versions file for the pinned providers in a scratch directory, runs `terraform init -backend=false` and `terraform providers schema -json` once per process, and keeps the resource schemas in memory. Measured here: 0.6 seconds and a 9.7 MB reply for the Google provider, from the plugin cache.
3. With `resource_type` alone it lists the arguments code may set (read-only ones are left out) and outlines every block. With `block` it gives one block's arguments with descriptions. With `find` it says where a setting sits, or that the resource type has none. An unknown resource type gets close matches.
4. The tool is `terraform_schema`, tier general, given to the target architect, the IaC author and the validator.
Checked against the real provider on 2026-10-07:
- `find="startup_cpu_boost"` on `google_cloud_run_v2_job`: "has no argument or block named like 'startup_cpu_boost'. It cannot be set on this resource type."
- the same on `google_cloud_run_v2_service`: `template.containers.resources.startup_cpu_boost (bool, optional)`.
Not done: generated code is not forced onto the pinned version. An agent that writes its own `required_providers` with another version is validated against that version. The patterns Skill gives it the pinned `versions.tf` to use.
Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `tests/test_providers.py` for the pin.

View File

@@ -0,0 +1,12 @@
# Step 19 — Implementation Detail
Built on 2026-10-07.
1. An `implementation.json` entry may be `{"check": "K-7", "unimplementable": "<what the resource type lacks>"}` in place of `file` and `address`. It accounts for the Check; an empty reason does not.
2. `unimplementable_checks` reads those entries. The router reads them as soon as deployment guidance is submitted, and a new playbook group, "the code cannot meet a Check", sits before "validation decided" and "a Phase is complete". Its action reopens the Target architecture with each Check's id and reason. That returns the code to draft, so the group stops applying.
3. A model Orchestrator is given the same list as `unimplementable_checks` in its context, and `agents/orchestrator.md` has the step.
4. The validator is told to confirm with `terraform_schema` before failing a Check for a missing setting, and to give not verifiable when the resource type has no such setting. Not verifiable already routes to the Target architecture.
Past the Revision limit the reopen is refused as any other is: the user decides, or a pre-approved Execution ends with warnings.
Tests: `tests/execution/test_requirements_chain.py` (two) and `tests/execution/test_router.py`, "a Check the code cannot meet goes back to the graph unvalidated".

View File

@@ -0,0 +1,7 @@
# Step 20 — Implementation Detail
Built on 2026-10-07. `models.timeout_seconds` in the provider bundle maps a role to the seconds one model request may take; `RoleSpec.model_timeout_seconds` carries it and `AdkAgentRuntime` uses it in place of `MODEL_TIMEOUT_SECONDS`.
`providers/gcp.yaml` sets the target architect to 420 and the IaC author and the packager to 300. These are starting values, not measurements. With step 17 the architect's replies are short, so its 420 should rarely be reached.
Tests: `tests/agents/test_adk_runtime.py`, `tests/agents/test_roles.py`, `tests/test_providers.py`.