docs: record the first runs with and without Terraform modules

Case 03's request once each way, both at 100%. Opted in, a module was used
for private Cloud SQL only and 14 Checks were met through its inputs. The
difference in time and cost between the two runs is their Revisions, not
the modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 22:47:09 +01:00
parent ef37f59bad
commit 10d23a5f91
3 changed files with 30 additions and 3 deletions

View File

@@ -13,7 +13,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 5 | `iter-05-jev-decisions/` | the router, intake, the reviews, pillar selection, Skill routing, the guardrail, meaning-aware change summaries and an assessment of every Execution are built (steps 2–5, 8, 10–12, 14, 15, 19), the newest ran in cases 05 and 06 on 2026-10-09; a lighter model for a first attempt was built, measured and left off (6); context selection and compaction were decided against on the numbers (7, 9); the cloud-read guard and readings as a node are planned (13, 18) | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
| 8 | `iter-08-terraform-modules/` | steps 1–5 built on 2026-10-09; no case has run with them; steps 6 and 7 wait on Azure and AWS provider bundles | Infrastructure code is plain resources by default. An Execution may opt into published Terraform modules: only from registry namespaces the provider bundle lists, at one exact version, with a lookup of what a module accepts and a Skill of four Google modules tested against the pinned provider. |
| 8 | `iter-08-terraform-modules/` | steps 1–5 built and run once each way on 2026-10-09, both at 100%; steps 6 and 7 wait on Azure and AWS provider bundles | Infrastructure code is plain resources by default. An Execution may opt into published Terraform modules: only from registry namespaces the provider bundle lists, at one exact version, with a lookup of what a module accepts and a Skill of four Google modules tested against the pinned provider. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.

View File

@@ -2,7 +2,7 @@
**Iteration:** 8, Terraform modules
**Intent:** Let generated infrastructure code call a published Terraform module where one replaces several resources that must be wired together, without losing the Requirements chain or opening a way for arbitrary code to be downloaded.
**Status:** steps 1 to 5 built on 2026-10-09; a case run each way is outstanding; steps 6 and 7 wait on those Cloud providers
**Status:** steps 1 to 5 built and run once each way on 2026-10-09; steps 6 and 7 wait on those Cloud providers
**This directory covers:** steps 1 to 7
## The owner's request (2026-10-09)
@@ -36,7 +36,7 @@ The owner's decision, 2026-10-09: **plain resources by default, with an opt-in t
| 3 | A Skill with the catalogue and the rule for when to use one | built: `skills/google-cloud-iac-modules/`, four references, each validated by a test |
| 4 | Checks through a module | built as guidance: `implementation.json` names the module input; the validator confirms what the input sets |
| 5 | Opt-in per Execution, plain resources by default | built: `terraform_modules` on a start, `/generate`, A2A metadata and an evaluation case; recorded in `execution.json` |
| 5a | Run a case each way and see what the agents do with it | not run |
| 5a | Run a case each way and see what the agents do with it | run once each way on 2026-10-09, both at 100%; a module was used for private Cloud SQL only |
| 6 | Azure: `aztfm` | waits on an Azure provider bundle |
| 7 | AWS: `terraform-aws-modules` | waits on an AWS provider bundle |

View File

@@ -60,3 +60,30 @@ Reference routing (iteration 5 step 8) does not choose within the modules Skill,
- A module version in the catalogue goes stale; nothing flags a newer one.
Tests: `tests/tools/test_modules.py` (24), `tests/test_iac_modules.py` (6), `tests/execution/test_terraform_modules.py` (7).
## Run on 2026-10-09
Case 03's request, once each way, side by side (`20261009T210931Z` plain, `20261009T210935Z` opted in as `case-07-approved-modules`). Both completed at 100%.
| | Plain resources | Modules opted in |
|---|---|---|
| Minutes | 35 | 22 |
| Tokens | 5.66 million | 3.08 million |
| Cost, about | $5.50 | $3.17 |
| Revisions of the Target architecture | 3 | 1 |
| Checks, all passed | 77 | 54 |
| Module calls / resources / lines of Terraform | 0 / 35 / 555 | 2 / 25 / 448 |
| Design quality score | 0.82 | 0.62 |
**The difference in time and cost is the Revisions, not the modules.** The two runs produced different designs before any code was written: 77 Checks against 54, three Revisions against one. One run each way cannot say modules are cheaper.
What the opted-in run does show:
- The IaC author used a module for one thing, private Cloud SQL (`sql-db//modules/postgresql` with `//modules/private_service_access`, 28.3.0), and wrote the network, load balancer, Cloud Run, storage, secrets, IAM and monitoring as plain resources. A VPC with one subnet did not get the network module.
- 14 of 54 Checks are met through module inputs, each recorded as `module.cloud_sql.<input>`; the validator passed all 14 citing the input's value.
- `terraform_module` was called four times by the author and once by the validator. No module was refused, and the code validated.
- For a Check met by leaving an input unset, the validator's evidence is that the module falls back to its default. That rests on the module's documentation.
The plain run was given neither the Skill nor the tool, and wrote no module.
The design quality score of 0.62 is below the bar of 0.67. It is scored on the design recommendations, written before any code.