Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-11.md
Jonathan Boniface 654370bb46 feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:32:57 +01:00

2.2 KiB

Step 11 — Implementation Detail

Built on 2026-10-07 as skills/google-cloud-iac-patterns/. The name carries the Cloud provider because the content does; the provider bundle assigns it to the IaC author (role_skills.iac-author), so another provider brings its own patterns Skill with no code change.

  1. SKILL.md says how to use the patterns and lists one reference per product. assets/versions.tf pins hashicorp/google 8.6.0, the version providers/gcp.yaml pins (step 18).
  2. Thirteen references, each a short list of what goes wrong and one self-contained pattern: enabling APIs, IAM, Cloud Run service, Cloud Run job, Pub/Sub, Cloud Storage, Secret Manager, Cloud KMS, Cloud SQL, VPC networking, Cloud Scheduler, Artifact Registry, Monitoring and Logging.
  3. Seeded from the 2026-10-07 run's lesson candidates. All 13 Terraform refusals were startup_cpu_boost or cpu_idle in the wrong block or on a Cloud Run job. The service reference says which block each setting sits in; the job reference lists what a job does not have.

Differs from the plan: the plan's acceptance is a fall in IaC author Revisions or tokens on the three cases. No evaluation has run, so that is unmeasured. What is verified is that the patterns are correct.

Tests: tests/test_iac_patterns.py runs terraform validate on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in SKILL.md, and checks the Skill and the bundle pin the same version.

Not done: Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from terraform_schema and the documentation.

Added on 2026-10-08: references/load-balancer.md, a fourteenth pattern: an external Application Load Balancer in front of Cloud Run with a TLS policy, a firewall policy, Identity-Aware Proxy and an HTTP redirect. Cases 02, 03 and 04 all used a load balancer with no pattern, and one run put ssl_policy on a backend service, which has no such setting; the reference says which link of the chain each setting lives on. It validates against the pinned provider.