Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/DETAIL.md
Jonathan Boniface c9f59be274 docs: accept ADR-0004, lessons and overlays arrive as Skills
Accepted by the owner on 2026-10-08 with its six decisions as written:
a person approves every lesson; a lesson must be shown to help by
evaluation; lessons are per Cloud provider; overlay Skills come only from
configured directories; each Execution records the Skills it loaded; and
lesson candidates are client data kept out of the repository.

- CONTEXT.md gains Lesson candidate, Lesson and Overlay Skill.
- Iteration 4 step 9 is complete; steps 10 and 12 name the decisions they
  must meet and are unblocked. Neither is built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:05:03 +01:00

207 lines
22 KiB
Markdown

# Plan Detail
Every step ends with the full suite passing. Steps that change agent behaviour (3, 6, 10, 11, 12, 13) also end with an evaluation run recorded in their step file, compared against the baseline in `iter-02-solution-architect-agent/steps/step-8.md`.
## Step 1: Structured validator findings
- **Report ref:** enhancement 3
- **Capability:** Validation findings as data the engine can route, count and learn from.
- **Rationale:** The validator reports findings as free markdown. The Orchestrator reads them once to decide what to reopen; nothing else can use them. They are the richest learning signal in an Execution.
- **Where:** `app/workflows/deliverables.py` (add `findings.json` to the `validation_results` attachments), `agents/validator.md`, `skills/iac-validation/SKILL.md` (the finding format), `app/tools/execution.py` (validate the attachment's shape at submission).
- **Acceptance criteria:** A `validation_results` submission without a parseable `findings.json` is refused with the reason. Each finding carries the Deliverable kind it traces to, a severity, the evidence and the rule. An empty list is valid and means a pass.
## Step 2: Carry the draft through `needs_input`
- **Report ref:** enhancement 9
- **Capability:** The user receives the draft they approved.
- **Rationale:** The Specialist protocol asks a Specialist to confirm its draft with the user before submitting. `needs_input` carries only questions and Specialists are stateless per delegation, so the approved draft is lost and written again on the next run.
- **Where:** `needs_input` in `app/tools/execution.py` (optional `draft` and attachments), `NeedsInput` in `app/adk/runtime.py`, `_outcome_record` and `_specialist_input` in `app/nodes/execution_nodes.py` (store the draft, return it in context), `app/states/execution_state.py`, `agents/partials/specialist-protocol.md`.
- **Acceptance criteria:** A Specialist restarted after the user approves its draft receives that draft in its context. A test shows the submitted content equals the approved draft when the fake runtime submits what it was given.
## Step 3: Reuse pillar advice across Revisions
- **Report ref:** enhancement 10
- **Capability:** A Revision of the design recommendations re-consults only the pillars its feedback touches.
- **Rationale:** Pillar advice is never stored. In case 01, two Revisions re-ran all six advisors each time: 18 advisor runs for one Deliverable.
- **Where:** `app/tools/pillars.py` (the tool already accepts `pillars`), the Execution state for stored advice, `agents/design-advisor.md` (on a Revision, name the affected pillars).
- **Acceptance criteria:** On a second attempt with stored advice, only the named pillars run and the tool returns stored advice for the rest. A changed brief for a pillar invalidates that pillar's stored advice.
## Step 4: Single source for Deliverable contracts
- **Report ref:** enhancement 11
- **Capability:** Required headings and attachments declared once, and contracts for the Deliverables that have none.
- **Rationale:** Required headings appear in each role file and again in `DeliverableSpec.sections`. `technical_decomposition`, `architecture_description` and the diagram's content accept any non-empty text.
- **Where:** `app/workflows/deliverables.py`, `_specialist_input` in `app/nodes/execution_nodes.py` (pass the contract in context), `agents/*.md` (remove the "Required headings" lines), `agents/partials/specialist-protocol.md`.
- **Acceptance criteria:** No role file lists required headings. Each Specialist's context carries its contract. The decomposition requires a component table naming the requirements each component serves.
## Step 5: Cap parallel Specialists
- **Report ref:** enhancement 14
- **Capability:** A bound on concurrent Specialist runs per Execution.
- **Rationale:** The spec's Risks section calls for a default cap of 4 against model rate limits. `orchestrator_turn` sends every delegated Deliverable at once, and the pillar fan-out adds six more.
- **Where:** `app/nodes/execution_nodes.py`, `app/tools/pillars.py`, a `MAX_PARALLEL_SPECIALISTS` setting in `app/config.py`.
- **Acceptance criteria:** A test with the fake runtime shows at most the configured number of Specialist runs in flight. The pillar fan-out counts toward the same cap.
## Step 6: Orchestrator step order
- **Report ref:** findings by agent, orchestrator row
- **Capability:** A failed validation goes back for Revision before the user is asked to approve it.
- **Rationale:** In `agents/orchestrator.md`, "A Phase is complete" (step 3) precedes "Validation found problems" (step 5), and the Orchestrator takes the first step that applies.
- **Where:** `agents/orchestrator.md`.
- **Acceptance criteria:** The validation-failure step sits above the Approval step. An evaluation run with a failing validation shows a reopen before any Approval request for the validation Phase.
## Step 7: Capture lesson candidates
- **Report ref:** enhancement 4
- **Capability:** A durable record, per Execution, of what went wrong and what fixed it.
- **Rationale:** The signals exist and are discarded. Revision feedback lives in the Ledger inside one checkpoint. Tool refusals (`Not submitted`, `Refused`, `terraform validate failed`) reach only the Python log through `REJECTION_PREFIXES` in `app/adk/adk_runtime.py`.
- **Where:** `app/recorder.py` (a `record_lesson_candidates` method on `ExecutionRecorder`), `finalize` and `end_with_warnings` in `app/nodes/execution_nodes.py`, the runtime for refusals.
- **Acceptance criteria:** A finished Execution writes one record holding each reopen with its reason and actor, each finding from step 1, each tool refusal with role and tool, each user change request and each exhausted budget. The record names the Execution, the Deliverable kind and the Skill versions loaded. It holds no secrets and no discovered cloud data.
## Step 8: Evaluation that can see improvement
- **Report ref:** enhancement 6
- **Capability:** An evaluation whose score moves when a Skill gets better or worse.
- **Rationale:** The 8 checks are structural and both scored runs reached 1.0. The loop's proof step has nothing to measure.
- **Where:** `eval/scoring.py`, `eval/solution_eval.py`, `eval/datasets/solution_cases.json`, `tests/eval/`.
- **Acceptance criteria:** New checks cover requirement-to-component traceability, coverage of each pillar Skill's checklist, and a graded rubric for recommendation quality. One interactive case checks that a Specialist asks before it submits. Revisions, seconds and tokens per case have recorded thresholds, and a run that exceeds one is reported as a regression. `case-02-high-scale-ingestion` has a scored run.
## Step 9: ADR: the lessons loop and overlay Skills
- **Report ref:** sequencing note
- **Capability:** A recorded decision on how Skills change and how they are chosen per Execution.
- **Rationale:** Steps 10 and 12 change which Skills an agent loads and who may edit them. That is an architectural decision, and ADR-0002 does not cover it.
- **Where:** `docs/adr/0004-*.md`.
- **Decisions the owner must make:** whether lessons need human review before they take effect (the plan assumes yes); where lessons Skills live and who owns them; whether an Execution may name overlay Skills, and who may supply them.
- **Acceptance criteria:** The ADR is accepted and steps 10 and 12 are updated to match it.
## Step 10: Lessons Skill and lesson curator
- **Report ref:** enhancement 5
- **Capability:** Each Execution can improve the next, under human review.
- **Rationale:** `eval/optimizer.py` is the only learning code. It maps four keywords to iteration 1 Skill names that the engine never loads.
- **Design:** The loop is capture (step 7), distil, review, prove, publish, retire.
- Distil: an offline `lesson-curator` role groups candidates across Executions and proposes lessons. It runs outside any Execution.
- Review: each proposal is an issue file under `.scratch/lessons/issues/` with `Status: needs-triage`.
- Prove: the evaluation from step 8 runs with and without the change.
- Publish: lessons land in a lessons Skill under `skills/`, one reference file per Deliverable kind; each lesson has its rule, reason, source Execution and date. The bundle's `role_skills` lists it.
- Retire: at each vendor sync, remove lessons the upstream Skill now covers.
- **Where:** `skills/<lessons-skill>/`, `agents/lesson-curator.md`, a curator entry point beside `eval/solution_eval.py`, `providers/gcp.yaml`, `scripts/sync-vendored-skills.sh`.
- **Acceptance criteria:** Vendored Skills stay byte-identical. No lesson reaches a Skill without an approved issue file. Each Execution records the lessons Skill version it loaded. One real lesson has gone through the whole loop with its evaluation comparison recorded.
- **From ADR-0004 (accepted 2026-10-08):** the lessons Skill is per Cloud provider and listed in that provider's bundle; a lesson is kept only if the evaluation with and without it holds quality and lowers Revisions, time or tokens; a lesson carries no client detail, and lesson candidates stay out of the repository.
## Step 11: IaC patterns Skill
- **Report ref:** enhancement 7
- **Capability:** The IaC author starts from known-good infrastructure code.
- **Rationale:** The IaC author is the most expensive role: 812 thousand tokens, 23 requests, 4 runs and 3 Revisions in case 03. It writes from a blank page each time.
- **Where:** `skills/<iac-patterns-skill>/` with one reference per product, `providers/gcp.yaml` (`role_skills.iac-author`).
- **Acceptance criteria:** On the three cases, quality checks hold and IaC author Revisions or tokens fall against the step 8 baseline.
## Step 12: Overlay Skills and an organisation standards Skill
- **Report ref:** enhancement 8
- **Capability:** New standards and clients need no code change.
- **Rationale:** Skills are fixed at start-up: `AdkAgentRuntime` loads every Skill once, and `role_skills` is static per provider. Only 2 of the 9 Skills are our own, and both describe tools.
- **Where:** `app/adk/adk_runtime.py`, `app/adk/roles.py`, `app/providers.py`, `ExecutionEngine.start` in `app/agent.py`, `app/a2a_server.py`, a standards Skill under `skills/`.
- **Acceptance criteria:** An Execution can name overlay Skills at start; each passes the same Agent Skills validation as bundled Skills, and an unknown or invalid one refuses the start. An organisation standards Skill (naming, tagging, approved products) is loaded by the roles the bundle lists.
- **From ADR-0004 (accepted 2026-10-08):** overlay Skills come only from directories the deployment configures; a caller names one by its Skill name and cannot upload one; the Execution records each overlay it loaded with a content hash. Skills are resolved per Execution, not once at start-up.
## Step 13: Trim Specialist context
- **Report ref:** enhancement 12
- **Capability:** Each Specialist receives what its Deliverable needs.
- **Rationale:** `inputs_of` gives a Specialist every Deliverable of every earlier Phase in full. The describer used 513 thousand tokens over 12 requests.
- **Where:** `inputs_of` in `app/workflows/deliverables.py`, `_specialist_input` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Direct Dependencies arrive in full; other earlier Deliverables arrive as summaries, with `read_deliverable`-style access to the full text on demand. Quality checks hold and describer tokens fall against the step 8 baseline.
## Step 14: Provider-owned contracts
- **Report ref:** enhancement 13
- **Capability:** A second Cloud provider is configuration, as ADR-0002 intends.
- **Rationale:** The six pillar headings, the eight solution-guide sections and the Terraform attachment names are hard-coded in `app/workflows/deliverables.py`. Other providers' frameworks name and count pillars differently. The bundle's pillar keys already differ from the headings (`performance-optimization` against `Performance efficiency`).
- **Where:** `providers/gcp.yaml`, `app/providers.py`, `app/workflows/deliverables.py`, `app/presentation.py`, `agents/iac-author.md`, `agents/validator.md`, `agents/design-advisor.md`.
- **Timing:** Build when a second Cloud provider is planned. ADR-0002 says no provider interface exists in code until then.
- **Acceptance criteria:** Pillar headings come from `pillar_skills`, guide sections from the bundle's `output_template`, and the IaC tool with its file contract from the bundle. `grep -rni "terraform" app/workflows app/nodes app/states` is empty, and no pillar or guide-section name appears as a literal in `app/workflows/deliverables.py`.
## Step 15: Group Executions by Customer case
- **Report ref:** none; requested by the owner on 2026-10-06
- **Capability:** Everything under `deliverables/` is grouped by Customer case, so a case ties each request to the artefacts generated for it.
- **Rationale:** Outputs were keyed only by Execution id. Nothing on disk said which customer workload an Execution belonged to or what it had been asked.
- **Where:** `app/states/execution_state.py` (the case id rule and the paths), `ExecutionEngine.start` in `app/agent.py`, `_write_outputs` in `app/nodes/execution_nodes.py`, `app/main.py`, `app/a2a_server.py`, `eval/solution_eval.py`, `CONTEXT.md`.
- **Acceptance criteria:** Every Execution's files are under `deliverables/cases/<customer_case>/executions/<execution_id>/`. Each has a manifest naming the case, the request and every file. Each case has an index of its Executions and their requests. A case id that is not a safe slug is refused before anything runs. A Customer case never feeds one Execution into another.
## Step 16: Cache Terraform providers between checks
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
- **Capability:** A provider is downloaded once and reused by every later `terraform validate` and `plan`.
- **Rationale:** Each check runs `terraform init` in a new scratch directory, so every call downloaded its providers again. The setting `TF_PLUGIN_CACHE_DIR` existed and was unset by default.
- **Where:** `TerraformRunner` in `app/tools/local.py`, `app/config.py`, `.env.example`.
- **Acceptance criteria:** The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.
## Step 17: A working copy for files and for the graph
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
- **Capability:** A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
- **Rationale:** In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every `terraform_fmt` and `terraform_validate` call carried every file, `terraform_fmt` returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
- **Where:** `app/tools/workspace.py` (new), `specialist_tools` in `app/tools/execution.py`, `app/tools/local.py`, `app/tools/toolkit.py`, `DeliverableSpec.working_copy` in `app/workflows/deliverables.py`, `render_tables` in `app/workflows/target_architecture.py`, `agents/target-architect.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
- **Acceptance criteria:** The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.
## Step 18: Provider schema lookup and one pinned provider version
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** An agent can ask what a resource type accepts, and get the provider's own answer.
- **Rationale:** All 13 Terraform refusals in the 2026-10-07 run were one mistake: `startup_cpu_boost` and `cpu_idle` in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
- **Where:** `TerraformRunner.schema` in `app/tools/local.py`, `terraform.required_providers` in `providers/gcp.yaml`, `app/providers.py`, the role files of the target architect, the IaC author and the validator.
- **Acceptance criteria:** `terraform_schema` lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.
## Step 19: A Check the code cannot meet goes back to the Target architecture
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
- **Rationale:** Check K-30 of the 2026-10-07 run asked for `startup_cpu_boost` on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
- **Where:** `check_implementation` and `unimplementable_checks` in `app/workflows/chain.py`, the playbook in `app/nodes/router.py`, `_orchestrator_context` in `app/nodes/execution_nodes.py`, `agents/orchestrator.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** `implementation.json` may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.
## Step 20: A model timeout per role
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A role whose replies are long is given longer for one model request than the rest.
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
## Step 21: A Deliverable stands while what it was written from is unchanged
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** A Revision upstream does not rewrite a Deliverable it leaves true.
- **Rationale:** In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
- **Where:** `DeliverableSpec.written_from` in `app/workflows/deliverables.py`, `graph_structure` in `app/workflows/chain.py`, `specialist_task` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.
## Step 22: Validation reports every gap in one pass
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** One validation finds the gaps in the Target architecture, not one validation per gap.
- **Rationale:** In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
## Step 23: Validation recorded a few verdicts at a time
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
- **Capability:** The validator's work survives a slow or lost model reply.
- **Rationale:** In the run cancelled on 2026-10-07 the validator's model request timed out five times at 180 seconds, about 15 minutes, each time while writing every verdict in one reply. Nothing it had judged was kept. The target architect had the same fault until iteration 4 step 17.
- **Where:** `verdict_tools` in `app/tools/workspace.py`, `specialist_tools` in `app/tools/execution.py`, `providers/gcp.yaml`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
- **Acceptance criteria:** Verdicts can be recorded over several calls and are attached as `verdicts.json` on submission. Verdicts recorded on the same code are kept across attempts; verdicts on code that has since changed are dropped. The validator has its own model timeout. On the cases, the validator's model timeouts fall to none.
## Step 24: The design checked against the provider's schema, in code
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
- **Capability:** A Check that names a setting its resource type lacks is caught before any code is written.
- **Rationale:** In the cancelled run two Revisions of the Target architecture were for Checks no code could meet: `ssl_policy` on a backend service, and a deny rule on a service account. The target architect had the schema lookup and did not use it for them. The earlier K-30 (`startup_cpu_boost` on a Cloud Run job) was the same fault.
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.