Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/DETAIL.md
Jonathan Boniface e9246828b3 feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:59:09 +01:00

14 KiB

Plan Detail

Every step ends with the full suite passing. Steps that change agent behaviour (3, 6, 10, 11, 12, 13) also end with an evaluation run recorded in their step file, compared against the baseline in iter-02-solution-architect-agent/steps/step-8.md.

Step 1: Structured validator findings

  • Report ref: enhancement 3
  • Capability: Validation findings as data the engine can route, count and learn from.
  • Rationale: The validator reports findings as free markdown. The Orchestrator reads them once to decide what to reopen; nothing else can use them. They are the richest learning signal in an Execution.
  • Where: app/workflows/deliverables.py (add findings.json to the validation_results attachments), agents/validator.md, skills/iac-validation/SKILL.md (the finding format), app/tools/execution.py (validate the attachment's shape at submission).
  • Acceptance criteria: A validation_results submission without a parseable findings.json is refused with the reason. Each finding carries the Deliverable kind it traces to, a severity, the evidence and the rule. An empty list is valid and means a pass.

Step 2: Carry the draft through needs_input

  • Report ref: enhancement 9
  • Capability: The user receives the draft they approved.
  • Rationale: The Specialist protocol asks a Specialist to confirm its draft with the user before submitting. needs_input carries only questions and Specialists are stateless per delegation, so the approved draft is lost and written again on the next run.
  • Where: needs_input in app/tools/execution.py (optional draft and attachments), NeedsInput in app/adk/runtime.py, _outcome_record and _specialist_input in app/nodes/execution_nodes.py (store the draft, return it in context), app/states/execution_state.py, agents/partials/specialist-protocol.md.
  • Acceptance criteria: A Specialist restarted after the user approves its draft receives that draft in its context. A test shows the submitted content equals the approved draft when the fake runtime submits what it was given.

Step 3: Reuse pillar advice across Revisions

  • Report ref: enhancement 10
  • Capability: A Revision of the design recommendations re-consults only the pillars its feedback touches.
  • Rationale: Pillar advice is never stored. In case 01, two Revisions re-ran all six advisors each time: 18 advisor runs for one Deliverable.
  • Where: app/tools/pillars.py (the tool already accepts pillars), the Execution state for stored advice, agents/design-advisor.md (on a Revision, name the affected pillars).
  • Acceptance criteria: On a second attempt with stored advice, only the named pillars run and the tool returns stored advice for the rest. A changed brief for a pillar invalidates that pillar's stored advice.

Step 4: Single source for Deliverable contracts

  • Report ref: enhancement 11
  • Capability: Required headings and attachments declared once, and contracts for the Deliverables that have none.
  • Rationale: Required headings appear in each role file and again in DeliverableSpec.sections. technical_decomposition, architecture_description and the diagram's content accept any non-empty text.
  • Where: app/workflows/deliverables.py, _specialist_input in app/nodes/execution_nodes.py (pass the contract in context), agents/*.md (remove the "Required headings" lines), agents/partials/specialist-protocol.md.
  • Acceptance criteria: No role file lists required headings. Each Specialist's context carries its contract. The decomposition requires a component table naming the requirements each component serves.

Step 5: Cap parallel Specialists

  • Report ref: enhancement 14
  • Capability: A bound on concurrent Specialist runs per Execution.
  • Rationale: The spec's Risks section calls for a default cap of 4 against model rate limits. orchestrator_turn sends every delegated Deliverable at once, and the pillar fan-out adds six more.
  • Where: app/nodes/execution_nodes.py, app/tools/pillars.py, a MAX_PARALLEL_SPECIALISTS setting in app/config.py.
  • Acceptance criteria: A test with the fake runtime shows at most the configured number of Specialist runs in flight. The pillar fan-out counts toward the same cap.

Step 6: Orchestrator step order

  • Report ref: findings by agent, orchestrator row
  • Capability: A failed validation goes back for Revision before the user is asked to approve it.
  • Rationale: In agents/orchestrator.md, "A Phase is complete" (step 3) precedes "Validation found problems" (step 5), and the Orchestrator takes the first step that applies.
  • Where: agents/orchestrator.md.
  • Acceptance criteria: The validation-failure step sits above the Approval step. An evaluation run with a failing validation shows a reopen before any Approval request for the validation Phase.

Step 7: Capture lesson candidates

  • Report ref: enhancement 4
  • Capability: A durable record, per Execution, of what went wrong and what fixed it.
  • Rationale: The signals exist and are discarded. Revision feedback lives in the Ledger inside one checkpoint. Tool refusals (Not submitted, Refused, terraform validate failed) reach only the Python log through REJECTION_PREFIXES in app/adk/adk_runtime.py.
  • Where: app/recorder.py (a record_lesson_candidates method on ExecutionRecorder), finalize and end_with_warnings in app/nodes/execution_nodes.py, the runtime for refusals.
  • Acceptance criteria: A finished Execution writes one record holding each reopen with its reason and actor, each finding from step 1, each tool refusal with role and tool, each user change request and each exhausted budget. The record names the Execution, the Deliverable kind and the Skill versions loaded. It holds no secrets and no discovered cloud data.

Step 8: Evaluation that can see improvement

  • Report ref: enhancement 6
  • Capability: An evaluation whose score moves when a Skill gets better or worse.
  • Rationale: The 8 checks are structural and both scored runs reached 1.0. The loop's proof step has nothing to measure.
  • Where: eval/scoring.py, eval/solution_eval.py, eval/datasets/solution_cases.json, tests/eval/.
  • Acceptance criteria: New checks cover requirement-to-component traceability, coverage of each pillar Skill's checklist, and a graded rubric for recommendation quality. One interactive case checks that a Specialist asks before it submits. Revisions, seconds and tokens per case have recorded thresholds, and a run that exceeds one is reported as a regression. case-02-high-scale-ingestion has a scored run.

Step 9: ADR: the lessons loop and overlay Skills

  • Report ref: sequencing note
  • Capability: A recorded decision on how Skills change and how they are chosen per Execution.
  • Rationale: Steps 10 and 12 change which Skills an agent loads and who may edit them. That is an architectural decision, and ADR-0002 does not cover it.
  • Where: docs/adr/0004-*.md.
  • Decisions the owner must make: whether lessons need human review before they take effect (the plan assumes yes); where lessons Skills live and who owns them; whether an Execution may name overlay Skills, and who may supply them.
  • Acceptance criteria: The ADR is accepted and steps 10 and 12 are updated to match it.

Step 10: Lessons Skill and lesson curator

  • Report ref: enhancement 5
  • Capability: Each Execution can improve the next, under human review.
  • Rationale: eval/optimizer.py is the only learning code. It maps four keywords to iteration 1 Skill names that the engine never loads.
  • Design: The loop is capture (step 7), distil, review, prove, publish, retire.
    • Distil: an offline lesson-curator role groups candidates across Executions and proposes lessons. It runs outside any Execution.
    • Review: each proposal is an issue file under .scratch/lessons/issues/ with Status: needs-triage.
    • Prove: the evaluation from step 8 runs with and without the change.
    • Publish: lessons land in a lessons Skill under skills/, one reference file per Deliverable kind; each lesson has its rule, reason, source Execution and date. The bundle's role_skills lists it.
    • Retire: at each vendor sync, remove lessons the upstream Skill now covers.
  • Where: skills/<lessons-skill>/, agents/lesson-curator.md, a curator entry point beside eval/solution_eval.py, providers/gcp.yaml, scripts/sync-vendored-skills.sh.
  • Acceptance criteria: Vendored Skills stay byte-identical. No lesson reaches a Skill without an approved issue file. Each Execution records the lessons Skill version it loaded. One real lesson has gone through the whole loop with its evaluation comparison recorded.

Step 11: IaC patterns Skill

  • Report ref: enhancement 7
  • Capability: The IaC author starts from known-good infrastructure code.
  • Rationale: The IaC author is the most expensive role: 812 thousand tokens, 23 requests, 4 runs and 3 Revisions in case 03. It writes from a blank page each time.
  • Where: skills/<iac-patterns-skill>/ with one reference per product, providers/gcp.yaml (role_skills.iac-author).
  • Acceptance criteria: On the three cases, quality checks hold and IaC author Revisions or tokens fall against the step 8 baseline.

Step 12: Overlay Skills and an organisation standards Skill

  • Report ref: enhancement 8
  • Capability: New standards and clients need no code change.
  • Rationale: Skills are fixed at start-up: AdkAgentRuntime loads every Skill once, and role_skills is static per provider. Only 2 of the 9 Skills are our own, and both describe tools.
  • Where: app/adk/adk_runtime.py, app/adk/roles.py, app/providers.py, ExecutionEngine.start in app/agent.py, app/a2a_server.py, a standards Skill under skills/.
  • Acceptance criteria: An Execution can name overlay Skills at start; each passes the same Agent Skills validation as bundled Skills, and an unknown or invalid one refuses the start. An organisation standards Skill (naming, tagging, approved products) is loaded by the roles the bundle lists.

Step 13: Trim Specialist context

  • Report ref: enhancement 12
  • Capability: Each Specialist receives what its Deliverable needs.
  • Rationale: inputs_of gives a Specialist every Deliverable of every earlier Phase in full. The describer used 513 thousand tokens over 12 requests.
  • Where: inputs_of in app/workflows/deliverables.py, _specialist_input in app/nodes/execution_nodes.py.
  • Acceptance criteria: Direct Dependencies arrive in full; other earlier Deliverables arrive as summaries, with read_deliverable-style access to the full text on demand. Quality checks hold and describer tokens fall against the step 8 baseline.

Step 14: Provider-owned contracts

  • Report ref: enhancement 13
  • Capability: A second Cloud provider is configuration, as ADR-0002 intends.
  • Rationale: The six pillar headings, the eight solution-guide sections and the Terraform attachment names are hard-coded in app/workflows/deliverables.py. Other providers' frameworks name and count pillars differently. The bundle's pillar keys already differ from the headings (performance-optimization against Performance efficiency).
  • Where: providers/gcp.yaml, app/providers.py, app/workflows/deliverables.py, app/presentation.py, agents/iac-author.md, agents/validator.md, agents/design-advisor.md.
  • Timing: Build when a second Cloud provider is planned. ADR-0002 says no provider interface exists in code until then.
  • Acceptance criteria: Pillar headings come from pillar_skills, guide sections from the bundle's output_template, and the IaC tool with its file contract from the bundle. grep -rni "terraform" app/workflows app/nodes app/states is empty, and no pillar or guide-section name appears as a literal in app/workflows/deliverables.py.

Step 15: Group Executions by Customer case

  • Report ref: none; requested by the owner on 2026-10-06
  • Capability: Everything under deliverables/ is grouped by Customer case, so a case ties each request to the artefacts generated for it.
  • Rationale: Outputs were keyed only by Execution id. Nothing on disk said which customer workload an Execution belonged to or what it had been asked.
  • Where: app/states/execution_state.py (the case id rule and the paths), ExecutionEngine.start in app/agent.py, _write_outputs in app/nodes/execution_nodes.py, app/main.py, app/a2a_server.py, eval/solution_eval.py, CONTEXT.md.
  • Acceptance criteria: Every Execution's files are under deliverables/cases/<customer_case>/executions/<execution_id>/. Each has a manifest naming the case, the request and every file. Each case has an index of its Executions and their requests. A case id that is not a safe slug is refused before anything runs. A Customer case never feeds one Execution into another.

Step 16: Cache Terraform providers between checks

  • Report ref: none; requested by the owner on 2026-10-07 to shorten a run
  • Capability: A provider is downloaded once and reused by every later terraform validate and plan.
  • Rationale: Each check runs terraform init in a new scratch directory, so every call downloaded its providers again. The setting TF_PLUGIN_CACHE_DIR existed and was unset by default.
  • Where: TerraformRunner in app/tools/local.py, app/config.py, .env.example.
  • Acceptance criteria: The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.