Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/DETAIL.md
Jonathan Boniface 5adc911a82 feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:58:38 +01:00

20 KiB

Plan Detail

Every step ends with the full suite passing. Steps that change agent behaviour (3, 6, 10, 11, 12, 13) also end with an evaluation run recorded in their step file, compared against the baseline in iter-02-solution-architect-agent/steps/step-8.md.

Step 1: Structured validator findings

  • Report ref: enhancement 3
  • Capability: Validation findings as data the engine can route, count and learn from.
  • Rationale: The validator reports findings as free markdown. The Orchestrator reads them once to decide what to reopen; nothing else can use them. They are the richest learning signal in an Execution.
  • Where: app/workflows/deliverables.py (add findings.json to the validation_results attachments), agents/validator.md, skills/iac-validation/SKILL.md (the finding format), app/tools/execution.py (validate the attachment's shape at submission).
  • Acceptance criteria: A validation_results submission without a parseable findings.json is refused with the reason. Each finding carries the Deliverable kind it traces to, a severity, the evidence and the rule. An empty list is valid and means a pass.

Step 2: Carry the draft through needs_input

  • Report ref: enhancement 9
  • Capability: The user receives the draft they approved.
  • Rationale: The Specialist protocol asks a Specialist to confirm its draft with the user before submitting. needs_input carries only questions and Specialists are stateless per delegation, so the approved draft is lost and written again on the next run.
  • Where: needs_input in app/tools/execution.py (optional draft and attachments), NeedsInput in app/adk/runtime.py, _outcome_record and _specialist_input in app/nodes/execution_nodes.py (store the draft, return it in context), app/states/execution_state.py, agents/partials/specialist-protocol.md.
  • Acceptance criteria: A Specialist restarted after the user approves its draft receives that draft in its context. A test shows the submitted content equals the approved draft when the fake runtime submits what it was given.

Step 3: Reuse pillar advice across Revisions

  • Report ref: enhancement 10
  • Capability: A Revision of the design recommendations re-consults only the pillars its feedback touches.
  • Rationale: Pillar advice is never stored. In case 01, two Revisions re-ran all six advisors each time: 18 advisor runs for one Deliverable.
  • Where: app/tools/pillars.py (the tool already accepts pillars), the Execution state for stored advice, agents/design-advisor.md (on a Revision, name the affected pillars).
  • Acceptance criteria: On a second attempt with stored advice, only the named pillars run and the tool returns stored advice for the rest. A changed brief for a pillar invalidates that pillar's stored advice.

Step 4: Single source for Deliverable contracts

  • Report ref: enhancement 11
  • Capability: Required headings and attachments declared once, and contracts for the Deliverables that have none.
  • Rationale: Required headings appear in each role file and again in DeliverableSpec.sections. technical_decomposition, architecture_description and the diagram's content accept any non-empty text.
  • Where: app/workflows/deliverables.py, _specialist_input in app/nodes/execution_nodes.py (pass the contract in context), agents/*.md (remove the "Required headings" lines), agents/partials/specialist-protocol.md.
  • Acceptance criteria: No role file lists required headings. Each Specialist's context carries its contract. The decomposition requires a component table naming the requirements each component serves.

Step 5: Cap parallel Specialists

  • Report ref: enhancement 14
  • Capability: A bound on concurrent Specialist runs per Execution.
  • Rationale: The spec's Risks section calls for a default cap of 4 against model rate limits. orchestrator_turn sends every delegated Deliverable at once, and the pillar fan-out adds six more.
  • Where: app/nodes/execution_nodes.py, app/tools/pillars.py, a MAX_PARALLEL_SPECIALISTS setting in app/config.py.
  • Acceptance criteria: A test with the fake runtime shows at most the configured number of Specialist runs in flight. The pillar fan-out counts toward the same cap.

Step 6: Orchestrator step order

  • Report ref: findings by agent, orchestrator row
  • Capability: A failed validation goes back for Revision before the user is asked to approve it.
  • Rationale: In agents/orchestrator.md, "A Phase is complete" (step 3) precedes "Validation found problems" (step 5), and the Orchestrator takes the first step that applies.
  • Where: agents/orchestrator.md.
  • Acceptance criteria: The validation-failure step sits above the Approval step. An evaluation run with a failing validation shows a reopen before any Approval request for the validation Phase.

Step 7: Capture lesson candidates

  • Report ref: enhancement 4
  • Capability: A durable record, per Execution, of what went wrong and what fixed it.
  • Rationale: The signals exist and are discarded. Revision feedback lives in the Ledger inside one checkpoint. Tool refusals (Not submitted, Refused, terraform validate failed) reach only the Python log through REJECTION_PREFIXES in app/adk/adk_runtime.py.
  • Where: app/recorder.py (a record_lesson_candidates method on ExecutionRecorder), finalize and end_with_warnings in app/nodes/execution_nodes.py, the runtime for refusals.
  • Acceptance criteria: A finished Execution writes one record holding each reopen with its reason and actor, each finding from step 1, each tool refusal with role and tool, each user change request and each exhausted budget. The record names the Execution, the Deliverable kind and the Skill versions loaded. It holds no secrets and no discovered cloud data.

Step 8: Evaluation that can see improvement

  • Report ref: enhancement 6
  • Capability: An evaluation whose score moves when a Skill gets better or worse.
  • Rationale: The 8 checks are structural and both scored runs reached 1.0. The loop's proof step has nothing to measure.
  • Where: eval/scoring.py, eval/solution_eval.py, eval/datasets/solution_cases.json, tests/eval/.
  • Acceptance criteria: New checks cover requirement-to-component traceability, coverage of each pillar Skill's checklist, and a graded rubric for recommendation quality. One interactive case checks that a Specialist asks before it submits. Revisions, seconds and tokens per case have recorded thresholds, and a run that exceeds one is reported as a regression. case-02-high-scale-ingestion has a scored run.

Step 9: ADR: the lessons loop and overlay Skills

  • Report ref: sequencing note
  • Capability: A recorded decision on how Skills change and how they are chosen per Execution.
  • Rationale: Steps 10 and 12 change which Skills an agent loads and who may edit them. That is an architectural decision, and ADR-0002 does not cover it.
  • Where: docs/adr/0004-*.md.
  • Decisions the owner must make: whether lessons need human review before they take effect (the plan assumes yes); where lessons Skills live and who owns them; whether an Execution may name overlay Skills, and who may supply them.
  • Acceptance criteria: The ADR is accepted and steps 10 and 12 are updated to match it.

Step 10: Lessons Skill and lesson curator

  • Report ref: enhancement 5
  • Capability: Each Execution can improve the next, under human review.
  • Rationale: eval/optimizer.py is the only learning code. It maps four keywords to iteration 1 Skill names that the engine never loads.
  • Design: The loop is capture (step 7), distil, review, prove, publish, retire.
    • Distil: an offline lesson-curator role groups candidates across Executions and proposes lessons. It runs outside any Execution.
    • Review: each proposal is an issue file under .scratch/lessons/issues/ with Status: needs-triage.
    • Prove: the evaluation from step 8 runs with and without the change.
    • Publish: lessons land in a lessons Skill under skills/, one reference file per Deliverable kind; each lesson has its rule, reason, source Execution and date. The bundle's role_skills lists it.
    • Retire: at each vendor sync, remove lessons the upstream Skill now covers.
  • Where: skills/<lessons-skill>/, agents/lesson-curator.md, a curator entry point beside eval/solution_eval.py, providers/gcp.yaml, scripts/sync-vendored-skills.sh.
  • Acceptance criteria: Vendored Skills stay byte-identical. No lesson reaches a Skill without an approved issue file. Each Execution records the lessons Skill version it loaded. One real lesson has gone through the whole loop with its evaluation comparison recorded.

Step 11: IaC patterns Skill

  • Report ref: enhancement 7
  • Capability: The IaC author starts from known-good infrastructure code.
  • Rationale: The IaC author is the most expensive role: 812 thousand tokens, 23 requests, 4 runs and 3 Revisions in case 03. It writes from a blank page each time.
  • Where: skills/<iac-patterns-skill>/ with one reference per product, providers/gcp.yaml (role_skills.iac-author).
  • Acceptance criteria: On the three cases, quality checks hold and IaC author Revisions or tokens fall against the step 8 baseline.

Step 12: Overlay Skills and an organisation standards Skill

  • Report ref: enhancement 8
  • Capability: New standards and clients need no code change.
  • Rationale: Skills are fixed at start-up: AdkAgentRuntime loads every Skill once, and role_skills is static per provider. Only 2 of the 9 Skills are our own, and both describe tools.
  • Where: app/adk/adk_runtime.py, app/adk/roles.py, app/providers.py, ExecutionEngine.start in app/agent.py, app/a2a_server.py, a standards Skill under skills/.
  • Acceptance criteria: An Execution can name overlay Skills at start; each passes the same Agent Skills validation as bundled Skills, and an unknown or invalid one refuses the start. An organisation standards Skill (naming, tagging, approved products) is loaded by the roles the bundle lists.

Step 13: Trim Specialist context

  • Report ref: enhancement 12
  • Capability: Each Specialist receives what its Deliverable needs.
  • Rationale: inputs_of gives a Specialist every Deliverable of every earlier Phase in full. The describer used 513 thousand tokens over 12 requests.
  • Where: inputs_of in app/workflows/deliverables.py, _specialist_input in app/nodes/execution_nodes.py.
  • Acceptance criteria: Direct Dependencies arrive in full; other earlier Deliverables arrive as summaries, with read_deliverable-style access to the full text on demand. Quality checks hold and describer tokens fall against the step 8 baseline.

Step 14: Provider-owned contracts

  • Report ref: enhancement 13
  • Capability: A second Cloud provider is configuration, as ADR-0002 intends.
  • Rationale: The six pillar headings, the eight solution-guide sections and the Terraform attachment names are hard-coded in app/workflows/deliverables.py. Other providers' frameworks name and count pillars differently. The bundle's pillar keys already differ from the headings (performance-optimization against Performance efficiency).
  • Where: providers/gcp.yaml, app/providers.py, app/workflows/deliverables.py, app/presentation.py, agents/iac-author.md, agents/validator.md, agents/design-advisor.md.
  • Timing: Build when a second Cloud provider is planned. ADR-0002 says no provider interface exists in code until then.
  • Acceptance criteria: Pillar headings come from pillar_skills, guide sections from the bundle's output_template, and the IaC tool with its file contract from the bundle. grep -rni "terraform" app/workflows app/nodes app/states is empty, and no pillar or guide-section name appears as a literal in app/workflows/deliverables.py.

Step 15: Group Executions by Customer case

  • Report ref: none; requested by the owner on 2026-10-06
  • Capability: Everything under deliverables/ is grouped by Customer case, so a case ties each request to the artefacts generated for it.
  • Rationale: Outputs were keyed only by Execution id. Nothing on disk said which customer workload an Execution belonged to or what it had been asked.
  • Where: app/states/execution_state.py (the case id rule and the paths), ExecutionEngine.start in app/agent.py, _write_outputs in app/nodes/execution_nodes.py, app/main.py, app/a2a_server.py, eval/solution_eval.py, CONTEXT.md.
  • Acceptance criteria: Every Execution's files are under deliverables/cases/<customer_case>/executions/<execution_id>/. Each has a manifest naming the case, the request and every file. Each case has an index of its Executions and their requests. A case id that is not a safe slug is refused before anything runs. A Customer case never feeds one Execution into another.

Step 16: Cache Terraform providers between checks

  • Report ref: none; requested by the owner on 2026-10-07 to shorten a run
  • Capability: A provider is downloaded once and reused by every later terraform validate and plan.
  • Rationale: Each check runs terraform init in a new scratch directory, so every call downloaded its providers again. The setting TF_PLUGIN_CACHE_DIR existed and was unset by default.
  • Where: TerraformRunner in app/tools/local.py, app/config.py, .env.example.
  • Acceptance criteria: The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.

Step 17: A working copy for files and for the graph

  • Report ref: none; requested by the owner on 2026-10-07 to shorten a run
  • Capability: A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
  • Rationale: In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every terraform_fmt and terraform_validate call carried every file, terraform_fmt returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
  • Where: app/tools/workspace.py (new), specialist_tools in app/tools/execution.py, app/tools/local.py, app/tools/toolkit.py, DeliverableSpec.working_copy in app/workflows/deliverables.py, render_tables in app/workflows/target_architecture.py, agents/target-architect.md, agents/iac-author.md, agents/validator.md, skills/iac-validation/SKILL.md.
  • Acceptance criteria: The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.

Step 18: Provider schema lookup and one pinned provider version

  • Report ref: none; requested by the owner on 2026-10-07
  • Capability: An agent can ask what a resource type accepts, and get the provider's own answer.
  • Rationale: All 13 Terraform refusals in the 2026-10-07 run were one mistake: startup_cpu_boost and cpu_idle in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
  • Where: TerraformRunner.schema in app/tools/local.py, terraform.required_providers in providers/gcp.yaml, app/providers.py, the role files of the target architect, the IaC author and the validator.
  • Acceptance criteria: terraform_schema lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.

Step 19: A Check the code cannot meet goes back to the Target architecture

  • Report ref: none; requested by the owner on 2026-10-07
  • Capability: A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
  • Rationale: Check K-30 of the 2026-10-07 run asked for startup_cpu_boost on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
  • Where: check_implementation and unimplementable_checks in app/workflows/chain.py, the playbook in app/nodes/router.py, _orchestrator_context in app/nodes/execution_nodes.py, agents/orchestrator.md, agents/iac-author.md, agents/validator.md, skills/requirements-chain/SKILL.md.
  • Acceptance criteria: implementation.json may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.

Step 20: A model timeout per role

  • Report ref: none; requested by the owner on 2026-10-07
  • Capability: A role whose replies are long is given longer for one model request than the rest.
  • Rationale: One MODEL_TIMEOUT_SECONDS (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
  • Where: models.timeout_seconds in providers/gcp.yaml, ModelConfig in app/providers.py, RoleSpec.model_timeout_seconds, AdkAgentRuntime._content_config.
  • Acceptance criteria: A role named in the bundle gets its own timeout; every other role gets the default.

Step 21: A Deliverable stands while what it was written from is unchanged

  • Report ref: none; requested by the owner on 2026-10-07 after case 02
  • Capability: A Revision upstream does not rewrite a Deliverable it leaves true.
  • Rationale: In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
  • Where: DeliverableSpec.written_from in app/workflows/deliverables.py, graph_structure in app/workflows/chain.py, specialist_task in app/nodes/execution_nodes.py.
  • Acceptance criteria: When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.

Step 22: Validation reports every gap in one pass

  • Report ref: none; requested by the owner on 2026-10-07 after case 02
  • Capability: One validation finds the gaps in the Target architecture, not one validation per gap.
  • Rationale: In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
  • Where: agents/validator.md, agents/target-architect.md, skills/requirements-chain/SKILL.md.
  • Acceptance criteria: The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.