Files
Jonathan Boniface e9246828b3 feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:59:09 +01:00

9.9 KiB

Plan Detail

Every step is built test-first with the fake runtime and ends with pytest, ruff check . and mypy . passing. File names for new attachments are proposals; keep them once a step uses them.

Step 1: Accept ADR-0005

  • Capability: A recorded decision to build on.
  • The three points the owner confirms:
    1. The user approves the Target architecture before code is written, which splits the solution architecture Phase and makes five Phases against the upstream Skill's four.
    2. A new Specialist produces the Target architecture.
    3. Code renders the diagram from the graph, and the diagrammer role is removed.
  • Acceptance criteria: ADR-0005's status is accepted, or it is amended and this plan is updated to match before any other step starts.

Step 2: Identify requirements, components and recommendations by id

  • Capability: Stable ids at the start of the Requirements chain, as data.
  • Rationale: A rule such as "every requirement has a Check" can only be enforced if requirements can be counted. Today requirements, components and recommendations exist only as prose.
  • Where: app/workflows/deliverables.py (new required attachments and their checks), agents/requirements-analyst.md, agents/decomposer.md, agents/design-advisor.md, agents/partials/specialist-protocol.md.
  • Design:
    • requirements attaches requirements.json: a list of {id, kind, text} with kind functional or non-functional.
    • technical_decomposition attaches components.json: a list of {id, name, requirements: [ids]}. This replaces the table contract from iteration 4 step 4.
    • design_recommendations attaches recommendations.json: a list of {id, pillar, text, components: [ids]}.
  • Acceptance criteria: Each attachment is refused when malformed, when an id repeats, or when it refers to an id that the Deliverable it depends on does not define. A requirement that no component serves is refused at the decomposition.

Step 3: The graph: format, parsing and structural rules

  • Capability: One definition of what a well-formed Target architecture is.
  • Where: new app/workflows/target_architecture.py, pure and without I/O, like the Ledger.
  • Design: target-architecture.json holds:
    • elements: {id, name, component, product, resource_type}.
    • relationships: {id, from, to, kind}.
    • checks: {id, applies_to, statement, source}, where applies_to is an Element or Relationship id and source is a requirement or recommendation id.
  • Acceptance criteria: Parsing refuses, with the offending id named: a repeated id; a Relationship whose from or to is not an Element; a Check whose applies_to is not in the graph; an Element with no Check; an empty statement. The module exposes the graph as typed data for steps 5, 7, 8 and 9.

Step 4: The Target architecture Deliverable, its Specialist and its Skill

  • Capability: A Deliverable that is the graph, produced by its own Specialist following a Skill.
  • Where: DeliverableKind.TARGET_ARCHITECTURE in app/workflows/deliverables.py, depending on product selection and design recommendations; a new role file in agents/; a new Skill under skills/ that defines the format and how to derive Elements, Relationships and Checks; providers/gcp.yaml (role_skills, model); app/adk/roles.py picks the role up from the dependency graph.
  • Rationale: ADR-0002: how to model an architecture is behaviour and belongs in a Skill. The role file stays provider-neutral.
  • Who writes it: the Specialist, never the user. The user's input is the Workflow request and their answers during requirements discovery; implementation-level detail is the agents' job, drawn from the pillar Skills and the provider's documentation.
  • Acceptance criteria: The Deliverable requires target-architecture.json, checked by step 3 at submission. Its content is a readable summary with one section per Element. A Check's statement is written so that it can be verified from the infrastructure code alone, and the Skill says so with examples.

Step 5: Completeness: the Requirements chain is a submission rule

  • Capability: A Target architecture with a broken chain cannot be submitted.
  • Rationale: This is the rule that makes the outcome understood up front. Without it the graph is one more document.
  • Where: check_submission currently sees only the Deliverable being submitted. It needs the Deliverables it depends on: extend it and specialist_tools in app/tools/execution.py to pass them.
  • Acceptance criteria: Submission is refused, naming each gap, when: a requirement id is the source of no Check, directly or through a component's Element; a component has no Element; a selected product appears on no Element; a recommendation is the source of no Check. A recommendation may be marked not applicable only with a stated reason, and that is listed for the user at Approval.

Step 6: Split the solution architecture Phase: design, then implementation

  • Capability: The user approves the Target architecture before code is written.
  • Where: PhaseId and DEPENDENCY_GRAPH in app/workflows/deliverables.py; agents/orchestrator.md; workflow.yaml (its test will fail until updated); app/card.py derives from the graph; app/presentation.py; eval/; the spec's Phase table.
  • Design: Design holds product selection, design recommendations and the Target architecture. Implementation holds the architecture description and the deployment guidance.
  • Acceptance criteria: No Deliverable in implementation can start until design is approved (a Ledger test). The Approval prompt for design summarises the graph: counts of Elements, Relationships and Checks, and any recommendation marked not applicable. A pre-approved Execution passes through as today.

Step 7: The diagram is rendered from the graph

  • Capability: A diagram that cannot disagree with the architecture.
  • Where: a renderer in app/workflows/target_architecture.py from Elements and Relationships to Mermaid; architecture.mmd is added to the Target architecture's attachments by the engine on submission; the architecture_diagram Deliverable, agents/diagrammer.md and its bundle entries are removed; app/presentation.py reads the diagram from the Target architecture so /generate's keys are unchanged.
  • Acceptance criteria: Rendering is deterministic and passes mermaid_check for any graph step 3 accepts, including names with characters Mermaid treats specially. Every Element and Relationship appears exactly once. The solution guide still carries the diagram in the template's section 4.2.

Step 8: Deployment guidance implements the graph

  • Capability: Code written against a known list, with each item accounted for.
  • Where: deployment_guidance depends on the Target architecture; agents/iac-author.md; skills/iac-validation/SKILL.md; a required attachment implementation.json: for each Check id, the file and resource address that satisfies it.
  • Acceptance criteria: Submission is refused when a Check id is missing from implementation.json or an id in it is not in the graph. The IaC author's context carries the graph, not the prose recommendations. terraform_validate must still pass before submission, as today.

Step 9: Validation gives a verdict per Check

  • Capability: A fixed bar: validation answers the graph's Checks and nothing else can fail the code.
  • Where: validation_results requires verdicts.json in place of findings.json: for each Check id, {verdict, evidence} with verdict pass, fail or not verifiable, plus a separate list proposed_additions of {statement, applies_to, reason}. agents/validator.md, skills/iac-validation/SKILL.md, app/states/lessons.py, eval/scoring.py.
  • Acceptance criteria: Submission is refused unless every Check has exactly one verdict with evidence. A proposed addition never counts as a failure. A not verifiable verdict is treated as a gap in the graph, not in the code. The dry-run result (terraform validate, and plan when permitted) is itself a Check on the deployment as a whole, so it has a verdict too.

Step 10: Revision routing: the code or the graph

  • Capability: Each validation outcome leads to exactly one kind of Revision.
  • Where: _orchestrator_context in app/nodes/execution_nodes.py gives the Orchestrator the failed Check ids and the proposed additions; agents/orchestrator.md.
  • Design: Failed Checks reopen the deployment guidance, with the failed ids as the reason. Proposed additions and not verifiable verdicts go to the user in an interactive Execution; in a pre-approved one the Orchestrator reopens the Target architecture only for an addition that names a requirement or recommendation id. Reopening the Target architecture returns the implementation Phase to draft and withdraws its Approval, by the Ledger's existing rule.
  • Acceptance criteria: Engine tests show both routes. With unchanged code and an unchanged graph, a second validation pass has the same Check ids as the first: the list cannot grow.

Step 11: Evaluation: prove the loop converges

  • Capability: Evidence that the outcome is understood before code is written.
  • Where: eval/scoring.py, eval/solution_eval.py, eval/datasets/solution_cases.json.
  • Design: New checks: every requirement reaches at least one Check with a pass verdict; no Check is left failed or not verifiable; the diagram matches the graph. The report records, per case, the number of validation passes and the Check ids failing at each.
  • Acceptance criteria: All benchmark cases complete and score at least 0.8. In each, the set of failing Check ids never grows from one validation pass to the next, and deployment guidance needs at most one Revision in at least two of the three cases. The thresholds left unset by iteration 4 step 8 are set from this run. The comparison with the stopped run of 2026-10-06 is written into this step's file.