Design is now approved before any infrastructure code is written (ADR-0005). A new target-architect Specialist produces one graph of Elements, Relationships and Checks; the diagram is rendered from it, deployment guidance must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations carry ids in data files, and a submission with a gap in the Requirements chain is refused. Also in this commit: - Customer cases group Executions, with an intent history per case - lesson candidates are captured when an Execution ends - evaluation gains chain, verdict, judged and interactive checks, and per-case thresholds - Terraform providers are cached between checks - the requirements analyst proposes defaults instead of asking - the legacy-baseline case and the checks that only read it are removed - ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed - plans for iterations 4 to 7 under .agents/plans/ Unproven with a real model: no evaluation run has completed on this design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
61 lines
6.8 KiB
Markdown
61 lines
6.8 KiB
Markdown
# Plan Detail
|
|
|
|
Every step is built test-first with the fake runtime and ends with `pytest`, `ruff check .` and `mypy .` passing.
|
|
|
|
## Step 1: Accept ADR-0006 and update the glossary
|
|
|
|
- **Capability:** A recorded decision, and vocabulary that no longer contradicts it.
|
|
- **The four points the owner confirms:** a Change request is a new Execution and never modifies its Baseline; the Baseline is a completed Execution of the same Customer case; the output is complete, not a patch; a Change request may itself be a Baseline.
|
|
- **Where:** `docs/adr/0006-*.md` status; `CONTEXT.md` entries for Execution, Session and Customer case, plus new entries **Change request** and **Baseline**; the sentence "An Execution starts clean" in `docs/adr/0004-*.md`.
|
|
- **Acceptance criteria:** The ADR is `accepted` or amended with this plan updated to match. No glossary entry says an Execution can never start from another.
|
|
|
|
## Step 2: Start an Execution against a Baseline
|
|
|
|
- **Capability:** A caller names a Baseline and the new Execution begins with its Deliverables.
|
|
- **Where:** `ExecutionEngine.start(baseline=...)` in `app/agent.py`; `/generate` body and A2A message metadata (`app/main.py`, `app/a2a_server.py`); `app/states/execution_state.py` (a `baseline` entry holding the Baseline's id and Deliverables); `execution.json` records the Baseline; `intent/history.json` records it on the new statement, so the history shows the line of versions.
|
|
- **Design:** The Baseline is loaded from its files under the Customer case, through `execution.json`, so it works across restarts and does not need the Baseline's checkpoint.
|
|
- **Acceptance criteria:** Start is refused, before anything runs, when the Baseline does not exist, is not completed, or belongs to another Customer case. An Execution started without a Baseline behaves exactly as today. The Baseline's files are byte-identical after the Change request completes.
|
|
|
|
## Step 3: Requirements revised with a change record
|
|
|
|
- **Capability:** The user's high-level statement of change becomes a precise change to the requirements.
|
|
- **Where:** `agents/requirements-analyst.md`; the requirements contract in `app/workflows/deliverables.py`; `agents/partials/specialist-protocol.md` (a `baseline` entry in the context).
|
|
- **Design:** `requirements.json` keeps the id of every requirement that still holds and gives each requirement a `change` of unchanged, added, changed or removed, with the reason. New requirements take new ids; an id is never reused.
|
|
- **Acceptance criteria:** Submission is refused when a Baseline requirement is missing from the revised list without being marked removed, or when a changed requirement has no reason. In an interactive Execution the analyst confirms the change record with the user before submitting.
|
|
|
|
## Step 4: Later Specialists revise from the Baseline
|
|
|
|
- **Capability:** Each Deliverable changes only where the changed requirements reach.
|
|
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py` (each Specialist receives the Baseline's version of its own Deliverable and the ids that changed upstream); every Specialist role file; the Target architecture Skill from iteration 6 step 4; `app/workflows/target_architecture.py` (a function that compares two graphs).
|
|
- **Design:** Ids are stable across versions: an Element, Relationship or Check that is unchanged keeps its id and its text. The engine computes what is reached by walking the Requirements chain from each changed requirement.
|
|
- **Acceptance criteria:** A test with the fake runtime shows that an Element or Check not reached by any changed requirement is identical to the Baseline's, and one that differs without being reached is reported in the change summary as an unrequested change. Validation still gives a verdict for every Check.
|
|
|
|
## Step 5: Approvals show the change
|
|
|
|
- **Capability:** The user approves what changed, not the whole architecture again.
|
|
- **Where:** `request_approval` in `app/tools/execution.py`, `app/presentation.py`, `agents/orchestrator.md`.
|
|
- **Acceptance criteria:** In a Change request, each Approval prompt lists what was added, changed and removed against the Baseline for that Phase's Deliverables, computed by the engine from the ids, not written by the Orchestrator.
|
|
|
|
## Step 6: Change summary and version line in the case
|
|
|
|
- **Capability:** A reader of a Customer case can see each version and what moved between them.
|
|
- **Where:** `_write_outputs` in `app/nodes/execution_nodes.py`; the packager role file.
|
|
- **Design:** Each Change request writes `changes.json` (requirements, Elements, Relationships and Checks added, changed and removed, each with its reason) and a readable summary in the solution guide. `intent/history.json` lists each statement with its Baseline.
|
|
- **Acceptance criteria:** The full Deliverables are written as for any Execution. `changes.json` matches a comparison of the two Executions' data files. Unrequested changes are listed separately from requested ones.
|
|
|
|
## Step 7: Evaluation: a change case
|
|
|
|
- **Capability:** Evidence that a change stays contained.
|
|
- **Where:** `eval/datasets/solution_cases.json`, `eval/solution_eval.py`, `eval/scoring.py`.
|
|
- **Design:** A case that runs a base request and then a Change request against it, for example retention moving from 30 days to 7 years.
|
|
- **Acceptance criteria:** The change case scores at least 0.8. New checks pass: every changed requirement reaches a changed or new Check; nothing outside its reach changed without being reported; the Change request used fewer tokens than its Baseline.
|
|
|
|
## Step 8: Remove the `legacy-baseline` case
|
|
|
|
- **Capability:** `deliverables/` holds only what the current engine produced.
|
|
- **Rationale:** `deliverables/cases/legacy-baseline/` is the output of the deleted iteration 1 pipeline: 68 Executions (62 of them from its tests and old benchmark) and its last shared copies in `as-is/`, `target/`, `validation/` and `guides/`. The owner chose on 2026-10-06 to keep it until this work is complete with the expected outcome, and to remove it then.
|
|
- **Gate:** Do this only when iteration 6 step 11 and step 7 here have both met their acceptance criteria. If either falls short, leave the case in place and say so.
|
|
- **Where:** `deliverables/cases/legacy-baseline/`; `scripts/validate_artifacts.py`, `tests/test_artifacts.py` and `tests/test_package.py`, which read its `as-is/`, `target/`, `validation/` and `guides/` folders and must be removed or pointed at a current Execution; `scripts/validate.sh`; `VALIDATION.md` step 1; the root `terraform/`, `docs/requirements.md`, `docs/architecture.md`, `docs/source-architecture.md` and `requirements.yaml`, which are the same pipeline's samples.
|
|
- **Guardrail:** Commit first, then delete in a commit of its own, so the removal can be reverted.
|
|
- **Acceptance criteria:** The case and the samples are gone, nothing in `scripts/` or `tests/` refers to them, and `pytest`, `ruff check .` and `mypy .` pass.
|