feat: establish the Target architecture as a graph before code

Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 13:59:09 +01:00
parent 7ee49ab352
commit e9246828b3
586 changed files with 5166 additions and 22432 deletions

View File

@@ -0,0 +1,12 @@
# Step 1 — Implementation Detail
1. `validation_results` now requires a `findings.json` attachment (`FINDINGS_ATTACHMENT` in `app/workflows/deliverables.py`).
2. `check_findings` validates it at submission: a JSON list of objects with `deliverable` (a Deliverable kind), `severity` (`critical`, `high`, `medium`, `low`), `evidence` and `rule`. `[]` is valid and means a pass. Each problem is refused by field, for example `findings.json[1].severity: must be one of ...`.
3. `agents/validator.md` tells the validator to attach it; the format and an example live in `skills/iac-validation/SKILL.md` under "Reporting findings".
4. The file is written with the other attachments to `deliverables/executions/<id>/findings.json`.
The check was added to `check_submission`, not to `app/tools/execution.py` as planned, so every contract rule stays in one function.
Tests: `tests/execution/test_deliverables.py` (missing attachment, empty list, well-formed finding, malformed JSON, per-field problems).
Not verified: no real-model run has confirmed the validator produces a well-formed file on its first submission. A malformed one costs a retry inside the validator's run, not a Revision.

View File

@@ -0,0 +1,19 @@
# Step 15 — Implementation Detail
1. **Customer case** is a glossary term (`CONTEXT.md`): a caller-supplied slug grouping the Executions for one customer workload. An Execution started without one belongs to `unassigned`, so every Execution sits under a case.
2. **Layout.** `deliverables/cases/<customer_case>/executions/<execution_id>/` holds an Execution's files. `execution_dir`, `case_dir` and `lessons_file` in `app/states/execution_state.py` are the only places these paths are built.
3. **`execution.json`** in each Execution folder names the case, the Execution, the Session, the Workflow request, and for each Deliverable its document and attachments.
4. **`intent/`** in each case folder records the customer's Intent and its history (`app/customer_case.py`). Each Workflow request is written as given to `intent/001.md`, `002.md` and so on when its Execution starts. `intent/history.json` lists every statement in order with the Execution it started, that Execution's status (`in_progress`, `completed`, `completed_with_warnings`) and the path to its files. This replaced the `case.json` index first built for this step; `execution.json` carries `intent_version`.
5. **Lesson candidates** are grouped the same way: `lessons/candidates/<customer_case>/<execution_id>.json`, and the record carries `customer_case`.
6. **Callers.** `ExecutionEngine.start(customer_case=...)`; `/generate` takes `customer_case` in its body and answers 400 for an unsafe one; A2A reads `customer_case` from the message metadata. The evaluation uses each benchmark case's id.
7. **The case id** must be 1 to 64 lowercase letters, digits or hyphens, starting and ending with a letter or digit. It becomes a directory name, so anything else is refused before the Execution starts.
8. **Existing content moved.** `deliverables/as-is`, `target`, `validation`, `guides` and the 68 old `executions` folders are now under `deliverables/cases/legacy-baseline/`, with an `intent/` folder built from the request recorded in each old run's `requirements.md`, ordered by file time. `scripts/validate_artifacts.py` and `tests/test_artifacts.py` look there.
Limits:
- An Execution that ends with warnings writes no Deliverable files (an existing rule). Its request is still in the Intent history, with its status and no path, and its lesson candidates are grouped under its case.
- The 68 old runs were asked 12 different things but share one case, `legacy-baseline`. They can be split into a case per request from `intent/history.json` if that matters.
- `/generate`'s response keys are unchanged, so it does not return the case; `execution.json` does.
- The database rows carry the case only inside the status details, not as a column.
Tests: `tests/execution/test_engine.py` (grouping, manifest, index, `unassigned`, refused ids), `tests/a2a/test_server.py` (`/generate`), `tests/execution/test_lessons.py`.

View File

@@ -0,0 +1,20 @@
# Step 16 — Implementation Detail
1. `TF_PLUGIN_CACHE_DIR` defaults to `.cache/terraform-plugins` under the repository root (already gitignored and kept out of the image build context). Set it empty to disable. `Settings.terraform_plugin_cache()` resolves it to an absolute path, since checks run in scratch directories.
2. `TerraformRunner` creates the directory on first use and passes it to Terraform.
3. It also sets `TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true`. A scratch directory has no dependency lock file, and without this Terraform downloads a cached provider again to verify it, which defeats the cache.
4. `terraform init` calls are serialised by a lock when the cache is in use, because Terraform's plugin cache is not safe for concurrent installs.
Measured on 2026-10-07 with a one-resource configuration on the Google provider (`~> 6.0`), on this machine:
| | Seconds per `terraform validate` |
|---|---|
| No cache | 7.3 to 7.8 |
| Cache, first use (downloads once) | 7.3 |
| Cache, later uses | 1.6 to 2.0 |
So each check after the first saves about 5.5 seconds. The stopped run of 2026-10-06 made 7 validate calls in 16 minutes and more through its Revisions; at 12 to 15 calls a case, with the scoring's own check, that is roughly 1 to 1.5 minutes a case. The saving is real but small against a 35 minute run; it grows on a slower network.
Costs: the cache holds each provider version once (117 MB for one Google provider version; the existing folder holds 600 MB across versions). In the container it is created under `/app/.cache` at run time and is lost with the container; pre-loading it into the image, or mounting a volume, would carry the saving across restarts and is not done.
Tests: `tests/tools/test_local.py`, "the provider cache", using a stand-in terraform binary.

View File

@@ -0,0 +1,10 @@
# Step 2 — Implementation Detail
1. `needs_input` accepts an optional `draft` and `attachments`; `NeedsInput` carries them (`app/adk/runtime.py`).
2. The Execution state holds `drafts` by Deliverable kind. `orchestrator_turn` stores a draft when it applies a `needs_input` outcome and drops it when that Deliverable is submitted.
3. A Specialist's context carries `draft` (`{content, attachments}` or `null`).
4. `agents/partials/specialist-protocol.md` tells Specialists to pass the complete draft with their confirmation question, to submit it unchanged once the latest answer approves it, and to apply requested changes to it.
Test: `test_a_draft_sent_with_questions_comes_back_with_the_answer` in `tests/execution/test_engine.py` shows the second run receiving the draft and the submitted content equalling it.
The draft is not shown to the user by the engine; the Specialist's question still carries its summary. Whether the full draft should appear in the prompt is an open product question.

View File

@@ -0,0 +1,12 @@
# Step 3 — Implementation Detail
1. **Working memory.** Each Deliverable has a JSON-compatible working memory kept in Execution state (`working`, merged by key so parallel Specialists each write their own). `ToolFactory` now takes it as a third argument, so tools can remember across attempts.
2. **Pillar advice is remembered** under `pillar_advice` in the design recommendations' working memory (`app/tools/pillars.py`).
3. **Semantics of `consult_pillar_advisors`:** omit `pillars` to consult every pillar afresh; name `pillars` to re-consult only those, with the remembered advice returned for the rest under a heading marked "(from your earlier consultation)". Failed advice is never remembered.
4. `agents/design-advisor.md` tells the design advisor to name only the pillars its feedback touches when revising.
Changed from the plan: the criterion "a changed brief for a pillar invalidates that pillar's stored advice" was not built. The brief is one text for all pillars and a model rewords it on every attempt, so hashing it would discard everything each time. Omitting `pillars` is the way to refresh all advice.
Tests: `tests/execution/test_pillars.py` (subset re-consult, full refresh, failed advice not stored, and one through the engine showing advice surviving a Revision).
Not verified: the evaluation run this step calls for. The saving depends on the design advisor following the new instruction; in case 01 it would cut up to 12 of 18 advisor runs.

View File

@@ -0,0 +1,15 @@
# Step 4 — Implementation Detail
1. `DeliverableSpec.contract()` returns the sections, attachments and table columns as data; every Specialist's context now carries it as `contract`.
2. Role files no longer list required headings. What remains in them is guidance the contract cannot express (what goes under a heading, verdicts, optional headings).
3. New contracts:
- `technical_decomposition`: a `Components` section and one table whose header names `component` and `requirement`.
- `architecture_description`: `Components` and `Flow` sections. Headings match by suffix, so "Data flow" and "Architecture components" satisfy them.
- Every required attachment must have content, which gives the diagram's `architecture.mmd` a minimum bar.
4. Tests share one contract-complete fixture, `sample_submission` in `app/adk/fake_runtime.py`.
Two existing tests used an empty `terraform/variables.tf` as a fixture; they now use a non-empty one.
Tests: `tests/execution/test_deliverables.py`, and `test_specialist_context_carries_its_contract` in `tests/execution/test_engine.py`.
Not verified with real models: the two new section contracts change what the decomposer and describer must write.

View File

@@ -0,0 +1,11 @@
# Step 5 — Implementation Detail
1. `RunLimiter` in `app/adk/runtime.py` caps agent runs in flight. `slot()` holds one; `lend()` gives the caller's slot up while its own fan-out runs.
2. `specialist_task` takes a slot for each Specialist run. The pillar tool lends the design advisor's slot and takes one per advisor, so the fan-out shares the cap and a cap of 1 cannot deadlock.
3. Setting `MAX_PARALLEL_SPECIALISTS` (default 4, as the spec's Risks section asked). `open_assembly` builds one limiter and gives it to both the engine and the tool factory.
Changed from the plan: the cap is per engine process, not per Execution. Model rate limits apply to the project, so one cap across concurrent Executions is the stricter and more useful bound.
An `ExecutionEngine` built without a limiter is uncapped, which keeps direct construction in tests unchanged.
Tests: cap of 1 gives a peak of 1 and no cap gives a peak of 2 (`tests/execution/test_engine.py`); the fan-out under a held slot with a cap of 1 completes with a peak of 1 (`tests/execution/test_pillars.py`).

View File

@@ -0,0 +1,6 @@
# Step 6 — Implementation Detail
1. `agents/orchestrator.md`: "Validation found problems" is now step 3, above "A Phase is complete" (step 4) and "Work is ready" (step 5).
2. The step now says when it applies (validation results submitted and not yet approved), to read them first, and to send failed work back before asking the user to approve it.
Not verified: the acceptance criterion needs a real-model evaluation run with a failing validation, showing a reopen before any Approval request for the validation Phase. No such run was made. The deterministic suite cannot test this, because the order is the model's choice.

View File

@@ -0,0 +1,18 @@
# Step 7 — Implementation Detail
1. `app/states/lessons.py`: `collect_lesson_candidates` builds one record when an Execution ends, from what the Execution already holds. Candidate types:
- `revision`: each reopen, with its reason and who asked.
- `revision_limit`: a reopen refused at the cap.
- `change_request`: the user's feedback on a Phase, with the Deliverables in it.
- `finding`: each entry of every submitted `findings.json`, with the attempt it came from.
- `refusal`: each tool reply that did nothing (`REJECTION_PREFIXES`, now in `app/adk/runtime.py`), with role, tool and Deliverable.
- `unfinished`: a Specialist that stopped or spent its budget.
2. Refusals are captured in the nodes, not the runtime: `_noting_refusals` wraps each tool, so any runtime, the fake included, produces them. Specialist refusals travel in the outcome record; the Orchestrator's are kept in state under `refusals`.
3. The record names the Execution, its status, each role's Skills and a version per Skill. `skill_version` in `app/skills/agent_skills.py` is a 12-character content hash of the Skill's files.
4. It is written twice: through `ExecutionRecorder.record_lesson_candidates` (the database recorder stores it as an artifact of type `lesson_candidates`), and to `lessons/candidates/<execution_id>.json` under the output root. That directory is gitignored.
Changed from the plan: the file is not inside `deliverables/executions/<id>/`. An Execution that ends with warnings must write no Deliverable files (an existing rule with a test), and those Executions are the ones most worth learning from.
On "no secrets and no discovered cloud data": a refusal from a `cloud_read` tool keeps only the words before its first colon, and every message is cut at 500 characters. Revision reasons, user feedback and finding evidence are kept whole, so a record can quote the user and name resources in generated Terraform. Treat the files as client data.
Tests: `tests/execution/test_lessons.py`.

View File

@@ -0,0 +1,36 @@
# Step 8 — Implementation Detail
Built, with two acceptance criteria still open (listed at the end).
1. **Structural checks added to every case** (`eval/scoring.py`):
- `components_trace_to_requirements`: every row of the decomposition's component table names a requirement.
- `no_blocking_findings`: the final `findings.json` holds no `critical` or `high` finding.
2. **Judged checks**, run with `--judge`:
- `pillar_checklists_covered`: each item of each pillar Skill's "Validation checklist" is put to the judge as a yes-or-no question against the design recommendations. The weakest pillar must reach 70%.
- `recommendations_quality`: four graded questions (specific, actionable, trade-offs, traceable); the mean must reach 0.67.
- `checklist_items` reads the checklists from the vendored Skills, nested bullets included, so a vendor sync updates the questions.
3. **Judge seam** (`eval/judge.py`): a `Judge` takes one document and a map of typed questions and returns a score from 0 to 1 each. `GeminiJudge` uses the bundle's default model, or `EVAL_JUDGE_MODEL`. The shape matches a decision model's, for iteration 5 step 10. One live call confirmed it works.
4. **Interactive case.** A case marked `"interactive": true` runs with a scripted user: `run_with_scripted_user` grants Approvals, denies cloud reads and answers questions by pattern. It adds the check `specialists_ask_before_submitting`. `case-04-interactive-web-app` is case 03 run this way.
5. **Thresholds.** A case may set `thresholds` for `revisions`, `seconds` and `tokens`. Exceeding one is printed as `REGRESSION`, listed in the report, and makes the exit code non-zero. It does not change the score.
`automatic_reply` in `app/nodes/execution_nodes.py` lost its leading underscore so the evaluation can reuse it.
Tests: `tests/eval/test_scoring.py`, with a fake judge.
Open:
- **Thresholds are not set.** They need a baseline per case from a completed run of the current code. None exists: see below.
- **No judged or interactive run yet.**
- **Judge stability** across repeated runs on the same output is unmeasured (iteration 5 step 10 names it).
## The evaluation run of 2026-10-06 (stopped, no score)
A three-case run was started on the code as of steps 1 to 6 and stopped by the owner after 35 minutes, during case 01. Cases 02 and 03 never started. Nothing was scored and no files were written, so it gives no baseline. What its log showed:
- Seven Deliverables were submitted on their first attempt in about 15 minutes, including the decomposition and description under their new contracts (step 4).
- The Orchestrator reopened deployment guidance after each failed validation without first asking for Approval of the validation Phase, which is what step 6 intended. Observed three times in one case.
- No Revision of the design recommendations occurred, so step 3 was not exercised.
- Validation did not converge: three passes gave 4, then 7, then 5 different findings, and deployment guidance used all three Revisions. The four findings of the first pass were all still present after the first Revision.
- The Orchestrator stopped without calling a tool four times and was nudged; one such turn took 234 seconds.
The non-convergence led to ADR-0005 (proposed): establish the Target architecture as an explicit graph with numbered Checks before code is written.