feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -9,9 +9,9 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.

View File

@@ -204,3 +204,27 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
## Step 25: An Execution that ends with warnings keeps its files
- **Report ref:** none; found by the runs of 2026-10-07 and 08
- **Capability:** What an unfinished Execution produced can be read afterwards.
- **Rationale:** Only a completed Execution wrote its Deliverables. Case 04's first run ended with warnings after 53 minutes and left nothing on disk: nine Deliverables' worth of work, and the evidence of what went wrong, existed only in the report's conversation log.
- **Where:** `_write_outputs` and `end_with_warnings` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** An Execution that ends with warnings writes every Deliverable it holds, with a manifest that carries the warnings, each Deliverable's status and each Phase's status, so nothing unapproved reads as final. The case's history points to the files.
## Step 26: Each pillar's checklist is reviewed and the review is in the Deliverable
- **Report ref:** none; found by the judged runs of 2026-10-08
- **Capability:** An architect sees every item of each pillar's checklist accounted for: addressed, and how, or not applicable, and why.
- **Rationale:** The judged check `pillar_checklists_covered` failed in all four cases of the judged batch, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The six checklists hold 78 items; a run produces 11 to 14 recommendations, each one configurable decision. Twelve recommendations cannot answer 78 items, many of which are an organisation's process and not a design decision, so the check could not pass and nothing told an architect which items had been considered.
- **Where:** `app/tools/pillars.py`, `submit_deliverable` in `app/tools/execution.py`, `agents/pillar-advisor.md`, `agents/design-advisor.md`.
- **Acceptance criteria:** Each pillar advisor returns, with its advice, a status and a note for every item of its Skill's checklist. The reviews are kept with the advice across Revisions and written into the design recommendations by code, once. On a judged run `pillar_checklists_covered` passes.
## Step 27: A user sees a Deliverable before it is first submitted
- **Report ref:** none; found by evaluation case 04 on 2026-10-08
- **Capability:** With a user present, no Deliverable is submitted that they have not been shown.
- **Rationale:** The Specialist protocol tells every Specialist to confirm its draft with the user before submitting. In case 04 the validator did not (first run), and the target architect did not after running out of tool calls on an earlier attempt (batch A). An instruction was not enough.
- **Where:** `submit_deliverable` in `app/tools/execution.py`, `specialist_task` in `app/nodes/execution_nodes.py`, `CONFIRM_DRAFTS` in `app/config.py`.
- **Acceptance criteria:** In an Execution with a user, a Specialist's submission is refused until the user has replied to it about that Deliverable at least once, and the refusal says how to ask. A pre-approved Execution is unaffected. It can be switched off.

View File

@@ -4,7 +4,7 @@
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 complete (ADR-0004 accepted 2026-10-08), which unblocks steps 10 and 12; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
**This directory covers:** steps 1 to 24
**This directory covers:** steps 1 to 27
## Sources of truth
@@ -19,12 +19,12 @@
|---|---|---|---|---|
| 1 | Structured validator findings | A | S | replaced by iteration 6 step 9 (verdicts per Check) |
| 2 | Carry the draft through `needs_input` | A | S | complete |
| 3 | Reuse pillar advice across Revisions | A | S | built; evaluation run outstanding |
| 3 | Reuse pillar advice across Revisions | A | S | built; no run has revised design recommendations, so unexercised by evaluation |
| 4 | Single source for Deliverable contracts | A | S | complete |
| 5 | Cap parallel Specialists | A | S | complete |
| 6 | Orchestrator step order | A | S | built; seen working in one stopped run, no completed evaluation |
| 7 | Capture lesson candidates | B | M | complete |
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
| 8 | Evaluation that can see improvement | B | M | complete: thresholds set and four judged runs done on 2026-10-08 |
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
| 10 | Lessons Skill and lesson curator | B | L | planned |
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
@@ -41,6 +41,9 @@
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; a hole found by a run and closed; not rerun |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -10,4 +10,7 @@ Differs from the plan: the plan's acceptance is a fall in IaC author Revisions o
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
Not done: Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
Added on 2026-10-08: `references/load-balancer.md`, a fourteenth pattern: an external Application Load Balancer in front of Cloud Run with a TLS policy, a firewall policy, Identity-Aware Proxy and an HTTP redirect. Cases 02, 03 and 04 all used a load balancer with no pattern, and one run put `ssl_policy` on a backend service, which has no such setting; the reference says which link of the chain each setting lives on. It validates against the pinned provider.

View File

@@ -0,0 +1,14 @@
# Step 25 — Implementation Detail
Built on 2026-10-08.
1. `_write_outputs` takes the Execution's final status and its Ledger. `end_with_warnings` now calls it, as `finalize` does.
2. `execution.json` gains `status` (`completed` or `completed_with_warnings`), `warnings`, `phases` (each Phase's status), and a `status` on every Deliverable: `submitted`, or `draft` when something it depends on was reopened after it was written.
3. With no Deliverable at all nothing is written, and the case's history records no files, as before.
4. The solution guide is written only when it exists; an Execution that stopped before packaging has none, and the manifest says so.
Not covered: an Execution whose process is killed. It never reaches either end node, so it still writes nothing. The evaluation's progress file (iteration 6 step 11) records its time and tokens, not its Deliverables.
Also on 2026-10-08, outside any step: the CI workflow installs `requirements.txt` and `requirements-dev.txt` before linting and testing. It installed only ruff, mypy and pytest, so its test step could not import the application.
Tests: `tests/execution/test_engine.py`, two updated.

View File

@@ -0,0 +1,17 @@
# Step 26 — Implementation Detail
Built on 2026-10-08.
1. `submit_advice` takes a `checklist`: one entry per item of the pillar Skill's validation checklist, with `status` (`addressed`, `not_applicable` or `not_addressed`) and a `note`. An entry with no item or another status is dropped.
2. The consultation keeps each pillar's review in the Deliverable's working memory (`pillar_checklists`) beside its advice, so a Revision that re-consults some pillars keeps the others' reviews. The design advisor is given the advice only, not the review.
3. `render_checklist_review` writes the reviews as one section, "Architecture framework checklist review", a table per pillar. `submit_deliverable` appends it to the design recommendations and replaces an earlier copy, so a Revision never stacks two.
4. A Skill with no checklist gets no review and the Deliverable no section: the mechanism is the provider's Skills' own, not Google Cloud's.
Checked on 2026-10-08 with two real advisors on case 03's brief: reliability returned all 8 of its items (6 addressed, 2 not applicable) and cost optimisation all 11 (11 addressed), in 74 seconds for the pair.
Two limits:
- A review says what the advisor's own advice covers. The design advisor reconciles the pillars and may leave something out, so "addressed" there is the advisor's claim, not a check of the final recommendations.
- A judged run of case 04 on 2026-10-08 scored `pillar_checklists_covered` at 100% on all six pillars, against 9% to 45% before. One run.
Tests: `tests/execution/test_pillars.py`, "each advisor reviews its pillar's checklist" (3).

View File

@@ -0,0 +1,13 @@
# Step 27 — Implementation Detail
Built on 2026-10-08.
1. `specialist_tools` takes `confirm_first`. While it is set, `submit_deliverable` checks the submission as usual, keeps the working copy, runs any review, and then refuses: the user has not seen this Deliverable; call `needs_input` with a summary and pass the content as `draft`.
2. `specialist_task` sets it when the Execution is not pre-approved and the user has not been shown a draft of the Deliverable. The Orchestrator's turn records a Deliverable in `drafts_shown` when a Specialist's question carries a draft. Once shown, submission is allowed, so a Revision is not asked about again.
The first version opened the rule once the user had answered any question about the Deliverable. The verification run of 2026-10-08 showed the hole: the target architect ran out of tool calls, the Orchestrator asked the user how to proceed, and that answer let the next attempt submit unseen.
3. `CONFIRM_DRAFTS` (default `true`) turns the rule on in the assembly. The engine's own default is off, so an engine built directly, as in tests, behaves as before.
**No interactive evaluation has run since the second version.** The case 04 check `specialists_ask_before_submitting` is the measure.
Tests: `tests/execution/test_engine.py`, "a user sees a Deliverable before it is first submitted" (3).

View File

@@ -78,7 +78,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | planned |
| 4 | Semantic checks at submission | J4 | a coverage review of the Target architecture built: one choice per listed condition; passes a live test on case 02's graph; off until a full case has run with it (`steps/step-4.md`); other checks planned |
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
| 5 | Finding-to-Deliverable routing | J2 | planned |
| 6 | Model tier per delegation | J5 | planned |
| 7 | Context selection | J6 | planned |
@@ -86,11 +86,11 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 9 | In-run context compaction | J10 | planned |
| 10 | Evaluation rubric and lesson triage | J8 | planned |
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
| 13 | Cloud-read call guard | J7 | planned |
| 14 | Input guardrail | J11 | planned |
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned |
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |

View File

@@ -49,3 +49,30 @@ Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04"
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
## Finished on 2026-10-08: the third case 04 fault, and intake
**A reply that restates what is agreed is an answer.** `_read_message` now shows the decider the agreed `requirements.json` and asks one more noul, `already_agreed`: does the message only confirm or repeat what those requirements say? At 0.75 or more the reply is closed as an answer, whatever `asks_change` says. Measured against live Jev with requirements holding the facts being restated:
| Reply | asks_change | already_agreed | Disposition |
|---|---|---|---|
| Restates the GDPR and secrets requirement | 0.12 | 0.78 | close |
| Restates the availability requirement | 0.26 | 0.48 | queue (asked once, then let go) |
| Restates the retention requirement | 0.52 | 0.81 | close |
| "Approved. Please proceed." | 0.04 | 0.70 | close |
| Raises availability to 99.99% | 0.85 | 0.01 | act |
| Changes retention from 2 years to 7 | 0.89 | 0.01 | act |
| Adds SMS notifications and a mobile app | 0.85 | 0.01 | act |
| Changes secret rotation to 30 days | 0.87 | 0.02 | act |
**Intake: a message that starts something is read first.** `app/nodes/intake.py` asks one choice of a message that arrives with no Execution in progress: new request, change request, or question. The A2A server (`app/a2a_server.py`) acts on it:
- a question, at 0.75 confidence or more, is answered with a short statement of what the agent does and starts nothing;
- a change request, when the Customer case has a completed Execution, is started as a new request with a note saying Change requests are not available yet (iteration 7 builds them);
- anything else, any reading that is not confident, and every pre-approved task starts an Execution as before.
Measured against live Jev on ten messages: two new requests, two questions, two change requests with an earlier Execution, a question about earlier work, a different new request for the same case, a change with nothing delivered (taken as new), and "hello" (0.53, taken as new). All ten as intended.
Not built: acting on a change request (iteration 7 step 2), and answering a question about earlier work from that work.
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).

View File

@@ -0,0 +1,20 @@
# Step 16 — Implementation Detail
The judge (`eval/judge.py`: `DeciderJudge` for Jev, `GeminiJudge` for a language model) was built earlier. Built on 2026-10-08: the calibration.
1. `eval/datasets/judge/` holds two saved design recommendations (cases 02 and 03, from runs that passed) and `labels.json`: 32 questions about them, 12 checklist items and the 4 quality-rubric questions for each document, with `label` empty.
2. `eval/calibrate_judge.py` asks both judges every question several times and reports, per judge, stability (the same answer to the same question across runs) and agreement with the labels, and agreement between the two judges. `--build` regenerates the question set. A new Jev client is used for each run, because the client caches identical requests and a cached answer would make stability look perfect.
Measured on 2026-10-08, three runs, no labels yet (`eval/datasets/judge/calibration.json`):
| | Jev | Gemini (the bundle's default model) |
|---|---|---|
| Stability | 1.00 | 0.92 |
| Mean answer | 0.37 | 0.32 |
| Agreement between the two | 0.91 | |
So Jev gave the same answer every time, the language-model judge changed its answer on about one question in twelve, and the two agree on nine in ten.
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).

View File

@@ -48,3 +48,14 @@ Turn it on for a full case before relying on it.
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.
## Added on 2026-10-08
**What code cannot show is listed apart.** The first full run with the coverage review raised conditions such as an availability percentage, which no Check can verify. A requirement or recommendation now lists under `conditions` only what infrastructure configuration can show, and under `outside_code` what it cannot: an outcome measured in operation, behaviour of the application's own code, a process. `check_requirements` and `check_recommendations` require at least one entry between the two. `chain_gaps` asks for no Check from a constraint with `"conditions": []` and something under `outside_code`. The target architect writes no Check for `outside_code`; the validator proposes nothing for it and names it as something to verify after deployment.
Measured before the split, on the designs of two runs that passed validation with nothing proposed, using the conditions the agents themselves wrote: the coverage review raised 14 of 66 (case 02) and 8 of 51 (case 03). About half were of the kind the split removes (availability, recovery time, throughput, latency, what the application logs). The rest look like real gaps the validator did not report (three zones, cost labels, a load balancer). It has not been measured since the split, which needs new runs.
**Requirements name no products.** `RequirementsReview` in `app/tools/requirements_review.py` asks one noul per requirement: does it name a cloud provider's product or managed service? It is a third review on the same seam, behind `JEV_REQUIREMENTS_REVIEW` (off), with its own threshold, `JEV_REQUIREMENTS_THRESHOLD`, 0.3. Measured against live Jev: across the 61 requirements of three passed runs the highest answer was 0.10, so nothing was raised; requirements written to name a product scored 0.93 (Cloud Run), 0.61 (Secret Manager) and 0.34 (a Pub/Sub topic), and a database engine or a region scored 0.10 or less. At 0.3 that is three of three caught and none of 65 raised wrongly. Three written examples is a small sample.
Tests: `tests/execution/test_coverage_review.py` (2 more), `tests/execution/test_requirements_chain.py` (2 more).

View File

@@ -2,7 +2,7 @@
**Iteration:** 6, the Target architecture graph
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
**Status:** steps 1 to 10 built; cases 01, 02 and 03 completed at 100% with real models on 2026-10-07 and 08 (`steps/step-11.md`); case 04, the interactive one, does not complete
**Status:** complete. Steps 1 to 10 built; all four cases, the interactive one included, completed twice with real models on 2026-10-08 and the thresholds are set (`steps/step-11.md`)
**This directory covers:** steps 1 to 11
## Order of work
@@ -56,7 +56,7 @@ flowchart TD
| 8 | Deployment guidance implements the graph | M | complete |
| 9 | Validation gives a verdict per Check | M | complete |
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding |
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
Size: S under a day, M a few days.

View File

@@ -162,3 +162,51 @@ Not yet changed:
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
## Changes to the evaluation after the unfinished run (2026-10-08)
- A scripted answer may name who it is for: `[pattern, answer, asker]`. Case 04's fourteen facts are tagged for `requirements`; any other Specialist, and the Orchestrator, gets "Approved. Please proceed." (`automatic_reply`, `eval/datasets/solution_cases.json`).
- The evaluation writes `<case>-progress.json` in the run folder after every agent run: seconds, runs, outcomes, tokens and the timeline so far. A run that is cut off leaves it behind with `"finished": false`.
## Runs of 2026-10-08, 14:25 UTC: all four cases, twice (`eval/results/20261008T142509Z/` and `...142514Z/`)
Two batches run side by side on the build with the case 04 changes, intake and the `outside_code` split (uncommitted at the time). Batch B added the judged checks. All eight runs completed; none ended with warnings.
| Case | Batch A | Batch B (judged) |
|---|---|---|
| 01 | 1.00, 31 min, 3.42 M tokens, 1 Revision | 0.92, 39 min, 4.06 M, 1 Revision |
| 02 | 1.00, 47 min, 7.02 M, 3 Revisions | 0.92, 24 min, 3.12 M, none |
| 03 | 1.00, 41 min, 6.33 M, 2 Revisions | 0.92, 29 min, 4.06 M, 1 Revision |
| 04 | 0.92, 45 min, 5.88 M, 1 Revision | 0.93, 33 min, 3.63 M, 1 Revision |
Estimated cost: $22.16 for batch A and $16.40 for batch B. Running two batches at once shares the machine and the model quota, so these times are longer than a case run alone.
- **Case 04 completes.** Both runs went through all five Phases with every draft confirmed by the user and no reply misread as a change. One Revision each: the code for a failed Check in batch B, the design for a proposed addition in batch A.
- **Every structural check passed in all eight runs**, with no model timeout, no loop and no `terraform validate` failure.
- **`pillar_checklists_covered` failed in all four judged runs**, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The cause is structural: 78 checklist items against 11 to 14 recommendations. Iteration 4 step 26 was built in answer; no judged run has happened since.
- **`specialists_ask_before_submitting` failed once**, in batch A's case 04: the target architect exhausted its 45 tool calls, and on its second attempt submitted without asking. The first submission with a user present is now refused until the draft has been put to them (`CONFIRM_DRAFTS`, on by default).
- **The same case varies widely.** Case 02 took 24 minutes with no Revision in one batch and 47 minutes with three in the other, at the same time on the same build. Revisions of the Target architecture for additions validation proposes remain the main cost.
### Thresholds
Set in `eval/datasets/solution_cases.json` from every completed run of each case on this build, at about a quarter above the worst seen:
| Case | Revisions | Seconds | Tokens |
|---|---|---|---|
| 01 | 2 | 3,000 | 5.0 M |
| 02 | 3 | 3,600 | 8.5 M |
| 03 | 3 | 3,200 | 8.0 M |
| 04 | 2 | 3,400 | 7.5 M |
They are alarms for a run that is clearly worse than any seen, not targets. With two to four runs a case and this much spread they are loose; tighten them as runs accumulate.
Acceptance criteria: met. All four cases complete and score at least 0.8; deployment guidance needed at most one Revision in every run; the thresholds are set. One criterion is met only loosely: the set of gaps did not always shrink from one validation to the next (case 02, 2026-10-07).
## Run of 2026-10-08, 17:20 UTC (`eval/results/20261008T172049Z/`): case 04, judged
To verify two changes made after the batches began: the pillar checklist review (iteration 4 step 26) and the confirm-first rule (step 27). Completed, 86% (12 of 14 checks), no Revision, 36 minutes, 4.10 million tokens.
- **`pillar_checklists_covered` passes: 100% on all six pillars**, against 9% to 45% in batch B's case 04. The review is in the Deliverable.
- `recommendations_quality` passed at 0.80 (specific 0.84, actionable 0.81, trade-offs 0.90, traceable 0.66).
- **`specialists_ask_before_submitting` failed again**, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (`drafts_shown`). Not rerun.
- **`requirements_name_no_products` failed** for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here.