feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -9,9 +9,9 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.