feat: keep an unchanged description, list conditions, review coverage with Jev

After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 20:58:38 +01:00
parent f1f6f1b83a
commit 5adc911a82
29 changed files with 1687 additions and 16 deletions

View File

@@ -9,9 +9,9 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–20 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–22 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; case 01 passes with real models; cases 02–04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.

View File

@@ -169,3 +169,20 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
## Step 21: A Deliverable stands while what it was written from is unchanged
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** A Revision upstream does not rewrite a Deliverable it leaves true.
- **Rationale:** In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
- **Where:** `DeliverableSpec.written_from` in `app/workflows/deliverables.py`, `graph_structure` in `app/workflows/chain.py`, `specialist_task` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.
## Step 22: Validation reports every gap in one pass
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** One validation finds the gaps in the Target architecture, not one validation per gap.
- **Rationale:** In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.

View File

@@ -4,7 +4,7 @@
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
**This directory covers:** steps 1 to 20
**This directory covers:** steps 1 to 22
## Sources of truth
@@ -37,6 +37,8 @@
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
| 20 | A model timeout per role | added | S | built |
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -0,0 +1,14 @@
# Step 21 — Implementation Detail
Built on 2026-10-07.
1. `DeliverableSpec.written_from` returns one comparable value for what a Deliverable is written from. The architecture description's is `graph_structure`: the graph's Elements and Relationships, without its Checks.
2. When a Specialist submits, `specialist_task` keeps the submission in the Deliverable's working memory with that value and the feedback it was written under.
3. When the Deliverable is started again and the value and the feedback are the same, no Specialist is run: the kept submission is returned as the attempt's outcome. The Ledger records a submission as for any attempt.
4. A note that a dependency was revised is not counted as feedback on the Deliverable. Anything else is, and has it written again.
Only the architecture description declares `written_from`. The code answers the Checks, so it is not kept; its working copy already makes its Revision cheap (27 and 88 seconds in case 02).
**Unrun:** no evaluation since. The saving expected on a case like 02 is the describer's second and third runs.
Tests: `tests/execution/test_coverage_review.py`, "a description stands while the shape of the design does" (4).

View File

@@ -0,0 +1,11 @@
# Step 22 — Implementation Detail
Changed on 2026-10-07. Instructions only; no code.
- `agents/validator.md`: review coverage once and completely, going through every requirement and recommendation in turn, and report every gap in that pass. After a Revision made for its additions, propose only what is still uncovered, and never what a Check already verifies in other words.
- `agents/target-architect.md`: one Check per requirement is rarely enough; write a Check for every condition a requirement states and every Element it applies to.
- `skills/requirements-chain/SKILL.md`: a "Coverage" rule for the graph, and "find every gap in one pass" for the verdicts.
Also changed: `agents/requirements-analyst.md` and `agents/design-advisor.md` now have every non-functional requirement and recommendation list its `conditions`, which the submission check requires. A review of those conditions by Jev is built and off by default: see `../../iter-05-jev-decisions/steps/step-4.md`.
**Unrun:** an instruction's effect is only known from an evaluation. Case 02 is the comparison: 2 Revisions of the Target architecture, with 1 and then 3 proposed additions.

View File

@@ -2,7 +2,7 @@
**Iteration:** 5, Jev decisions
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
**Status:** steps 2, 11 and 12 built (2026-10-07) and used live in case 02, where Jev judged two gaps; step 4 built in part and switched off; the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
**This directory covers:** steps 1 to 19
## What Jev is
@@ -78,7 +78,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | planned |
| 4 | Semantic checks at submission | J4 | planned |
| 4 | Semantic checks at submission | J4 | a coverage review of the Target architecture built: one choice per listed condition; passes a live test on case 02's graph; off until a full case has run with it (`steps/step-4.md`); other checks planned |
| 5 | Finding-to-Deliverable routing | J2 | planned |
| 6 | Model tier per delegation | J5 | planned |
| 7 | Context selection | J6 | planned |

View File

@@ -0,0 +1,50 @@
# Step 4 — Implementation Detail
Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.
What was built:
1. A seam: `SubmissionReview` in `app/adk/runtime.py`, passed through `ExecutionEngine`, `build_execution_graph` and `make_execution_nodes` to `specialist_tools`. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
2. `CoverageReview` in `app/tools/coverage.py`, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
3. `JEV_COVERAGE_REVIEW`, default `false`. The assembly installs the review only when it is `true` and a Jev key is set.
## Why the first form failed
Case 02 (`eval/results/20261007T190702Z/`) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:
| Question asked | On the complete graph | On the graph with 4 known gaps |
|---|---|---|
| "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 |
| "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 |
| Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged |
Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.
## The form that works: one choice per condition
The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.
1. Requirements and recommendations list their `conditions`: one entry for each separate thing that must be true. `check_requirements` and `check_recommendations` refuse a non-functional requirement or a recommendation without them. `conditions_of` reads an older item as one condition, its text.
2. `CoverageReview` asks, per condition, "which one Check verifies this condition?" with that source's Check ids and `none` as the options. A condition answered `none` with a probability of at least `JEV_COVERAGE_THRESHOLD` (0.6) is raised, as `NFR-1: no Check verifies "<condition>" (its Checks: ...)`.
Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):
| | Complete graph | Graph with the 4 gaps |
|---|---|---|
| The 4 gap conditions | a Check chosen each time | `none` each time, probability 0.72 to 1.00 |
| The 5 covered conditions | a Check chosen; `none` at most 0.40 | a Check chosen; `none` at most 0.45 |
| Raised at 0.6 | 0 | 4 of 4, and nothing else |
Each call: about 0.25 seconds and 2,000 tokens. `tests/execution/test_coverage_live.py` repeats this through `CoverageReview` itself and is opt-in (`LIVE_JEV=1`); it passed on 2026-10-07.
Still off (`JEV_COVERAGE_REVIEW=false`), because three things are unproven:
- The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
- Nine conditions from one case is a small sample; one covered condition scored `none` at 0.45 against a threshold of 0.6.
- A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.
Turn it on for a full case before relying on it.
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.

View File

@@ -2,7 +2,7 @@
**Iteration:** 6, the Target architecture graph
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
**Status:** steps 1 to 10 built and covered by the deterministic suite; case 01 completed at 100% with no Revision on 2026-10-07 (`steps/step-11.md`); cases 02 to 04 have not run on this build
**Status:** steps 1 to 10 built and covered by the deterministic suite; cases 01 and 02 completed at 100% on 2026-10-07 (`steps/step-11.md`), case 02 after two Revisions of the Target architecture; cases 03 and 04 have not run on this build
**This directory covers:** steps 1 to 11
## Order of work
@@ -56,7 +56,7 @@ flowchart TD
| 8 | Deployment guidance implements the graph | M | complete |
| 9 | Validation gives a verdict per Check | M | complete |
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
| 11 | Evaluation: prove the loop converges | M | case 01 passes (100%, no Revision); cases 02 to 04 and the thresholds outstanding |
| 11 | Evaluation: prove the loop converges | M | cases 01 and 02 pass at 100%; cases 03 and 04 and the thresholds outstanding |
Size: S under a day, M a few days.

View File

@@ -79,3 +79,17 @@ What went wrong: the target architect exhausted its 30 tool calls on its first a
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.
## Run of 2026-10-07, 19:07 UTC (`eval/results/20261007T190702Z/`)
Case 02, on commit `f1f6f1b`. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.
- The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
- The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
- The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
- The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
- By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.
Changed after the run, not yet run: iteration 4 steps 21 and 22.
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.