feat: keep an unchanged description, list conditions, review coverage with Jev

After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 20:58:38 +01:00
parent f1f6f1b83a
commit 5adc911a82
29 changed files with 1687 additions and 16 deletions

View File

@@ -169,3 +169,20 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
## Step 21: A Deliverable stands while what it was written from is unchanged
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** A Revision upstream does not rewrite a Deliverable it leaves true.
- **Rationale:** In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
- **Where:** `DeliverableSpec.written_from` in `app/workflows/deliverables.py`, `graph_structure` in `app/workflows/chain.py`, `specialist_task` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.
## Step 22: Validation reports every gap in one pass
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
- **Capability:** One validation finds the gaps in the Target architecture, not one validation per gap.
- **Rationale:** In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.

View File

@@ -4,7 +4,7 @@
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
**This directory covers:** steps 1 to 20
**This directory covers:** steps 1 to 22
## Sources of truth
@@ -37,6 +37,8 @@
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
| 20 | A model timeout per role | added | S | built |
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -0,0 +1,14 @@
# Step 21 — Implementation Detail
Built on 2026-10-07.
1. `DeliverableSpec.written_from` returns one comparable value for what a Deliverable is written from. The architecture description's is `graph_structure`: the graph's Elements and Relationships, without its Checks.
2. When a Specialist submits, `specialist_task` keeps the submission in the Deliverable's working memory with that value and the feedback it was written under.
3. When the Deliverable is started again and the value and the feedback are the same, no Specialist is run: the kept submission is returned as the attempt's outcome. The Ledger records a submission as for any attempt.
4. A note that a dependency was revised is not counted as feedback on the Deliverable. Anything else is, and has it written again.
Only the architecture description declares `written_from`. The code answers the Checks, so it is not kept; its working copy already makes its Revision cheap (27 and 88 seconds in case 02).
**Unrun:** no evaluation since. The saving expected on a case like 02 is the describer's second and third runs.
Tests: `tests/execution/test_coverage_review.py`, "a description stands while the shape of the design does" (4).

View File

@@ -0,0 +1,11 @@
# Step 22 — Implementation Detail
Changed on 2026-10-07. Instructions only; no code.
- `agents/validator.md`: review coverage once and completely, going through every requirement and recommendation in turn, and report every gap in that pass. After a Revision made for its additions, propose only what is still uncovered, and never what a Check already verifies in other words.
- `agents/target-architect.md`: one Check per requirement is rarely enough; write a Check for every condition a requirement states and every Element it applies to.
- `skills/requirements-chain/SKILL.md`: a "Coverage" rule for the graph, and "find every gap in one pass" for the verdicts.
Also changed: `agents/requirements-analyst.md` and `agents/design-advisor.md` now have every non-functional requirement and recommendation list its `conditions`, which the submission check requires. A review of those conditions by Jev is built and off by default: see `../../iter-05-jev-decisions/steps/step-4.md`.
**Unrun:** an instruction's effect is only known from an evaluation. Case 02 is the comparison: 2 Revisions of the Target architecture, with 1 and then 3 proposed additions.