feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target architecture for gaps validation found one pass at a time): - A Deliverable stands while what it was written from is unchanged. The architecture description is written from the Elements and Relationships, so a Revision that only adds or rewords Checks no longer runs the describer. - Non-functional requirements and recommendations list their conditions, one entry per separate thing that must be true; submission requires them. - A coverage review at submission (JEV_COVERAGE_REVIEW, off by default): for each condition, Jev chooses which Check verifies it, or none. Asked as one broad yes-or-no per requirement it could not tell a complete design from one with known gaps; asked as a choice per condition it raised all four known gaps of case 02 and nothing on the complete graph. An opt-in live test (LIVE_JEV=1) repeats that. - The validator is told to report every gap in its first pass. Plans record the case 02 run and both forms of the review with figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -169,3 +169,20 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
|
||||
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
|
||||
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
|
||||
|
||||
## Step 21: A Deliverable stands while what it was written from is unchanged
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
|
||||
- **Capability:** A Revision upstream does not rewrite a Deliverable it leaves true.
|
||||
- **Rationale:** In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
|
||||
- **Where:** `DeliverableSpec.written_from` in `app/workflows/deliverables.py`, `graph_structure` in `app/workflows/chain.py`, `specialist_task` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.
|
||||
|
||||
## Step 22: Validation reports every gap in one pass
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
|
||||
- **Capability:** One validation finds the gaps in the Target architecture, not one validation per gap.
|
||||
- **Rationale:** In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
|
||||
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 20
|
||||
**This directory covers:** steps 1 to 22
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -37,6 +37,8 @@
|
||||
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-21.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-21.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 21 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `DeliverableSpec.written_from` returns one comparable value for what a Deliverable is written from. The architecture description's is `graph_structure`: the graph's Elements and Relationships, without its Checks.
|
||||
2. When a Specialist submits, `specialist_task` keeps the submission in the Deliverable's working memory with that value and the feedback it was written under.
|
||||
3. When the Deliverable is started again and the value and the feedback are the same, no Specialist is run: the kept submission is returned as the attempt's outcome. The Ledger records a submission as for any attempt.
|
||||
4. A note that a dependency was revised is not counted as feedback on the Deliverable. Anything else is, and has it written again.
|
||||
|
||||
Only the architecture description declares `written_from`. The code answers the Checks, so it is not kept; its working copy already makes its Revision cheap (27 and 88 seconds in case 02).
|
||||
|
||||
**Unrun:** no evaluation since. The saving expected on a case like 02 is the describer's second and third runs.
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, "a description stands while the shape of the design does" (4).
|
||||
11
.agents/plans/iter-04-learning-and-skills/steps/step-22.md
Normal file
11
.agents/plans/iter-04-learning-and-skills/steps/step-22.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Step 22 — Implementation Detail
|
||||
|
||||
Changed on 2026-10-07. Instructions only; no code.
|
||||
|
||||
- `agents/validator.md`: review coverage once and completely, going through every requirement and recommendation in turn, and report every gap in that pass. After a Revision made for its additions, propose only what is still uncovered, and never what a Check already verifies in other words.
|
||||
- `agents/target-architect.md`: one Check per requirement is rarely enough; write a Check for every condition a requirement states and every Element it applies to.
|
||||
- `skills/requirements-chain/SKILL.md`: a "Coverage" rule for the graph, and "find every gap in one pass" for the verdicts.
|
||||
|
||||
Also changed: `agents/requirements-analyst.md` and `agents/design-advisor.md` now have every non-functional requirement and recommendation list its `conditions`, which the submission check requires. A review of those conditions by Jev is built and off by default: see `../../iter-05-jev-decisions/steps/step-4.md`.
|
||||
|
||||
**Unrun:** an instruction's effect is only known from an evaluation. Case 02 is the comparison: 2 Revisions of the Target architecture, with 1 and then 3 proposed additions.
|
||||
Reference in New Issue
Block a user