feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target architecture for gaps validation found one pass at a time): - A Deliverable stands while what it was written from is unchanged. The architecture description is written from the Elements and Relationships, so a Revision that only adds or rewords Checks no longer runs the describer. - Non-functional requirements and recommendations list their conditions, one entry per separate thing that must be true; submission requires them. - A coverage review at submission (JEV_COVERAGE_REVIEW, off by default): for each condition, Jev chooses which Check verifies it, or none. Asked as one broad yes-or-no per requirement it could not tell a complete design from one with known gaps; asked as a choice per condition it raised all four known gaps of case 02 and nothing on the complete graph. An opt-in live test (LIVE_JEV=1) repeats that. - The validator is told to report every gap in its first pass. Plans record the case 02 run and both forms of the review with figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
50
.agents/plans/iter-05-jev-decisions/steps/step-4.md
Normal file
50
.agents/plans/iter-05-jev-decisions/steps/step-4.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.
|
||||
|
||||
What was built:
|
||||
|
||||
1. A seam: `SubmissionReview` in `app/adk/runtime.py`, passed through `ExecutionEngine`, `build_execution_graph` and `make_execution_nodes` to `specialist_tools`. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
|
||||
2. `CoverageReview` in `app/tools/coverage.py`, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
|
||||
3. `JEV_COVERAGE_REVIEW`, default `false`. The assembly installs the review only when it is `true` and a Jev key is set.
|
||||
|
||||
## Why the first form failed
|
||||
|
||||
Case 02 (`eval/results/20261007T190702Z/`) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:
|
||||
|
||||
| Question asked | On the complete graph | On the graph with 4 known gaps |
|
||||
|---|---|---|
|
||||
| "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 |
|
||||
| "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 |
|
||||
| Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged |
|
||||
|
||||
Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.
|
||||
|
||||
## The form that works: one choice per condition
|
||||
|
||||
The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.
|
||||
|
||||
1. Requirements and recommendations list their `conditions`: one entry for each separate thing that must be true. `check_requirements` and `check_recommendations` refuse a non-functional requirement or a recommendation without them. `conditions_of` reads an older item as one condition, its text.
|
||||
2. `CoverageReview` asks, per condition, "which one Check verifies this condition?" with that source's Check ids and `none` as the options. A condition answered `none` with a probability of at least `JEV_COVERAGE_THRESHOLD` (0.6) is raised, as `NFR-1: no Check verifies "<condition>" (its Checks: ...)`.
|
||||
|
||||
Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):
|
||||
|
||||
| | Complete graph | Graph with the 4 gaps |
|
||||
|---|---|---|
|
||||
| The 4 gap conditions | a Check chosen each time | `none` each time, probability 0.72 to 1.00 |
|
||||
| The 5 covered conditions | a Check chosen; `none` at most 0.40 | a Check chosen; `none` at most 0.45 |
|
||||
| Raised at 0.6 | 0 | 4 of 4, and nothing else |
|
||||
|
||||
Each call: about 0.25 seconds and 2,000 tokens. `tests/execution/test_coverage_live.py` repeats this through `CoverageReview` itself and is opt-in (`LIVE_JEV=1`); it passed on 2026-10-07.
|
||||
|
||||
Still off (`JEV_COVERAGE_REVIEW=false`), because three things are unproven:
|
||||
|
||||
- The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
|
||||
- Nine conditions from one case is a small sample; one covered condition scored `none` at 0.45 against a threshold of 0.6.
|
||||
- A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.
|
||||
|
||||
Turn it on for a full case before relying on it.
|
||||
|
||||
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.
|
||||
Reference in New Issue
Block a user