After case 02 (completed at 100%, but with two Revisions of the Target architecture for gaps validation found one pass at a time): - A Deliverable stands while what it was written from is unchanged. The architecture description is written from the Elements and Relationships, so a Revision that only adds or rewords Checks no longer runs the describer. - Non-functional requirements and recommendations list their conditions, one entry per separate thing that must be true; submission requires them. - A coverage review at submission (JEV_COVERAGE_REVIEW, off by default): for each condition, Jev chooses which Check verifies it, or none. Asked as one broad yes-or-no per requirement it could not tell a complete design from one with known gaps; asked as a choice per condition it raised all four known gaps of case 02 and nothing on the complete graph. An opt-in live test (LIVE_JEV=1) repeats that. - The validator is told to report every gap in its first pass. Plans record the case 02 run and both forms of the review with figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.4 KiB
Step 4 — Implementation Detail
Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.
What was built:
- A seam:
SubmissionReviewinapp/adk/runtime.py, passed throughExecutionEngine,build_execution_graphandmake_execution_nodestospecialist_tools. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission. CoverageReviewinapp/tools/coverage.py, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.JEV_COVERAGE_REVIEW, defaultfalse. The assembly installs the review only when it istrueand a Jev key is set.
Why the first form failed
Case 02 (eval/results/20261007T190702Z/) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:
| Question asked | On the complete graph | On the graph with 4 known gaps |
|---|---|---|
| "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 |
| "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 |
| Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged |
Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.
The form that works: one choice per condition
The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.
- Requirements and recommendations list their
conditions: one entry for each separate thing that must be true.check_requirementsandcheck_recommendationsrefuse a non-functional requirement or a recommendation without them.conditions_ofreads an older item as one condition, its text. CoverageReviewasks, per condition, "which one Check verifies this condition?" with that source's Check ids andnoneas the options. A condition answerednonewith a probability of at leastJEV_COVERAGE_THRESHOLD(0.6) is raised, asNFR-1: no Check verifies "<condition>" (its Checks: ...).
Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):
| Complete graph | Graph with the 4 gaps | |
|---|---|---|
| The 4 gap conditions | a Check chosen each time | none each time, probability 0.72 to 1.00 |
| The 5 covered conditions | a Check chosen; none at most 0.40 |
a Check chosen; none at most 0.45 |
| Raised at 0.6 | 0 | 4 of 4, and nothing else |
Each call: about 0.25 seconds and 2,000 tokens. tests/execution/test_coverage_live.py repeats this through CoverageReview itself and is opt-in (LIVE_JEV=1); it passed on 2026-10-07.
Still off (JEV_COVERAGE_REVIEW=false), because three things are unproven:
- The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
- Nine conditions from one case is a small sample; one covered condition scored
noneat 0.45 against a threshold of 0.6. - A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.
Turn it on for a full case before relying on it.
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
Tests: tests/execution/test_coverage_review.py, with a scripted decider.