feat: keep an unchanged description, list conditions, review coverage with Jev

After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 20:58:38 +01:00
parent f1f6f1b83a
commit 5adc911a82
29 changed files with 1687 additions and 16 deletions

View File

@@ -0,0 +1,50 @@
# Step 4 — Implementation Detail
Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.
What was built:
1. A seam: `SubmissionReview` in `app/adk/runtime.py`, passed through `ExecutionEngine`, `build_execution_graph` and `make_execution_nodes` to `specialist_tools`. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
2. `CoverageReview` in `app/tools/coverage.py`, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
3. `JEV_COVERAGE_REVIEW`, default `false`. The assembly installs the review only when it is `true` and a Jev key is set.
## Why the first form failed
Case 02 (`eval/results/20261007T190702Z/`) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:
| Question asked | On the complete graph | On the graph with 4 known gaps |
|---|---|---|
| "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 |
| "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 |
| Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged |
Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.
## The form that works: one choice per condition
The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.
1. Requirements and recommendations list their `conditions`: one entry for each separate thing that must be true. `check_requirements` and `check_recommendations` refuse a non-functional requirement or a recommendation without them. `conditions_of` reads an older item as one condition, its text.
2. `CoverageReview` asks, per condition, "which one Check verifies this condition?" with that source's Check ids and `none` as the options. A condition answered `none` with a probability of at least `JEV_COVERAGE_THRESHOLD` (0.6) is raised, as `NFR-1: no Check verifies "<condition>" (its Checks: ...)`.
Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):
| | Complete graph | Graph with the 4 gaps |
|---|---|---|
| The 4 gap conditions | a Check chosen each time | `none` each time, probability 0.72 to 1.00 |
| The 5 covered conditions | a Check chosen; `none` at most 0.40 | a Check chosen; `none` at most 0.45 |
| Raised at 0.6 | 0 | 4 of 4, and nothing else |
Each call: about 0.25 seconds and 2,000 tokens. `tests/execution/test_coverage_live.py` repeats this through `CoverageReview` itself and is opt-in (`LIVE_JEV=1`); it passed on 2026-10-07.
Still off (`JEV_COVERAGE_REVIEW=false`), because three things are unproven:
- The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
- Nine conditions from one case is a small sample; one covered condition scored `none` at 0.45 against a threshold of 0.6.
- A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.
Turn it on for a full case before relying on it.
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.