# Step 4 — Implementation Detail Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here. What was built: 1. A seam: `SubmissionReview` in `app/adk/runtime.py`, passed through `ExecutionEngine`, `build_execution_graph` and `make_execution_nodes` to `specialist_tools`. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission. 2. `CoverageReview` in `app/tools/coverage.py`, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state. 3. `JEV_COVERAGE_REVIEW`, default `false`. The assembly installs the review only when it is `true` and a Jev key is set. ## Why the first form failed Case 02 (`eval/results/20261007T190702Z/`) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed: | Question asked | On the complete graph | On the graph with 4 known gaps | |---|---|---| | "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 | | "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 | | Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged | Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens. ## The form that works: one choice per condition The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no. 1. Requirements and recommendations list their `conditions`: one entry for each separate thing that must be true. `check_requirements` and `check_recommendations` refuse a non-functional requirement or a recommendation without them. `conditions_of` reads an older item as one condition, its text. 2. `CoverageReview` asks, per condition, "which one Check verifies this condition?" with that source's Check ids and `none` as the options. A condition answered `none` with a probability of at least `JEV_COVERAGE_THRESHOLD` (0.6) is raised, as `NFR-1: no Check verifies "" (its Checks: ...)`. Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered): | | Complete graph | Graph with the 4 gaps | |---|---|---| | The 4 gap conditions | a Check chosen each time | `none` each time, probability 0.72 to 1.00 | | The 5 covered conditions | a Check chosen; `none` at most 0.40 | a Check chosen; `none` at most 0.45 | | Raised at 0.6 | 0 | 4 of 4, and nothing else | Each call: about 0.25 seconds and 2,000 tokens. `tests/execution/test_coverage_live.py` repeats this through `CoverageReview` itself and is opt-in (`LIVE_JEV=1`); it passed on 2026-10-07. Still off (`JEV_COVERAGE_REVIEW=false`), because three things are unproven: - The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list. - Nine conditions from one case is a small sample; one covered condition scored `none` at 0.45 against a threshold of 0.6. - A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element. Turn it on for a full case before relying on it. Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task). Tests: `tests/execution/test_coverage_review.py`, with a scripted decider. ## Added on 2026-10-08 **What code cannot show is listed apart.** The first full run with the coverage review raised conditions such as an availability percentage, which no Check can verify. A requirement or recommendation now lists under `conditions` only what infrastructure configuration can show, and under `outside_code` what it cannot: an outcome measured in operation, behaviour of the application's own code, a process. `check_requirements` and `check_recommendations` require at least one entry between the two. `chain_gaps` asks for no Check from a constraint with `"conditions": []` and something under `outside_code`. The target architect writes no Check for `outside_code`; the validator proposes nothing for it and names it as something to verify after deployment. Measured before the split, on the designs of two runs that passed validation with nothing proposed, using the conditions the agents themselves wrote: the coverage review raised 14 of 66 (case 02) and 8 of 51 (case 03). About half were of the kind the split removes (availability, recovery time, throughput, latency, what the application logs). The rest look like real gaps the validator did not report (three zones, cost labels, a load balancer). It has not been measured since the split, which needs new runs. **Requirements name no products.** `RequirementsReview` in `app/tools/requirements_review.py` asks one noul per requirement: does it name a cloud provider's product or managed service? It is a third review on the same seam, behind `JEV_REQUIREMENTS_REVIEW` (off), with its own threshold, `JEV_REQUIREMENTS_THRESHOLD`, 0.3. Measured against live Jev: across the 61 requirements of three passed runs the highest answer was 0.10, so nothing was raised; requirements written to name a product scored 0.93 (Cloud Run), 0.61 (Secret Manager) and 0.34 (a Pub/Sub topic), and a database engine or a region scored 0.10 or less. At 0.3 that is three of three caught and none of 65 raised wrongly. Three written examples is a small sample. Tests: `tests/execution/test_coverage_review.py` (2 more), `tests/execution/test_requirements_chain.py` (2 more).