Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-4.md
Jonathan Boniface 5adc911a82 feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:58:38 +01:00

4.4 KiB

Step 4 — Implementation Detail

Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.

What was built:

  1. A seam: SubmissionReview in app/adk/runtime.py, passed through ExecutionEngine, build_execution_graph and make_execution_nodes to specialist_tools. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
  2. CoverageReview in app/tools/coverage.py, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
  3. JEV_COVERAGE_REVIEW, default false. The assembly installs the review only when it is true and a Jev key is set.

Why the first form failed

Case 02 (eval/results/20261007T190702Z/) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:

Question asked On the complete graph On the graph with 4 known gaps
"Does it state a condition none of these Checks verifies?" at 0.75 raised 7 of 28 raised 6; caught 1 of the 4
"Do these Checks verify everything it asks?" (probability of yes) mean 0.33 over the other 24 the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33
Score: none, part, all mean 1.24 of 2 over the other 24 the 4 fell by 0.06 to 0.55; the others unchanged

Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.

The form that works: one choice per condition

The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.

  1. Requirements and recommendations list their conditions: one entry for each separate thing that must be true. check_requirements and check_recommendations refuse a non-functional requirement or a recommendation without them. conditions_of reads an older item as one condition, its text.
  2. CoverageReview asks, per condition, "which one Check verifies this condition?" with that source's Check ids and none as the options. A condition answered none with a probability of at least JEV_COVERAGE_THRESHOLD (0.6) is raised, as NFR-1: no Check verifies "<condition>" (its Checks: ...).

Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):

Complete graph Graph with the 4 gaps
The 4 gap conditions a Check chosen each time none each time, probability 0.72 to 1.00
The 5 covered conditions a Check chosen; none at most 0.40 a Check chosen; none at most 0.45
Raised at 0.6 0 4 of 4, and nothing else

Each call: about 0.25 seconds and 2,000 tokens. tests/execution/test_coverage_live.py repeats this through CoverageReview itself and is opt-in (LIVE_JEV=1); it passed on 2026-10-07.

Still off (JEV_COVERAGE_REVIEW=false), because three things are unproven:

  • The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
  • Nine conditions from one case is a small sample; one covered condition scored none at 0.45 against a threshold of 0.6.
  • A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.

Turn it on for a full case before relying on it.

Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).

Tests: tests/execution/test_coverage_review.py, with a scripted decider.