Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-4.md
Jonathan Boniface 654370bb46 feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:32:57 +01:00

6.5 KiB

Step 4 — Implementation Detail

Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.

What was built:

  1. A seam: SubmissionReview in app/adk/runtime.py, passed through ExecutionEngine, build_execution_graph and make_execution_nodes to specialist_tools. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
  2. CoverageReview in app/tools/coverage.py, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
  3. JEV_COVERAGE_REVIEW, default false. The assembly installs the review only when it is true and a Jev key is set.

Why the first form failed

Case 02 (eval/results/20261007T190702Z/) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:

Question asked On the complete graph On the graph with 4 known gaps
"Does it state a condition none of these Checks verifies?" at 0.75 raised 7 of 28 raised 6; caught 1 of the 4
"Do these Checks verify everything it asks?" (probability of yes) mean 0.33 over the other 24 the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33
Score: none, part, all mean 1.24 of 2 over the other 24 the 4 fell by 0.06 to 0.55; the others unchanged

Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.

The form that works: one choice per condition

The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.

  1. Requirements and recommendations list their conditions: one entry for each separate thing that must be true. check_requirements and check_recommendations refuse a non-functional requirement or a recommendation without them. conditions_of reads an older item as one condition, its text.
  2. CoverageReview asks, per condition, "which one Check verifies this condition?" with that source's Check ids and none as the options. A condition answered none with a probability of at least JEV_COVERAGE_THRESHOLD (0.6) is raised, as NFR-1: no Check verifies "<condition>" (its Checks: ...).

Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):

Complete graph Graph with the 4 gaps
The 4 gap conditions a Check chosen each time none each time, probability 0.72 to 1.00
The 5 covered conditions a Check chosen; none at most 0.40 a Check chosen; none at most 0.45
Raised at 0.6 0 4 of 4, and nothing else

Each call: about 0.25 seconds and 2,000 tokens. tests/execution/test_coverage_live.py repeats this through CoverageReview itself and is opt-in (LIVE_JEV=1); it passed on 2026-10-07.

Still off (JEV_COVERAGE_REVIEW=false), because three things are unproven:

  • The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
  • Nine conditions from one case is a small sample; one covered condition scored none at 0.45 against a threshold of 0.6.
  • A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.

Turn it on for a full case before relying on it.

Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).

Tests: tests/execution/test_coverage_review.py, with a scripted decider.

Added on 2026-10-08

What code cannot show is listed apart. The first full run with the coverage review raised conditions such as an availability percentage, which no Check can verify. A requirement or recommendation now lists under conditions only what infrastructure configuration can show, and under outside_code what it cannot: an outcome measured in operation, behaviour of the application's own code, a process. check_requirements and check_recommendations require at least one entry between the two. chain_gaps asks for no Check from a constraint with "conditions": [] and something under outside_code. The target architect writes no Check for outside_code; the validator proposes nothing for it and names it as something to verify after deployment.

Measured before the split, on the designs of two runs that passed validation with nothing proposed, using the conditions the agents themselves wrote: the coverage review raised 14 of 66 (case 02) and 8 of 51 (case 03). About half were of the kind the split removes (availability, recovery time, throughput, latency, what the application logs). The rest look like real gaps the validator did not report (three zones, cost labels, a load balancer). It has not been measured since the split, which needs new runs.

Requirements name no products. RequirementsReview in app/tools/requirements_review.py asks one noul per requirement: does it name a cloud provider's product or managed service? It is a third review on the same seam, behind JEV_REQUIREMENTS_REVIEW (off), with its own threshold, JEV_REQUIREMENTS_THRESHOLD, 0.3. Measured against live Jev: across the 61 requirements of three passed runs the highest answer was 0.10, so nothing was raised; requirements written to name a product scored 0.93 (Cloud Run), 0.61 (Secret Manager) and 0.34 (a Pub/Sub topic), and a database engine or a region scored 0.10 or less. At 0.3 that is three of three caught and none of 65 raised wrongly. Three written examples is a small sample.

Tests: tests/execution/test_coverage_review.py (2 more), tests/execution/test_requirements_chain.py (2 more).