feat: a design-time coverage review, measured and left off

A reviewer role reads the requirements and recommendations against the Checks
when the Target architecture is first submitted. Three runs of case 03: it did
not stop the validator sending the design back, and the designs it reviewed
ended with 87 and 93 Checks against 52, so they cost more. It stays behind
DESIGN_COVERAGE_REVIEW, off (iteration 6 step 13).

Also: the schema review no longer reads `true` or `false` in backticks as a
setting name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 12:33:06 +01:00
parent 1cf1234143
commit ee1ba38867
13 changed files with 401 additions and 6 deletions

View File

@@ -22,7 +22,7 @@ from langchain_core.runnables import RunnableConfig
from langgraph.checkpoint.base import BaseCheckpointSaver
from langgraph.types import Command
from app.adk.roles import load_roles
from app.adk.roles import COVERAGE_REVIEWER, load_roles
from app.adk.runtime import (
AgentRuntime,
Decider,
@@ -50,6 +50,7 @@ from app.skills.agent_skills import discover_skills, skill_version
from app.states.execution_state import check_customer_case
from app.states.ledger import Ledger
from app.tools.coverage import CoverageReview
from app.tools.design_review import DesignCoverageReview
from app.tools.execution import ORCHESTRATOR
from app.tools.jev import JevDecider, JevLimits
from app.tools.local import TerraformRunner, local_tools
@@ -481,6 +482,10 @@ async def open_assembly(
reviews.append(
RequirementsReview(decider, act=settings.JEV_REQUIREMENTS_THRESHOLD)
)
if settings.DESIGN_COVERAGE_REVIEW and COVERAGE_REVIEWER in roles:
# A model reads the requirements against the Checks, once, before the
# design is approved: the reading the validator would make after the code.
reviews.append(DesignCoverageReview(runtime, roles[COVERAGE_REVIEWER]))
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider):
# Optional, and off by default: Jev says which Check verifies each condition.
reviews.append(CoverageReview(decider, none_at=settings.JEV_COVERAGE_THRESHOLD))