feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
19
app/agent.py
19
app/agent.py
@@ -52,6 +52,7 @@ from app.tools.execution import ORCHESTRATOR
|
||||
from app.tools.jev import JevDecider, JevLimits
|
||||
from app.tools.local import TerraformRunner, local_tools
|
||||
from app.tools.mcp import McpCatalog, auth_from_config
|
||||
from app.tools.schema_review import Reviews, SchemaReview
|
||||
from app.tools.toolkit import make_tool_factory
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
from app.workflows.execution_graph import build_execution_graph
|
||||
@@ -323,6 +324,16 @@ async def open_assembly(
|
||||
f"ORCHESTRATOR must be router or model, not {settings.ORCHESTRATOR!r}"
|
||||
)
|
||||
|
||||
# What is looked at in a design before it is submitted. Each raises what it finds
|
||||
# once and never blocks.
|
||||
reviews: list[SubmissionReview] = []
|
||||
if bundle.terraform_providers:
|
||||
# In code: every setting a Check names exists on its resource type.
|
||||
reviews.append(SchemaReview(runner))
|
||||
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider):
|
||||
# Optional, and off by default: Jev says which Check verifies each condition.
|
||||
reviews.append(CoverageReview(decider, none_at=settings.JEV_COVERAGE_THRESHOLD))
|
||||
|
||||
async with open_checkpointer(settings) as checkpointer:
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
@@ -338,13 +349,7 @@ async def open_assembly(
|
||||
name: skill_version(meta.path) for name, meta in bundle.skills.items()
|
||||
},
|
||||
orchestrator=orchestrator,
|
||||
# Optional, and off by default: Jev reviews a design's coverage before
|
||||
# it is submitted. Without it a submission stands on its structure.
|
||||
submission_review=CoverageReview(
|
||||
decider, none_at=settings.JEV_COVERAGE_THRESHOLD
|
||||
)
|
||||
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider)
|
||||
else None,
|
||||
submission_review=Reviews(reviews) if reviews else None,
|
||||
)
|
||||
try:
|
||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
||||
|
||||
Reference in New Issue
Block a user