feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target architecture for gaps validation found one pass at a time): - A Deliverable stands while what it was written from is unchanged. The architecture description is written from the Elements and Relationships, so a Revision that only adds or rewords Checks no longer runs the describer. - Non-functional requirements and recommendations list their conditions, one entry per separate thing that must be true; submission requires them. - A coverage review at submission (JEV_COVERAGE_REVIEW, off by default): for each condition, Jev chooses which Check verifies it, or none. Asked as one broad yes-or-no per requirement it could not tell a complete design from one with known gaps; asked as a choice per condition it raised all four known gaps of case 02 and nothing on the complete graph. An opt-in live test (LIVE_JEV=1) repeats that. - The validator is told to report every gap in its first pass. Plans record the case 02 run and both forms of the review with figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -79,3 +79,17 @@ What went wrong: the target architect exhausted its 30 tool calls on its first a
|
||||
|
||||
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.
|
||||
|
||||
## Run of 2026-10-07, 19:07 UTC (`eval/results/20261007T190702Z/`)
|
||||
|
||||
Case 02, on commit `f1f6f1b`. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.
|
||||
|
||||
- The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
|
||||
- The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
|
||||
- The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
|
||||
- The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
|
||||
- By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.
|
||||
|
||||
Changed after the run, not yet run: iteration 4 steps 21 and 22.
|
||||
|
||||
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user