feat: record verdicts in pieces and check the design against the schema

From the case 02 run cancelled at 66 minutes on 2026-10-07:

- The validator records verdicts a few Checks at a time (verdict_put) and
  they are attached on submission. It timed out five times writing them in
  one reply. Verdicts on the same code survive a run cut short; verdicts on
  changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
  looked up on its Element's resource type, and an unknown resource type is
  reported. Two Revisions in that run were for Checks no code could meet.
  On the two designs that passed validation it raises 2 of 96 Checks, both
  one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.

Plans record both steps and the cancelled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 09:39:13 +01:00
parent 5adc911a82
commit 58e778ca7a
16 changed files with 691 additions and 14 deletions

View File

@@ -93,3 +93,14 @@ Changed after the run, not yet run: iteration 4 steps 21 and 22.
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
## Run of 2026-10-07, 19:58 UTC, cancelled (`.cache/eval-logs/`; no report)
Case 02 again on commit `5adc911`, with `JEV_COVERAGE_REVIEW=true`. Cancelled by the owner at 66 minutes, during a fifth validation. No score and no token count were written.
- The Target architecture was reopened 3 times (the limit): twice for a Check the code's author reported as unimplementable, once for additions validation proposed. The code was reopened twice for failed Checks. Target architect 5 runs, IaC author 6, validator 5.
- 6 model timeouts, 5 of them the validator writing all its verdicts in one reply: about 15 minutes.
- The coverage review raised 5, 10, 11 and 9 of 61 conditions over four submissions. Validation still proposed a TLS policy and an access grant afterwards. The agents had written conditions no infrastructure code can prove, such as an availability percentage, and Jev answered "none" to those. It stays off.
- What held: the description was kept on two Revisions (iteration 4 step 21); the unimplementable-Check route sent the design back before any validation was spent (step 19); code fixes took 1 to 5 minutes from the working copy; no `terraform validate` failure.
Changed after it, not yet run: iteration 4 steps 23 and 24.