chore: raise the validator's budget and record the runs of cases 02, 03 and 04
- Validator budget 15 to 30: recording verdicts in pieces for 100 Checks used all 15 calls in one attempt. - Plans record three runs on 2026-10-08: case 02 completed at 100% with no Revision in 24 minutes; case 03 completed at 100% with two Revisions of the design in 40 minutes; case 04 (interactive) ended with warnings at 75%, on two router faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit. Neither is fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -104,3 +104,36 @@ Case 02 again on commit `5adc911`, with `JEV_COVERAGE_REVIEW=true`. Cancelled by
|
||||
|
||||
Changed after it, not yet run: iteration 4 steps 23 and 24.
|
||||
|
||||
## Run of 2026-10-08, 08:39 UTC (`eval/results/20261008T083917Z/`)
|
||||
|
||||
Case 02 on commit `58e778c`, coverage review off. Completed, 100% (11 of 11 checks), no warnings, no Revision: about 24 minutes, an estimated $3.52.
|
||||
|
||||
- The graph had 21 Elements, 22 Relationships and 102 Checks, against 54 in the run of 19:07 the day before. The requirements and recommendations now list their conditions, and the target architect wrote a Check for each. Validation passed all 102 and proposed nothing.
|
||||
- No Check was unimplementable, and the schema review raised nothing at submission.
|
||||
- No model timeout. The validator exhausted its 15 tool calls on its first attempt with verdicts recorded; its second attempt carried on from them and submitted in 68 seconds. Its budget was raised to 30 after the run.
|
||||
- The target architect took 388 seconds and 29 requests (21 `graph_put`, 15 schema lookups): now the slowest and most expensive step.
|
||||
|
||||
Case 02 across its three runs: 30 minutes with 2 Revisions; cancelled at 66 minutes; 24 minutes with none. One clean run does not show the variance is gone.
|
||||
|
||||
Acceptance criteria: cases 01 and 02 complete at 1.0, case 02 now with no Revision. Cases 03 and 04 were started after this run.
|
||||
|
||||
## Run of 2026-10-08, 10:33 UTC (`eval/results/20261008T103338Z/`)
|
||||
|
||||
Cases 03 and 04, on `58e778c` with the validator's budget raised to 30 (uncommitted).
|
||||
|
||||
**Case 03 (internal web app, pre-approved): completed, 100%, no warnings.** 40 minutes (25 to the first validation), 5.21 million tokens, an estimated $5.10. The code passed every Check in all three validations. The Target architecture was revised twice for proposed additions: region choices for a carbon-free-energy requirement, then a role grant for database access. The description was kept on the first Revision and rewritten on the second, which added an Element. No timeout, no `terraform validate` failure, no unimplementable Check. The validator was the most expensive role (31 requests, about $1.78).
|
||||
|
||||
**Case 04 (the same workload, interactive): completed with warnings, 75%.** 53 minutes, 8.72 million tokens, an estimated $8.11. It stopped at the limit of 100 Orchestrator turns. Three checks failed: completed, the guide (never packaged), and `specialists_ask_before_submitting` for validation results.
|
||||
|
||||
What went wrong in case 04, in order:
|
||||
|
||||
1. **A reply to a Specialist's question was read as a change request.** The scripted user replies with `Q: <the question> A: <the answer>`. The router gives that whole text to Jev as the user's message. Twice, a describer's question asking the user to confirm its draft, echoed back, was judged a request to change the Target architecture (asks_change 0.76 to 0.78), and the design was reopened with the question as the reason. With one real unimplementable Check, that used all three Revisions.
|
||||
2. **At the Revision limit the router loops.** Validation then proposed an addition; reopening was refused at the limit; the router asked the user; the reply was again read as a change request for the same Deliverable; the reopen was refused again. 40 escalations in two seconds, each question nesting the last, until the turn limit. No model ran in the loop, but Jev was called each time (73 judged decisions in the run).
|
||||
3. The validator submitted without confirming a draft with the user, which the interactive protocol asks of every Specialist.
|
||||
|
||||
What held in case 04: every Specialist's questions reached the user and their answers came back; four Approvals were granted by structured decisions; the unimplementable-Check route fired once, correctly; the code validated.
|
||||
|
||||
Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not complete. "At most one Revision of deployment guidance in two of three cases" holds (none in 01, 02 and 03). The thresholds remain unset until case 04 passes.
|
||||
|
||||
Not yet fixed: the two router faults above.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user