feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -162,3 +162,51 @@ Not yet changed:
|
||||
|
||||
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
|
||||
|
||||
## Changes to the evaluation after the unfinished run (2026-10-08)
|
||||
|
||||
- A scripted answer may name who it is for: `[pattern, answer, asker]`. Case 04's fourteen facts are tagged for `requirements`; any other Specialist, and the Orchestrator, gets "Approved. Please proceed." (`automatic_reply`, `eval/datasets/solution_cases.json`).
|
||||
- The evaluation writes `<case>-progress.json` in the run folder after every agent run: seconds, runs, outcomes, tokens and the timeline so far. A run that is cut off leaves it behind with `"finished": false`.
|
||||
|
||||
## Runs of 2026-10-08, 14:25 UTC: all four cases, twice (`eval/results/20261008T142509Z/` and `...142514Z/`)
|
||||
|
||||
Two batches run side by side on the build with the case 04 changes, intake and the `outside_code` split (uncommitted at the time). Batch B added the judged checks. All eight runs completed; none ended with warnings.
|
||||
|
||||
| Case | Batch A | Batch B (judged) |
|
||||
|---|---|---|
|
||||
| 01 | 1.00, 31 min, 3.42 M tokens, 1 Revision | 0.92, 39 min, 4.06 M, 1 Revision |
|
||||
| 02 | 1.00, 47 min, 7.02 M, 3 Revisions | 0.92, 24 min, 3.12 M, none |
|
||||
| 03 | 1.00, 41 min, 6.33 M, 2 Revisions | 0.92, 29 min, 4.06 M, 1 Revision |
|
||||
| 04 | 0.92, 45 min, 5.88 M, 1 Revision | 0.93, 33 min, 3.63 M, 1 Revision |
|
||||
|
||||
Estimated cost: $22.16 for batch A and $16.40 for batch B. Running two batches at once shares the machine and the model quota, so these times are longer than a case run alone.
|
||||
|
||||
- **Case 04 completes.** Both runs went through all five Phases with every draft confirmed by the user and no reply misread as a change. One Revision each: the code for a failed Check in batch B, the design for a proposed addition in batch A.
|
||||
- **Every structural check passed in all eight runs**, with no model timeout, no loop and no `terraform validate` failure.
|
||||
- **`pillar_checklists_covered` failed in all four judged runs**, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The cause is structural: 78 checklist items against 11 to 14 recommendations. Iteration 4 step 26 was built in answer; no judged run has happened since.
|
||||
- **`specialists_ask_before_submitting` failed once**, in batch A's case 04: the target architect exhausted its 45 tool calls, and on its second attempt submitted without asking. The first submission with a user present is now refused until the draft has been put to them (`CONFIRM_DRAFTS`, on by default).
|
||||
- **The same case varies widely.** Case 02 took 24 minutes with no Revision in one batch and 47 minutes with three in the other, at the same time on the same build. Revisions of the Target architecture for additions validation proposes remain the main cost.
|
||||
|
||||
### Thresholds
|
||||
|
||||
Set in `eval/datasets/solution_cases.json` from every completed run of each case on this build, at about a quarter above the worst seen:
|
||||
|
||||
| Case | Revisions | Seconds | Tokens |
|
||||
|---|---|---|---|
|
||||
| 01 | 2 | 3,000 | 5.0 M |
|
||||
| 02 | 3 | 3,600 | 8.5 M |
|
||||
| 03 | 3 | 3,200 | 8.0 M |
|
||||
| 04 | 2 | 3,400 | 7.5 M |
|
||||
|
||||
They are alarms for a run that is clearly worse than any seen, not targets. With two to four runs a case and this much spread they are loose; tighten them as runs accumulate.
|
||||
|
||||
Acceptance criteria: met. All four cases complete and score at least 0.8; deployment guidance needed at most one Revision in every run; the thresholds are set. One criterion is met only loosely: the set of gaps did not always shrink from one validation to the next (case 02, 2026-10-07).
|
||||
|
||||
## Run of 2026-10-08, 17:20 UTC (`eval/results/20261008T172049Z/`): case 04, judged
|
||||
|
||||
To verify two changes made after the batches began: the pillar checklist review (iteration 4 step 26) and the confirm-first rule (step 27). Completed, 86% (12 of 14 checks), no Revision, 36 minutes, 4.10 million tokens.
|
||||
|
||||
- **`pillar_checklists_covered` passes: 100% on all six pillars**, against 9% to 45% in batch B's case 04. The review is in the Deliverable.
|
||||
- `recommendations_quality` passed at 0.80 (specific 0.84, actionable 0.81, trade-offs 0.90, traceable 0.66).
|
||||
- **`specialists_ask_before_submitting` failed again**, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (`drafts_shown`). Not rerun.
|
||||
- **`requirements_name_no_products` failed** for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user