feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -204,3 +204,27 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
## Step 25: An Execution that ends with warnings keeps its files
- **Report ref:** none; found by the runs of 2026-10-07 and 08
- **Capability:** What an unfinished Execution produced can be read afterwards.
- **Rationale:** Only a completed Execution wrote its Deliverables. Case 04's first run ended with warnings after 53 minutes and left nothing on disk: nine Deliverables' worth of work, and the evidence of what went wrong, existed only in the report's conversation log.
- **Where:** `_write_outputs` and `end_with_warnings` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** An Execution that ends with warnings writes every Deliverable it holds, with a manifest that carries the warnings, each Deliverable's status and each Phase's status, so nothing unapproved reads as final. The case's history points to the files.
## Step 26: Each pillar's checklist is reviewed and the review is in the Deliverable
- **Report ref:** none; found by the judged runs of 2026-10-08
- **Capability:** An architect sees every item of each pillar's checklist accounted for: addressed, and how, or not applicable, and why.
- **Rationale:** The judged check `pillar_checklists_covered` failed in all four cases of the judged batch, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The six checklists hold 78 items; a run produces 11 to 14 recommendations, each one configurable decision. Twelve recommendations cannot answer 78 items, many of which are an organisation's process and not a design decision, so the check could not pass and nothing told an architect which items had been considered.
- **Where:** `app/tools/pillars.py`, `submit_deliverable` in `app/tools/execution.py`, `agents/pillar-advisor.md`, `agents/design-advisor.md`.
- **Acceptance criteria:** Each pillar advisor returns, with its advice, a status and a note for every item of its Skill's checklist. The reviews are kept with the advice across Revisions and written into the design recommendations by code, once. On a judged run `pillar_checklists_covered` passes.
## Step 27: A user sees a Deliverable before it is first submitted
- **Report ref:** none; found by evaluation case 04 on 2026-10-08
- **Capability:** With a user present, no Deliverable is submitted that they have not been shown.
- **Rationale:** The Specialist protocol tells every Specialist to confirm its draft with the user before submitting. In case 04 the validator did not (first run), and the target architect did not after running out of tool calls on an earlier attempt (batch A). An instruction was not enough.
- **Where:** `submit_deliverable` in `app/tools/execution.py`, `specialist_task` in `app/nodes/execution_nodes.py`, `CONFIRM_DRAFTS` in `app/config.py`.
- **Acceptance criteria:** In an Execution with a user, a Specialist's submission is refused until the user has replied to it about that Deliverable at least once, and the refusal says how to ask. A pre-approved Execution is unaffected. It can be switched off.