Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-27.md
Jonathan Boniface 654370bb46 feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:32:57 +01:00

1.3 KiB

Step 27 — Implementation Detail

Built on 2026-10-08.

  1. specialist_tools takes confirm_first. While it is set, submit_deliverable checks the submission as usual, keeps the working copy, runs any review, and then refuses: the user has not seen this Deliverable; call needs_input with a summary and pass the content as draft.

  2. specialist_task sets it when the Execution is not pre-approved and the user has not been shown a draft of the Deliverable. The Orchestrator's turn records a Deliverable in drafts_shown when a Specialist's question carries a draft. Once shown, submission is allowed, so a Revision is not asked about again.

    The first version opened the rule once the user had answered any question about the Deliverable. The verification run of 2026-10-08 showed the hole: the target architect ran out of tool calls, the Orchestrator asked the user how to proceed, and that answer let the next attempt submit unseen.

  3. CONFIRM_DRAFTS (default true) turns the rule on in the assembly. The engine's own default is off, so an engine built directly, as in tests, behaves as before.

No interactive evaluation has run since the second version. The case 04 check specialists_ask_before_submitting is the measure.

Tests: tests/execution/test_engine.py, "a user sees a Deliverable before it is first submitted" (3).