feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -10,4 +10,7 @@ Differs from the plan: the plan's acceptance is a fall in IaC author Revisions o
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
Not done: Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
Added on 2026-10-08: `references/load-balancer.md`, a fourteenth pattern: an external Application Load Balancer in front of Cloud Run with a TLS policy, a firewall policy, Identity-Aware Proxy and an HTTP redirect. Cases 02, 03 and 04 all used a load balancer with no pattern, and one run put `ssl_policy` on a backend service, which has no such setting; the reference says which link of the chain each setting lives on. It validates against the pinned provider.

View File

@@ -0,0 +1,14 @@
# Step 25 — Implementation Detail
Built on 2026-10-08.
1. `_write_outputs` takes the Execution's final status and its Ledger. `end_with_warnings` now calls it, as `finalize` does.
2. `execution.json` gains `status` (`completed` or `completed_with_warnings`), `warnings`, `phases` (each Phase's status), and a `status` on every Deliverable: `submitted`, or `draft` when something it depends on was reopened after it was written.
3. With no Deliverable at all nothing is written, and the case's history records no files, as before.
4. The solution guide is written only when it exists; an Execution that stopped before packaging has none, and the manifest says so.
Not covered: an Execution whose process is killed. It never reaches either end node, so it still writes nothing. The evaluation's progress file (iteration 6 step 11) records its time and tokens, not its Deliverables.
Also on 2026-10-08, outside any step: the CI workflow installs `requirements.txt` and `requirements-dev.txt` before linting and testing. It installed only ruff, mypy and pytest, so its test step could not import the application.
Tests: `tests/execution/test_engine.py`, two updated.

View File

@@ -0,0 +1,17 @@
# Step 26 — Implementation Detail
Built on 2026-10-08.
1. `submit_advice` takes a `checklist`: one entry per item of the pillar Skill's validation checklist, with `status` (`addressed`, `not_applicable` or `not_addressed`) and a `note`. An entry with no item or another status is dropped.
2. The consultation keeps each pillar's review in the Deliverable's working memory (`pillar_checklists`) beside its advice, so a Revision that re-consults some pillars keeps the others' reviews. The design advisor is given the advice only, not the review.
3. `render_checklist_review` writes the reviews as one section, "Architecture framework checklist review", a table per pillar. `submit_deliverable` appends it to the design recommendations and replaces an earlier copy, so a Revision never stacks two.
4. A Skill with no checklist gets no review and the Deliverable no section: the mechanism is the provider's Skills' own, not Google Cloud's.
Checked on 2026-10-08 with two real advisors on case 03's brief: reliability returned all 8 of its items (6 addressed, 2 not applicable) and cost optimisation all 11 (11 addressed), in 74 seconds for the pair.
Two limits:
- A review says what the advisor's own advice covers. The design advisor reconciles the pillars and may leave something out, so "addressed" there is the advisor's claim, not a check of the final recommendations.
- A judged run of case 04 on 2026-10-08 scored `pillar_checklists_covered` at 100% on all six pillars, against 9% to 45% before. One run.
Tests: `tests/execution/test_pillars.py`, "each advisor reviews its pillar's checklist" (3).

View File

@@ -0,0 +1,13 @@
# Step 27 — Implementation Detail
Built on 2026-10-08.
1. `specialist_tools` takes `confirm_first`. While it is set, `submit_deliverable` checks the submission as usual, keeps the working copy, runs any review, and then refuses: the user has not seen this Deliverable; call `needs_input` with a summary and pass the content as `draft`.
2. `specialist_task` sets it when the Execution is not pre-approved and the user has not been shown a draft of the Deliverable. The Orchestrator's turn records a Deliverable in `drafts_shown` when a Specialist's question carries a draft. Once shown, submission is allowed, so a Revision is not asked about again.
The first version opened the rule once the user had answered any question about the Deliverable. The verification run of 2026-10-08 showed the hole: the target architect ran out of tool calls, the Orchestrator asked the user how to proceed, and that answer let the next attempt submit unseen.
3. `CONFIRM_DRAFTS` (default `true`) turns the rule on in the assembly. The engine's own default is off, so an engine built directly, as in tests, behaves as before.
**No interactive evaluation has run since the second version.** The case 04 check `specialists_ask_before_submitting` is the measure.
Tests: `tests/execution/test_engine.py`, "a user sees a Deliverable before it is first submitted" (3).