feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,4 +10,7 @@ Differs from the plan: the plan's acceptance is a fall in IaC author Revisions o
|
||||
|
||||
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
|
||||
|
||||
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
Not done: Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
|
||||
Added on 2026-10-08: `references/load-balancer.md`, a fourteenth pattern: an external Application Load Balancer in front of Cloud Run with a TLS policy, a firewall policy, Identity-Aware Proxy and an HTTP redirect. Cases 02, 03 and 04 all used a load balancer with no pattern, and one run put `ssl_policy` on a backend service, which has no such setting; the reference says which link of the chain each setting lives on. It validates against the pinned provider.
|
||||
|
||||
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-25.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-25.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 25 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `_write_outputs` takes the Execution's final status and its Ledger. `end_with_warnings` now calls it, as `finalize` does.
|
||||
2. `execution.json` gains `status` (`completed` or `completed_with_warnings`), `warnings`, `phases` (each Phase's status), and a `status` on every Deliverable: `submitted`, or `draft` when something it depends on was reopened after it was written.
|
||||
3. With no Deliverable at all nothing is written, and the case's history records no files, as before.
|
||||
4. The solution guide is written only when it exists; an Execution that stopped before packaging has none, and the manifest says so.
|
||||
|
||||
Not covered: an Execution whose process is killed. It never reaches either end node, so it still writes nothing. The evaluation's progress file (iteration 6 step 11) records its time and tokens, not its Deliverables.
|
||||
|
||||
Also on 2026-10-08, outside any step: the CI workflow installs `requirements.txt` and `requirements-dev.txt` before linting and testing. It installed only ruff, mypy and pytest, so its test step could not import the application.
|
||||
|
||||
Tests: `tests/execution/test_engine.py`, two updated.
|
||||
17
.agents/plans/iter-04-learning-and-skills/steps/step-26.md
Normal file
17
.agents/plans/iter-04-learning-and-skills/steps/step-26.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Step 26 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `submit_advice` takes a `checklist`: one entry per item of the pillar Skill's validation checklist, with `status` (`addressed`, `not_applicable` or `not_addressed`) and a `note`. An entry with no item or another status is dropped.
|
||||
2. The consultation keeps each pillar's review in the Deliverable's working memory (`pillar_checklists`) beside its advice, so a Revision that re-consults some pillars keeps the others' reviews. The design advisor is given the advice only, not the review.
|
||||
3. `render_checklist_review` writes the reviews as one section, "Architecture framework checklist review", a table per pillar. `submit_deliverable` appends it to the design recommendations and replaces an earlier copy, so a Revision never stacks two.
|
||||
4. A Skill with no checklist gets no review and the Deliverable no section: the mechanism is the provider's Skills' own, not Google Cloud's.
|
||||
|
||||
Checked on 2026-10-08 with two real advisors on case 03's brief: reliability returned all 8 of its items (6 addressed, 2 not applicable) and cost optimisation all 11 (11 addressed), in 74 seconds for the pair.
|
||||
|
||||
Two limits:
|
||||
|
||||
- A review says what the advisor's own advice covers. The design advisor reconciles the pillars and may leave something out, so "addressed" there is the advisor's claim, not a check of the final recommendations.
|
||||
- A judged run of case 04 on 2026-10-08 scored `pillar_checklists_covered` at 100% on all six pillars, against 9% to 45% before. One run.
|
||||
|
||||
Tests: `tests/execution/test_pillars.py`, "each advisor reviews its pillar's checklist" (3).
|
||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-27.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-27.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 27 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `specialist_tools` takes `confirm_first`. While it is set, `submit_deliverable` checks the submission as usual, keeps the working copy, runs any review, and then refuses: the user has not seen this Deliverable; call `needs_input` with a summary and pass the content as `draft`.
|
||||
2. `specialist_task` sets it when the Execution is not pre-approved and the user has not been shown a draft of the Deliverable. The Orchestrator's turn records a Deliverable in `drafts_shown` when a Specialist's question carries a draft. Once shown, submission is allowed, so a Revision is not asked about again.
|
||||
|
||||
The first version opened the rule once the user had answered any question about the Deliverable. The verification run of 2026-10-08 showed the hole: the target architect ran out of tool calls, the Orchestrator asked the user how to proceed, and that answer let the next attempt submit unseen.
|
||||
3. `CONFIRM_DRAFTS` (default `true`) turns the rule on in the assembly. The engine's own default is off, so an engine built directly, as in tests, behaves as before.
|
||||
|
||||
**No interactive evaluation has run since the second version.** The case 04 check `specialists_ask_before_submitting` is the measure.
|
||||
|
||||
Tests: `tests/execution/test_engine.py`, "a user sees a Deliverable before it is first submitted" (3).
|
||||
Reference in New Issue
Block a user