feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s

- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 21:27:09 +01:00
parent 567aede5ca
commit e5e1647ff1
35 changed files with 497 additions and 39 deletions

View File

@@ -17,3 +17,13 @@ Differs from the plan:
`case-06-organisation-standards` in the evaluation dataset runs case 03's request under the example overlay. It has not been run.
Tests: `tests/execution/test_overlay_skills.py` (8).
## Run on 2026-10-09
`case-06-organisation-standards` (results `20261009T171333Z`): completed, 100%, 36 minutes, 3.66 million tokens, one Revision of the Target architecture. `execution.json` records the overlay with its hash. The code follows it: every resource is named `${var.workload}-${var.environment}-<purpose>`, the only region is `europe-west1`, and the labels are set. The same request without the overlay last took 22 minutes and 2.32 million tokens (`20261009T103827Z`); one run each, so the difference is not attributed.
No evaluation check scores whether an overlay was followed; this was read from the code by hand.
## An evaluation check, later on 2026-10-09
A case may now carry `code_standards`: regular expressions the infrastructure code must contain or must not, scored as `code_follows_standards`. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.