feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s
Some checks failed
validation / verify (push) Failing after 13s
- Schema review: a dotted path is judged only when it starts at a name the resource type has, so a database flag or a service name is no longer read as a missing setting. Lesson L-04, which worked around that, is retired. - Skill routing: in a Change request the question says what changed. - Evaluation: a case may list code_standards the infrastructure code must meet; case 06 checks its overlay's regions, naming and labels. - Model tiers (iteration 5 step 6): a lighter model for a role's first attempt, named in the provider bundle. Measured on case 03 and left off: Flash saved five cents on the IaC author and took twice the requests. - Steps 7 and 9 decided against on the numbers, recorded in the plans. - Plans: the runs of cases 05 and 06, and the first proof of the lessons on case 03 (no Revision, 22 minutes against 28, quality held). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -28,7 +28,7 @@
|
||||
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
|
||||
| 10 | Lessons Skill and lesson curator | B | L | the loop is built; six lessons proposed from 21 Executions, approved by the owner and published on 2026-10-09; not yet proved by an evaluation with and without (`steps/step-10.md`) |
|
||||
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
|
||||
| 12 | Overlay Skills and an organisation standards Skill | C | M | built, with an example standards Skill and an evaluation case; not run with real models |
|
||||
| 12 | Overlay Skills and an organisation standards Skill | C | M | built, with an example standards Skill; case 06 passed with it on 2026-10-09 and the code follows the standards |
|
||||
| 13 | Trim Specialist context | C | M | built; measured on case 01 only, together with steps 11 and 17 |
|
||||
| 14 | Provider-owned contracts | C | M | deferred by its own plan until a second Cloud provider is planned (ADR-0002) |
|
||||
| 15 | Group Executions by Customer case | added | S | complete |
|
||||
|
||||
@@ -50,3 +50,26 @@ The owner approved all six on 2026-10-09 and they were published: five for the t
|
||||
|
||||
Still open: retiring lessons when the upstream Skills change is not automated, and no lesson has been proved.
|
||||
|
||||
## Run on 2026-10-09
|
||||
|
||||
Case 06 ran with the six lessons loaded and still needed one Revision of the Target architecture, as case 03 did before the lessons existed. That neither proves nor disproves them: it also carried an overlay. `prove` has not been run.
|
||||
|
||||
## L-04 retired on 2026-10-09
|
||||
|
||||
L-04 told the target architect to word Checks so the schema review would not mistake a value for a setting. The review was at fault: it read `cloudsql.iam_authentication` (a database flag) and `run.googleapis.com` (a service) as settings, and raised both again in case 06 with the lesson loaded. It now judges a dotted path only when the path starts at a name the resource type has (`_is_missing` in `app/tools/schema_review.py`). The lesson is removed from the Skill; five remain.
|
||||
|
||||
## First proof, 2026-10-09
|
||||
|
||||
Case 03, one run each way, side by side, with the five lessons that remain (`20261009T195603Z` without, `20261009T195607Z` with):
|
||||
|
||||
| | Without lessons | With lessons |
|
||||
|---|---|---|
|
||||
| Score | 100% | 100% |
|
||||
| Revisions | 1, of the Target architecture | 0 |
|
||||
| Minutes | 28 | 22 |
|
||||
| Tokens | 3.00 million | 2.32 million |
|
||||
| Cost, about | $3.47 | $2.44 |
|
||||
|
||||
`prove` recorded this in the proposals as "kept". One pair suggests a direction and does not settle it: a third run the same evening, with lessons, did need a Revision of the Target architecture. `prove` also wrote to L-04's proposal, which is retired; it should skip retired lessons.
|
||||
|
||||
Neither run raised the schema review's false alarm, which had been raised in case 06 before the fix.
|
||||
|
||||
@@ -17,3 +17,13 @@ Differs from the plan:
|
||||
`case-06-organisation-standards` in the evaluation dataset runs case 03's request under the example overlay. It has not been run.
|
||||
|
||||
Tests: `tests/execution/test_overlay_skills.py` (8).
|
||||
|
||||
## Run on 2026-10-09
|
||||
|
||||
`case-06-organisation-standards` (results `20261009T171333Z`): completed, 100%, 36 minutes, 3.66 million tokens, one Revision of the Target architecture. `execution.json` records the overlay with its hash. The code follows it: every resource is named `${var.workload}-${var.environment}-<purpose>`, the only region is `europe-west1`, and the labels are set. The same request without the overlay last took 22 minutes and 2.32 million tokens (`20261009T103827Z`); one run each, so the difference is not attributed.
|
||||
|
||||
No evaluation check scores whether an overlay was followed; this was read from the code by hand.
|
||||
|
||||
## An evaluation check, later on 2026-10-09
|
||||
|
||||
A case may now carry `code_standards`: regular expressions the infrastructure code must contain or must not, scored as `code_follows_standards`. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.
|
||||
|
||||
Reference in New Issue
Block a user