feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s

- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 21:27:09 +01:00
parent 567aede5ca
commit e5e1647ff1
35 changed files with 497 additions and 39 deletions

View File

@@ -14,3 +14,7 @@ Measured against live Jev on ten texts: product documentation, how-to steps writ
The decider's budget is counted per Execution through a context variable set when a Specialist starts.
Tests: `tests/tools/test_guardrail.py` (7).
## Run on 2026-10-09
Ran in cases 05 and 06: nothing was withheld and no risk was flagged, in two Executions' worth of documentation lookups. No false alarm; nothing in these runs tried to instruct the agent, so the catching side is still only calibrated.

View File

@@ -25,3 +25,7 @@ At 0.25 the three rewordings are read as such and no real change is. The dropped
Differs from the plan: requirements are not asked about. They carry their own `change` and reason, and an `unchanged` one must keep its text exactly (iteration 7 step 3). The Baseline's verdict is not carried forward for a reworded Check: validation still gives a verdict on every Check.
Tests: `tests/execution/test_change_requests.py`, "changed, or only reworded?" (2).
## Run on 2026-10-09
Ran in case 05's Change request. The Specialists changed no existing item's text: Checks and recommendations were added and removed by id, so there was nothing to settle and nothing listed as reworded.

View File

@@ -16,3 +16,7 @@ Differs from the plan:
The judge behind it was measured in step 16: Jev answered the same question the same way every time (1.00) and agreed with Gemini on 0.91 of questions. Agreement with a person still awaits the owner's 32 labels.
Tests: `tests/execution/test_assessment.py` (5).
## Run on 2026-10-09
Both Executions carry an assessment in `execution.json`: case 06 a mean of 0.74 (specific 0.76, actionable 0.77, trade-offs 0.75, traceable 0.69); case 05's Change request 0.90. Both meet the bar of 0.67.

View File

@@ -21,3 +21,7 @@ Differs from the plan: the plan asked a yes or no per pillar. Measured against l
Operations and cost are picked for almost every real change, so the saving is two to five advisors of six, not five of six.
Tests: `tests/execution/test_pillars.py` (3), `tests/execution/test_change_requests.py` (2).
## Run on 2026-10-09
`case-05-change-retention` (results `20261009T171336Z`): the Change request consulted 5 pillar advisors, not 6, which is what calibration predicted for this change. The Change request took about 18 minutes and 1.30 million tokens against the Baseline's 30 minutes and 3.09 million (42%), passed every change check, and changed nothing unrequested (0 of 89).

View File

@@ -0,0 +1,37 @@
# Step 6 — Implementation Detail
Built on 2026-10-09, **off by default** (`MODEL_TIERS`) until measured.
Where case 06's cost went (`20261009T171333Z`, about $3.84 at the prices in the plan map):
| Role | Model | Requests | Prompt tokens | Of them cached | Cost |
|---|---|---|---|---|---|
| validator | Pro | 22 | 1,025,080 | 82% | $1.04 |
| target-architect | Pro | 24 | 686,263 | 69% | $1.02 |
| iac-author | Pro | 16 | 434,234 | 60% | $0.63 |
| describer | Flash | 13 | 675,284 | 86% | $0.16 |
| packager | Flash | 4 | 233,373 | 72% | $0.17 |
| the other six | mixed | 38 | 371,088 | | $0.82 |
The three roles on the Pro model are 70% of the cost. For each, output and thinking tokens cost about as much as the prompt.
1. `models.first_attempt` in the provider bundle names a lighter model for a role's first attempt at a Deliverable. With `MODEL_TIERS` on, the role takes it when nothing has been asked of the Deliverable again: there is no feedback on it, and no earlier attempt ended without one. Any other attempt takes the role's own model.
2. Only the IaC author is named, on Flash. Its work is checked twice by something else: `terraform validate`, and the validator.
3. The validator is deliberately not named, though it costs the most. It is the check on the code. A weaker validator that passed everything would raise the evaluation's score, so the evaluation could not tell a saving from a loss.
4. The target architect is not named: its mistakes are what cause Revisions, which cost far more than its own tokens.
5. Usage is reported per role and model, so a run shows what each tier used.
Differs from the plan: **there is no Jev question.** The plan asks Jev to pick a tier from the task. There is no data yet saying which tasks a lighter model handles, so any threshold would be a guess. The rule is the plan's second sentence alone: a Revision after a failed attempt goes to the stronger tier.
Expected saving if quality holds: about $0.40 of a $3.84 run, roughly 10%. Tests: `tests/execution/test_model_tiers.py` (3).
## Measured on 2026-10-09: no saving, so it stays off
Case 03, one run each, side by side (`20261009T195607Z` without tiers, `20261009T195611Z` with):
| | Pro throughout | Flash for the first attempt |
|---|---|---|
| IaC author's first attempt | 17 requests, $0.44 | 35 requests, $0.39 |
| Case result | 100% | 100% |
Flash wrote code that validated and passed every Check, but took twice the requests to get there: 35 of its budget of 40, and 336 seconds. Five cents saved, at the edge of the tool budget, is not a saving worth the risk of an attempt that runs out. `MODEL_TIERS` stays off. The run with tiers also had a Revision of the Target architecture that the other did not, which is unrelated to the tier and makes the whole-run totals ($3.02 against $2.44) not comparable.

View File

@@ -0,0 +1,11 @@
# Step 7 — Decision
Decided on 2026-10-09: **not to be built as planned.** The numbers do not support it.
The plan passes only the relevant sections of earlier Deliverables to the describer and the packager, to cut their tokens.
- **The describer's prompt is not earlier Deliverables.** In case 06 it was given the Target architecture, about 7 thousand tokens, and its prompt averaged 52 thousand per request. The rest is its Skill and the results of seven documentation searches, which accumulate. Trimming the Target architecture would save about a tenth of its prompt.
- **The packager needs what it is given.** It assembles the guide from every Deliverable's prose, in four requests.
- **Both are on Flash.** Together they cost $0.33 of a $3.84 run, and 80% of their prompt is cached at a tenth of the price.
What would cut the describer's tokens is fewer or shorter documentation searches, which is a change to its role file, not a Jev decision.

View File

@@ -25,3 +25,15 @@ Differs from the plan:
- **Not measured against its acceptance criterion.** Whether `load_skill_resource` calls fall needs a case run with it and one without.
Tests: `tests/execution/test_skill_routing.py` (5).
## Run on 2026-10-09
Ran in cases 05 and 06. The IaC author was named 7 references for the web application (service, SQL, IAM, load balancer, networking, project services, secrets) and 9 for the event-driven service; on the Change request, 3 (KMS, monitoring, Pub/Sub). Both cases passed. Whether `load_skill_resource` calls fell was not measured.
One to look at: for the retention change, whose subject is a bucket, the Cloud Storage reference was not among the three named.
## Corrected and changed later on 2026-10-09
The "one to look at" above was misread. The three references (KMS, monitoring, Pub/Sub) were named for a Revision inside the Baseline, from that Revision's feedback. The Change request itself was named nine, Cloud Storage among them.
What it did show is that a Change request's question listed every Element and not what changed. The task now says what has changed since delivery. Against live Jev on case 05's change, Cloud Storage rose from 1.63 to 1.95, the highest of all, and the load balancer, monitoring and Secret Manager patterns fell below the threshold.

View File

@@ -0,0 +1,11 @@
# Step 9 — Decision
Decided on 2026-10-09: **not to be built as planned.** It would be likely to raise cost, not lower it.
The plan drops or truncates old tool results inside one agent run before each model request.
- **The prompt is already mostly cached.** In case 06 the validator's prompt was 82% cached, the target architect's 69% and the IaC author's 60%, and a cached token costs a tenth of an uncached one.
- **Caching depends on the prompt's beginning staying the same.** Removing or shortening an earlier tool result changes everything after it, so the request that follows is paid at the full price.
- **The arithmetic.** A request of 46 thousand tokens, 82% cached, costs about $0.024 on the Pro model. Cut to 30 thousand with the cache lost, it costs $0.060.
It could still help where a run nears the model's context limit, which no run has. The cost that remains in these roles is output and thinking tokens, and the number of requests; neither is what this step addresses.