- Schema review: a dotted path is judged only when it starts at a name the resource type has, so a database flag or a service name is no longer read as a missing setting. Lesson L-04, which worked around that, is retired. - Skill routing: in a Change request the question says what changed. - Evaluation: a case may list code_standards the infrastructure code must meet; case 06 checks its overlay's regions, naming and labels. - Model tiers (iteration 5 step 6): a lighter model for a role's first attempt, named in the provider bundle. Measured on case 03 and left off: Flash saved five cents on the IaC author and took twice the requests. - Steps 7 and 9 decided against on the numbers, recorded in the plans. - Plans: the runs of cases 05 and 06, and the first proof of the lessons on case 03 (no Revision, 22 minutes against 28, quality held). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.8 KiB
Step 6 — Implementation Detail
Built on 2026-10-09, off by default (MODEL_TIERS) until measured.
Where case 06's cost went (20261009T171333Z, about $3.84 at the prices in the plan map):
| Role | Model | Requests | Prompt tokens | Of them cached | Cost |
|---|---|---|---|---|---|
| validator | Pro | 22 | 1,025,080 | 82% | $1.04 |
| target-architect | Pro | 24 | 686,263 | 69% | $1.02 |
| iac-author | Pro | 16 | 434,234 | 60% | $0.63 |
| describer | Flash | 13 | 675,284 | 86% | $0.16 |
| packager | Flash | 4 | 233,373 | 72% | $0.17 |
| the other six | mixed | 38 | 371,088 | $0.82 |
The three roles on the Pro model are 70% of the cost. For each, output and thinking tokens cost about as much as the prompt.
models.first_attemptin the provider bundle names a lighter model for a role's first attempt at a Deliverable. WithMODEL_TIERSon, the role takes it when nothing has been asked of the Deliverable again: there is no feedback on it, and no earlier attempt ended without one. Any other attempt takes the role's own model.- Only the IaC author is named, on Flash. Its work is checked twice by something else:
terraform validate, and the validator. - The validator is deliberately not named, though it costs the most. It is the check on the code. A weaker validator that passed everything would raise the evaluation's score, so the evaluation could not tell a saving from a loss.
- The target architect is not named: its mistakes are what cause Revisions, which cost far more than its own tokens.
- Usage is reported per role and model, so a run shows what each tier used.
Differs from the plan: there is no Jev question. The plan asks Jev to pick a tier from the task. There is no data yet saying which tasks a lighter model handles, so any threshold would be a guess. The rule is the plan's second sentence alone: a Revision after a failed attempt goes to the stronger tier.
Expected saving if quality holds: about $0.40 of a $3.84 run, roughly 10%. Tests: tests/execution/test_model_tiers.py (3).
Measured on 2026-10-09: no saving, so it stays off
Case 03, one run each, side by side (20261009T195607Z without tiers, 20261009T195611Z with):
| Pro throughout | Flash for the first attempt | |
|---|---|---|
| IaC author's first attempt | 17 requests, $0.44 | 35 requests, $0.39 |
| Case result | 100% | 100% |
Flash wrote code that validated and passed every Check, but took twice the requests to get there: 35 of its budget of 40, and 336 seconds. Five cents saved, at the edge of the tool budget, is not a saving worth the risk of an attempt that runs out. MODEL_TIERS stays off. The run with tiers also had a Revision of the Target architecture that the other did not, which is unrelated to the tier and makes the whole-run totals ($3.02 against $2.44) not comparable.