Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-6.md
Jonathan Boniface e5e1647ff1
Some checks failed
validation / verify (push) Failing after 13s
feat: prove the lessons, check overlays in evaluation, and settle the cost steps
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00

2.8 KiB

Step 6 — Implementation Detail

Built on 2026-10-09, off by default (MODEL_TIERS) until measured.

Where case 06's cost went (20261009T171333Z, about $3.84 at the prices in the plan map):

Role Model Requests Prompt tokens Of them cached Cost
validator Pro 22 1,025,080 82% $1.04
target-architect Pro 24 686,263 69% $1.02
iac-author Pro 16 434,234 60% $0.63
describer Flash 13 675,284 86% $0.16
packager Flash 4 233,373 72% $0.17
the other six mixed 38 371,088 $0.82

The three roles on the Pro model are 70% of the cost. For each, output and thinking tokens cost about as much as the prompt.

  1. models.first_attempt in the provider bundle names a lighter model for a role's first attempt at a Deliverable. With MODEL_TIERS on, the role takes it when nothing has been asked of the Deliverable again: there is no feedback on it, and no earlier attempt ended without one. Any other attempt takes the role's own model.
  2. Only the IaC author is named, on Flash. Its work is checked twice by something else: terraform validate, and the validator.
  3. The validator is deliberately not named, though it costs the most. It is the check on the code. A weaker validator that passed everything would raise the evaluation's score, so the evaluation could not tell a saving from a loss.
  4. The target architect is not named: its mistakes are what cause Revisions, which cost far more than its own tokens.
  5. Usage is reported per role and model, so a run shows what each tier used.

Differs from the plan: there is no Jev question. The plan asks Jev to pick a tier from the task. There is no data yet saying which tasks a lighter model handles, so any threshold would be a guess. The rule is the plan's second sentence alone: a Revision after a failed attempt goes to the stronger tier.

Expected saving if quality holds: about $0.40 of a $3.84 run, roughly 10%. Tests: tests/execution/test_model_tiers.py (3).

Measured on 2026-10-09: no saving, so it stays off

Case 03, one run each, side by side (20261009T195607Z without tiers, 20261009T195611Z with):

Pro throughout Flash for the first attempt
IaC author's first attempt 17 requests, $0.44 35 requests, $0.39
Case result 100% 100%

Flash wrote code that validated and passed every Check, but took twice the requests to get there: 35 of its budget of 40, and 336 seconds. Five cents saved, at the edge of the tool budget, is not a saving worth the risk of an attempt that runs out. MODEL_TIERS stays off. The run with tiers also had a Revision of the Target architecture that the other did not, which is unrelated to the tier and makes the whole-run totals ($3.02 against $2.44) not comparable.