Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-19.md
Jonathan Boniface e5e1647ff1
Some checks failed
validation / verify (push) Failing after 13s
feat: prove the lessons, check overlays in evaluation, and settle the cost steps
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00

1.9 KiB

Step 19 — Implementation Detail

Built on 2026-10-09, on by default (JEV_ASSESS_EXECUTIONS). No Execution has run with it.

  1. app/tools/assessment.py holds the graded rubric the evaluation uses for design recommendations (specific, actionable, trade-offs, traceable) and the bar (a mean of 0.67). eval/scoring.py and eval/judge.py now take the rubric, the bar and the grades from there, so a live Execution and a benchmark case are asked the same questions.
  2. When an Execution ends, completed or with warnings, its design recommendations are put to the decider once. The four scores, their mean and whether it meets the bar are kept in the Execution's state and in execution.json under assessment. A mean below the bar is logged as a warning.
  3. It reports only. Nothing stops or changes, and a failure to assess leaves the Execution with no assessment.
  4. Risk flags (step 14) are already in execution.json under risks.

Differs from the plan:

  • Only the quality rubric is asked. Checklist coverage is already settled inside every Execution by the pillar checklist review (iteration 4), in code. The structural checks need a case's expected products, which a live Execution does not have.
  • Nothing is sent as feedback on a trace. That part waits on step 17 and the owner's decision on LangSmith.
  • No alerting. A low score is a log line and a field in the record; nothing watches for it.

The judge behind it was measured in step 16: Jev answered the same question the same way every time (1.00) and agreed with Gemini on 0.91 of questions. Agreement with a person still awaits the owner's 32 labels.

Tests: tests/execution/test_assessment.py (5).

Run on 2026-10-09

Both Executions carry an assessment in execution.json: case 06 a mean of 0.74 (specific 0.76, actionable 0.77, trade-offs 0.75, traceable 0.69); case 05's Change request 0.90. Both meet the bar of 0.67.