Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-8.md
Jonathan Boniface e5e1647ff1
Some checks failed
validation / verify (push) Failing after 13s
feat: prove the lessons, check overlays in evaluation, and settle the cost steps
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00

3.4 KiB

Step 8 — Implementation Detail

Built on 2026-10-09, on by default (JEV_SKILL_ROUTING). No Execution has run with it.

  1. app/tools/skill_routing.py. A Skill with four or more reference files is one worth routing within; each reference is described by its title and first paragraph. Today that is google-cloud-iac-patterns (14 references), which only the IaC author loads.
  2. When a Specialist whose Skills include such a Skill is started, the engine asks one graded question per reference: how much does this task need a reference on this (not needed, might help, needed). The task is the Deliverable, any feedback, and the Target architecture's Elements with their products and resource types.
  3. References at JEV_SKILL_THRESHOLD (1.25 on that 0 to 2 scale) or above are named in the Specialist's context as read_first. The protocol says to read those first, and that it is a place to start, not a limit.
  4. With no decision nothing is named, and the Specialist searches its Skills as before.

Measured against live Jev on four tasks, each reference scored 0 to 2:

Task Named at 1.25 Highest not named
Web app: Cloud Run, Cloud SQL, Secret Manager, load balancer, IAM, VPC service, SQL, secrets, load balancer, IAM, networking, project services, Artifact Registry KMS 0.72
Nightly batch: Cloud Run job, Scheduler, Storage, IAM job, scheduler, storage, IAM, project services, Artifact Registry Secret Manager 0.90
Events: Pub/Sub, Cloud Run, Artifact Registry, KMS, Monitoring all seven it needs Secret Manager 1.09
A Revision: one failed Check on a bucket's retention storage only project services 0.96

The Cloud Run service pattern scored 1.39 and 1.42 where it was needed, which is why the threshold is 1.25 and not 1.5. The cost of that is the Artifact Registry pattern being named for two designs that run containers and did not list a registry.

Differs from the plan:

  • A graded question per reference, not one Choice. A task needs several references, and a Choice names one.
  • Overlay Skills are not selected by Jev. The caller or the deployment names them (iteration 4 step 12): which organisation's standards apply is not something to infer.
  • Not measured against its acceptance criterion. Whether load_skill_resource calls fall needs a case run with it and one without.

Tests: tests/execution/test_skill_routing.py (5).

Run on 2026-10-09

Ran in cases 05 and 06. The IaC author was named 7 references for the web application (service, SQL, IAM, load balancer, networking, project services, secrets) and 9 for the event-driven service; on the Change request, 3 (KMS, monitoring, Pub/Sub). Both cases passed. Whether load_skill_resource calls fell was not measured.

One to look at: for the retention change, whose subject is a bucket, the Cloud Storage reference was not among the three named.

Corrected and changed later on 2026-10-09

The "one to look at" above was misread. The three references (KMS, monitoring, Pub/Sub) were named for a Revision inside the Baseline, from that Revision's feedback. The Change request itself was named nine, Cloud Storage among them.

What it did show is that a Change request's question listed every Element and not what changed. The task now says what has changed since delivery. Against live Jev on case 05's change, Cloud Storage rose from 1.63 to 1.95, the highest of all, and the load balancer, monitoring and Secret Manager patterns fell below the threshold.