Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-12.md
Jonathan Boniface e5e1647ff1
Some checks failed
validation / verify (push) Failing after 13s
feat: prove the lessons, check overlays in evaluation, and settle the cost steps
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00

3.0 KiB

Step 12 — Implementation Detail

Built on 2026-10-09. Not run with real models.

  1. An overlay Skill is an ordinary Agent Skill in a folder the deployment names: OVERLAY_SKILLS_DIRS (default skills/overlays). discover_overlays in app/agent.py reads them with the same validation as the provider's Skills and refuses one that reuses a provider Skill's name.
  2. ExecutionEngine.start(overlay_skills=[...]) names the overlays for one Execution. DEFAULT_OVERLAY_SKILLS names the ones every Execution loads. An unknown name, or anything but a list of names, refuses the start before anything runs or is recorded. A caller names an overlay; nothing lets it supply one.
  3. Every Specialist is given the Execution's overlays on top of its own Skills, and its context lists them as overlay_skills. The protocol says an overlay wins over the provider's general guidance, and a requirement the user stated wins over an overlay.
  4. The Execution records each overlay with the content hash it had: in its state, in execution.json and in its lesson candidates.
  5. Callers: overlay_skills in the /generate body, in A2A message metadata, and on an evaluation case.
  6. skills/overlays/example-organisation-standards/ is an example: naming, four mandatory labels, approved regions, three prohibitions. It is marked as an example, is not loaded by default, and is not any real organisation's policy.

Differs from the plan:

  • Skills are still loaded once at start-up. The runtime loads every provider Skill and every overlay in the configured folders when it starts; an Execution chooses among them. An overlay added to the folder needs a restart.
  • No organisation standards Skill is loaded by default. The plan's criterion "loaded by the roles the bundle lists" is met by naming it, not by the bundle: whose standards apply is the deployment's and the caller's to say.

case-06-organisation-standards in the evaluation dataset runs case 03's request under the example overlay. It has not been run.

Tests: tests/execution/test_overlay_skills.py (8).

Run on 2026-10-09

case-06-organisation-standards (results 20261009T171333Z): completed, 100%, 36 minutes, 3.66 million tokens, one Revision of the Target architecture. execution.json records the overlay with its hash. The code follows it: every resource is named ${var.workload}-${var.environment}-<purpose>, the only region is europe-west1, and the labels are set. The same request without the overlay last took 22 minutes and 2.32 million tokens (20261009T103827Z); one run each, so the difference is not attributed.

No evaluation check scores whether an overlay was followed; this was read from the code by hand.

An evaluation check, later on 2026-10-09

A case may now carry code_standards: regular expressions the infrastructure code must contain or must not, scored as code_follows_standards. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.