- Schema review: a dotted path is judged only when it starts at a name the resource type has, so a database flag or a service name is no longer read as a missing setting. Lesson L-04, which worked around that, is retired. - Skill routing: in a Change request the question says what changed. - Evaluation: a case may list code_standards the infrastructure code must meet; case 06 checks its overlay's regions, naming and labels. - Model tiers (iteration 5 step 6): a lighter model for a role's first attempt, named in the provider bundle. Measured on case 03 and left off: Flash saved five cents on the IaC author and took twice the requests. - Steps 7 and 9 decided against on the numbers, recorded in the plans. - Plans: the runs of cases 05 and 06, and the first proof of the lessons on case 03 (no Revision, 22 minutes against 28, quality held). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.0 KiB
Step 12 — Implementation Detail
Built on 2026-10-09. Not run with real models.
- An overlay Skill is an ordinary Agent Skill in a folder the deployment names:
OVERLAY_SKILLS_DIRS(defaultskills/overlays).discover_overlaysinapp/agent.pyreads them with the same validation as the provider's Skills and refuses one that reuses a provider Skill's name. ExecutionEngine.start(overlay_skills=[...])names the overlays for one Execution.DEFAULT_OVERLAY_SKILLSnames the ones every Execution loads. An unknown name, or anything but a list of names, refuses the start before anything runs or is recorded. A caller names an overlay; nothing lets it supply one.- Every Specialist is given the Execution's overlays on top of its own Skills, and its context lists them as
overlay_skills. The protocol says an overlay wins over the provider's general guidance, and a requirement the user stated wins over an overlay. - The Execution records each overlay with the content hash it had: in its state, in
execution.jsonand in its lesson candidates. - Callers:
overlay_skillsin the/generatebody, in A2A message metadata, and on an evaluation case. skills/overlays/example-organisation-standards/is an example: naming, four mandatory labels, approved regions, three prohibitions. It is marked as an example, is not loaded by default, and is not any real organisation's policy.
Differs from the plan:
- Skills are still loaded once at start-up. The runtime loads every provider Skill and every overlay in the configured folders when it starts; an Execution chooses among them. An overlay added to the folder needs a restart.
- No organisation standards Skill is loaded by default. The plan's criterion "loaded by the roles the bundle lists" is met by naming it, not by the bundle: whose standards apply is the deployment's and the caller's to say.
case-06-organisation-standards in the evaluation dataset runs case 03's request under the example overlay. It has not been run.
Tests: tests/execution/test_overlay_skills.py (8).
Run on 2026-10-09
case-06-organisation-standards (results 20261009T171333Z): completed, 100%, 36 minutes, 3.66 million tokens, one Revision of the Target architecture. execution.json records the overlay with its hash. The code follows it: every resource is named ${var.workload}-${var.environment}-<purpose>, the only region is europe-west1, and the labels are set. The same request without the overlay last took 22 minutes and 2.32 million tokens (20261009T103827Z); one run each, so the difference is not attributed.
No evaluation check scores whether an overlay was followed; this was read from the code by hand.
An evaluation check, later on 2026-10-09
A case may now carry code_standards: regular expressions the infrastructure code must contain or must not, scored as code_follows_standards. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.