5.8 KiB
Step 12 — Implementation Detail
Built on 2026-10-09. Not run with real models.
- An overlay Skill is an ordinary Agent Skill in a folder the deployment names:
OVERLAY_SKILLS_DIRS(defaultskills/overlays).discover_overlaysinapp/agent.pyreads them with the same validation as the provider's Skills and refuses one that reuses a provider Skill's name. ExecutionEngine.start(overlay_skills=[...])names the overlays for one Execution.DEFAULT_OVERLAY_SKILLSnames the ones every Execution loads. An unknown name, or anything but a list of names, refuses the start before anything runs or is recorded. A caller names an overlay; nothing lets it supply one.- Every Specialist is given the Execution's overlays on top of its own Skills, and its context lists them as
overlay_skills. The protocol says an overlay wins over the provider's general guidance, and a requirement the user stated wins over an overlay. - The Execution records each overlay with the content hash it had: in its state, in
execution.jsonand in its lesson candidates. - Callers:
overlay_skillsin the/generatebody, in A2A message metadata, and on an evaluation case. skills/overlays/example-organisation-standards/is an example: naming, four mandatory labels, approved regions, three prohibitions. It is marked as an example, is not loaded by default, and is not any real organisation's policy.
Differs from the plan:
- Skills are still loaded once at start-up. The runtime loads every provider Skill and every overlay in the configured folders when it starts; an Execution chooses among them. An overlay added to the folder needs a restart.
- No organisation standards Skill is loaded by default. The plan's criterion "loaded by the roles the bundle lists" is met by naming it, not by the bundle: whose standards apply is the deployment's and the caller's to say.
case-06-organisation-standards in the evaluation dataset runs case 03's request under the example overlay. It has not been run.
Tests: tests/execution/test_overlay_skills.py (8).
Run on 2026-10-09
case-06-organisation-standards (results 20261009T171333Z): completed, 100%, 36 minutes, 3.66 million tokens, one Revision of the Target architecture. execution.json records the overlay with its hash. The code follows it: every resource is named ${var.workload}-${var.environment}-<purpose>, the only region is europe-west1, and the labels are set. The same request without the overlay last took 22 minutes and 2.32 million tokens (20261009T103827Z); one run each, so the difference is not attributed.
No evaluation check scores whether an overlay was followed; this was read from the code by hand.
An evaluation check, later on 2026-10-09
A case may now carry code_standards: regular expressions the infrastructure code must contain or must not, scored as code_follows_standards. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.
Rerun on 2026-10-10: case 06 failed, case 04 passed
Both rerun after the week's changes (20261010T123446Z case 04, 20261010T123450Z case 06).
Case 04, interactive: completed, 12 of 12 checks, 34 minutes, about $4.58, one Revision of the Target architecture. Nothing in routing, intake, the guardrail or the writing of files as it goes broke it.
Case 06, overlay: ended with warnings, 67%, 42 minutes, about $7.72. The Target architecture reached its Revision limit of 3 and no guide was packaged. Two separate causes:
- A verdict that was wrong, and no way to say so. Check K-10 says a role is granted to the service identity. It was. The validator failed it because the identity also held three other roles that other Checks require. The IaC author, sent the failure, could change nothing, and recorded the Check as unimplementable with the reason "the validator insists on only this role". That sent the design back for a fault it did not have. The same Check passed in the validation between, on the same grants: the validator was not consistent. The other two Revisions were ordinary ones (labels and names the overlay asks for, and a resource whose name cannot contain hyphens).
- The evaluation's naming check was too narrow.
code_follows_standardswanted a variable with "env" in its name inside each resource name. The code built names fromlocal.name_prefix, itself made fromvar.workloadandvar.environment, which follows the standard. The check now asks that names are built from a variable or local, and that the environment comes from a variable; this run's code and the earlier run's both pass it.
Changed, not yet run:
- The validator is told to judge each Check by its own words: a Check that something is granted or set passes when it is, whatever is beside it, unless it says "only". What else it thinks should hold is a proposed addition, never a failure.
- The IaC author is told what to do when a Check failed and the code already meets its words: change nothing, keep the implementation entry, name the Check it disputes, and not record it as unimplementable.
An overlay also makes a design larger: this run ended with over 120 Checks, against 50 to 90 for the same request without one.
Case 06 again, with the fixes (2026-10-10)
20261010T134034Z: completed, 100%, 45 minutes, about $5.43, 126 Checks, all passed. code_follows_standards passed. Two Revisions of the Target architecture, both ordinary: locations the validator wanted checked (L-10's subject, the persistence regions of a topic, among them), and two Checks on one secret that contradicted each other.
No Check was failed for something it does not say, and none was disputed, so the two new instructions were not put to the test: the run shows they do no harm, not that they work.