Files
solution_architecture_agent/docs/adr/0004-lessons-and-overlays-arrive-as-skills.md
Jonathan Boniface c84dbdea20 feat: the solution architecture agent on the shared Execution engine
The reference agent's subject, cloud solution architecture, as an agent in
the operating model agent's pattern: the Execution engine (from
orchestration_engine) and one Workflow in app/solution_architecture,
built for the settings and provider bundle by assembly.workflow_for.

Copied from gcp_solution_architecture_agent at 8b397cd without its own
engine: its role files, Skills, provider bundle, ADRs, vocabulary, scripts
and evaluation datasets. The image takes the engine from a named build
context and installs Terraform. It is called through the Orchestration
engine, and its files are what the operating model agent digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:49:28 +01:00

4.3 KiB

status
status
accepted

Lessons and overlays arrive as Skills

An Execution starts clean (CONTEXT.md) unless it is a Change request with a named Baseline (ADR-0006), so nothing one Execution learns may flow into the next as hidden state. What an Execution gets wrong is still worth keeping. We decide that everything that changes how agents behave between Executions is a Skill change, of one of two kinds.

  • Lessons. Each Execution records lesson candidates: Revisions and their reasons, validator findings, tool refusals, the user's change requests, and work that stopped. An offline curator proposes lessons from candidates across Executions. A person approves each one. An approved lesson is added to a lessons Skill that the provider bundle assigns to roles.
  • Overlays. An organisation's standards or one engagement's constraints are written as their own Skills. The provider bundle lists the ones every Execution loads. A caller may name further overlay Skills when it starts an Execution.

Vendored Skills stay unmodified (ADR-0002); lessons and overlays sit beside them.

Decisions

Accepted by the owner on 2026-10-08, agreeing to each of these:

  1. No lesson takes effect without a person's approval. The curator proposes; it never edits a Skill that agents load. We prefer slower learning to an agent that rewrites its own instructions.
  2. A lesson must be shown to help. It is kept only if the evaluation, run with and without it, holds quality and lowers Revisions, time or tokens.
  3. Lessons are per Cloud provider. The lessons Skill is listed in that provider's bundle, since most lessons name its products.
  4. Overlay Skills come only from directories the deployment configures. A caller names an overlay by its Skill name; it cannot upload one. Each passes the same Agent Skills validation as bundled Skills, and an unknown name refuses the Execution at the start.
  5. Every Execution records the Skills it loaded and a content hash of each, so a result can be traced to the instructions that produced it.
  6. Lesson candidates are client data. They quote user feedback and name resources. They are stored with the Execution's records, never in the repository, and a lesson is written so that it carries no client detail.

Considered Options

  • Memory carried between Executions (a vector store of past Deliverables or feedback, retrieved at run time): rejected. It breaks "every Execution starts clean", can leak one client's content into another's Execution, and cannot be reviewed or rolled back as a diff.
  • Automatic Skill edits (the curator applies lessons directly): rejected for now. Without review, one bad Execution can degrade every later one, and nothing ties a regression to its cause.
  • Editing the vendored Skills in place: rejected by ADR-0002. Upstream changes must arrive as a clean diff.
  • Overlays as prompt text passed by the caller: rejected. Free text bypasses Skill validation and progressive loading, and it is an injection path.

Consequences

  • Learning is slow by design: a lesson reaches agents only after capture, proposal, review and an evaluation run.
  • The evaluation must be able to see a difference a Skill makes. Its structural checks cannot, so the judged checks and the thresholds on Revisions, time and tokens become part of the loop.
  • The glossary gains Lesson candidate, Lesson and Overlay Skill.
  • Skills can no longer be loaded once at start-up for all Executions: the runtime must resolve a role's Skills per Execution.

As built (2026-10-09)

Two points differ from the text above; the decisions stand.

  • Default overlays are a deployment setting, not part of the provider bundle. DEFAULT_OVERLAY_SKILLS names the overlays every Execution loads. Whose standards apply belongs to the deployment and the caller, not to a Cloud provider.
  • Skills are still loaded once at start-up. The runtime loads every provider Skill and every overlay in the configured folders when it starts, and an Execution chooses among them. An overlay added to a folder needs a restart.

Decision 2 has a first result: on one benchmark case run each way, the lessons held quality and removed a Revision. One lesson was retired because the fault was in a review, not in the agents.