Iteration 4: - Overlay Skills (step 12): an Execution names an organisation's standards Skill from the folders the deployment configures; recorded with its version. An example standards Skill and an evaluation case that uses it. - Lessons (step 10): a prove command that compares a run with lessons against one without, and triage of evidence an existing lesson already covers. Iteration 5: - Step 8: the references of a Specialist's Skills its task needs are named in its context as read_first. - Step 14: a remote MCP server's reply that instructs the agent is withheld; the user's own text is flagged as a risk and nothing more. - Step 15: in a Change request, items only reworded are listed apart from real changes. - Step 19: each Execution's design recommendations are judged when it ends and the scores kept in its record. The evaluation takes its rubric from the same place. Each question was calibrated against live Jev; none has run in a case. Steps 6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.9 KiB
Step 10 — Implementation Detail
Built on 2026-10-09: the loop, and its first six lessons, which the owner approved the same day and which are published. None is proved yet: no evaluation has run with and without them, so the step's last acceptance criterion is open.
The loop
| Stage | Where | What it does |
|---|---|---|
| Capture | app/states/lessons.py (step 7) |
Each Execution records its lesson candidates. |
| Distil | app/learning/evidence.py |
Reads every Execution's candidates where it wrote them and groups what recurred in at least two Executions, with ids and values removed from the description. A refusal the Execution's own permissions caused, and the router's own questions, are not evidence. |
app/learning/curator.py, agents/lesson-curator.md |
The lesson-curator role, outside any Execution, proposes a lesson for each group worth one, or says why not. A proposal naming a Customer case or an Execution is refused. |
|
| Review | app/learning/proposals.py |
Each proposal is .scratch/lessons/issues/NN-<slug>.md, Status: needs-triage, with its rule, reason and evidence as counts. A person sets ready-for-agent to approve or wontfix to reject. |
| Publish | app/learning/skill.py |
Writes an approved lesson into skills/google-cloud-lessons/references/<deliverable>.md and rewrites the index in SKILL.md. Refuses anything not approved. |
| Prove | LESSONS_ENABLED |
false leaves the lessons Skill out of every role, so the evaluation can run with and without. The comparison goes under the proposal's ## Proof. |
| Retire | app/learning/skill.py |
Removes a lesson by id; an emptied reference is deleted. |
eval/curate_lessons.py runs it: propose, status, publish, retire L-NN.
A Specialist is given the lessons Skill only once it holds a reference for that Specialist's Deliverable (load_roles), so nobody loads an empty Skill. The provider bundle names the Skill (lessons_skill), which makes lessons per Cloud provider. The protocol tells a Specialist with a lessons Skill to read its Deliverable's reference before drafting. Each Execution already records the version of every Skill it loaded.
Differs from the plan
- A lesson's source is its proposal, not an Execution. The plan said each lesson records its source Execution. An Execution id names a Customer case, and ADR-0004 says a lesson carries no client detail, so the Skill records the proposal number and the proposal records how many Executions the evidence came from.
- Retiring at each vendor sync is not automated.
retireexists; nothing yet compares lessons with a new upstream Skill. - Proving is a manual comparison of two evaluation runs. Nothing computes it or writes it into the proposal.
First proposals
Run on 2026-10-09 over the candidates of 21 Executions (eval/results/), which held 14 recurring problems. The curator proposed six and passed over eight:
| Id | Deliverable | Lesson | Evidence |
|---|---|---|---|
| L-01 | Target architecture | A Check for every explicit constraint in a requirement, not one broad Check | 14 of 21 Executions, 58 times |
| L-02 | Target architecture | Every Element has a Check before submitting | 13 of 21, 13 times |
| L-03 | Target architecture | Confirm the resource type supports the argument a Check names | 3 of 21 |
| L-04 | Target architecture | Name the block's argument, not the expected value, as the setting | 2 of 21 |
| L-05 | Deployment guidance | Include required arguments in alert-policy blocks | 2 of 21 |
| L-06 | Target architecture | not_applicable is for recommendations, never requirements |
2 of 21 |
The owner approved all six on 2026-10-09 and they were published: five for the target architect, one for the IaC author, which are now the two roles given the lessons Skill. Several restate what instructions or checks already say (L-02 is refused at submission today; L-03 is what the schema review does), which is the kind of thing a reviewer should weigh: a lesson costs tokens on every Execution.
Tests
tests/test_lessons_loop.py (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill.
Added later on 2026-10-09
- Proving is a command.
eval.curate_lessons prove --without <report.json> --with <report.json>compares two evaluation reports case by case (score, Revisions, minutes, tokens), says whether quality held and anything fell, and writes the comparison into every published proposal's## Proof(app/learning/proof.py). The two runs are still made by hand, one withLESSONS_ENABLED=false. - Triage. Before the curator runs, each new group is put to the decider with the lessons that exist: which one would have prevented it, or none. A group a lesson covers is not put to the curator again (
_not_already_coveredinapp/learning/curator.py; iteration 5 step 10).
Still open: retiring lessons when the upstream Skills change is not automated, and no lesson has been proved.