feat: the lessons loop, and six lessons approved and published

Capture, distil, review, publish, retire (ADR-0004). Lesson candidates of
past Executions are grouped into what recurred; a lesson-curator role,
outside any Execution, proposes a lesson per group worth one; each proposal
is an issue file a person approves or rejects; an approved lesson is
written into the provider's lessons Skill, which a Specialist is given only
once it holds lessons for that Specialist's Deliverable. LESSONS_ENABLED
off runs without them, for an evaluation with and without.

The curator proposed six lessons from 21 Executions. The owner approved all
six on 2026-10-09 and they are published: five for the target architect,
one for the IaC author. None is proved by an evaluation yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 15:19:32 +01:00
parent 59a641fded
commit a7bc5310f3
29 changed files with 1560 additions and 6 deletions

View File

@@ -0,0 +1,44 @@
# Step 10 — Implementation Detail
Built on 2026-10-09: the loop, and its first six lessons, which the owner approved the same day and which are published. **None is proved yet**: no evaluation has run with and without them, so the step's last acceptance criterion is open.
## The loop
| Stage | Where | What it does |
|---|---|---|
| Capture | `app/states/lessons.py` (step 7) | Each Execution records its lesson candidates. |
| Distil | `app/learning/evidence.py` | Reads every Execution's candidates where it wrote them and groups what recurred in at least two Executions, with ids and values removed from the description. A refusal the Execution's own permissions caused, and the router's own questions, are not evidence. |
| | `app/learning/curator.py`, `agents/lesson-curator.md` | The `lesson-curator` role, outside any Execution, proposes a lesson for each group worth one, or says why not. A proposal naming a Customer case or an Execution is refused. |
| Review | `app/learning/proposals.py` | Each proposal is `.scratch/lessons/issues/NN-<slug>.md`, `Status: needs-triage`, with its rule, reason and evidence as counts. A person sets `ready-for-agent` to approve or `wontfix` to reject. |
| Publish | `app/learning/skill.py` | Writes an approved lesson into `skills/google-cloud-lessons/references/<deliverable>.md` and rewrites the index in `SKILL.md`. Refuses anything not approved. |
| Prove | `LESSONS_ENABLED` | `false` leaves the lessons Skill out of every role, so the evaluation can run with and without. The comparison goes under the proposal's `## Proof`. |
| Retire | `app/learning/skill.py` | Removes a lesson by id; an emptied reference is deleted. |
`eval/curate_lessons.py` runs it: `propose`, `status`, `publish`, `retire L-NN`.
A Specialist is given the lessons Skill only once it holds a reference for that Specialist's Deliverable (`load_roles`), so nobody loads an empty Skill. The provider bundle names the Skill (`lessons_skill`), which makes lessons per Cloud provider. The protocol tells a Specialist with a lessons Skill to read its Deliverable's reference before drafting. Each Execution already records the version of every Skill it loaded.
## Differs from the plan
- **A lesson's source is its proposal, not an Execution.** The plan said each lesson records its source Execution. An Execution id names a Customer case, and ADR-0004 says a lesson carries no client detail, so the Skill records the proposal number and the proposal records how many Executions the evidence came from.
- **Retiring at each vendor sync is not automated.** `retire` exists; nothing yet compares lessons with a new upstream Skill.
- **Proving is a manual comparison** of two evaluation runs. Nothing computes it or writes it into the proposal.
## First proposals
Run on 2026-10-09 over the candidates of 21 Executions (`eval/results/`), which held 14 recurring problems. The curator proposed six and passed over eight:
| Id | Deliverable | Lesson | Evidence |
|---|---|---|---|
| L-01 | Target architecture | A Check for every explicit constraint in a requirement, not one broad Check | 14 of 21 Executions, 58 times |
| L-02 | Target architecture | Every Element has a Check before submitting | 13 of 21, 13 times |
| L-03 | Target architecture | Confirm the resource type supports the argument a Check names | 3 of 21 |
| L-04 | Target architecture | Name the block's argument, not the expected value, as the setting | 2 of 21 |
| L-05 | Deployment guidance | Include required arguments in alert-policy blocks | 2 of 21 |
| L-06 | Target architecture | `not_applicable` is for recommendations, never requirements | 2 of 21 |
The owner approved all six on 2026-10-09 and they were published: five for the target architect, one for the IaC author, which are now the two roles given the lessons Skill. Several restate what instructions or checks already say (L-02 is refused at submission today; L-03 is what the schema review does), which is the kind of thing a reviewer should weigh: a lesson costs tokens on every Execution.
## Tests
`tests/test_lessons_loop.py` (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill.