feat: the lessons loop, and six lessons approved and published

Capture, distil, review, publish, retire (ADR-0004). Lesson candidates of
past Executions are grouped into what recurred; a lesson-curator role,
outside any Execution, proposes a lesson per group worth one; each proposal
is an issue file a person approves or rejects; an approved lesson is
written into the provider's lessons Skill, which a Specialist is given only
once it holds lessons for that Specialist's Deliverable. LESSONS_ENABLED
off runs without them, for an evaluation with and without.

The curator proposed six lessons from 21 Executions. The owner approved all
six on 2026-10-09 and they are published: five for the target architect,
one for the IaC author. None is proved by an evaluation yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 15:19:32 +01:00
parent 59a641fded
commit a7bc5310f3
29 changed files with 1560 additions and 6 deletions

View File

@@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); the lessons loop is built and its first six lessons are approved and published, not yet proved (step 10); 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |

View File

@@ -26,7 +26,7 @@
| 7 | Capture lesson candidates | B | M | complete |
| 8 | Evaluation that can see improvement | B | M | complete: thresholds set and four judged runs done on 2026-10-08 |
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
| 10 | Lessons Skill and lesson curator | B | L | planned |
| 10 | Lessons Skill and lesson curator | B | L | the loop is built; six lessons proposed from 21 Executions, approved by the owner and published on 2026-10-09; not yet proved by an evaluation with and without (`steps/step-10.md`) |
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
| 13 | Trim Specialist context | C | M | built; measured on case 01 only, together with steps 11 and 17 |

View File

@@ -0,0 +1,44 @@
# Step 10 — Implementation Detail
Built on 2026-10-09: the loop, and its first six lessons, which the owner approved the same day and which are published. **None is proved yet**: no evaluation has run with and without them, so the step's last acceptance criterion is open.
## The loop
| Stage | Where | What it does |
|---|---|---|
| Capture | `app/states/lessons.py` (step 7) | Each Execution records its lesson candidates. |
| Distil | `app/learning/evidence.py` | Reads every Execution's candidates where it wrote them and groups what recurred in at least two Executions, with ids and values removed from the description. A refusal the Execution's own permissions caused, and the router's own questions, are not evidence. |
| | `app/learning/curator.py`, `agents/lesson-curator.md` | The `lesson-curator` role, outside any Execution, proposes a lesson for each group worth one, or says why not. A proposal naming a Customer case or an Execution is refused. |
| Review | `app/learning/proposals.py` | Each proposal is `.scratch/lessons/issues/NN-<slug>.md`, `Status: needs-triage`, with its rule, reason and evidence as counts. A person sets `ready-for-agent` to approve or `wontfix` to reject. |
| Publish | `app/learning/skill.py` | Writes an approved lesson into `skills/google-cloud-lessons/references/<deliverable>.md` and rewrites the index in `SKILL.md`. Refuses anything not approved. |
| Prove | `LESSONS_ENABLED` | `false` leaves the lessons Skill out of every role, so the evaluation can run with and without. The comparison goes under the proposal's `## Proof`. |
| Retire | `app/learning/skill.py` | Removes a lesson by id; an emptied reference is deleted. |
`eval/curate_lessons.py` runs it: `propose`, `status`, `publish`, `retire L-NN`.
A Specialist is given the lessons Skill only once it holds a reference for that Specialist's Deliverable (`load_roles`), so nobody loads an empty Skill. The provider bundle names the Skill (`lessons_skill`), which makes lessons per Cloud provider. The protocol tells a Specialist with a lessons Skill to read its Deliverable's reference before drafting. Each Execution already records the version of every Skill it loaded.
## Differs from the plan
- **A lesson's source is its proposal, not an Execution.** The plan said each lesson records its source Execution. An Execution id names a Customer case, and ADR-0004 says a lesson carries no client detail, so the Skill records the proposal number and the proposal records how many Executions the evidence came from.
- **Retiring at each vendor sync is not automated.** `retire` exists; nothing yet compares lessons with a new upstream Skill.
- **Proving is a manual comparison** of two evaluation runs. Nothing computes it or writes it into the proposal.
## First proposals
Run on 2026-10-09 over the candidates of 21 Executions (`eval/results/`), which held 14 recurring problems. The curator proposed six and passed over eight:
| Id | Deliverable | Lesson | Evidence |
|---|---|---|---|
| L-01 | Target architecture | A Check for every explicit constraint in a requirement, not one broad Check | 14 of 21 Executions, 58 times |
| L-02 | Target architecture | Every Element has a Check before submitting | 13 of 21, 13 times |
| L-03 | Target architecture | Confirm the resource type supports the argument a Check names | 3 of 21 |
| L-04 | Target architecture | Name the block's argument, not the expected value, as the setting | 2 of 21 |
| L-05 | Deployment guidance | Include required arguments in alert-policy blocks | 2 of 21 |
| L-06 | Target architecture | `not_applicable` is for recommendations, never requirements | 2 of 21 |
The owner approved all six on 2026-10-09 and they were published: five for the target architect, one for the IaC author, which are now the two roles given the lessons Skill. Several restate what instructions or checks already say (L-02 is refused at submission today; L-03 is what the schema review does), which is the kind of thing a reviewer should weigh: a lesson costs tokens on every Execution.
## Tests
`tests/test_lessons_loop.py` (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill.