Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-10.md
Jonathan Boniface 5feaaa6271
Some checks failed
validation / verify (push) Failing after 10s
feat: lessons from what recurs about one setting, and three more approved
Evidence was grouped only by the kind of problem, and a group gets one lesson,
so what kept recurring inside "validation proposed a Check" could not be
seen. A proposed or unimplementable Check is now also counted under each
provider setting or resource type it names; a subject seen in two or more
Executions is its own group and reaches the curator past triage.

Over 32 Executions that gave five proposals. The owner approved three, now
published: L-07 location Checks, L-08 Cloud SQL backup encryption, L-09
backend service capabilities. L-10 is held for more evidence and L-11
rejected.

Also: retiring a lesson marks its proposal, so publish no longer brings a
retired lesson back. L-04 had returned to the Skill that way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:47:18 +01:00

8.6 KiB

Step 10 — Implementation Detail

Built on 2026-10-09: the loop, and its first six lessons, which the owner approved the same day and which are published. None is proved yet: no evaluation has run with and without them, so the step's last acceptance criterion is open.

The loop

Stage Where What it does
Capture app/states/lessons.py (step 7) Each Execution records its lesson candidates.
Distil app/learning/evidence.py Reads every Execution's candidates where it wrote them and groups what recurred in at least two Executions, with ids and values removed from the description. A refusal the Execution's own permissions caused, and the router's own questions, are not evidence.
app/learning/curator.py, agents/lesson-curator.md The lesson-curator role, outside any Execution, proposes a lesson for each group worth one, or says why not. A proposal naming a Customer case or an Execution is refused.
Review app/learning/proposals.py Each proposal is .scratch/lessons/issues/NN-<slug>.md, Status: needs-triage, with its rule, reason and evidence as counts. A person sets ready-for-agent to approve or wontfix to reject.
Publish app/learning/skill.py Writes an approved lesson into skills/google-cloud-lessons/references/<deliverable>.md and rewrites the index in SKILL.md. Refuses anything not approved.
Prove LESSONS_ENABLED false leaves the lessons Skill out of every role, so the evaluation can run with and without. The comparison goes under the proposal's ## Proof.
Retire app/learning/skill.py Removes a lesson by id; an emptied reference is deleted.

eval/curate_lessons.py runs it: propose, status, publish, retire L-NN.

A Specialist is given the lessons Skill only once it holds a reference for that Specialist's Deliverable (load_roles), so nobody loads an empty Skill. The provider bundle names the Skill (lessons_skill), which makes lessons per Cloud provider. The protocol tells a Specialist with a lessons Skill to read its Deliverable's reference before drafting. Each Execution already records the version of every Skill it loaded.

Differs from the plan

  • A lesson's source is its proposal, not an Execution. The plan said each lesson records its source Execution. An Execution id names a Customer case, and ADR-0004 says a lesson carries no client detail, so the Skill records the proposal number and the proposal records how many Executions the evidence came from.
  • Retiring at each vendor sync is not automated. retire exists; nothing yet compares lessons with a new upstream Skill.
  • Proving is a manual comparison of two evaluation runs. Nothing computes it or writes it into the proposal.

First proposals

Run on 2026-10-09 over the candidates of 21 Executions (eval/results/), which held 14 recurring problems. The curator proposed six and passed over eight:

Id Deliverable Lesson Evidence
L-01 Target architecture A Check for every explicit constraint in a requirement, not one broad Check 14 of 21 Executions, 58 times
L-02 Target architecture Every Element has a Check before submitting 13 of 21, 13 times
L-03 Target architecture Confirm the resource type supports the argument a Check names 3 of 21
L-04 Target architecture Name the block's argument, not the expected value, as the setting 2 of 21
L-05 Deployment guidance Include required arguments in alert-policy blocks 2 of 21
L-06 Target architecture not_applicable is for recommendations, never requirements 2 of 21

The owner approved all six on 2026-10-09 and they were published: five for the target architect, one for the IaC author, which are now the two roles given the lessons Skill. Several restate what instructions or checks already say (L-02 is refused at submission today; L-03 is what the schema review does), which is the kind of thing a reviewer should weigh: a lesson costs tokens on every Execution.

Tests

tests/test_lessons_loop.py (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill.

Added later on 2026-10-09

  • Proving is a command. eval.curate_lessons prove --without <report.json> --with <report.json> compares two evaluation reports case by case (score, Revisions, minutes, tokens), says whether quality held and anything fell, and writes the comparison into every published proposal's ## Proof (app/learning/proof.py). The two runs are still made by hand, one with LESSONS_ENABLED=false.
  • Triage. Before the curator runs, each new group is put to the decider with the lessons that exist: which one would have prevented it, or none. A group a lesson covers is not put to the curator again (_not_already_covered in app/learning/curator.py; iteration 5 step 10).

Still open: retiring lessons when the upstream Skills change is not automated, and no lesson has been proved.

Run on 2026-10-09

Case 06 ran with the six lessons loaded and still needed one Revision of the Target architecture, as case 03 did before the lessons existed. That neither proves nor disproves them: it also carried an overlay. prove has not been run.

L-04 retired on 2026-10-09

L-04 told the target architect to word Checks so the schema review would not mistake a value for a setting. The review was at fault: it read cloudsql.iam_authentication (a database flag) and run.googleapis.com (a service) as settings, and raised both again in case 06 with the lesson loaded. It now judges a dotted path only when the path starts at a name the resource type has (_is_missing in app/tools/schema_review.py). The lesson is removed from the Skill; five remain.

First proof, 2026-10-09

Case 03, one run each way, side by side, with the five lessons that remain (20261009T195603Z without, 20261009T195607Z with):

Without lessons With lessons
Score 100% 100%
Revisions 1, of the Target architecture 0
Minutes 28 22
Tokens 3.00 million 2.32 million
Cost, about $3.47 $2.44

prove recorded this in the proposals as "kept". One pair suggests a direction and does not settle it: a third run the same evening, with lessons, did need a Revision of the Target architecture. prove also wrote to L-04's proposal, which is retired; it should skip retired lessons.

Neither run raised the schema review's false alarm, which had been raised in case 06 before the fix.

Housekeeping, 2026-10-10

  • prove skips retired lessons. It wrote its comparison into L-04's proposal after L-04 had been retired. It now writes only for lessons the Skill holds. L-04's proposal is corrected.
  • Lessons are flagged when the vendored Skills change. The lessons Skill keeps upstream.json: the version of each vendored Skill the lessons were last read against. status names the ones that have changed since and says to read each lesson again; reviewed records that a person has. A test fails until then, so a sync of the vendored Skills cannot go unnoticed. The first publish records the versions; after that only reviewed moves them.
  • status says "compared", not "proved", and shows a retired lesson as retired. One pair of runs is a comparison.

Tests: tests/test_lessons_loop.py.

Evidence is also grouped by what it is about (2026-10-10)

Running propose over 32 Executions gave nothing useful. Evidence was grouped by the kind of problem, and a group gets one lesson: "validation proposed a Check the design lacked" was 20 of 32 Executions and already had L-01. What recurred inside it could not be seen.

  • A proposed Check, and a Check sent back as unimplementable, is now also counted under each provider setting or resource type its text names (_subjects in app/learning/evidence.py). A subject seen in two or more Executions is its own group, with its own key. A setting's name is the provider's vocabulary, not a client's.
  • These narrower groups skip triage: each recurred with the general lessons published. The curator is shown the existing lessons and may still decline.
  • The curator's instructions say such a lesson may name the setting or resource type.

Over the same 32 Executions this gave 22 groups and five proposals, L-07 to L-11, all awaiting the owner: location Checks, Cloud SQL backup encryption, backend service capabilities, Pub/Sub persistence regions, and customer-managed keys.

A first run of propose before this change produced one proposal, about giving edit_file enough context. It was removed to run again cleanly and was not proposed the second time.

Tests: tests/test_lessons_loop.py (3 more).