diff --git a/.agents/plans/README.md b/.agents/plans/README.md index 60d2ee4..31c1caf 100644 --- a/.agents/plans/README.md +++ b/.agents/plans/README.md @@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio | 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. | | 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. | | 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. | -| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | +| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); the lessons loop is built and its first six lessons are approved and published, not yet proved (step 10); 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | | 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. | | 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. | | 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | diff --git a/.agents/plans/iter-04-learning-and-skills/MAP.md b/.agents/plans/iter-04-learning-and-skills/MAP.md index 17ed329..770015f 100644 --- a/.agents/plans/iter-04-learning-and-skills/MAP.md +++ b/.agents/plans/iter-04-learning-and-skills/MAP.md @@ -26,7 +26,7 @@ | 7 | Capture lesson candidates | B | M | complete | | 8 | Evaluation that can see improvement | B | M | complete: thresholds set and four judged runs done on 2026-10-08 | | 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 | -| 10 | Lessons Skill and lesson curator | B | L | planned | +| 10 | Lessons Skill and lesson curator | B | L | the loop is built; six lessons proposed from 21 Executions, approved by the owner and published on 2026-10-09; not yet proved by an evaluation with and without (`steps/step-10.md`) | | 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured | | 12 | Overlay Skills and an organisation standards Skill | C | M | planned | | 13 | Trim Specialist context | C | M | built; measured on case 01 only, together with steps 11 and 17 | diff --git a/.agents/plans/iter-04-learning-and-skills/steps/step-10.md b/.agents/plans/iter-04-learning-and-skills/steps/step-10.md new file mode 100644 index 0000000..ab8cf10 --- /dev/null +++ b/.agents/plans/iter-04-learning-and-skills/steps/step-10.md @@ -0,0 +1,44 @@ +# Step 10 — Implementation Detail + +Built on 2026-10-09: the loop, and its first six lessons, which the owner approved the same day and which are published. **None is proved yet**: no evaluation has run with and without them, so the step's last acceptance criterion is open. + +## The loop + +| Stage | Where | What it does | +|---|---|---| +| Capture | `app/states/lessons.py` (step 7) | Each Execution records its lesson candidates. | +| Distil | `app/learning/evidence.py` | Reads every Execution's candidates where it wrote them and groups what recurred in at least two Executions, with ids and values removed from the description. A refusal the Execution's own permissions caused, and the router's own questions, are not evidence. | +| | `app/learning/curator.py`, `agents/lesson-curator.md` | The `lesson-curator` role, outside any Execution, proposes a lesson for each group worth one, or says why not. A proposal naming a Customer case or an Execution is refused. | +| Review | `app/learning/proposals.py` | Each proposal is `.scratch/lessons/issues/NN-.md`, `Status: needs-triage`, with its rule, reason and evidence as counts. A person sets `ready-for-agent` to approve or `wontfix` to reject. | +| Publish | `app/learning/skill.py` | Writes an approved lesson into `skills/google-cloud-lessons/references/.md` and rewrites the index in `SKILL.md`. Refuses anything not approved. | +| Prove | `LESSONS_ENABLED` | `false` leaves the lessons Skill out of every role, so the evaluation can run with and without. The comparison goes under the proposal's `## Proof`. | +| Retire | `app/learning/skill.py` | Removes a lesson by id; an emptied reference is deleted. | + +`eval/curate_lessons.py` runs it: `propose`, `status`, `publish`, `retire L-NN`. + +A Specialist is given the lessons Skill only once it holds a reference for that Specialist's Deliverable (`load_roles`), so nobody loads an empty Skill. The provider bundle names the Skill (`lessons_skill`), which makes lessons per Cloud provider. The protocol tells a Specialist with a lessons Skill to read its Deliverable's reference before drafting. Each Execution already records the version of every Skill it loaded. + +## Differs from the plan + +- **A lesson's source is its proposal, not an Execution.** The plan said each lesson records its source Execution. An Execution id names a Customer case, and ADR-0004 says a lesson carries no client detail, so the Skill records the proposal number and the proposal records how many Executions the evidence came from. +- **Retiring at each vendor sync is not automated.** `retire` exists; nothing yet compares lessons with a new upstream Skill. +- **Proving is a manual comparison** of two evaluation runs. Nothing computes it or writes it into the proposal. + +## First proposals + +Run on 2026-10-09 over the candidates of 21 Executions (`eval/results/`), which held 14 recurring problems. The curator proposed six and passed over eight: + +| Id | Deliverable | Lesson | Evidence | +|---|---|---|---| +| L-01 | Target architecture | A Check for every explicit constraint in a requirement, not one broad Check | 14 of 21 Executions, 58 times | +| L-02 | Target architecture | Every Element has a Check before submitting | 13 of 21, 13 times | +| L-03 | Target architecture | Confirm the resource type supports the argument a Check names | 3 of 21 | +| L-04 | Target architecture | Name the block's argument, not the expected value, as the setting | 2 of 21 | +| L-05 | Deployment guidance | Include required arguments in alert-policy blocks | 2 of 21 | +| L-06 | Target architecture | `not_applicable` is for recommendations, never requirements | 2 of 21 | + +The owner approved all six on 2026-10-09 and they were published: five for the target architect, one for the IaC author, which are now the two roles given the lessons Skill. Several restate what instructions or checks already say (L-02 is refused at submission today; L-03 is what the schema review does), which is the kind of thing a reviewer should weigh: a lesson costs tokens on every Execution. + +## Tests + +`tests/test_lessons_loop.py` (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill. diff --git a/.env.example b/.env.example index 31ebc51..8e66a78 100644 --- a/.env.example +++ b/.env.example @@ -41,6 +41,8 @@ ORCHESTRATOR_BUDGET=40 SPECIALIST_BUDGET=15 # With a user present, a Deliverable is shown to them before it is first submitted. CONFIRM_DRAFTS=true +# Give Specialists the lessons Skill. Set false to run without anything learned. +LESSONS_ENABLED=true MAX_REVISIONS=3 MAX_ORCHESTRATOR_TURNS=100 MAX_PARALLEL_SPECIALISTS=4 diff --git a/.scratch/lessons/issues/01-create-specific-checks-for-all-constraints.md b/.scratch/lessons/issues/01-create-specific-checks-for-all-constraints.md new file mode 100644 index 0000000..7611d74 --- /dev/null +++ b/.scratch/lessons/issues/01-create-specific-checks-for-all-constraints.md @@ -0,0 +1,27 @@ +# L-01: Create Specific Checks for All Constraints + +Status: ready-for-agent +Deliverable: target_architecture +Evidence-key: 13ebd6fd65 +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +Create specific Checks for every explicit constraint, metric, or capability mentioned in a requirement, rather than relying on a single broad Check. + +## Why + +If detailed constraints are missed, the validation phase will reject the design for incomplete coverage, forcing a revision. + +## Evidence + +Seen in 14 of 21 Executions, 58 times in all: validation proposed a Check the design lacked. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/.scratch/lessons/issues/02-assign-checks-to-every-element.md b/.scratch/lessons/issues/02-assign-checks-to-every-element.md new file mode 100644 index 0000000..74360f2 --- /dev/null +++ b/.scratch/lessons/issues/02-assign-checks-to-every-element.md @@ -0,0 +1,27 @@ +# L-02: Assign Checks to Every Element + +Status: ready-for-agent +Deliverable: target_architecture +Evidence-key: 93d74c1245 +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +Give every Element at least one Check before you submit. + +## Why + +The submission tool will reject the architecture if any Element lacks a Check, wasting an attempt. + +## Evidence + +Seen in 13 of 21 Executions, 13 times in all: submit_deliverable refused: Not submitted. The Deliverable is missing: target-architecture.json: Element … has no Check; target-architectu. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/.scratch/lessons/issues/03-verify-resource-arguments-for-checks.md b/.scratch/lessons/issues/03-verify-resource-arguments-for-checks.md new file mode 100644 index 0000000..e8236e2 --- /dev/null +++ b/.scratch/lessons/issues/03-verify-resource-arguments-for-checks.md @@ -0,0 +1,27 @@ +# L-03: Verify Resource Arguments for Checks + +Status: ready-for-agent +Deliverable: target_architecture +Evidence-key: f1e7c68393 +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +Verify that the specific resource type used for an Element actually supports the configuration argument you specify in a Check. Remember that access policies and certain configurations often require entirely separate resources. + +## Why + +If a Check specifies an argument that does not exist on the resource, it cannot be implemented, forcing an architectural revision. + +## Evidence + +Seen in 3 of 21 Executions, 3 times in all: revised because a Check named something its resource cannot do. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/.scratch/lessons/issues/04-avoid-using-values-as-setting-names.md b/.scratch/lessons/issues/04-avoid-using-values-as-setting-names.md new file mode 100644 index 0000000..aaf13fd --- /dev/null +++ b/.scratch/lessons/issues/04-avoid-using-values-as-setting-names.md @@ -0,0 +1,27 @@ +# L-04: Avoid Using Values as Setting Names + +Status: ready-for-agent +Deliverable: target_architecture +Evidence-key: 22ff1f5fed +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +When writing a Check for a configuration block that takes a name and value, explicitly state the block's argument name instead of writing the expected value as if it were the setting itself. + +## Why + +Automated reviews may mistake specific expected values for invalid setting names, causing warnings and submission delays. + +## Evidence + +Seen in 2 of 21 Executions, 4 times in all: submit_deliverable refused: Not submitted. A review found: Check …: google_sql_database_instance has no setting named cloudsql.iam_authent. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/.scratch/lessons/issues/05-include-required-arguments-in-alert-policies.md b/.scratch/lessons/issues/05-include-required-arguments-in-alert-policies.md new file mode 100644 index 0000000..937641a --- /dev/null +++ b/.scratch/lessons/issues/05-include-required-arguments-in-alert-policies.md @@ -0,0 +1,27 @@ +# L-05: Include Required Arguments in Alert Policies + +Status: ready-for-agent +Deliverable: deployment_guidance +Evidence-key: 2d4ff48141 +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +Always verify and include all required arguments (such as time window definitions) when configuring nested blocks in monitoring alert policies. + +## Why + +Missing required arguments in nested blocks will cause Terraform validation to fail, preventing submission. + +## Evidence + +Seen in 2 of 21 Executions, 2 times in all: terraform_validate refused: terraform validate failed: Error: Missing required argument on main.tf line …, in resource … …: …: condition_m. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/.scratch/lessons/issues/06-do-not-mark-requirements-as-not-applicable.md b/.scratch/lessons/issues/06-do-not-mark-requirements-as-not-applicable.md new file mode 100644 index 0000000..032d459 --- /dev/null +++ b/.scratch/lessons/issues/06-do-not-mark-requirements-as-not-applicable.md @@ -0,0 +1,27 @@ +# L-06: Do Not Mark Requirements as Not Applicable + +Status: ready-for-agent +Deliverable: target_architecture +Evidence-key: cd5ae98124 +Proposed: 2026-10-09 +Published: 2026-10-09 + +## Rule + +Do not list Requirements in the not applicable field, as this field is strictly for Design Recommendations. Ensure every Requirement has at least one Check. + +## Why + +The submission tool will reject the deliverable if a Requirement lacks a Check or is incorrectly marked as not applicable. + +## Evidence + +Seen in 2 of 21 Executions, 2 times in all: submit_deliverable refused: Not submitted. The Deliverable is missing: Requirement … has no Check; Requirement … has no Check; Requirement. + +## Proof + +Not yet measured. + +## Comments + +Approved by the owner on 2026-10-09. diff --git a/README.md b/README.md index c988804..7263418 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ Execution records (status, Ledger events, submitted Deliverables) go to PostgreS Behaviour lives in Agent Skills (agentskills.io format) at the repository root, per ADR-0002: - `skills/vendor/google/`: the upstream solution-architecture Skill and six Well-Architected Framework pillar Skills, vendored unmodified. - `skills/cloud-discovery/`, `skills/iac-validation/` and `skills/requirements-chain/`: our own Skills for the tooling and the data files. +- `skills/google-cloud-lessons/`: lessons learned from earlier Executions, written by the lessons loop (`app/learning/`, `eval/curate_lessons.py`) from proposals a person approved. Empty until one is. - `skills/google-cloud-iac-patterns/`: Terraform patterns for Google Cloud products, each validated by a test against the provider version `providers/gcp.yaml` pins. `app/skills/agent_skills.py` discovers and validates them; the ADK runtime (`app/adk/adk_runtime.py`) loads them on demand. `providers/gcp.yaml` assigns Skills, tools, MCP servers and models to each role in `agents/`. diff --git a/agents/lesson-curator.md b/agents/lesson-curator.md new file mode 100644 index 0000000..40572f4 --- /dev/null +++ b/agents/lesson-curator.md @@ -0,0 +1,27 @@ +--- +role: lesson-curator +tools: [] +budget: 60 +--- +You review what went wrong, the same way, across many solution-architecture Executions, and propose lessons: rules that would have prevented it. You run outside any Execution. A person approves each lesson before any agent sees it, so propose only what you would defend to them. + +Your task message carries JSON context: + +- `executions`: how many Executions the evidence comes from. +- `groups`: what recurred. Each has a `group` key, the `deliverable` it concerns, `what_happened` in general terms, how many `executions` it occurred in and how many `occurrences` in all, and up to three `examples` as they were recorded. +- `published_or_proposed`: lessons that already exist or are awaiting review. + +For each group, decide whether a rule given to the Specialist that produces that Deliverable would have prevented it. + +- Call `propose_lesson` when it would. Name the Deliverable whose Specialist must follow the rule, which is the one that could have prevented the problem, not always the one where it surfaced. A Check that validation had to propose is prevented by the target architect. +- Call `no_lesson` when it would not: the cause is the tooling or the engine and no instruction would change it, it is too rare or too particular to one workload, or an existing lesson already covers it. + +A good lesson: + +- States one rule, as an instruction, that a Specialist can act on without knowing what happened: "Give every Element at least one Check before you submit", not "be more careful with Checks". +- Says why in one or two sentences: what goes wrong without it and what that costs. +- Is general. It names no customer, no project, no Execution, no resource name and no value from the examples. The examples are evidence for you; nothing from them may be quoted. +- Does not repeat what the Specialist's instructions or tools already enforce. When a submission is already refused for something, a lesson helps only if it prevents the attempt. +- Is worth its cost. Every lesson is read on every Execution from then on. Propose few. + +Call `finish` when every group has a proposal or a reason for none. diff --git a/agents/partials/specialist-protocol.md b/agents/partials/specialist-protocol.md index 28c164c..f34cdf4 100644 --- a/agents/partials/specialist-protocol.md +++ b/agents/partials/specialist-protocol.md @@ -16,7 +16,7 @@ Your task message carries JSON context: Steps: -1. Load the Skill available to you and find the part that covers your Deliverable. Read the references and assets it points to with `load_skill_resource`. +1. Load the Skill available to you and find the part that covers your Deliverable. Read the references and assets it points to with `load_skill_resource`. When a lessons Skill is among your Skills, load it too and read its reference for your Deliverable: those are rules learned from earlier work, and you follow each one that applies. 2. Ground your work in `dependencies`, `feedback` and `answers`. When you need information only the user has, call `needs_input` with all your questions at once, each saying why it matters. Your run ends there, and you are started again with the answers. 3. Draft the Deliverable to its `contract`. When `pre_approved` is `false`, confirm the draft with the user before submitting: call `needs_input` with a short summary of the draft, asking them to approve it or request changes, and pass the complete draft as `draft` with its `attachments`. When `draft` is set and the latest entry in `answers` approves it, submit that draft as it is; when the user asked for changes, apply them to that draft. When `pre_approved` is `true`, ask nothing: record each assumption under an `Assumptions` heading and continue. 4. Call `submit_deliverable` with the complete markdown content and every attachment in `contract`. When it reports what is missing, add it and submit again. diff --git a/app/adk/roles.py b/app/adk/roles.py index fbf8cb9..f2c4bfa 100644 --- a/app/adk/roles.py +++ b/app/adk/roles.py @@ -7,6 +7,7 @@ include. The provider bundle supplies each role's Skills, MCP servers and model. from __future__ import annotations +import dataclasses from pathlib import Path from app.adk.runtime import RoleSpec @@ -17,11 +18,19 @@ from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind ORCHESTRATOR = "orchestrator" PILLAR_ADVISOR = "pillar-advisor" REPLY_WRITER = "reply-writer" +LESSON_CURATOR = "lesson-curator" # runs outside any Execution PARTIALS_DIR = "partials" _ALLOWED_FIELDS = {"role", "deliverable", "tools", "budget", "includes"} +def _deliverables_with_lessons(bundle: ProviderBundle) -> set[str]: + if not bundle.lessons_skill: + return set() + folder = bundle.skills[bundle.lessons_skill].path / "references" + return {path.stem for path in folder.glob("*.md")} if folder.is_dir() else set() + + class RoleConfigError(ValueError): """A role file is missing or invalid.""" @@ -33,9 +42,14 @@ def load_roles( orchestrator_budget: int = 40, specialist_budget: int = 15, include_preview_mcp: bool = True, + lessons: bool = True, ) -> dict[str, RoleSpec]: + """`lessons` off leaves the lessons Skill out of every role, so an evaluation + can run with and without what has been learned.""" specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()} required = {ORCHESTRATOR, PILLAR_ADVISOR, REPLY_WRITER, *specialists} + optional = {LESSON_CURATOR} + learned = _deliverables_with_lessons(bundle) if lessons else set() roles = {} for path in sorted(agents_dir.glob("*.md")): @@ -54,7 +68,13 @@ def load_roles( raise RoleConfigError( f"{agents_dir}: missing role files for {', '.join(missing)}" ) - unexpected = sorted(set(roles) - required) + for name, kind in specialists.items(): + if kind.value in learned and bundle.lessons_skill: + # Only a Specialist whose Deliverable has lessons is given the Skill. + roles[name] = dataclasses.replace( + roles[name], skills=(*roles[name].skills, bundle.lessons_skill) + ) + unexpected = sorted(set(roles) - required - optional) if unexpected: raise RoleConfigError( f"{agents_dir}: unexpected role files {', '.join(unexpected)}" diff --git a/app/agent.py b/app/agent.py index c0993f1..c35b5cb 100644 --- a/app/agent.py +++ b/app/agent.py @@ -293,6 +293,7 @@ async def open_assembly( orchestrator_budget=settings.ORCHESTRATOR_BUDGET, specialist_budget=settings.SPECIALIST_BUDGET, include_preview_mcp=settings.MCP_ENABLE_PREVIEW, + lessons=settings.LESSONS_ENABLED, ) if runtime is None: from app.adk.adk_runtime import AdkAgentRuntime diff --git a/app/config.py b/app/config.py index d031e57..89b45c8 100644 --- a/app/config.py +++ b/app/config.py @@ -54,6 +54,9 @@ class Settings(BaseSettings): A2A_PUBLIC_URL: str | None = None # advertised in the agent card ORCHESTRATOR_BUDGET: int = 40 SPECIALIST_BUDGET: int = 15 + # Give Specialists the lessons Skill. Off runs without anything learned, which + # is how the evaluation shows whether a lesson helps. + LESSONS_ENABLED: bool = True # With a user present, a Specialist's first submission of a Deliverable is refused # until it has put its draft to them. Off: Specialists are only told to. CONFIRM_DRAFTS: bool = True diff --git a/app/learning/__init__.py b/app/learning/__init__.py new file mode 100644 index 0000000..daccdee --- /dev/null +++ b/app/learning/__init__.py @@ -0,0 +1,17 @@ +"""The lessons loop (ADR-0004): what Executions got wrong becomes reviewed Skill changes. + + capture each Execution records lesson candidates app/states/lessons.py + distil recurring candidates are grouped, and a curator + proposes a lesson for each group worth one evidence.py, curator.py + review each proposal is an issue file a person decides proposals.py + publish an approved lesson is written into the lessons + Skill the provider bundle assigns to roles skill.py + prove the evaluation runs with and without lessons + retire a lesson that did not help, or that the upstream + Skills now cover, is removed skill.py + +Nothing here runs inside an Execution, and no lesson reaches an agent without a +person's approval. Lesson candidates are client data: they are read from the +Executions' records and never copied into the repository; a lesson and its issue +file carry counts and a general description only. +""" diff --git a/app/learning/curator.py b/app/learning/curator.py new file mode 100644 index 0000000..60fc2fe --- /dev/null +++ b/app/learning/curator.py @@ -0,0 +1,164 @@ +"""The lesson curator: proposes a lesson for each recurring problem worth one. + +It runs outside any Execution, on the evidence of many. It only proposes: every +proposal is written as an issue file for a person to decide (ADR-0004). +""" + +from __future__ import annotations + +import logging +from collections.abc import Sequence +from pathlib import Path +from typing import Any + +from app.adk.runtime import ( + AgentRuntime, + AgentTask, + RoleSpec, + ToolResult, + ToolSpec, + TurnEnded, +) +from app.learning.evidence import Group +from app.learning.proposals import ISSUES, Proposal, load, propose +from app.workflows.deliverables import DeliverableKind + +LESSON_CURATOR = "lesson-curator" +MAX_GROUPS = 25 + +logger = logging.getLogger(__name__) + + +def _evidence(group: Group, executions: int) -> str: + return ( + f"Seen in {len(group.executions)} of {executions} Executions, " + f"{group.occurrences} times in all: {group.description}." + ) + + +async def curate( + runtime: AgentRuntime, + role: RoleSpec, + groups: Sequence[Group], + *, + executions: int, + identifiers: set[str] = frozenset(), # type: ignore[assignment] + folder: Path = ISSUES, +) -> list[Proposal]: + """Run the curator over the groups that have no proposal yet; return what it + proposed. Groups it judged not worth a lesson are left for the next run.""" + existing = load(folder) + decided = {proposal.evidence_key for proposal in existing} + fresh = [group for group in groups if group.key not in decided][:MAX_GROUPS] + if not fresh: + return [] + by_key = {group.key: group for group in fresh} + kinds = [kind.value for kind in DeliverableKind] + proposed: list[Proposal] = [] + declined: dict[str, str] = {} + + async def propose_lesson( + group: str, deliverable: str, title: str, rule: str, reason: str + ) -> ToolResult: + if group not in by_key: + return ToolResult(f"Refused: {group} is not one of the groups given.") + if deliverable not in kinds: + return ToolResult( + f"Refused: deliverable must be one of {', '.join(kinds)}." + ) + if any(p.evidence_key == group for p in proposed): + return ToolResult(f"Refused: {group} already has a proposal.") + text = f"{title} {rule} {reason}" + leaked = sorted(name for name in identifiers if name in text) + if leaked: + return ToolResult( + "Refused: a lesson carries no client or Execution detail; remove " + + ", ".join(leaked) + ) + if not (title.strip() and rule.strip() and reason.strip()): + return ToolResult("Refused: a lesson needs a title, a rule and a reason.") + proposal = propose( + title=title, + deliverable=deliverable, + rule=rule, + reason=reason, + evidence=_evidence(by_key[group], executions), + evidence_key=group, + folder=folder, + ) + proposed.append(proposal) + return ToolResult(f"Proposed {proposal.id} for a person to review.") + + async def no_lesson(group: str, why: str) -> ToolResult: + if group not in by_key: + return ToolResult(f"Refused: {group} is not one of the groups given.") + declined[group] = why + return ToolResult(f"Noted: no lesson for {group}.") + + async def finish() -> ToolResult: + return ToolResult("Done.", TurnEnded("finish")) + + text = {"type": "string"} + tools = [ + ToolSpec( + "propose_lesson", + "Propose one lesson for a group. A person reviews it before any agent " + "sees it.", + { + "type": "object", + "properties": { + "group": text, + "deliverable": {"type": "string", "enum": kinds}, + "title": text, + "rule": text, + "reason": text, + }, + "required": ["group", "deliverable", "title", "rule", "reason"], + }, + propose_lesson, + tier="execution", + ), + ToolSpec( + "no_lesson", + "Say that a group is not worth a lesson, and why.", + { + "type": "object", + "properties": {"group": text, "why": text}, + "required": ["group", "why"], + }, + no_lesson, + tier="execution", + ), + ToolSpec( + "finish", + "End your turn once every group has a proposal or a reason for none.", + {"type": "object", "properties": {}}, + finish, + tier="execution", + ), + ] + context: dict[str, Any] = { + "executions": executions, + "groups": [group.as_context() for group in fresh], + "published_or_proposed": [ + { + "id": p.id, + "deliverable": p.deliverable, + "rule": p.rule, + "status": p.status, + } + for p in existing + ], + } + await runtime.run( + role, + AgentTask( + role=role.name, + prompt="Propose lessons from what recurred across these Executions.", + context=context, + ), + tools, + ) + for group, why in declined.items(): + logger.info("no lesson for %s: %s", group, why) + return proposed diff --git a/app/learning/evidence.py b/app/learning/evidence.py new file mode 100644 index 0000000..8d832aa --- /dev/null +++ b/app/learning/evidence.py @@ -0,0 +1,155 @@ +"""Lesson candidates across Executions, grouped into what recurs.""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +EXAMPLES = 3 +EXAMPLE_LIMIT = 400 +# Ids and numbers vary between Executions; what recurs is the shape around them. +_VARIABLE = re.compile(r"`[^`]*`|\"[^\"]*\"|\b[A-Z]{1,4}-[\w.-]+\b|\b\d[\w.]*\b") +# A refusal the Execution's own permissions caused is policy, not a mistake. +_POLICY = ("Refused: reading live cloud resources", "Permission to read live cloud") + + +@dataclass +class Group: + """One thing that went wrong the same way in several Executions.""" + + key: str # stable across runs of the curator: the same evidence, the same key + kind: str # refusal | revision | proposed_addition | failed_check | ... + deliverable: str | None + description: str # general: no id, name or value from any Execution + executions: set[str] = field(default_factory=set) + occurrences: int = 0 + examples: list[str] = field(default_factory=list) # client data: never published + + def as_context(self) -> dict[str, Any]: + return { + "group": self.key, + "kind": self.kind, + "deliverable": self.deliverable, + "what_happened": self.description, + "executions": len(self.executions), + "occurrences": self.occurrences, + "examples": self.examples, + } + + +def read_candidates(roots: Iterable[Path]) -> list[dict[str, Any]]: + """Every Execution's lesson-candidate record under the given folders.""" + records = [] + seen: set[Path] = set() + for root in roots: + for path in sorted(Path(root).rglob("lessons/candidates/*/*.json")): + # One folder may lie inside another that was also given. + if path.resolve() in seen: + continue + seen.add(path.resolve()) + try: + record = json.loads(path.read_text(encoding="utf-8")) + except ValueError: + continue + if isinstance(record, dict) and isinstance(record.get("candidates"), list): + records.append(record) + return records + + +def _general(text: str, limit: int = 110) -> str: + return " ".join(_VARIABLE.sub("…", text).split())[:limit] + + +def _shape(candidate: Mapping[str, Any]) -> tuple[str, str | None, str] | None: + """(kind, Deliverable, general description) of a candidate, or None to ignore it.""" + kind = str(candidate.get("type")) + deliverable = candidate.get("deliverable") + if kind == "refusal": + message = str(candidate.get("message", "")) + if message.strip() == "Refused" or message.startswith(_POLICY): + return None + tool = candidate.get("tool") + return kind, deliverable, f"{tool} refused: {_general(message)}" + if kind in ("revision", "revision_limit"): + reason = str(candidate.get("reason", "")) + if reason.startswith("Q: ") or "revision limit" in reason: + return None # the router's own questions, recorded as reasons + if reason.startswith("Add a Check"): + why = "validation proposed a Check the design lacked" + elif "cannot be implemented" in reason: + why = "a Check named something its resource cannot do" + elif " failed" in reason[:60]: + why = "the code failed a Check" + else: + why = _general(reason, 70) + return "revision", deliverable, f"revised because {why}" + if kind == "proposed_addition": + return ( + kind, + "target_architecture", + "validation proposed a Check the design lacked", + ) + if kind in ("failed_check", "unverifiable_check"): + what = "failed" if kind == "failed_check" else "could not be verified" + return kind, "deployment_guidance", f"a Check {what} at validation" + if kind == "unfinished": + return kind, deliverable, f"the Specialist {candidate.get('reason')}" + if kind == "change_request": + return kind, None, "the user asked for changes at an Approval" + return None + + +def group_candidates( + records: Sequence[Mapping[str, Any]], *, min_executions: int = 2 +) -> list[Group]: + """What recurred in at least `min_executions` Executions, most widespread first.""" + groups: dict[str, Group] = {} + for record in records: + execution = str(record.get("execution_id")) + for candidate in record.get("candidates", ()): + if not isinstance(candidate, Mapping): + continue + shape = _shape(candidate) + if shape is None: + continue + kind, deliverable, description = shape + key = hashlib.sha256( + f"{kind}|{deliverable}|{description}".encode() + ).hexdigest()[:10] + group = groups.setdefault(key, Group(key, kind, deliverable, description)) + group.executions.add(execution) + group.occurrences += 1 + example = str( + candidate.get("message") + or candidate.get("reason") + or candidate.get("statement") + or candidate.get("evidence") + or candidate.get("feedback") + or "" + )[:EXAMPLE_LIMIT] + if ( + example + and example not in group.examples + and len(group.examples) < EXAMPLES + ): + group.examples.append(example) + return sorted( + (g for g in groups.values() if len(g.executions) >= min_executions), + key=lambda g: (-len(g.executions), -g.occurrences, g.key), + ) + + +def identifiers(records: Sequence[Mapping[str, Any]]) -> set[str]: + """Names that identify a client or an Execution, which no lesson may carry.""" + found = set() + for record in records: + for name in ("execution_id", "customer_case"): + value = record.get(name) + if isinstance(value, str) and len(value) >= 4 and value != "unassigned": + found.add(value) + return found diff --git a/app/learning/proposals.py b/app/learning/proposals.py new file mode 100644 index 0000000..cf17385 --- /dev/null +++ b/app/learning/proposals.py @@ -0,0 +1,167 @@ +"""Lesson proposals as issue files: where a person reviews each one. + + .scratch/lessons/issues/NN-.md + +A proposal starts as `needs-triage`. A person approves it by setting its Status to +`ready-for-agent`, or rejects it with `wontfix`. Only an approved proposal can be +published. The file carries the rule, its reason and how widespread the evidence +was, and nothing that identifies a client or an Execution. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, replace +from datetime import UTC, date, datetime +from pathlib import Path + +NEEDS_TRIAGE = "needs-triage" +APPROVED = "ready-for-agent" +REJECTED = "wontfix" +STATUSES = (NEEDS_TRIAGE, "needs-info", APPROVED, "ready-for-human", REJECTED) +ISSUES = Path(".scratch") / "lessons" / "issues" + +_FIELD = re.compile(r"^([A-Z][\w-]*):\s*(.*)$", re.MULTILINE) +_SECTION = re.compile(r"^## (.+?)\s*$", re.MULTILINE) + + +@dataclass(frozen=True) +class Proposal: + number: int + title: str + deliverable: str + rule: str + reason: str + evidence: str # counts and a general description + evidence_key: str + status: str = NEEDS_TRIAGE + proposed: str = "" + published: str = "" + proof: str = "" + comments: str = "" + path: Path | None = None + + @property + def id(self) -> str: + return f"L-{self.number:02d}" + + @property + def approved(self) -> bool: + return self.status == APPROVED + + +def _slug(title: str) -> str: + return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")[:60] or "lesson" + + +def render(proposal: Proposal) -> str: + lines = [ + f"# {proposal.id}: {proposal.title}", + "", + f"Status: {proposal.status}", + f"Deliverable: {proposal.deliverable}", + f"Evidence-key: {proposal.evidence_key}", + f"Proposed: {proposal.proposed}", + ] + if proposal.published: + lines.append(f"Published: {proposal.published}") + lines += [ + "", + "## Rule", + "", + proposal.rule.strip(), + "", + "## Why", + "", + proposal.reason.strip(), + "", + "## Evidence", + "", + proposal.evidence.strip(), + "", + "## Proof", + "", + proposal.proof.strip() or "Not yet measured.", + "", + "## Comments", + "", + proposal.comments.strip(), + ] + return "\n".join(lines).rstrip() + "\n" + + +def parse(path: Path) -> Proposal: + text = path.read_text(encoding="utf-8") + heading = re.match(r"# L-(\d+): (.+)", text) + if not heading: + raise ValueError(f"{path}: not a lesson proposal") + head = text.split("\n## ", 1)[0] + fields = {name: value.strip() for name, value in _FIELD.findall(head)} + sections: dict[str, str] = {} + matches = list(_SECTION.finditer(text)) + for index, match in enumerate(matches): + end = matches[index + 1].start() if index + 1 < len(matches) else len(text) + sections[match.group(1)] = text[match.end() : end].strip() + status = fields.get("Status", NEEDS_TRIAGE) + if status not in STATUSES: + raise ValueError(f"{path}: Status must be one of {', '.join(STATUSES)}") + proof = sections.get("Proof", "") + return Proposal( + number=int(heading.group(1)), + title=heading.group(2).strip(), + deliverable=fields.get("Deliverable", ""), + rule=sections.get("Rule", ""), + reason=sections.get("Why", ""), + evidence=sections.get("Evidence", ""), + evidence_key=fields.get("Evidence-key", ""), + status=status, + proposed=fields.get("Proposed", ""), + published=fields.get("Published", ""), + proof="" if proof == "Not yet measured." else proof, + comments=sections.get("Comments", ""), + path=path, + ) + + +def load(folder: Path = ISSUES) -> list[Proposal]: + if not folder.is_dir(): + return [] + return sorted( + (parse(path) for path in folder.glob("[0-9]*-*.md")), key=lambda p: p.number + ) + + +def write(proposal: Proposal, folder: Path = ISSUES) -> Proposal: + folder.mkdir(parents=True, exist_ok=True) + path = proposal.path or folder / f"{proposal.number:02d}-{_slug(proposal.title)}.md" + path.write_text(render(proposal), encoding="utf-8") + return replace(proposal, path=path) + + +def propose( + *, + title: str, + deliverable: str, + rule: str, + reason: str, + evidence: str, + evidence_key: str, + folder: Path = ISSUES, + today: date | None = None, +) -> Proposal: + """A new proposal, numbered after the last, awaiting a person's decision.""" + existing = load(folder) + number = max((p.number for p in existing), default=0) + 1 + return write( + Proposal( + number=number, + title=title.strip(), + deliverable=deliverable, + rule=rule, + reason=reason, + evidence=evidence, + evidence_key=evidence_key, + proposed=(today or datetime.now(UTC).date()).isoformat(), + ), + folder, + ) diff --git a/app/learning/skill.py b/app/learning/skill.py new file mode 100644 index 0000000..5543c53 --- /dev/null +++ b/app/learning/skill.py @@ -0,0 +1,142 @@ +"""The lessons Skill: where approved lessons are published, and retired from. + +A lessons Skill is an ordinary Agent Skill, per Cloud provider, that the provider +bundle names. It has one reference file per Deliverable that has lessons, so a +Specialist reads only the lessons for its own work. Both the reference files and +the index in SKILL.md are written here, from approved proposals, and never by hand. +""" + +from __future__ import annotations + +import re +from dataclasses import replace +from datetime import UTC, date, datetime +from pathlib import Path + +from app.learning.proposals import Proposal, write + +REFERENCES = "references" +_INDEX = re.compile(r".*", re.DOTALL) +_ENTRY = re.compile(r"^## (L-\d+): .*?(?=^## L-\d+: |\Z)", re.MULTILINE | re.DOTALL) + + +class NotApproved(ValueError): + """A lesson cannot be published: no person has approved its proposal.""" + + +def reference(skill_dir: Path, deliverable: str) -> Path: + return skill_dir / REFERENCES / f"{deliverable}.md" + + +def deliverables_with_lessons(skill_dir: Path) -> list[str]: + folder = skill_dir / REFERENCES + if not folder.is_dir(): + return [] + return sorted( + path.stem + for path in folder.glob("*.md") + if _ENTRY.search(path.read_text(encoding="utf-8")) + ) + + +def published(skill_dir: Path) -> dict[str, str]: + """Lesson id -> the Deliverable whose reference holds it.""" + found = {} + for deliverable in deliverables_with_lessons(skill_dir): + text = reference(skill_dir, deliverable).read_text(encoding="utf-8") + for match in _ENTRY.finditer(text): + found[match.group(1)] = deliverable + return found + + +def _entry(proposal: Proposal, today: str) -> str: + return ( + f"## {proposal.id}: {proposal.title}\n\n" + f"**Rule.** {' '.join(proposal.rule.split())}\n\n" + f"**Why.** {' '.join(proposal.reason.split())}\n\n" + f"Published {today}, from lesson proposal {proposal.number:02d}.\n" + ) + + +def _write_reference(skill_dir: Path, deliverable: str, entries: list[str]) -> None: + path = reference(skill_dir, deliverable) + if not entries: + path.unlink(missing_ok=True) + return + path.parent.mkdir(parents=True, exist_ok=True) + title = deliverable.replace("_", " ") + path.write_text( + f"# Lessons: {title}\n\n" + "Rules learned from earlier work. Follow every one that applies.\n\n" + + "\n".join(entries), + encoding="utf-8", + ) + + +def _entries(skill_dir: Path, deliverable: str) -> dict[str, str]: + path = reference(skill_dir, deliverable) + if not path.is_file(): + return {} + text = path.read_text(encoding="utf-8") + return {m.group(1): m.group(0).rstrip() + "\n" for m in _ENTRY.finditer(text)} + + +def _reindex(skill_dir: Path) -> None: + """Rewrite the list of references in SKILL.md from the files that exist.""" + skill = skill_dir / "SKILL.md" + rows = [ + f"| {name.replace('_', ' ')} | `{REFERENCES}/{name}.md` | " + f"{len(_entries(skill_dir, name))} |" + for name in deliverables_with_lessons(skill_dir) + ] + table = ( + "| Deliverable | Reference | Lessons |\n| --- | --- | --- |\n" + "\n".join(rows) + if rows + else "No lessons have been published yet." + ) + text = skill.read_text(encoding="utf-8") + skill.write_text( + _INDEX.sub(f"\n{table}\n", text), + encoding="utf-8", + ) + + +def publish( + proposal: Proposal, skill_dir: Path, *, today: date | None = None +) -> Proposal: + """Write an approved lesson into the Skill and mark its proposal published. + + Raises NotApproved for any proposal a person has not approved: nothing reaches + an agent on a curator's say-so. + """ + if not proposal.approved: + raise NotApproved( + f"{proposal.id} is {proposal.status}; a person approves a lesson by " + "setting its Status to ready-for-agent" + ) + if not proposal.deliverable or not proposal.rule.strip(): + raise NotApproved(f"{proposal.id} has no Deliverable or no rule") + stamp = (today or datetime.now(UTC).date()).isoformat() + for deliverable in deliverables_with_lessons(skill_dir): + if deliverable != proposal.deliverable: + entries = _entries(skill_dir, deliverable) + if entries.pop(proposal.id, None): # it moved to another Deliverable + _write_reference(skill_dir, deliverable, list(entries.values())) + entries = _entries(skill_dir, proposal.deliverable) + entries[proposal.id] = _entry(proposal, stamp) + ordered = [entries[key] for key in sorted(entries)] + _write_reference(skill_dir, proposal.deliverable, ordered) + _reindex(skill_dir) + return write(replace(proposal, published=proposal.published or stamp)) + + +def retire(lesson_id: str, skill_dir: Path) -> bool: + """Remove a lesson from the Skill. Returns whether it was there.""" + deliverable = published(skill_dir).get(lesson_id) + if deliverable is None: + return False + entries = _entries(skill_dir, deliverable) + entries.pop(lesson_id) + _write_reference(skill_dir, deliverable, [entries[key] for key in sorted(entries)]) + _reindex(skill_dir) + return True diff --git a/app/providers.py b/app/providers.py index 46c675f..1208b2b 100644 --- a/app/providers.py +++ b/app/providers.py @@ -69,6 +69,9 @@ class ProviderBundle: # The Terraform providers generated code is written for, pinned so the schema # lookup, the patterns and validation agree: {name: {source, version}}. terraform_providers: Mapping[str, Mapping[str, str]] = field(default_factory=dict) + # The Skill approved lessons are published in (ADR-0004); None when the provider + # has none. A Specialist is given it once it holds lessons for its Deliverable. + lessons_skill: str | None = None def mcp_servers_for_role( self, role: str, include_preview: bool = True @@ -194,6 +197,10 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle: raise fail(f"pillar {pillar!r} references unknown skill {skill_name!r}") pillar_skills[str(pillar)] = skill_name + lessons_skill = config.get("lessons_skill") or None + if lessons_skill is not None and lessons_skill not in skills: + raise fail(f"'lessons_skill' references unknown skill {lessons_skill!r}") + return ProviderBundle( name=name, skills_dirs=skills_dirs, @@ -207,4 +214,5 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle: role_skills=role_skills, pillar_skills=pillar_skills, terraform_providers=terraform_providers, + lessons_skill=lessons_skill, ) diff --git a/eval/curate_lessons.py b/eval/curate_lessons.py new file mode 100644 index 0000000..d2d7215 --- /dev/null +++ b/eval/curate_lessons.py @@ -0,0 +1,155 @@ +"""The lessons loop, from the command line (ADR-0004). + + propose group the lesson candidates of past Executions and have the curator + propose a lesson for each recurring problem worth one. Proposals are + written to .scratch/lessons/issues/ as `needs-triage`. + status list every proposal and where it stands. + publish write every approved proposal (Status: ready-for-agent) into the + provider's lessons Skill. + retire remove a lesson from the Skill, by id. + +Usage: + .venv/bin/python -m eval.curate_lessons propose [--from ...] [--min 2] + .venv/bin/python -m eval.curate_lessons status + .venv/bin/python -m eval.curate_lessons publish + .venv/bin/python -m eval.curate_lessons retire L-03 + +A person approves a proposal by editing its file: set `Status: ready-for-agent`, or +`wontfix` to reject it. Nothing is published without that. After publishing, prove +the lesson: run the evaluation with LESSONS_ENABLED=false and again with it on, and +record the comparison under the proposal's `## Proof`. Retire a lesson that does +not hold quality and lower Revisions, time or tokens. + +`propose` costs model usage. Lesson candidates are client data; they are read +where the Executions wrote them and never copied. +""" + +from __future__ import annotations + +import asyncio +import sys +from collections.abc import Sequence +from pathlib import Path + +from app.config import get_settings +from app.learning.curator import LESSON_CURATOR, curate +from app.learning.evidence import group_candidates, identifiers, read_candidates +from app.learning.proposals import ISSUES, load +from app.learning.skill import NotApproved, publish, published, retire +from app.providers import ProviderBundle, load_provider + +ROOT = Path(__file__).resolve().parent.parent +# Where Executions write their candidates: under each evaluation run, and under the +# server's output root. +DEFAULT_SOURCES = (ROOT / "eval" / "results", ROOT / "lessons" / "candidates" / "..") + + +def _skill_dir(bundle: ProviderBundle) -> Path: + if not bundle.lessons_skill: + raise SystemExit(f"The {bundle.name} provider bundle names no lessons_skill.") + return bundle.skills[bundle.lessons_skill].path + + +def _options(arguments: Sequence[str], name: str) -> list[str]: + return [ + arguments[index + 1] + for index, argument in enumerate(arguments[:-1]) + if argument == name + ] + + +async def propose_lessons(arguments: Sequence[str]) -> int: + from app.adk.adk_runtime import AdkAgentRuntime + from app.adk.roles import load_roles + + settings = get_settings() + bundle = load_provider(settings.CLOUD_PROVIDER) + sources = [Path(p) for p in _options(arguments, "--from")] or list(DEFAULT_SOURCES) + minimum = int((_options(arguments, "--min") or ["2"])[0]) + records = read_candidates(sources) + groups = group_candidates(records, min_executions=minimum) + print( + f"{len(records)} Executions, {len(groups)} recurring problems " + f"(in at least {minimum} Executions each)." + ) + if not groups: + return 0 + roles = load_roles(ROOT / "agents", bundle) + runtime = AdkAgentRuntime( + skill_dirs={name: meta.path for name, meta in bundle.skills.items()}, + model_timeout_seconds=settings.MODEL_TIMEOUT_SECONDS, + ) + proposed = await curate( + runtime, + roles[LESSON_CURATOR], + groups, + executions=len(records), + identifiers=identifiers(records), + folder=ROOT / ISSUES, + ) + for proposal in proposed: + print(f" {proposal.id} [{proposal.deliverable}] {proposal.title}") + print( + f"{len(proposed)} proposed in {ROOT / ISSUES}. Review each one and set its " + "Status to ready-for-agent or wontfix." + ) + return 0 + + +def show_status() -> int: + bundle = load_provider(get_settings().CLOUD_PROVIDER) + live = published(_skill_dir(bundle)) + proposals = load(ROOT / ISSUES) + if not proposals: + print("No lesson has been proposed.") + for proposal in proposals: + where = "in the Skill" if proposal.id in live else "not in the Skill" + proof = "proved" if proposal.proof else "not proved" + print( + f"{proposal.id} {proposal.status:16s} {where:16s} {proof:11s} " + f"[{proposal.deliverable}] {proposal.title}" + ) + return 0 + + +def publish_approved() -> int: + bundle = load_provider(get_settings().CLOUD_PROVIDER) + skill_dir = _skill_dir(bundle) + done = 0 + for proposal in load(ROOT / ISSUES): + try: + publish(proposal, skill_dir) + except NotApproved: + continue + done += 1 + print(f" published {proposal.id} [{proposal.deliverable}] {proposal.title}") + print(f"{done} lessons are in {skill_dir.name}.") + return 0 + + +def retire_lesson(arguments: Sequence[str]) -> int: + if not arguments: + print("Name the lesson to retire, such as L-03.", file=sys.stderr) + return 2 + bundle = load_provider(get_settings().CLOUD_PROVIDER) + gone = retire(arguments[0], _skill_dir(bundle)) + print(f"Retired {arguments[0]}." if gone else f"{arguments[0]} is not published.") + return 0 if gone else 1 + + +def main(arguments: Sequence[str]) -> int: + command, rest = (arguments[0], arguments[1:]) if arguments else ("status", []) + if command == "propose": + return asyncio.run(propose_lessons(rest)) + if command == "status": + return show_status() + if command == "publish": + return publish_approved() + if command == "retire": + return retire_lesson(rest) + print(__doc__, file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/providers/gcp.yaml b/providers/gcp.yaml index cb7face..91dbbbc 100644 --- a/providers/gcp.yaml +++ b/providers/gcp.yaml @@ -21,6 +21,7 @@ models: target-architect: gemini/gemini-3.1-pro-preview iac-author: gemini/gemini-3.1-pro-preview validator: gemini/gemini-3.1-pro-preview + lesson-curator: gemini/gemini-3.1-pro-preview # Seconds one model request may take, for roles whose replies are long. Other # roles get MODEL_TIMEOUT_SECONDS. The target architect timed out at 180 s in both # attempts of the 2026-10-07 run. @@ -43,6 +44,10 @@ role_skills: validator: [google-cloud-solution-architecture, requirements-chain, iac-validation] packager: [google-cloud-solution-architecture] +# Where approved lessons are published (ADR-0004). A Specialist is given this Skill +# once it holds lessons for that Specialist's Deliverable. +lessons_skill: google-cloud-lessons + # One advisor per architecture-framework pillar; each loads only its pillar's Skill. pillar_skills: security: google-cloud-waf-security diff --git a/skills/google-cloud-lessons/SKILL.md b/skills/google-cloud-lessons/SKILL.md new file mode 100644 index 0000000..7338d1f --- /dev/null +++ b/skills/google-cloud-lessons/SKILL.md @@ -0,0 +1,24 @@ +--- +name: google-cloud-lessons +description: >- + Rules learned from earlier solution-architecture work on Google Cloud, one set + per Deliverable. Each was proposed from problems that recurred across + Executions, approved by a person, and kept only while evaluation shows it + helps. Use before drafting a Deliverable, to read the lessons for that + Deliverable and follow them. +metadata: + scope: every Deliverable +--- + +# Lessons + +Each reference below holds the lessons for one Deliverable. Read the one for the Deliverable you are producing, with `load_skill_resource`, before you draft, and follow every rule in it that applies. A rule here takes precedence over a general habit; your role's instructions and the contract `submit_deliverable` checks take precedence over a rule here. + + +| Deliverable | Reference | Lessons | +| --- | --- | --- | +| deployment guidance | `references/deployment_guidance.md` | 1 | +| target architecture | `references/target_architecture.md` | 5 | + + +This Skill is written by the lessons loop (`app/learning/`), from proposals a person approved. Do not edit it by hand: change the proposal and publish again. diff --git a/skills/google-cloud-lessons/references/deployment_guidance.md b/skills/google-cloud-lessons/references/deployment_guidance.md new file mode 100644 index 0000000..a1784ce --- /dev/null +++ b/skills/google-cloud-lessons/references/deployment_guidance.md @@ -0,0 +1,11 @@ +# Lessons: deployment guidance + +Rules learned from earlier work. Follow every one that applies. + +## L-05: Include Required Arguments in Alert Policies + +**Rule.** Always verify and include all required arguments (such as time window definitions) when configuring nested blocks in monitoring alert policies. + +**Why.** Missing required arguments in nested blocks will cause Terraform validation to fail, preventing submission. + +Published 2026-10-09, from lesson proposal 05. diff --git a/skills/google-cloud-lessons/references/target_architecture.md b/skills/google-cloud-lessons/references/target_architecture.md new file mode 100644 index 0000000..1717120 --- /dev/null +++ b/skills/google-cloud-lessons/references/target_architecture.md @@ -0,0 +1,43 @@ +# Lessons: target architecture + +Rules learned from earlier work. Follow every one that applies. + +## L-01: Create Specific Checks for All Constraints + +**Rule.** Create specific Checks for every explicit constraint, metric, or capability mentioned in a requirement, rather than relying on a single broad Check. + +**Why.** If detailed constraints are missed, the validation phase will reject the design for incomplete coverage, forcing a revision. + +Published 2026-10-09, from lesson proposal 01. + +## L-02: Assign Checks to Every Element + +**Rule.** Give every Element at least one Check before you submit. + +**Why.** The submission tool will reject the architecture if any Element lacks a Check, wasting an attempt. + +Published 2026-10-09, from lesson proposal 02. + +## L-03: Verify Resource Arguments for Checks + +**Rule.** Verify that the specific resource type used for an Element actually supports the configuration argument you specify in a Check. Remember that access policies and certain configurations often require entirely separate resources. + +**Why.** If a Check specifies an argument that does not exist on the resource, it cannot be implemented, forcing an architectural revision. + +Published 2026-10-09, from lesson proposal 03. + +## L-04: Avoid Using Values as Setting Names + +**Rule.** When writing a Check for a configuration block that takes a name and value, explicitly state the block's argument name instead of writing the expected value as if it were the setting itself. + +**Why.** Automated reviews may mistake specific expected values for invalid setting names, causing warnings and submission delays. + +Published 2026-10-09, from lesson proposal 04. + +## L-06: Do Not Mark Requirements as Not Applicable + +**Rule.** Do not list Requirements in the not applicable field, as this field is strictly for Design Recommendations. Ensure every Requirement has at least one Check. + +**Why.** The submission tool will reject the deliverable if a Requirement lacks a Check or is incorrectly marked as not applicable. + +Published 2026-10-09, from lesson proposal 06. diff --git a/tests/agents/test_roles.py b/tests/agents/test_roles.py index f652c25..f88e0d7 100644 --- a/tests/agents/test_roles.py +++ b/tests/agents/test_roles.py @@ -21,13 +21,15 @@ def test_every_engine_role_and_the_pillar_advisor_is_defined(bundle): expected = {"orchestrator", "reply-writer", PILLAR_ADVISOR} expected |= {spec.specialist for spec in DEPENDENCY_GRAPH.values()} - assert set(roles) == expected + # The lesson curator runs outside any Execution; the engine does not need it. + assert set(roles) == expected | {"lesson-curator"} for role in roles.values(): assert role.instructions.strip() def test_roles_take_skills_mcp_servers_and_models_from_the_bundle(bundle): - roles = load_roles(AGENTS_DIR, bundle) + # Without lessons: what the bundle's role_skills assigns, whatever has been learned. + roles = load_roles(AGENTS_DIR, bundle, lessons=False) assert roles["orchestrator"].skills == ("google-cloud-solution-architecture",) assert roles[PILLAR_ADVISOR].skills == () diff --git a/tests/test_lessons_loop.py b/tests/test_lessons_loop.py new file mode 100644 index 0000000..df26d86 --- /dev/null +++ b/tests/test_lessons_loop.py @@ -0,0 +1,401 @@ +"""The lessons loop (ADR-0004): capture, distil, review, publish, retire. + +No lesson reaches a Skill, and so an agent, without an issue file a person approved. +""" + +import json +import re +from datetime import date +from pathlib import Path + +import pytest + +from app.adk.fake_runtime import Call, FakeAgentRuntime +from app.adk.roles import load_roles +from app.adk.runtime import RoleSpec +from app.learning.curator import LESSON_CURATOR, curate +from app.learning.evidence import group_candidates, identifiers, read_candidates +from app.learning.proposals import ( + APPROVED, + NEEDS_TRIAGE, + REJECTED, + load, + parse, + propose, + write, +) +from app.learning.skill import ( + NotApproved, + deliverables_with_lessons, + publish, + published, + retire, +) +from app.providers import load_provider + +AGENTS_DIR = Path(__file__).parents[1] / "agents" +CURATOR = RoleSpec(LESSON_CURATOR, "Propose lessons.", budget=30) +DAY = date(2026, 10, 9) + + +def record(execution: str, *candidates: dict, case: str = "acme-webhooks") -> dict: + return { + "execution_id": execution, + "customer_case": case, + "candidates": list(candidates), + } + + +def refused(message: str, deliverable="target_architecture") -> dict: + return { + "type": "refusal", + "deliverable": deliverable, + "tool": "submit_deliverable", + "message": message, + } + + +NO_CHECK = "Not submitted. The Deliverable is missing: Element E-{n} has no Check" +RECORDS = [ + record("run-1", refused(NO_CHECK.format(n=3)), refused(NO_CHECK.format(n=9))), + record("run-2", refused(NO_CHECK.format(n=14))), + record( + "run-3", + refused(NO_CHECK.format(n=2)), + refused("Refused"), # a cloud read the Execution was not permitted + refused("Not submitted. Something that happened once."), + ), +] + + +# --- distil ---------------------------------------------------------------------------- + + +def test_what_recurs_across_executions_is_grouped_whatever_its_ids(): + [group] = group_candidates(RECORDS, min_executions=2) + + assert group.deliverable == "target_architecture" and group.kind == "refusal" + assert group.executions == {"run-1", "run-2", "run-3"} + assert group.occurrences == 4 + assert "E-3" not in group.description and "has no Check" in group.description + assert len(group.examples) == 3 + + +def test_a_one_off_and_a_refusal_by_policy_are_not_evidence(): + groups = group_candidates(RECORDS, min_executions=1) + + descriptions = [group.description for group in groups] + assert len(groups) == 2 + assert not any( + description.endswith("refused: Refused") for description in descriptions + ) + assert group_candidates(RECORDS[:1], min_executions=2) == [] + + +def test_the_same_evidence_has_the_same_key_on_every_run(): + first = group_candidates(RECORDS)[0].key + again = group_candidates(list(reversed(RECORDS)))[0].key + + assert first == again + + +def test_candidates_are_read_where_the_executions_wrote_them(tmp_path): + folder = tmp_path / "run" / "lessons" / "candidates" / "acme-webhooks" + folder.mkdir(parents=True) + (folder / "run-1.json").write_text(json.dumps(RECORDS[0])) + (folder / "broken.json").write_text("not json") + + # A folder given twice, or inside another that is given, is read once. + records = read_candidates([tmp_path, tmp_path / "run"]) + + assert [r["execution_id"] for r in records] == ["run-1"] + assert identifiers(records) == {"run-1", "acme-webhooks"} + + +# --- review: proposals are issue files --------------------------------------------------- + + +def proposal(folder: Path, **overrides): + values = { + "title": "Give every Element a Check", + "deliverable": "target_architecture", + "rule": "Before you submit, give every Element at least one Check.", + "reason": "A graph with an unchecked Element is refused, which costs a turn.", + "evidence": "Seen in 3 of 3 Executions, 4 times in all.", + "evidence_key": "abc1234567", + "folder": folder, + "today": DAY, + } + return propose(**{**values, **overrides}) + + +def test_a_proposal_is_an_issue_file_awaiting_a_persons_decision(tmp_path): + made = proposal(tmp_path) + + assert made.id == "L-01" and made.status == NEEDS_TRIAGE + assert made.path.name == "01-give-every-element-a-check.md" + text = made.path.read_text() + assert "Status: needs-triage" in text and "## Rule" in text and "## Proof" in text + assert parse(made.path) == made + assert proposal(tmp_path, title="Another").id == "L-02" + + +def test_a_persons_edits_to_the_file_are_what_is_read(tmp_path): + made = proposal(tmp_path) + text = made.path.read_text().replace("Status: needs-triage", "Status: wontfix") + made.path.write_text(text + "\nNot worth the tokens.\n") + + [read] = load(tmp_path) + + assert read.status == REJECTED and not read.approved + assert read.comments == "Not worth the tokens." + + +def test_an_unknown_status_is_an_error_not_an_approval(tmp_path): + made = proposal(tmp_path) + made.path.write_text(made.path.read_text().replace("needs-triage", "approved")) + + with pytest.raises(ValueError, match="Status must be one of"): + load(tmp_path) + + +# --- publish and retire ------------------------------------------------------------------ + + +def lessons_skill(tmp_path: Path) -> Path: + source = Path(__file__).parents[1] / "skills" / "google-cloud-lessons" / "SKILL.md" + skill = tmp_path / "google-cloud-lessons" + skill.mkdir() + # The real Skill's text, with whatever has been published since taken out. + empty = re.sub( + r".*", + "\nNo lessons have been published yet.\n" + "", + source.read_text(), + flags=re.DOTALL, + ) + (skill / "SKILL.md").write_text(empty) + return skill + + +def approve(made, folder: Path): + from dataclasses import replace + + return write(replace(made, status=APPROVED), folder) + + +def test_no_lesson_reaches_the_skill_without_an_approved_proposal(tmp_path): + skill = lessons_skill(tmp_path) + made = proposal(tmp_path / "issues") + + with pytest.raises(NotApproved, match="L-01 is needs-triage"): + publish(made, skill) + + assert published(skill) == {} + assert "No lessons have been published yet." in (skill / "SKILL.md").read_text() + + +def test_an_approved_lesson_is_written_for_its_deliverable_and_indexed(tmp_path): + skill = lessons_skill(tmp_path) + made = approve(proposal(tmp_path / "issues"), tmp_path / "issues") + + done = publish(made, skill, today=DAY) + + reference = (skill / "references" / "target_architecture.md").read_text() + assert "## L-01: Give every Element a Check" in reference + assert "**Rule.** Before you submit, give every Element at least one Check." in ( + reference + ) + assert "Published 2026-10-09, from lesson proposal 01." in reference + index = (skill / "SKILL.md").read_text() + assert "| target architecture | `references/target_architecture.md` | 1 |" in index + assert "No lessons have been published yet." not in index + assert done.published == "2026-10-09" + assert "Published: 2026-10-09" in done.path.read_text() + + +def test_publishing_again_replaces_a_lesson_and_never_duplicates_it(tmp_path): + from dataclasses import replace + + skill = lessons_skill(tmp_path) + issues = tmp_path / "issues" + first = publish(approve(proposal(issues), issues), skill, today=DAY) + second = approve( + proposal(issues, title="Name settings exactly", evidence_key="def"), issues + ) + publish(second, skill, today=DAY) + + publish(replace(first, rule="Give each Element two Checks."), skill, today=DAY) + + reference = (skill / "references" / "target_architecture.md").read_text() + assert reference.count("## L-01:") == 1 and reference.count("## L-02:") == 1 + assert "Give each Element two Checks." in reference + assert published(skill) == { + "L-01": "target_architecture", + "L-02": "target_architecture", + } + + +def test_a_retired_lesson_is_removed_and_an_empty_reference_goes_with_it(tmp_path): + skill = lessons_skill(tmp_path) + issues = tmp_path / "issues" + publish(approve(proposal(issues), issues), skill, today=DAY) + + assert retire("L-01", skill) is True + assert retire("L-01", skill) is False + + assert deliverables_with_lessons(skill) == [] + assert not (skill / "references" / "target_architecture.md").exists() + assert "No lessons have been published yet." in (skill / "SKILL.md").read_text() + + +# --- the curator only proposes ----------------------------------------------------------- + + +def curator_calls(*calls: Call) -> FakeAgentRuntime: + return FakeAgentRuntime({LESSON_CURATOR: [[*calls, Call("finish")]]}) + + +LESSON = { + "deliverable": "target_architecture", + "title": "Give every Element a Check", + "rule": "Before you submit, give every Element at least one Check.", + "reason": "An unchecked Element is refused, which costs a turn.", +} + + +async def test_the_curator_is_given_the_evidence_and_its_proposals_await_review( + tmp_path, +): + groups = group_candidates(RECORDS) + runtime = curator_calls(Call("propose_lesson", {"group": groups[0].key, **LESSON})) + + proposed = await curate(runtime, CURATOR, groups, executions=3, folder=tmp_path) + + [made] = proposed + assert made.status == NEEDS_TRIAGE and made.evidence_key == groups[0].key + assert made.evidence.startswith("Seen in 3 of 3 Executions, 4 times in all:") + # The issue file holds counts and a general description, never the examples. + assert "E-3" not in made.path.read_text() + context = runtime.runs[0].task.context + assert context["executions"] == 3 and context["groups"][0]["occurrences"] == 4 + assert len(context["groups"][0]["examples"]) == 3 + + +async def test_a_lesson_that_names_a_client_or_an_execution_is_refused(tmp_path): + groups = group_candidates(RECORDS) + leaky = {**LESSON, "reason": "It happened to acme-webhooks in run-2."} + runtime = curator_calls(Call("propose_lesson", {"group": groups[0].key, **leaky})) + + proposed = await curate( + runtime, + CURATOR, + groups, + executions=3, + identifiers=identifiers(RECORDS), + folder=tmp_path, + ) + + assert proposed == [] and load(tmp_path) == [] + refusal = runtime.runs[0].results[0][1] + assert refusal.startswith("Refused: a lesson carries no client or Execution") + assert "acme-webhooks" in refusal and "run-2" in refusal + + +async def test_evidence_already_decided_is_not_put_to_the_curator_again(tmp_path): + groups = group_candidates(RECORDS) + first = curator_calls(Call("propose_lesson", {"group": groups[0].key, **LESSON})) + await curate(first, CURATOR, groups, executions=3, folder=tmp_path) + again = FakeAgentRuntime({}) + + proposed = await curate(again, CURATOR, groups, executions=3, folder=tmp_path) + + assert proposed == [] and again.runs == [] # nothing new: the curator is not run + + +async def test_a_group_the_curator_passes_over_makes_no_proposal(tmp_path): + groups = group_candidates(RECORDS) + runtime = curator_calls( + Call("no_lesson", {"group": groups[0].key, "why": "The engine refuses it."}), + Call("propose_lesson", {"group": "nope", **LESSON}), + ) + + proposed = await curate(runtime, CURATOR, groups, executions=3, folder=tmp_path) + + assert proposed == [] + assert runtime.runs[0].results[1][1].startswith("Refused: nope is not one") + + +# --- a Specialist is given the lessons for its own Deliverable --------------------------- + + +def bundle_with_lessons(tmp_path: Path, *lessons_for: str): + """The real bundle, with its lessons Skill replaced by one that has lessons.""" + import dataclasses + + skill = lessons_skill(tmp_path) + issues = tmp_path / "issues" + for number, kind in enumerate(lessons_for): + made = approve( + proposal(issues, deliverable=kind, evidence_key=f"k{number}"), issues + ) + publish(made, skill, today=DAY) + bundle = load_provider("gcp") + name = bundle.lessons_skill + assert name is not None + skills = { + **bundle.skills, + name: dataclasses.replace(bundle.skills[name], path=skill), + } + return dataclasses.replace(bundle, skills=skills) + + +def test_a_specialist_is_given_the_lessons_skill_once_it_has_lessons_for_it(tmp_path): + bundle = bundle_with_lessons(tmp_path, "target_architecture") + + roles = load_roles(AGENTS_DIR, bundle) + + assert roles["target-architect"].skills[-1] == "google-cloud-lessons" + # A Specialist with nothing to read is not sent to load it. + assert "google-cloud-lessons" not in roles["iac-author"].skills + assert "google-cloud-lessons" not in roles["orchestrator"].skills + + +def test_lessons_can_be_switched_off_to_prove_what_they_change(tmp_path): + bundle = bundle_with_lessons(tmp_path, "target_architecture") + + roles = load_roles(AGENTS_DIR, bundle, lessons=False) + + assert all("google-cloud-lessons" not in role.skills for role in roles.values()) + + +def test_the_real_bundle_names_its_lessons_skill_and_gives_it_only_where_it_has_lessons(): + bundle = load_provider("gcp") + roles = load_roles(AGENTS_DIR, bundle) + + assert bundle.lessons_skill == "google-cloud-lessons" + learned = set(deliverables_with_lessons(bundle.skills["google-cloud-lessons"].path)) + given = { + name for name, role in roles.items() if "google-cloud-lessons" in role.skills + } + from app.workflows.deliverables import DEPENDENCY_GRAPH + + assert given == { + spec.specialist + for kind, spec in DEPENDENCY_GRAPH.items() + if kind.value in learned + } + + +def test_every_published_lesson_has_an_approved_proposal(): + """The real Skill and the real proposals: nothing is in one without the other.""" + from app.learning.proposals import ISSUES + + root = Path(__file__).parents[1] + live = published(root / "skills" / "google-cloud-lessons") + approved = {p.id: p for p in load(root / ISSUES) if p.approved} + + assert set(live) <= set(approved) + for lesson, deliverable in live.items(): + assert approved[lesson].deliverable == deliverable + assert approved[lesson].published