feat: lessons from what recurs about one setting, and three more approved
Some checks failed
validation / verify (push) Failing after 10s
Some checks failed
validation / verify (push) Failing after 10s
Evidence was grouped only by the kind of problem, and a group gets one lesson, so what kept recurring inside "validation proposed a Check" could not be seen. A proposed or unimplementable Check is now also counted under each provider setting or resource type it names; a subject seen in two or more Executions is its own group and reaches the curator past triage. Over 32 Executions that gave five proposals. The owner approved three, now published: L-07 location Checks, L-08 Cloud SQL backup encryption, L-09 backend service capabilities. L-10 is held for more evidence and L-11 rejected. Also: retiring a lesson marks its proposal, so publish no longer brings a retired lesson back. L-04 had returned to the Skill that way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -81,3 +81,17 @@ Neither run raised the schema review's false alarm, which had been raised in cas
|
||||
- **`status` says "compared", not "proved",** and shows a retired lesson as retired. One pair of runs is a comparison.
|
||||
|
||||
Tests: `tests/test_lessons_loop.py`.
|
||||
|
||||
## Evidence is also grouped by what it is about (2026-10-10)
|
||||
|
||||
Running `propose` over 32 Executions gave nothing useful. Evidence was grouped by the kind of problem, and a group gets one lesson: "validation proposed a Check the design lacked" was 20 of 32 Executions and already had L-01. What recurred inside it could not be seen.
|
||||
|
||||
- A proposed Check, and a Check sent back as unimplementable, is now also counted under each provider setting or resource type its text names (`_subjects` in `app/learning/evidence.py`). A subject seen in two or more Executions is its own group, with its own key. A setting's name is the provider's vocabulary, not a client's.
|
||||
- These narrower groups skip triage: each recurred with the general lessons published. The curator is shown the existing lessons and may still decline.
|
||||
- The curator's instructions say such a lesson may name the setting or resource type.
|
||||
|
||||
Over the same 32 Executions this gave 22 groups and five proposals, L-07 to L-11, all awaiting the owner: location Checks, Cloud SQL backup encryption, backend service capabilities, Pub/Sub persistence regions, and customer-managed keys.
|
||||
|
||||
A first run of `propose` before this change produced one proposal, about giving `edit_file` enough context. It was removed to run again cleanly and was not proposed the second time.
|
||||
|
||||
Tests: `tests/test_lessons_loop.py` (3 more).
|
||||
|
||||
@@ -5,6 +5,7 @@ Deliverable: target_architecture
|
||||
Evidence-key: 22ff1f5fed
|
||||
Proposed: 2026-10-09
|
||||
Published: 2026-10-09
|
||||
Retired: 2026-10-09
|
||||
|
||||
## Rule
|
||||
|
||||
|
||||
27
.scratch/lessons/issues/07-verify-location-constraints.md
Normal file
27
.scratch/lessons/issues/07-verify-location-constraints.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# L-07: Verify location constraints
|
||||
|
||||
Status: ready-for-agent
|
||||
Deliverable: target_architecture
|
||||
Evidence-key: 493b0a95d9
|
||||
Proposed: 2026-10-10
|
||||
Published: 2026-10-10
|
||||
|
||||
## Rule
|
||||
|
||||
When a requirement specifies where resources must be provisioned, such as a designated region, multi-zone placement, or a low-carbon location, create a Check for the `region` or `location` argument on every applicable Element.
|
||||
|
||||
## Why
|
||||
|
||||
Validation frequently catches missing location constraints. Without this Check, resources may be provisioned in default regions, violating compliance or sustainability requirements.
|
||||
|
||||
## Evidence
|
||||
|
||||
Seen in 5 of 32 Executions, 8 times in all: validation proposed a Check about `location` that the design lacked.
|
||||
|
||||
## Proof
|
||||
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
Approved by the owner on 2026-10-10. It covers which region is set, which code can show. It does not settle whether a region is low-carbon, which code cannot show.
|
||||
27
.scratch/lessons/issues/08-cloud-sql-backup-encryption.md
Normal file
27
.scratch/lessons/issues/08-cloud-sql-backup-encryption.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# L-08: Cloud SQL backup encryption
|
||||
|
||||
Status: ready-for-agent
|
||||
Deliverable: target_architecture
|
||||
Evidence-key: 08d64df049
|
||||
Proposed: 2026-10-10
|
||||
Published: 2026-10-10
|
||||
|
||||
## Rule
|
||||
|
||||
When writing a Check for CMEK encryption on a `google_sql_database_instance`, do not require an `encryption_key_name` argument for backup storage, as the instance-level `encryption_key_name` automatically encrypts backups as well.
|
||||
|
||||
## Why
|
||||
|
||||
The database instance resource does not have a separate encryption key argument for backups. Asking for one causes validation to reject the Check as un-implementable.
|
||||
|
||||
## Evidence
|
||||
|
||||
Seen in 3 of 32 Executions, 3 times in all: a Check about `google_sql_database_instance` asked for something that resource cannot do.
|
||||
|
||||
## Proof
|
||||
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
Approved by the owner on 2026-10-10.
|
||||
27
.scratch/lessons/issues/09-backend-service-capabilities.md
Normal file
27
.scratch/lessons/issues/09-backend-service-capabilities.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# L-09: Backend Service capabilities
|
||||
|
||||
Status: ready-for-agent
|
||||
Deliverable: target_architecture
|
||||
Evidence-key: 1cacec9a09
|
||||
Proposed: 2026-10-10
|
||||
Published: 2026-10-10
|
||||
|
||||
## Rule
|
||||
|
||||
When writing Checks for a `google_compute_backend_service`, do not verify IAM roles inside the backend service or require health checks when it is configured with a Serverless NEG.
|
||||
|
||||
## Why
|
||||
|
||||
Roles are verified via IAM bindings outside the backend service, and Serverless NEGs do not support health checks. Invalid Checks cause the design to be rejected during validation.
|
||||
|
||||
## Evidence
|
||||
|
||||
Seen in 3 of 32 Executions, 3 times in all: a Check about `google_compute_backend_service` asked for something that resource cannot do.
|
||||
|
||||
## Proof
|
||||
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
Approved by the owner on 2026-10-10.
|
||||
26
.scratch/lessons/issues/10-pub-sub-persistence-regions.md
Normal file
26
.scratch/lessons/issues/10-pub-sub-persistence-regions.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# L-10: Pub/Sub persistence regions
|
||||
|
||||
Status: needs-info
|
||||
Deliverable: target_architecture
|
||||
Evidence-key: ba78caba9a
|
||||
Proposed: 2026-10-10
|
||||
|
||||
## Rule
|
||||
|
||||
When a requirement dictates that a messaging queue or topic must replicate across zones or stay within a specific region, create a Check for the `message_storage_policy.allowed_persistence_regions` argument on the topic Element.
|
||||
|
||||
## Why
|
||||
|
||||
Without this Check, topics might use default storage policies that violate geographical or high-availability requirements.
|
||||
|
||||
## Evidence
|
||||
|
||||
Seen in 2 of 32 Executions, 4 times in all: validation proposed a Check about `allowed_persistence_regions` that the design lacked.
|
||||
|
||||
## Proof
|
||||
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
Held by the owner on 2026-10-10, on the agent's view: seen in 2 of 32 Executions, and L-07 partly covers it. To be looked at again if it recurs.
|
||||
26
.scratch/lessons/issues/11-cmek-encryption-checks.md
Normal file
26
.scratch/lessons/issues/11-cmek-encryption-checks.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# L-11: CMEK encryption checks
|
||||
|
||||
Status: wontfix
|
||||
Deliverable: target_architecture
|
||||
Evidence-key: 5cd8dc3cda
|
||||
Proposed: 2026-10-10
|
||||
|
||||
## Rule
|
||||
|
||||
Whenever a requirement mandates encryption at rest, explicitly create a Check ensuring that the `kms_key_name` (or the equivalent CMEK argument) is configured on every applicable Element.
|
||||
|
||||
## Why
|
||||
|
||||
Without an explicit Check, elements may default to Google-managed keys, leading to validation failures against security constraints.
|
||||
|
||||
## Evidence
|
||||
|
||||
Seen in 2 of 32 Executions, 2 times in all: validation proposed a Check about `kms_key_name` that the design lacked.
|
||||
|
||||
## Proof
|
||||
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
Rejected by the owner on 2026-10-10, on the agent's view: encryption at rest is on by default, and this rule would put customer-managed keys on designs whose requirements did not ask for them.
|
||||
@@ -8,7 +8,7 @@ You review what went wrong, the same way, across many solution-architecture Exec
|
||||
Your task message carries JSON context:
|
||||
|
||||
- `executions`: how many Executions the evidence comes from.
|
||||
- `groups`: what recurred. Each has a `group` key, the `deliverable` it concerns, `what_happened` in general terms, how many `executions` it occurred in and how many `occurrences` in all, and up to three `examples` as they were recorded.
|
||||
- `groups`: what recurred. Each has a `group` key, the `deliverable` it concerns, `what_happened` in general terms, how many `executions` it occurred in and how many `occurrences` in all, and up to three `examples` as they were recorded. Some groups are about one setting or resource type of the Cloud provider, named in `what_happened`: the same kind of problem recurring on that subject.
|
||||
- `published_or_proposed`: lessons that already exist or are awaiting review.
|
||||
|
||||
For each group, decide whether a rule given to the Specialist that produces that Deliverable would have prevented it.
|
||||
@@ -20,6 +20,7 @@ A good lesson:
|
||||
|
||||
- States one rule, as an instruction, that a Specialist can act on without knowing what happened: "Give every Element at least one Check before you submit", not "be more careful with Checks".
|
||||
- Says why in one or two sentences: what goes wrong without it and what that costs.
|
||||
- For a group about one setting or resource type, may name that setting or resource type and say what to do about it: that is the Cloud provider's vocabulary, not a client's. A general lesson that already exists does not cover it if the problem went on recurring.
|
||||
- Is general. It names no customer, no project, no Execution, no resource name and no value from the examples. The examples are evidence for you; nothing from them may be quoted.
|
||||
- Does not repeat what the Specialist's instructions or tools already enforce. When a submission is already refused for something, a lesson helps only if it prevents the attempt.
|
||||
- Is worth its cost. Every lesson is read on every Execution from then on. Propose few.
|
||||
|
||||
@@ -101,7 +101,12 @@ async def curate(
|
||||
decided = {proposal.evidence_key for proposal in existing}
|
||||
fresh = [group for group in groups if group.key not in decided][:MAX_GROUPS]
|
||||
if decider is not None and existing:
|
||||
fresh = await _not_already_covered(decider, fresh, existing, covered_at)
|
||||
# A group about one setting is narrower than any lesson about its kind of
|
||||
# problem, and it recurred with those lessons published: it goes to the
|
||||
# curator, who is shown the lessons that exist and may still decline.
|
||||
broad = [group for group in fresh if group.subject is None]
|
||||
kept = await _not_already_covered(decider, broad, existing, covered_at)
|
||||
fresh = [g for g in fresh if g.subject is not None or g in kept]
|
||||
if not fresh:
|
||||
return []
|
||||
by_key = {group.key: group for group in fresh}
|
||||
|
||||
@@ -10,6 +10,8 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.tools.schema_review import named_settings
|
||||
|
||||
EXAMPLES = 3
|
||||
EXAMPLE_LIMIT = 400
|
||||
# Ids and numbers vary between Executions; what recurs is the shape around them.
|
||||
@@ -29,6 +31,9 @@ class Group:
|
||||
executions: set[str] = field(default_factory=set)
|
||||
occurrences: int = 0
|
||||
examples: list[str] = field(default_factory=list) # client data: never published
|
||||
# The provider setting or resource type the problem was about, for a group
|
||||
# narrowed to one subject. None for a group of every problem of its kind.
|
||||
subject: str | None = None
|
||||
|
||||
def as_context(self) -> dict[str, Any]:
|
||||
return {
|
||||
@@ -61,6 +66,10 @@ def read_candidates(roots: Iterable[Path]) -> list[dict[str, Any]]:
|
||||
return records
|
||||
|
||||
|
||||
# A subject shorter than this is a stray word, not a setting.
|
||||
_SUBJECT_MIN = 4
|
||||
|
||||
|
||||
def _general(text: str, limit: int = 110) -> str:
|
||||
return " ".join(_VARIABLE.sub("…", text).split())[:limit]
|
||||
|
||||
@@ -104,6 +113,31 @@ def _shape(candidate: Mapping[str, Any]) -> tuple[str, str | None, str] | None:
|
||||
return None
|
||||
|
||||
|
||||
def _subjects(candidate: Mapping[str, Any]) -> tuple[str, list[str]] | None:
|
||||
"""For a problem about what a Check says: (how to describe it, the settings and
|
||||
resource types its text names).
|
||||
|
||||
Problems of one kind are one group, and one group gets one lesson. "Validation
|
||||
proposed a Check" was 20 of 32 Executions and a single lesson; what recurred
|
||||
inside it, such as the same setting missed again and again, was invisible. A
|
||||
setting's name is the provider's vocabulary, not a client's, so it can be the
|
||||
subject of a narrower group.
|
||||
"""
|
||||
kind = str(candidate.get("type"))
|
||||
if kind == "proposed_addition":
|
||||
text = str(candidate.get("statement", ""))
|
||||
what = "validation proposed a Check about `{}` that the design lacked"
|
||||
elif kind == "revision" and "cannot be implemented" in str(
|
||||
candidate.get("reason", "")
|
||||
):
|
||||
text = str(candidate.get("reason", ""))
|
||||
what = "a Check about `{}` asked for something that resource cannot do"
|
||||
else:
|
||||
return None
|
||||
names = {name.split(".")[-1] for name in named_settings(text)}
|
||||
return what, sorted(name for name in names if len(name) >= _SUBJECT_MIN)
|
||||
|
||||
|
||||
def group_candidates(
|
||||
records: Sequence[Mapping[str, Any]], *, min_executions: int = 2
|
||||
) -> list[Group]:
|
||||
@@ -138,6 +172,23 @@ def group_candidates(
|
||||
and len(group.examples) < EXAMPLES
|
||||
):
|
||||
group.examples.append(example)
|
||||
narrowed = _subjects(candidate)
|
||||
for subject in narrowed[1] if narrowed else ():
|
||||
about = narrowed[0].format(subject) # type: ignore[index]
|
||||
subkey = hashlib.sha256(
|
||||
f"{kind}|{deliverable}|{about}".encode()
|
||||
).hexdigest()[:10]
|
||||
narrow = groups.setdefault(
|
||||
subkey, Group(subkey, kind, deliverable, about, subject=subject)
|
||||
)
|
||||
narrow.executions.add(execution)
|
||||
narrow.occurrences += 1
|
||||
if (
|
||||
example
|
||||
and example not in narrow.examples
|
||||
and len(narrow.examples) < EXAMPLES
|
||||
):
|
||||
narrow.examples.append(example)
|
||||
return sorted(
|
||||
(g for g in groups.values() if len(g.executions) >= min_executions),
|
||||
key=lambda g: (-len(g.executions), -g.occurrences, g.key),
|
||||
|
||||
@@ -37,6 +37,9 @@ class Proposal:
|
||||
status: str = NEEDS_TRIAGE
|
||||
proposed: str = ""
|
||||
published: str = ""
|
||||
# The day it was taken out of the Skill again. A retired lesson is not
|
||||
# published a second time, whatever its Status still says.
|
||||
retired: str = ""
|
||||
proof: str = ""
|
||||
comments: str = ""
|
||||
path: Path | None = None
|
||||
@@ -47,7 +50,7 @@ class Proposal:
|
||||
|
||||
@property
|
||||
def approved(self) -> bool:
|
||||
return self.status == APPROVED
|
||||
return self.status == APPROVED and not self.retired
|
||||
|
||||
|
||||
def _slug(title: str) -> str:
|
||||
@@ -65,6 +68,8 @@ def render(proposal: Proposal) -> str:
|
||||
]
|
||||
if proposal.published:
|
||||
lines.append(f"Published: {proposal.published}")
|
||||
if proposal.retired:
|
||||
lines.append(f"Retired: {proposal.retired}")
|
||||
lines += [
|
||||
"",
|
||||
"## Rule",
|
||||
@@ -117,6 +122,7 @@ def parse(path: Path) -> Proposal:
|
||||
status=status,
|
||||
proposed=fields.get("Proposed", ""),
|
||||
published=fields.get("Published", ""),
|
||||
retired=fields.get("Retired", ""),
|
||||
proof="" if proof == "Not yet measured." else proof,
|
||||
comments=sections.get("Comments", ""),
|
||||
path=path,
|
||||
|
||||
@@ -111,6 +111,8 @@ def publish(
|
||||
Raises NotApproved for any proposal a person has not approved: nothing reaches
|
||||
an agent on a curator's say-so.
|
||||
"""
|
||||
if proposal.retired:
|
||||
raise NotApproved(f"{proposal.id} was retired on {proposal.retired}")
|
||||
if not proposal.approved:
|
||||
raise NotApproved(
|
||||
f"{proposal.id} is {proposal.status}; a person approves a lesson by "
|
||||
|
||||
@@ -35,6 +35,8 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import replace
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from app.agent import open_decider
|
||||
@@ -42,7 +44,7 @@ from app.config import get_settings
|
||||
from app.learning.curator import LESSON_CURATOR, curate
|
||||
from app.learning.evidence import group_candidates, identifiers, read_candidates
|
||||
from app.learning.proof import compare, record, render
|
||||
from app.learning.proposals import ISSUES, load
|
||||
from app.learning.proposals import ISSUES, load, write
|
||||
from app.learning.skill import (
|
||||
NotApproved,
|
||||
mark_reviewed,
|
||||
@@ -141,7 +143,7 @@ def show_status() -> int:
|
||||
"in the Skill"
|
||||
if proposal.id in live
|
||||
else "retired"
|
||||
if proposal.published
|
||||
if proposal.retired
|
||||
else "not in the Skill"
|
||||
)
|
||||
# One comparison suggests a direction; it does not prove a lesson.
|
||||
@@ -211,6 +213,10 @@ def retire_lesson(arguments: Sequence[str]) -> int:
|
||||
return 2
|
||||
bundle = load_provider(get_settings().CLOUD_PROVIDER)
|
||||
gone = retire(arguments[0], _skill_dir(bundle))
|
||||
for proposal in load(ROOT / ISSUES):
|
||||
if proposal.id == arguments[0] and not proposal.retired:
|
||||
# Marked on its proposal, so the next publish does not bring it back.
|
||||
write(replace(proposal, retired=datetime.now(UTC).date().isoformat()))
|
||||
print(f"Retired {arguments[0]}." if gone else f"{arguments[0]} is not published.")
|
||||
return 0 if gone else 1
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ Each reference below holds the lessons for one Deliverable. Read the one for the
|
||||
| Deliverable | Reference | Lessons |
|
||||
| --- | --- | --- |
|
||||
| deployment guidance | `references/deployment_guidance.md` | 1 |
|
||||
| target architecture | `references/target_architecture.md` | 4 |
|
||||
| target architecture | `references/target_architecture.md` | 7 |
|
||||
<!-- lessons:end -->
|
||||
|
||||
This Skill is written by the lessons loop (`app/learning/`), from proposals a person approved. Do not edit it by hand: change the proposal and publish again.
|
||||
|
||||
@@ -8,4 +8,4 @@ Rules learned from earlier work. Follow every one that applies.
|
||||
|
||||
**Why.** Missing required arguments in nested blocks will cause Terraform validation to fail, preventing submission.
|
||||
|
||||
Published 2026-10-09, from lesson proposal 05.
|
||||
Published 2026-10-10, from lesson proposal 05.
|
||||
|
||||
@@ -8,7 +8,7 @@ Rules learned from earlier work. Follow every one that applies.
|
||||
|
||||
**Why.** If detailed constraints are missed, the validation phase will reject the design for incomplete coverage, forcing a revision.
|
||||
|
||||
Published 2026-10-09, from lesson proposal 01.
|
||||
Published 2026-10-10, from lesson proposal 01.
|
||||
|
||||
## L-02: Assign Checks to Every Element
|
||||
|
||||
@@ -16,7 +16,7 @@ Published 2026-10-09, from lesson proposal 01.
|
||||
|
||||
**Why.** The submission tool will reject the architecture if any Element lacks a Check, wasting an attempt.
|
||||
|
||||
Published 2026-10-09, from lesson proposal 02.
|
||||
Published 2026-10-10, from lesson proposal 02.
|
||||
|
||||
## L-03: Verify Resource Arguments for Checks
|
||||
|
||||
@@ -24,7 +24,7 @@ Published 2026-10-09, from lesson proposal 02.
|
||||
|
||||
**Why.** If a Check specifies an argument that does not exist on the resource, it cannot be implemented, forcing an architectural revision.
|
||||
|
||||
Published 2026-10-09, from lesson proposal 03.
|
||||
Published 2026-10-10, from lesson proposal 03.
|
||||
|
||||
## L-06: Do Not Mark Requirements as Not Applicable
|
||||
|
||||
@@ -32,4 +32,28 @@ Published 2026-10-09, from lesson proposal 03.
|
||||
|
||||
**Why.** The submission tool will reject the deliverable if a Requirement lacks a Check or is incorrectly marked as not applicable.
|
||||
|
||||
Published 2026-10-09, from lesson proposal 06.
|
||||
Published 2026-10-10, from lesson proposal 06.
|
||||
|
||||
## L-07: Verify location constraints
|
||||
|
||||
**Rule.** When a requirement specifies where resources must be provisioned, such as a designated region, multi-zone placement, or a low-carbon location, create a Check for the `region` or `location` argument on every applicable Element.
|
||||
|
||||
**Why.** Validation frequently catches missing location constraints. Without this Check, resources may be provisioned in default regions, violating compliance or sustainability requirements.
|
||||
|
||||
Published 2026-10-10, from lesson proposal 07.
|
||||
|
||||
## L-08: Cloud SQL backup encryption
|
||||
|
||||
**Rule.** When writing a Check for CMEK encryption on a `google_sql_database_instance`, do not require an `encryption_key_name` argument for backup storage, as the instance-level `encryption_key_name` automatically encrypts backups as well.
|
||||
|
||||
**Why.** The database instance resource does not have a separate encryption key argument for backups. Asking for one causes validation to reject the Check as un-implementable.
|
||||
|
||||
Published 2026-10-10, from lesson proposal 08.
|
||||
|
||||
## L-09: Backend Service capabilities
|
||||
|
||||
**Rule.** When writing Checks for a `google_compute_backend_service`, do not verify IAM roles inside the backend service or require health checks when it is configured with a Serverless NEG.
|
||||
|
||||
**Why.** Roles are verified via IAM bindings outside the backend service, and Serverless NEGs do not support health checks. Invalid Checks cause the design to be rejected during validation.
|
||||
|
||||
Published 2026-10-10, from lesson proposal 09.
|
||||
|
||||
@@ -10,9 +10,9 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.adk.fake_runtime import Call, FakeAgentRuntime
|
||||
from app.adk.fake_runtime import Call, FakeAgentRuntime, FakeDecider
|
||||
from app.adk.roles import load_roles
|
||||
from app.adk.runtime import RoleSpec
|
||||
from app.adk.runtime import Answer, RoleSpec
|
||||
from app.learning.curator import LESSON_CURATOR, curate
|
||||
from app.learning.evidence import group_candidates, identifiers, read_candidates
|
||||
from app.learning.proposals import (
|
||||
@@ -563,3 +563,101 @@ def test_the_published_lessons_were_reviewed_against_todays_vendored_skills():
|
||||
|
||||
assert upstream_reviewed(skill), "run: python -m eval.curate_lessons reviewed"
|
||||
assert upstream_changes(skill, upstream_versions(bundle)) == []
|
||||
|
||||
|
||||
# --- what recurs inside a kind of problem ------------------------------------------------
|
||||
|
||||
|
||||
def addition(statement: str) -> dict:
|
||||
return {"type": "proposed_addition", "statement": statement, "serves": "NFR-1"}
|
||||
|
||||
|
||||
def test_the_same_setting_missed_in_several_executions_is_its_own_group():
|
||||
records = [
|
||||
record("run-1", addition("`deletion_protection` is true on the database")),
|
||||
record("run-2", addition("deletion_protection is enabled"), addition("x y")),
|
||||
record("run-3", addition("`min_tls_version` is TLS_1_2")),
|
||||
]
|
||||
|
||||
groups = {group.subject: group for group in group_candidates(records)}
|
||||
|
||||
# Every proposed Check is one broad group; the setting missed twice is another.
|
||||
assert len(groups[None].executions) == 3
|
||||
narrow = groups["deletion_protection"]
|
||||
assert len(narrow.executions) == 2 and narrow.kind == "proposed_addition"
|
||||
assert narrow.description == (
|
||||
"validation proposed a Check about `deletion_protection` that the design lacked"
|
||||
)
|
||||
assert narrow.key != groups[None].key
|
||||
# Seen in one Execution only, a setting is not a pattern.
|
||||
assert "min_tls_version" not in groups
|
||||
|
||||
|
||||
def test_a_resource_that_checks_keep_asking_too_much_of_is_its_own_group():
|
||||
def cannot(why: str) -> dict:
|
||||
return {
|
||||
"type": "revision",
|
||||
"deliverable": "target_architecture",
|
||||
"reason": f"Check K-1 cannot be implemented as written ({why}).",
|
||||
}
|
||||
|
||||
records = [
|
||||
record("run-1", cannot("example_database has no backup key argument")),
|
||||
record("run-2", cannot("example_database has no query access flag")),
|
||||
]
|
||||
|
||||
subjects = {group.subject for group in group_candidates(records)}
|
||||
|
||||
assert subjects == {None, "example_database"}
|
||||
|
||||
|
||||
async def test_a_group_about_one_setting_reaches_the_curator_past_triage(tmp_path):
|
||||
issues = tmp_path / "issues"
|
||||
approve(proposal(issues), issues) # a general lesson already exists
|
||||
records = [
|
||||
record("run-1", addition("`deletion_protection` is true")),
|
||||
record("run-2", addition("`deletion_protection` is true")),
|
||||
]
|
||||
groups = group_candidates(records)
|
||||
# Triage says the general lesson covers everything it is asked about.
|
||||
decider = FakeDecider(
|
||||
{group.key: Answer("L-01", 0.95) for group in groups if group.subject is None}
|
||||
)
|
||||
seen: list[str] = []
|
||||
|
||||
def turn(task):
|
||||
seen.extend(group["what_happened"] for group in task.context["groups"])
|
||||
return [Call("finish")]
|
||||
|
||||
runtime = FakeAgentRuntime({LESSON_CURATOR: [turn]})
|
||||
|
||||
await curate(
|
||||
runtime,
|
||||
CURATOR,
|
||||
groups,
|
||||
executions=2,
|
||||
folder=issues,
|
||||
decider=decider,
|
||||
)
|
||||
|
||||
assert seen == [
|
||||
"validation proposed a Check about `deletion_protection` that the design lacked"
|
||||
]
|
||||
|
||||
|
||||
def test_a_retired_lesson_is_not_published_again(tmp_path):
|
||||
"""Retiring left the proposal approved, so the next publish put the lesson
|
||||
back: L-04 returned to the Skill that way on 2026-10-10."""
|
||||
from dataclasses import replace
|
||||
|
||||
skill = lessons_skill(tmp_path)
|
||||
issues = tmp_path / "issues"
|
||||
made = publish(approve(proposal(issues), issues), skill, today=DAY)
|
||||
retire(made.id, skill)
|
||||
retired = write(replace(parse(made.path), retired="2026-10-09"), issues)
|
||||
|
||||
assert parse(retired.path).retired == "2026-10-09"
|
||||
assert not retired.approved
|
||||
with pytest.raises(NotApproved, match="was retired on 2026-10-09"):
|
||||
publish(retired, skill)
|
||||
assert published(skill) == {}
|
||||
|
||||
Reference in New Issue
Block a user