chore: housekeeping from the runs of 2026-10-09
Some checks failed
validation / verify (push) Failing after 12s

- An Execution's files are written as each Deliverable is submitted, marked
  in_progress, so a killed process or one waiting for its user leaves them on
  disk. A write removes files an earlier write left that it no longer has.
- A question about delivered work is answered from that Execution's solution
  guide by the reply writer; a question about the service is answered as
  before. Nothing is started for either.
- Lessons: prove writes only for lessons the Skill holds; the Skill records
  the vendored Skill versions the lessons were read against, status names any
  that changed, and a test fails until they are reviewed.
- Four more code patterns: Firestore, BigQuery, Eventarc and GKE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 11:44:38 +01:00
parent 10d23a5f91
commit 71da0ed504
24 changed files with 1109 additions and 43 deletions

View File

@@ -44,6 +44,7 @@
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -73,3 +73,11 @@ Case 03, one run each way, side by side, with the five lessons that remain (`202
`prove` recorded this in the proposals as "kept". One pair suggests a direction and does not settle it: a third run the same evening, with lessons, did need a Revision of the Target architecture. `prove` also wrote to L-04's proposal, which is retired; it should skip retired lessons.
Neither run raised the schema review's false alarm, which had been raised in case 06 before the fix.
## Housekeeping, 2026-10-10
- **`prove` skips retired lessons.** It wrote its comparison into L-04's proposal after L-04 had been retired. It now writes only for lessons the Skill holds. L-04's proposal is corrected.
- **Lessons are flagged when the vendored Skills change.** The lessons Skill keeps `upstream.json`: the version of each vendored Skill the lessons were last read against. `status` names the ones that have changed since and says to read each lesson again; `reviewed` records that a person has. A test fails until then, so a sync of the vendored Skills cannot go unnoticed. The first publish records the versions; after that only `reviewed` moves them.
- **`status` says "compared", not "proved",** and shows a retired lesson as retired. One pair of runs is a comparison.
Tests: `tests/test_lessons_loop.py`.

View File

@@ -0,0 +1,21 @@
# Step 28 — Housekeeping (2026-10-10)
Small things found during the runs of 2026-10-09. Not run with real models unless said.
## Files are written as an Execution goes
An Execution's folder was written only when it ended, so a killed process, or an Execution left waiting for its user, left nothing on disk.
- Each time a Deliverable is submitted, the folder is written with what exists, and `execution.json` says `in_progress`. The end of the Execution writes it again with its final status.
- An in-progress Execution is refused as a Baseline, as any Execution that did not complete is.
- Because the folder is now written more than once, a write removes the files an earlier write recorded and this one does not have. Without that, a Terraform file removed in a Revision would have stayed beside the files that replaced it. Only files the Execution itself wrote are removed.
Tests: `tests/execution/test_outputs_as_it_goes.py` (2).
## Four more code patterns
`skills/google-cloud-iac-patterns/` gains Firestore, BigQuery, Eventarc and Google Kubernetes Engine, 18 in all. Each validates against `hashicorp/google` 8.6.0 in `tests/test_iac_patterns.py`.
## The two explainer pages
Both now describe the opt-in to published Terraform modules.

View File

@@ -76,3 +76,14 @@ Not built: acting on a change request (iteration 7 step 2), and answering a ques
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).
## A question about delivered work is answered from it (2026-10-10)
Intake answered every question with a description of the service. A question read as about work already delivered in the caller's Customer case is now answered from that Execution's solution guide.
1. When intake reads a message as a question and the case has a completed Execution, a second yes or no is asked: is this about an architecture already delivered, or about the service in general. At 0.6 or above it is about delivered work.
2. `DeliveredAnswers` in `app/nodes/intake.py` loads the latest completed Execution and gives its solution guide and the question to the reply writer. `agents/reply-writer.md` now covers this second task: answer from the guide alone, say so when it does not say, name the section, propose no change.
3. The A2A reply carries `about`, the Execution it was answered from. Nothing is started. When no answer could be written the caller is told which Execution's documents to read.
Measured against live Jev on twelve messages: six about delivered work scored 0.37 to 0.96, five of them 0.73 or more ("what does the load balancer in my architecture do?" scored 0.37 and would get the general answer). Five about the service scored 0.07 to 0.16. "Is my data stored anywhere?" scored 0.74: it is ambiguous, and would get an answer from the guide. **No real model has written such an answer yet.**
Tests: `tests/execution/test_intake.py` (3 more), `tests/a2a/test_server.py` (3 more).

View File

@@ -20,15 +20,7 @@ Seen in 2 of 21 Executions, 4 times in all: submit_deliverable refused: Not subm
## Proof
Evaluation without lessons: `20261009T195603Z`; with: `20261009T195607Z`.
| Case | Score | Revisions | Minutes | Tokens (M) |
| --- | --- | --- | --- | --- |
| case-03-internal-web-app | 1.00 → 1.00 | 1 → 0 | 28 → 22 | 3.00 → 2.32 |
Quality held and revisions, seconds, tokens fell: kept.
This compares every published lesson together, on one run each way. Runs of the same case vary, so one pair suggests a direction and does not settle it.
Not yet measured.
## Comments

View File

@@ -2,9 +2,11 @@
role: reply-writer
tools: []
---
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. The decision has already been made; you put it into words and decide nothing.
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. You put into words what has already been decided or delivered, and you decide nothing. Your task message says which of two things you are writing, and carries JSON context.
Your task message carries JSON context: `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
## A summary to approve a Phase from
The context has `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
Write a summary a busy architect can approve from:
@@ -13,4 +15,14 @@ Write a summary a busy architect can approve from:
- Name anything they should look at before approving: failed or unverifiable Checks, open gaps, recommendations treated as not applicable, Deliverables that were revised.
- Use plain sentences and no headings. Keep it under 120 words.
Call `write_reply` with the summary. You are done when it accepts it.
## An answer about delivered work
The context has `question`, what the user asked, and `delivered`: the `request` the architecture was designed for and its `solution_guide`.
- Answer from the solution guide alone. Give the choice or the fact, and the reason the guide gives for it.
- When the guide does not say, say that it does not. Add nothing from general knowledge about the products, and do not guess.
- Name the section of the guide the answer comes from.
- Propose no change. If the user seems to want one, say they can ask for it as a change to this architecture.
- Use plain sentences and no headings. Keep it under 150 words.
Call `write_reply` with the text. You are done when it accepts it.

View File

@@ -31,7 +31,15 @@ from google.protobuf import json_format
from app.agent import ExecutionEngine, Interaction
from app.nodes.execution_nodes import GUIDE_FILENAME
from app.nodes.intake import ABOUT, CHANGE_REQUEST, QUESTION, Intake, change_note
from app.nodes.intake import (
ABOUT,
CHANGE_REQUEST,
QUESTION,
DeliveredAnswers,
Intake,
change_note,
no_answer_note,
)
from app.presentation import describe_prompt
from app.workflows.deliverables import DeliverableKind
@@ -41,7 +49,13 @@ STRUCTURED_REPLY_KEYS = ("approval", "permission")
class SolutionArchitectExecutor(AgentExecutor):
def __init__(self, engine: ExecutionEngine, intake: Intake | None = None) -> None:
def __init__(
self,
engine: ExecutionEngine,
intake: Intake | None = None,
answers: DeliveredAnswers | None = None,
) -> None:
self._answers = answers
self._engine = engine
self._intake = intake
@@ -75,14 +89,21 @@ class SolutionArchitectExecutor(AgentExecutor):
scope=task_id,
)
if reading.kind == QUESTION:
text, about = ABOUT, None
if reading.about_delivered and reading.latest and self._answers:
# About what was delivered: answered from its files.
about = reading.latest
text = await self._answers.answer(
context.get_user_input(),
str(options.get("customer_case")),
about,
) or no_answer_note(about)
data = {"status": "answered", "intake": QUESTION}
if about:
data["about"] = about
await updater.complete(
updater.new_agent_message(
[
new_text_part(ABOUT),
new_data_part(
{"status": "answered", "intake": QUESTION}
),
]
[new_text_part(text), new_data_part(data)]
)
)
return

View File

@@ -36,7 +36,7 @@ from app.baseline import load_baseline
from app.checkpointing import open_checkpointer
from app.config import Settings
from app.customer_case import record_intent
from app.nodes.intake import Intake
from app.nodes.intake import DeliveredAnswers, Intake
from app.nodes.router import (
REPLY_WRITER,
OrchestratorPolicy,
@@ -328,6 +328,8 @@ class Assembly:
mcp_gaps: dict[str, str]
decider: Decider = field(default_factory=NoDecider)
intake: Intake | None = None # reads a message that starts something
# Answers a question about delivered work from its files.
answers: DeliveredAnswers | None = None
def discover_overlays(settings: Settings, bundle: ProviderBundle) -> dict[str, Any]:
@@ -529,6 +531,9 @@ async def open_assembly(
output_root or settings.BASE_DIR,
act=settings.JEV_ACT_THRESHOLD,
),
DeliveredAnswers(
runtime, roles[REPLY_WRITER], output_root or settings.BASE_DIR
),
)
finally:
if isinstance(decider, JevDecider):

View File

@@ -7,7 +7,7 @@ are ordinary evaluation reports; one is made with LESSONS_ENABLED=false.
from __future__ import annotations
import json
from collections.abc import Mapping, Sequence
from collections.abc import Collection, Mapping, Sequence
from dataclasses import replace
from pathlib import Path
from typing import Any
@@ -106,6 +106,12 @@ def render(result: Mapping[str, Any], without: Path, with_lessons: Path) -> str:
return "\n".join(lines)
def record(proposals: Sequence[Proposal], proof: str) -> list[Proposal]:
"""Write the comparison into every published proposal."""
return [write(replace(p, proof=proof)) for p in proposals if p.published]
def record(
proposals: Sequence[Proposal], proof: str, live: Collection[str]
) -> list[Proposal]:
"""Write the comparison into the proposal of every lesson in the Skill.
`live` is the ids the Skill holds now. A retired lesson was not part of what
was compared, so its proposal is left as it was.
"""
return [write(replace(p, proof=proof)) for p in proposals if p.id in live]

View File

@@ -8,7 +8,9 @@ the index in SKILL.md are written here, from approved proposals, and never by ha
from __future__ import annotations
import json
import re
from collections.abc import Mapping
from dataclasses import replace
from datetime import UTC, date, datetime
from pathlib import Path
@@ -130,6 +132,45 @@ def publish(
return write(replace(proposal, published=proposal.published or stamp))
# The versions of the Skills the lessons were written beside, kept in the lessons
# Skill. A lesson is a correction to how agents follow those Skills, so when one
# of them changes each lesson is worth a second look: it may now be said upstream,
# or contradict it.
UPSTREAM_FILE = "upstream.json"
def upstream_reviewed(skill_dir: Path) -> dict[str, str]:
"""The upstream Skill versions the lessons were last reviewed against."""
path = skill_dir / UPSTREAM_FILE
if not path.is_file():
return {}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except ValueError:
return {}
return {str(k): str(v) for k, v in data.items()} if isinstance(data, dict) else {}
def mark_reviewed(skill_dir: Path, current: Mapping[str, str]) -> None:
"""Record that the lessons stand against these upstream Skill versions."""
(skill_dir / UPSTREAM_FILE).write_text(
json.dumps(dict(sorted(current.items())), indent=2) + "\n", encoding="utf-8"
)
def upstream_changes(skill_dir: Path, current: Mapping[str, str]) -> list[str]:
"""The upstream Skills that changed, appeared or went since the lessons were
last reviewed. Empty when there are no lessons, or nothing was ever recorded."""
reviewed = upstream_reviewed(skill_dir)
if not reviewed or not published(skill_dir):
return []
return sorted(
name
for name in set(reviewed) | set(current)
if reviewed.get(name) != current.get(name)
)
def retire(lesson_id: str, skill_dir: Path) -> bool:
"""Remove a lesson from the Skill. Returns whether it was there."""
deliverable = published(skill_dir).get(lesson_id)

View File

@@ -89,7 +89,7 @@ def create_app(
async with assembly_factory(settings) as assembly:
handler = DefaultRequestHandlerV2(
agent_executor=SolutionArchitectExecutor(
assembly.engine, assembly.intake
assembly.engine, assembly.intake, assembly.answers
),
task_store=InMemoryTaskStore(),
agent_card=card,

View File

@@ -15,7 +15,7 @@ import time
from collections.abc import Awaitable, Callable, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from typing import Any, cast
from langgraph.types import Command, Send, interrupt
@@ -37,7 +37,7 @@ from app.adk.runtime import (
TurnEnded,
)
from app.baseline import WORKING_FILENAME
from app.customer_case import close_intent
from app.customer_case import IN_PROGRESS, close_intent
from app.nodes.router import ModelOrchestrator, OrchestratorPolicy, TurnView
from app.recorder import ExecutionRecorder
from app.states.execution_state import (
@@ -207,11 +207,13 @@ def make_execution_nodes(
drafts = dict(state.get("drafts", {}))
outcomes = state.get("outcomes", [])
arrived = False
for record in outcomes[state.get("outcomes_applied", 0) :]:
kind = DeliverableKind(record["kind"])
try:
if record["type"] == "submitted":
ledger.submit(kind, record["attempt"], actor=record["role"])
arrived = True
deliverables[kind.value] = {
"content": record["content"],
"attachments": record["attachments"],
@@ -252,6 +254,19 @@ def make_execution_nodes(
]
pending_questions.append({"kind": kind.value, "questions": questions})
if arrived:
# What has been submitted is on disk as soon as it exists, marked as in
# progress. A process that is killed leaves that much behind.
try:
_write_outputs(
output_root,
cast(ExecutionState, {**state, "deliverables": deliverables}),
IN_PROGRESS,
ledger,
)
except OSError:
logger.exception("Writing the Execution's files so far failed")
turns = state.get("turns", 0) + 1
update: dict[str, Any] = {
"deliverables": deliverables,
@@ -1214,6 +1229,26 @@ def _case(state: ExecutionState) -> str:
return state.get("customer_case") or UNASSIGNED_CASE
def _written_before(out_dir: Path) -> set[str]:
"""The files an earlier write of this Execution's folder recorded."""
try:
manifest = json.loads((out_dir / MANIFEST_FILENAME).read_text("utf-8"))
except (OSError, ValueError):
return set()
if not isinstance(manifest, dict):
return set()
names = {manifest.get("solution_guide"), manifest.get("changes")}
for entry in (manifest.get("deliverables") or {}).values():
if isinstance(entry, dict):
names.add(entry.get("document"))
names.update(entry.get("attachments") or [])
return {
name
for name in names
if isinstance(name, str) and name and ".." not in Path(name).parts
}
def _write_outputs(
output_root: Path,
state: ExecutionState,
@@ -1267,6 +1302,11 @@ def _write_outputs(
indent=2,
ensure_ascii=False,
)
# The folder is written more than once as an Execution goes. A file an earlier
# write left that this one does not have, such as a Terraform file removed in a
# Revision, must not stay beside the files that replaced it.
for stale in _written_before(out_dir) - set(files):
(out_dir / stale).unlink(missing_ok=True)
for name, content in files.items():
path = out_dir / name
path.parent.mkdir(parents=True, exist_ok=True)

View File

@@ -6,6 +6,7 @@ one step earlier, for a message that arrives with no Execution in progress:
new request design this workload start an Execution
change request change what was delivered name the Baseline to change (ADR-0006)
question what can you do, how long... answer; start nothing
why did you choose X? answer from what was delivered
A change is never started from earlier work on a guess: the caller chooses the
Baseline. So a message read as a change is answered with the Execution it would
@@ -24,7 +25,17 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
from app.adk.runtime import Decider, Question
from app.adk.runtime import (
AgentRuntime,
AgentTask,
Decider,
Question,
RoleSpec,
Submitted,
ToolResult,
ToolSpec,
)
from app.baseline import BaselineError, load_baseline
from app.customer_case import IN_PROGRESS, read_history
from app.states.execution_state import UNASSIGNED_CASE
@@ -53,7 +64,18 @@ def change_note(baseline: str) -> str:
)
def no_answer_note(execution_id: str) -> str:
"""What a caller is told when a question about delivered work went unanswered."""
return (
"I could not answer that from the documents delivered in Execution "
f"{execution_id}. Its solution guide and Deliverables hold what was "
"decided and why. Nothing was started for this message."
)
_MESSAGE_LIMIT = 8_000
# How much of the delivered guide the answer is written from.
_GUIDE_LIMIT = 80_000
_EARLIER_SHOWN = 5
@@ -64,6 +86,8 @@ class IntakeReading:
judged: bool = False # False when no decision was available
earlier: int = 0 # finished Executions of the same Customer case
latest: str | None = None # the most recent of them: what a change would change
# A question about what was delivered, as opposed to about the service.
about_delivered: bool = False
def finished_intents(output_root: Path, customer_case: str) -> list[Mapping[str, Any]]:
@@ -85,11 +109,18 @@ class Intake:
output_root: Path,
*,
act: float = 0.75,
about_delivered_at: float = 0.6,
) -> None:
self._decider = decider
self._output_root = output_root
# How sure the reading must be before anything but a new Execution is done.
self._act = act
# How likely a question must be about delivered work to be answered from
# it. Measured on 2026-10-10: six such questions scored 0.37 to 0.96, five
# of them 0.73 or more; six about the service scored 0.07 to 0.16 and one
# ambiguous one ("is my data stored anywhere?") 0.74. A wrong yes costs an
# answer that says the documents do not cover it.
self._about_delivered_at = about_delivered_at
async def read(
self, message: str, customer_case: str | None = None, scope: str = ""
@@ -123,9 +154,38 @@ class Intake:
kind = NEW_REQUEST # there is nothing delivered to change
logger.info("[%s] intake: %s (%.2f)", scope, kind, confidence)
latest = str(earlier[-1].get("execution_id")) if earlier else None
return IntakeReading(
kind, confidence, judged=True, earlier=len(earlier), latest=latest
about = (
await self._about_delivered(text, earlier, scope)
if kind == QUESTION and latest
else False
)
return IntakeReading(
kind,
confidence,
judged=True,
earlier=len(earlier),
latest=latest,
about_delivered=about,
)
async def _about_delivered(
self, text: str, earlier: Sequence[Mapping[str, Any]], scope: str
) -> bool:
answers = await self._decider.decide(
_state(text, earlier),
{
"about_delivered": Question(
"noul",
"Is the user asking about an architecture already delivered to "
"them: what it contains, why something was chosen, or what "
"validation found? Answer no when they ask what this service "
"does or how it works in general.",
)
},
scope=scope,
)
answer = (answers or {}).get("about_delivered")
return answer is not None and float(answer.value) >= self._about_delivered_at
def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
@@ -135,3 +195,65 @@ def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
lines.append(f"- {request}")
lines.append(f"The user's message: {message[:_MESSAGE_LIMIT]}")
return "\n".join(lines)
class DeliveredAnswers:
"""Answers a question about delivered work from its documents, on the reply
writer. It reads one Execution's files and starts nothing."""
def __init__(
self, runtime: AgentRuntime, role: RoleSpec, output_root: Path
) -> None:
self._runtime = runtime
self._role = role
self._output_root = output_root
async def answer(
self, question: str, customer_case: str, execution_id: str
) -> str | None:
"""The answer, or None when the documents could not be read or the writer
gave none."""
try:
delivered = load_baseline(self._output_root, customer_case, execution_id)
except BaselineError:
return None
guide = str(
(delivered["deliverables"].get("solution_guide") or {}).get("content", "")
)
if not guide.strip():
return None
async def write_reply(text: str) -> ToolResult:
if not text.strip():
return ToolResult("Not submitted: the reply is empty.")
return ToolResult("Written.", Submitted(text))
tool = ToolSpec(
"write_reply",
"Give the text the user will read. Ends your turn.",
{
"type": "object",
"properties": {"text": {"type": "string"}},
"required": ["text"],
},
write_reply,
tier="execution",
)
task = AgentTask(
role=self._role.name,
prompt="Answer the user's question about the architecture delivered to them.",
context={
"question": question[:_MESSAGE_LIMIT],
"delivered": {
"execution_id": execution_id,
"request": delivered.get("workflow_request", ""),
"solution_guide": guide[:_GUIDE_LIMIT],
},
},
)
try:
outcome = await self._runtime.run(self._role, task, [tool])
except Exception: # a question must still get a reply
logger.exception("Answering from delivered work failed")
return None
return outcome.content if isinstance(outcome, Submitted) else None

View File

@@ -9,6 +9,8 @@
prove compare an evaluation run without lessons with one run with them,
and write the comparison into every published proposal.
retire remove a lesson from the Skill, by id.
reviewed record that the lessons were read against the vendored Skills as
they are now. `status` says when those have changed since.
Usage:
.venv/bin/python -m eval.curate_lessons propose [--from <folder> ...] [--min 2]
@@ -16,6 +18,7 @@ Usage:
.venv/bin/python -m eval.curate_lessons publish
.venv/bin/python -m eval.curate_lessons prove --without <report.json> --with <report.json>
.venv/bin/python -m eval.curate_lessons retire L-03
.venv/bin/python -m eval.curate_lessons reviewed
A person approves a proposal by editing its file: set `Status: ready-for-agent`, or
`wontfix` to reject it. Nothing is published without that. After publishing, prove
@@ -40,8 +43,17 @@ from app.learning.curator import LESSON_CURATOR, curate
from app.learning.evidence import group_candidates, identifiers, read_candidates
from app.learning.proof import compare, record, render
from app.learning.proposals import ISSUES, load
from app.learning.skill import NotApproved, publish, published, retire
from app.learning.skill import (
NotApproved,
mark_reviewed,
publish,
published,
retire,
upstream_changes,
upstream_reviewed,
)
from app.providers import ProviderBundle, load_provider
from app.skills.agent_skills import skill_version
ROOT = Path(__file__).resolve().parent.parent
# Where Executions write their candidates: under each evaluation run, and under the
@@ -55,6 +67,16 @@ def _skill_dir(bundle: ProviderBundle) -> Path:
return bundle.skills[bundle.lessons_skill].path
def upstream_versions(bundle: ProviderBundle) -> dict[str, str]:
"""The vendored Skills, each with the version it has now. They are what the
lessons correct the use of, and they change when upstream is synced."""
return {
name: skill_version(meta.path)
for name, meta in bundle.skills.items()
if "vendor" in meta.path.parts
}
def _options(arguments: Sequence[str], name: str) -> list[str]:
return [
arguments[index + 1]
@@ -115,12 +137,34 @@ def show_status() -> int:
if not proposals:
print("No lesson has been proposed.")
for proposal in proposals:
where = "in the Skill" if proposal.id in live else "not in the Skill"
proof = "proved" if proposal.proof else "not proved"
where = (
"in the Skill"
if proposal.id in live
else "retired"
if proposal.published
else "not in the Skill"
)
# One comparison suggests a direction; it does not prove a lesson.
proof = "compared" if proposal.proof else "not compared"
print(
f"{proposal.id} {proposal.status:16s} {where:16s} {proof:11s} "
f"{proposal.id} {proposal.status:16s} {where:16s} {proof:12s} "
f"[{proposal.deliverable}] {proposal.title}"
)
changed = upstream_changes(_skill_dir(bundle), upstream_versions(bundle))
if changed:
print(
f"\nVendored Skills changed since the lessons were last reviewed: "
f"{', '.join(changed)}.\nRead each lesson in the Skill against them: "
"retire one the Skill now says itself or contradicts. Then run "
"`reviewed`."
)
return 0
def mark_lessons_reviewed() -> int:
bundle = load_provider(get_settings().CLOUD_PROVIDER)
mark_reviewed(_skill_dir(bundle), upstream_versions(bundle))
print("Recorded: the lessons stand against the vendored Skills as they are now.")
return 0
@@ -135,6 +179,10 @@ def publish_approved() -> int:
continue
done += 1
print(f" published {proposal.id} [{proposal.deliverable}] {proposal.title}")
if done and not upstream_reviewed(skill_dir):
# The first lessons: what they were written beside is recorded once. After
# that only a person's review moves it on.
mark_reviewed(skill_dir, upstream_versions(bundle))
print(f"{done} lessons are in {skill_dir.name}.")
return 0
@@ -150,7 +198,8 @@ def prove_lessons(arguments: Sequence[str]) -> int:
first, second = Path(without[0]), Path(with_lessons[0])
result = compare(first, second)
proof = render(result, first, second)
recorded = record(load(ROOT / ISSUES), proof)
bundle = load_provider(get_settings().CLOUD_PROVIDER)
recorded = record(load(ROOT / ISSUES), proof, published(_skill_dir(bundle)))
print(proof)
print(f"\nRecorded in {len(recorded)} published proposals.")
return 0 if result["helps"] else 1
@@ -178,6 +227,8 @@ def main(arguments: Sequence[str]) -> int:
return prove_lessons(rest)
if command == "retire":
return retire_lesson(rest)
if command == "reviewed":
return mark_lessons_reviewed()
print(__doc__, file=sys.stderr)
return 2

View File

@@ -4,10 +4,10 @@ description: >-
Terraform patterns for Google Cloud products that validate against one pinned
version of the hashicorp/google provider: Cloud Run services and jobs,
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring and API
enablement. Use when writing Terraform for a Google Cloud product, before
writing the resource, to start from a pattern whose arguments are known to be
in the right place.
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring,
Firestore, BigQuery, Eventarc, GKE and API enablement. Use when writing
Terraform for a Google Cloud product, before writing the resource, to start
from a pattern whose arguments are known to be in the right place.
metadata:
scope: deployment guidance
provider: hashicorp/google 8.6.0
@@ -39,6 +39,10 @@ Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0.
| Cloud Scheduler | `references/cloud-scheduler.md` |
| Artifact Registry | `references/artifact-registry.md` |
| Cloud Monitoring and Logging | `references/monitoring.md` |
| Firestore | `references/firestore.md` |
| BigQuery | `references/bigquery.md` |
| Eventarc | `references/eventarc.md` |
| Google Kubernetes Engine | `references/gke.md` |
A product with no reference has no pattern yet: write it from `terraform_schema` and the product documentation.

View File

@@ -0,0 +1,92 @@
# BigQuery
A dataset, a partitioned and clustered table, and the grants for a writer and a reader.
- `location` is a region or `EU`/`US`, and cannot be changed. Queries cannot join datasets in different locations.
- `delete_contents_on_destroy = false` makes Terraform refuse to remove a dataset that still has tables.
- A table's `deletion_protection` is true unless set; take it from a variable so a test deployment can be removed.
- `schema` is a JSON string. Changing a column's type or mode replaces the table.
- Partition on a `DATE` or `TIMESTAMP` column with `time_partitioning`, and set `require_partition_filter` so a query cannot scan every partition. `expiration_ms` there removes old partitions.
- Grant on the dataset with `google_bigquery_dataset_iam_member`. A principal that runs queries also needs `roles/bigquery.jobUser` on the project.
- Do not mix `google_bigquery_dataset_iam_member` with `access` blocks on the dataset: each removes what the other sets.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "writer_service_account_email" {
type = string
description = "Service account that loads rows."
}
variable "reader_service_account_email" {
type = string
description = "Service account that queries the dataset."
}
variable "deletion_protection" {
type = bool
description = "Whether the table is protected from deletion."
default = true
}
resource "google_bigquery_dataset" "analytics" {
project = var.project_id
dataset_id = "analytics"
location = var.region
delete_contents_on_destroy = false
default_table_expiration_ms = null
max_time_travel_hours = 168
}
resource "google_bigquery_table" "events" {
project = var.project_id
dataset_id = google_bigquery_dataset.analytics.dataset_id
table_id = "events"
deletion_protection = var.deletion_protection
require_partition_filter = true
clustering = ["event_type"]
time_partitioning {
type = "DAY"
field = "event_time"
expiration_ms = 34560000000
}
schema = jsonencode([
{ name = "event_id", type = "STRING", mode = "REQUIRED" },
{ name = "event_type", type = "STRING", mode = "REQUIRED" },
{ name = "event_time", type = "TIMESTAMP", mode = "REQUIRED" },
{ name = "payload", type = "JSON", mode = "NULLABLE" }
])
}
resource "google_bigquery_dataset_iam_member" "writer" {
project = var.project_id
dataset_id = google_bigquery_dataset.analytics.dataset_id
role = "roles/bigquery.dataEditor"
member = "serviceAccount:${var.writer_service_account_email}"
}
resource "google_bigquery_dataset_iam_member" "reader" {
project = var.project_id
dataset_id = google_bigquery_dataset.analytics.dataset_id
role = "roles/bigquery.dataViewer"
member = "serviceAccount:${var.reader_service_account_email}"
}
resource "google_project_iam_member" "reader_runs_queries" {
project = var.project_id
role = "roles/bigquery.jobUser"
member = "serviceAccount:${var.reader_service_account_email}"
}
```

View File

@@ -0,0 +1,129 @@
# Eventarc
A trigger that delivers events to a Cloud Run service: one for messages on a Pub/Sub topic, one for objects finalised in a bucket.
- A trigger has its own service account. It needs `roles/eventarc.eventReceiver` on the project and `roles/run.invoker` on the service it calls.
- For a Pub/Sub trigger, name an existing topic under `transport.pubsub.topic`; left out, Eventarc creates a topic you do not manage.
- For a Cloud Storage trigger, `location` must be the bucket's location, and the Cloud Storage service agent needs `roles/pubsub.publisher` on the project before the trigger is created.
- `matching_criteria` attributes are exact. `type` is always present; a storage event also needs `bucket`.
- The destination service is named, with its `region`; `path` is optional.
- The service must not allow unauthenticated calls for this to be the only way in: grant `roles/run.invoker` to the trigger's service account only.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "service_name" {
type = string
description = "Name of the Cloud Run service that receives the events."
}
variable "topic_id" {
type = string
description = "Id of the Pub/Sub topic whose messages are delivered."
}
variable "bucket_name" {
type = string
description = "Bucket whose new objects are delivered. It must be in var.region."
}
data "google_storage_project_service_account" "this" {
project = var.project_id
}
resource "google_service_account" "trigger" {
project = var.project_id
account_id = "eventarc-trigger"
display_name = "Eventarc trigger"
}
resource "google_project_iam_member" "trigger_receives_events" {
project = var.project_id
role = "roles/eventarc.eventReceiver"
member = "serviceAccount:${google_service_account.trigger.email}"
}
resource "google_cloud_run_v2_service_iam_member" "trigger_invokes_service" {
project = var.project_id
location = var.region
name = var.service_name
role = "roles/run.invoker"
member = "serviceAccount:${google_service_account.trigger.email}"
}
resource "google_project_iam_member" "storage_publishes_events" {
project = var.project_id
role = "roles/pubsub.publisher"
member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"
}
resource "google_eventarc_trigger" "on_message" {
project = var.project_id
name = "on-message"
location = var.region
service_account = google_service_account.trigger.email
matching_criteria {
attribute = "type"
value = "google.cloud.pubsub.topic.v1.messagePublished"
}
transport {
pubsub {
topic = var.topic_id
}
}
destination {
cloud_run_service {
service = var.service_name
region = var.region
path = "/events/message"
}
}
depends_on = [
google_project_iam_member.trigger_receives_events,
google_cloud_run_v2_service_iam_member.trigger_invokes_service,
]
}
resource "google_eventarc_trigger" "on_object" {
project = var.project_id
name = "on-object"
location = var.region
service_account = google_service_account.trigger.email
matching_criteria {
attribute = "type"
value = "google.cloud.storage.object.v1.finalized"
}
matching_criteria {
attribute = "bucket"
value = var.bucket_name
}
destination {
cloud_run_service {
service = var.service_name
region = var.region
path = "/events/object"
}
}
depends_on = [
google_project_iam_member.trigger_receives_events,
google_project_iam_member.storage_publishes_events,
google_cloud_run_v2_service_iam_member.trigger_invokes_service,
]
}
```

View File

@@ -0,0 +1,74 @@
# Firestore
A Firestore database in Native mode, with point-in-time recovery, a scheduled backup, a composite index and the grant a workload needs.
- A project's first database is named `(default)`. Any other name creates an additional database, which clients must name when they connect.
- `location_id` is a region or a multi-region such as `eur3`, and cannot be changed afterwards.
- `type` is `FIRESTORE_NATIVE` or `DATASTORE_MODE`, and cannot be changed once the database holds data.
- `delete_protection_state` is the guard in the API; `deletion_policy = "DELETE"` is needed before Terraform will remove a database at all.
- `point_in_time_recovery_enablement` keeps seven days of versions. A backup schedule is separate and keeps whole backups for `retention`, in seconds with an `s`.
- A workload reads and writes with `roles/datastore.user`, granted on the project. There is no per-database role; restrict with an IAM condition on the database name.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "app_service_account_email" {
type = string
description = "Service account of the workload that uses the database."
}
resource "google_firestore_database" "app" {
project = var.project_id
name = "app"
location_id = var.region
type = "FIRESTORE_NATIVE"
concurrency_mode = "OPTIMISTIC"
point_in_time_recovery_enablement = "POINT_IN_TIME_RECOVERY_ENABLED"
delete_protection_state = "DELETE_PROTECTION_ENABLED"
deletion_policy = "ABANDON"
}
resource "google_firestore_backup_schedule" "daily" {
project = var.project_id
database = google_firestore_database.app.name
retention = "1209600s"
daily_recurrence {}
}
resource "google_firestore_index" "orders_by_customer" {
project = var.project_id
database = google_firestore_database.app.name
collection = "orders"
fields {
field_path = "customer_id"
order = "ASCENDING"
}
fields {
field_path = "created_at"
order = "DESCENDING"
}
}
resource "google_project_iam_member" "app_uses_firestore" {
project = var.project_id
role = "roles/datastore.user"
member = "serviceAccount:${var.app_service_account_email}"
condition {
title = "only-the-app-database"
expression = "resource.name == \"${google_firestore_database.app.id}\" || resource.name.startsWith(\"${google_firestore_database.app.id}/\")"
}
}
```

View File

@@ -0,0 +1,109 @@
# Google Kubernetes Engine
A private Autopilot cluster on an existing subnet, with its node service account and Workload Identity for one workload.
- `enable_autopilot = true` makes Google manage the nodes. Node pools, machine types and node counts are then not yours to set: leave out `node_pool`, `node_config` and `initial_node_count`.
- A regional `location` gives a control plane in three zones. A zone gives one.
- The subnet needs two secondary ranges, for Pods and Services, named in `ip_allocation_policy`.
- `enable_private_nodes = true` gives nodes no public address; they then need Cloud NAT to pull images from outside Google.
- `master_authorized_networks_config` limits who can reach the control plane. With `enable_private_endpoint = true` it has no public address at all.
- Give nodes a dedicated service account with `roles/container.defaultNodeServiceAccount`, not the Compute Engine default account.
- Autopilot has Workload Identity on. A Kubernetes service account acts as a Google service account through `roles/iam.workloadIdentityUser`, with the member `serviceAccount:PROJECT.svc.id.goog[NAMESPACE/KSA]`.
- `deletion_protection` is true unless set; take it from a variable.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "network_id" {
type = string
description = "Id of the VPC the cluster is in."
}
variable "subnetwork_id" {
type = string
description = "Id of the subnet for nodes. It has secondary ranges named pods and services."
}
variable "admin_cidr" {
type = string
description = "Range allowed to reach the control plane."
}
variable "deletion_protection" {
type = bool
description = "Whether the cluster is protected from deletion."
default = true
}
resource "google_service_account" "nodes" {
project = var.project_id
account_id = "gke-nodes"
display_name = "GKE nodes"
}
resource "google_project_iam_member" "nodes_default_role" {
project = var.project_id
role = "roles/container.defaultNodeServiceAccount"
member = "serviceAccount:${google_service_account.nodes.email}"
}
resource "google_container_cluster" "workload" {
project = var.project_id
name = "workload"
location = var.region
enable_autopilot = true
deletion_protection = var.deletion_protection
network = var.network_id
subnetwork = var.subnetwork_id
ip_allocation_policy {
cluster_secondary_range_name = "pods"
services_secondary_range_name = "services"
}
private_cluster_config {
enable_private_nodes = true
enable_private_endpoint = false
}
master_authorized_networks_config {
cidr_blocks {
cidr_block = var.admin_cidr
display_name = "administrators"
}
}
cluster_autoscaling {
auto_provisioning_defaults {
service_account = google_service_account.nodes.email
}
}
release_channel {
channel = "REGULAR"
}
depends_on = [google_project_iam_member.nodes_default_role]
}
resource "google_service_account" "app" {
project = var.project_id
account_id = "orders-app"
display_name = "Orders application"
}
resource "google_service_account_iam_member" "app_workload_identity" {
service_account_id = google_service_account.app.name
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project_id}.svc.id.goog[orders/orders-app]"
}
```

View File

@@ -0,0 +1,9 @@
{
"google-cloud-solution-architecture": "f5b8cdac4060",
"google-cloud-waf-cost-optimization": "b3febbf52e5c",
"google-cloud-waf-operational-excellence": "bdc6a9f415f6",
"google-cloud-waf-performance-optimization": "89a4947438cf",
"google-cloud-waf-reliability": "9baf33bd5b75",
"google-cloud-waf-security": "e791a44cda99",
"google-cloud-waf-sustainability": "7d0a6921cb59"
}

View File

@@ -75,7 +75,7 @@ def complete_run_turns() -> dict:
def client_for(
tmp_path: Path, turns: dict, decider=None
tmp_path: Path, turns: dict, decider=None, answers=None
) -> tuple[TestClient, FakeAgentRuntime]:
bundle = load_provider("gcp")
roles = load_roles(AGENTS_DIR, bundle)
@@ -96,6 +96,7 @@ def client_for(
roles,
{"compute": "API not enabled"},
intake=Intake(decider, tmp_path) if decider else None,
answers=answers,
)
return TestClient(create_app(get_settings(), fake_assembly)), runtime
@@ -398,6 +399,74 @@ def delivered_case(tmp_path: Path, case: str = "acme") -> None:
)
class Answers:
"""Stands in for the reply writer answering from delivered files."""
def __init__(self, reply):
self.reply = reply
self.asked: list[tuple[str, str, str]] = []
async def answer(self, question, customer_case, execution_id):
self.asked.append((question, customer_case, execution_id))
return self.reply
ABOUT_DELIVERED = {
"intake": Answer("question", 0.98),
"about_delivered": Answer(0.92, 0.9),
}
def test_a_question_about_delivered_work_is_answered_from_it(tmp_path):
delivered_case(tmp_path)
answers = Answers("Cloud SQL, because bookings are relational.")
client, runtime = client_for(tmp_path, {}, FakeDecider(ABOUT_DELIVERED), answers)
with client:
task = send(
client,
[{"text": "Why did you choose Cloud SQL?"}],
metadata={"customer_case": "acme"},
)
text, data = status_parts(task)
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
assert text == "Cloud SQL, because bookings are relational."
assert data == {"status": "answered", "intake": "question", "about": "v1"}
assert answers.asked == [("Why did you choose Cloud SQL?", "acme", "v1")]
assert runtime.runs == []
def test_a_question_that_could_not_be_answered_says_where_to_look(tmp_path):
delivered_case(tmp_path)
client, _ = client_for(tmp_path, {}, FakeDecider(ABOUT_DELIVERED), Answers(None))
with client:
task = send(
client, [{"text": "Why Cloud SQL?"}], metadata={"customer_case": "acme"}
)
text, data = status_parts(task)
assert "could not answer that from the documents delivered in Execution v1" in text
assert data["about"] == "v1"
def test_a_question_about_the_service_is_still_answered_in_general(tmp_path):
delivered_case(tmp_path)
decider = FakeDecider(
{"intake": Answer("question", 0.98), "about_delivered": Answer(0.08, 0.9)}
)
answers = Answers("unused")
client, _ = client_for(tmp_path, {}, decider, answers)
with client:
task = send(
client, [{"text": "What can you do?"}], metadata={"customer_case": "acme"}
)
text, data = status_parts(task)
assert text.startswith("I design cloud solution architectures.")
assert data == {"status": "answered", "intake": "question"}
assert answers.asked == []
def test_a_change_to_delivered_work_names_the_baseline_and_starts_nothing(tmp_path):
delivered_case(tmp_path)
decider = FakeDecider({"intake": Answer("change_request", 0.98)})

View File

@@ -62,3 +62,100 @@ async def test_a_change_needs_something_delivered_to_change(tmp_path):
assert "- Design thing 1." in decider.asked[0][0]
assert "Design thing 2." not in decider.asked[0][0] # still in progress
assert without.kind == NEW_REQUEST and no_case.kind == NEW_REQUEST
# --- a question about what was delivered ------------------------------------------------
async def test_a_question_about_delivered_work_is_told_from_one_about_the_service(
tmp_path,
):
delivered(tmp_path, "acme", ["completed"])
about = FakeDecider(
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.9, 0.9)}
)
general = FakeDecider(
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.1, 0.9)}
)
first = await Intake(about, tmp_path).read("Why Cloud SQL?", "acme")
second = await Intake(general, tmp_path).read("What can you do?", "acme")
assert first.kind == QUESTION and first.about_delivered and first.latest == "v1"
assert second.kind == QUESTION and not second.about_delivered
async def test_with_nothing_delivered_a_question_is_never_about_delivered_work(
tmp_path,
):
decider = FakeDecider(
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.99, 0.9)}
)
reading = await Intake(decider, tmp_path).read("Why Cloud SQL?", "new-client")
assert not reading.about_delivered
assert len(decider.asked) == 1 # the second question was not asked
async def test_an_answer_is_written_from_the_delivered_guide_and_nothing_else(
tmp_path,
):
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
from app.adk.runtime import RoleSpec
from app.agent import ExecutionEngine
from app.nodes.intake import DeliveredAnswers
from app.nodes.router import Router, Thresholds
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH
roles = {
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
**{
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
for spec in DEPENDENCY_GRAPH.values()
},
}
seen: dict = {}
def writer(task):
seen.update(task.context)
return [Call("write_reply", {"text": "Cloud SQL, for relational bookings."})]
runtime = FakeAgentRuntime(
{
**{
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
},
"reply-writer": [writer],
}
)
engine = ExecutionEngine(
runtime=runtime,
roles=roles,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
)
await engine.start(
"Design bookings.", customer_case="acme", pre_approved=True, execution_id="v1"
)
runs_before = len(runtime.runs)
answers = DeliveredAnswers(
runtime, RoleSpec("reply-writer", "Write.", budget=3), tmp_path
)
answer = await answers.answer("Why Cloud SQL?", "acme", "v1")
assert answer == "Cloud SQL, for relational bookings."
assert seen["question"] == "Why Cloud SQL?"
assert seen["delivered"]["execution_id"] == "v1"
assert seen["delivered"]["request"] == "Design bookings."
assert seen["delivered"]["solution_guide"]
assert len(runtime.runs) == runs_before + 1 # one writer run; no Execution
# An Execution that is not there gives no answer, not an error.
assert await answers.answer("Why?", "acme", "missing") is None

View File

@@ -0,0 +1,100 @@
"""An Execution's files are on disk as soon as each Deliverable is submitted.
Before this they were written only when an Execution ended, so a process that was
killed, or an Execution left waiting for its user, had nothing to show.
"""
import json
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import FakeAgentRuntime, sample_submission
from app.adk.runtime import NoDecider, RoleSpec
from app.agent import ExecutionEngine
from app.baseline import BaselineError, load_baseline
from app.nodes.execution_nodes import _write_outputs
from app.nodes.router import Router, Thresholds
from app.recorder import InMemoryRecorder
from app.states.ledger import Ledger
from app.workflows.deliverables import DEPENDENCY_GRAPH
ROLES = {
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
**{
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
for spec in DEPENDENCY_GRAPH.values()
},
}
def folder(tmp_path, execution_id: str):
return tmp_path / "deliverables" / "cases" / "acme" / "executions" / execution_id
async def test_an_execution_waiting_for_its_user_has_written_what_exists(tmp_path):
engine = ExecutionEngine(
runtime=FakeAgentRuntime(
{
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
}
),
roles=ROLES,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
)
waiting = await engine.start("Ingest", customer_case="acme", execution_id="v1")
# It stopped for the first Approval: only the first Phase has been produced.
assert waiting.status == "awaiting_user"
out = folder(tmp_path, "v1")
manifest = json.loads((out / "execution.json").read_text())
assert manifest["status"] == "in_progress"
assert set(manifest["deliverables"]) == {"requirements", "technical_decomposition"}
assert (out / "deliverables" / "requirements.md").read_text()
assert (out / "requirements.json").is_file()
assert manifest["solution_guide"] is None
# Unfinished work is never something a Change request can start from.
try:
load_baseline(tmp_path, "acme", "v1")
except BaselineError as refused:
assert "did not complete (in_progress)" in str(refused)
else:
raise AssertionError("an in-progress Execution was accepted as a Baseline")
def test_a_file_an_earlier_write_left_is_removed_with_the_revision(tmp_path):
state = {
"execution_id": "v1",
"customer_case": "acme",
"workflow_request": "Ingest",
"deliverables": {
"deployment_guidance": {
"content": "guide",
"attachments": {
"terraform/main.tf": "one",
"terraform/queue.tf": "two",
},
}
},
}
out = folder(tmp_path, "v1")
_write_outputs(tmp_path, state, "in_progress", Ledger())
assert (out / "terraform" / "queue.tf").is_file()
(out / "notes.txt").write_text("a person's own file")
# The Revision folded the queue into main.tf.
state["deliverables"]["deployment_guidance"]["attachments"] = {
"terraform/main.tf": "one and two"
}
_write_outputs(tmp_path, state, "completed", Ledger())
assert not (out / "terraform" / "queue.tf").exists()
assert (out / "terraform" / "main.tf").read_text() == "one and two"
# Only files this Execution wrote are ever removed.
assert (out / "notes.txt").is_file()
assert json.loads((out / "execution.json").read_text())["status"] == "completed"

View File

@@ -498,16 +498,68 @@ def test_lessons_that_cost_quality_or_save_nothing_are_not_shown_to_help(tmp_pat
assert "nothing fell" in render(compare(base, same), base, same)
def test_the_comparison_is_written_into_published_proposals_only(tmp_path):
def test_the_comparison_is_written_only_for_lessons_in_the_skill(tmp_path):
from app.learning.proof import record
skill = lessons_skill(tmp_path)
issues = tmp_path / "issues"
live = publish(approve(proposal(issues), issues), skill, today=DAY)
waiting = proposal(issues, title="Another", evidence_key="zzz")
gone = publish(
approve(proposal(issues, title="Retired", evidence_key="yyy"), issues),
skill,
today=DAY,
)
retire(gone.id, skill)
recorded = record([live, waiting], "| a table |")
recorded = record([live, waiting, gone], "| a table |", published(skill))
assert [p.id for p in recorded] == ["L-01"]
assert parse(live.path).proof == "| a table |"
assert parse(waiting.path).proof == ""
# A retired lesson was not part of what was compared.
assert parse(gone.path).proof == ""
# --- lessons are read again when the Skills beside them change ---------------------------
def test_a_changed_upstream_skill_is_flagged_until_the_lessons_are_reviewed(tmp_path):
from app.learning.skill import mark_reviewed, upstream_changes
skill = lessons_skill(tmp_path)
issues = tmp_path / "issues"
publish(approve(proposal(issues), issues), skill, today=DAY)
then = {"architecture": "aaa", "security": "bbb"}
mark_reviewed(skill, then)
assert upstream_changes(skill, then) == []
now = {"architecture": "aaa", "security": "ccc", "cost": "ddd"}
assert upstream_changes(skill, now) == ["cost", "security"]
mark_reviewed(skill, now)
assert upstream_changes(skill, now) == []
def test_nothing_is_flagged_with_no_lessons_or_nothing_recorded(tmp_path):
from app.learning.skill import mark_reviewed, upstream_changes
skill = lessons_skill(tmp_path)
assert upstream_changes(skill, {"architecture": "aaa"}) == []
mark_reviewed(skill, {"architecture": "aaa"})
# No lesson is published, so there is nothing to review.
assert upstream_changes(skill, {"architecture": "zzz"}) == []
def test_the_published_lessons_were_reviewed_against_todays_vendored_skills():
"""Syncing the vendored Skills changes what the lessons sit beside. This fails
until a person has read each lesson again and run `curate_lessons reviewed`."""
from app.learning.skill import upstream_changes, upstream_reviewed
from app.providers import load_provider
from eval.curate_lessons import upstream_versions
bundle = load_provider("gcp")
skill = bundle.skills[bundle.lessons_skill].path
assert upstream_reviewed(skill), "run: python -m eval.curate_lessons reviewed"
assert upstream_changes(skill, upstream_versions(bundle)) == []