chore: housekeeping from the runs of 2026-10-09
Some checks failed
validation / verify (push) Failing after 12s
Some checks failed
validation / verify (push) Failing after 12s
- An Execution's files are written as each Deliverable is submitted, marked in_progress, so a killed process or one waiting for its user leaves them on disk. A write removes files an earlier write left that it no longer has. - A question about delivered work is answered from that Execution's solution guide by the reply writer; a question about the service is answered as before. Nothing is started for either. - Lessons: prove writes only for lessons the Skill holds; the Skill records the vendored Skill versions the lessons were read against, status names any that changed, and a test fails until they are reviewed. - Four more code patterns: Firestore, BigQuery, Eventarc and GKE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -44,6 +44,7 @@
|
||||
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
|
||||
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
|
||||
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
|
||||
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
@@ -73,3 +73,11 @@ Case 03, one run each way, side by side, with the five lessons that remain (`202
|
||||
`prove` recorded this in the proposals as "kept". One pair suggests a direction and does not settle it: a third run the same evening, with lessons, did need a Revision of the Target architecture. `prove` also wrote to L-04's proposal, which is retired; it should skip retired lessons.
|
||||
|
||||
Neither run raised the schema review's false alarm, which had been raised in case 06 before the fix.
|
||||
|
||||
## Housekeeping, 2026-10-10
|
||||
|
||||
- **`prove` skips retired lessons.** It wrote its comparison into L-04's proposal after L-04 had been retired. It now writes only for lessons the Skill holds. L-04's proposal is corrected.
|
||||
- **Lessons are flagged when the vendored Skills change.** The lessons Skill keeps `upstream.json`: the version of each vendored Skill the lessons were last read against. `status` names the ones that have changed since and says to read each lesson again; `reviewed` records that a person has. A test fails until then, so a sync of the vendored Skills cannot go unnoticed. The first publish records the versions; after that only `reviewed` moves them.
|
||||
- **`status` says "compared", not "proved",** and shows a retired lesson as retired. One pair of runs is a comparison.
|
||||
|
||||
Tests: `tests/test_lessons_loop.py`.
|
||||
|
||||
21
.agents/plans/iter-04-learning-and-skills/steps/step-28.md
Normal file
21
.agents/plans/iter-04-learning-and-skills/steps/step-28.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Step 28 — Housekeeping (2026-10-10)
|
||||
|
||||
Small things found during the runs of 2026-10-09. Not run with real models unless said.
|
||||
|
||||
## Files are written as an Execution goes
|
||||
|
||||
An Execution's folder was written only when it ended, so a killed process, or an Execution left waiting for its user, left nothing on disk.
|
||||
|
||||
- Each time a Deliverable is submitted, the folder is written with what exists, and `execution.json` says `in_progress`. The end of the Execution writes it again with its final status.
|
||||
- An in-progress Execution is refused as a Baseline, as any Execution that did not complete is.
|
||||
- Because the folder is now written more than once, a write removes the files an earlier write recorded and this one does not have. Without that, a Terraform file removed in a Revision would have stayed beside the files that replaced it. Only files the Execution itself wrote are removed.
|
||||
|
||||
Tests: `tests/execution/test_outputs_as_it_goes.py` (2).
|
||||
|
||||
## Four more code patterns
|
||||
|
||||
`skills/google-cloud-iac-patterns/` gains Firestore, BigQuery, Eventarc and Google Kubernetes Engine, 18 in all. Each validates against `hashicorp/google` 8.6.0 in `tests/test_iac_patterns.py`.
|
||||
|
||||
## The two explainer pages
|
||||
|
||||
Both now describe the opt-in to published Terraform modules.
|
||||
@@ -76,3 +76,14 @@ Not built: acting on a change request (iteration 7 step 2), and answering a ques
|
||||
|
||||
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).
|
||||
|
||||
## A question about delivered work is answered from it (2026-10-10)
|
||||
|
||||
Intake answered every question with a description of the service. A question read as about work already delivered in the caller's Customer case is now answered from that Execution's solution guide.
|
||||
|
||||
1. When intake reads a message as a question and the case has a completed Execution, a second yes or no is asked: is this about an architecture already delivered, or about the service in general. At 0.6 or above it is about delivered work.
|
||||
2. `DeliveredAnswers` in `app/nodes/intake.py` loads the latest completed Execution and gives its solution guide and the question to the reply writer. `agents/reply-writer.md` now covers this second task: answer from the guide alone, say so when it does not say, name the section, propose no change.
|
||||
3. The A2A reply carries `about`, the Execution it was answered from. Nothing is started. When no answer could be written the caller is told which Execution's documents to read.
|
||||
|
||||
Measured against live Jev on twelve messages: six about delivered work scored 0.37 to 0.96, five of them 0.73 or more ("what does the load balancer in my architecture do?" scored 0.37 and would get the general answer). Five about the service scored 0.07 to 0.16. "Is my data stored anywhere?" scored 0.74: it is ambiguous, and would get an answer from the guide. **No real model has written such an answer yet.**
|
||||
|
||||
Tests: `tests/execution/test_intake.py` (3 more), `tests/a2a/test_server.py` (3 more).
|
||||
|
||||
@@ -20,15 +20,7 @@ Seen in 2 of 21 Executions, 4 times in all: submit_deliverable refused: Not subm
|
||||
|
||||
## Proof
|
||||
|
||||
Evaluation without lessons: `20261009T195603Z`; with: `20261009T195607Z`.
|
||||
|
||||
| Case | Score | Revisions | Minutes | Tokens (M) |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| case-03-internal-web-app | 1.00 → 1.00 | 1 → 0 | 28 → 22 | 3.00 → 2.32 |
|
||||
|
||||
Quality held and revisions, seconds, tokens fell: kept.
|
||||
|
||||
This compares every published lesson together, on one run each way. Runs of the same case vary, so one pair suggests a direction and does not settle it.
|
||||
Not yet measured.
|
||||
|
||||
## Comments
|
||||
|
||||
|
||||
@@ -2,9 +2,11 @@
|
||||
role: reply-writer
|
||||
tools: []
|
||||
---
|
||||
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. The decision has already been made; you put it into words and decide nothing.
|
||||
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. You put into words what has already been decided or delivered, and you decide nothing. Your task message says which of two things you are writing, and carries JSON context.
|
||||
|
||||
Your task message carries JSON context: `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
|
||||
## A summary to approve a Phase from
|
||||
|
||||
The context has `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
|
||||
|
||||
Write a summary a busy architect can approve from:
|
||||
|
||||
@@ -13,4 +15,14 @@ Write a summary a busy architect can approve from:
|
||||
- Name anything they should look at before approving: failed or unverifiable Checks, open gaps, recommendations treated as not applicable, Deliverables that were revised.
|
||||
- Use plain sentences and no headings. Keep it under 120 words.
|
||||
|
||||
Call `write_reply` with the summary. You are done when it accepts it.
|
||||
## An answer about delivered work
|
||||
|
||||
The context has `question`, what the user asked, and `delivered`: the `request` the architecture was designed for and its `solution_guide`.
|
||||
|
||||
- Answer from the solution guide alone. Give the choice or the fact, and the reason the guide gives for it.
|
||||
- When the guide does not say, say that it does not. Add nothing from general knowledge about the products, and do not guess.
|
||||
- Name the section of the guide the answer comes from.
|
||||
- Propose no change. If the user seems to want one, say they can ask for it as a change to this architecture.
|
||||
- Use plain sentences and no headings. Keep it under 150 words.
|
||||
|
||||
Call `write_reply` with the text. You are done when it accepts it.
|
||||
|
||||
@@ -31,7 +31,15 @@ from google.protobuf import json_format
|
||||
|
||||
from app.agent import ExecutionEngine, Interaction
|
||||
from app.nodes.execution_nodes import GUIDE_FILENAME
|
||||
from app.nodes.intake import ABOUT, CHANGE_REQUEST, QUESTION, Intake, change_note
|
||||
from app.nodes.intake import (
|
||||
ABOUT,
|
||||
CHANGE_REQUEST,
|
||||
QUESTION,
|
||||
DeliveredAnswers,
|
||||
Intake,
|
||||
change_note,
|
||||
no_answer_note,
|
||||
)
|
||||
from app.presentation import describe_prompt
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
@@ -41,7 +49,13 @@ STRUCTURED_REPLY_KEYS = ("approval", "permission")
|
||||
|
||||
|
||||
class SolutionArchitectExecutor(AgentExecutor):
|
||||
def __init__(self, engine: ExecutionEngine, intake: Intake | None = None) -> None:
|
||||
def __init__(
|
||||
self,
|
||||
engine: ExecutionEngine,
|
||||
intake: Intake | None = None,
|
||||
answers: DeliveredAnswers | None = None,
|
||||
) -> None:
|
||||
self._answers = answers
|
||||
self._engine = engine
|
||||
self._intake = intake
|
||||
|
||||
@@ -75,14 +89,21 @@ class SolutionArchitectExecutor(AgentExecutor):
|
||||
scope=task_id,
|
||||
)
|
||||
if reading.kind == QUESTION:
|
||||
text, about = ABOUT, None
|
||||
if reading.about_delivered and reading.latest and self._answers:
|
||||
# About what was delivered: answered from its files.
|
||||
about = reading.latest
|
||||
text = await self._answers.answer(
|
||||
context.get_user_input(),
|
||||
str(options.get("customer_case")),
|
||||
about,
|
||||
) or no_answer_note(about)
|
||||
data = {"status": "answered", "intake": QUESTION}
|
||||
if about:
|
||||
data["about"] = about
|
||||
await updater.complete(
|
||||
updater.new_agent_message(
|
||||
[
|
||||
new_text_part(ABOUT),
|
||||
new_data_part(
|
||||
{"status": "answered", "intake": QUESTION}
|
||||
),
|
||||
]
|
||||
[new_text_part(text), new_data_part(data)]
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
@@ -36,7 +36,7 @@ from app.baseline import load_baseline
|
||||
from app.checkpointing import open_checkpointer
|
||||
from app.config import Settings
|
||||
from app.customer_case import record_intent
|
||||
from app.nodes.intake import Intake
|
||||
from app.nodes.intake import DeliveredAnswers, Intake
|
||||
from app.nodes.router import (
|
||||
REPLY_WRITER,
|
||||
OrchestratorPolicy,
|
||||
@@ -328,6 +328,8 @@ class Assembly:
|
||||
mcp_gaps: dict[str, str]
|
||||
decider: Decider = field(default_factory=NoDecider)
|
||||
intake: Intake | None = None # reads a message that starts something
|
||||
# Answers a question about delivered work from its files.
|
||||
answers: DeliveredAnswers | None = None
|
||||
|
||||
|
||||
def discover_overlays(settings: Settings, bundle: ProviderBundle) -> dict[str, Any]:
|
||||
@@ -529,6 +531,9 @@ async def open_assembly(
|
||||
output_root or settings.BASE_DIR,
|
||||
act=settings.JEV_ACT_THRESHOLD,
|
||||
),
|
||||
DeliveredAnswers(
|
||||
runtime, roles[REPLY_WRITER], output_root or settings.BASE_DIR
|
||||
),
|
||||
)
|
||||
finally:
|
||||
if isinstance(decider, JevDecider):
|
||||
|
||||
@@ -7,7 +7,7 @@ are ordinary evaluation reports; one is made with LESSONS_ENABLED=false.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping, Sequence
|
||||
from collections.abc import Collection, Mapping, Sequence
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -106,6 +106,12 @@ def render(result: Mapping[str, Any], without: Path, with_lessons: Path) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def record(proposals: Sequence[Proposal], proof: str) -> list[Proposal]:
|
||||
"""Write the comparison into every published proposal."""
|
||||
return [write(replace(p, proof=proof)) for p in proposals if p.published]
|
||||
def record(
|
||||
proposals: Sequence[Proposal], proof: str, live: Collection[str]
|
||||
) -> list[Proposal]:
|
||||
"""Write the comparison into the proposal of every lesson in the Skill.
|
||||
|
||||
`live` is the ids the Skill holds now. A retired lesson was not part of what
|
||||
was compared, so its proposal is left as it was.
|
||||
"""
|
||||
return [write(replace(p, proof=proof)) for p in proposals if p.id in live]
|
||||
|
||||
@@ -8,7 +8,9 @@ the index in SKILL.md are written here, from approved proposals, and never by ha
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import replace
|
||||
from datetime import UTC, date, datetime
|
||||
from pathlib import Path
|
||||
@@ -130,6 +132,45 @@ def publish(
|
||||
return write(replace(proposal, published=proposal.published or stamp))
|
||||
|
||||
|
||||
# The versions of the Skills the lessons were written beside, kept in the lessons
|
||||
# Skill. A lesson is a correction to how agents follow those Skills, so when one
|
||||
# of them changes each lesson is worth a second look: it may now be said upstream,
|
||||
# or contradict it.
|
||||
UPSTREAM_FILE = "upstream.json"
|
||||
|
||||
|
||||
def upstream_reviewed(skill_dir: Path) -> dict[str, str]:
|
||||
"""The upstream Skill versions the lessons were last reviewed against."""
|
||||
path = skill_dir / UPSTREAM_FILE
|
||||
if not path.is_file():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except ValueError:
|
||||
return {}
|
||||
return {str(k): str(v) for k, v in data.items()} if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def mark_reviewed(skill_dir: Path, current: Mapping[str, str]) -> None:
|
||||
"""Record that the lessons stand against these upstream Skill versions."""
|
||||
(skill_dir / UPSTREAM_FILE).write_text(
|
||||
json.dumps(dict(sorted(current.items())), indent=2) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def upstream_changes(skill_dir: Path, current: Mapping[str, str]) -> list[str]:
|
||||
"""The upstream Skills that changed, appeared or went since the lessons were
|
||||
last reviewed. Empty when there are no lessons, or nothing was ever recorded."""
|
||||
reviewed = upstream_reviewed(skill_dir)
|
||||
if not reviewed or not published(skill_dir):
|
||||
return []
|
||||
return sorted(
|
||||
name
|
||||
for name in set(reviewed) | set(current)
|
||||
if reviewed.get(name) != current.get(name)
|
||||
)
|
||||
|
||||
|
||||
def retire(lesson_id: str, skill_dir: Path) -> bool:
|
||||
"""Remove a lesson from the Skill. Returns whether it was there."""
|
||||
deliverable = published(skill_dir).get(lesson_id)
|
||||
|
||||
@@ -89,7 +89,7 @@ def create_app(
|
||||
async with assembly_factory(settings) as assembly:
|
||||
handler = DefaultRequestHandlerV2(
|
||||
agent_executor=SolutionArchitectExecutor(
|
||||
assembly.engine, assembly.intake
|
||||
assembly.engine, assembly.intake, assembly.answers
|
||||
),
|
||||
task_store=InMemoryTaskStore(),
|
||||
agent_card=card,
|
||||
|
||||
@@ -15,7 +15,7 @@ import time
|
||||
from collections.abc import Awaitable, Callable, Mapping, MutableMapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, cast
|
||||
|
||||
from langgraph.types import Command, Send, interrupt
|
||||
|
||||
@@ -37,7 +37,7 @@ from app.adk.runtime import (
|
||||
TurnEnded,
|
||||
)
|
||||
from app.baseline import WORKING_FILENAME
|
||||
from app.customer_case import close_intent
|
||||
from app.customer_case import IN_PROGRESS, close_intent
|
||||
from app.nodes.router import ModelOrchestrator, OrchestratorPolicy, TurnView
|
||||
from app.recorder import ExecutionRecorder
|
||||
from app.states.execution_state import (
|
||||
@@ -207,11 +207,13 @@ def make_execution_nodes(
|
||||
drafts = dict(state.get("drafts", {}))
|
||||
|
||||
outcomes = state.get("outcomes", [])
|
||||
arrived = False
|
||||
for record in outcomes[state.get("outcomes_applied", 0) :]:
|
||||
kind = DeliverableKind(record["kind"])
|
||||
try:
|
||||
if record["type"] == "submitted":
|
||||
ledger.submit(kind, record["attempt"], actor=record["role"])
|
||||
arrived = True
|
||||
deliverables[kind.value] = {
|
||||
"content": record["content"],
|
||||
"attachments": record["attachments"],
|
||||
@@ -252,6 +254,19 @@ def make_execution_nodes(
|
||||
]
|
||||
pending_questions.append({"kind": kind.value, "questions": questions})
|
||||
|
||||
if arrived:
|
||||
# What has been submitted is on disk as soon as it exists, marked as in
|
||||
# progress. A process that is killed leaves that much behind.
|
||||
try:
|
||||
_write_outputs(
|
||||
output_root,
|
||||
cast(ExecutionState, {**state, "deliverables": deliverables}),
|
||||
IN_PROGRESS,
|
||||
ledger,
|
||||
)
|
||||
except OSError:
|
||||
logger.exception("Writing the Execution's files so far failed")
|
||||
|
||||
turns = state.get("turns", 0) + 1
|
||||
update: dict[str, Any] = {
|
||||
"deliverables": deliverables,
|
||||
@@ -1214,6 +1229,26 @@ def _case(state: ExecutionState) -> str:
|
||||
return state.get("customer_case") or UNASSIGNED_CASE
|
||||
|
||||
|
||||
def _written_before(out_dir: Path) -> set[str]:
|
||||
"""The files an earlier write of this Execution's folder recorded."""
|
||||
try:
|
||||
manifest = json.loads((out_dir / MANIFEST_FILENAME).read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return set()
|
||||
if not isinstance(manifest, dict):
|
||||
return set()
|
||||
names = {manifest.get("solution_guide"), manifest.get("changes")}
|
||||
for entry in (manifest.get("deliverables") or {}).values():
|
||||
if isinstance(entry, dict):
|
||||
names.add(entry.get("document"))
|
||||
names.update(entry.get("attachments") or [])
|
||||
return {
|
||||
name
|
||||
for name in names
|
||||
if isinstance(name, str) and name and ".." not in Path(name).parts
|
||||
}
|
||||
|
||||
|
||||
def _write_outputs(
|
||||
output_root: Path,
|
||||
state: ExecutionState,
|
||||
@@ -1267,6 +1302,11 @@ def _write_outputs(
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
# The folder is written more than once as an Execution goes. A file an earlier
|
||||
# write left that this one does not have, such as a Terraform file removed in a
|
||||
# Revision, must not stay beside the files that replaced it.
|
||||
for stale in _written_before(out_dir) - set(files):
|
||||
(out_dir / stale).unlink(missing_ok=True)
|
||||
for name, content in files.items():
|
||||
path = out_dir / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -6,6 +6,7 @@ one step earlier, for a message that arrives with no Execution in progress:
|
||||
new request design this workload start an Execution
|
||||
change request change what was delivered name the Baseline to change (ADR-0006)
|
||||
question what can you do, how long... answer; start nothing
|
||||
why did you choose X? answer from what was delivered
|
||||
|
||||
A change is never started from earlier work on a guess: the caller chooses the
|
||||
Baseline. So a message read as a change is answered with the Execution it would
|
||||
@@ -24,7 +25,17 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
from app.adk.runtime import (
|
||||
AgentRuntime,
|
||||
AgentTask,
|
||||
Decider,
|
||||
Question,
|
||||
RoleSpec,
|
||||
Submitted,
|
||||
ToolResult,
|
||||
ToolSpec,
|
||||
)
|
||||
from app.baseline import BaselineError, load_baseline
|
||||
from app.customer_case import IN_PROGRESS, read_history
|
||||
from app.states.execution_state import UNASSIGNED_CASE
|
||||
|
||||
@@ -53,7 +64,18 @@ def change_note(baseline: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
def no_answer_note(execution_id: str) -> str:
|
||||
"""What a caller is told when a question about delivered work went unanswered."""
|
||||
return (
|
||||
"I could not answer that from the documents delivered in Execution "
|
||||
f"{execution_id}. Its solution guide and Deliverables hold what was "
|
||||
"decided and why. Nothing was started for this message."
|
||||
)
|
||||
|
||||
|
||||
_MESSAGE_LIMIT = 8_000
|
||||
# How much of the delivered guide the answer is written from.
|
||||
_GUIDE_LIMIT = 80_000
|
||||
_EARLIER_SHOWN = 5
|
||||
|
||||
|
||||
@@ -64,6 +86,8 @@ class IntakeReading:
|
||||
judged: bool = False # False when no decision was available
|
||||
earlier: int = 0 # finished Executions of the same Customer case
|
||||
latest: str | None = None # the most recent of them: what a change would change
|
||||
# A question about what was delivered, as opposed to about the service.
|
||||
about_delivered: bool = False
|
||||
|
||||
|
||||
def finished_intents(output_root: Path, customer_case: str) -> list[Mapping[str, Any]]:
|
||||
@@ -85,11 +109,18 @@ class Intake:
|
||||
output_root: Path,
|
||||
*,
|
||||
act: float = 0.75,
|
||||
about_delivered_at: float = 0.6,
|
||||
) -> None:
|
||||
self._decider = decider
|
||||
self._output_root = output_root
|
||||
# How sure the reading must be before anything but a new Execution is done.
|
||||
self._act = act
|
||||
# How likely a question must be about delivered work to be answered from
|
||||
# it. Measured on 2026-10-10: six such questions scored 0.37 to 0.96, five
|
||||
# of them 0.73 or more; six about the service scored 0.07 to 0.16 and one
|
||||
# ambiguous one ("is my data stored anywhere?") 0.74. A wrong yes costs an
|
||||
# answer that says the documents do not cover it.
|
||||
self._about_delivered_at = about_delivered_at
|
||||
|
||||
async def read(
|
||||
self, message: str, customer_case: str | None = None, scope: str = ""
|
||||
@@ -123,9 +154,38 @@ class Intake:
|
||||
kind = NEW_REQUEST # there is nothing delivered to change
|
||||
logger.info("[%s] intake: %s (%.2f)", scope, kind, confidence)
|
||||
latest = str(earlier[-1].get("execution_id")) if earlier else None
|
||||
return IntakeReading(
|
||||
kind, confidence, judged=True, earlier=len(earlier), latest=latest
|
||||
about = (
|
||||
await self._about_delivered(text, earlier, scope)
|
||||
if kind == QUESTION and latest
|
||||
else False
|
||||
)
|
||||
return IntakeReading(
|
||||
kind,
|
||||
confidence,
|
||||
judged=True,
|
||||
earlier=len(earlier),
|
||||
latest=latest,
|
||||
about_delivered=about,
|
||||
)
|
||||
|
||||
async def _about_delivered(
|
||||
self, text: str, earlier: Sequence[Mapping[str, Any]], scope: str
|
||||
) -> bool:
|
||||
answers = await self._decider.decide(
|
||||
_state(text, earlier),
|
||||
{
|
||||
"about_delivered": Question(
|
||||
"noul",
|
||||
"Is the user asking about an architecture already delivered to "
|
||||
"them: what it contains, why something was chosen, or what "
|
||||
"validation found? Answer no when they ask what this service "
|
||||
"does or how it works in general.",
|
||||
)
|
||||
},
|
||||
scope=scope,
|
||||
)
|
||||
answer = (answers or {}).get("about_delivered")
|
||||
return answer is not None and float(answer.value) >= self._about_delivered_at
|
||||
|
||||
|
||||
def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
|
||||
@@ -135,3 +195,65 @@ def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
|
||||
lines.append(f"- {request}")
|
||||
lines.append(f"The user's message: {message[:_MESSAGE_LIMIT]}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class DeliveredAnswers:
|
||||
"""Answers a question about delivered work from its documents, on the reply
|
||||
writer. It reads one Execution's files and starts nothing."""
|
||||
|
||||
def __init__(
|
||||
self, runtime: AgentRuntime, role: RoleSpec, output_root: Path
|
||||
) -> None:
|
||||
self._runtime = runtime
|
||||
self._role = role
|
||||
self._output_root = output_root
|
||||
|
||||
async def answer(
|
||||
self, question: str, customer_case: str, execution_id: str
|
||||
) -> str | None:
|
||||
"""The answer, or None when the documents could not be read or the writer
|
||||
gave none."""
|
||||
try:
|
||||
delivered = load_baseline(self._output_root, customer_case, execution_id)
|
||||
except BaselineError:
|
||||
return None
|
||||
guide = str(
|
||||
(delivered["deliverables"].get("solution_guide") or {}).get("content", "")
|
||||
)
|
||||
if not guide.strip():
|
||||
return None
|
||||
|
||||
async def write_reply(text: str) -> ToolResult:
|
||||
if not text.strip():
|
||||
return ToolResult("Not submitted: the reply is empty.")
|
||||
return ToolResult("Written.", Submitted(text))
|
||||
|
||||
tool = ToolSpec(
|
||||
"write_reply",
|
||||
"Give the text the user will read. Ends your turn.",
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"text": {"type": "string"}},
|
||||
"required": ["text"],
|
||||
},
|
||||
write_reply,
|
||||
tier="execution",
|
||||
)
|
||||
task = AgentTask(
|
||||
role=self._role.name,
|
||||
prompt="Answer the user's question about the architecture delivered to them.",
|
||||
context={
|
||||
"question": question[:_MESSAGE_LIMIT],
|
||||
"delivered": {
|
||||
"execution_id": execution_id,
|
||||
"request": delivered.get("workflow_request", ""),
|
||||
"solution_guide": guide[:_GUIDE_LIMIT],
|
||||
},
|
||||
},
|
||||
)
|
||||
try:
|
||||
outcome = await self._runtime.run(self._role, task, [tool])
|
||||
except Exception: # a question must still get a reply
|
||||
logger.exception("Answering from delivered work failed")
|
||||
return None
|
||||
return outcome.content if isinstance(outcome, Submitted) else None
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
prove compare an evaluation run without lessons with one run with them,
|
||||
and write the comparison into every published proposal.
|
||||
retire remove a lesson from the Skill, by id.
|
||||
reviewed record that the lessons were read against the vendored Skills as
|
||||
they are now. `status` says when those have changed since.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python -m eval.curate_lessons propose [--from <folder> ...] [--min 2]
|
||||
@@ -16,6 +18,7 @@ Usage:
|
||||
.venv/bin/python -m eval.curate_lessons publish
|
||||
.venv/bin/python -m eval.curate_lessons prove --without <report.json> --with <report.json>
|
||||
.venv/bin/python -m eval.curate_lessons retire L-03
|
||||
.venv/bin/python -m eval.curate_lessons reviewed
|
||||
|
||||
A person approves a proposal by editing its file: set `Status: ready-for-agent`, or
|
||||
`wontfix` to reject it. Nothing is published without that. After publishing, prove
|
||||
@@ -40,8 +43,17 @@ from app.learning.curator import LESSON_CURATOR, curate
|
||||
from app.learning.evidence import group_candidates, identifiers, read_candidates
|
||||
from app.learning.proof import compare, record, render
|
||||
from app.learning.proposals import ISSUES, load
|
||||
from app.learning.skill import NotApproved, publish, published, retire
|
||||
from app.learning.skill import (
|
||||
NotApproved,
|
||||
mark_reviewed,
|
||||
publish,
|
||||
published,
|
||||
retire,
|
||||
upstream_changes,
|
||||
upstream_reviewed,
|
||||
)
|
||||
from app.providers import ProviderBundle, load_provider
|
||||
from app.skills.agent_skills import skill_version
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
# Where Executions write their candidates: under each evaluation run, and under the
|
||||
@@ -55,6 +67,16 @@ def _skill_dir(bundle: ProviderBundle) -> Path:
|
||||
return bundle.skills[bundle.lessons_skill].path
|
||||
|
||||
|
||||
def upstream_versions(bundle: ProviderBundle) -> dict[str, str]:
|
||||
"""The vendored Skills, each with the version it has now. They are what the
|
||||
lessons correct the use of, and they change when upstream is synced."""
|
||||
return {
|
||||
name: skill_version(meta.path)
|
||||
for name, meta in bundle.skills.items()
|
||||
if "vendor" in meta.path.parts
|
||||
}
|
||||
|
||||
|
||||
def _options(arguments: Sequence[str], name: str) -> list[str]:
|
||||
return [
|
||||
arguments[index + 1]
|
||||
@@ -115,12 +137,34 @@ def show_status() -> int:
|
||||
if not proposals:
|
||||
print("No lesson has been proposed.")
|
||||
for proposal in proposals:
|
||||
where = "in the Skill" if proposal.id in live else "not in the Skill"
|
||||
proof = "proved" if proposal.proof else "not proved"
|
||||
where = (
|
||||
"in the Skill"
|
||||
if proposal.id in live
|
||||
else "retired"
|
||||
if proposal.published
|
||||
else "not in the Skill"
|
||||
)
|
||||
# One comparison suggests a direction; it does not prove a lesson.
|
||||
proof = "compared" if proposal.proof else "not compared"
|
||||
print(
|
||||
f"{proposal.id} {proposal.status:16s} {where:16s} {proof:11s} "
|
||||
f"{proposal.id} {proposal.status:16s} {where:16s} {proof:12s} "
|
||||
f"[{proposal.deliverable}] {proposal.title}"
|
||||
)
|
||||
changed = upstream_changes(_skill_dir(bundle), upstream_versions(bundle))
|
||||
if changed:
|
||||
print(
|
||||
f"\nVendored Skills changed since the lessons were last reviewed: "
|
||||
f"{', '.join(changed)}.\nRead each lesson in the Skill against them: "
|
||||
"retire one the Skill now says itself or contradicts. Then run "
|
||||
"`reviewed`."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def mark_lessons_reviewed() -> int:
|
||||
bundle = load_provider(get_settings().CLOUD_PROVIDER)
|
||||
mark_reviewed(_skill_dir(bundle), upstream_versions(bundle))
|
||||
print("Recorded: the lessons stand against the vendored Skills as they are now.")
|
||||
return 0
|
||||
|
||||
|
||||
@@ -135,6 +179,10 @@ def publish_approved() -> int:
|
||||
continue
|
||||
done += 1
|
||||
print(f" published {proposal.id} [{proposal.deliverable}] {proposal.title}")
|
||||
if done and not upstream_reviewed(skill_dir):
|
||||
# The first lessons: what they were written beside is recorded once. After
|
||||
# that only a person's review moves it on.
|
||||
mark_reviewed(skill_dir, upstream_versions(bundle))
|
||||
print(f"{done} lessons are in {skill_dir.name}.")
|
||||
return 0
|
||||
|
||||
@@ -150,7 +198,8 @@ def prove_lessons(arguments: Sequence[str]) -> int:
|
||||
first, second = Path(without[0]), Path(with_lessons[0])
|
||||
result = compare(first, second)
|
||||
proof = render(result, first, second)
|
||||
recorded = record(load(ROOT / ISSUES), proof)
|
||||
bundle = load_provider(get_settings().CLOUD_PROVIDER)
|
||||
recorded = record(load(ROOT / ISSUES), proof, published(_skill_dir(bundle)))
|
||||
print(proof)
|
||||
print(f"\nRecorded in {len(recorded)} published proposals.")
|
||||
return 0 if result["helps"] else 1
|
||||
@@ -178,6 +227,8 @@ def main(arguments: Sequence[str]) -> int:
|
||||
return prove_lessons(rest)
|
||||
if command == "retire":
|
||||
return retire_lesson(rest)
|
||||
if command == "reviewed":
|
||||
return mark_lessons_reviewed()
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
@@ -4,10 +4,10 @@ description: >-
|
||||
Terraform patterns for Google Cloud products that validate against one pinned
|
||||
version of the hashicorp/google provider: Cloud Run services and jobs,
|
||||
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
|
||||
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring and API
|
||||
enablement. Use when writing Terraform for a Google Cloud product, before
|
||||
writing the resource, to start from a pattern whose arguments are known to be
|
||||
in the right place.
|
||||
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring,
|
||||
Firestore, BigQuery, Eventarc, GKE and API enablement. Use when writing
|
||||
Terraform for a Google Cloud product, before writing the resource, to start
|
||||
from a pattern whose arguments are known to be in the right place.
|
||||
metadata:
|
||||
scope: deployment guidance
|
||||
provider: hashicorp/google 8.6.0
|
||||
@@ -39,6 +39,10 @@ Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0.
|
||||
| Cloud Scheduler | `references/cloud-scheduler.md` |
|
||||
| Artifact Registry | `references/artifact-registry.md` |
|
||||
| Cloud Monitoring and Logging | `references/monitoring.md` |
|
||||
| Firestore | `references/firestore.md` |
|
||||
| BigQuery | `references/bigquery.md` |
|
||||
| Eventarc | `references/eventarc.md` |
|
||||
| Google Kubernetes Engine | `references/gke.md` |
|
||||
|
||||
A product with no reference has no pattern yet: write it from `terraform_schema` and the product documentation.
|
||||
|
||||
|
||||
92
skills/google-cloud-iac-patterns/references/bigquery.md
Normal file
92
skills/google-cloud-iac-patterns/references/bigquery.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# BigQuery
|
||||
|
||||
A dataset, a partitioned and clustered table, and the grants for a writer and a reader.
|
||||
|
||||
- `location` is a region or `EU`/`US`, and cannot be changed. Queries cannot join datasets in different locations.
|
||||
- `delete_contents_on_destroy = false` makes Terraform refuse to remove a dataset that still has tables.
|
||||
- A table's `deletion_protection` is true unless set; take it from a variable so a test deployment can be removed.
|
||||
- `schema` is a JSON string. Changing a column's type or mode replaces the table.
|
||||
- Partition on a `DATE` or `TIMESTAMP` column with `time_partitioning`, and set `require_partition_filter` so a query cannot scan every partition. `expiration_ms` there removes old partitions.
|
||||
- Grant on the dataset with `google_bigquery_dataset_iam_member`. A principal that runs queries also needs `roles/bigquery.jobUser` on the project.
|
||||
- Do not mix `google_bigquery_dataset_iam_member` with `access` blocks on the dataset: each removes what the other sets.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "writer_service_account_email" {
|
||||
type = string
|
||||
description = "Service account that loads rows."
|
||||
}
|
||||
|
||||
variable "reader_service_account_email" {
|
||||
type = string
|
||||
description = "Service account that queries the dataset."
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
type = bool
|
||||
description = "Whether the table is protected from deletion."
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "google_bigquery_dataset" "analytics" {
|
||||
project = var.project_id
|
||||
dataset_id = "analytics"
|
||||
location = var.region
|
||||
delete_contents_on_destroy = false
|
||||
|
||||
default_table_expiration_ms = null
|
||||
max_time_travel_hours = 168
|
||||
}
|
||||
|
||||
resource "google_bigquery_table" "events" {
|
||||
project = var.project_id
|
||||
dataset_id = google_bigquery_dataset.analytics.dataset_id
|
||||
table_id = "events"
|
||||
deletion_protection = var.deletion_protection
|
||||
|
||||
require_partition_filter = true
|
||||
clustering = ["event_type"]
|
||||
|
||||
time_partitioning {
|
||||
type = "DAY"
|
||||
field = "event_time"
|
||||
expiration_ms = 34560000000
|
||||
}
|
||||
|
||||
schema = jsonencode([
|
||||
{ name = "event_id", type = "STRING", mode = "REQUIRED" },
|
||||
{ name = "event_type", type = "STRING", mode = "REQUIRED" },
|
||||
{ name = "event_time", type = "TIMESTAMP", mode = "REQUIRED" },
|
||||
{ name = "payload", type = "JSON", mode = "NULLABLE" }
|
||||
])
|
||||
}
|
||||
|
||||
resource "google_bigquery_dataset_iam_member" "writer" {
|
||||
project = var.project_id
|
||||
dataset_id = google_bigquery_dataset.analytics.dataset_id
|
||||
role = "roles/bigquery.dataEditor"
|
||||
member = "serviceAccount:${var.writer_service_account_email}"
|
||||
}
|
||||
|
||||
resource "google_bigquery_dataset_iam_member" "reader" {
|
||||
project = var.project_id
|
||||
dataset_id = google_bigquery_dataset.analytics.dataset_id
|
||||
role = "roles/bigquery.dataViewer"
|
||||
member = "serviceAccount:${var.reader_service_account_email}"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "reader_runs_queries" {
|
||||
project = var.project_id
|
||||
role = "roles/bigquery.jobUser"
|
||||
member = "serviceAccount:${var.reader_service_account_email}"
|
||||
}
|
||||
```
|
||||
129
skills/google-cloud-iac-patterns/references/eventarc.md
Normal file
129
skills/google-cloud-iac-patterns/references/eventarc.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Eventarc
|
||||
|
||||
A trigger that delivers events to a Cloud Run service: one for messages on a Pub/Sub topic, one for objects finalised in a bucket.
|
||||
|
||||
- A trigger has its own service account. It needs `roles/eventarc.eventReceiver` on the project and `roles/run.invoker` on the service it calls.
|
||||
- For a Pub/Sub trigger, name an existing topic under `transport.pubsub.topic`; left out, Eventarc creates a topic you do not manage.
|
||||
- For a Cloud Storage trigger, `location` must be the bucket's location, and the Cloud Storage service agent needs `roles/pubsub.publisher` on the project before the trigger is created.
|
||||
- `matching_criteria` attributes are exact. `type` is always present; a storage event also needs `bucket`.
|
||||
- The destination service is named, with its `region`; `path` is optional.
|
||||
- The service must not allow unauthenticated calls for this to be the only way in: grant `roles/run.invoker` to the trigger's service account only.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "service_name" {
|
||||
type = string
|
||||
description = "Name of the Cloud Run service that receives the events."
|
||||
}
|
||||
|
||||
variable "topic_id" {
|
||||
type = string
|
||||
description = "Id of the Pub/Sub topic whose messages are delivered."
|
||||
}
|
||||
|
||||
variable "bucket_name" {
|
||||
type = string
|
||||
description = "Bucket whose new objects are delivered. It must be in var.region."
|
||||
}
|
||||
|
||||
data "google_storage_project_service_account" "this" {
|
||||
project = var.project_id
|
||||
}
|
||||
|
||||
resource "google_service_account" "trigger" {
|
||||
project = var.project_id
|
||||
account_id = "eventarc-trigger"
|
||||
display_name = "Eventarc trigger"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "trigger_receives_events" {
|
||||
project = var.project_id
|
||||
role = "roles/eventarc.eventReceiver"
|
||||
member = "serviceAccount:${google_service_account.trigger.email}"
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_service_iam_member" "trigger_invokes_service" {
|
||||
project = var.project_id
|
||||
location = var.region
|
||||
name = var.service_name
|
||||
role = "roles/run.invoker"
|
||||
member = "serviceAccount:${google_service_account.trigger.email}"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "storage_publishes_events" {
|
||||
project = var.project_id
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"
|
||||
}
|
||||
|
||||
resource "google_eventarc_trigger" "on_message" {
|
||||
project = var.project_id
|
||||
name = "on-message"
|
||||
location = var.region
|
||||
service_account = google_service_account.trigger.email
|
||||
|
||||
matching_criteria {
|
||||
attribute = "type"
|
||||
value = "google.cloud.pubsub.topic.v1.messagePublished"
|
||||
}
|
||||
|
||||
transport {
|
||||
pubsub {
|
||||
topic = var.topic_id
|
||||
}
|
||||
}
|
||||
|
||||
destination {
|
||||
cloud_run_service {
|
||||
service = var.service_name
|
||||
region = var.region
|
||||
path = "/events/message"
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
google_project_iam_member.trigger_receives_events,
|
||||
google_cloud_run_v2_service_iam_member.trigger_invokes_service,
|
||||
]
|
||||
}
|
||||
|
||||
resource "google_eventarc_trigger" "on_object" {
|
||||
project = var.project_id
|
||||
name = "on-object"
|
||||
location = var.region
|
||||
service_account = google_service_account.trigger.email
|
||||
|
||||
matching_criteria {
|
||||
attribute = "type"
|
||||
value = "google.cloud.storage.object.v1.finalized"
|
||||
}
|
||||
|
||||
matching_criteria {
|
||||
attribute = "bucket"
|
||||
value = var.bucket_name
|
||||
}
|
||||
|
||||
destination {
|
||||
cloud_run_service {
|
||||
service = var.service_name
|
||||
region = var.region
|
||||
path = "/events/object"
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
google_project_iam_member.trigger_receives_events,
|
||||
google_project_iam_member.storage_publishes_events,
|
||||
google_cloud_run_v2_service_iam_member.trigger_invokes_service,
|
||||
]
|
||||
}
|
||||
```
|
||||
74
skills/google-cloud-iac-patterns/references/firestore.md
Normal file
74
skills/google-cloud-iac-patterns/references/firestore.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Firestore
|
||||
|
||||
A Firestore database in Native mode, with point-in-time recovery, a scheduled backup, a composite index and the grant a workload needs.
|
||||
|
||||
- A project's first database is named `(default)`. Any other name creates an additional database, which clients must name when they connect.
|
||||
- `location_id` is a region or a multi-region such as `eur3`, and cannot be changed afterwards.
|
||||
- `type` is `FIRESTORE_NATIVE` or `DATASTORE_MODE`, and cannot be changed once the database holds data.
|
||||
- `delete_protection_state` is the guard in the API; `deletion_policy = "DELETE"` is needed before Terraform will remove a database at all.
|
||||
- `point_in_time_recovery_enablement` keeps seven days of versions. A backup schedule is separate and keeps whole backups for `retention`, in seconds with an `s`.
|
||||
- A workload reads and writes with `roles/datastore.user`, granted on the project. There is no per-database role; restrict with an IAM condition on the database name.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "app_service_account_email" {
|
||||
type = string
|
||||
description = "Service account of the workload that uses the database."
|
||||
}
|
||||
|
||||
resource "google_firestore_database" "app" {
|
||||
project = var.project_id
|
||||
name = "app"
|
||||
location_id = var.region
|
||||
type = "FIRESTORE_NATIVE"
|
||||
|
||||
concurrency_mode = "OPTIMISTIC"
|
||||
point_in_time_recovery_enablement = "POINT_IN_TIME_RECOVERY_ENABLED"
|
||||
delete_protection_state = "DELETE_PROTECTION_ENABLED"
|
||||
deletion_policy = "ABANDON"
|
||||
}
|
||||
|
||||
resource "google_firestore_backup_schedule" "daily" {
|
||||
project = var.project_id
|
||||
database = google_firestore_database.app.name
|
||||
retention = "1209600s"
|
||||
|
||||
daily_recurrence {}
|
||||
}
|
||||
|
||||
resource "google_firestore_index" "orders_by_customer" {
|
||||
project = var.project_id
|
||||
database = google_firestore_database.app.name
|
||||
collection = "orders"
|
||||
|
||||
fields {
|
||||
field_path = "customer_id"
|
||||
order = "ASCENDING"
|
||||
}
|
||||
|
||||
fields {
|
||||
field_path = "created_at"
|
||||
order = "DESCENDING"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "app_uses_firestore" {
|
||||
project = var.project_id
|
||||
role = "roles/datastore.user"
|
||||
member = "serviceAccount:${var.app_service_account_email}"
|
||||
|
||||
condition {
|
||||
title = "only-the-app-database"
|
||||
expression = "resource.name == \"${google_firestore_database.app.id}\" || resource.name.startsWith(\"${google_firestore_database.app.id}/\")"
|
||||
}
|
||||
}
|
||||
```
|
||||
109
skills/google-cloud-iac-patterns/references/gke.md
Normal file
109
skills/google-cloud-iac-patterns/references/gke.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Google Kubernetes Engine
|
||||
|
||||
A private Autopilot cluster on an existing subnet, with its node service account and Workload Identity for one workload.
|
||||
|
||||
- `enable_autopilot = true` makes Google manage the nodes. Node pools, machine types and node counts are then not yours to set: leave out `node_pool`, `node_config` and `initial_node_count`.
|
||||
- A regional `location` gives a control plane in three zones. A zone gives one.
|
||||
- The subnet needs two secondary ranges, for Pods and Services, named in `ip_allocation_policy`.
|
||||
- `enable_private_nodes = true` gives nodes no public address; they then need Cloud NAT to pull images from outside Google.
|
||||
- `master_authorized_networks_config` limits who can reach the control plane. With `enable_private_endpoint = true` it has no public address at all.
|
||||
- Give nodes a dedicated service account with `roles/container.defaultNodeServiceAccount`, not the Compute Engine default account.
|
||||
- Autopilot has Workload Identity on. A Kubernetes service account acts as a Google service account through `roles/iam.workloadIdentityUser`, with the member `serviceAccount:PROJECT.svc.id.goog[NAMESPACE/KSA]`.
|
||||
- `deletion_protection` is true unless set; take it from a variable.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "network_id" {
|
||||
type = string
|
||||
description = "Id of the VPC the cluster is in."
|
||||
}
|
||||
|
||||
variable "subnetwork_id" {
|
||||
type = string
|
||||
description = "Id of the subnet for nodes. It has secondary ranges named pods and services."
|
||||
}
|
||||
|
||||
variable "admin_cidr" {
|
||||
type = string
|
||||
description = "Range allowed to reach the control plane."
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
type = bool
|
||||
description = "Whether the cluster is protected from deletion."
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "google_service_account" "nodes" {
|
||||
project = var.project_id
|
||||
account_id = "gke-nodes"
|
||||
display_name = "GKE nodes"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "nodes_default_role" {
|
||||
project = var.project_id
|
||||
role = "roles/container.defaultNodeServiceAccount"
|
||||
member = "serviceAccount:${google_service_account.nodes.email}"
|
||||
}
|
||||
|
||||
resource "google_container_cluster" "workload" {
|
||||
project = var.project_id
|
||||
name = "workload"
|
||||
location = var.region
|
||||
enable_autopilot = true
|
||||
deletion_protection = var.deletion_protection
|
||||
|
||||
network = var.network_id
|
||||
subnetwork = var.subnetwork_id
|
||||
|
||||
ip_allocation_policy {
|
||||
cluster_secondary_range_name = "pods"
|
||||
services_secondary_range_name = "services"
|
||||
}
|
||||
|
||||
private_cluster_config {
|
||||
enable_private_nodes = true
|
||||
enable_private_endpoint = false
|
||||
}
|
||||
|
||||
master_authorized_networks_config {
|
||||
cidr_blocks {
|
||||
cidr_block = var.admin_cidr
|
||||
display_name = "administrators"
|
||||
}
|
||||
}
|
||||
|
||||
cluster_autoscaling {
|
||||
auto_provisioning_defaults {
|
||||
service_account = google_service_account.nodes.email
|
||||
}
|
||||
}
|
||||
|
||||
release_channel {
|
||||
channel = "REGULAR"
|
||||
}
|
||||
|
||||
depends_on = [google_project_iam_member.nodes_default_role]
|
||||
}
|
||||
|
||||
resource "google_service_account" "app" {
|
||||
project = var.project_id
|
||||
account_id = "orders-app"
|
||||
display_name = "Orders application"
|
||||
}
|
||||
|
||||
resource "google_service_account_iam_member" "app_workload_identity" {
|
||||
service_account_id = google_service_account.app.name
|
||||
role = "roles/iam.workloadIdentityUser"
|
||||
member = "serviceAccount:${var.project_id}.svc.id.goog[orders/orders-app]"
|
||||
}
|
||||
```
|
||||
9
skills/google-cloud-lessons/upstream.json
Normal file
9
skills/google-cloud-lessons/upstream.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"google-cloud-solution-architecture": "f5b8cdac4060",
|
||||
"google-cloud-waf-cost-optimization": "b3febbf52e5c",
|
||||
"google-cloud-waf-operational-excellence": "bdc6a9f415f6",
|
||||
"google-cloud-waf-performance-optimization": "89a4947438cf",
|
||||
"google-cloud-waf-reliability": "9baf33bd5b75",
|
||||
"google-cloud-waf-security": "e791a44cda99",
|
||||
"google-cloud-waf-sustainability": "7d0a6921cb59"
|
||||
}
|
||||
@@ -75,7 +75,7 @@ def complete_run_turns() -> dict:
|
||||
|
||||
|
||||
def client_for(
|
||||
tmp_path: Path, turns: dict, decider=None
|
||||
tmp_path: Path, turns: dict, decider=None, answers=None
|
||||
) -> tuple[TestClient, FakeAgentRuntime]:
|
||||
bundle = load_provider("gcp")
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
@@ -96,6 +96,7 @@ def client_for(
|
||||
roles,
|
||||
{"compute": "API not enabled"},
|
||||
intake=Intake(decider, tmp_path) if decider else None,
|
||||
answers=answers,
|
||||
)
|
||||
|
||||
return TestClient(create_app(get_settings(), fake_assembly)), runtime
|
||||
@@ -398,6 +399,74 @@ def delivered_case(tmp_path: Path, case: str = "acme") -> None:
|
||||
)
|
||||
|
||||
|
||||
class Answers:
|
||||
"""Stands in for the reply writer answering from delivered files."""
|
||||
|
||||
def __init__(self, reply):
|
||||
self.reply = reply
|
||||
self.asked: list[tuple[str, str, str]] = []
|
||||
|
||||
async def answer(self, question, customer_case, execution_id):
|
||||
self.asked.append((question, customer_case, execution_id))
|
||||
return self.reply
|
||||
|
||||
|
||||
ABOUT_DELIVERED = {
|
||||
"intake": Answer("question", 0.98),
|
||||
"about_delivered": Answer(0.92, 0.9),
|
||||
}
|
||||
|
||||
|
||||
def test_a_question_about_delivered_work_is_answered_from_it(tmp_path):
|
||||
delivered_case(tmp_path)
|
||||
answers = Answers("Cloud SQL, because bookings are relational.")
|
||||
client, runtime = client_for(tmp_path, {}, FakeDecider(ABOUT_DELIVERED), answers)
|
||||
with client:
|
||||
task = send(
|
||||
client,
|
||||
[{"text": "Why did you choose Cloud SQL?"}],
|
||||
metadata={"customer_case": "acme"},
|
||||
)
|
||||
|
||||
text, data = status_parts(task)
|
||||
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||
assert text == "Cloud SQL, because bookings are relational."
|
||||
assert data == {"status": "answered", "intake": "question", "about": "v1"}
|
||||
assert answers.asked == [("Why did you choose Cloud SQL?", "acme", "v1")]
|
||||
assert runtime.runs == []
|
||||
|
||||
|
||||
def test_a_question_that_could_not_be_answered_says_where_to_look(tmp_path):
|
||||
delivered_case(tmp_path)
|
||||
client, _ = client_for(tmp_path, {}, FakeDecider(ABOUT_DELIVERED), Answers(None))
|
||||
with client:
|
||||
task = send(
|
||||
client, [{"text": "Why Cloud SQL?"}], metadata={"customer_case": "acme"}
|
||||
)
|
||||
|
||||
text, data = status_parts(task)
|
||||
assert "could not answer that from the documents delivered in Execution v1" in text
|
||||
assert data["about"] == "v1"
|
||||
|
||||
|
||||
def test_a_question_about_the_service_is_still_answered_in_general(tmp_path):
|
||||
delivered_case(tmp_path)
|
||||
decider = FakeDecider(
|
||||
{"intake": Answer("question", 0.98), "about_delivered": Answer(0.08, 0.9)}
|
||||
)
|
||||
answers = Answers("unused")
|
||||
client, _ = client_for(tmp_path, {}, decider, answers)
|
||||
with client:
|
||||
task = send(
|
||||
client, [{"text": "What can you do?"}], metadata={"customer_case": "acme"}
|
||||
)
|
||||
|
||||
text, data = status_parts(task)
|
||||
assert text.startswith("I design cloud solution architectures.")
|
||||
assert data == {"status": "answered", "intake": "question"}
|
||||
assert answers.asked == []
|
||||
|
||||
|
||||
def test_a_change_to_delivered_work_names_the_baseline_and_starts_nothing(tmp_path):
|
||||
delivered_case(tmp_path)
|
||||
decider = FakeDecider({"intake": Answer("change_request", 0.98)})
|
||||
|
||||
@@ -62,3 +62,100 @@ async def test_a_change_needs_something_delivered_to_change(tmp_path):
|
||||
assert "- Design thing 1." in decider.asked[0][0]
|
||||
assert "Design thing 2." not in decider.asked[0][0] # still in progress
|
||||
assert without.kind == NEW_REQUEST and no_case.kind == NEW_REQUEST
|
||||
|
||||
|
||||
# --- a question about what was delivered ------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_question_about_delivered_work_is_told_from_one_about_the_service(
|
||||
tmp_path,
|
||||
):
|
||||
delivered(tmp_path, "acme", ["completed"])
|
||||
about = FakeDecider(
|
||||
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.9, 0.9)}
|
||||
)
|
||||
general = FakeDecider(
|
||||
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.1, 0.9)}
|
||||
)
|
||||
|
||||
first = await Intake(about, tmp_path).read("Why Cloud SQL?", "acme")
|
||||
second = await Intake(general, tmp_path).read("What can you do?", "acme")
|
||||
|
||||
assert first.kind == QUESTION and first.about_delivered and first.latest == "v1"
|
||||
assert second.kind == QUESTION and not second.about_delivered
|
||||
|
||||
|
||||
async def test_with_nothing_delivered_a_question_is_never_about_delivered_work(
|
||||
tmp_path,
|
||||
):
|
||||
decider = FakeDecider(
|
||||
{"intake": Answer(QUESTION, 0.98), "about_delivered": Answer(0.99, 0.9)}
|
||||
)
|
||||
|
||||
reading = await Intake(decider, tmp_path).read("Why Cloud SQL?", "new-client")
|
||||
|
||||
assert not reading.about_delivered
|
||||
assert len(decider.asked) == 1 # the second question was not asked
|
||||
|
||||
|
||||
async def test_an_answer_is_written_from_the_delivered_guide_and_nothing_else(
|
||||
tmp_path,
|
||||
):
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
|
||||
from app.adk.runtime import RoleSpec
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.intake import DeliveredAnswers
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
|
||||
roles = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
**{
|
||||
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
seen: dict = {}
|
||||
|
||||
def writer(task):
|
||||
seen.update(task.context)
|
||||
return [Call("write_reply", {"text": "Cloud SQL, for relational bookings."})]
|
||||
|
||||
runtime = FakeAgentRuntime(
|
||||
{
|
||||
**{
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
},
|
||||
"reply-writer": [writer],
|
||||
}
|
||||
)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=roles,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
)
|
||||
await engine.start(
|
||||
"Design bookings.", customer_case="acme", pre_approved=True, execution_id="v1"
|
||||
)
|
||||
runs_before = len(runtime.runs)
|
||||
answers = DeliveredAnswers(
|
||||
runtime, RoleSpec("reply-writer", "Write.", budget=3), tmp_path
|
||||
)
|
||||
|
||||
answer = await answers.answer("Why Cloud SQL?", "acme", "v1")
|
||||
|
||||
assert answer == "Cloud SQL, for relational bookings."
|
||||
assert seen["question"] == "Why Cloud SQL?"
|
||||
assert seen["delivered"]["execution_id"] == "v1"
|
||||
assert seen["delivered"]["request"] == "Design bookings."
|
||||
assert seen["delivered"]["solution_guide"]
|
||||
assert len(runtime.runs) == runs_before + 1 # one writer run; no Execution
|
||||
# An Execution that is not there gives no answer, not an error.
|
||||
assert await answers.answer("Why?", "acme", "missing") is None
|
||||
|
||||
100
tests/execution/test_outputs_as_it_goes.py
Normal file
100
tests/execution/test_outputs_as_it_goes.py
Normal file
@@ -0,0 +1,100 @@
|
||||
"""An Execution's files are on disk as soon as each Deliverable is submitted.
|
||||
|
||||
Before this they were written only when an Execution ended, so a process that was
|
||||
killed, or an Execution left waiting for its user, had nothing to show.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import FakeAgentRuntime, sample_submission
|
||||
from app.adk.runtime import NoDecider, RoleSpec
|
||||
from app.agent import ExecutionEngine
|
||||
from app.baseline import BaselineError, load_baseline
|
||||
from app.nodes.execution_nodes import _write_outputs
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.states.ledger import Ledger
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
|
||||
ROLES = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
**{
|
||||
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def folder(tmp_path, execution_id: str):
|
||||
return tmp_path / "deliverables" / "cases" / "acme" / "executions" / execution_id
|
||||
|
||||
|
||||
async def test_an_execution_waiting_for_its_user_has_written_what_exists(tmp_path):
|
||||
engine = ExecutionEngine(
|
||||
runtime=FakeAgentRuntime(
|
||||
{
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
),
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
)
|
||||
|
||||
waiting = await engine.start("Ingest", customer_case="acme", execution_id="v1")
|
||||
|
||||
# It stopped for the first Approval: only the first Phase has been produced.
|
||||
assert waiting.status == "awaiting_user"
|
||||
out = folder(tmp_path, "v1")
|
||||
manifest = json.loads((out / "execution.json").read_text())
|
||||
assert manifest["status"] == "in_progress"
|
||||
assert set(manifest["deliverables"]) == {"requirements", "technical_decomposition"}
|
||||
assert (out / "deliverables" / "requirements.md").read_text()
|
||||
assert (out / "requirements.json").is_file()
|
||||
assert manifest["solution_guide"] is None
|
||||
# Unfinished work is never something a Change request can start from.
|
||||
try:
|
||||
load_baseline(tmp_path, "acme", "v1")
|
||||
except BaselineError as refused:
|
||||
assert "did not complete (in_progress)" in str(refused)
|
||||
else:
|
||||
raise AssertionError("an in-progress Execution was accepted as a Baseline")
|
||||
|
||||
|
||||
def test_a_file_an_earlier_write_left_is_removed_with_the_revision(tmp_path):
|
||||
state = {
|
||||
"execution_id": "v1",
|
||||
"customer_case": "acme",
|
||||
"workflow_request": "Ingest",
|
||||
"deliverables": {
|
||||
"deployment_guidance": {
|
||||
"content": "guide",
|
||||
"attachments": {
|
||||
"terraform/main.tf": "one",
|
||||
"terraform/queue.tf": "two",
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
out = folder(tmp_path, "v1")
|
||||
|
||||
_write_outputs(tmp_path, state, "in_progress", Ledger())
|
||||
assert (out / "terraform" / "queue.tf").is_file()
|
||||
(out / "notes.txt").write_text("a person's own file")
|
||||
|
||||
# The Revision folded the queue into main.tf.
|
||||
state["deliverables"]["deployment_guidance"]["attachments"] = {
|
||||
"terraform/main.tf": "one and two"
|
||||
}
|
||||
_write_outputs(tmp_path, state, "completed", Ledger())
|
||||
|
||||
assert not (out / "terraform" / "queue.tf").exists()
|
||||
assert (out / "terraform" / "main.tf").read_text() == "one and two"
|
||||
# Only files this Execution wrote are ever removed.
|
||||
assert (out / "notes.txt").is_file()
|
||||
assert json.loads((out / "execution.json").read_text())["status"] == "completed"
|
||||
@@ -498,16 +498,68 @@ def test_lessons_that_cost_quality_or_save_nothing_are_not_shown_to_help(tmp_pat
|
||||
assert "nothing fell" in render(compare(base, same), base, same)
|
||||
|
||||
|
||||
def test_the_comparison_is_written_into_published_proposals_only(tmp_path):
|
||||
def test_the_comparison_is_written_only_for_lessons_in_the_skill(tmp_path):
|
||||
from app.learning.proof import record
|
||||
|
||||
skill = lessons_skill(tmp_path)
|
||||
issues = tmp_path / "issues"
|
||||
live = publish(approve(proposal(issues), issues), skill, today=DAY)
|
||||
waiting = proposal(issues, title="Another", evidence_key="zzz")
|
||||
gone = publish(
|
||||
approve(proposal(issues, title="Retired", evidence_key="yyy"), issues),
|
||||
skill,
|
||||
today=DAY,
|
||||
)
|
||||
retire(gone.id, skill)
|
||||
|
||||
recorded = record([live, waiting], "| a table |")
|
||||
recorded = record([live, waiting, gone], "| a table |", published(skill))
|
||||
|
||||
assert [p.id for p in recorded] == ["L-01"]
|
||||
assert parse(live.path).proof == "| a table |"
|
||||
assert parse(waiting.path).proof == ""
|
||||
# A retired lesson was not part of what was compared.
|
||||
assert parse(gone.path).proof == ""
|
||||
|
||||
|
||||
# --- lessons are read again when the Skills beside them change ---------------------------
|
||||
|
||||
|
||||
def test_a_changed_upstream_skill_is_flagged_until_the_lessons_are_reviewed(tmp_path):
|
||||
from app.learning.skill import mark_reviewed, upstream_changes
|
||||
|
||||
skill = lessons_skill(tmp_path)
|
||||
issues = tmp_path / "issues"
|
||||
publish(approve(proposal(issues), issues), skill, today=DAY)
|
||||
then = {"architecture": "aaa", "security": "bbb"}
|
||||
mark_reviewed(skill, then)
|
||||
|
||||
assert upstream_changes(skill, then) == []
|
||||
now = {"architecture": "aaa", "security": "ccc", "cost": "ddd"}
|
||||
assert upstream_changes(skill, now) == ["cost", "security"]
|
||||
|
||||
mark_reviewed(skill, now)
|
||||
assert upstream_changes(skill, now) == []
|
||||
|
||||
|
||||
def test_nothing_is_flagged_with_no_lessons_or_nothing_recorded(tmp_path):
|
||||
from app.learning.skill import mark_reviewed, upstream_changes
|
||||
|
||||
skill = lessons_skill(tmp_path)
|
||||
assert upstream_changes(skill, {"architecture": "aaa"}) == []
|
||||
mark_reviewed(skill, {"architecture": "aaa"})
|
||||
# No lesson is published, so there is nothing to review.
|
||||
assert upstream_changes(skill, {"architecture": "zzz"}) == []
|
||||
|
||||
|
||||
def test_the_published_lessons_were_reviewed_against_todays_vendored_skills():
|
||||
"""Syncing the vendored Skills changes what the lessons sit beside. This fails
|
||||
until a person has read each lesson again and run `curate_lessons reviewed`."""
|
||||
from app.learning.skill import upstream_changes, upstream_reviewed
|
||||
from app.providers import load_provider
|
||||
from eval.curate_lessons import upstream_versions
|
||||
|
||||
bundle = load_provider("gcp")
|
||||
skill = bundle.skills[bundle.lessons_skill].path
|
||||
|
||||
assert upstream_reviewed(skill), "run: python -m eval.curate_lessons reviewed"
|
||||
assert upstream_changes(skill, upstream_versions(bundle)) == []
|
||||
|
||||
Reference in New Issue
Block a user