feat: the solution architecture agent on the shared Execution engine
The reference agent's subject, cloud solution architecture, as an agent in the operating model agent's pattern: the Execution engine (from orchestration_engine) and one Workflow in app/solution_architecture, built for the settings and provider bundle by assembly.workflow_for. Copied from gcp_solution_architecture_agent at 8b397cd without its own engine: its role files, Skills, provider bundle, ADRs, vocabulary, scripts and evaluation datasets. The image takes the engine from a named build context and installs Terraform. It is called through the Orchestration engine, and its files are what the operating model agent digests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
---
|
||||
status: superseded by ADR-0003
|
||||
---
|
||||
|
||||
# LangGraph orchestrates Phases, not Google ADK
|
||||
|
||||
Phases are orchestrated by a LangGraph `StateGraph`, as the KAB Workflow Agent standard (`kyndryl-agent-builder/docs/KAB_WORKFLOW_AGENT_IMPLEMENTATION_GUIDE.md`) requires, rather than by Google ADK `Runner`/`LoopAgent`. The earlier "ADK" orchestration was a local compatibility layer that never used the real `google-adk` package, so it added a pretend seam with one adapter and diverged from the platform standard; we deleted it. Real ADK remains welcome *inside* a Phase (e.g. an `LlmAgent` for LLM-driven design) — just not as the thing that sequences Phases.
|
||||
|
||||
> **Reopened (2026-09-15).** [ADR-0002](0002-skills-are-the-portable-unit-of-behaviour.md) makes Skills, loaded by a framework's own skill runtime, the portable unit of behaviour. This ADR assumed our code sequences the Phases. If the Skill sequences them instead, the choice here reduces to which skill runtime to wire first, and "ADK only inside a Phase" no longer holds. Still true regardless: the compat stubs are fiction and should go.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Real `google-adk` as orchestrator**: rejected; departs from the KAB workflow standard and would require async event-stream plumbing for no capability LangGraph lacks (conditional edges cover the Review round).
|
||||
- **Plain in-process pipeline, no framework**: rejected; neither KAB template uses it, and LangGraph already gives us conditional routing and a path to checkpointing if a human approval gate is added later.
|
||||
25
docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md
Normal file
25
docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Skills are the portable unit of behaviour
|
||||
|
||||
This agent exists to replace the [`google-cloud-solution-architecture`](https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/SKILL.md) skill with an agent that runs anywhere. Its solution-architecture behaviour lives in Skills that follow the [Agent Skills specification](https://agentskills.io/specification) (`SKILL.md` plus `references/`, `assets/` and `scripts/`). The agent framework's own skill runtime loads them: Google ADK's `SkillToolset`, or LangChain's skills pattern (Deep Agents `skills=` / `SkillsMiddleware`). Our code supplies the tooling those Skills call on: grounding, IaC validation and environment scanning. It doesn't restate the work the Skills describe.
|
||||
|
||||
Google Cloud is the first and, for now, only Cloud provider. Other hyperscalers should be addable as new Skills plus new tooling, without rewriting the agent's core.
|
||||
|
||||
## Refinements (2026-09-15)
|
||||
|
||||
- **Hybrid sequencing.** Our code owns Phase boundaries and the Approvals between Phases. Within a Phase, the Skill, run by the skill runtime, decides the work. We rejected letting the Skill sequence everything: Phase checkpoints give us resumable, recordable, testable progress. We rejected code-owned Phase bodies because they would bypass the Skill.
|
||||
- **Hierarchical and ReAct, not linear.** Work isn't a fixed chain. Agents are organised in a hierarchy that follows the dependencies between Deliverables. Work whose dependencies are satisfied may proceed independently, and every agent reasons and acts in a ReAct loop of thought, tool call and observation. An Orchestrator sits above one Specialist per Deliverable, and a Specialist fans out further only where its work splits into independent parts. Code declares the dependency graph, and the Orchestrator chooses what to do next within it. The engine is recorded in [ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md).
|
||||
- **Interactive by default.** Executions are multi-turn with user Approval, as the upstream Skill requires. A pre-approved mode serves batch and evaluation runs.
|
||||
- **Vendored upstream Skill.** `google-cloud-solution-architecture` is copied unmodified and pinned to an upstream commit. Our additions live in separate Skills for our own tooling, so upstream changes arrive as a diff.
|
||||
- **Discovery through the provider's own MCP servers.** Current-state discovery and documentation grounding go through the remote MCP servers each Cloud provider hosts, for example Google Cloud's Resource Manager, Asset Inventory, Compute, GKE and Developer Knowledge servers, and the AWS MCP Server. Code carries no hand-written cloud API clients. Only reviewed read-only MCP tools are exposed, and reading a user's cloud requires their permission.
|
||||
- **Cloud provider as configuration.** A Cloud provider is a configured bundle of Skills, tools and remote MCP servers. There's no provider interface in code until a second Cloud provider exists, but core module names carry no provider prefix.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Framework-specific code is confined to wiring a skill runtime and registering tools, so moving to another framework is a new wiring, not a rewrite.
|
||||
- Skills in `app/skills/` must conform to the specification: kebab-case `name` matching the directory, a `description` of when to use the Skill, and no custom `phase:` key. They don't today. (2026-10-06: those five Skills were deleted with the original pipeline. The engine loads only the conforming Skills under `skills/`, and `app/skills/` holds the discovery and validation code.)
|
||||
- This reopens [ADR-0001](0001-langgraph-orchestrates-phases-not-google-adk.md) and the Phase pipeline spec (`.scratch/phase-pipeline/spec.md`). [ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) settles the engine and supersedes ADR-0001.
|
||||
- Both known skill runtimes are young: ADK marks Skills *Experimental* (Python ≥ 1.25.0), and Deep Agents skills are new. Pin versions.
|
||||
27
docs/adr/0003-langgraph-holds-state-adk-agents-reason.md
Normal file
27
docs/adr/0003-langgraph-holds-state-adk-agents-reason.md
Normal file
@@ -0,0 +1,27 @@
|
||||
---
|
||||
status: accepted
|
||||
supersedes: ADR-0001
|
||||
---
|
||||
|
||||
# LangGraph holds Execution state; ADK agents reason and act
|
||||
|
||||
An Execution is a hierarchy of ReAct agents shaped by the dependencies between Deliverables ([ADR-0002](0002-skills-are-the-portable-unit-of-behaviour.md)).
|
||||
|
||||
- **LangGraph holds the Execution.** A LangGraph `StateGraph` holds the dependency graph and each Deliverable's state. It checkpoints after every step and pauses with `interrupt()` whenever the user must answer or approve. This follows the KAB workflow standard and the platform's `langgraph-workflow-agent`.
|
||||
- **ADK agents reason and act.** The Orchestrator and every Specialist is an ADK `LlmAgent`, each running its own ReAct loop, with a `SkillToolset` for progressive Skill loading. This follows the platform's `google-adk-base-agent/app/skills_integration.py`.
|
||||
- **Agents sit behind one interface.** Each agent runs behind a single agent-runtime interface, so LangChain Deep Agents can be added as a second implementation without touching the graph.
|
||||
|
||||
The dependency graph is declared as data in code. The Orchestrator picks what to do next through tools that refuse work whose dependencies aren't met. Dependencies are guaranteed by code, while ordering and parallelism are the model's choice.
|
||||
|
||||
> **Amended (2026-10-07) by [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), accepted.** The Orchestrator is now a router: it decides by readings and a playbook in code, with Jev for the judgements, and runs no language model to choose an action. That replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected and still reason on language models.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **ADK end to end** (root `LlmAgent` with `sub_agents`/`AgentTool`, and ADK sessions for pauses): rejected. It departs from the KAB workflow standard, and LangGraph's checkpointed `interrupt()` is the proven platform path for resumable user Approvals.
|
||||
- **Deep Agents end to end** (`create_deep_agent(skills=, subagents=)`): viable, and it compiles to LangGraph. It was deferred rather than rejected: ADK `SkillToolset` already has working prior art on this platform, and Deep Agents stays reachable as the second implementation of the agent-runtime interface.
|
||||
- **Code runs the dependency graph with no Orchestrator model**: rejected. The top level wouldn't be ReAct, and the Skill's conversational flexibility (such as skipping work the user says is already approved) would need hand-coding.
|
||||
|
||||
## Consequences
|
||||
|
||||
- There are two frameworks in one process. Framework types stay inside the agent-runtime implementation and never appear in graph state.
|
||||
- ADK Skills are *Experimental* (Python ≥ 1.25.0), so pin the version.
|
||||
46
docs/adr/0004-lessons-and-overlays-arrive-as-skills.md
Normal file
46
docs/adr/0004-lessons-and-overlays-arrive-as-skills.md
Normal file
@@ -0,0 +1,46 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Lessons and overlays arrive as Skills
|
||||
|
||||
An Execution starts clean ([CONTEXT.md](../../CONTEXT.md)) unless it is a Change request with a named Baseline ([ADR-0006](0006-a-change-request-starts-from-a-baseline-execution.md)), so nothing one Execution learns may flow into the next as hidden state. What an Execution gets wrong is still worth keeping. We decide that everything that changes how agents behave between Executions is a Skill change, of one of two kinds.
|
||||
|
||||
- **Lessons.** Each Execution records **lesson candidates**: Revisions and their reasons, validator findings, tool refusals, the user's change requests, and work that stopped. An offline curator proposes lessons from candidates across Executions. A person approves each one. An approved lesson is added to a lessons Skill that the provider bundle assigns to roles.
|
||||
- **Overlays.** An organisation's standards or one engagement's constraints are written as their own Skills. The provider bundle lists the ones every Execution loads. A caller may name further overlay Skills when it starts an Execution.
|
||||
|
||||
Vendored Skills stay unmodified ([ADR-0002](0002-skills-are-the-portable-unit-of-behaviour.md)); lessons and overlays sit beside them.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-08, agreeing to each of these:
|
||||
|
||||
1. **No lesson takes effect without a person's approval.** The curator proposes; it never edits a Skill that agents load. We prefer slower learning to an agent that rewrites its own instructions.
|
||||
2. **A lesson must be shown to help.** It is kept only if the evaluation, run with and without it, holds quality and lowers Revisions, time or tokens.
|
||||
3. **Lessons are per Cloud provider.** The lessons Skill is listed in that provider's bundle, since most lessons name its products.
|
||||
4. **Overlay Skills come only from directories the deployment configures.** A caller names an overlay by its Skill name; it cannot upload one. Each passes the same Agent Skills validation as bundled Skills, and an unknown name refuses the Execution at the start.
|
||||
5. **Every Execution records the Skills it loaded and a content hash of each,** so a result can be traced to the instructions that produced it.
|
||||
6. **Lesson candidates are client data.** They quote user feedback and name resources. They are stored with the Execution's records, never in the repository, and a lesson is written so that it carries no client detail.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Memory carried between Executions** (a vector store of past Deliverables or feedback, retrieved at run time): rejected. It breaks "every Execution starts clean", can leak one client's content into another's Execution, and cannot be reviewed or rolled back as a diff.
|
||||
- **Automatic Skill edits** (the curator applies lessons directly): rejected for now. Without review, one bad Execution can degrade every later one, and nothing ties a regression to its cause.
|
||||
- **Editing the vendored Skills in place**: rejected by ADR-0002. Upstream changes must arrive as a clean diff.
|
||||
- **Overlays as prompt text passed by the caller**: rejected. Free text bypasses Skill validation and progressive loading, and it is an injection path.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Learning is slow by design: a lesson reaches agents only after capture, proposal, review and an evaluation run.
|
||||
- The evaluation must be able to see a difference a Skill makes. Its structural checks cannot, so the judged checks and the thresholds on Revisions, time and tokens become part of the loop.
|
||||
- The glossary gains **Lesson candidate**, **Lesson** and **Overlay Skill**.
|
||||
- Skills can no longer be loaded once at start-up for all Executions: the runtime must resolve a role's Skills per Execution.
|
||||
|
||||
## As built (2026-10-09)
|
||||
|
||||
Two points differ from the text above; the decisions stand.
|
||||
|
||||
- **Default overlays are a deployment setting, not part of the provider bundle.** `DEFAULT_OVERLAY_SKILLS` names the overlays every Execution loads. Whose standards apply belongs to the deployment and the caller, not to a Cloud provider.
|
||||
- **Skills are still loaded once at start-up.** The runtime loads every provider Skill and every overlay in the configured folders when it starts, and an Execution chooses among them. An overlay added to a folder needs a restart.
|
||||
|
||||
Decision 2 has a first result: on one benchmark case run each way, the lessons held quality and removed a Revision. One lesson was retired because the fault was in a review, not in the agents.
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# The target architecture is an explicit graph, and code comes after it
|
||||
|
||||
Today the design passes from Specialist to Specialist as prose, and the infrastructure code is written before anyone has stated what it must satisfy. In an evaluation run on 2026-10-06 the validator found 4 problems, then 7, then 5 different ones, across three Revisions of the same Terraform: the bar was discovered after the code existed, and it moved. We decide that the **Target architecture** is one explicit graph, held as data, that is established before any infrastructure code is written, and that everything after it is derived from it and checked against it.
|
||||
|
||||
The graph has three kinds of thing:
|
||||
|
||||
- **Elements**: every component of the solution and the resource that realises it, with the product selected for it.
|
||||
- **Relationships**: how Elements connect: calls, publishes to, reads from, is granted access to.
|
||||
- **Checks**: on each Element and Relationship, the numbered, verifiable conditions it must meet. Each Check names the requirement or design recommendation it comes from.
|
||||
|
||||
This makes the **Requirements chain** explicit: every requirement leads to the components that serve it, to the Elements and Checks that realise it, to the code that implements them, and to a validation verdict. A link that is missing is refused, not discovered later.
|
||||
|
||||
## Who provides what
|
||||
|
||||
The user states what they want to achieve, at a high level, in the Workflow request, and answers the questions that only they can answer. The agents produce everything in the graph: every Element, Relationship and Check is derived by Specialists from the requirements, the Cloud provider's architecture framework and its documented best practices. The user never writes or edits the graph. They approve it, from a summary, before code is written.
|
||||
|
||||
## What changes
|
||||
|
||||
- **A new Deliverable, the Target architecture,** follows product selection and design recommendations. Its graph is an attachment; its content is the readable summary.
|
||||
- **Completeness is a submission rule.** The Target architecture is refused when a requirement, a selected product or a design recommendation has no Element or Check, or when a Relationship or Check refers to an Element that is not in the graph.
|
||||
- **Later Deliverables are views of the graph.** The architecture diagram is rendered from its Elements and Relationships. The architecture description describes them. The deployment guidance implements every Element, Relationship and Check.
|
||||
- **Validation is a verdict per Check,** pass or fail with evidence, not a list the validator composes on each pass. Something the validator notices that the graph lacks is a proposed addition to the Target architecture. It cannot fail the code. If it matters, the Target architecture is revised, and the code follows.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **The user approves the Target architecture before code is written.** The solution architecture Phase splits in two: design, which ends with the Target architecture, and implementation, which holds the deployment guidance. The upstream Skill has four Phases; this makes five, and is a deliberate departure from it.
|
||||
2. **A new Specialist produces the Target architecture.** The design advisor stays responsible for recommendations only.
|
||||
3. **Code renders the diagram from the graph.** No model draws it, so the picture cannot disagree with the architecture, and the diagrammer role goes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Keep the prose chain and tighten the validator's instructions**: rejected. The validator would still decide what to check after the code exists, and nothing would make two passes agree.
|
||||
- **A build plan for the infrastructure code only**, beside the architecture: rejected. It would be a second description of the same system, free to drift from the diagram and the description. The graph has to be the architecture itself.
|
||||
- **Let the infrastructure code be the model** and derive the diagram and checks from Terraform: rejected. It puts code first, which is the order that failed, and it ties the architecture to one infrastructure-as-code tool.
|
||||
- **Raise the Revision limit**: rejected. It spends more on a loop that has no fixed target.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Requirements reach code through data, not prose, so a requirement that nothing implements is caught when the Target architecture is submitted.
|
||||
- A Revision either closes a known Check or it does not. Validation findings stop growing between passes.
|
||||
- More is decided before code: the Specialist that writes the Target architecture carries work the infrastructure author and validator did implicitly, and a gap in the graph now costs a Revision of the Target architecture and everything after it.
|
||||
- The Phase list is no longer the upstream Skill's. The solution guide still follows the upstream template; only where Approvals fall changes.
|
||||
- The vocabulary of the graph is part of the glossary: Target architecture, Element, Relationship, Check, Requirements chain. "Dependency" keeps its existing meaning, between Deliverables.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# A change request starts from a baseline Execution
|
||||
|
||||
A user's requirements change after an architecture has been delivered. They need to say what has changed, at a high level, and get the architecture updated, not rebuilt from nothing with no link to what was agreed before. Today that is impossible by design: every Execution starts clean, and a Session or Customer case only groups Executions. We decide that a caller may start an Execution as a **Change request** against a **Baseline**: a completed Execution of the same Customer case. The new Execution begins from the Baseline's approved Deliverables and the user's description of what has changed, and it delivers a full, updated set plus a statement of what changed and why.
|
||||
|
||||
This is the one, explicit exception to "every Execution starts clean". It is named by the caller, limited to the same Customer case, and recorded in the Execution's manifest. Nothing flows between Executions unless a Baseline is named.
|
||||
|
||||
## How a Change request runs
|
||||
|
||||
- **The user says what changed,** in the Workflow request, at the same high level as an original request: "retention goes from 30 days to 7 years", "it must now also serve a second region". It becomes the next statement in the Customer case's Intent history.
|
||||
- **The agents work out the consequences.** The requirements analyst revises the Baseline's requirements, keeping every id that still holds and marking each requirement added, changed or removed. Each later Specialist starts from the Baseline's version of its Deliverable and changes only what the changed requirements reach.
|
||||
- **The Requirements chain shows the reach.** Because requirements, components, Elements and Checks carry ids ([ADR-0005](0005-the-target-architecture-is-an-explicit-graph.md)), a changed requirement leads to the Elements and Checks it touches. Everything it does not reach is carried over unchanged.
|
||||
- **The user approves the change, not the whole.** Each Approval shows what was added, changed and removed against the Baseline.
|
||||
- **Validation covers every Check,** carried over or new, so an unchanged part that a change has broken is still caught.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **A Change request is a new Execution,** with its own id, Approvals and files. The Baseline is never modified, so every delivered version of a Customer case's architecture remains on record.
|
||||
2. **The Baseline must be a completed Execution of the same Customer case.** One case can never start from another's work.
|
||||
3. **The output is complete, not a patch.** A Change request delivers the full updated Deliverables and a change summary, so the latest Execution of a case always stands on its own.
|
||||
4. **A Change request may itself be a Baseline,** so a case's Executions form a line of versions.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Reopen the completed Execution** and revise its Deliverables in place: rejected. It destroys the record of what was delivered and approved, and an Execution that never ends cannot be audited.
|
||||
- **Start clean and ask the user to restate everything** with the change included: rejected. It loses every decision and Approval already made, changes parts nobody asked to change, and gives no account of what differs.
|
||||
- **Carry context automatically within a Customer case:** rejected. Hidden carry-over is what "starts clean" exists to prevent; the caller must choose the Baseline.
|
||||
- **Deliver only a diff** of the Deliverables: rejected. The reader would have to assemble the current architecture from a chain of patches.
|
||||
|
||||
## Consequences
|
||||
|
||||
- "Every Execution starts clean" becomes "every Execution starts clean unless the caller names a Baseline". The glossary entries for Execution, Session and Customer case change to say so, and gain **Change request** and **Baseline**.
|
||||
- This depends on ADR-0005. Without ids along the Requirements chain there is nothing to trace a change through, and a Change request would be a rewrite with extra context.
|
||||
- A Change request against a Baseline produced by older Skills is revised under the Skills loaded now; the change summary can therefore include changes the user did not ask for. They must be shown as such.
|
||||
- Lesson candidates gain a strong signal: what users change after delivery.
|
||||
90
docs/adr/0007-jev-is-the-orchestrators-decision-point.md
Normal file
90
docs/adr/0007-jev-is-the-orchestrators-decision-point.md
Normal file
@@ -0,0 +1,90 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Jev is the Orchestrator's decision point, and the Orchestrator is a router
|
||||
|
||||
[ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev ([Jev in AI agent loops](https://vercel.com/i/jev-agent-control)): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.
|
||||
|
||||
## Who owns what
|
||||
|
||||
| Responsibility | Owner in the pattern | Here |
|
||||
|---|---|---|
|
||||
| Interpret the request | A model with supplied context | The requirements analyst and the Specialists after it |
|
||||
| Propose the next action | The decision model | Jev, asked typed questions about the Execution's state |
|
||||
| Check permission | Application code | The Ledger: Dependencies, Approvals, the Revision limit, the cloud-read gate |
|
||||
| Execute the action | The tool runner | The Orchestrator's existing tools, called by code |
|
||||
| Update state | Application code | The Execution graph and its checkpoint |
|
||||
| Explain the result | A generative model | One small model call that writes an Approval summary or a question, from data it is given |
|
||||
|
||||
## The Orchestrator is a router
|
||||
|
||||
With Jev as its decision point, the Orchestrator's job is routing: it reads what has arrived and the state of the Execution, asks Jev which route applies, and dispatches. It does no work of its own. There are four routes to choose.
|
||||
|
||||
| What is routed | The routes Jev chooses between | Who is dispatched |
|
||||
|---|---|---|
|
||||
| A message that starts something | a new Workflow request; a Change request against a Baseline; a question about work already delivered; not something this agent does | a full Execution; an Execution from the Baseline; the reply writer, from the delivered files; a polite refusal |
|
||||
| A message inside an Execution | an answer to a waiting question; a request to change submitted work, and which Deliverable; anything else | the Specialist that asked; a Revision of that Deliverable; a question back to the user |
|
||||
| A validation outcome | revise the code; revise the Target architecture; proceed | the IaC author; the target architect; the Approval |
|
||||
| Ready work, in a Change request | which Deliverables the change reaches, and which carry over from the Baseline unchanged | only the Specialists whose Deliverables are reached |
|
||||
|
||||
Which Specialist produces a given Deliverable is still fixed by the dependency graph: Jev chooses the route, never the worker for it. And the Ledger still refuses a route whose Dependencies or Approvals are not met.
|
||||
|
||||
## How a turn is decided: readings, disposition, playbook
|
||||
|
||||
The owner's model for this is a four-stage pipeline: readings of what arrived, a disposition worked out in code, further readings of the state, and a playbook that takes the first group that applies and, within it, the strongest action whose conditions hold, escalating when no group applies.
|
||||
|
||||
1. **Readings.** Facts about the turn. Most are read from the Ledger and the Execution's data by code: are questions waiting, is a Phase complete, is work ready, which Checks failed, did a Check fail twice running. The ones that need judgement are typed answers from Jev: is the user asking for a change (yes or no), which Deliverable would have to change first (a choice), how specific are they (a score), does a gap in the Target architecture leave a requirement unmet (yes or no). Jev reads; it never picks the action.
|
||||
2. **Disposition.** Code turns a judged reading into act, queue or close by threshold: a probability of 0.75 or more is acted on, 0.15 or less is dropped, and anything between is put to the user. A choice with confidence under 0.6 is not relied on. The thresholds are settings.
|
||||
3. **Playbook.** Ordered groups; the first that applies is taken, and within it the strongest action whose conditions hold:
|
||||
|
||||
| Group | Actions, strongest first |
|
||||
|---|---|
|
||||
| The user asked for a change | revise the Deliverable it affects; else ask them to confirm which and what |
|
||||
| Questions are waiting | relay them |
|
||||
| Validation decided | revise the Target architecture (a Check failed twice running, or a gap that matters); else revise the code (a Check failed); else ask about the gaps; else proceed |
|
||||
| A Phase is complete | request its Approval |
|
||||
| Work is ready | delegate all of it |
|
||||
| Packaging is approved | finish |
|
||||
| No group applies | escalate: ask the user, or end with warnings when there is none |
|
||||
|
||||
4. **Permission.** Every action goes through the Orchestrator's tools, so the Ledger refuses what is not allowed whatever was decided, and the playbook then takes the next action that is. A typed answer can still be wrong.
|
||||
5. **Record.** Each decision is kept in the Execution's state: the turn, the group, the action and what Jev answered with its confidence.
|
||||
6. **Reply.** A generative model writes only what the user reads, the summary they approve a Phase from, out of data it is given. It runs after the decision and cannot change it. When it fails, the summary is assembled from the same data.
|
||||
|
||||
A failed call to Jev is not an answer. The client returns nothing, and the reading falls back to a rule that acts only on what is certain: a change is acted on when the Deliverable is named outright, and a gap is revised for when it names a real requirement or cannot be verified.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **No language model decides what the Orchestrator does.** Jev reads, and the playbook in code decides. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
|
||||
2. **Low confidence goes to the user, not to another model.** Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
|
||||
3. **Jev being unreachable does not stop an Execution.** After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
|
||||
4. **What is sent to Jev is the Execution's state, not its content:** Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
|
||||
5. **The Orchestrator routes; Jev chooses the route.** That covers what kind of work a message starts, where a message inside an Execution goes, where a validation outcome goes, and, in a Change request, which Deliverables are reached. Jev never chooses which Specialist produces a Deliverable: the dependency graph fixes that.
|
||||
6. **In a new Execution every Deliverable is still produced.** Routing does not skip a Specialist for a new Workflow request, because each later Deliverable is checked against the ones before it. Skipping happens only in a Change request, where the Baseline supplies what is carried over.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **A smaller language model as Orchestrator:** rejected by the owner. It keeps the slow, open-ended turn and the stalls, at lower cost.
|
||||
- **No generative model at all, with fixed wording for summaries and questions:** this ADR's first draft. Rejected to follow the pattern, which pairs Jev with a generative model for replies: an Approval summary is what a person decides from and reads better written than assembled.
|
||||
- **Code only, with fixed rules:** rejected. It cannot tell what a user's free-text reply is asking for.
|
||||
- **Code for the mechanical steps and the existing model for judgement:** rejected in favour of one decision point. It keeps two ways of deciding and the large model in the loop.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Jev is reached through TypeSafe AI's Python SDK, `typesafe-sdk`, pinned exactly (owner, 2026-10-07). `app/tools/jev.py` is the only module that imports it.
|
||||
- The Orchestrator as a language-model agent is kept behind the setting `ORCHESTRATOR=model`, for the tests that drive the Ledger's rules in arbitrary order and for comparing the two in an evaluation. The default is `router`. Remove the setting once the router is proven.
|
||||
|
||||
- The Orchestrator's instructions in `agents/orchestrator.md` become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.
|
||||
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
|
||||
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.
|
||||
- An unusual request that fits none of the questions is handled by asking the user, where a language model might have improvised.
|
||||
|
||||
## As built (2026-10-09)
|
||||
|
||||
- **Jev is used beyond the Orchestrator.** It also chooses the pillars a change touches and the Skill references a Specialist reads first, checks remote MCP replies and the user's text for instructions aimed at the agent, tells a rewording from a change, and scores each Execution's design recommendations. Each has a fallback in code and is a setting.
|
||||
- **Decision 4 covers Orchestrator decisions only.** These other uses do send content: design recommendations, Check and Element text, remote MCP replies and the Workflow request. Jev's data retention is therefore a question about Deliverable content, and it is still unanswered by the vendor.
|
||||
- **Jev did not become a hard dependency.** Without a key, or when a call fails, every decision takes its fallback and the Execution continues.
|
||||
- **The fourth route, which Deliverables a change reaches, is decided in code** from the changed requirements and the Requirements chain, not by Jev.
|
||||
59
docs/adr/0008-scoping-starts-from-a-scope-spec.md
Normal file
59
docs/adr/0008-scoping-starts-from-a-scope-spec.md
Normal file
@@ -0,0 +1,59 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Scoping starts from a scope spec
|
||||
|
||||
An Execution began from the customer's request, a sentence or two saved word for word, and the requirements analyst filled everything else with assumptions. Most of what went wrong downstream traced to what nobody had stated, and nothing told the customer how much of their design rested on assumption. We decide that an engagement is scoped from a **scope spec**: a structured file that states what the customer wants and the facts a design depends on, which may arrive complete, partly filled or not at all, and which the customer approves before requirements begin.
|
||||
|
||||
The Intent stays the customer's own words. The scope spec sits beside it as the agreed, structured form.
|
||||
|
||||
## What the spec is
|
||||
|
||||
- **A file of fields from a catalogue.** A Skill holds the catalogue: each field's type, when it must be settled, the question to ask for it, and guidance for a default. The catalogue names no Cloud provider. A provider's bundle supplies the allowed values for the fields it leaves to a provider, in that provider's terms.
|
||||
- **Every field says one of four things.** A value is the customer's statement. `ask`, or the field left out, means not known. `decide` leaves the choice to the agent, recorded as an assumption. `not_applicable` asks and assumes nothing.
|
||||
- **Whether a field is settled is read by code.** No model decides what is missing.
|
||||
- **Fields are essential, important or optional,** and which is which depends on the kind of engagement: a workload, a platform foundation or a migration.
|
||||
- **It carries the customer's own words and requirements:** the statement verbatim, the requirements they stated with their own ids, and what they excluded.
|
||||
- **It carries how to run the agent,** so one file starts an Execution.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-11, with each of these:
|
||||
|
||||
1. **The file supplied is never altered.** The agent writes a completed copy beside it, with what it assumed, what is still unknown, where each value came from, and a status.
|
||||
2. **A choice the customer states binds the design.** A product, a topology or a setting written in the spec is a constraint every Specialist honours. The rule that requirements name no products is about what the agents introduce, not what the customer has already decided.
|
||||
3. **A platform foundation is in scope,** alongside the application workloads the agent was built on.
|
||||
4. **An engagement and a Customer case are the same thing.** A spec may name either; an engagement reference is turned into the case's id.
|
||||
5. **The spec is provider-agnostic in shape.** Provider vocabulary enters only as values a provider's bundle allows.
|
||||
6. **An essential field that is not settled refuses a start that has nobody to ask.** With someone present it is asked for.
|
||||
7. **Scoping becomes a Phase the customer approves,** before requirements discovery.
|
||||
8. **Where the spec and a prose request disagree, the spec wins.**
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Judge the free-text request against a list of topics** with a model or Jev: the first proposal. Rejected once a spec was on the table: a field that is filled or not is a fact, and a judgement about prose is not.
|
||||
- **One file the agent completes in place:** rejected. An author must be able to see that what they wrote is what was used.
|
||||
- **Keep product names out of the spec, as out of requirements:** rejected. A customer who has decided on an interconnect or a named appliance has stated a constraint; refusing to record it would lose it.
|
||||
- **A separate spec for platform foundations:** rejected. One catalogue with tiers by kind serves both, and a migration is usually some of each.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The Requirements chain gains a start: a requirement traces to the customer's own requirement id or to a spec field.
|
||||
- The requirements analyst stops being where questions are asked and assumptions made. That moves to scoping.
|
||||
- A Change request may be an edited spec, compared field by field in code.
|
||||
- An evaluation case may be a spec, in place of answers matched by pattern.
|
||||
- The glossary gains **Scope spec** and **Scoping**, and Customer case says that an engagement is the same thing.
|
||||
- A platform foundation is new work for the agents. It needs its own evaluation case before it is relied on.
|
||||
- The spec holds client names and details. Like lesson candidates it is client data: kept with the Customer case, never in the repository, apart from the fictional benchmark specs.
|
||||
|
||||
This is built in stages; `.agents/plans/iter-09-scoping/` says which are done.
|
||||
|
||||
## As built (2026-10-12)
|
||||
|
||||
- Built: the catalogue and reading a spec in code; scoping as the first Phase, completed in conversation or by assumption; a spec drafted from a sentence; a stated choice refused unless a requirement carries it; the status following the Execution; a Change request starting from the delivered scope.
|
||||
- **Answers are given in conversation.** Supplying an edited file part-way through is not handled.
|
||||
- **A Change request is still stated in prose.** Comparing an edited spec with the delivered one, field by field, is not built.
|
||||
- **Exclusions are not checked in code.** They are free text; the Specialists are told, and in the first landing-zone run they complied.
|
||||
- **An organisation's standards are not part of the spec.** They still arrive as overlay Skills ([ADR-0004](0004-lessons-and-overlays-arrive-as-skills.md)). Carrying them in the spec as stated requirements was discussed on 2026-10-12 and left undecided; it would amend that ADR.
|
||||
- A platform foundation has run twice with real models, as `case-08-landing-zone`.
|
||||
51
docs/agents/domain.md
Normal file
51
docs/agents/domain.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Domain Docs
|
||||
|
||||
How the engineering skills should consume this repo's domain documentation when exploring the codebase.
|
||||
|
||||
## Before exploring, read these
|
||||
|
||||
- **`CONTEXT.md`** at the repo root, or
|
||||
- **`CONTEXT-MAP.md`** at the repo root if it exists: it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
|
||||
- **`docs/adr/`**: read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
|
||||
|
||||
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
|
||||
|
||||
## File structure
|
||||
|
||||
Single-context repo (most repos):
|
||||
|
||||
```
|
||||
/
|
||||
├── CONTEXT.md
|
||||
├── docs/adr/
|
||||
│ ├── 0001-event-sourced-orders.md
|
||||
│ └── 0002-postgres-for-write-model.md
|
||||
└── src/
|
||||
```
|
||||
|
||||
Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
|
||||
|
||||
```
|
||||
/
|
||||
├── CONTEXT-MAP.md
|
||||
├── docs/adr/ ← system-wide decisions
|
||||
└── src/
|
||||
├── ordering/
|
||||
│ ├── CONTEXT.md
|
||||
│ └── docs/adr/ ← context-specific decisions
|
||||
└── billing/
|
||||
├── CONTEXT.md
|
||||
└── docs/adr/
|
||||
```
|
||||
|
||||
## Use the glossary's vocabulary
|
||||
|
||||
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
|
||||
|
||||
If the concept you need isn't in the glossary yet, that's a signal: either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
|
||||
|
||||
## Flag ADR conflicts
|
||||
|
||||
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
|
||||
|
||||
> _Contradicts ADR-0007 (event-sourced orders), but worth reopening because…_
|
||||
30
docs/agents/issue-tracker.md
Normal file
30
docs/agents/issue-tracker.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Issue tracker: Local Markdown
|
||||
|
||||
Issues and specs for this repo live as markdown files in `.scratch/`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- One feature per directory: `.scratch/<feature-slug>/`
|
||||
- The spec is `.scratch/<feature-slug>/spec.md`
|
||||
- Implementation issues are one file per ticket at `.scratch/<feature-slug>/issues/<NN>-<slug>.md`, numbered from `01`, never a single combined tickets file
|
||||
- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings)
|
||||
- Comments and conversation history append to the bottom of the file under a `## Comments` heading
|
||||
|
||||
## When a skill says "publish to the issue tracker"
|
||||
|
||||
Create a new file under `.scratch/<feature-slug>/` (creating the directory if needed).
|
||||
|
||||
## When a skill says "fetch the relevant ticket"
|
||||
|
||||
Read the file at the referenced path. The user will normally pass the path or the issue number directly.
|
||||
|
||||
## Wayfinding operations
|
||||
|
||||
Used by `/wayfinder`. The **map** is a file with one **child** file per ticket.
|
||||
|
||||
- **Map**: `.scratch/<effort>/map.md` (the Notes / Decisions-so-far / Fog body).
|
||||
- **Child ticket**: `.scratch/<effort>/issues/NN-<slug>.md`, numbered from `01`, with the question in the body. A `Type:` line records the ticket type (`research`/`prototype`/`grilling`/`task`); a `Status:` line records `claimed`/`resolved`.
|
||||
- **Blocking**: a `Blocked by: NN, NN` line near the top. A ticket is unblocked when every file it lists is `resolved`.
|
||||
- **Frontier**: scan `.scratch/<effort>/issues/` for files that are open, unblocked, and unclaimed; first by number wins.
|
||||
- **Claim**: set `Status: claimed` and save before any work.
|
||||
- **Resolve**: append the answer under an `## Answer` heading, set `Status: resolved`, then append a context pointer (gist + link) to the map's Decisions-so-far in `map.md`.
|
||||
15
docs/agents/triage-labels.md
Normal file
15
docs/agents/triage-labels.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Triage Labels
|
||||
|
||||
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker.
|
||||
|
||||
| Label in mattpocock/skills | Label in our tracker | Meaning |
|
||||
| -------------------------- | -------------------- | ---------------------------------------- |
|
||||
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
|
||||
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||
| `wontfix` | `wontfix` | Will not be actioned |
|
||||
|
||||
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
|
||||
|
||||
Edit the right-hand column to match whatever vocabulary you actually use.
|
||||
68
docs/operations/mcp-access.md
Normal file
68
docs/operations/mcp-access.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Remote MCP access: Google Cloud
|
||||
|
||||
How to give the solution architect agent safe, read-only access to Google Cloud's hosted MCP servers. The servers the agent uses, and the only tools it may call on each, are listed in [`providers/gcp.yaml`](../../providers/gcp.yaml).
|
||||
|
||||
## How the agent limits itself
|
||||
|
||||
- **Allowlist.** An agent is only ever offered tools named in a server's `allow` list. Every other tool on the server is invisible to it.
|
||||
- **Read-only check at startup.** If any allowlisted tool doesn't declare `readOnlyHint: true`, the agent refuses to start.
|
||||
- **Content excluded.** Tools that return data content are excluded even though they are read-only: Cloud SQL `execute_sql_readonly`, Cloud Storage `read_object`/`read_text`/`list_objects`, and generic Kubernetes reads that can return Secrets (`get_k8s_resource`, `describe_k8s_resource`).
|
||||
- **Permission per Execution.** Tools that read your environment (`cloud_read`) run only after the user grants permission in that Execution. Pre-approved Executions deny it unless the caller passes `allow_cloud_read=true`, in the request or in the scope spec's `run` section. When it is denied, the cloud-read tools are not offered to the agents at all.
|
||||
|
||||
- **Replies are checked.** When Jev is configured, a server's reply that reads as instructions to the agent, as opposed to information, is withheld from the Specialist and logged (`JEV_GUARDRAIL`). Without Jev, replies pass through.
|
||||
|
||||
These are defence in depth. Enforce read-only access in Google Cloud as well.
|
||||
|
||||
## Set up a project the agent may read
|
||||
|
||||
1. **Enable the APIs** of the products you want discovered. A product's MCP endpoint is available once its API is enabled, for example `compute.googleapis.com`, `container.googleapis.com`, `run.googleapis.com`, `sqladmin.googleapis.com`, `pubsub.googleapis.com`, `cloudasset.googleapis.com`, `monitoring.googleapis.com` and `networkmanagement.googleapis.com`. A disabled API shows up as a recorded discovery gap, not a failure.
|
||||
2. **Create a dedicated service account** for the agent, and grant it only:
|
||||
- `roles/mcp.toolUser`, to call MCP tools
|
||||
- `roles/viewer`, or narrower viewer roles for the products in scope
|
||||
- `roles/cloudasset.viewer`, for asset inventory and IAM analysis
|
||||
3. **Deny read-write tool calls** for every principal with an IAM deny policy. Save the policy as `policy.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"rules": [
|
||||
{
|
||||
"denyRule": {
|
||||
"deniedPrincipals": ["principalSet://goog/public:all"],
|
||||
"deniedPermissions": ["mcp.googleapis.com/tools.call"],
|
||||
"denialCondition": {
|
||||
"title": "Deny read-write tools",
|
||||
"expression": "api.getAttribute('mcp.googleapis.com/tool.isReadOnly', false) == false"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Then attach it to the project:
|
||||
|
||||
```bash
|
||||
gcloud iam policies create mcp-deny-policy \
|
||||
--attachment-point=cloudresourcemanager.googleapis.com/projects/PROJECT_ID \
|
||||
--kind=denypolicies \
|
||||
--policy-file=policy.json
|
||||
```
|
||||
|
||||
4. **Give the agent credentials.** It authenticates with Application Default Credentials: the attached service account on Cloud Run or GKE, or `GOOGLE_APPLICATION_CREDENTIALS` locally. Set `MCP_QUOTA_PROJECT` when quota should be charged to a project other than the credentials' project.
|
||||
|
||||
For local development, `gcloud auth application-default login` supplies ADC for your user.
|
||||
|
||||
## Reviewing the allowlists
|
||||
|
||||
Upstream tool sets change. To review them:
|
||||
|
||||
1. Refresh the snapshot. `tools/list` works without credentials, and the script lists tools without calling any:
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/list-mcp-tools.py gcp --json --no-auth > providers/reviews/gcp-mcp-tools.json
|
||||
```
|
||||
|
||||
2. Read the diff. For each new read-only tool, decide whether it returns inventory or configuration metadata (allow it) or data content or secrets (leave it out).
|
||||
3. Update the `allow` lists in `providers/gcp.yaml`. `tests/test_providers.py` fails if an allowlisted tool is missing from the snapshot or isn't read-only.
|
||||
|
||||
Last reviewed 2026-09-16, against all 12 servers.
|
||||
Reference in New Issue
Block a user