docs: bring the README, runbook, glossary and ADR notes up to date
Some checks failed
validation / verify (push) Failing after 11s

The README now covers the router, Change requests, overlay Skills, Jev's
uses and the current layout. VALIDATION.md no longer points at the removed
root Terraform. ADR-0004 and ADR-0007 gain dated "As built" notes; ADR-0007's
says which uses of Jev send Deliverable content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 21:33:38 +01:00
parent e5e1647ff1
commit a8ed21b89d
7 changed files with 65 additions and 19 deletions

View File

@@ -49,7 +49,7 @@ _Avoid_: review round, retry, loop iteration
### Agents
**Orchestrator**:
The top-level agent of an Execution, and the only agent that talks to the user. It reasons about which Deliverables are ready to produce, delegates them to Specialists, and asks the user for Approvals.
The top-level agent of an Execution, and the only agent that talks to the user. It routes: it works out which Deliverables are ready to produce, delegates them to Specialists, and asks the user for Approvals. It produces no Deliverable itself.
_Avoid_: supervisor, manager, root agent, router
**Specialist**:

View File

@@ -1,3 +1,3 @@
# Contributing
Keep requirements product-neutral until the architecture phase. Do not commit state files, plans, credentials, secret values, or real project identifiers. Update the guide and validation evidence when changing Terraform. Run the commands in `VALIDATION.md` before proposing a change.
Read `CLAUDE.md` and `.agents/plans/README.md` first. Behaviour belongs in Skills and cloud-specific knowledge in Skills and the provider bundle, never in the agent's core. Keep requirements product-neutral until the architecture phase. Do not commit state files, plans, credentials, secret values, real project identifiers or lesson candidates, which are client data. Never edit a vendored Skill. Run the commands in `VALIDATION.md` before proposing a change, and record a finished step in `.agents/plans/`.

View File

@@ -1,6 +1,6 @@
# GCP Solution Architecture Agent (`gcp_solution_architecture_agent`)
A hierarchical ReAct agent that turns a request for a cloud workload into a reviewed solution architecture: requirements, a target design with diagram and infrastructure-as-code, validation results, and a packaged guide. It replaces the [`google-cloud-solution-architecture`](https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/SKILL.md) skill with an agent that runs anywhere. Google Cloud is the first Cloud provider.
A hierarchical agent that turns a request for a cloud workload into a reviewed solution architecture: requirements, a target design with diagram and infrastructure-as-code, validation results, and a packaged guide. It replaces the [`google-cloud-solution-architecture`](https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/SKILL.md) skill with an agent that runs anywhere. Google Cloud is the first Cloud provider.
Vocabulary is defined in [CONTEXT.md](CONTEXT.md), decisions in [docs/adr/](docs/adr/), and the plan and history of each iteration in [.agents/plans/](.agents/plans/README.md).
@@ -9,7 +9,7 @@ Vocabulary is defined in [CONTEXT.md](CONTEXT.md), decisions in [docs/adr/](docs
## 🌟 Architecture & Features
### 1. Orchestrator and Specialists (`agents/`)
One Orchestrator coordinates an Execution and is the only agent that talks to the user. It delegates each Deliverable to one Specialist, and independent Deliverables run in parallel:
One Orchestrator coordinates an Execution and is the only agent that talks to the user. It is a router ([ADR-0007](docs/adr/0007-jev-is-the-orchestrators-decision-point.md)): code reads the Execution's state, a decision model (Jev) answers the closed questions that need judgement, and a playbook in code picks the action. A language model only writes what the user reads. It delegates each Deliverable to one Specialist, and independent Deliverables run in parallel:
- **Requirements discovery:** `requirements-analyst`, `decomposer`.
- **Solution architecture (design):** `product-selector`, `design-advisor` (which consults one `pillar-advisor` per architecture-framework pillar), `target-architect`.
- **Solution implementation:** `describer`, `iac-author`.
@@ -18,6 +18,8 @@ One Orchestrator coordinates an Execution and is the only agent that talks to th
Each Phase needs the user's explicit Approval before work that depends on it starts. A pre-approved mode serves batch and evaluation runs.
A **Change request** ([ADR-0006](docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md)) starts from a completed Execution in the same Customer case, its Baseline, and delivers the full updated Deliverables with `changes.json`: what changed, what was only reworded, and anything changed that nobody asked for.
The design Phase ends with the **Target architecture**: one explicit graph of Elements, Relationships and Checks ([ADR-0005](docs/adr/0005-the-target-architecture-is-an-explicit-graph.md)). The user approves it before any infrastructure code is written. The diagram is drawn from it by code, the Terraform must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations each carry ids in a data file, so every requirement can be traced through to its verdict; a submission with a missing link is refused.
### 2. LangGraph state, ADK agents (`app/`)
@@ -33,17 +35,28 @@ Execution records (status, Ledger events, submitted Deliverables) go to PostgreS
Behaviour lives in Agent Skills (agentskills.io format) at the repository root, per ADR-0002:
- `skills/vendor/google/`: the upstream solution-architecture Skill and six Well-Architected Framework pillar Skills, vendored unmodified.
- `skills/cloud-discovery/`, `skills/iac-validation/` and `skills/requirements-chain/`: our own Skills for the tooling and the data files.
- `skills/google-cloud-lessons/`: lessons learned from earlier Executions, written by the lessons loop (`app/learning/`, `eval/curate_lessons.py`) from proposals a person approved. Empty until one is.
- `skills/google-cloud-lessons/`: lessons learned from earlier Executions, written by the lessons loop (`app/learning/`, `eval/curate_lessons.py`) from proposals a person approved ([ADR-0004](docs/adr/0004-lessons-and-overlays-arrive-as-skills.md)). Set `LESSONS_ENABLED=false` to run without them.
- `skills/overlays/`: overlay Skills, each an organisation's standards or one engagement's constraints. An Execution names the ones it wants; `DEFAULT_OVERLAY_SKILLS` names the ones every Execution loads. The one here is an example, not anyone's policy.
- `skills/google-cloud-iac-patterns/`: Terraform patterns for Google Cloud products, each validated by a test against the provider version `providers/gcp.yaml` pins.
`app/skills/agent_skills.py` discovers and validates them; the ADK runtime (`app/adk/adk_runtime.py`) loads them on demand. `providers/gcp.yaml` assigns Skills, tools, MCP servers and models to each role in `agents/`.
### 5. A2A Server (`app/main.py`)
### 5. Decisions by Jev (`app/tools/jev.py`)
Jev is a hosted decision model that answers typed, closed questions with a probability. It is optional: without `TYPESAFE_API_KEY` every decision takes its fallback in code. Beyond routing the Orchestrator it is used to:
- read a message that starts something: a new request, a Change request, or a question (`app/nodes/intake.py`);
- choose which pillar advisors a change needs consulting again, and which Skill references a Specialist should read first;
- withhold a remote MCP server's reply that reads as instructions to the agent, and flag the same in the user's text;
- tell a rewording from a real change in a Change request;
- score each Execution's design recommendations when it ends, kept in `execution.json` under `assessment`.
Every setting, with its threshold, is described in `.env.example`. What each use measured against the live service is in `.agents/plans/iter-05-jev-decisions/`.
### 6. A2A Server (`app/main.py`)
The container starts the hierarchical Execution engine (ADR-0003) behind an A2A server:
- `POST /a2a`: A2A JSON-RPC for interactive, multi-turn Executions with Approvals.
- `GET /.well-known/agent-card.json`: A2A agent card.
- `GET /health`: Health signal, the Cloud provider and any MCP discovery gaps.
- `POST /generate`: One-shot, pre-approved Execution for batch and evaluation callers. Pass `customer_case` to group it with the case's other Executions; over A2A, send `customer_case` in the message metadata.
- `POST /generate`: One-shot, pre-approved Execution for batch and evaluation callers. Pass `customer_case` to group it with the case's other Executions, `baseline` (an Execution id in that case) to make it a Change request, and `overlay_skills` (a list of names) to apply overlay Skills. Over A2A, send the same fields in the message metadata.
The original linear pipeline and its REST routes (`/card`, `/validate`, `/sessions/{session_id}`) were removed on 2026-10-06; see `.agents/plans/iter-03-kab-layout/`.
@@ -66,6 +79,8 @@ pytest -q
python3 -m eval.solution_eval
```
Copy `.env.example` to `.env` for local settings. The server and the evaluation read it at start-up, and export its model credentials (`GOOGLE_*`) to the process.
### Start the A2A Server
```bash
python3 -m app.main --host 0.0.0.0 --port 8080
@@ -127,18 +142,22 @@ curl -X POST http://localhost:8080/generate \
gcp_solution_architecture_agent/
├── agents/ # Framework-neutral role files, one per agent
├── providers/ # Cloud provider bundles (gcp.yaml)
├── skills/ # Agent Skills: vendored upstream + our tooling Skills
├── skills/ # Agent Skills: vendored upstream, our own, lessons, overlays
├── app/ # KAB workflow-agent layout
│ ├── main.py # A2A server entrypoint (python -m app.main)
│ ├── a2a_server.py # Maps A2A tasks to Executions and their prompts
│ ├── agent.py # ExecutionEngine and its assembly (composition root)
│ ├── baseline.py # Loading the Execution a Change request starts from
│ ├── card.py # A2A agent card
│ ├── states/ # ExecutionState and the Ledger
│ ├── nodes/ # Execution graph nodes (orchestrator, specialist, user)
│ ├── workflows/ # Execution graph wiring and the Deliverable dependency graph
│ ├── nodes/ # Execution graph nodes, the router and message intake
│ ├── workflows/ # Execution graph wiring, the Deliverable dependency graph,
│ │ # the Requirements chain and change summaries
│ ├── adk/ # Agent-runtime interface, ADK runtime, role loader
│ ├── skills/ # Agent Skills discovery and validation
│ ├── tools/ # Execution tools, local tools, remote MCP, pillar fan-out
│ ├── tools/ # Execution tools, local tools, remote MCP, pillar fan-out,
│ │ # the Jev client and each decision that uses it
│ ├── learning/ # The lessons loop: evidence, proposals, publishing, proof
│ ├── providers.py # Provider bundle loader
│ ├── recorder.py # Execution recording
│ ├── checkpointing.py # LangGraph checkpointer selection
@@ -148,7 +167,10 @@ gcp_solution_architecture_agent/
├── eval/ # Evaluation
│ ├── solution_eval.py # Real-model evaluation of the Execution engine
│ ├── scoring.py # Checks applied to a finished Execution
│ └── datasets/ # Benchmark cases (JSON)
│ ├── judge.py # The judges for the judged checks (Jev, or a model)
│ ├── calibrate_judge.py # Judges against a person's labels, and against each other
│ ├── curate_lessons.py # propose, publish, prove and retire lessons
│ └── datasets/ # Benchmark cases (JSON) and the judge's labelled set
├── deliverables/
│ └── cases/ # Everything generated, grouped by Customer case
│ └── <customer_case>/
@@ -156,14 +178,17 @@ gcp_solution_architecture_agent/
│ │ ├── 001.md, 002.md # Each request as the customer stated it, in order
│ │ └── history.json # Every request, the Execution it started and how it ended
│ └── executions/<execution_id>/
│ ├── execution.json # Request, status and files of this Execution
│ ├── execution.json # Request, status, Baseline, overlay Skills,
│ │ # risks, assessment and files of this Execution
│ ├── solution-architecture-guide.md
│ ├── deliverables/<kind>.md # One per Deliverable
│ ├── requirements.json, components.json, products.json, recommendations.json
│ ├── target-architecture.json # Elements, Relationships and Checks
│ ├── architecture.mmd # Drawn from the graph
│ ├── terraform/, implementation.json # The code, and which part meets each Check
│ └── verdicts.json # Validation's verdict on each Check
│ ├── verdicts.json # Validation's verdict on each Check
│ ├── working.json # What a Change request carries forward
│ └── changes.json # In a Change request: what changed and why
├── lessons/candidates/<customer_case>/ # Lesson candidates per Execution (gitignored)
├── docs/ # ADRs, agent-skill conventions and operations notes
├── scripts/ # Vendored-Skill sync, MCP tool listing

View File

@@ -9,16 +9,19 @@ pytest -v
# 2. Real-model evaluation of the engine (needs model credentials; costs usage)
python3 -m eval.solution_eval
# 3. Optional Terraform formatting & validation
terraform -chdir=terraform fmt -check -recursive
terraform -chdir=terraform init -backend=false
terraform -chdir=terraform validate
# 3. Optional: validate the Terraform an Execution generated
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform init -backend=false
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform validate
```
`init -backend=false` may download the pinned provider constraint but does not create infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
The repository holds no Terraform of its own. The tests need the `terraform` binary (1.14.3 in CI) to validate the code patterns in `skills/google-cloud-iac-patterns/`. `init -backend=false` downloads the pinned provider and creates no infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
## Evaluation options
- `python3 -m eval.solution_eval --judge` adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations.
- `python3 -m eval.solution_eval case-04-interactive-web-app` runs one case with a scripted user in the loop.
- Several cases run side by side as separate processes; each writes its own folder under `eval/results/`, with a progress file while it runs.
- A case with `change` runs its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case with `overlay_skills` names overlay Skills, and `code_standards` lists what the generated code must and must not contain.
- `LESSONS_ENABLED=false python3 -m eval.solution_eval <case>` runs without the lessons Skill. `python3 -m eval.curate_lessons prove --without <report.json> --with <report.json>` compares the two and records the result in the lesson proposals.
- `python3 -m eval.calibrate_judge` measures the judges' stability and agreement. Agreement with a person needs the labels in `eval/datasets/judge/labels.json` filled in.
- A case's `thresholds` in `eval/datasets/solution_cases.json` bound its revisions, seconds and tokens; exceeding one is reported as a regression.

View File

@@ -35,3 +35,12 @@ Accepted by the owner on 2026-10-08, agreeing to each of these:
- The evaluation must be able to see a difference a Skill makes. Its structural checks cannot, so the judged checks and the thresholds on Revisions, time and tokens become part of the loop.
- The glossary gains **Lesson candidate**, **Lesson** and **Overlay Skill**.
- Skills can no longer be loaded once at start-up for all Executions: the runtime must resolve a role's Skills per Execution.
## As built (2026-10-09)
Two points differ from the text above; the decisions stand.
- **Default overlays are a deployment setting, not part of the provider bundle.** `DEFAULT_OVERLAY_SKILLS` names the overlays every Execution loads. Whose standards apply belongs to the deployment and the caller, not to a Cloud provider.
- **Skills are still loaded once at start-up.** The runtime loads every provider Skill and every overlay in the configured folders when it starts, and an Execution chooses among them. An overlay added to a folder needs a restart.
Decision 2 has a first result: on one benchmark case run each way, the lessons held quality and removed a Revision. One lesson was retired because the fault was in a review, not in the agents.

View File

@@ -81,3 +81,10 @@ Accepted by the owner on 2026-10-07, with each of these:
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.
- An unusual request that fits none of the questions is handled by asking the user, where a language model might have improvised.
## As built (2026-10-09)
- **Jev is used beyond the Orchestrator.** It also chooses the pillars a change touches and the Skill references a Specialist reads first, checks remote MCP replies and the user's text for instructions aimed at the agent, tells a rewording from a change, and scores each Execution's design recommendations. Each has a fallback in code and is a setting.
- **Decision 4 covers Orchestrator decisions only.** These other uses do send content: design recommendations, Check and Element text, remote MCP replies and the Workflow request. Jev's data retention is therefore a question about Deliverable content, and it is still unanswered by the vendor.
- **Jev did not become a hard dependency.** Without a key, or when a call fails, every decision takes its fallback and the Execution continues.
- **The fourth route, which Deliverables a change reaches, is decided in code** from the changed requirements and the Requirements chain, not by Jev.

View File

@@ -9,6 +9,8 @@ How to give the solution architect agent safe, read-only access to Google Cloud'
- **Content excluded.** Tools that return data content are excluded even though they are read-only: Cloud SQL `execute_sql_readonly`, Cloud Storage `read_object`/`read_text`/`list_objects`, and generic Kubernetes reads that can return Secrets (`get_k8s_resource`, `describe_k8s_resource`).
- **Permission per Execution.** Tools that read your environment (`cloud_read`) run only after the user grants permission in that Execution. Pre-approved Executions deny it unless the caller passes `allow_cloud_read=true`.
- **Replies are checked.** When Jev is configured, a server's reply that reads as instructions to the agent, as opposed to information, is withheld from the Specialist and logged (`JEV_GUARDRAIL`). Without Jev, replies pass through.
These are defence in depth. Enforce read-only access in Google Cloud as well.
## Set up a project the agent may read