This commit is contained in:
65
.agents/plans/iter-02-solution-architect-agent/DETAIL.md
Normal file
65
.agents/plans/iter-02-solution-architect-agent/DETAIL.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# Plan Detail
|
||||
|
||||
The spec is the full design. Each entry here gives the step's purpose, the rule it must keep, and its acceptance criteria. Paths are current (post iteration 3).
|
||||
|
||||
## Step 1: Environment and baseline
|
||||
|
||||
- **Capability:** A reproducible environment and a recorded test baseline to compare every later step against.
|
||||
- **Rationale:** Later steps claim "the suite stays green"; that needs a starting count.
|
||||
- **Where:** `.venv/`, `requirements.txt`, `requirements-dev.txt`.
|
||||
- **Acceptance criteria:** The existing suite runs and its result is recorded in the spec's comments.
|
||||
|
||||
## Step 2: Vendored Skills and provider bundle
|
||||
|
||||
- **Capability:** Upstream Skills available unmodified, and a Cloud provider expressed as configuration.
|
||||
- **Rationale:** ADR-0002. Upstream changes must arrive as a diff, and a second Cloud provider must be addable without touching the core.
|
||||
- **Where:** `skills/vendor/google/` (pinned to upstream commit `81a31a6`, with `VENDOR.md`, `LICENSE`, `SHA256SUMS`), `scripts/sync-vendored-skills.sh`, `providers/gcp.yaml`, `app/providers.py`, `app/skills/agent_skills.py`.
|
||||
- **Acceptance criteria:** Vendored files match their checksums. Every Skill under `skills/` passes the Agent Skills frontmatter rules. Only `app/providers.py` names a provider.
|
||||
|
||||
## Step 3: Deliverables and Ledger
|
||||
|
||||
- **Capability:** The dependency graph of Deliverables as data, and a pure Ledger that enforces acceptance, Approval and Revision rules.
|
||||
- **Rationale:** ADR-0003. Dependencies are guaranteed by code; ordering is the model's choice.
|
||||
- **Where:** `app/workflows/deliverables.py` (the only place Phases, Deliverables and Dependencies are declared), `app/states/ledger.py`.
|
||||
- **Acceptance criteria:** A Dependency inside a Phase is met on submission; across Phases only on Approval. Reopening a Deliverable returns it and its dependents to draft and withdraws affected Approvals. Revisions are capped.
|
||||
|
||||
## Step 4: Execution graph
|
||||
|
||||
- **Capability:** A checkpointed LangGraph Execution that pauses for the user, behind one `ExecutionEngine` interface.
|
||||
- **Rationale:** ADR-0003. Resumable, recordable progress at every step, with framework types kept out of state.
|
||||
- **Where:** `app/adk/runtime.py` (interface), `app/adk/fake_runtime.py`, `app/states/execution_state.py`, `app/tools/execution.py`, `app/nodes/execution_nodes.py`, `app/workflows/execution_graph.py`, `app/agent.py`, `app/recorder.py`, `app/checkpointing.py`.
|
||||
- **Acceptance criteria:** Only `orchestrator_turn` and `await_user` change the Ledger. No Approval is ever granted from free text. Cloud-read tools are unreachable without permission. All of it is tested with the fake runtime and no model.
|
||||
|
||||
## Step 5: ADK runtime and role files
|
||||
|
||||
- **Capability:** Real ADK `LlmAgent`s running role files with scoped Skills, behind the runtime interface.
|
||||
- **Rationale:** ADR-0003. Each agent runs its own ReAct loop; Skills load on demand through `SkillToolset`.
|
||||
- **Where:** `app/adk/adk_runtime.py` (the only module that imports `google.adk`), `app/adk/roles.py`, `agents/*.md`, `agents/partials/specialist-protocol.md`, `app/tools/pillars.py`.
|
||||
- **Acceptance criteria:** Role files carry no Skill, MCP server or model names; the bundle supplies them. `run_skill_script` is never offered. Each pillar advisor loads only its own pillar Skill.
|
||||
|
||||
## Step 6: Tools and remote MCP servers
|
||||
|
||||
- **Capability:** Local check tools and the Cloud provider's hosted MCP servers, exposed through reviewed read-only allowlists.
|
||||
- **Rationale:** ADR-0002. No hand-written cloud API clients; reading a user's cloud needs their permission.
|
||||
- **Where:** `app/tools/local.py`, `app/tools/mcp.py`, `app/tools/toolkit.py`, `providers/gcp.yaml` (`mcp_servers`, `role_mcp_servers`), `skills/cloud-discovery/`, `skills/iac-validation/`, `docs/operations/mcp-access.md`, `providers/reviews/gcp-mcp-tools.json`.
|
||||
- **Acceptance criteria:** Only allowlisted tools are exposed. No tool can apply, deploy or run a script. An unavailable server becomes a recorded discovery gap, not a failure.
|
||||
|
||||
## Step 7: Interfaces
|
||||
|
||||
- **Capability:** Interactive Executions over A2A, and a pre-approved one-shot `/generate`.
|
||||
- **Rationale:** Spec decision D28.
|
||||
- **Where:** `app/main.py`, `app/a2a_server.py`, `app/card.py`, `app/presentation.py`.
|
||||
- **Acceptance criteria:** Approvals travel as structured data. The agent card is derived from the dependency graph. `/generate` response keys are unchanged from iteration 1.
|
||||
|
||||
## Step 8: Evaluation
|
||||
|
||||
- **Capability:** A real-model evaluation of the whole engine with scripted user answers.
|
||||
- **Rationale:** The gate for cleanup: proceed only when every case scores at least 0.8.
|
||||
- **Where:** `eval/solution_eval.py`, `eval/scoring.py`, `eval/datasets/solution_cases.json`, `tests/eval/`.
|
||||
- **Acceptance criteria:** Each benchmark case completes and reaches `PASS_THRESHOLD` (0.8) on the 8 checks in `eval/scoring.py`.
|
||||
|
||||
## Step 9: Cleanup
|
||||
|
||||
- **Capability:** One code path: the legacy pipeline removed and the new engine the only engine.
|
||||
- **Status:** Done in iteration 3, by a different route. On 2026-10-06 the engine first moved into the KAB folder layout, and the legacy pipeline was then deleted (`iter-03-kab-layout/steps/step-7.md`).
|
||||
- **Not done from the spec's list:** the sample artefacts the pipeline once wrote (`docs/*.md`, root `terraform/`, `deliverables/{as-is,target,validation,guides}/`) are still checked in.
|
||||
38
.agents/plans/iter-02-solution-architect-agent/MAP.md
Normal file
38
.agents/plans/iter-02-solution-architect-agent/MAP.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 2, solution architect agent
|
||||
**Intent:** Replace the linear template pipeline with a portable, hierarchical ReAct agent that runs the upstream `google-cloud-solution-architecture` Skill's workflow. An Orchestrator reasons over a code-declared dependency graph of Deliverables and delegates to one Specialist per Deliverable. Behaviour lives in Agent Skills; the Cloud provider is configuration.
|
||||
**Branch:** `feat/solution-architect-agent`
|
||||
**This directory covers:** steps 1 to 9, the spec's build increments 1 to 9
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `.scratch/solution-architect-agent/spec.md`: the design, the decisions table, the acceptance list, and a dated `## Comments` entry per finished increment. Its module paths predate iteration 3; translate them with the table in `../README.md`.
|
||||
- `docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md` and `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: the two decisions every step serves.
|
||||
- `CONTEXT.md`: the glossary. Use its terms.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | Increment | Status | Commit |
|
||||
|---|---|---|---|
|
||||
| 1 | Environment and baseline | complete | `0851171` |
|
||||
| 2 | Vendored Skills and provider bundle | complete | `2976dfa` |
|
||||
| 3 | Deliverables and Ledger | complete | `c841094` |
|
||||
| 4 | Execution graph | complete | `d9214c6` |
|
||||
| 5 | ADK runtime and role files | complete | `f040493` |
|
||||
| 6 | Tools and remote MCP servers | complete | `ed104e5` |
|
||||
| 7 | Interfaces (A2A, `/generate`) | complete | `ad371b2` |
|
||||
| 8 | Evaluation | complete, uncommitted | none |
|
||||
| 9 | Cleanup | done in iteration 3 | none |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: depends on step 1
|
||||
- Step 3: depends on step 1
|
||||
- Step 4: depends on step 3
|
||||
- Step 5: depends on step 2, step 4
|
||||
- Step 6: depends on step 2, step 5
|
||||
- Step 7: depends on step 4, step 5, step 6
|
||||
- Step 8: depends on step 7
|
||||
- Step 9: depends on step 8 passing its threshold
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 1 — Implementation Detail
|
||||
|
||||
1. Created `.venv/` on Python 3.12.3 and installed both requirements files. `google-adk` resolved to 2.9.0, `langgraph` to 1.2.11.
|
||||
2. Recorded the baseline: 30 tests passed across the iteration 1 suite.
|
||||
3. Found that the iteration 1 "ADK" layer never used `google-adk`: `HAS_NATIVE_ADK` stayed false even with the package installed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 1".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
1. Vendored 7 upstream Skills (solution architecture plus six Well-Architected Framework pillars) at commit `81a31a6`, with checksums verified by `tests/test_vendored_skills.py`.
|
||||
2. Added the provider bundle `providers/gcp.yaml` and its loader, and the Agent Skills frontmatter validator.
|
||||
3. Suite: 65 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 2".
|
||||
@@ -0,0 +1,8 @@
|
||||
# Step 3 — Implementation Detail
|
||||
|
||||
1. Declared 9 Deliverables in 4 Phases as data, with section and attachment contracts checked by `check_submission`.
|
||||
2. Built the Ledger as pure code with no I/O, serialisable into graph state.
|
||||
3. `technical_decomposition` and `architecture_description` were left with a non-empty-content contract only. Iteration 4 step 4 revisits this.
|
||||
4. Suite: 102 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 3".
|
||||
@@ -0,0 +1,9 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
1. Built the graph test-first: `orchestrator_turn`, parallel `specialist_task` via `Send`, `await_user` via `interrupt`, `finalize`, `end_with_warnings`.
|
||||
2. Added `ExecutionEngine` (`start`, `resume`, `interaction`) as the single interface; the graph is an internal detail behind it.
|
||||
3. Added the `Stopped` outcome, the single-writer rule for Specialist outcomes, and `MAX_ORCHESTRATOR_TURNS` as the loop guard.
|
||||
4. Verified the tests bite by breaking the cloud-read gate, the permission prompt, question relay and the submission contract in turn; each break failed a test.
|
||||
5. Suite: 121 passed.
|
||||
|
||||
Full record, including prompt and reply shapes: spec `## Comments`, "2026-09-15 · Increment 4".
|
||||
@@ -0,0 +1,8 @@
|
||||
# Step 5 — Implementation Detail
|
||||
|
||||
1. Implemented `AdkAgentRuntime`: a fresh `LlmAgent` and in-memory session per run, tool-call budgets, and nudges when an agent stops without ending its turn.
|
||||
2. Wrote the 11 role files and the shared Specialist protocol; the role loader rejects any field the bundle should own.
|
||||
3. Added the design advisor's fan-out to six pillar advisors.
|
||||
4. Suite: 158 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 5".
|
||||
@@ -0,0 +1,9 @@
|
||||
# Step 6 — Implementation Detail
|
||||
|
||||
1. Added local tools `mermaid_check`, `terraform_fmt`, `terraform_validate` and `terraform_plan`, each running on a scratch copy.
|
||||
2. Added the remote MCP catalog with Application Default Credentials and per-server allowlists, reviewed on 2026-09-16 against each server's live tool list.
|
||||
3. Excluded data-content tools even where read-only (SQL queries, object reads, generic Kubernetes reads).
|
||||
4. Wrote the `cloud-discovery` and `iac-validation` Skills.
|
||||
5. Suite: 189 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 6".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 7 — Implementation Detail
|
||||
|
||||
1. Added the A2A server (JSON-RPC at `/a2a`, agent card, `/health`) and the executor that maps A2A tasks to Executions and their prompts.
|
||||
2. Kept `/generate` as a pre-approved one-shot Execution.
|
||||
3. Suite: 206 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 7".
|
||||
@@ -0,0 +1,22 @@
|
||||
# Step 8 — Implementation Detail
|
||||
|
||||
This step has no entry in the spec's comments; this file is its record.
|
||||
|
||||
1. Added `eval/solution_eval.py`: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
|
||||
2. Added `eval/scoring.py` with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
|
||||
3. Added 3 cases in `eval/datasets/solution_cases.json`. Results are written to `eval/results/<timestamp>/` (gitignored).
|
||||
|
||||
Recorded runs:
|
||||
|
||||
| Run | Case | Score | Seconds | Revisions |
|
||||
|---|---|---|---|---|
|
||||
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
|
||||
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
|
||||
|
||||
No scored run of `case-02-high-scale-ingestion` is on record.
|
||||
|
||||
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
|
||||
|
||||
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
|
||||
|
||||
Suite at the end of this step: 226 passed.
|
||||
Reference in New Issue
Block a user