latest refactoring
Some checks failed
validation / verify (push) Failing after 10s

This commit is contained in:
2026-10-06 20:22:37 +01:00
parent e94b2493e7
commit 7ee49ab352
414 changed files with 15768 additions and 3896 deletions

View File

@@ -0,0 +1,65 @@
# Plan Detail
The spec is the full design. Each entry here gives the step's purpose, the rule it must keep, and its acceptance criteria. Paths are current (post iteration 3).
## Step 1: Environment and baseline
- **Capability:** A reproducible environment and a recorded test baseline to compare every later step against.
- **Rationale:** Later steps claim "the suite stays green"; that needs a starting count.
- **Where:** `.venv/`, `requirements.txt`, `requirements-dev.txt`.
- **Acceptance criteria:** The existing suite runs and its result is recorded in the spec's comments.
## Step 2: Vendored Skills and provider bundle
- **Capability:** Upstream Skills available unmodified, and a Cloud provider expressed as configuration.
- **Rationale:** ADR-0002. Upstream changes must arrive as a diff, and a second Cloud provider must be addable without touching the core.
- **Where:** `skills/vendor/google/` (pinned to upstream commit `81a31a6`, with `VENDOR.md`, `LICENSE`, `SHA256SUMS`), `scripts/sync-vendored-skills.sh`, `providers/gcp.yaml`, `app/providers.py`, `app/skills/agent_skills.py`.
- **Acceptance criteria:** Vendored files match their checksums. Every Skill under `skills/` passes the Agent Skills frontmatter rules. Only `app/providers.py` names a provider.
## Step 3: Deliverables and Ledger
- **Capability:** The dependency graph of Deliverables as data, and a pure Ledger that enforces acceptance, Approval and Revision rules.
- **Rationale:** ADR-0003. Dependencies are guaranteed by code; ordering is the model's choice.
- **Where:** `app/workflows/deliverables.py` (the only place Phases, Deliverables and Dependencies are declared), `app/states/ledger.py`.
- **Acceptance criteria:** A Dependency inside a Phase is met on submission; across Phases only on Approval. Reopening a Deliverable returns it and its dependents to draft and withdraws affected Approvals. Revisions are capped.
## Step 4: Execution graph
- **Capability:** A checkpointed LangGraph Execution that pauses for the user, behind one `ExecutionEngine` interface.
- **Rationale:** ADR-0003. Resumable, recordable progress at every step, with framework types kept out of state.
- **Where:** `app/adk/runtime.py` (interface), `app/adk/fake_runtime.py`, `app/states/execution_state.py`, `app/tools/execution.py`, `app/nodes/execution_nodes.py`, `app/workflows/execution_graph.py`, `app/agent.py`, `app/recorder.py`, `app/checkpointing.py`.
- **Acceptance criteria:** Only `orchestrator_turn` and `await_user` change the Ledger. No Approval is ever granted from free text. Cloud-read tools are unreachable without permission. All of it is tested with the fake runtime and no model.
## Step 5: ADK runtime and role files
- **Capability:** Real ADK `LlmAgent`s running role files with scoped Skills, behind the runtime interface.
- **Rationale:** ADR-0003. Each agent runs its own ReAct loop; Skills load on demand through `SkillToolset`.
- **Where:** `app/adk/adk_runtime.py` (the only module that imports `google.adk`), `app/adk/roles.py`, `agents/*.md`, `agents/partials/specialist-protocol.md`, `app/tools/pillars.py`.
- **Acceptance criteria:** Role files carry no Skill, MCP server or model names; the bundle supplies them. `run_skill_script` is never offered. Each pillar advisor loads only its own pillar Skill.
## Step 6: Tools and remote MCP servers
- **Capability:** Local check tools and the Cloud provider's hosted MCP servers, exposed through reviewed read-only allowlists.
- **Rationale:** ADR-0002. No hand-written cloud API clients; reading a user's cloud needs their permission.
- **Where:** `app/tools/local.py`, `app/tools/mcp.py`, `app/tools/toolkit.py`, `providers/gcp.yaml` (`mcp_servers`, `role_mcp_servers`), `skills/cloud-discovery/`, `skills/iac-validation/`, `docs/operations/mcp-access.md`, `providers/reviews/gcp-mcp-tools.json`.
- **Acceptance criteria:** Only allowlisted tools are exposed. No tool can apply, deploy or run a script. An unavailable server becomes a recorded discovery gap, not a failure.
## Step 7: Interfaces
- **Capability:** Interactive Executions over A2A, and a pre-approved one-shot `/generate`.
- **Rationale:** Spec decision D28.
- **Where:** `app/main.py`, `app/a2a_server.py`, `app/card.py`, `app/presentation.py`.
- **Acceptance criteria:** Approvals travel as structured data. The agent card is derived from the dependency graph. `/generate` response keys are unchanged from iteration 1.
## Step 8: Evaluation
- **Capability:** A real-model evaluation of the whole engine with scripted user answers.
- **Rationale:** The gate for cleanup: proceed only when every case scores at least 0.8.
- **Where:** `eval/solution_eval.py`, `eval/scoring.py`, `eval/datasets/solution_cases.json`, `tests/eval/`.
- **Acceptance criteria:** Each benchmark case completes and reaches `PASS_THRESHOLD` (0.8) on the 8 checks in `eval/scoring.py`.
## Step 9: Cleanup
- **Capability:** One code path: the legacy pipeline removed and the new engine the only engine.
- **Status:** Done in iteration 3, by a different route. On 2026-10-06 the engine first moved into the KAB folder layout, and the legacy pipeline was then deleted (`iter-03-kab-layout/steps/step-7.md`).
- **Not done from the spec's list:** the sample artefacts the pipeline once wrote (`docs/*.md`, root `terraform/`, `deliverables/{as-is,target,validation,guides}/`) are still checked in.

View File

@@ -0,0 +1,38 @@
# Plan Map
**Iteration:** 2, solution architect agent
**Intent:** Replace the linear template pipeline with a portable, hierarchical ReAct agent that runs the upstream `google-cloud-solution-architecture` Skill's workflow. An Orchestrator reasons over a code-declared dependency graph of Deliverables and delegates to one Specialist per Deliverable. Behaviour lives in Agent Skills; the Cloud provider is configuration.
**Branch:** `feat/solution-architect-agent`
**This directory covers:** steps 1 to 9, the spec's build increments 1 to 9
## Sources of truth
- `.scratch/solution-architect-agent/spec.md`: the design, the decisions table, the acceptance list, and a dated `## Comments` entry per finished increment. Its module paths predate iteration 3; translate them with the table in `../README.md`.
- `docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md` and `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: the two decisions every step serves.
- `CONTEXT.md`: the glossary. Use its terms.
## Status
| Step | Increment | Status | Commit |
|---|---|---|---|
| 1 | Environment and baseline | complete | `0851171` |
| 2 | Vendored Skills and provider bundle | complete | `2976dfa` |
| 3 | Deliverables and Ledger | complete | `c841094` |
| 4 | Execution graph | complete | `d9214c6` |
| 5 | ADK runtime and role files | complete | `f040493` |
| 6 | Tools and remote MCP servers | complete | `ed104e5` |
| 7 | Interfaces (A2A, `/generate`) | complete | `ad371b2` |
| 8 | Evaluation | complete, uncommitted | none |
| 9 | Cleanup | done in iteration 3 | none |
## Dependency graph
- Step 1: no dependencies
- Step 2: depends on step 1
- Step 3: depends on step 1
- Step 4: depends on step 3
- Step 5: depends on step 2, step 4
- Step 6: depends on step 2, step 5
- Step 7: depends on step 4, step 5, step 6
- Step 8: depends on step 7
- Step 9: depends on step 8 passing its threshold

View File

@@ -0,0 +1,7 @@
# Step 1 — Implementation Detail
1. Created `.venv/` on Python 3.12.3 and installed both requirements files. `google-adk` resolved to 2.9.0, `langgraph` to 1.2.11.
2. Recorded the baseline: 30 tests passed across the iteration 1 suite.
3. Found that the iteration 1 "ADK" layer never used `google-adk`: `HAS_NATIVE_ADK` stayed false even with the package installed.
Full record: spec `## Comments`, "2026-09-15 · Increment 1".

View File

@@ -0,0 +1,7 @@
# Step 2 — Implementation Detail
1. Vendored 7 upstream Skills (solution architecture plus six Well-Architected Framework pillars) at commit `81a31a6`, with checksums verified by `tests/test_vendored_skills.py`.
2. Added the provider bundle `providers/gcp.yaml` and its loader, and the Agent Skills frontmatter validator.
3. Suite: 65 passed.
Full record: spec `## Comments`, "2026-09-15 · Increment 2".

View File

@@ -0,0 +1,8 @@
# Step 3 — Implementation Detail
1. Declared 9 Deliverables in 4 Phases as data, with section and attachment contracts checked by `check_submission`.
2. Built the Ledger as pure code with no I/O, serialisable into graph state.
3. `technical_decomposition` and `architecture_description` were left with a non-empty-content contract only. Iteration 4 step 4 revisits this.
4. Suite: 102 passed.
Full record: spec `## Comments`, "2026-09-15 · Increment 3".

View File

@@ -0,0 +1,9 @@
# Step 4 — Implementation Detail
1. Built the graph test-first: `orchestrator_turn`, parallel `specialist_task` via `Send`, `await_user` via `interrupt`, `finalize`, `end_with_warnings`.
2. Added `ExecutionEngine` (`start`, `resume`, `interaction`) as the single interface; the graph is an internal detail behind it.
3. Added the `Stopped` outcome, the single-writer rule for Specialist outcomes, and `MAX_ORCHESTRATOR_TURNS` as the loop guard.
4. Verified the tests bite by breaking the cloud-read gate, the permission prompt, question relay and the submission contract in turn; each break failed a test.
5. Suite: 121 passed.
Full record, including prompt and reply shapes: spec `## Comments`, "2026-09-15 · Increment 4".

View File

@@ -0,0 +1,8 @@
# Step 5 — Implementation Detail
1. Implemented `AdkAgentRuntime`: a fresh `LlmAgent` and in-memory session per run, tool-call budgets, and nudges when an agent stops without ending its turn.
2. Wrote the 11 role files and the shared Specialist protocol; the role loader rejects any field the bundle should own.
3. Added the design advisor's fan-out to six pillar advisors.
4. Suite: 158 passed.
Full record: spec `## Comments`, "2026-09-16 · Increment 5".

View File

@@ -0,0 +1,9 @@
# Step 6 — Implementation Detail
1. Added local tools `mermaid_check`, `terraform_fmt`, `terraform_validate` and `terraform_plan`, each running on a scratch copy.
2. Added the remote MCP catalog with Application Default Credentials and per-server allowlists, reviewed on 2026-09-16 against each server's live tool list.
3. Excluded data-content tools even where read-only (SQL queries, object reads, generic Kubernetes reads).
4. Wrote the `cloud-discovery` and `iac-validation` Skills.
5. Suite: 189 passed.
Full record: spec `## Comments`, "2026-09-16 · Increment 6".

View File

@@ -0,0 +1,7 @@
# Step 7 — Implementation Detail
1. Added the A2A server (JSON-RPC at `/a2a`, agent card, `/health`) and the executor that maps A2A tasks to Executions and their prompts.
2. Kept `/generate` as a pre-approved one-shot Execution.
3. Suite: 206 passed.
Full record: spec `## Comments`, "2026-09-16 · Increment 7".

View File

@@ -0,0 +1,22 @@
# Step 8 — Implementation Detail
This step has no entry in the spec's comments; this file is its record.
1. Added `eval/solution_eval.py`: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
2. Added `eval/scoring.py` with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
3. Added 3 cases in `eval/datasets/solution_cases.json`. Results are written to `eval/results/<timestamp>/` (gitignored).
Recorded runs:
| Run | Case | Score | Seconds | Revisions |
|---|---|---|---|---|
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
No scored run of `case-02-high-scale-ingestion` is on record.
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
Suite at the end of this step: 226 passed.