This commit is contained in:
54
.agents/plans/README.md
Normal file
54
.agents/plans/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Plans
|
||||
|
||||
One directory per iteration of this agent. Each holds the plan for that iteration and, for finished steps, the record of what was built. Read the row for the iteration you are working in, then its `MAP.md`.
|
||||
|
||||
## Iterations
|
||||
|
||||
| # | Directory | Status | What it is |
|
||||
|---|---|---|---|
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (uncommitted) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. |
|
||||
| 5 | `iter-05-jev-decisions/` | planned, gated on open questions | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
|
||||
Two designs sit between iterations 1 and 2 and have no plan directory. Commit `a24a44e` wrapped the pipeline in a compatibility layer that imitated Google ADK without using it (deleted in iteration 3). `.scratch/phase-pipeline/spec.md` then designed a code-sequenced Phase pipeline; ADR-0002 reopened it before any of it was built, and its decision log (D1 to D46) explains how iteration 2's design was reached.
|
||||
|
||||
## Files in an iteration
|
||||
|
||||
- `MAP.md`: the intent, each step's status, the dependency graph between steps, and the sources of truth to read.
|
||||
- `DETAIL.md`: per step, the capability, why it exists (with evidence), where the change lands, and the acceptance criteria.
|
||||
- `steps/step-N.md`: written when step N is finished. What was built, what differed from the plan, and how it was verified. A step with no file is not done.
|
||||
|
||||
Steps are numbered from 1 in iterations 2 onward. Iteration 1 numbers from 0.
|
||||
|
||||
## Working in an iteration
|
||||
|
||||
1. Read `MAP.md`, then the `DETAIL.md` entry for your step and every step it depends on.
|
||||
2. Read the sources of truth `MAP.md` names. Plans point at specs and ADRs; they repeat only what those lack.
|
||||
3. Build the step until every acceptance criterion in its `DETAIL.md` entry holds and `.venv/bin/python -m pytest -q`, `.venv/bin/ruff check .` and `.venv/bin/mypy .` all pass.
|
||||
4. Write `steps/step-N.md` and set the step's status in `MAP.md` to `complete`. When the last step closes, update the iteration's row above.
|
||||
|
||||
A new iteration gets the next number, a directory `iter-NN-<slug>/`, and a row above.
|
||||
|
||||
## Module paths changed in iteration 3
|
||||
|
||||
Iteration 2's spec and ADR text name the modules by their old paths. The code is here now:
|
||||
|
||||
| Old path | Current path |
|
||||
|---|---|
|
||||
| `app/execution/deliverables.py` | `app/workflows/deliverables.py` |
|
||||
| `app/execution/graph.py` | `app/nodes/execution_nodes.py` (nodes) and `app/workflows/execution_graph.py` (wiring) |
|
||||
| `app/execution/engine.py`, `app/assembly.py` | `app/agent.py` |
|
||||
| `app/execution/state.py` | `app/states/execution_state.py` |
|
||||
| `app/execution/ledger.py` | `app/states/ledger.py` |
|
||||
| `app/execution/tools.py`, `toolkit.py`, `pillars.py` | `app/tools/execution.py`, `app/tools/toolkit.py`, `app/tools/pillars.py` |
|
||||
| `app/execution/recorder.py`, `checkpointing.py`, `presentation.py` | `app/recorder.py`, `app/checkpointing.py`, `app/presentation.py` |
|
||||
| `app/agents/{runtime,adk_runtime,fake_runtime,roles}.py` | `app/adk/` (same file names) |
|
||||
| `app/agent_skills.py` | `app/skills/agent_skills.py` |
|
||||
| `app/a2a/server.py`, `executor.py`, `card.py` | `app/main.py`, `app/a2a_server.py`, `app/card.py` |
|
||||
| the pre-rebuild `app/agent.py`, `app/card.py`, `app/adk/`, `app/nodes/`, `app/workflows/`, `app/skills/`, `app/states/` deleted; in git history up to `e94b249` |
|
||||
| `tests/test_{adk_architecture,agent_api,graph_workflow,mcp_developer_knowledge,skill_loader}.py` | deleted |
|
||||
| `eval/{metrics,optimizer,eval_harness,test_eval_harness}.py`, `eval/datasets/benchmark_cases.json` | deleted |
|
||||
|
||||
`app/adk/`, `app/nodes/`, `app/workflows/`, `app/skills/` and `app/states/` therefore mean different code before and after iteration 3. In text written before 2026-10-06, those names refer to the deleted original pipeline.
|
||||
65
.agents/plans/iter-02-solution-architect-agent/DETAIL.md
Normal file
65
.agents/plans/iter-02-solution-architect-agent/DETAIL.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# Plan Detail
|
||||
|
||||
The spec is the full design. Each entry here gives the step's purpose, the rule it must keep, and its acceptance criteria. Paths are current (post iteration 3).
|
||||
|
||||
## Step 1: Environment and baseline
|
||||
|
||||
- **Capability:** A reproducible environment and a recorded test baseline to compare every later step against.
|
||||
- **Rationale:** Later steps claim "the suite stays green"; that needs a starting count.
|
||||
- **Where:** `.venv/`, `requirements.txt`, `requirements-dev.txt`.
|
||||
- **Acceptance criteria:** The existing suite runs and its result is recorded in the spec's comments.
|
||||
|
||||
## Step 2: Vendored Skills and provider bundle
|
||||
|
||||
- **Capability:** Upstream Skills available unmodified, and a Cloud provider expressed as configuration.
|
||||
- **Rationale:** ADR-0002. Upstream changes must arrive as a diff, and a second Cloud provider must be addable without touching the core.
|
||||
- **Where:** `skills/vendor/google/` (pinned to upstream commit `81a31a6`, with `VENDOR.md`, `LICENSE`, `SHA256SUMS`), `scripts/sync-vendored-skills.sh`, `providers/gcp.yaml`, `app/providers.py`, `app/skills/agent_skills.py`.
|
||||
- **Acceptance criteria:** Vendored files match their checksums. Every Skill under `skills/` passes the Agent Skills frontmatter rules. Only `app/providers.py` names a provider.
|
||||
|
||||
## Step 3: Deliverables and Ledger
|
||||
|
||||
- **Capability:** The dependency graph of Deliverables as data, and a pure Ledger that enforces acceptance, Approval and Revision rules.
|
||||
- **Rationale:** ADR-0003. Dependencies are guaranteed by code; ordering is the model's choice.
|
||||
- **Where:** `app/workflows/deliverables.py` (the only place Phases, Deliverables and Dependencies are declared), `app/states/ledger.py`.
|
||||
- **Acceptance criteria:** A Dependency inside a Phase is met on submission; across Phases only on Approval. Reopening a Deliverable returns it and its dependents to draft and withdraws affected Approvals. Revisions are capped.
|
||||
|
||||
## Step 4: Execution graph
|
||||
|
||||
- **Capability:** A checkpointed LangGraph Execution that pauses for the user, behind one `ExecutionEngine` interface.
|
||||
- **Rationale:** ADR-0003. Resumable, recordable progress at every step, with framework types kept out of state.
|
||||
- **Where:** `app/adk/runtime.py` (interface), `app/adk/fake_runtime.py`, `app/states/execution_state.py`, `app/tools/execution.py`, `app/nodes/execution_nodes.py`, `app/workflows/execution_graph.py`, `app/agent.py`, `app/recorder.py`, `app/checkpointing.py`.
|
||||
- **Acceptance criteria:** Only `orchestrator_turn` and `await_user` change the Ledger. No Approval is ever granted from free text. Cloud-read tools are unreachable without permission. All of it is tested with the fake runtime and no model.
|
||||
|
||||
## Step 5: ADK runtime and role files
|
||||
|
||||
- **Capability:** Real ADK `LlmAgent`s running role files with scoped Skills, behind the runtime interface.
|
||||
- **Rationale:** ADR-0003. Each agent runs its own ReAct loop; Skills load on demand through `SkillToolset`.
|
||||
- **Where:** `app/adk/adk_runtime.py` (the only module that imports `google.adk`), `app/adk/roles.py`, `agents/*.md`, `agents/partials/specialist-protocol.md`, `app/tools/pillars.py`.
|
||||
- **Acceptance criteria:** Role files carry no Skill, MCP server or model names; the bundle supplies them. `run_skill_script` is never offered. Each pillar advisor loads only its own pillar Skill.
|
||||
|
||||
## Step 6: Tools and remote MCP servers
|
||||
|
||||
- **Capability:** Local check tools and the Cloud provider's hosted MCP servers, exposed through reviewed read-only allowlists.
|
||||
- **Rationale:** ADR-0002. No hand-written cloud API clients; reading a user's cloud needs their permission.
|
||||
- **Where:** `app/tools/local.py`, `app/tools/mcp.py`, `app/tools/toolkit.py`, `providers/gcp.yaml` (`mcp_servers`, `role_mcp_servers`), `skills/cloud-discovery/`, `skills/iac-validation/`, `docs/operations/mcp-access.md`, `providers/reviews/gcp-mcp-tools.json`.
|
||||
- **Acceptance criteria:** Only allowlisted tools are exposed. No tool can apply, deploy or run a script. An unavailable server becomes a recorded discovery gap, not a failure.
|
||||
|
||||
## Step 7: Interfaces
|
||||
|
||||
- **Capability:** Interactive Executions over A2A, and a pre-approved one-shot `/generate`.
|
||||
- **Rationale:** Spec decision D28.
|
||||
- **Where:** `app/main.py`, `app/a2a_server.py`, `app/card.py`, `app/presentation.py`.
|
||||
- **Acceptance criteria:** Approvals travel as structured data. The agent card is derived from the dependency graph. `/generate` response keys are unchanged from iteration 1.
|
||||
|
||||
## Step 8: Evaluation
|
||||
|
||||
- **Capability:** A real-model evaluation of the whole engine with scripted user answers.
|
||||
- **Rationale:** The gate for cleanup: proceed only when every case scores at least 0.8.
|
||||
- **Where:** `eval/solution_eval.py`, `eval/scoring.py`, `eval/datasets/solution_cases.json`, `tests/eval/`.
|
||||
- **Acceptance criteria:** Each benchmark case completes and reaches `PASS_THRESHOLD` (0.8) on the 8 checks in `eval/scoring.py`.
|
||||
|
||||
## Step 9: Cleanup
|
||||
|
||||
- **Capability:** One code path: the legacy pipeline removed and the new engine the only engine.
|
||||
- **Status:** Done in iteration 3, by a different route. On 2026-10-06 the engine first moved into the KAB folder layout, and the legacy pipeline was then deleted (`iter-03-kab-layout/steps/step-7.md`).
|
||||
- **Not done from the spec's list:** the sample artefacts the pipeline once wrote (`docs/*.md`, root `terraform/`, `deliverables/{as-is,target,validation,guides}/`) are still checked in.
|
||||
38
.agents/plans/iter-02-solution-architect-agent/MAP.md
Normal file
38
.agents/plans/iter-02-solution-architect-agent/MAP.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 2, solution architect agent
|
||||
**Intent:** Replace the linear template pipeline with a portable, hierarchical ReAct agent that runs the upstream `google-cloud-solution-architecture` Skill's workflow. An Orchestrator reasons over a code-declared dependency graph of Deliverables and delegates to one Specialist per Deliverable. Behaviour lives in Agent Skills; the Cloud provider is configuration.
|
||||
**Branch:** `feat/solution-architect-agent`
|
||||
**This directory covers:** steps 1 to 9, the spec's build increments 1 to 9
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `.scratch/solution-architect-agent/spec.md`: the design, the decisions table, the acceptance list, and a dated `## Comments` entry per finished increment. Its module paths predate iteration 3; translate them with the table in `../README.md`.
|
||||
- `docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md` and `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: the two decisions every step serves.
|
||||
- `CONTEXT.md`: the glossary. Use its terms.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | Increment | Status | Commit |
|
||||
|---|---|---|---|
|
||||
| 1 | Environment and baseline | complete | `0851171` |
|
||||
| 2 | Vendored Skills and provider bundle | complete | `2976dfa` |
|
||||
| 3 | Deliverables and Ledger | complete | `c841094` |
|
||||
| 4 | Execution graph | complete | `d9214c6` |
|
||||
| 5 | ADK runtime and role files | complete | `f040493` |
|
||||
| 6 | Tools and remote MCP servers | complete | `ed104e5` |
|
||||
| 7 | Interfaces (A2A, `/generate`) | complete | `ad371b2` |
|
||||
| 8 | Evaluation | complete, uncommitted | none |
|
||||
| 9 | Cleanup | done in iteration 3 | none |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: depends on step 1
|
||||
- Step 3: depends on step 1
|
||||
- Step 4: depends on step 3
|
||||
- Step 5: depends on step 2, step 4
|
||||
- Step 6: depends on step 2, step 5
|
||||
- Step 7: depends on step 4, step 5, step 6
|
||||
- Step 8: depends on step 7
|
||||
- Step 9: depends on step 8 passing its threshold
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 1 — Implementation Detail
|
||||
|
||||
1. Created `.venv/` on Python 3.12.3 and installed both requirements files. `google-adk` resolved to 2.9.0, `langgraph` to 1.2.11.
|
||||
2. Recorded the baseline: 30 tests passed across the iteration 1 suite.
|
||||
3. Found that the iteration 1 "ADK" layer never used `google-adk`: `HAS_NATIVE_ADK` stayed false even with the package installed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 1".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
1. Vendored 7 upstream Skills (solution architecture plus six Well-Architected Framework pillars) at commit `81a31a6`, with checksums verified by `tests/test_vendored_skills.py`.
|
||||
2. Added the provider bundle `providers/gcp.yaml` and its loader, and the Agent Skills frontmatter validator.
|
||||
3. Suite: 65 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 2".
|
||||
@@ -0,0 +1,8 @@
|
||||
# Step 3 — Implementation Detail
|
||||
|
||||
1. Declared 9 Deliverables in 4 Phases as data, with section and attachment contracts checked by `check_submission`.
|
||||
2. Built the Ledger as pure code with no I/O, serialisable into graph state.
|
||||
3. `technical_decomposition` and `architecture_description` were left with a non-empty-content contract only. Iteration 4 step 4 revisits this.
|
||||
4. Suite: 102 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-15 · Increment 3".
|
||||
@@ -0,0 +1,9 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
1. Built the graph test-first: `orchestrator_turn`, parallel `specialist_task` via `Send`, `await_user` via `interrupt`, `finalize`, `end_with_warnings`.
|
||||
2. Added `ExecutionEngine` (`start`, `resume`, `interaction`) as the single interface; the graph is an internal detail behind it.
|
||||
3. Added the `Stopped` outcome, the single-writer rule for Specialist outcomes, and `MAX_ORCHESTRATOR_TURNS` as the loop guard.
|
||||
4. Verified the tests bite by breaking the cloud-read gate, the permission prompt, question relay and the submission contract in turn; each break failed a test.
|
||||
5. Suite: 121 passed.
|
||||
|
||||
Full record, including prompt and reply shapes: spec `## Comments`, "2026-09-15 · Increment 4".
|
||||
@@ -0,0 +1,8 @@
|
||||
# Step 5 — Implementation Detail
|
||||
|
||||
1. Implemented `AdkAgentRuntime`: a fresh `LlmAgent` and in-memory session per run, tool-call budgets, and nudges when an agent stops without ending its turn.
|
||||
2. Wrote the 11 role files and the shared Specialist protocol; the role loader rejects any field the bundle should own.
|
||||
3. Added the design advisor's fan-out to six pillar advisors.
|
||||
4. Suite: 158 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 5".
|
||||
@@ -0,0 +1,9 @@
|
||||
# Step 6 — Implementation Detail
|
||||
|
||||
1. Added local tools `mermaid_check`, `terraform_fmt`, `terraform_validate` and `terraform_plan`, each running on a scratch copy.
|
||||
2. Added the remote MCP catalog with Application Default Credentials and per-server allowlists, reviewed on 2026-09-16 against each server's live tool list.
|
||||
3. Excluded data-content tools even where read-only (SQL queries, object reads, generic Kubernetes reads).
|
||||
4. Wrote the `cloud-discovery` and `iac-validation` Skills.
|
||||
5. Suite: 189 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 6".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 7 — Implementation Detail
|
||||
|
||||
1. Added the A2A server (JSON-RPC at `/a2a`, agent card, `/health`) and the executor that maps A2A tasks to Executions and their prompts.
|
||||
2. Kept `/generate` as a pre-approved one-shot Execution.
|
||||
3. Suite: 206 passed.
|
||||
|
||||
Full record: spec `## Comments`, "2026-09-16 · Increment 7".
|
||||
@@ -0,0 +1,22 @@
|
||||
# Step 8 — Implementation Detail
|
||||
|
||||
This step has no entry in the spec's comments; this file is its record.
|
||||
|
||||
1. Added `eval/solution_eval.py`: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
|
||||
2. Added `eval/scoring.py` with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
|
||||
3. Added 3 cases in `eval/datasets/solution_cases.json`. Results are written to `eval/results/<timestamp>/` (gitignored).
|
||||
|
||||
Recorded runs:
|
||||
|
||||
| Run | Case | Score | Seconds | Revisions |
|
||||
|---|---|---|---|---|
|
||||
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
|
||||
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
|
||||
|
||||
No scored run of `case-02-high-scale-ingestion` is on record.
|
||||
|
||||
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
|
||||
|
||||
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
|
||||
|
||||
Suite at the end of this step: 226 passed.
|
||||
55
.agents/plans/iter-03-kab-layout/DETAIL.md
Normal file
55
.agents/plans/iter-03-kab-layout/DETAIL.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Plan Detail
|
||||
|
||||
## Step 1: Container starts the Execution engine
|
||||
|
||||
- **Capability:** The image runs the reviewed and evaluated engine.
|
||||
- **Rationale:** The `Dockerfile` started `python -m app.agent`, the iteration 1 pipeline, while every evaluation ran the new engine.
|
||||
- **Where:** `Dockerfile`, `README.md`.
|
||||
- **Acceptance criteria:** The container's main process is the A2A server. `/health` and `/.well-known/agent-card.json` answer on the container port. Existing `-p 8080:8080` run commands still work.
|
||||
|
||||
## Step 2: Legacy pipeline moved to `app/legacy/`
|
||||
|
||||
- **Capability:** The KAB folder names are free for the new engine, with the old pipeline still importable and tested.
|
||||
- **Rationale:** The owner chose to keep the old code rather than delete it (2026-10-06). The old and new code cannot share `app/nodes/`, `app/states/`, `app/workflows/`.
|
||||
- **Where:** `app/legacy/`, the five legacy tests in `tests/`, `eval/eval_harness.py`, `SKILLS_DIR` in `app/config.py`.
|
||||
- **Acceptance criteria:** Every legacy test passes against `app.legacy.*`. Nothing outside `app/legacy/`, those tests and the legacy eval files imports `app.legacy`.
|
||||
|
||||
## Step 3: Engine moved into the KAB folders
|
||||
|
||||
- **Capability:** The engine's modules sit where the KAB workflow standard expects them.
|
||||
- **Rationale:** Platform consistency: decomposers, reviewers and other agents on the platform look for state in `app/states/`, nodes in `app/nodes/`, graphs in `app/workflows/`, the manager in `app/agent.py`.
|
||||
- **Where:** The path table in `../README.md`.
|
||||
- **Acceptance criteria:** Behaviour is unchanged: the full suite passes with only import paths edited in tests. `google.adk` is still imported only by `app/adk/adk_runtime.py`. Package `__init__.py` files import nothing, so `app.states` can import `app.workflows.deliverables` without a cycle.
|
||||
|
||||
## Step 4: Graph file split into nodes and wiring
|
||||
|
||||
- **Capability:** Node logic in `app/nodes/`, graph assembly in `app/workflows/`.
|
||||
- **Rationale:** The KAB layout separates stage nodes from graph construction; the engine had both in one 560-line file.
|
||||
- **Where:** `app/nodes/execution_nodes.py` (`make_execution_nodes` returns an `ExecutionNodes` of the five node callables), `app/workflows/execution_graph.py` (`build_execution_graph`), `ToolFactory` in `app/adk/runtime.py`.
|
||||
- **Acceptance criteria:** `build_execution_graph` keeps its signature. The nodes still route with `Command` and `Send`, so the wiring declares only the fixed edges.
|
||||
|
||||
## Step 5: Refresh paths in the spec and ADR text
|
||||
|
||||
- **Capability:** Design documents that name modules where they are.
|
||||
- **Rationale:** `.scratch/solution-architect-agent/spec.md` (module layout, acceptance list, comments) and the ADRs still use pre-move paths. An agent following them will look in `app/legacy/` or in folders that no longer exist.
|
||||
- **Where:** `.scratch/solution-architect-agent/spec.md` sections "Module layout" and "Acceptance"; `docs/adr/0002-*.md` consequence about `app/skills/`; `docs/adr/0003-*.md`.
|
||||
- **Acceptance criteria:** The spec's module layout matches the tree. Each acceptance command in the spec runs against current paths. Dated `## Comments` entries stay as written, since they are history; one note at the top of the comments points to `../README.md` for translation.
|
||||
|
||||
## Step 6: Finish iteration 2's cleanup list
|
||||
|
||||
- **Capability:** The parts of iteration 2 step 9 that still apply once the legacy code is kept.
|
||||
- **Rationale:** The spec's cleanup list mixed deletions (now dropped) with changes the new engine still needs.
|
||||
- **Where and what:**
|
||||
- `workflow.yaml`: rewrite its phases to the four upstream Phases and their Deliverables.
|
||||
- `app/config.py`, `.env.example`: move settings only `app/legacy/` reads (`HOST`, `PORT`, `GCP_PROJECT_ID`, `GCP_REGION`, `ADK_*`, `DEVELOPER_KNOWLEDGE_MCP_*`, `SKILLS_DIR`, `EVAL_DATASET_PATH`, the OpenAI and Azure keys) under a clearly marked legacy block. Keep `POSTGRES_*` and `DATABASE_URL`: the Postgres checkpointer builds its URL from them.
|
||||
- `mypy.ini`, `ruff.toml`: exclude `app/legacy/` and the legacy eval files, so `ruff check .` and `mypy .` report on live code only (198 and 38 findings today, almost all legacy).
|
||||
- `Dockerfile`: stop copying what only the legacy pipeline reads, once step 7 is decided.
|
||||
- **Acceptance criteria:** `ruff check .` and `mypy .` are clean or list only known, recorded exceptions. `workflow.yaml` names the four Phases in `app/workflows/deliverables.py`. The suite passes.
|
||||
|
||||
## Step 7: Decide the future of `app/legacy/`
|
||||
|
||||
- **Capability:** One code path, or a deliberate, documented second one.
|
||||
- **Rationale:** Two pipelines cost every reader a detour, and the legacy one holds what ADR-0002 rules out: hand-written cloud clients, non-conformant Skills, a pretend ADK layer.
|
||||
- **Options for the owner:** delete `app/legacy/` with its tests and the legacy eval files; or keep it frozen and excluded from lint, types and the image.
|
||||
- **Guardrail:** Deleting files with uncommitted changes was blocked once by the permission system. Commit first, then delete in a commit of its own.
|
||||
- **Acceptance criteria:** The decision is recorded here. If deleted: `grep -rn "app\.legacy" app eval tests` is empty and the suite passes.
|
||||
34
.agents/plans/iter-03-kab-layout/MAP.md
Normal file
34
.agents/plans/iter-03-kab-layout/MAP.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 3, KAB layout
|
||||
**Intent:** Make the deployed container run the hierarchical Execution engine, and house that engine in the KAB workflow-agent folder layout (`app/agent.py`, `app/main.py`, `app/a2a_server.py`, `app/states/`, `app/nodes/`, `app/workflows/`), with the iteration 1 pipeline first parked under `app/legacy/` (steps 2 to 6) and then deleted (step 7).
|
||||
**Branch:** `feat/solution-architect-agent` (all of this iteration is uncommitted as of 2026-10-06)
|
||||
**This directory covers:** steps 1 to 7
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `../README.md`: the old-to-new module path table.
|
||||
- `../docs/KAB_WORKFLOW_AGENT_IMPLEMENTATION_GUIDE.md` and `../langgraph-workflow-agent/app/`, both relative to the repository root: the layout being matched.
|
||||
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: the engine the layout must keep unchanged.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Status |
|
||||
|---|---|---|
|
||||
| 1 | Container starts the Execution engine | complete |
|
||||
| 2 | Legacy pipeline moved to `app/legacy/` | complete |
|
||||
| 3 | Engine moved into the KAB folders | complete |
|
||||
| 4 | Graph file split into nodes and wiring | complete |
|
||||
| 5 | Refresh paths in the spec and ADR text | complete |
|
||||
| 6 | Finish iteration 2's cleanup list | complete; two items carried forward in `steps/step-6.md` |
|
||||
| 7 | Decide the future of `app/legacy/` | complete: deleted on the owner's instruction |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: no dependencies
|
||||
- Step 3: depends on step 2
|
||||
- Step 4: depends on step 3
|
||||
- Step 5: depends on step 4
|
||||
- Step 6: depends on step 4
|
||||
- Step 7: depends on step 6
|
||||
16
.agents/plans/iter-03-kab-layout/steps/step-1.md
Normal file
16
.agents/plans/iter-03-kab-layout/steps/step-1.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Step 1 — Implementation Detail
|
||||
|
||||
1. `Dockerfile`: the start command became the A2A server; the image sets `A2A_HOST=0.0.0.0` and `A2A_PORT=8080`; only port 8080 is exposed.
|
||||
2. The health check reads `A2A_PORT` and allows 60 seconds to start, because `/health` answers only after the provider bundle and MCP catalogs load.
|
||||
3. `README.md`: server section, start command, container example (now passing `GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION`, `A2A_PUBLIC_URL`) and test calls.
|
||||
|
||||
Verified: built with podman and ran the image with `MCP_ENABLED=false`; `/health` and the agent card returned 200 and process 1 was the server.
|
||||
|
||||
Consequences for callers:
|
||||
|
||||
- `/card`, `/validate` and `/sessions/{session_id}` are no longer served. The card is at `/.well-known/agent-card.json`.
|
||||
- `/generate` now runs a full pre-approved Execution with real models, about 45 minutes in the evaluation runs.
|
||||
- Without `A2A_PUBLIC_URL` the card advertises `http://0.0.0.0:8080/a2a`.
|
||||
- Podman's default image format ignores `HEALTHCHECK`.
|
||||
|
||||
Not verified: a full Execution with real models and MCP servers inside the container.
|
||||
9
.agents/plans/iter-03-kab-layout/steps/step-2.md
Normal file
9
.agents/plans/iter-03-kab-layout/steps/step-2.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
1. Moved `app/agent.py`, `app/card.py`, `app/adk/`, `app/nodes/`, `app/workflows/`, `app/skills/`, `app/states/` to `app/legacy/`, and `gcp_scanner.py`, `mcp_developer_knowledge.py`, `validation_tools.py` to `app/legacy/tools/`.
|
||||
2. Rewrote imports to `app.legacy.*` in the legacy code, its five tests (`test_adk_architecture`, `test_agent_api`, `test_graph_workflow`, `test_mcp_developer_knowledge`, `test_skill_loader`) and `eval/eval_harness.py`.
|
||||
3. Pointed `SKILLS_DIR` at `app/legacy/skills`, and the paths quoted in `eval/optimizer.py` likewise.
|
||||
|
||||
Verified: 226 passed before any engine file moved; `python -m eval.eval_harness` still runs.
|
||||
|
||||
Nothing was deleted. A first attempt to delete these files was blocked by the permission system because 25 of them held uncommitted changes.
|
||||
10
.agents/plans/iter-03-kab-layout/steps/step-3.md
Normal file
10
.agents/plans/iter-03-kab-layout/steps/step-3.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Step 3 — Implementation Detail
|
||||
|
||||
1. Moved each engine module to the path in `../../README.md`. `app/execution/`, `app/agents/` and `app/a2a/` no longer exist.
|
||||
2. Merged `engine.py` and `assembly.py` into `app/agent.py`: `ExecutionEngine`, `Interaction`, then `Assembly` and `open_assembly`.
|
||||
3. The entry point is `python -m app.main`; `Dockerfile` and `README.md` use it.
|
||||
4. Left the test folders (`tests/execution/`, `tests/agents/`, `tests/a2a/`) where they were; only their imports changed.
|
||||
|
||||
Verified: 226 passed; the rebuilt image starts `python -m app.main` and answers `/health` and the agent card.
|
||||
|
||||
Side effect: ruff's import sort and unused-import fix ran across `app`, `eval`, `tests` and `scripts`, so legacy and test files show import-only diffs.
|
||||
9
.agents/plans/iter-03-kab-layout/steps/step-4.md
Normal file
9
.agents/plans/iter-03-kab-layout/steps/step-4.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
1. `app/nodes/execution_nodes.py` keeps the five nodes as closures over the runtime, roles, recorder and tool factory, built by `make_execution_nodes` and returned as a frozen `ExecutionNodes`. The helper functions and `GUIDE_FILENAME` moved with them.
|
||||
2. `app/workflows/execution_graph.py` holds `build_execution_graph`, which builds the nodes and declares the fixed edges.
|
||||
3. `ToolFactory` moved to `app/adk/runtime.py`, beside `ToolSpec`, so `app/tools/toolkit.py` no longer imports the nodes module.
|
||||
|
||||
The node bodies were moved verbatim; no logic changed.
|
||||
|
||||
Verified: 226 passed; the moved files are ruff-clean.
|
||||
11
.agents/plans/iter-03-kab-layout/steps/step-5.md
Normal file
11
.agents/plans/iter-03-kab-layout/steps/step-5.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Step 5 — Implementation Detail
|
||||
|
||||
1. `.scratch/solution-architect-agent/spec.md`: replaced the "Module layout" tree with the current one, and updated every module path in the design sections (Deliverables, Ledger, Execution graph, tools, agent runtime, A2A, recording) and in the Risks section.
|
||||
2. Rewrote two acceptance commands to match the tree:
|
||||
- `rg -n "google\.adk" app -g '!app/legacy/**' | grep -v adk_runtime.py` is empty.
|
||||
- `rg -n "app\.legacy" app eval tests -g '!**/legacy/**'` is empty.
|
||||
3. Marked build increment 9 as replaced, with a pointer here; its original list stays below the note.
|
||||
4. Left the dated `## Comments` entries as written and added one note above them pointing to `../../README.md` for path translation.
|
||||
5. `docs/adr/0002-*.md`: added a dated note to the `app/skills/` consequence. ADR-0003 names no repository paths and needed no change.
|
||||
|
||||
Verified: both acceptance commands return nothing (run with `grep`).
|
||||
19
.agents/plans/iter-03-kab-layout/steps/step-6.md
Normal file
19
.agents/plans/iter-03-kab-layout/steps/step-6.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Step 6 — Implementation Detail
|
||||
|
||||
1. **Legacy grouped in three folders.** The five legacy tests moved to `tests/legacy/` and the offline harness (`metrics.py`, `optimizer.py`, `eval_harness.py`, `test_eval_harness.py`) to `eval/legacy/`. Run it with `python -m eval.legacy.eval_harness`. `eval/datasets/benchmark_cases.json` stayed where `EVAL_DATASET_PATH` points.
|
||||
2. **Lint and types on live code only.** `ruff.toml` and `mypy.ini` excluded `app/legacy`, `eval/legacy` and `tests/legacy` (exclusions removed again in step 7). `ruff check .` and `mypy .` are now clean, down from 198 and 38 findings.
|
||||
3. **Live findings fixed, none suppressed by config:**
|
||||
- `app/database.py`: modern annotations, an unused import removed, and each broad `except` marked with its reason (connection failure falls back to SQLite; persistence is best-effort and logged).
|
||||
- `app/workflows/execution_graph.py`: every node is cast for `add_node`, since the nodes are now typed callables.
|
||||
- `scripts/list-mcp-tools.py`, `scripts/validate_artifacts.py`, `tests/test_assembly.py`, `tests/test_package.py`, `tests/agents/test_adk_engine.py`, `tests/agents/test_adk_runtime.py`: small typing and lint fixes.
|
||||
- `requirements-dev.txt`: added `types-protobuf` and `types-psycopg2`.
|
||||
4. **Settings.** `app/config.py` and `.env.example` list live settings first and the legacy-only ones under a marked block. No setting was removed or renamed.
|
||||
5. **`workflow.yaml`** names the four Phases and their Deliverables. `tests/test_workflow_manifest.py` fails if it drifts from `app/workflows/deliverables.py`.
|
||||
6. **Image.** `Dockerfile` copies `app/`, `agents/`, `providers/`, `skills/` and `workflow.yaml` only, and creates `deliverables/`. A new `.dockerignore` keeps `app/legacy`, `.venv`, `.git`, caches, databases and past results out of the build context.
|
||||
|
||||
Verified: 227 passed; ruff and mypy clean; the rebuilt image contains no `app/legacy`, `eval/`, `docs/` or `terraform/`, and `/health` answers.
|
||||
|
||||
Still open from iteration 2's cleanup list:
|
||||
|
||||
- **Writes to shared paths.** The engine writes only under `deliverables/executions/<execution_id>/`. The legacy pipeline still writes `deliverables/{as-is,target,validation,guides}/` and `docs/*.md`, and `tests/legacy/` runs it, so a test run can modify tracked files. Step 7 deleted the legacy folders, which closed this item.
|
||||
- **CI.** `.github/workflows/ci.yml` pins `ruff==0.6.9` and `mypy==1.11.2` and installs no project dependencies; the local tools are newer (ruff 0.16.7). CI was not run for this step.
|
||||
15
.agents/plans/iter-03-kab-layout/steps/step-7.md
Normal file
15
.agents/plans/iter-03-kab-layout/steps/step-7.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Step 7 — Implementation Detail
|
||||
|
||||
Decision: **the original pipeline is deleted.** The owner asked for its removal on 2026-10-06, after it had first been kept frozen in `app/legacy/`, `tests/legacy/` and `eval/legacy/` (step 6).
|
||||
|
||||
1. Deleted `app/legacy/`, `tests/legacy/`, `eval/legacy/` and `eval/datasets/benchmark_cases.json`.
|
||||
2. Removed the legacy-only settings from `app/config.py` and `.env.example`: `HOST`, `PORT`, the OpenAI and Azure keys, `LLM_TEMPERATURE`, `GOOGLE_APPLICATION_CREDENTIALS`, `GCP_PROJECT_ID`, `GCP_REGION`, `ADK_*`, `DEVELOPER_KNOWLEDGE_MCP_*`, `SKILLS_DIR`, `EVAL_DATASET_PATH`. Google client libraries still read `GOOGLE_APPLICATION_CREDENTIALS` from the environment themselves.
|
||||
3. Removed the legacy exclusions from `ruff.toml`, `mypy.ini` and `.dockerignore`.
|
||||
4. Removed four dependencies no live module imports: `langchain`, `langchain-openai`, `sqlalchemy`, `asyncpg`.
|
||||
5. Rewrote the top of `README.md` around the Orchestrator, Specialists, Skills and A2A, and pointed `VALIDATION.md` at `eval.solution_eval`.
|
||||
|
||||
Verified: 206 passed (227 less the 21 legacy tests); `ruff check .` and `mypy .` clean; `grep -rn "app\.legacy\|eval\.legacy" app eval tests scripts` is empty; the image rebuilt from the trimmed `requirements.txt` imports every live module, including both checkpointers, and answers `/health` and the agent card.
|
||||
|
||||
Recovering the old code: the committed versions are in git history at their pre-move paths (`app/adk/`, `app/nodes/`, `app/workflows/`, `app/skills/`, `app/states/`, `app/agent.py`, `app/card.py`) up to commit `e94b249`. Uncommitted formatting changes to those files were not kept.
|
||||
|
||||
Left in place, because they are checked-in sample artefacts and not pipeline code: `scripts/validate_artifacts.py`, `tests/test_artifacts.py`, `tests/test_package.py`, the root `terraform/`, `docs/*.md`, `deliverables/{as-is,target,validation,guides}/`, `requirements.yaml`. `app/database.py` keeps two methods only the old pipeline called (`get_session`, `save_evaluation`).
|
||||
123
.agents/plans/iter-04-learning-and-skills/DETAIL.md
Normal file
123
.agents/plans/iter-04-learning-and-skills/DETAIL.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Plan Detail
|
||||
|
||||
Every step ends with the full suite passing. Steps that change agent behaviour (3, 6, 10, 11, 12, 13) also end with an evaluation run recorded in their step file, compared against the baseline in `iter-02-solution-architect-agent/steps/step-8.md`.
|
||||
|
||||
## Step 1: Structured validator findings
|
||||
|
||||
- **Report ref:** enhancement 3
|
||||
- **Capability:** Validation findings as data the engine can route, count and learn from.
|
||||
- **Rationale:** The validator reports findings as free markdown. The Orchestrator reads them once to decide what to reopen; nothing else can use them. They are the richest learning signal in an Execution.
|
||||
- **Where:** `app/workflows/deliverables.py` (add `findings.json` to the `validation_results` attachments), `agents/validator.md`, `skills/iac-validation/SKILL.md` (the finding format), `app/tools/execution.py` (validate the attachment's shape at submission).
|
||||
- **Acceptance criteria:** A `validation_results` submission without a parseable `findings.json` is refused with the reason. Each finding carries the Deliverable kind it traces to, a severity, the evidence and the rule. An empty list is valid and means a pass.
|
||||
|
||||
## Step 2: Carry the draft through `needs_input`
|
||||
|
||||
- **Report ref:** enhancement 9
|
||||
- **Capability:** The user receives the draft they approved.
|
||||
- **Rationale:** The Specialist protocol asks a Specialist to confirm its draft with the user before submitting. `needs_input` carries only questions and Specialists are stateless per delegation, so the approved draft is lost and written again on the next run.
|
||||
- **Where:** `needs_input` in `app/tools/execution.py` (optional `draft` and attachments), `NeedsInput` in `app/adk/runtime.py`, `_outcome_record` and `_specialist_input` in `app/nodes/execution_nodes.py` (store the draft, return it in context), `app/states/execution_state.py`, `agents/partials/specialist-protocol.md`.
|
||||
- **Acceptance criteria:** A Specialist restarted after the user approves its draft receives that draft in its context. A test shows the submitted content equals the approved draft when the fake runtime submits what it was given.
|
||||
|
||||
## Step 3: Reuse pillar advice across Revisions
|
||||
|
||||
- **Report ref:** enhancement 10
|
||||
- **Capability:** A Revision of the design recommendations re-consults only the pillars its feedback touches.
|
||||
- **Rationale:** Pillar advice is never stored. In case 01, two Revisions re-ran all six advisors each time: 18 advisor runs for one Deliverable.
|
||||
- **Where:** `app/tools/pillars.py` (the tool already accepts `pillars`), the Execution state for stored advice, `agents/design-advisor.md` (on a Revision, name the affected pillars).
|
||||
- **Acceptance criteria:** On a second attempt with stored advice, only the named pillars run and the tool returns stored advice for the rest. A changed brief for a pillar invalidates that pillar's stored advice.
|
||||
|
||||
## Step 4: Single source for Deliverable contracts
|
||||
|
||||
- **Report ref:** enhancement 11
|
||||
- **Capability:** Required headings and attachments declared once, and contracts for the Deliverables that have none.
|
||||
- **Rationale:** Required headings appear in each role file and again in `DeliverableSpec.sections`. `technical_decomposition`, `architecture_description` and the diagram's content accept any non-empty text.
|
||||
- **Where:** `app/workflows/deliverables.py`, `_specialist_input` in `app/nodes/execution_nodes.py` (pass the contract in context), `agents/*.md` (remove the "Required headings" lines), `agents/partials/specialist-protocol.md`.
|
||||
- **Acceptance criteria:** No role file lists required headings. Each Specialist's context carries its contract. The decomposition requires a component table naming the requirements each component serves.
|
||||
|
||||
## Step 5: Cap parallel Specialists
|
||||
|
||||
- **Report ref:** enhancement 14
|
||||
- **Capability:** A bound on concurrent Specialist runs per Execution.
|
||||
- **Rationale:** The spec's Risks section calls for a default cap of 4 against model rate limits. `orchestrator_turn` sends every delegated Deliverable at once, and the pillar fan-out adds six more.
|
||||
- **Where:** `app/nodes/execution_nodes.py`, `app/tools/pillars.py`, a `MAX_PARALLEL_SPECIALISTS` setting in `app/config.py`.
|
||||
- **Acceptance criteria:** A test with the fake runtime shows at most the configured number of Specialist runs in flight. The pillar fan-out counts toward the same cap.
|
||||
|
||||
## Step 6: Orchestrator step order
|
||||
|
||||
- **Report ref:** findings by agent, orchestrator row
|
||||
- **Capability:** A failed validation goes back for Revision before the user is asked to approve it.
|
||||
- **Rationale:** In `agents/orchestrator.md`, "A Phase is complete" (step 3) precedes "Validation found problems" (step 5), and the Orchestrator takes the first step that applies.
|
||||
- **Where:** `agents/orchestrator.md`.
|
||||
- **Acceptance criteria:** The validation-failure step sits above the Approval step. An evaluation run with a failing validation shows a reopen before any Approval request for the validation Phase.
|
||||
|
||||
## Step 7: Capture lesson candidates
|
||||
|
||||
- **Report ref:** enhancement 4
|
||||
- **Capability:** A durable record, per Execution, of what went wrong and what fixed it.
|
||||
- **Rationale:** The signals exist and are discarded. Revision feedback lives in the Ledger inside one checkpoint. Tool refusals (`Not submitted`, `Refused`, `terraform validate failed`) reach only the Python log through `REJECTION_PREFIXES` in `app/adk/adk_runtime.py`.
|
||||
- **Where:** `app/recorder.py` (a `record_lesson_candidates` method on `ExecutionRecorder`), `finalize` and `end_with_warnings` in `app/nodes/execution_nodes.py`, the runtime for refusals.
|
||||
- **Acceptance criteria:** A finished Execution writes one record holding each reopen with its reason and actor, each finding from step 1, each tool refusal with role and tool, each user change request and each exhausted budget. The record names the Execution, the Deliverable kind and the Skill versions loaded. It holds no secrets and no discovered cloud data.
|
||||
|
||||
## Step 8: Evaluation that can see improvement
|
||||
|
||||
- **Report ref:** enhancement 6
|
||||
- **Capability:** An evaluation whose score moves when a Skill gets better or worse.
|
||||
- **Rationale:** The 8 checks are structural and both scored runs reached 1.0. The loop's proof step has nothing to measure.
|
||||
- **Where:** `eval/scoring.py`, `eval/solution_eval.py`, `eval/datasets/solution_cases.json`, `tests/eval/`.
|
||||
- **Acceptance criteria:** New checks cover requirement-to-component traceability, coverage of each pillar Skill's checklist, and a graded rubric for recommendation quality. One interactive case checks that a Specialist asks before it submits. Revisions, seconds and tokens per case have recorded thresholds, and a run that exceeds one is reported as a regression. `case-02-high-scale-ingestion` has a scored run.
|
||||
|
||||
## Step 9: ADR: the lessons loop and overlay Skills
|
||||
|
||||
- **Report ref:** sequencing note
|
||||
- **Capability:** A recorded decision on how Skills change and how they are chosen per Execution.
|
||||
- **Rationale:** Steps 10 and 12 change which Skills an agent loads and who may edit them. That is an architectural decision, and ADR-0002 does not cover it.
|
||||
- **Where:** `docs/adr/0004-*.md`.
|
||||
- **Decisions the owner must make:** whether lessons need human review before they take effect (the plan assumes yes); where lessons Skills live and who owns them; whether an Execution may name overlay Skills, and who may supply them.
|
||||
- **Acceptance criteria:** The ADR is accepted and steps 10 and 12 are updated to match it.
|
||||
|
||||
## Step 10: Lessons Skill and lesson curator
|
||||
|
||||
- **Report ref:** enhancement 5
|
||||
- **Capability:** Each Execution can improve the next, under human review.
|
||||
- **Rationale:** `eval/optimizer.py` is the only learning code. It maps four keywords to iteration 1 Skill names that the engine never loads.
|
||||
- **Design:** The loop is capture (step 7), distil, review, prove, publish, retire.
|
||||
- Distil: an offline `lesson-curator` role groups candidates across Executions and proposes lessons. It runs outside any Execution.
|
||||
- Review: each proposal is an issue file under `.scratch/lessons/issues/` with `Status: needs-triage`.
|
||||
- Prove: the evaluation from step 8 runs with and without the change.
|
||||
- Publish: lessons land in a lessons Skill under `skills/`, one reference file per Deliverable kind; each lesson has its rule, reason, source Execution and date. The bundle's `role_skills` lists it.
|
||||
- Retire: at each vendor sync, remove lessons the upstream Skill now covers.
|
||||
- **Where:** `skills/<lessons-skill>/`, `agents/lesson-curator.md`, a curator entry point beside `eval/solution_eval.py`, `providers/gcp.yaml`, `scripts/sync-vendored-skills.sh`.
|
||||
- **Acceptance criteria:** Vendored Skills stay byte-identical. No lesson reaches a Skill without an approved issue file. Each Execution records the lessons Skill version it loaded. One real lesson has gone through the whole loop with its evaluation comparison recorded.
|
||||
|
||||
## Step 11: IaC patterns Skill
|
||||
|
||||
- **Report ref:** enhancement 7
|
||||
- **Capability:** The IaC author starts from known-good infrastructure code.
|
||||
- **Rationale:** The IaC author is the most expensive role: 812 thousand tokens, 23 requests, 4 runs and 3 Revisions in case 03. It writes from a blank page each time.
|
||||
- **Where:** `skills/<iac-patterns-skill>/` with one reference per product, `providers/gcp.yaml` (`role_skills.iac-author`).
|
||||
- **Acceptance criteria:** On the three cases, quality checks hold and IaC author Revisions or tokens fall against the step 8 baseline.
|
||||
|
||||
## Step 12: Overlay Skills and an organisation standards Skill
|
||||
|
||||
- **Report ref:** enhancement 8
|
||||
- **Capability:** New standards and clients need no code change.
|
||||
- **Rationale:** Skills are fixed at start-up: `AdkAgentRuntime` loads every Skill once, and `role_skills` is static per provider. Only 2 of the 9 Skills are our own, and both describe tools.
|
||||
- **Where:** `app/adk/adk_runtime.py`, `app/adk/roles.py`, `app/providers.py`, `ExecutionEngine.start` in `app/agent.py`, `app/a2a_server.py`, a standards Skill under `skills/`.
|
||||
- **Acceptance criteria:** An Execution can name overlay Skills at start; each passes the same Agent Skills validation as bundled Skills, and an unknown or invalid one refuses the start. An organisation standards Skill (naming, tagging, approved products) is loaded by the roles the bundle lists.
|
||||
|
||||
## Step 13: Trim Specialist context
|
||||
|
||||
- **Report ref:** enhancement 12
|
||||
- **Capability:** Each Specialist receives what its Deliverable needs.
|
||||
- **Rationale:** `inputs_of` gives a Specialist every Deliverable of every earlier Phase in full. The describer used 513 thousand tokens over 12 requests.
|
||||
- **Where:** `inputs_of` in `app/workflows/deliverables.py`, `_specialist_input` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Direct Dependencies arrive in full; other earlier Deliverables arrive as summaries, with `read_deliverable`-style access to the full text on demand. Quality checks hold and describer tokens fall against the step 8 baseline.
|
||||
|
||||
## Step 14: Provider-owned contracts
|
||||
|
||||
- **Report ref:** enhancement 13
|
||||
- **Capability:** A second Cloud provider is configuration, as ADR-0002 intends.
|
||||
- **Rationale:** The six pillar headings, the eight solution-guide sections and the Terraform attachment names are hard-coded in `app/workflows/deliverables.py`. Other providers' frameworks name and count pillars differently. The bundle's pillar keys already differ from the headings (`performance-optimization` against `Performance efficiency`).
|
||||
- **Where:** `providers/gcp.yaml`, `app/providers.py`, `app/workflows/deliverables.py`, `app/presentation.py`, `agents/iac-author.md`, `agents/validator.md`, `agents/design-advisor.md`.
|
||||
- **Timing:** Build when a second Cloud provider is planned. ADR-0002 says no provider interface exists in code until then.
|
||||
- **Acceptance criteria:** Pillar headings come from `pillar_skills`, guide sections from the bundle's `output_template`, and the IaC tool with its file contract from the bundle. `grep -rni "terraform" app/workflows app/nodes app/states` is empty, and no pillar or guide-section name appears as a literal in `app/workflows/deliverables.py`.
|
||||
48
.agents/plans/iter-04-learning-and-skills/MAP.md
Normal file
48
.agents/plans/iter-04-learning-and-skills/MAP.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 4, learning and Skills
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** planned; no step started
|
||||
**This directory covers:** steps 1 to 14
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- Review report (2026-10-06): https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. Each step's `Report ref` names its enhancement number there. The report uses pre-iteration-3 paths.
|
||||
- `iter-02-solution-architect-agent/steps/step-8.md`: the evaluation figures quoted as evidence.
|
||||
- `docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md`: every step must keep behaviour in Skills and vendored Skills unmodified.
|
||||
- `CONTEXT.md`: "every Execution starts clean". Learning arrives as a reviewed Skill change, never as memory carried between Executions.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Group | Size | Status |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Structured validator findings | A | S | planned |
|
||||
| 2 | Carry the draft through `needs_input` | A | S | planned |
|
||||
| 3 | Reuse pillar advice across Revisions | A | S | planned |
|
||||
| 4 | Single source for Deliverable contracts | A | S | planned |
|
||||
| 5 | Cap parallel Specialists | A | S | planned |
|
||||
| 6 | Orchestrator step order | A | S | planned |
|
||||
| 7 | Capture lesson candidates | B | M | planned |
|
||||
| 8 | Evaluation that can see improvement | B | M | planned |
|
||||
| 9 | ADR: the lessons loop and overlay Skills | B | S | planned, owner's decision |
|
||||
| 10 | Lessons Skill and lesson curator | B | L | planned |
|
||||
| 11 | IaC patterns Skill | B | M | planned |
|
||||
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
|
||||
| 13 | Trim Specialist context | C | M | planned |
|
||||
| 14 | Provider-owned contracts | C | M | planned |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Steps 1 to 6: no dependencies; any order
|
||||
- Step 7: depends on step 1
|
||||
- Step 8: no dependencies
|
||||
- Step 9: no dependencies
|
||||
- Step 10: depends on step 7, step 8, step 9
|
||||
- Step 11: depends on step 8
|
||||
- Step 12: depends on step 9
|
||||
- Step 13: depends on step 8
|
||||
- Step 14: depends on step 4
|
||||
|
||||
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.
|
||||
151
.agents/plans/iter-05-jev-decisions/DETAIL.md
Normal file
151
.agents/plans/iter-05-jev-decisions/DETAIL.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# Plan Detail
|
||||
|
||||
Rules for every step from 3 onward:
|
||||
|
||||
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
|
||||
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
|
||||
- **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14).
|
||||
- With no API key configured, the agent behaves exactly as it does without this iteration.
|
||||
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
|
||||
|
||||
## Step 1: Resolve the open questions
|
||||
|
||||
- **Capability:** The facts needed to send client data to Jev.
|
||||
- **Questions for the owner or vendor:**
|
||||
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
|
||||
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
|
||||
- Rate limits and typical latency.
|
||||
- **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
|
||||
|
||||
## Step 2: Decider interface and Jev client with limits
|
||||
|
||||
- **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake.
|
||||
- **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly.
|
||||
- **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`.
|
||||
- **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
|
||||
|
||||
| Practice | Default |
|
||||
|---|---|
|
||||
| Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question |
|
||||
| Send every question about one state in one call | one call per state |
|
||||
| Timeout | 2 seconds connect, 5 seconds total |
|
||||
| Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap |
|
||||
| Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 |
|
||||
| Semaphore on calls in flight | 8 |
|
||||
| Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down |
|
||||
| Cache answers by a hash of model, state and questions | in memory, per Execution |
|
||||
| Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 |
|
||||
| Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 |
|
||||
| Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always |
|
||||
| API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` |
|
||||
|
||||
- **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback".
|
||||
|
||||
## Step 3: Pillars a Revision touches
|
||||
|
||||
- **Report ref:** J3
|
||||
- **Capability:** The feedback text selects which pillar advisors re-run.
|
||||
- **Design:** One call with a Noul per pillar: "Does this feedback require changes to the <pillar> recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`.
|
||||
- **Fallback:** all pillars.
|
||||
- **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`.
|
||||
- **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
|
||||
|
||||
## Step 4: Semantic checks at submission
|
||||
|
||||
- **Report ref:** J4, and Jev's "task-completion check" use case
|
||||
- **Capability:** Submission contracts that check meaning, not only headings.
|
||||
- **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
|
||||
- **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
|
||||
- **Fallback:** accept on the structural contract alone.
|
||||
- **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle.
|
||||
- **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
|
||||
|
||||
## Step 5: Finding-to-Deliverable routing
|
||||
|
||||
- **Report ref:** J2
|
||||
- **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity.
|
||||
- **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`.
|
||||
- **Fallback:** the validator's own stated Deliverable.
|
||||
- **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
|
||||
|
||||
## Step 6: Model tier per delegation
|
||||
|
||||
- **Report ref:** J5, and Jev's "model routing" use case
|
||||
- **Capability:** The cheaper model tier where the task allows.
|
||||
- **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
|
||||
- **Fallback:** the role's configured model.
|
||||
- **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`.
|
||||
- **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
|
||||
|
||||
## Step 7: Context selection
|
||||
|
||||
- **Report ref:** J6, and Jev's "repo context selection" use case
|
||||
- **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full.
|
||||
- **Design:** A Noul per section: "Is this section needed to produce the <kind> Deliverable?" Sections below threshold are passed as one-line summaries.
|
||||
- **Fallback:** the rule from iteration 4 step 13.
|
||||
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged.
|
||||
|
||||
## Step 8: Skill routing
|
||||
|
||||
- **Report ref:** J9, and Jev's "agent skill routing" use case
|
||||
- **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
|
||||
- **Design:** A Choice over the role's Skills and their references, with a `none` option.
|
||||
- **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today.
|
||||
- **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`.
|
||||
- **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged.
|
||||
|
||||
## Step 9: In-run context compaction
|
||||
|
||||
- **Report ref:** J10, and Jev's "context compaction" use case
|
||||
- **Capability:** Old tool results inside one agent run are kept, truncated or dropped.
|
||||
- **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each.
|
||||
- **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result.
|
||||
- **Fallback:** keep everything.
|
||||
- **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook.
|
||||
- **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
|
||||
|
||||
## Step 10: Evaluation rubric and lesson triage
|
||||
|
||||
- **Report ref:** J8
|
||||
- **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates.
|
||||
- **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
|
||||
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
|
||||
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
|
||||
|
||||
## Step 11: ADR: Orchestrator routing
|
||||
|
||||
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
|
||||
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
|
||||
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
|
||||
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
|
||||
|
||||
## Step 12: Orchestrator routing
|
||||
|
||||
- **Report ref:** J1
|
||||
- **Capability:** The Orchestrator model runs only when judgement is needed.
|
||||
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
|
||||
- **Fallback:** the Orchestrator model, as today.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
|
||||
|
||||
## Step 13: Cloud-read call guard
|
||||
|
||||
- **Report ref:** J7, and Jev's "tool-call risk gate" use case
|
||||
- **Capability:** A check before each cloud-read call that it stays within the scope the user named.
|
||||
- **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
|
||||
- **Fallback:** fail closed: ask the user.
|
||||
- **Where:** `gate_cloud_read` in `app/tools/execution.py`.
|
||||
- **Depends on step 1:** tool arguments name the user's cloud resources.
|
||||
- **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user.
|
||||
|
||||
## Step 14: Input guardrail
|
||||
|
||||
- **Report ref:** J11, and Jev's "LLM guardrail" use case
|
||||
- **Capability:** Injected instructions are caught before a Specialist reads them.
|
||||
- **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
|
||||
- **Fallback:** fail closed: withhold the text and tell the Orchestrator.
|
||||
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
|
||||
- **Depends on step 1:** MCP results hold discovered cloud state.
|
||||
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.
|
||||
73
.agents/plans/iter-05-jev-decisions/MAP.md
Normal file
73
.agents/plans/iter-05-jev-decisions/MAP.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1
|
||||
**This directory covers:** steps 1 to 14
|
||||
|
||||
## What Jev is
|
||||
|
||||
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
|
||||
|
||||
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
|
||||
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
|
||||
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
|
||||
|
||||
Facts from the vendor's documentation, read 2026-10-06:
|
||||
|
||||
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
|
||||
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
|
||||
- Price: $0.42 per million input tokens; output free.
|
||||
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
|
||||
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
|
||||
|
||||
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
|
||||
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
|
||||
- `iter-04-learning-and-skills/`: several steps here build on steps there.
|
||||
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
| 6 | Model tier per delegation | J5 | planned |
|
||||
| 7 | Context selection | J6 | planned |
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Orchestrator routing | none | planned, owner's decision |
|
||||
| 12 | Orchestrator routing | J1 | planned |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: depends on step 1
|
||||
- Step 3: depends on step 2, iteration 4 step 3
|
||||
- Step 4: depends on step 2
|
||||
- Step 5: depends on step 2, iteration 4 step 1
|
||||
- Step 6: depends on step 2
|
||||
- Step 7: depends on step 2, iteration 4 step 13
|
||||
- Step 8: depends on step 2, iteration 4 step 12
|
||||
- Step 9: depends on step 2
|
||||
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
|
||||
- Step 11: depends on step 4 having evaluation results
|
||||
- Step 12: depends on step 2, step 11
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
Reference in New Issue
Block a user