Some checks failed
check agents / check (push) Failing after 13s
The cp1-solution-architecture-agent and the sa-aws/azure/gcp-cloud MCP servers, a separate stream from the individual and sub-agent patterns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
427 lines
44 KiB
Markdown
427 lines
44 KiB
Markdown
# Plan: Solution Architecture agent on the modernization platform
|
|
|
|
| | |
|
|
|---|---|
|
|
| Date | 2026-10-30, revision 4 |
|
|
| Status | `mcp-servers/` is built and tested locally (section 4.13). Step 0 is done as far as it can go without the cluster (section 4.14). Steps 1 to 5 are done, except what needs a built image. Step 6 is partly done (section 6). |
|
|
| Source | `kyndryl-agent-builder/gcp_solution_architecture_agent` (read only) |
|
|
| Platform reviewed | `kyndryl-agent-builder/modernization-platform` @ `c89aedcb9` (2026-10-29), runtime 0.37.0. The other checkout, `platform-engineering/modernization-platform`, is older. |
|
|
| Target runtime | `agent-runtime-adk:0.35.7`, the version `cp1-agents` pins. |
|
|
| Boundary | Every new file sits under `outcome-pattern/solution_architecture/`. Changes needed elsewhere are listed in section 8, not made. |
|
|
|
|
**Changed in revision 2, at the owner's request:** every model is `claude-sonnet-5-5`; Azure, AWS and Google Cloud are all in the first version; live conversation stays as an option; Jev stays as an option.
|
|
|
|
**Changed in revision 3, at the owner's request:** the agent replicates the source with full feature parity. Everything the source does is in v1 (section 5). The engine is still rebuilt on KAB so that it integrates. The only intended differences are the ones the owner chose, such as one model for every role.
|
|
|
|
**Changed in revision 4, at the owner's request:** the MCP layer is built like for like inside this folder, one server per hyperscaler. This replaces the plan to use and extend the shared discovery servers and `cloud-knowledge`. Sections 4.4, 4.12 and 8 are updated to match.
|
|
|
|
**Changed in revision 5, at the owner's request:** the Execution runs on real LangGraph, as in the source, not on a hand-written driver. See section 4.15.
|
|
|
|
## 1. Summary
|
|
|
|
Build one KAB agent that delivers the same outcome as the source: a request goes in, and a reviewed solution architecture comes out, with Deliverables, Target architecture, Terraform, validation, work breakdown and Solution guide. An Execution targets one Cloud provider, chosen from Azure, AWS and Google Cloud.
|
|
|
|
Don't lift the source's container onto the platform. Its engine conflicts with the runtime: A2A server, LangGraph checkpointing, `interrupt()` approvals, database and direct model and MCP clients. Keep what makes the source valuable and rebuild the execution layer on platform features.
|
|
|
|
| Source | Size | Plan |
|
|
|---|---|---|
|
|
| Deterministic core: `scope`, `deliverables`, `target_architecture`, `chain`, `work_breakdown`, `changes`, `ledger` | about 3,000 lines | **Port**, made provider-neutral, with tests |
|
|
| Router and its playbook (`app/nodes/router.py`) | about 900 lines | **Port.** It's now needed for conversation and for Jev |
|
|
| Jev and its reviews: `jev`, `meaning`, `skill_routing`, `guardrail`, `assessment`, `design_review`, `schema_review`, `coverage` | about 1,200 lines | **Port** behind the source's `Decider` interface |
|
|
| Change requests: `baseline`, `customer_case` and the Change route | about 250 lines, plus `changes` above | **Port** |
|
|
| Lessons loop: `app/learning/*`, `states/lessons`, `eval/curate_lessons.py` | about 1,000 lines | **Port** |
|
|
| Evaluation harness: `eval/` scoring, judge, calibration | | **Port** as scripts that run against the deployed agent |
|
|
| 16 role files in `agents/` | | **Port** as sub-agent prompts, made provider-neutral |
|
|
| Google Skills, including vendored ones | 49 files | **Port** with the licence and checksums |
|
|
| Azure and AWS Skills | none exist | **New** (section 4.8) |
|
|
| 8 evaluation cases | | **Port**, then add Azure and AWS cases |
|
|
| `execution_nodes`, `app/adk/*`, `a2a_server`, `database`, `checkpointing` | about 3,400 lines | **Replace** with the platform |
|
|
|
|
## 2. Platform constraints that shape the design
|
|
|
|
| Constraint | Where | Consequence |
|
|
|---|---|---|
|
|
| The runtime fails any run that pauses with LangGraph `interrupt()`, and keeps no checkpointer | `a2a/server.py` `_refuse_interrupt` | The source's Approvals can't move as they are |
|
|
| Agents keep nothing between requests | `continuation.py` | Execution state has to live outside the process |
|
|
| A workflow step is reviewed after it runs: continue, stop, rerun with comments (`hitl_feedback`), or replace the output. Up to 5 reruns per step, and 50 steps per workflow | `generic_workflow_executor.py`, `workflow_template_validator.py` | In a workflow, each Phase is a step and each Approval a review point |
|
|
| Run chat fills in a workflow's inputs and handles its reviews. Catalog chat answers about the catalog. I found no KAB screen for talking directly to a deployed agent | `services/run_chat/`, `api/routes/catalog_chat.py` | Live conversation needs an A2A client outside those screens (section 4.6) |
|
|
| Agents reach only the model and MCP gateways, plus hosts in `AGENT_EGRESS_ALLOW` | `egress.py` | Jev needs an explicit allowance |
|
|
| Sub-agents have a prompt and functions only, with no MCP tools or Skills | Discovery `.agents/CONTEXT.md` | Skills and guidance need a code path (section 4.4) |
|
|
| An agent with its own `app/agent.py` gets no sub-agent registry or resume | `sub_agents.py`, `continuation.py` | Use a spec agent, not a wrapped graph |
|
|
| `cloud-knowledge` takes fixed pillars and topics only, and `mcp-servers/README.md` forbids free-text fields | `cloud-knowledge.server.yaml` | The agent uses its own servers for free-text search (section 4.13) |
|
|
| A declared MCP tool the gateway doesn't offer stops the agent at startup | runtime README | Tool names must be confirmed first |
|
|
| Requests have a wall-clock limit (`AGENT_MAX_DURATION_SECONDS`) | `configmap.yaml` | One full Execution took 22 to 35 minutes, so Phases need separate requests |
|
|
|
|
## 3. Approach and alternatives
|
|
|
|
**Chosen: an ADK spec agent with two ways in, a workflow and a live conversation, sharing one Phase runner and one workspace.**
|
|
|
|
| Alternative | Why not |
|
|
|---|---|
|
|
| Keep the source's container and register it by URL | No platform limits, tracing or cost logs, and it breaks egress |
|
|
| `framework: langgraph` with the graph in `app/agent.py` | Loses `ask_sub_agent` and the runtime's handling of Specialists. Real LangGraph runs inside the ADK agent's hooks instead (section 4.15) |
|
|
| One agent per Deliverable, chained in a workflow | That's the `individual-agents/` pattern, and it rules out live conversation |
|
|
| Port `orchestration_engine` and `solution_architecture_agent` | A cleaner split with the same conflicts. Compare module by module when porting (step 2). |
|
|
|
|
## 4. Design
|
|
|
|
### 4.1 Folder layout
|
|
|
|
`solution_architecture/` has the same depth as a Feature folder, so the shared scripts work with it unchanged.
|
|
|
|
```
|
|
outcome-pattern/solution_architecture/
|
|
├── CLAUDE.md README.md .agents/PLAN.md
|
|
├── cp1-solution-architecture-agent/
|
|
│ ├── agent.yaml prompt.md Dockerfile RELEASE_NOTES.md README.md CLAUDE.md
|
|
│ ├── schemas/{input,output}.json
|
|
│ ├── prompts/<specialist>.md from agents/*.md
|
|
│ ├── skills/{common,gcp,azure,aws}/
|
|
│ ├── providers/{gcp,azure,aws}.yaml
|
|
│ ├── app/
|
|
│ │ ├── business_logic.py hooks: bind the run, route the message, build the reply
|
|
│ │ ├── kab_runtime.py runs a role through ask_sub_agent
|
|
│ │ ├── engine/ the source's engine on LangGraph
|
|
│ │ ├── skill_reader.py role-scoped Skill loading
|
|
│ │ ├── terraform.py fmt, validate and schema lookup
|
|
│ │ ├── decisions/ Decider, NoDecider, Jev and its reviews
|
|
│ │ ├── learning/ Lesson candidates and curation
|
|
│ │ └── core/ the ported deterministic modules
|
|
│ ├── tests/ eval/ kubernetes/ docs/adr/ .agents/CONTEXT.md
|
|
├── mcp-servers/ built: see its README.md
|
|
│ ├── sa-gcp-cloud/ sa-azure-cloud/ sa-aws-cloud/
|
|
│ └── terraform-plan/ not built: plan with a project's credential
|
|
├── workflows/cp1-solution-architecture.json
|
|
└── scripts/{build_workflow.py, dry_run.py, curate_lessons.py}
|
|
```
|
|
|
|
`terraform-plan` sits here because only this agent uses it, which the repository's rule allows: a server belongs at the root when more than one Feature or pattern needs it.
|
|
|
|
The agent's Skills are listed in `agent.yaml` by folder name, so nested folders need checking in step 1. If they're refused, use flat names such as `azure-waf-security`.
|
|
|
|
### 4.2 Source to platform mapping
|
|
|
|
| Source concept | On the platform |
|
|
|---|---|
|
|
| Execution | A workflow run, or a conversation's A2A `contextId` |
|
|
| Approval | A review point (workflow), or an approving message (conversation) |
|
|
| Request for changes (Revision) | `hitl_feedback` rerun, or a message. Both reach the same router route, and code reopens the Deliverable and its dependants. |
|
|
| Pre-approved Execution | Workflow review points switched off, or `pre_approved` in a conversation's first message |
|
|
| Orchestrator | The ported router and Ledger, in the source's LangGraph graph |
|
|
| Specialist | `sub_agents:` entry, `as_tool: false`, called with `ask_sub_agent` |
|
|
| LangGraph state and checkpoints | Real LangGraph, with `VolumeSaver` on `execution-storage-pvc` (section 4.15) |
|
|
| Customer case | `engagement`: `/data/executions/solution-architecture/<engagement>/<run>/` |
|
|
| Scope spec file | KAB upload or artifact step, or inline in a conversation message |
|
|
| Model per role | One model, `claude-sonnet-5-5`, through LiteLLM |
|
|
| Google's hosted MCP servers | The same servers through `sa-gcp-cloud`, plus `sa-azure-cloud` and `sa-aws-cloud`, through the MCP gateway |
|
|
| Cloud-read permission | `allow_cloud_read`, and the project's stored credential |
|
|
|
|
### 4.3 How a Phase runs
|
|
|
|
1. `pre_process` binds the run, verifies the manifest chain, takes the run's lock and imports files.
|
|
2. The router reads the request or message and the Execution's state, and picks a route. With Jev on, Jev informs that choice; with it off, the playbook's rules decide.
|
|
3. The graph asks the Ledger what's ready, and runs ready Specialists in parallel with a concurrency limit.
|
|
4. Each result goes through the source's submission checks. A refusal returns to the Specialist with the reason, up to the Revision limit.
|
|
5. `post_process` builds the reply from the workspace. It owns `status`, `deliverables`, `open_questions` and `manifest_head`, so a model can't assert them.
|
|
|
|
The root agent runs no model to choose work on a JSON request, matching source ADR-0007. A plain-text message needs one model call to reach this code (section 4.14).
|
|
|
|
Every finished step is checkpointed, so a stopped request loses nothing it finished.
|
|
|
|
### 4.4 Skills and guidance for Specialists
|
|
|
|
- `read_skill(name, reference?)` returns Skill text, limited to what the provider bundle grants that role.
|
|
- `search_cloud_docs(query)` replicates the source's free-text search, for the product selector, describer, target architect, design advisor and pillar advisors. It calls the documentation tools on the Execution's provider server, and relies on `call_tool` working from a sub-agent's function (spike b). If not, code fetches documents per Deliverable before the Specialist starts.
|
|
- **This is an accepted exception to a repository rule (owner decision, 2026-10-30).** The root `mcp-servers/README.md` limits knowledge queries to fixed lists, so client detail can't reach a vendor. Here a model writes the query. The rule still holds for the shared servers.
|
|
- **Queries aren't scrubbed (owner decision, 2026-10-30).** They go to the vendor as the model wrote them. Specialists' prompts tell them to keep client detail out of searches, but that's guidance, not enforcement.
|
|
- **Safeguards that remain:** every query is logged, and replies pass through `_scrub`. A per-deployment setting on the agent turns `search_cloud_docs` off, and it starts off until a deployment enables it. The ADR records the decision.
|
|
|
|
### 4.5 Terraform
|
|
|
|
The agent runs `fmt`, `validate`, schema lookup and `plan`, never `apply`, as the source does. The Dockerfile adds a checksum-verified Terraform binary and a mirror of the three pinned providers (`hashicorp/google`, `hashicorp/azurerm`, `hashicorp/aws`), so validation needs no network.
|
|
|
|
- **`plan`** needs a customer's cloud credential and network access to their cloud, and agents get neither. So a new `terraform-plan` MCP server runs it, using the read-only credential stored per project at the gateway. It takes the generated files, runs `plan` only, and has no apply path. As in the source, it's a `cloud_read` tool, available only when `allow_cloud_read` is granted.
|
|
- **Published modules need the network.** Terraform has no offline mirror for registry modules, so the earlier plan to bake them in doesn't work. `terraform init` and the module lookup both reach `registry.terraform.io`, so a deployment that sets `TERRAFORM_MODULES` must allow that host in `AGENT_EGRESS_ALLOW` and its network policy. It's off by default, as in the source. Unlisted namespaces are still refused.
|
|
|
|
The three provider mirrors add 260 MB, so one image serves all three clouds. Kyndryl's CI may also hold the Dockerfile to a stricter shape, so confirm this before raising a pull request there. The fallback is a Terraform MCP server, which would sit outside this folder.
|
|
|
|
### 4.6 Live conversation
|
|
|
|
The agent answers two kinds of caller, and the request says which one it is: a workflow sends `phase`, and a conversation sends a message without one.
|
|
|
|
```
|
|
Workflow: step(phase) ──────────► Phase runner ──► structured reply ──► review point
|
|
Conversation: message + contextId ──► router ──┬─► question or approval request (reply writer)
|
|
└─► Phase runner ──► progress notes, then summary
|
|
```
|
|
|
|
- **One Execution per `contextId`.** The workspace maps it to the run, so any replica can take the next turn and a restart loses nothing.
|
|
- **What a turn does.** The router routes the message as in the source: an answer to a waiting question, a request to change submitted work, an approval, a question about delivered work, a new request, or something out of scope. It asks one set of questions at a time and waits.
|
|
- **A turn running a Phase.** The caller sees `progress()` notes throughout. If the turn reaches its time limit, finished Deliverables are kept, the reply says so, and the next message continues.
|
|
- **Concurrency.** A lock on the run stops a second message starting work at the same time. It gets a plain "still working" reply.
|
|
- **Approvals are recorded** in the manifest with the message that gave them, and the caller's identity when the request carries one.
|
|
|
|
**The gap is the client.** Direct conversation works from any A2A client, such as the platform's A2A Inspector or the template's `test_chatbot.py`, but those are developer tools, not a customer screen. Two options to close it, neither assumed in v1:
|
|
|
|
1. Spike (j): raise a question as an action-review pause, so it appears in KAB's run chat during a workflow. It's a stretch of a feature built for tool approvals, a pause inside a sub-agent reruns that sub-agent, and a step allows 10 pauses, so only the root code could ask.
|
|
2. A KAB change to add direct chat with an agent. It belongs to the platform team and sits outside this folder.
|
|
|
|
Without either, a workflow run in the KAB UI still falls back to answering questions in review comments, within the 5 reruns.
|
|
|
|
### 4.7 Workflow
|
|
|
|
`scripts/build_workflow.py` generates this, using `kab_front_steps.py`:
|
|
|
|
```
|
|
upload / artifacts → scoping ⏸ → requirements ⏸ → design ⏸ → implementation ⏸ → validation ⏸ → packaging ⏸
|
|
```
|
|
|
|
Discovery's output can come in through `select_artifacts`, which links the two CP1 agents. Phase timeouts start at 30 minutes, with `AGENT_MAX_DURATION_SECONDS` at 1740. Measure these in step 9.
|
|
|
|
### 4.8 Three hyperscalers
|
|
|
|
The engine mostly separates providers already, but the source built only for Google. `deliverables.py` hard-codes the six pillar headings, the eight guide sections and the Terraform attachment names. The source's own step 14 ("provider-owned contracts") deferred that fix until a second provider existed, so it now comes first.
|
|
|
|
| Per provider | Google Cloud | Azure | AWS |
|
|
|---|---|---|---|
|
|
| Bundle: scope values, models, roles, Terraform pins | port `gcp.yaml` | new | new |
|
|
| Well-Architected pillars | 6, vendored Skills | 5 pillars, new Skills | 6 pillars, new Skills |
|
|
| Solution guide template | vendored | new | new |
|
|
| IaC patterns Skill | port | new | new |
|
|
| Terraform modules Skill | port | new | new |
|
|
| Lessons Skill | port | starts empty | starts empty |
|
|
| Discovery server | `gcp-discovery` | `azure-discovery` | `aws-discovery` |
|
|
| Guidance | `cloud-knowledge`, provider `gcp` | provider `azure` | provider `aws` |
|
|
| Evaluation | 8 ported cases | at least 3 new | at least 3 new |
|
|
|
|
- **The rules are a checklist:** `grep -rni "terraform\|google\|gcp" app/core app/engine` should come back empty, and no pillar or section name may be a literal in `deliverables.py`.
|
|
- **Azure and AWS Skills are the biggest new content.** Google's come from its published `google/skills` repository. Use an official equivalent where one exists under a usable licence; otherwise write them from each provider's published Well-Architected Framework. Google's IaC patterns (100 KB) and modules (32 KB) are the guide to size.
|
|
- **Scope catalogue values** such as `shared_vpc_hub_and_spoke` become provider-specific in the bundles: Azure hub-and-spoke or Virtual WAN, AWS Transit Gateway.
|
|
- **Discovery tools are declared with their provider prefix.** `agent.yaml` is static, so it declares all three servers, and code offers only the Execution's provider. Each server must be registered in the tenant, and a customer credential is stored per project.
|
|
- **Multi-cloud designs are out of scope:** one Execution, one provider.
|
|
|
|
### 4.9 Jev, optional
|
|
|
|
Jev stays off unless `TYPESAFE_API_KEY` is set. The source's `Decider` interface is ported with it: `NoDecider` makes every decision take its fallback, and `JevDecider` calls the service. Each review keeps its own `JEV_*` switch and threshold.
|
|
|
|
- **Without a key:** nothing calls Jev, so the agent behaves the same on the platform with no Jev hosting or allowance.
|
|
- **To turn it on:** add `typesafe-sdk==0.7.2` to the image, put the key in the agent's Secret, set `TYPESAFE_BASE_URL`, add that host to `AGENT_EGRESS_ALLOW`, and allow it in any NetworkPolicy.
|
|
- **Data leaves the platform's gateways.** Jev receives requirements and user messages, which may be client information, and bypasses the gateway's logging and cost tracking. Each customer deployment should approve this before it's turned on. Record that in the ADR.
|
|
- **Evaluation runs both ways,** so the fallback isn't left untested.
|
|
|
|
### 4.10 Models
|
|
|
|
Every role, including the reply writer, uses `claude-sonnet-5-5`, set once as `LITELLM_MODEL` with no per-role overrides. `MODEL_TIERS` is removed.
|
|
|
|
- Discovery's critic deliberately runs on a different model from the agents it judges. Here the validator and coverage reviewer share the authors' model, so the checks written in code matter more.
|
|
- **Spike (f) failed on 2026-11-01: the cluster's LiteLLM doesn't serve `claude-sonnet-5-5`.** The agent can't run on it until the model is added there.
|
|
- Prompts were tuned on Gemini, so quality has to be measured, not assumed (step 9).
|
|
|
|
### 4.11 Change requests, lessons and overlays
|
|
|
|
- **Change requests** start a new Execution from a completed Baseline in the same engagement. The Baseline's run folder is read and never modified. The Execution delivers the full updated Deliverables with `changes.json`, as in the source.
|
|
- **Lesson candidates** are recorded in the workspace as they happen: Revisions and their reasons, tool refusals, and a user's requests for changes.
|
|
- **Curation stays offline and human-approved.** `scripts/curate_lessons.py` proposes Lessons from the volume, a person approves them, evaluation has to show they help, and they're published into each provider's lessons Skill. Publishing means a new image version. Azure and AWS lessons start empty and fill the same way.
|
|
- **Overlay Skills** are named when an Execution starts, and come only from configured folders: `skills/overlays/` in the image, plus an optional folder on the volume set by `OVERLAY_SKILLS_DIRS`. `read_skill` refuses anything else.
|
|
|
|
### 4.12 Discovery server review (2026-10-30)
|
|
|
|
**Superseded by 4.13 for this agent:** the gaps below are what the agent's own servers now fill. The table stays as the record of why.
|
|
|
|
**Result: the shared servers are not at parity.** The three servers are read from their definitions and code, not run against a cloud. They cover inventory, topology and policy well, but miss live metrics and much of the identity picture. All three also depend on a credential path that doesn't work yet.
|
|
|
|
The source allows 39 cloud-read tools across 11 Google servers, plus 3 documentation tools. The servers each define 13 tools, one a connection check, built for estate discovery, so depth differs even where the names match.
|
|
|
|
| Source capability | Google Cloud | Azure | AWS |
|
|
|---|---|---|---|
|
|
| Projects, accounts, structure | Yes | Yes | Yes |
|
|
| Asset inventory | Yes | Yes | Yes, needs AWS Config recording |
|
|
| Compute, disks, instance groups | Yes | Yes | Yes |
|
|
| Kubernetes clusters, node pools | Yes | Yes | Yes |
|
|
| Serverless | Yes | Yes | Yes |
|
|
| Databases, storage | Yes | Yes | Yes |
|
|
| Network topology | Yes, beyond the source | Yes | Yes |
|
|
| Recommendations | Yes, beyond the source | Yes | Yes |
|
|
| Org policies | Yes, beyond the source | Yes | Yes |
|
|
| Detailed configuration of one resource | **Unproven:** Cloud Asset search replaces each product's `get_*` | Likely: Resource Graph `properties` | Likely: Config record |
|
|
| Instance templates, Pub/Sub, alerts, dashboards | **Unproven:** only through a generic asset type | **Unproven:** through generic resource type | **Unproven:** through Config |
|
|
| Reservations, commitments | **Unproven**, same route | Not checked | Not checked |
|
|
| IAM on resources | Partial: lists bindings | **Missing:** no role assignments; `authorizationresources` isn't an allowed table | Partial: trust and resource policies, not role permissions |
|
|
| Who has access (`analyze_iam_policy`) | **Missing** | **Missing** | **Missing** |
|
|
| Roles and deny policies | **Missing.** The server plan says deny policies are covered, but the code doesn't read them | **Missing** | **Missing** |
|
|
| Live metrics (`list_timeseries`, metric descriptors) | **Missing** | **Missing**, planned "later" | **Missing** |
|
|
| Connectivity tests (validator role) | **Missing** | **Missing** | **Missing** |
|
|
| Available Kubernetes versions | **Missing** | **Missing** | **Missing** |
|
|
|
|
**Impact.** The missing metrics weaken sizing from real usage. The missing identity tools weaken the current-state step "identities and their roles", worst on Azure. The source's prompts don't name these tools, so nothing breaks; Specialists would record discovery gaps instead, as `cloud-discovery` tells them to. Connectivity tests matter least: the validator's prompt relies on Terraform tools.
|
|
|
|
**Status facts, including corrections:**
|
|
|
|
| Fact | State |
|
|
|---|---|
|
|
| Credentials | **Blocker.** The Vault Service isn't deployed on the dev cluster, so no stored credential can reach a server through the gateway. `docs/platform-issues/vault-service-not-deployed/`. This also blocks the agent's own servers and `terraform-plan`. Microsoft Learn itself needs no credential, but the gateway may still require a stored connection |
|
|
| Real-cloud testing | Azure once, on one dev subscription, bypassing the gateway. AWS and Google Cloud have only been tested against simulations, with no sandbox |
|
|
| Deployed and registered | Yes, in tenant `CP1 Agentic Landing Zone`. Not connected to any agent |
|
|
| Stale docs | `.agents/CONTEXT.md` says none is tested on a real cloud. The server plan says Azure doesn't back off on 429, but `azure_support.py` now does, unverified live |
|
|
| Rate limits | AWS calls don't use the shared back-off helper. Whether boto3's own retries are enough is unchecked |
|
|
| Documentation search | Endpoints exist for all three, in `knowledge_support.py`: `microsoft_docs_search`, `aws___search_documentation`, `search_documents`. The agent's servers call these |
|
|
|
|
### 4.13 The agent's own MCP servers (built 2026-10-30)
|
|
|
|
`mcp-servers/` holds three servers, rendered with the platform's builder, reusing the shared `_common/` code read-only. Nothing outside this folder changed. See its `README.md`.
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| `sa-gcp-cloud` | 44 tools: all 42 of the source's, plus a connection check and `gcp_describe_tool` |
|
|
| `sa-azure-cloud` | 40 tools, `sa-aws-cloud` 44 tools, matched by capability. No equivalent of `answer_query` |
|
|
| Local tests | All pass on the three servers, with upstreams simulated |
|
|
| Google tools, live | All 42 are offered and declare `readOnlyHint`, from anonymous `tools/list` on 2026-10-30. No tool has been called |
|
|
| AWS operations | Every operation, result key and paging token exists in boto3's service models. No call to AWS |
|
|
| Azure documentation, live | `azure_search_documents` and `azure_get_documents` both worked through the server on 2026-10-30 with a generic query, and each query was logged |
|
|
|
|
**Not proven:**
|
|
|
|
- No server has been called with a customer credential, against any real estate.
|
|
- Whether Google's hosted servers accept the `cloud-platform.read-only` token `gcp_support.py` issues. The source used its own Application Default Credentials.
|
|
- The AWS documentation tool `aws___read_documentation` and the sign-in to it. That name is from memory, and the server refuses cleanly if it isn't offered.
|
|
- Azure's Resource Graph queries and ARM API versions have never run, including `AuthorizationResources`.
|
|
- Images aren't built, nothing is deployed or registered, and no agent uses these servers.
|
|
|
|
**One credential per project.** These servers hold the customer's credential, so documentation search needs a stored connection too, unlike the source, where it ran on the agent's own identity. Google and AWS have optional documentation key fields for Kyndryl's keys. `search_cloud_docs` therefore can't run for a project until its connection is stored, which depends on the Vault Service.
|
|
|
|
### 4.14 Step 0 results (2026-10-30)
|
|
|
|
Run on the real 0.35.7 runtime source (commit `c0b4a4bf8`, ADK 2.12.0) on a local machine, with stub model and MCP gateway servers. This wasn't the image or the cluster. "Open" spikes need the cluster, which needs the owner's approval each time.
|
|
|
|
| Spike | Result | Consequence |
|
|
|---|---|---|
|
|
| (a) `ask_sub_agent` in `pre_process`, with `Reply` | **Pass.** No root model call. Two sub-agents ran in parallel: 2.2 s against 3 s in sequence | The design in 4.3 stands |
|
|
| (b) `call_tool` from a sub-agent's function | **Pass.** Tenant and project headers reach the gateway | `search_cloud_docs` as designed. No fallback needed |
|
|
| (c) `hitl_feedback` | **Partial.** It reaches `pre_process` if it's in the payload and declared in `schemas/input.json`. Delivery by the workflow engine is untested | Declare it in the schema |
|
|
| (d) Terraform mirror | **Pass for size:** 260 MB for all three providers, so one image. Running as the runtime user is untested | Pins chosen for now: `google` 8.6.0 (the source's), `azurerm` 4.40.0, `aws` 6.10.0 |
|
|
| (h) Plain text | **Changes the design.** A message with no JSON skips `pre_process` and the input contract, and goes straight to the root model | See below |
|
|
| (h) Free-text reply | **Works**, with a logged warning if an output contract is declared. Replies must carry JSON to pass it | Conversation replies are JSON with a `reply` field |
|
|
| (i) `contextId` | **Pass.** Hooks read it from `structlog.contextvars`, as `context_id` | Works, but it's not a documented interface. Fall back to `conversation_id` in the payload |
|
|
| Streaming | **Pass.** `progress()` notes arrive during `pre_process` | The design in 4.6 stands |
|
|
| (e), (f), (g), (k), (m), and (c) in the engine | **Open** | Need the cluster |
|
|
| (j) Questions in run chat | **Open, not worth doing yet.** It needs an action review, and the source shows those exist to approve tool calls | Revisit after v1 |
|
|
|
|
**Plain text in conversations.** Any A2A client can send prose, and prose never reaches `pre_process`. So the root agent gets one function, `run_turn(message)`, and its prompt says to call it with the user's message and return the result unchanged. This was tested: the function ran and could call `ask_sub_agent`. That adds one model call per plain-text turn, and relies on the model calling it. A client that sends `{"message": ...}` skips that call. The tests must check that a turn which doesn't call `run_turn` is refused in `post_process`.
|
|
|
|
### 4.15 LangGraph inside the agent (decided by the owner, 2026-10-30)
|
|
|
|
- **How it runs.** The graph is the source's, compiled in `app/engine/execution_graph.py` and run from `pre_process`. `interrupt()` pauses it there, and the hook returns the prompt as a `Reply`, so the runtime's refusal of `interrupt()` is never reached.
|
|
- **Checkpoints.** `app/engine/checkpoints.py` is a `BaseCheckpointSaver` on the shared volume, with write-once files and a file lock. There's no Postgres, and SQLite isn't safe on a network volume with several replicas. This replaces the hash-chained workspace for state, so checkpoints have no tamper check. Delivered files are written as in the source.
|
|
- **Time limits.** The engine reads `astream` one step at a time and stops between steps when time runs short. `carry_on` continues from the last checkpoint.
|
|
- **Versions.** `langgraph` 1.2.11, `langgraph-checkpoint` 4.2.0 and `langchain-core` 1.6.3, the source's, are installed into `/agent/vendor`. They install beside ADK 2.12.0 in a local environment. The image isn't built, so the Dockerfile's `pip` stage is untested.
|
|
|
|
**Verified on the 0.35.7 runtime source with a stub model:** a JSON request and a plain-text message both ran a sub-agent through `use_tool`, inside LangGraph, inside `pre_process`. The Execution paused, and a later request resumed it from the volume. With the stub, the scope analyst only ever asks a question, so this run stopped at scoping. All six Phases have run through the hooks with fake Specialists, but not on the runtime.
|
|
|
|
### 4.16 Official Skills for Azure and AWS (reviewed 2026-10-30)
|
|
|
|
Nothing is vendored yet. Licences are as each repository states them; Kyndryl's legal or open-source review hasn't seen them. Commits are the ones reviewed.
|
|
|
|
| Repository | Commit | Licence | What it offers | Verdict |
|
|
|---|---|---|---|---|
|
|
| `microsoft/skills`: `cloud-solution-architect` | `3f5b461b` | MIT | About 130 KB of written guidance: principles, styles, 44 patterns, technology choices, mission-critical | **Vendor.** Closest to Google's solution architecture Skill. No output template |
|
|
| `MicrosoftDocs/agent-skills`: `azure-well-architected` | `95544200` | CC BY 4.0 for content, MIT for code | One 30 KB index of Microsoft Learn URLs, no guidance of its own | Vendor with attribution, only useful with documentation access |
|
|
| `microsoft/azure-skills`: `azure-reliability`, cost, compliance, `azure-enterprise-infra-planner` | `fbda857c` | MIT | Candidates, not read in depth. The repository holds 220 script files | Review each; any Skill with `scripts/` is refused |
|
|
| `aws/agent-toolkit-for-aws`: service Skills | `8eee01d9` | Apache-2.0 | Official and maintained. `aws-security` (288 KB), `aws-iam`, `aws-compute`, `aws-database`, `aws-storage`, `aws-serverless`, `aws-networking` have no scripts | **Vendor** those that fit, with `NOTICE`. `aws-observability` has 26 scripts, so it needs trimming, which breaks "unmodified" |
|
|
| `aws/agent-toolkit-for-aws`: `aws-well-architected-review` | `8eee01d9` | Apache-2.0 | Reviews existing code and IaC. Reads the framework live and writes corpus files | **Don't use as a pillar Skill.** Wrong job, and heavy. Its references are useful background |
|
|
| `aws-samples/sample-well-architected-skills-and-steering` | `e81835b1` | MIT-0 | Six-pillar samples | **Don't use:** deprecated, and marked not for production |
|
|
| `awslabs/agent-plugins` | `fbefb376` | Apache-2.0 | Superseded by the Agent Toolkit | Don't use |
|
|
| `hashicorp/agent-skills`: `terraform-style-guide`, `azure-verified-modules` | `bc6b14b7` | MPL-2.0 | Terraform style and Azure modules. Most of the rest is for provider authors | Vendor unmodified, once legal has looked at MPL-2.0 |
|
|
|
|
**Still to write, because nothing official exists:** one Skill per pillar for Azure (5) and AWS (6), to match `pillar_skills`; a Solution guide template for each; IaC patterns for each; a modules Skill for AWS.
|
|
|
|
**Consequences:**
|
|
|
|
- The pillar Skills must carry their checklists themselves. `CLOUD_DOCS_SEARCH` is off by default, and Microsoft's index and AWS's review both depend on live documentation.
|
|
- Write them in our own words from each provider's published framework, and cite it.
|
|
- Vendored Skills are instructions a model follows. Pin each commit, keep checksums, and read every file before it's added, as with Google's.
|
|
- **Bug found in `sa-aws-cloud`:** AWS's documentation tool is `aws___read_documentation`, as assumed, but it takes `{"requests": [{"url": ..., "max_length": ...}]}`. `aws_get_documents` sends `{"url": ...}`, so it needs fixing in step 6.
|
|
|
|
## 5. Scope
|
|
|
|
Parity is the rule: anything the source does is in v1 unless this table says otherwise. Step 0 produces a line-by-line parity checklist from the source's README, `providers/gcp.yaml` and `.env.example`, and step 9 closes it.
|
|
|
|
| In v1 | Replaced by an equivalent | Intentionally different |
|
|
|---|---|---|
|
|
| All 6 Phases and 11 Deliverables | Own A2A server and `/generate`: the runtime, and a pre-approved run | One model for every role, so no `MODEL_TIERS` |
|
|
| Target architecture graph, Checks, Requirements chain | Postgres and SQLite records: the workspace and KAB's run records | Azure and AWS added |
|
|
| Work breakdown and Solution guide | Postgres or SQLite checkpoints: `VolumeSaver` on the volume | KAB workflow route added |
|
|
| Live conversation, Approvals and Revisions | Google's hosted servers: the same servers through `sa-gcp-cloud` | |
|
|
| Change requests and Baselines | | |
|
|
| Lessons loop, curation and proof | | |
|
|
| Overlay Skills | | |
|
|
| Terraform `fmt`, `validate`, `plan`, schema lookup, pinned modules | | |
|
|
| Free-text documentation search | | |
|
|
| Jev and every review, off unless configured | | |
|
|
| Evaluation harness and judge | | |
|
|
|
|
Still not included, because the source doesn't do them either: multi-cloud in one Execution, and a customer-ready chat screen in KAB.
|
|
|
|
## 6. Build steps
|
|
|
|
| # | Step | Done when |
|
|
|---|---|---|
|
|
| 0 | **Spikes on `agent-runtime-adk:0.35.7`:** (a) `ask_sub_agent` in `pre_process` with `Reply`, and parallel calls; (b) `call_tool` from a sub-agent's function; (c) rerun delivers `hitl_feedback` in an imported workflow; (d) Terraform with three provider mirrors, as the runtime user, and the image size; (e) the agent's three servers registered at the gateway, with their exact tool names, and each server in 4.13 called on a real estate per provider; (f) `claude-sonnet-5-5` on LiteLLM; (g) writes to the volume; (h) a plain-text message passes the input contract, and a free-text reply passes the output contract; (i) `contextId` reaches the hooks across turns; (j) a question through action review shows in run chat; (k) Jev reachable from the cluster; (m) a server can run `plan` with a per-project credential | Each has a recorded result, the design is updated, and the parity checklist exists. Features were read from 0.37.0 source, so 0.35.7 may differ. |
|
|
| 1 | Skeleton agent from `subagent-pattern`, with one stub Specialist | `check_agents.py` passes |
|
|
| 2 | Port the core and tests, then make it provider-owned (source step 14) | Ported tests pass; the grep in 4.8 is empty |
|
|
| 3 | Workspace, Ledger, router, Phase runner, hooks, `NoDecider` | `dry_run.py` completes all 6 Phases with fake Specialists, as a workflow and as a conversation, including a Revision and a resume from another process |
|
|
| 4 | Specialists and Google Cloud: prompts, `read_skill`, guidance, bundle | Roles load only their granted Skills; vendored checksums match |
|
|
| 5 | Terraform and validation | A known-bad case is refused and a known-good one passes offline, in the built image, for all three providers |
|
|
| 6 | Azure and AWS: bundles, Skills, templates, evaluation cases | A dry run completes for each provider with no code change |
|
|
| 7 | Parity features: Change requests, lessons loop, overlay Skills, `terraform-plan` server, `search_cloud_docs` wired to the built servers | Ported tests pass; `terraform-plan` passes its tests and refuses what it must; a Change request dry run leaves its Baseline unchanged |
|
|
| 8 | Jev: port the reviews, and write the workflow export, ADRs, README and user guide | Tests pass with and without a key; `kab_import_workflow.py` validates without `--apply` |
|
|
| 9 | Deploy to dev, run evaluation with real models, tune timeouts. **Needs your approval for cluster changes.** | All 8 Google cases compare well with the source; 3 cases pass on each of Azure and AWS; one full conversation runs to a Solution guide; the parity checklist is closed |
|
|
|
|
**Progress, 2026-10-30:**
|
|
|
|
- **Step 1 done.** The skeleton passes `check_agents.py`. It holds one stub Specialist, and every request returns `failed` until step 3.
|
|
- **Step 2 done for the code.** The core and provider loader are ported, with 80 tests passing and the provider grep empty on the core modules. Pillars and code files come from `ProviderContract`, which a bundle supplies. `tests/core/test_scope_spec.py` and two tests in `test_work_breakdown.py` drive the source's engine, so they return in step 3. `sa`-server mapping in `providers/gcp.yaml` is step 4: it still lists Google's URLs. Azure and AWS bundles are step 6.
|
|
|
|
- **Step 3 done.** The router, nodes, Ledger, intake, change requests, lessons recording and Jev's reviews are ported, and `turn` in `app/business_logic.py` serves both routes. 257 tests pass, 251 of them the source's own or ported from it. `tests/test_turns.py` runs a whole Execution through the hooks with a fresh engine for every request, including a Revision. A last `complete` workflow step ends the Execution. Seven source test files wait for `toolkit` and the Terraform tools (`tests/conftest.py`). `assemble` uses `NoDecider` and no tools yet, and Intake isn't wired into a conversation's first message: step 4 and step 8.
|
|
|
|
- **Step 4 done for Google Cloud.** All 16 roles' prompts are in, with 14 sub-agents declared. `load_skill` and `load_skill_resource` give each role its own Skills only. `app/mcp_tools.py` gives each role its bundle's tools on `sa-gcp-cloud`, and a failed call is a discovery gap, as in the source. `before_model` strips the root model's tools down to `run_turn`, confirmed on the runtime. `mcp:` declares `sa-gcp-cloud`, so **the agent won't start until that server is registered at the gateway**.
|
|
- **Step 5 done, except the image.** The Terraform tools and the schema review are ported. `fmt` and `validate` ran offline against a real mirror on all three providers, and bad code was refused. `terraform_plan` isn't offered until the `terraform-plan` server exists (step 7). The Dockerfile hasn't been built.
|
|
|
|
- **Step 6 partly done.** Azure and AWS bundles load with all roles. Written in Google's shape: 5 Azure and 6 AWS pillar Skills (3 to 6 KB each, against Google's 7 to 17 KB), a solution architecture Skill and guide template for each (derived from Google's, Apache-2.0, marked in their metadata), and IaC patterns (7 Azure, 8 AWS, against Google's 18), each passing `terraform validate` offline. A conversation keeps its cloud after the first message. The `aws_get_documents` bug is fixed. **Still to do:** vendoring (Microsoft's references need reading; AWS's service Skills turned out to be mostly routers to Skills we don't hold, so they may not be worth it), Azure and AWS evaluation cases, module Skills (`terraform_module` is refused on both for now), and a review of the Skills by someone who knows each cloud. Whole Executions on Azure and AWS have run through a fake Specialist only as far as the first Approval.
|
|
|
|
ADRs to write: LangGraph inside the agent and `VolumeSaver`; pattern and two ways in; Approvals as review points and messages; live conversation; Jev as an option; Skills for sub-agents; Terraform in the image; `terraform plan` through a server; free-text search exception; provider-owned contracts; one model for every role.
|
|
|
|
## 7. Risks
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|---|---|---|
|
|
| Spike (a), (b) or (h) fails | The runner, guidance or conversation changes shape | Fallbacks in 4.3 and 4.4; a separate agent for conversation if (h) fails; spikes run before any porting |
|
|
| Azure and AWS have no proven designs behind them | Weaker first results than Google, which went through ten iterations | New evaluation cases; describe them as first-version until they pass |
|
|
| Free-text search could carry client detail to a vendor, accepted by the owner | Queries aren't scrubbed, so a customer's name, project or anything else a model writes can reach the vendor | Prompt guidance, the query log and the per-deployment switch in 4.4. Check customer contracts before enabling it for a deployment |
|
|
| `terraform-plan` handles a customer credential | A credential or plan output could leak, or a plan could show secrets | Read-only credential per project, no apply path, every reply scrubbed, `allow_cloud_read` required |
|
|
| Parity makes v1 much larger | Longer before anything can be used | Run one Google Cloud case with real models after step 5, with your approval for the cluster change, so problems show early |
|
|
| No customer-ready conversation screen | Live conversation stays a developer route | Spike (j); raise with the platform team |
|
|
| One model authors and checks | Shared blind spots | Code checks stay authoritative; the Requirements chain is enforced in code |
|
|
| A Phase overruns a turn or step | Lost time | Write-once records make the rerun resume |
|
|
| Jev sends client data outside the gateways | Compliance | Off by default; approval per deployment |
|
|
| Client data on a shared volume | Retention not agreed | Same open question as Discovery ADR 0010 |
|
|
| Vault Service isn't deployed | No customer credential reaches any server, so cloud reads and `terraform plan` can't run | Needs the platform deployment's owner. Everything else works without it, with cloud reads recorded as discovery gaps |
|
|
| The agent's servers have never met a real cloud with a credential, and AWS and Google Cloud have no sandbox | Tools may fail at first use | A sandbox per provider (D7 in the server plan); spike (e) |
|
|
| Two sets of hyperscaler servers now exist | Fixes must be made twice, and customers may need to grant access to both | Shared `_common/` limits this; revisit if the Discovery agents adopt the same tools |
|
|
| Vendored Google Skills are third-party | Licence obligations | Keep `LICENSE`, `SHA256SUMS`, `VENDOR.md`; don't edit them |
|
|
|
|
## 8. Needed outside this folder (not done here)
|
|
|
|
Until these change, the agent can be checked, and its workflow validated, by passing explicit paths.
|
|
|
|
| File | Change | Effect until then |
|
|
|---|---|---|
|
|
| `scripts/render_k8s.py` | Add `outcome-pattern` to `PATTERNS` | **Blocks deployment through the normal route** |
|
|
| `deploy/crucible-kab-dev.yaml` | Add the agent's entry | Blocks deployment |
|
|
| `scripts/render_k8s.py` and `deploy/crucible-kab-dev.yaml` | Find and list the servers in this folder; the script looks only in the root `mcp-servers/` | **Blocks deploying this folder's servers** |
|
|
| `mcp-servers/README.md` | Note that this folder's servers are an accepted exception to the fixed-list rule | Rule and servers disagree |
|
|
| `docs/customer-cloud-access.md` | The roles these servers need, such as Google's `roles/mcp.toolUser`, monitoring reads and AWS `cloudwatch` | Customers grant too little |
|
|
| `.agents/CONTEXT.md`, `docs/hyperscaler-mcp-servers-plan-2026-10-30.md` | Correct the stale facts in 4.12 | Docs mislead |
|
|
| KAB dev cluster | Deploy the Vault Service. Platform owner, not this repository | **Blocks every stored credential** |
|
|
| `.gitea/workflows/check.yml` | Add `outcome-pattern` to the three `find` commands, a dry-run step and these servers' tests | CI doesn't check this agent |
|
|
| `CLAUDE.md`, `README.md`, `.agents/CONTEXT.md` | Describe the third pattern | Docs say there are two |
|