ALZ-F01 Discovery: package step, value scorecard, application grouping, cases

- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent f659564208
commit 9e503885cc
211 changed files with 18170 additions and 158 deletions

View File

@@ -4,9 +4,10 @@ Facts for this Feature's individual-agents pattern, for coding agents. Repositor
## Pattern
- **Eight declarative agents, one per ALZ-DSC story.** Each declares its own read-only MCP tools under `mcp:` (allowed for top-level agents; only sub-agents can't hold MCP tools).
- **Eleven agents:** one per ALZ-DSC story, plus the report, the Well-Architected review and the package. Each declares its own read-only MCP tools under `mcp:` (allowed for top-level agents; only sub-agents can't hold MCP tools). The package agent declares none.
- **The workflow (`workflows/cp1-alz-discovery-individual.json`) is the orchestration:** step order through `parallelGroup` and `executionLevel`, and data through `from_steps` (`"<input>": "sN.<output field>"`). The step map is `STEPS` in `scripts/build_workflow.py`.
- **Nothing persists between steps** except their replies (ADR 0010). There's no workspace, no hooks and no enforced verdict; the critic is gated by a human approval (ADR 0006).
- **Nothing persists between steps** except their replies (ADR 0010). There's no workspace. Hooks answer the steps their ADRs name. The review point is on the report step (ADR 0012); the pages and the package (`s13`, ADR 0020) run only after it.
- **The package is Discovery's hand-over to the Assessment workflow (F02), its only consumer.** The solution architecture agent (`outcome-pattern/`) is a separate stream: build no hand-over to it.
## Sources of truth
@@ -22,4 +23,5 @@ Facts for this Feature's individual-agents pattern, for coding agents. Repositor
- **Runtime:** `cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7`. Images are pushed to `cruciblecoredevacr.azurecr.io/kab/cp1/<agent>:<version>`.
- **Cluster:** `crucible-kab-dev-aks`, namespace `agents`, kubeconfig `/mnt/c/Users/JonathanBoniface/.kube/config`. The gateways are in `core-components`: LiteLLM on `:4000`, the MCP gateway on `:8000`. There's no `agent-platform-connections` ConfigMap, so the deploy profile sets the gateway URLs.
- **An agent needs its own `MCP_AGENT_API_KEY`** (Secret `<agent>-secrets`), issued when it's created in KAB. Without it, every request fails at startup.
- **As of 2026-10-29, MCP server registration fails** in the MCP gateway, because KAB's default tenant `6ada663eb5b88053a9e43723` isn't mapped there (it fails row-level security). This blocks every agent in both patterns.
- **The `agents` namespace allows 30 Deployments** (ResourceQuota `kab-eng-agents`) and is full. On 2026-11-05, to deploy Assessment, `cp4-spike-echo-agent` and the sub-agent pattern's `cp1-alz-discovery-orchestrator-agent` were deleted at the owner's choice; their manifests are in the engineering hub's `reports/cp1/deleted-deployment-*-2026-11-05.yaml`. A new agent needs a free slot.
- **MCP servers and agent keys belong to the CP1 tenant** (`6ae37d13999438461a529409`). Registration still fails in KAB's default tenant, which these agents don't use.

View File

@@ -11,17 +11,31 @@ Status values: `done`, `blocked (<on whom>)`, `todo`.
| 3 | Workflow export: 8 steps, `from_steps` hand-offs, critic approval | `build_workflow.py` asserts the fields on both sides; same keys as cp4's export | done |
| 4 | Deploy profile and kustomize render | `render_k8s.py` + `kustomize build` (3 objects each) | done |
| 5 | Docs: README, user guides, ADRs 0001–0010; agent context | Review | done |
| 6 | Fix the MCP gateway tenant mapping so MCP servers can be registered | Registration saves | blocked (KAB platform admin) |
| 6 | Register the MCP servers. Done in the CP1 tenant, under the nine logical names, not by fixing the default tenant | Gateway reports 15 healthy servers | done (2026-11-01) |
| 7 | Build and push the eight images | `podman push` | done (2026-10-29, on the pinned runtime 0.35.7) |
| 8 | Create the eight agents in KAB; store each `MCP_AGENT_API_KEY` | Secrets exist; no file holds a key | blocked (step 6) |
| 8 | Store each `MCP_AGENT_API_KEY`, using gateway keys from `scripts/kab_bootstrap_discovery.py`. The keys expire after 30 days. Whether the agents also need creating in KAB for the workflow is unconfirmed | Secrets exist; no file holds a key; each pod resolves its MCP tools at startup | done (2026-11-01) |
| 9 | `kubectl diff -k`, then `apply -k`, for each agent | Pods ready; agent cards served | done (2026-10-29): all 8 Ready, cards served; requests fail at startup until step 8 |
| 9a | Uploads and artifacts (ADR 0011): `source_documents` / `reference_documents` on the planner and inventory (0.2.0, regenerated), front steps in `build_workflow.py`, `scripts/kab_import_workflow.py` | cp4 check 0 problems; runtime parse; eval validation; builder asserts wiring | done (2026-10-29) |
| 9b | Move into `kab-agents-and-workflows/cp1-agents` (`individual-agents/<Feature>/`); shared scripts, MCP server and deploy profile at the repository root; `repo: cp1-agents-individual` kept per agent (Deployment selector) | `check_agents.py` 0 problems; `build_workflow.py` output identical; `kubectl diff -k` empty for all 8 | done (2026-10-29) |
| 9c | Timeouts: every workflow step 20 minutes (`AGENT_STEP_TIMEOUT_MINUTES`, shared); `AGENT_MAX_DURATION_SECONDS` 1140 in each ConfigMap and in the scaffold | Export rebuilt; `kubectl diff -k` shows only that line for all 8 | done (2026-10-29); not yet applied |
| 10 | Run each agent's eval cases, then the workflow on `ENG-MERIDIAN-001` | All cases pass; the critic step waits for approval | todo |
| 11 | Compare with `sub-agents/ALZ-F01: Discovery Workflow Agent/` and record which pattern is adopted | An ADR in both patterns | todo (product owner) |
| 9c | Timeouts: every workflow step 20 minutes (`AGENT_STEP_TIMEOUT_MINUTES`, shared); `AGENT_MAX_DURATION_SECONDS` 1140 in each ConfigMap and in the scaffold | Export rebuilt; `kubectl diff -k` shows only that line for all 8 | done (2026-11-01): applied to all 8, and running |
| 10 | Run each agent's eval cases, then the workflow on `ENG-MERIDIAN-001` | All cases pass; the critic step waits for approval | done (2026-11-01): evals 8 of 8; workflow run `exec_fbf1d3af1290` scored 27 of 29 on `workflows/cases/meridian-baseline`. Five findings to fix are in the engineering hub report `reports/cp1/discovery-run-quality-2026-11-01.md` |
| 10a | Fix the five findings from the first run (`reports/cp1/discovery-run-quality-2026-11-01.md`): planner 0.4.0, scan, network, dependency and critic 0.2.0, sample server 0.1.2 | Workflow case `meridian-baseline`, now 33 checks; run `exec_fbf1d3af1290` scores 26 of 33 against it | done (2026-11-01): `RUN-007` scored 36 of 36 with critic verdict `pass`. Agent evals not rerun since these changes |
| 10b | Report step (ADR 0012): `cp1-discovery-report-agent`, rendered in code; the approval gate moves to it; case now 41 checks; Markdown for the record, HTML for the customer | `scripts/render_discovery_report.py` wrote the `RUN-007` report | done (2026-11-01): deployed; `wf_06af245b-dc0e-4bf5-90ff-ab74a4a4330f`; `RUN-008` scored 41 of 41 with verdict `pass`. Publishing to KAB not yet tried |
| 10c | Missing sources: `source_gaps` from the planner, respected by CMDB, classification, critic and report; case `snapshot-missing-sources` | `RUN-GCP-002` passed 8 of 8 on `wf_795e1918-e240-4854-b31e-282a1ab655b8` | done (2026-11-01); Meridian `RUN-009` 41 of 42; inventory 0.3.1 deployed, then `RUN-010` |
| 10d | Report narrative: a model writes a marked executive summary and recommendations from the structured findings only. Code keeps every finding; a check confirms each cited id exists and that source gaps and the verdict are stated; the Engagement Lead approves it at the gate. Test prompt injection on it. Start after the sub-agent pattern's first live run, so the patterns compare cleanly. Update ADR 0012 | Cited ids checked in code; injection test passes | todo |
| 11 | Live provider (ADR 0013): `provider` input, live tools on 5 agents, Live provider prompt sections (`scripts/add_live_provider.py`); then build, deploy and run on GCP `delry-cp1-prj-1239-a31894bc` | `check_agents.py` 0 problems; workflow export rebuilt; live GCP run | done (2026-11-03): deployed; live runs on GCP and, from 2026-11-04, Azure |
| 11a | First live GCP run `exec_0f8629066e13` (verdict `pass`, approved by the user). Fixes: dependencies keep out-of-scope targets (0.4.2), planner never scopes projects or subscriptions (0.7.3), critic checks dropped dependencies and structural scope entries and runs on `gpt-5.4` (0.4.3) | Rerun on GCP | done (2026-11-03): deployed; `RUN-GCP-LIVE-004` (`exec_42fc8affdbdd`) held all four fixes |
| 11b | Well-Architected review (ADR 0014): new code-only agent `cp1-well-architected-review-agent` 0.1.0, workflow steps `s11` and `s12`; richer report (0.6.0) from `provider_attributes` (scan 0.4.0, inventory 0.5.0) and `topology_summary` (network 0.4.0) | `tests/test_waf.py` 6 passed; report rendered from `RUN-GCP-LIVE-002` data; `check_agents.py` 10 folders, 0 problems; export 13 steps | done (2026-11-03): deployed with its MCP key; `exec_42fc8affdbdd` completed all 13 steps |
| 12 | Value plan phase 0 (ADR 0020, `docs/discovery-value-plan.md`): the value scorecard (`cp1-discovery-package-agent/app/scorecard.py`) in `evaluate_workflow_run.py`; `value:` and `limits:` in both Azure cases (`azure-crucible-kab-dev` gains its `case.yaml`); `scripts/collect_run_usage.py` | Baseline on `exec_14956d524a8b`: 24% filled, 27% filled or declared, 6 of 7 run checks; usage 106k tokens, 10 calls, 108 s, within limits | done (2026-11-05) |
| 12a | Value plan phase 1 (ADR 0020): `cp1-discovery-package-agent` 0.1.0, step `s13` after the report's review point, 14 steps; the context report gains its 2026-11-05 wiring section | `tests/test_package.py` 8 passed; `check_context_doc.py` matches; `check_agents.py` 11 folders, 0 problems; `exec_14956d524a8b` 46 of 46 and `exec_d1c242609f82` 19 of 19 | done (2026-11-05): deployed, registered in the agent catalog (`discover-and-register`), workflow `wf_05565d2d` updated to 14 steps |
| 12b | F02 verifies the package's manifest in code (F02's own ADR), and selects one artifact | An F02 run from one package id | todo (F02) |
| 12c | Value plan phases 2–7 | `docs/discovery-value-plan.md` | todo; phase 2 waits on the owner's review of ADR 0016 |
| 13 | Applications and asset owners (ADR 0021, value plan phase 3, first part): CMDB 0.7.0 (owner, business service, support group per CI), dependency mapping 0.6.0 (`app/applications.py`, sub-agent `propose_application_groups`), report 0.9.0 (Applications section), package 0.2.0 (`applications` part, scorecard 2); both Azure cases gain application checks and raised value bars | dependency 20, report 8, package 9, CMDB 16 tests passed; `check_agents.py` 0 problems; `check_context_doc.py` matches | done (2026-11-05): deployed; `exec_58ba6a71d00b` 56 of 56 (46% filled, 7 of 7 run checks, 33% of findings with an owner) and `exec_925194d79c47` 21 of 21 |
| 14 | Compare with `sub-agents/ALZ-F01: Discovery Workflow Agent/` and record which pattern is adopted | An ADR in both patterns | todo (product owner) |
## Open questions (owner)
- **Platform team:** the tenant mapping (step 6). Does the executor externalise large step outputs between steps? (ADR 0010)
- **Product owner:** confirm the Proposed ADRs: 0003, 0004, 0005, 0006, 0007, 0008, 0010.
- ~~**Blocks the critic:** `claude-sonnet-5-5` isn't served on the cluster's LiteLLM.~~ Resolved 2026-11-03: the critic runs on `gpt-5.4` (ADR 0008, Accepted).
- **Platform team:** does the executor externalise large step outputs between steps? (ADR 0010)
- **Product owner:** confirm the Proposed ADRs: 0003, 0004, 0005, 0006, 0007, 0010, and review ADR 0016 field by field (it carries the run context to solution architecture, which is a separate stream).