ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,17 +11,31 @@ Status values: `done`, `blocked (<on whom>)`, `todo`.
|
||||
| 3 | Workflow export: 8 steps, `from_steps` hand-offs, critic approval | `build_workflow.py` asserts the fields on both sides; same keys as cp4's export | done |
|
||||
| 4 | Deploy profile and kustomize render | `render_k8s.py` + `kustomize build` (3 objects each) | done |
|
||||
| 5 | Docs: README, user guides, ADRs 0001–0010; agent context | Review | done |
|
||||
| 6 | Fix the MCP gateway tenant mapping so MCP servers can be registered | Registration saves | blocked (KAB platform admin) |
|
||||
| 6 | Register the MCP servers. Done in the CP1 tenant, under the nine logical names, not by fixing the default tenant | Gateway reports 15 healthy servers | done (2026-11-01) |
|
||||
| 7 | Build and push the eight images | `podman push` | done (2026-10-29, on the pinned runtime 0.35.7) |
|
||||
| 8 | Create the eight agents in KAB; store each `MCP_AGENT_API_KEY` | Secrets exist; no file holds a key | blocked (step 6) |
|
||||
| 8 | Store each `MCP_AGENT_API_KEY`, using gateway keys from `scripts/kab_bootstrap_discovery.py`. The keys expire after 30 days. Whether the agents also need creating in KAB for the workflow is unconfirmed | Secrets exist; no file holds a key; each pod resolves its MCP tools at startup | done (2026-11-01) |
|
||||
| 9 | `kubectl diff -k`, then `apply -k`, for each agent | Pods ready; agent cards served | done (2026-10-29): all 8 Ready, cards served; requests fail at startup until step 8 |
|
||||
| 9a | Uploads and artifacts (ADR 0011): `source_documents` / `reference_documents` on the planner and inventory (0.2.0, regenerated), front steps in `build_workflow.py`, `scripts/kab_import_workflow.py` | cp4 check 0 problems; runtime parse; eval validation; builder asserts wiring | done (2026-10-29) |
|
||||
| 9b | Move into `kab-agents-and-workflows/cp1-agents` (`individual-agents/<Feature>/`); shared scripts, MCP server and deploy profile at the repository root; `repo: cp1-agents-individual` kept per agent (Deployment selector) | `check_agents.py` 0 problems; `build_workflow.py` output identical; `kubectl diff -k` empty for all 8 | done (2026-10-29) |
|
||||
| 9c | Timeouts: every workflow step 20 minutes (`AGENT_STEP_TIMEOUT_MINUTES`, shared); `AGENT_MAX_DURATION_SECONDS` 1140 in each ConfigMap and in the scaffold | Export rebuilt; `kubectl diff -k` shows only that line for all 8 | done (2026-10-29); not yet applied |
|
||||
| 10 | Run each agent's eval cases, then the workflow on `ENG-MERIDIAN-001` | All cases pass; the critic step waits for approval | todo |
|
||||
| 11 | Compare with `sub-agents/ALZ-F01: Discovery Workflow Agent/` and record which pattern is adopted | An ADR in both patterns | todo (product owner) |
|
||||
| 9c | Timeouts: every workflow step 20 minutes (`AGENT_STEP_TIMEOUT_MINUTES`, shared); `AGENT_MAX_DURATION_SECONDS` 1140 in each ConfigMap and in the scaffold | Export rebuilt; `kubectl diff -k` shows only that line for all 8 | done (2026-11-01): applied to all 8, and running |
|
||||
| 10 | Run each agent's eval cases, then the workflow on `ENG-MERIDIAN-001` | All cases pass; the critic step waits for approval | done (2026-11-01): evals 8 of 8; workflow run `exec_fbf1d3af1290` scored 27 of 29 on `workflows/cases/meridian-baseline`. Five findings to fix are in the engineering hub report `reports/cp1/discovery-run-quality-2026-11-01.md` |
|
||||
| 10a | Fix the five findings from the first run (`reports/cp1/discovery-run-quality-2026-11-01.md`): planner 0.4.0, scan, network, dependency and critic 0.2.0, sample server 0.1.2 | Workflow case `meridian-baseline`, now 33 checks; run `exec_fbf1d3af1290` scores 26 of 33 against it | done (2026-11-01): `RUN-007` scored 36 of 36 with critic verdict `pass`. Agent evals not rerun since these changes |
|
||||
| 10b | Report step (ADR 0012): `cp1-discovery-report-agent`, rendered in code; the approval gate moves to it; case now 41 checks; Markdown for the record, HTML for the customer | `scripts/render_discovery_report.py` wrote the `RUN-007` report | done (2026-11-01): deployed; `wf_06af245b-dc0e-4bf5-90ff-ab74a4a4330f`; `RUN-008` scored 41 of 41 with verdict `pass`. Publishing to KAB not yet tried |
|
||||
| 10c | Missing sources: `source_gaps` from the planner, respected by CMDB, classification, critic and report; case `snapshot-missing-sources` | `RUN-GCP-002` passed 8 of 8 on `wf_795e1918-e240-4854-b31e-282a1ab655b8` | done (2026-11-01); Meridian `RUN-009` 41 of 42; inventory 0.3.1 deployed, then `RUN-010` |
|
||||
| 10d | Report narrative: a model writes a marked executive summary and recommendations from the structured findings only. Code keeps every finding; a check confirms each cited id exists and that source gaps and the verdict are stated; the Engagement Lead approves it at the gate. Test prompt injection on it. Start after the sub-agent pattern's first live run, so the patterns compare cleanly. Update ADR 0012 | Cited ids checked in code; injection test passes | todo |
|
||||
| 11 | Live provider (ADR 0013): `provider` input, live tools on 5 agents, Live provider prompt sections (`scripts/add_live_provider.py`); then build, deploy and run on GCP `delry-cp1-prj-1239-a31894bc` | `check_agents.py` 0 problems; workflow export rebuilt; live GCP run | done (2026-11-03): deployed; live runs on GCP and, from 2026-11-04, Azure |
|
||||
| 11a | First live GCP run `exec_0f8629066e13` (verdict `pass`, approved by the user). Fixes: dependencies keep out-of-scope targets (0.4.2), planner never scopes projects or subscriptions (0.7.3), critic checks dropped dependencies and structural scope entries and runs on `gpt-5.4` (0.4.3) | Rerun on GCP | done (2026-11-03): deployed; `RUN-GCP-LIVE-004` (`exec_42fc8affdbdd`) held all four fixes |
|
||||
| 11b | Well-Architected review (ADR 0014): new code-only agent `cp1-well-architected-review-agent` 0.1.0, workflow steps `s11` and `s12`; richer report (0.6.0) from `provider_attributes` (scan 0.4.0, inventory 0.5.0) and `topology_summary` (network 0.4.0) | `tests/test_waf.py` 6 passed; report rendered from `RUN-GCP-LIVE-002` data; `check_agents.py` 10 folders, 0 problems; export 13 steps | done (2026-11-03): deployed with its MCP key; `exec_42fc8affdbdd` completed all 13 steps |
|
||||
| 12 | Value plan phase 0 (ADR 0020, `docs/discovery-value-plan.md`): the value scorecard (`cp1-discovery-package-agent/app/scorecard.py`) in `evaluate_workflow_run.py`; `value:` and `limits:` in both Azure cases (`azure-crucible-kab-dev` gains its `case.yaml`); `scripts/collect_run_usage.py` | Baseline on `exec_14956d524a8b`: 24% filled, 27% filled or declared, 6 of 7 run checks; usage 106k tokens, 10 calls, 108 s, within limits | done (2026-11-05) |
|
||||
| 12a | Value plan phase 1 (ADR 0020): `cp1-discovery-package-agent` 0.1.0, step `s13` after the report's review point, 14 steps; the context report gains its 2026-11-05 wiring section | `tests/test_package.py` 8 passed; `check_context_doc.py` matches; `check_agents.py` 11 folders, 0 problems; `exec_14956d524a8b` 46 of 46 and `exec_d1c242609f82` 19 of 19 | done (2026-11-05): deployed, registered in the agent catalog (`discover-and-register`), workflow `wf_05565d2d` updated to 14 steps |
|
||||
| 12b | F02 verifies the package's manifest in code (F02's own ADR), and selects one artifact | An F02 run from one package id | todo (F02) |
|
||||
| 12c | Value plan phases 2–7 | `docs/discovery-value-plan.md` | todo; phase 2 waits on the owner's review of ADR 0016 |
|
||||
| 13 | Applications and asset owners (ADR 0021, value plan phase 3, first part): CMDB 0.7.0 (owner, business service, support group per CI), dependency mapping 0.6.0 (`app/applications.py`, sub-agent `propose_application_groups`), report 0.9.0 (Applications section), package 0.2.0 (`applications` part, scorecard 2); both Azure cases gain application checks and raised value bars | dependency 20, report 8, package 9, CMDB 16 tests passed; `check_agents.py` 0 problems; `check_context_doc.py` matches | done (2026-11-05): deployed; `exec_58ba6a71d00b` 56 of 56 (46% filled, 7 of 7 run checks, 33% of findings with an owner) and `exec_925194d79c47` 21 of 21 |
|
||||
| 14 | Compare with `sub-agents/ALZ-F01: Discovery Workflow Agent/` and record which pattern is adopted | An ADR in both patterns | todo (product owner) |
|
||||
|
||||
## Open questions (owner)
|
||||
|
||||
- **Platform team:** the tenant mapping (step 6). Does the executor externalise large step outputs between steps? (ADR 0010)
|
||||
- **Product owner:** confirm the Proposed ADRs: 0003, 0004, 0005, 0006, 0007, 0008, 0010.
|
||||
- ~~**Blocks the critic:** `claude-sonnet-5-5` isn't served on the cluster's LiteLLM.~~ Resolved 2026-11-03: the critic runs on `gpt-5.4` (ADR 0008, Accepted).
|
||||
|
||||
- **Platform team:** does the executor externalise large step outputs between steps? (ADR 0010)
|
||||
- **Product owner:** confirm the Proposed ADRs: 0003, 0004, 0005, 0006, 0007, 0010, and review ADR 0016 field by field (it carries the run context to solution architecture, which is a separate stream).
|
||||
|
||||
Reference in New Issue
Block a user