ALZ-F01 Discovery: package step, value scorecard, application grouping, cases

- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent f659564208
commit 9e503885cc
211 changed files with 18170 additions and 158 deletions

View File

@@ -0,0 +1,98 @@
# Discovery output value: plan
| | |
|---|---|
| Date | 2026-11-05 |
| Status | Phases 0 and 1 live ([ADR 0020](adr/0020.md)); the rest is planned |
| Decided by | Product owner, 2026-11-05: go ahead, with Discovery and Assessment kept as separate workflows; combining them is decided later |
## 1. Purpose and boundaries
Make the live Discovery workflow's output useful to the people and workflow that consume it, using ideas from the
original `gcp_solution_architecture_agent`.
- **Consumers:** the Engagement Lead at the review point, and the Assessment workflow (ALZ-F02), assuming F02 is
still required. No other workflow or pattern consumes Discovery's output.
- **Separate streams:** the solution architecture agent (`outcome-pattern/`) and the sub-agent pattern are
independent. No hand-over is built to them; only ideas are borrowed from the original agent, never its code or
engine.
- **The owner's order holds:** Azure working (done 2026-11-05), then the package step, then Discovery meets every
F02 requirement.
- **Discovery and Assessment stay separate workflows.** Whether to also offer one combined run is decided later.
## 2. What "meaningful value" means
| Consumer | The output must answer |
|---|---|
| Engagement Lead | What's there, what's wrong, what's missing, and what must be decided |
| F02 Assessment | For each application: its assets, dependencies, support status, data class, compliance, cost and owner, each with evidence |
The **value scorecard** (`cp1-discovery-package-agent/app/scorecard.py`) measures this for every run, in code: F02
fields filled, declared or missing; F02's run-level checks; decisions left for a person; findings with an owner and
an action. Cases also carry limits on tokens, model calls and agent seconds.
**Baseline**, case `azure-crucible-kab-dev-cmdb`, run `exec_14956d524a8b`, 2026-11-05 (46 of 46 checks):
| Measure | Value |
|---|---|
| F02 fields filled | 24% |
| F02 fields filled or declared | 27% |
| Missing everywhere | application, owner, support status, end of support, compliance scope, annual cost |
| F02 run checks | 6 of 7 (no applications list) |
| Decisions for a person | 9 |
| Findings with an owner / an action | 0% / 100% |
| Cost | about 106k tokens, 10 model calls, 108 agent seconds |
| Package size | about 100 KB |
## 3. Borrowed from the original
| Idea | In the original | In Discovery |
|---|---|---|
| Field meanings: `ask`, `decide`, `not_applicable`, provenance, a catalogue, a `check` command | ADR-0008, `app/scope.py` | The run context (phase 2) |
| An explicit graph, and an id chain code won't break | ADR-0005, the requirements chain | Application nodes in the ADR 0019 graph (phase 3); evidence on every finding (phase 4) |
| A deliverable manifest; customer questions | `execution.json`, `customer-questions.md` | The package manifest (phase 1); decisions needed (phase 4) |
| A Baseline and its changes | ADR-0006, `changes.json` | Re-discovery against the last approved run (phase 5) |
| An evaluation harness | `solution_eval`, judge calibration, limits | Value bars and limits (phase 0); a judge (phase 6) |
| Lessons and overlays | ADR-0004 | Phase 7 |
**Left behind:** its own server, database, LangGraph and `interrupt()` (they conflict with KAB); one model for every
role (Discovery's critic stays on a different model); Jev and unscrubbed free-text search (data would leave the
gateways).
## 4. Phases
Sizes are relative. "F02-only" marks work that only F02 justifies.
| # | Phase | Size | What | Done when | Status |
|---|---|---|---|---|---|
| 0 | Value baseline | S | The scorecard; `value:` bars and `limits:` in both Azure cases; `scripts/collect_run_usage.py` | Both cases have a baseline | **built** (ADR 0020) |
| 1 | Discovery package | M, F02-only | Step `s13`, `cp1-discovery-package-agent`: every part unchanged, a manifest with sha256 per part, completeness and the scorecard | An F02 run starts from one artifact id | **live** (ADR 0020): both Azure cases pass with complete packages; F02's own change to verify the manifest is next |
| 2 | Run context field meanings | M | A catalogue for `context.yaml` (value, `ask`, `decide`, `not_applicable`; essential, important, optional) covering compliance scope, criticality, owner, business services; `check_context.py`; provenance per fact; assumptions shown apart | A `decide` gives a visible assumption; an unset essential field stops the run before it scans | todo. ADR 0016 (Proposed) needs the owner's review first: it carries the file to solution architecture |
| 3 | Close F02's data gaps | L, mostly F02-only | Applications as graph nodes (CMDB business service, then tags, then resource group or naming, then model-proposed and judged in code); CMDB owners onto records; end-of-life rules in code over `provider_attributes`; cost through fixed-shape queries on the existing discovery servers; per-application summaries for size; the repositories source | The Azure case reaches the agreed F02 bar | **applications and owners built** ([ADR 0021](adr/0021.md)): CMDB→tag→container→judged model, owners from the CMDB or an owner tag; expected 46% filled and 7 of 7 run checks. End-of-life, cost and per-application summaries still todo |
| 4 | A report that leads to decisions | M | First page: decisions needed, top risks, open questions. Each finding has a severity, an owner (else "unowned"), an action and its consumer; code refuses a finding without evidence or an action | 100% of findings with an owner and an action | todo |
| 5 | Re-discovery against a Baseline | M | A run may name an earlier approved execution; a code-only diff step gives added, removed and changed assets, edges and drift | Identical runs give an empty diff; a planted change shows | todo |
| 6 | Evaluation harness | M | A report judge on a different model, calibrated on 10–20 labelled sections; runs with and without a feature | Each phase's effect is measured | todo |
| 7 | Lessons and overlays | M | Repeated critic findings become approved rules; customer standards arrive as overlays | — | later |
Phases 0, 2, 4, 5 and 6 are worth doing whatever happens to F02; 1 and 3 are justified by F02.
## 5. Decisions for the owner
1. **Is F02 still required?** It gates the rest of phase 3.
2. **ADR 0016:** the owner will go through its fields one by one before phase 2 starts: [run-context-field-review.md](run-context-field-review.md).
3. ~~**Application grouping**~~ decided 2026-11-05: CMDB, then tag, then resource group, then a judged model
proposal, with model groups reviewed (ADR 0021).
4. **Cost and end-of-life sources:** cost queries on the existing servers (needs Cost Management Reader or the
equivalent) or an uploaded file; end-of-life from a table in code or the provider's lifecycle data.
5. **The value bar** that counts as "Discovery meets F02".
6. **Later:** a combined Discovery and Assessment run, beside the separate workflows.
## 6. Risks
| Risk | Mitigation |
|---|---|
| The package outgrows KAB's 5 MB inline limit | Per-application summaries (phase 3), or a platform zip step |
| Wrong application groups mislead F02's scoring | Checks in code, the critic, and review of model-proposed groups |
| Cost and lifecycle need permissions beyond Reader | Update `docs/customer-cloud-access.md`; a missing permission is a declared gap |
| Borrowing turns into porting | Each borrowing gets its own ADR; ideas only |
| Model cost grows with new steps | The limits in each case; code-only steps wherever possible |