Files
cp1-agents/individual-agents/ALZ-F01: Discovery Workflow Agent/docs/discovery-value-plan.md
Jonathan Boniface 9e503885cc ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

99 lines
7.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Discovery output value: plan
| | |
|---|---|
| Date | 2026-11-05 |
| Status | Phases 0 and 1 live ([ADR 0020](adr/0020.md)); the rest is planned |
| Decided by | Product owner, 2026-11-05: go ahead, with Discovery and Assessment kept as separate workflows; combining them is decided later |
## 1. Purpose and boundaries
Make the live Discovery workflow's output useful to the people and workflow that consume it, using ideas from the
original `gcp_solution_architecture_agent`.
- **Consumers:** the Engagement Lead at the review point, and the Assessment workflow (ALZ-F02), assuming F02 is
still required. No other workflow or pattern consumes Discovery's output.
- **Separate streams:** the solution architecture agent (`outcome-pattern/`) and the sub-agent pattern are
independent. No hand-over is built to them; only ideas are borrowed from the original agent, never its code or
engine.
- **The owner's order holds:** Azure working (done 2026-11-05), then the package step, then Discovery meets every
F02 requirement.
- **Discovery and Assessment stay separate workflows.** Whether to also offer one combined run is decided later.
## 2. What "meaningful value" means
| Consumer | The output must answer |
|---|---|
| Engagement Lead | What's there, what's wrong, what's missing, and what must be decided |
| F02 Assessment | For each application: its assets, dependencies, support status, data class, compliance, cost and owner, each with evidence |
The **value scorecard** (`cp1-discovery-package-agent/app/scorecard.py`) measures this for every run, in code: F02
fields filled, declared or missing; F02's run-level checks; decisions left for a person; findings with an owner and
an action. Cases also carry limits on tokens, model calls and agent seconds.
**Baseline**, case `azure-crucible-kab-dev-cmdb`, run `exec_14956d524a8b`, 2026-11-05 (46 of 46 checks):
| Measure | Value |
|---|---|
| F02 fields filled | 24% |
| F02 fields filled or declared | 27% |
| Missing everywhere | application, owner, support status, end of support, compliance scope, annual cost |
| F02 run checks | 6 of 7 (no applications list) |
| Decisions for a person | 9 |
| Findings with an owner / an action | 0% / 100% |
| Cost | about 106k tokens, 10 model calls, 108 agent seconds |
| Package size | about 100 KB |
## 3. Borrowed from the original
| Idea | In the original | In Discovery |
|---|---|---|
| Field meanings: `ask`, `decide`, `not_applicable`, provenance, a catalogue, a `check` command | ADR-0008, `app/scope.py` | The run context (phase 2) |
| An explicit graph, and an id chain code won't break | ADR-0005, the requirements chain | Application nodes in the ADR 0019 graph (phase 3); evidence on every finding (phase 4) |
| A deliverable manifest; customer questions | `execution.json`, `customer-questions.md` | The package manifest (phase 1); decisions needed (phase 4) |
| A Baseline and its changes | ADR-0006, `changes.json` | Re-discovery against the last approved run (phase 5) |
| An evaluation harness | `solution_eval`, judge calibration, limits | Value bars and limits (phase 0); a judge (phase 6) |
| Lessons and overlays | ADR-0004 | Phase 7 |
**Left behind:** its own server, database, LangGraph and `interrupt()` (they conflict with KAB); one model for every
role (Discovery's critic stays on a different model); Jev and unscrubbed free-text search (data would leave the
gateways).
## 4. Phases
Sizes are relative. "F02-only" marks work that only F02 justifies.
| # | Phase | Size | What | Done when | Status |
|---|---|---|---|---|---|
| 0 | Value baseline | S | The scorecard; `value:` bars and `limits:` in both Azure cases; `scripts/collect_run_usage.py` | Both cases have a baseline | **built** (ADR 0020) |
| 1 | Discovery package | M, F02-only | Step `s13`, `cp1-discovery-package-agent`: every part unchanged, a manifest with sha256 per part, completeness and the scorecard | An F02 run starts from one artifact id | **live** (ADR 0020): both Azure cases pass with complete packages; F02's own change to verify the manifest is next |
| 2 | Run context field meanings | M | A catalogue for `context.yaml` (value, `ask`, `decide`, `not_applicable`; essential, important, optional) covering compliance scope, criticality, owner, business services; `check_context.py`; provenance per fact; assumptions shown apart | A `decide` gives a visible assumption; an unset essential field stops the run before it scans | todo. ADR 0016 (Proposed) needs the owner's review first: it carries the file to solution architecture |
| 3 | Close F02's data gaps | L, mostly F02-only | Applications as graph nodes (CMDB business service, then tags, then resource group or naming, then model-proposed and judged in code); CMDB owners onto records; end-of-life rules in code over `provider_attributes`; cost through fixed-shape queries on the existing discovery servers; per-application summaries for size; the repositories source | The Azure case reaches the agreed F02 bar | **applications and owners built** ([ADR 0021](adr/0021.md)): CMDB→tag→container→judged model, owners from the CMDB or an owner tag; expected 46% filled and 7 of 7 run checks. End-of-life, cost and per-application summaries still todo |
| 4 | A report that leads to decisions | M | First page: decisions needed, top risks, open questions. Each finding has a severity, an owner (else "unowned"), an action and its consumer; code refuses a finding without evidence or an action | 100% of findings with an owner and an action | todo |
| 5 | Re-discovery against a Baseline | M | A run may name an earlier approved execution; a code-only diff step gives added, removed and changed assets, edges and drift | Identical runs give an empty diff; a planted change shows | todo |
| 6 | Evaluation harness | M | A report judge on a different model, calibrated on 10–20 labelled sections; runs with and without a feature | Each phase's effect is measured | todo |
| 7 | Lessons and overlays | M | Repeated critic findings become approved rules; customer standards arrive as overlays | — | later |
Phases 0, 2, 4, 5 and 6 are worth doing whatever happens to F02; 1 and 3 are justified by F02.
## 5. Decisions for the owner
1. **Is F02 still required?** It gates the rest of phase 3.
2. **ADR 0016:** the owner will go through its fields one by one before phase 2 starts: [run-context-field-review.md](run-context-field-review.md).
3. ~~**Application grouping**~~ decided 2026-11-05: CMDB, then tag, then resource group, then a judged model
proposal, with model groups reviewed (ADR 0021).
4. **Cost and end-of-life sources:** cost queries on the existing servers (needs Cost Management Reader or the
equivalent) or an uploaded file; end-of-life from a table in code or the provider's lifecycle data.
5. **The value bar** that counts as "Discovery meets F02".
6. **Later:** a combined Discovery and Assessment run, beside the separate workflows.
## 6. Risks
| Risk | Mitigation |
|---|---|
| The package outgrows KAB's 5 MB inline limit | Per-application summaries (phase 3), or a platform zip step |
| Wrong application groups mislead F02's scoring | Checks in code, the critic, and review of model-proposed groups |
| Cost and lifecycle need permissions beyond Reader | Update `docs/customer-cloud-access.md`; a missing permission is a declared gap |
| Borrowing turns into porting | Each borrowing gets its own ADR; ideas only |
| Model cost grows with new steps | The limits in each case; code-only steps wherever possible |