Files
cp1-agents/individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/README.md
Jonathan Boniface 9e503885cc ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

4.5 KiB

Workflow test cases

Test data for judging what a whole Discovery workflow run produces. Each agent's eval/cases.yaml tests that agent alone. These cases test the workflow: the data each step produced, what survived the hand-offs, and whether the run reached the right conclusions.

Each case folder holds:

File What it is
case.yaml The run's inputs, and the facts a correct run must show
brief.txt The file uploaded to the run
context.yaml For a live case: the run context the run starts from (--context)
other files Files the context's uploads: lists, uploaded after it, such as cmdb-export.csv

The expected facts come from the data the run reads, never from an earlier run. For meridian-baseline, that's mcp-servers/alz-discovery-sample.server.yaml. If that data changes, update the case.

Run a case

From the repository root, with a fresh token from scripts/kab_token.py:

F="individual-agents/ALZ-F01: Discovery Workflow Agent"
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
  --input engagement_id=ENG-MERIDIAN-001 --input run_id=RUN-001 --file "$F/workflows/cases/meridian-baseline/brief.txt"
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/meridian-baseline/case.yaml" <folder>/steps.json

A live case starts from its context file instead, and the files its uploads: lists are uploaded after it:

python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> --context "$F/workflows/cases/azure-crucible-kab-dev-cmdb/context.yaml"
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
python3 "$F/scripts/collect_run_usage.py" <execution id> --out <folder>/usage.json     # read-only kubectl logs
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/azure-crucible-kab-dev-cmdb/case.yaml" <folder>/steps.json --usage <folder>/usage.json
Case What it covers
meridian-baseline, meridian-without-itsm, snapshot-missing-sources The Meridian sample estate (provider: sample)
gcp-live, azure-live Context files for live runs on a whole project or subscription; no checks
azure-crucible-kab-dev Live Azure, one resource group, no CMDB or ITSM: every gap declared, and a complete package
azure-crucible-kab-dev-cmdb The same group with an uploaded CMDB export and relationships, planting drift, ghosts, a dangling relationship and a prompt injection

Reading the result

  • Critical: a failure means the output can't be trusted, whatever else passed.
  • Major: a real gap in the result.
  • Review: a person must judge it. The checker only lists these.
  • Value scorecard (ADR 0020): printed for every run, from the package step's manifest, or computed the same way from the step outputs when the run has no package. It shows how much of what the Assessment workflow needs the run gives, the decisions it leaves for a person, and how many findings name an owner and an action.
  • Value bar and limits: a case's value: and limits: entries fail as major.

A passing case shows the run found these facts. It doesn't prove the wider output is good, so read the review items as well.

Adding a case

Add a folder with its own case.yaml. Each check names a step (s1 to s9, s11, s13), a field and one test. A nested field is named with dots, such as manifest.complete on the package step.

Test Passes when
contains, contains_all The field's JSON has this text, or all of these
not_contains The field's JSON doesn't have this text
equals The field is this value
min_items The field's list has at least this many items
item Items matching match (a regular expression over each item's JSON) exist, and each passes the other tests given, such as contains. With match alone, at least one item matches. With absent: true, no item matches

A case may also hold:

Key Entries Fails when
value: f02_filled_min, f02_filled_or_declared_min, f02_missing_max, f02_run_met_min, findings_with_owner_min, findings_with_action_min The run's scorecard is below the bar (above it, for _max)
limits: tokens, model_calls, seconds (agent time, from --usage) The run used more. Without --usage they're reported as not measured

Set a bar at the baseline so a change can't lower it, and raise it as the scorecard improves. Never lower a bar to make a run pass.