- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.5 KiB
Workflow test cases
Test data for judging what a whole Discovery workflow run produces. Each agent's eval/cases.yaml tests that agent alone. These cases test the workflow: the data each step produced, what survived the hand-offs, and whether the run reached the right conclusions.
Each case folder holds:
| File | What it is |
|---|---|
case.yaml |
The run's inputs, and the facts a correct run must show |
brief.txt |
The file uploaded to the run |
context.yaml |
For a live case: the run context the run starts from (--context) |
| other files | Files the context's uploads: lists, uploaded after it, such as cmdb-export.csv |
The expected facts come from the data the run reads, never from an earlier run. For meridian-baseline, that's mcp-servers/alz-discovery-sample.server.yaml. If that data changes, update the case.
Run a case
From the repository root, with a fresh token from scripts/kab_token.py:
F="individual-agents/ALZ-F01: Discovery Workflow Agent"
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
--input engagement_id=ENG-MERIDIAN-001 --input run_id=RUN-001 --file "$F/workflows/cases/meridian-baseline/brief.txt"
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/meridian-baseline/case.yaml" <folder>/steps.json
A live case starts from its context file instead, and the files its uploads: lists are uploaded after it:
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> --context "$F/workflows/cases/azure-crucible-kab-dev-cmdb/context.yaml"
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
python3 "$F/scripts/collect_run_usage.py" <execution id> --out <folder>/usage.json # read-only kubectl logs
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/azure-crucible-kab-dev-cmdb/case.yaml" <folder>/steps.json --usage <folder>/usage.json
| Case | What it covers |
|---|---|
meridian-baseline, meridian-without-itsm, snapshot-missing-sources |
The Meridian sample estate (provider: sample) |
gcp-live, azure-live |
Context files for live runs on a whole project or subscription; no checks |
azure-crucible-kab-dev |
Live Azure, one resource group, no CMDB or ITSM: every gap declared, and a complete package |
azure-crucible-kab-dev-cmdb |
The same group with an uploaded CMDB export and relationships, planting drift, ghosts, a dangling relationship and a prompt injection |
Reading the result
- Critical: a failure means the output can't be trusted, whatever else passed.
- Major: a real gap in the result.
- Review: a person must judge it. The checker only lists these.
- Value scorecard (ADR 0020): printed for every run, from the package step's manifest, or computed the same way from the step outputs when the run has no package. It shows how much of what the Assessment workflow needs the run gives, the decisions it leaves for a person, and how many findings name an owner and an action.
- Value bar and limits: a case's
value:andlimits:entries fail as major.
A passing case shows the run found these facts. It doesn't prove the wider output is good, so read the review items as well.
Adding a case
Add a folder with its own case.yaml. Each check names a step (s1 to s9, s11, s13), a field and one test. A
nested field is named with dots, such as manifest.complete on the package step.
| Test | Passes when |
|---|---|
contains, contains_all |
The field's JSON has this text, or all of these |
not_contains |
The field's JSON doesn't have this text |
equals |
The field is this value |
min_items |
The field's list has at least this many items |
item |
Items matching match (a regular expression over each item's JSON) exist, and each passes the other tests given, such as contains. With match alone, at least one item matches. With absent: true, no item matches |
A case may also hold:
| Key | Entries | Fails when |
|---|---|---|
value: |
f02_filled_min, f02_filled_or_declared_min, f02_missing_max, f02_run_met_min, findings_with_owner_min, findings_with_action_min |
The run's scorecard is below the bar (above it, for _max) |
limits: |
tokens, model_calls, seconds (agent time, from --usage) |
The run used more. Without --usage they're reported as not measured |
Set a bar at the baseline so a change can't lower it, and raise it as the scorecard improves. Never lower a bar to make a run pass.