# Workflow test cases Test data for judging what a whole Discovery workflow run produces. Each agent's `eval/cases.yaml` tests that agent alone. These cases test the workflow: the data each step produced, what survived the hand-offs, and whether the run reached the right conclusions. Each case folder holds: | File | What it is | |---|---| | `case.yaml` | The run's inputs, and the facts a correct run must show | | `brief.txt` | The file uploaded to the run | | `context.yaml` | For a live case: the run context the run starts from (`--context`) | | other files | Files the context's `uploads:` lists, uploaded after it, such as `cmdb-export.csv` | The expected facts come from the data the run reads, never from an earlier run. For `meridian-baseline`, that's `mcp-servers/alz-discovery-sample.server.yaml`. If that data changes, update the case. ## Run a case From the repository root, with a fresh token from `scripts/kab_token.py`: ```sh F="individual-agents/ALZ-F01: Discovery Workflow Agent" python3 scripts/kab_run_workflow.py start --workflow --project \ --input engagement_id=ENG-MERIDIAN-001 --input run_id=RUN-001 --file "$F/workflows/cases/meridian-baseline/brief.txt" python3 scripts/kab_run_workflow.py outputs --out python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/meridian-baseline/case.yaml" /steps.json ``` A live case starts from its context file instead, and the files its `uploads:` lists are uploaded after it: ```sh python3 scripts/kab_run_workflow.py start --workflow --project --context "$F/workflows/cases/azure-crucible-kab-dev-cmdb/context.yaml" python3 scripts/kab_run_workflow.py outputs --out python3 "$F/scripts/collect_run_usage.py" --out /usage.json # read-only kubectl logs python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/azure-crucible-kab-dev-cmdb/case.yaml" /steps.json --usage /usage.json ``` | Case | What it covers | |---|---| | `meridian-baseline`, `meridian-without-itsm`, `snapshot-missing-sources` | The Meridian sample estate (`provider: sample`) | | `gcp-live`, `azure-live` | Context files for live runs on a whole project or subscription; no checks | | `azure-crucible-kab-dev` | Live Azure, one resource group, no CMDB or ITSM: every gap declared, and a complete package | | `azure-crucible-kab-dev-cmdb` | The same group with an uploaded CMDB export and relationships, planting drift, ghosts, a dangling relationship and a prompt injection | ## Reading the result - **Critical:** a failure means the output can't be trusted, whatever else passed. - **Major:** a real gap in the result. - **Review:** a person must judge it. The checker only lists these. - **Value scorecard** (ADR 0020): printed for every run, from the package step's manifest, or computed the same way from the step outputs when the run has no package. It shows how much of what the Assessment workflow needs the run gives, the decisions it leaves for a person, and how many findings name an owner and an action. - **Value bar and limits:** a case's `value:` and `limits:` entries fail as major. A passing case shows the run found these facts. It doesn't prove the wider output is good, so read the `review` items as well. ## Adding a case Add a folder with its own `case.yaml`. Each check names a step (`s1` to `s9`, `s11`, `s13`), a field and one test. A nested field is named with dots, such as `manifest.complete` on the package step. | Test | Passes when | |---|---| | `contains`, `contains_all` | The field's JSON has this text, or all of these | | `not_contains` | The field's JSON doesn't have this text | | `equals` | The field is this value | | `min_items` | The field's list has at least this many items | | `item` | Items matching `match` (a regular expression over each item's JSON) exist, and each passes the other tests given, such as `contains`. With `match` alone, at least one item matches. With `absent: true`, no item matches | A case may also hold: | Key | Entries | Fails when | |---|---|---| | `value:` | `f02_filled_min`, `f02_filled_or_declared_min`, `f02_missing_max`, `f02_run_met_min`, `findings_with_owner_min`, `findings_with_action_min` | The run's scorecard is below the bar (above it, for `_max`) | | `limits:` | `tokens`, `model_calls`, `seconds` (agent time, from `--usage`) | The run used more. Without `--usage` they're reported as not measured | Set a bar at the baseline so a change can't lower it, and raise it as the scorecard improves. Never lower a bar to make a run pass.