- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
78 lines
4.5 KiB
Markdown
78 lines
4.5 KiB
Markdown
# Workflow test cases
|
|
|
|
Test data for judging what a whole Discovery workflow run produces. Each agent's `eval/cases.yaml` tests that agent alone. These cases test the workflow: the data each step produced, what survived the hand-offs, and whether the run reached the right conclusions.
|
|
|
|
Each case folder holds:
|
|
|
|
| File | What it is |
|
|
|---|---|
|
|
| `case.yaml` | The run's inputs, and the facts a correct run must show |
|
|
| `brief.txt` | The file uploaded to the run |
|
|
| `context.yaml` | For a live case: the run context the run starts from (`--context`) |
|
|
| other files | Files the context's `uploads:` lists, uploaded after it, such as `cmdb-export.csv` |
|
|
|
|
The expected facts come from the data the run reads, never from an earlier run. For `meridian-baseline`, that's `mcp-servers/alz-discovery-sample.server.yaml`. If that data changes, update the case.
|
|
|
|
## Run a case
|
|
|
|
From the repository root, with a fresh token from `scripts/kab_token.py`:
|
|
|
|
```sh
|
|
F="individual-agents/ALZ-F01: Discovery Workflow Agent"
|
|
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
|
|
--input engagement_id=ENG-MERIDIAN-001 --input run_id=RUN-001 --file "$F/workflows/cases/meridian-baseline/brief.txt"
|
|
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
|
|
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/meridian-baseline/case.yaml" <folder>/steps.json
|
|
```
|
|
|
|
A live case starts from its context file instead, and the files its `uploads:` lists are uploaded after it:
|
|
|
|
```sh
|
|
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> --context "$F/workflows/cases/azure-crucible-kab-dev-cmdb/context.yaml"
|
|
python3 scripts/kab_run_workflow.py outputs <execution id> --out <folder>
|
|
python3 "$F/scripts/collect_run_usage.py" <execution id> --out <folder>/usage.json # read-only kubectl logs
|
|
python3 "$F/scripts/evaluate_workflow_run.py" "$F/workflows/cases/azure-crucible-kab-dev-cmdb/case.yaml" <folder>/steps.json --usage <folder>/usage.json
|
|
```
|
|
|
|
| Case | What it covers |
|
|
|---|---|
|
|
| `meridian-baseline`, `meridian-without-itsm`, `snapshot-missing-sources` | The Meridian sample estate (`provider: sample`) |
|
|
| `gcp-live`, `azure-live` | Context files for live runs on a whole project or subscription; no checks |
|
|
| `azure-crucible-kab-dev` | Live Azure, one resource group, no CMDB or ITSM: every gap declared, and a complete package |
|
|
| `azure-crucible-kab-dev-cmdb` | The same group with an uploaded CMDB export and relationships, planting drift, ghosts, a dangling relationship and a prompt injection |
|
|
|
|
## Reading the result
|
|
|
|
- **Critical:** a failure means the output can't be trusted, whatever else passed.
|
|
- **Major:** a real gap in the result.
|
|
- **Review:** a person must judge it. The checker only lists these.
|
|
- **Value scorecard** (ADR 0020): printed for every run, from the package step's manifest, or computed the same way
|
|
from the step outputs when the run has no package. It shows how much of what the Assessment workflow needs the run
|
|
gives, the decisions it leaves for a person, and how many findings name an owner and an action.
|
|
- **Value bar and limits:** a case's `value:` and `limits:` entries fail as major.
|
|
|
|
A passing case shows the run found these facts. It doesn't prove the wider output is good, so read the `review` items as well.
|
|
|
|
## Adding a case
|
|
|
|
Add a folder with its own `case.yaml`. Each check names a step (`s1` to `s9`, `s11`, `s13`), a field and one test. A
|
|
nested field is named with dots, such as `manifest.complete` on the package step.
|
|
|
|
| Test | Passes when |
|
|
|---|---|
|
|
| `contains`, `contains_all` | The field's JSON has this text, or all of these |
|
|
| `not_contains` | The field's JSON doesn't have this text |
|
|
| `equals` | The field is this value |
|
|
| `min_items` | The field's list has at least this many items |
|
|
| `item` | Items matching `match` (a regular expression over each item's JSON) exist, and each passes the other tests given, such as `contains`. With `match` alone, at least one item matches. With `absent: true`, no item matches |
|
|
|
|
A case may also hold:
|
|
|
|
| Key | Entries | Fails when |
|
|
|---|---|---|
|
|
| `value:` | `f02_filled_min`, `f02_filled_or_declared_min`, `f02_missing_max`, `f02_run_met_min`, `findings_with_owner_min`, `findings_with_action_min` | The run's scorecard is below the bar (above it, for `_max`) |
|
|
| `limits:` | `tokens`, `model_calls`, `seconds` (agent time, from `--usage`) | The run used more. Without `--usage` they're reported as not measured |
|
|
|
|
Set a bar at the baseline so a change can't lower it, and raise it as the scorecard improves. Never lower a bar to
|
|
make a run pass.
|