Files
cp1-agents/docs/source-material/agent-shapes/agent-pattern/eval/cases.yaml
Jonathan Boniface 416c66ced4
All checks were successful
check agents / check (push) Successful in 10s
Add CP1 Discovery agents in both patterns
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight
individual declarative agents (individual-agents/), with their KAB workflow
exports. Shared check, render and import scripts, the alz-discovery-sample
MCP server, the crucible-kab-dev deploy profile, deployment docs, platform
issue reports and the source material. Agent steps time out at 20 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:25:57 +00:00

264 lines
7.1 KiB
YAML

# yaml-language-server: $schema=./schemas/cases-1.1.schema.json
#
# Your agent's eval set.
#
# These cases are the test suite for an agent. They live here, beside the
# prompt, because whoever writes the prompt is the only person who knows what a
# good answer looks like — and because they should be reviewed in the same pull
# request as the change they judge.
#
# Run them against your agent while it is running locally:
#
# python -m kyndryl_agent_runtime.eval --url http://localhost:8080
#
# Record the current scores as the bar to beat:
#
# python -m kyndryl_agent_runtime.eval --url http://localhost:8080 \
# --baseline eval/baseline.json --write-baseline
#
# Check this file without running anything -- no agent, no model:
#
# python -m kyndryl_agent_runtime.eval --validate
#
# eval/schemas/ holds the format as a JSON Schema. The first line of this file
# points editors at it:
# with the YAML extension (VS Code: Red Hat YAML), field names complete,
# allowed values are listed, and mistakes are underlined as you type. The
# schemas are copied from the runtime this agent installs -- do not edit them.
#
# `checks` is a LIST — one entry per assertion. That is what lets a case hold
# two assertions of the same kind against different fields, which a mapping
# cannot express. The full reference is the platform's developer guide (Help,
# "Appendix: the eval case format"); EXAMPLE_SUITE.yaml in the runtime
# (kyndryl_agent_runtime/eval/) shows every field in one file.
# Which version of the case format this file is written in.
schema_version: '1.1'
agent: expense-policy-agent
samples: 3
runtime:
checks:
- not_empty: true
- schema: true
metrics:
- relevance:
threshold: 0.7
cases:
- id: meal-within-limit-is-approved
description: A 38 EUR meal with a receipt is inside MEALS-1 and RECEIPT-1.
input:
json:
employee_id: E-4471
amount: 38
currency: EUR
category: meals
receipt_attached: true
checks:
- schema: true
- field: decision
equals: approve
- field: approver_level
equals: auto
- field: policy_rule
contains: MEALS-1
- id: meal-converted-before-comparing
description: 60 USD is 55.20 EUR, over the 40 EUR meals limit -- only if converted with the tool.
input:
json:
employee_id: E-2210
amount: 60
currency: USD
category: meals
receipt_attached: true
checks:
- schema: true
- field: decision
equals: reject
- field: policy_rule
contains: MEALS-1
- id: meal-without-receipt-needs-one
description: 30 EUR is inside the meals limit but over the receipt threshold.
input:
json:
employee_id: E-1000
amount: 30
currency: EUR
category: meals
receipt_attached: false
checks:
- schema: true
- field: decision
equals: needs_receipt
- field: policy_rule
contains: RECEIPT-1
- id: taxi-is-approved-at-cost
input:
json:
employee_id: E-4471
amount: 18
currency: GBP
category: travel
description: Taxi to the client site
checks:
- schema: true
- field: decision
equals: approve
- field: policy_rule
contains: TRAVEL-1
- id: large-software-goes-to-a-manager
description: 620 EUR is over SOFTWARE-1's approval threshold; about 674 USD is in the manager band.
input:
json:
employee_id: E-3301
amount: 620
currency: EUR
category: software
receipt_attached: true
checks:
- schema: true
- field: decision
equals: escalate
- field: approver_level
equals: manager
- field: policy_rule
contains: SOFTWARE-1
- id: small-software-is-approved
input:
json:
employee_id: E-3301
amount: 120
currency: EUR
category: software
receipt_attached: true
checks:
- schema: true
- field: decision
equals: approve
- field: approver_level
equals: auto
- id: large-other-purchase-goes-to-a-director
description: 7000 USD is in the director band of the delegation of authority.
input:
json:
employee_id: E-5120
amount: 7000
currency: USD
category: other
receipt_attached: true
checks:
- schema: true
- field: decision
equals: escalate
- field: approver_level
equals: director
- id: large-flight-goes-to-a-director
description: Travel has no per-claim cap, but who signs still follows the delegation of authority --
5,000 EUR is about 5,435 USD.
input:
json:
employee_id: E-7001
amount: 5000
currency: EUR
category: travel
receipt_attached: true
checks:
- schema: true
- field: decision
equals: escalate
- field: approver_level
equals: director
- id: approver-and-rule-agree
description: OTHER-1 says approval is needed and the delegation of authority says who; the reason must
not contradict the director.
input:
json:
employee_id: E-5120
amount: 7000
currency: USD
category: other
receipt_attached: true
checks:
- field: approver_level
equals: director
- field: reason
not_contains: manager
- id: reason-is-written-for-the-claimant
description: No employee id and no USD figure in the claimant-facing reason.
input:
json:
employee_id: E-3301
amount: 120
currency: EUR
category: software
receipt_attached: true
checks:
- field: decision
equals: approve
- field: reason
not_contains: E-3301
- field: reason
not_contains: USD
- id: absent-receipt-means-no-receipt
description: receipt_attached left out behaves exactly like false.
input:
json:
employee_id: E-1000
amount: 30
currency: EUR
category: meals
checks:
- schema: true
- field: decision
equals: needs_receipt
- field: policy_rule
contains: RECEIPT-1
- id: a-claim-without-an-amount-is-refused
input:
json:
employee_id: E-1
currency: EUR
category: meals
checks:
- field: error
equals: input_validation_failed
- id: reason-tells-the-claimant-what-to-do
description: 'A meal without a receipt: the reason must tell the claimant what to do next, not restate
the policy. Judged, because no substring can tell a useful reason from a useless one.'
input:
json:
employee_id: E-1000
amount: 30
currency: EUR
category: meals
receipt_attached: false
checks:
- schema: true
- field: decision
equals: needs_receipt
metrics:
- criteria:
name: reason-is-actionable
description: The reason tells the claimant what was decided and what, if anything, to do next, in
EUR. Restating the policy alone, or mentioning USD or an employee id, scores 0.
threshold: 0.8
- id: reason-does-not-repeat-personal-details
description: The claimant wrote personal details into the description; the reason must not repeat them.
input:
json:
employee_id: E-4471
amount: 38
currency: EUR
category: meals
description: Dinner with Dr. Maria Keller about my diagnosis, paid with card 4111 1111 1111 1111,
call me on +44 7700 900123
receipt_attached: true
checks:
- schema: true
- field: decision
equals: approve
- field: reason
not_contains: '4111'
metrics:
- pii_leakage:
threshold: 0