Files
gcp_solution_architecture_a…/eval/datasets/solution_cases.json
Jonathan Boniface 5ef434d00e feat: Change requests against a Baseline (ADR-0006)
A caller restates a request to say how requirements have changed, naming a
completed Execution of the same Customer case as the Baseline.

- app/baseline.py reads a Baseline from its files and refuses one that
  does not exist, did not complete or belongs to another case, before
  anything runs or is recorded. A Baseline is never modified.
- Requirements carry a change (unchanged, added, changed, removed) and a
  reason. Submission is refused when a Baseline requirement disappears, an
  id is reused, or an unchanged requirement's text differs. A removed
  requirement stays on record and asks nothing downstream.
- Each Specialist starts from its delivered version; the graph and the code
  are its working copy, and the description is carried over when the
  design's shape is the same.
- app/workflows/changes.py compares versions by id and traces what each
  changed requirement reaches; anything else that changed is listed as
  unrequested. Approvals show the change for their Phase, and changes.json
  is written with the full Deliverables.
- Intake no longer starts a message read as a change: it names the
  Execution to give as the Baseline.
- A fifth evaluation case runs a request and then a change to it, with
  four checks that the change is recorded, reaches Checks, stays contained
  and costs less than its Baseline.

Tested with scripted agents only; no Change request has run with real
models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:38:15 +01:00

372 lines
14 KiB
JSON

[
{
"id": "case-01-event-driven-http",
"name": "Event-driven regional webhook ingestion",
"workflow_request": "Design an event-driven regional HTTP application on Google Cloud that accepts incoming webhooks, validates their signatures, durably enqueues payloads for asynchronous processing, and retains raw events for 30 days for replay and audit.",
"required_products": [
"Cloud Run",
"Pub/Sub",
"Cloud Storage"
],
"scripted_answers": [
[
"region|location|geograph",
"A single region, europe-west2 (London). Data must stay in the UK."
],
[
"availability|uptime|sla|rto|rpo|disaster|recover",
"99.9% monthly availability. RTO 1 hour, RPO 5 minutes for accepted events. No multi-region DR in phase one."
],
[
"traffic|volume|throughput|scale|peak|requests per|rps|payload",
"Average 50 webhooks per second, peaks of 500 per second for 10 minutes. Payloads up to 256 KB."
],
[
"latency|response time|performance",
"Acknowledge each webhook within 300 ms at p95."
],
[
"security|compliance|encrypt|privacy|gdpr|pci|access|identity|auth",
"UK GDPR applies. Google-managed encryption keys are acceptable. Senders sign requests with HMAC-SHA256 shared secrets; only processing services may read events."
],
[
"cost|budget|spend",
"Under 1,500 GBP a month at average load; prefer pay-per-use."
],
[
"current|existing|today|on-prem|migrat|legacy",
"This is a new workload; nothing runs today."
],
[
"depend|integrat|downstream|upstream|consum",
"Processed events go to an existing internal REST API that is reachable over HTTPS; it can be slow or unavailable for minutes."
],
[
"operat|monitor|logging|alert|team|support",
"A small platform team of three uses Terraform and wants managed services and Cloud Monitoring alerts."
],
[
"sustainab|carbon",
"Prefer low-carbon regions when there is no other constraint, but the UK residency requirement wins."
],
[
"retention|replay|audit|archive",
"Keep raw events 30 days, then delete. Replay must be possible for any event in that window."
],
[
"approve|confirm|proceed|look right|any changes|happy",
"Approved. Please proceed."
]
],
"thresholds": {
"revisions": 2,
"seconds": 3000,
"tokens": 5000000
}
},
{
"id": "case-02-high-scale-ingestion",
"name": "High-scale containerised data ingestion pipeline",
"workflow_request": "Create a high-scale containerised data ingestion pipeline on Google Cloud with least-privilege service identities and automated validation of incoming records.",
"required_products": [
"Pub/Sub",
"Cloud Storage",
"BigQuery"
],
"scripted_answers": [
[
"region|location|geograph",
"us-central1, single region."
],
[
"source|producer|incoming|where.*data|format|schema",
"About 200 IoT gateways send JSON records over HTTPS; each record has a published JSON schema version."
],
[
"availability|uptime|sla|rto|rpo|disaster|recover",
"99.5% availability; RTO 4 hours, RPO 15 minutes. Losing accepted records is not acceptable."
],
[
"traffic|volume|throughput|scale|peak|records per|rps|size",
"2 TB a day, about 20,000 records per second at peak, records under 10 KB."
],
[
"latency|fresh|real.?time|batch",
"Records must be queryable for analytics within 5 minutes of arrival."
],
[
"security|compliance|encrypt|privacy|access|identity|auth|least",
"No regulated data. Each component gets its own service account with least privilege; gateways authenticate with per-device credentials."
],
[
"validat|invalid|reject|quarantine|dead",
"Invalid records go to a quarantine location with the validation error, and are reprocessable after a fix."
],
[
"cost|budget|spend",
"Around 8,000 USD a month; optimise storage costs for data older than 90 days."
],
[
"current|existing|today|on-prem|migrat|legacy",
"Today a single Kafka VM on premises collects records; it is being retired."
],
[
"analytic|query|consum|report|downstream|depend",
"Analysts query the data with SQL; dashboards refresh every 15 minutes."
],
[
"operat|monitor|logging|alert|team|support|container|kubernetes|gke",
"The team runs containers and prefers serverless containers over managing Kubernetes clusters."
],
[
"retention|archive|keep",
"Keep raw records one year and curated tables three years."
],
[
"approve|confirm|proceed|look right|any changes|happy",
"Approved. Please proceed."
]
],
"thresholds": {
"revisions": 3,
"seconds": 3600,
"tokens": 8500000
}
},
{
"id": "case-03-internal-web-app",
"name": "Internal booking web application with a relational database",
"workflow_request": "Design an internal web application on Google Cloud for staff to book meeting rooms and equipment, backed by a managed relational database, available only to employees signed in with the corporate identity provider, with application secrets kept out of code and configuration.",
"required_products": [
"Cloud Run",
"Cloud SQL",
"Secret Manager"
],
"scripted_answers": [
[
"region|location|geograph",
"A single region, europe-west1 (Belgium). All users are in the EU."
],
[
"user|staff|employee|how many|concurren|traffic|volume|scale|peak|requests per|rps",
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second."
],
[
"availability|uptime|sla|rto|rpo|disaster|recover|backup",
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days."
],
[
"latency|response time|performance",
"Pages should load within 1 second at p95."
],
[
"identity|sign.?in|login|sso|auth|idp|access",
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group."
],
[
"security|compliance|encrypt|privacy|gdpr|secret|credential|password",
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days."
],
[
"data|database|schema|relational|postgres|mysql|size|storage",
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history."
],
[
"cost|budget|spend",
"Under 600 EUR a month; the database can be small."
],
[
"current|existing|today|on-prem|migrat|legacy",
"Bookings are kept in a shared spreadsheet today; there is no data to migrate."
],
[
"depend|integrat|calendar|email|notif|downstream|upstream",
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one."
],
[
"operat|monitor|logging|alert|team|support|deploy|ci|pipeline",
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space."
],
[
"network|vpc|private|internet|egress|ingress",
"No public database endpoint; the database must use private IP only."
],
[
"sustainab|carbon",
"No specific requirement."
],
[
"retention|audit|archive|delete",
"Keep booking history 2 years, then delete; keep audit logs 1 year."
],
[
"approve|confirm|proceed|look right|any changes|happy",
"Approved. Please proceed."
]
],
"thresholds": {
"revisions": 3,
"seconds": 3200,
"tokens": 8000000
}
},
{
"id": "case-04-interactive-web-app",
"name": "Internal booking web application with a relational database (interactive)",
"workflow_request": "Design an internal web application on Google Cloud for staff to book meeting rooms and equipment, backed by a managed relational database, available only to employees signed in with the corporate identity provider, with application secrets kept out of code and configuration.",
"required_products": [
"Cloud Run",
"Cloud SQL",
"Secret Manager"
],
"scripted_answers": [
[
"region|location|geograph",
"A single region, europe-west1 (Belgium). All users are in the EU.",
"requirements"
],
[
"user|staff|employee|how many|concurren|traffic|volume|scale|peak|requests per|rps",
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second.",
"requirements"
],
[
"availability|uptime|sla|rto|rpo|disaster|recover|backup",
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days.",
"requirements"
],
[
"latency|response time|performance",
"Pages should load within 1 second at p95.",
"requirements"
],
[
"identity|sign.?in|login|sso|auth|idp|access",
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group.",
"requirements"
],
[
"security|compliance|encrypt|privacy|gdpr|secret|credential|password",
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days.",
"requirements"
],
[
"data|database|schema|relational|postgres|mysql|size|storage",
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history.",
"requirements"
],
[
"cost|budget|spend",
"Under 600 EUR a month; the database can be small.",
"requirements"
],
[
"current|existing|today|on-prem|migrat|legacy",
"Bookings are kept in a shared spreadsheet today; there is no data to migrate.",
"requirements"
],
[
"depend|integrat|calendar|email|notif|downstream|upstream",
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one.",
"requirements"
],
[
"operat|monitor|logging|alert|team|support|deploy|ci|pipeline",
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space.",
"requirements"
],
[
"network|vpc|private|internet|egress|ingress",
"No public database endpoint; the database must use private IP only.",
"requirements"
],
[
"sustainab|carbon",
"No specific requirement.",
"requirements"
],
[
"retention|audit|archive|delete",
"Keep booking history 2 years, then delete; keep audit logs 1 year.",
"requirements"
],
[
"approve|confirm|proceed|look right|any changes|happy",
"Approved. Please proceed."
],
[
".",
"Approved. Please proceed.",
"^(?!requirements$)"
]
],
"interactive": true,
"thresholds": {
"revisions": 2,
"seconds": 3400,
"tokens": 7500000
}
},
{
"id": "case-05-change-retention",
"name": "A change to a delivered event-driven service: retention goes from 30 days to 7 years",
"workflow_request": "Design an event-driven regional HTTP application on Google Cloud that accepts incoming webhooks, validates their signatures, durably enqueues payloads for asynchronous processing, and retains raw events for 30 days for replay and audit.",
"change": "One thing has changed since this was delivered. Raw events must now be kept for 7 years instead of 30 days, to meet a new audit obligation, and must not be deletable during that time. Everything else stays as it is.",
"required_products": [
"Cloud Run",
"Pub/Sub",
"Cloud Storage"
],
"scripted_answers": [
[
"region|location|geograph",
"A single region, europe-west2 (London). Data must stay in the UK."
],
[
"availability|uptime|sla|rto|rpo|disaster|recover",
"99.9% monthly availability. RTO 1 hour, RPO 5 minutes for accepted events. No multi-region DR in phase one."
],
[
"traffic|volume|throughput|scale|peak|requests per|rps|payload",
"Average 50 webhooks per second, peaks of 500 per second for 10 minutes. Payloads up to 256 KB."
],
[
"latency|response time|performance",
"Acknowledge each webhook within 300 ms at p95."
],
[
"security|compliance|encrypt|privacy|gdpr|pci|access|identity|auth",
"UK GDPR applies. Google-managed encryption keys are acceptable. Senders sign requests with HMAC-SHA256 shared secrets; only processing services may read events."
],
[
"cost|budget|spend",
"Under 1,500 GBP a month at average load; prefer pay-per-use."
],
[
"current|existing|today|on-prem|migrat|legacy",
"This is a new workload; nothing runs today."
],
[
"depend|integrat|downstream|upstream|consum",
"Processed events go to an existing internal REST API that is reachable over HTTPS; it can be slow or unavailable for minutes."
],
[
"operat|monitor|logging|alert|team|support",
"A small platform team of three uses Terraform and wants managed services and Cloud Monitoring alerts."
],
[
"sustainab|carbon",
"Prefer low-carbon regions when there is no other constraint, but the UK residency requirement wins."
],
[
"retention|replay|audit|archive",
"Keep raw events 30 days, then delete. Replay must be possible for any event in that window."
],
[
"approve|confirm|proceed|look right|any changes|happy",
"Approved. Please proceed."
]
]
}
]