feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -57,7 +57,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 2,
|
||||
"seconds": 3000,
|
||||
"tokens": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-02-high-scale-ingestion",
|
||||
@@ -121,7 +126,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 3,
|
||||
"seconds": 3600,
|
||||
"tokens": 8500000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-03-internal-web-app",
|
||||
@@ -193,7 +203,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 3,
|
||||
"seconds": 3200,
|
||||
"tokens": 8000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-04-interactive-web-app",
|
||||
@@ -207,65 +222,89 @@
|
||||
"scripted_answers": [
|
||||
[
|
||||
"region|location|geograph",
|
||||
"A single region, europe-west1 (Belgium). All users are in the EU."
|
||||
"A single region, europe-west1 (Belgium). All users are in the EU.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"user|staff|employee|how many|concurren|traffic|volume|scale|peak|requests per|rps",
|
||||
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second."
|
||||
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"availability|uptime|sla|rto|rpo|disaster|recover|backup",
|
||||
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days."
|
||||
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"latency|response time|performance",
|
||||
"Pages should load within 1 second at p95."
|
||||
"Pages should load within 1 second at p95.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"identity|sign.?in|login|sso|auth|idp|access",
|
||||
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group."
|
||||
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"security|compliance|encrypt|privacy|gdpr|secret|credential|password",
|
||||
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days."
|
||||
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"data|database|schema|relational|postgres|mysql|size|storage",
|
||||
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history."
|
||||
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"cost|budget|spend",
|
||||
"Under 600 EUR a month; the database can be small."
|
||||
"Under 600 EUR a month; the database can be small.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"current|existing|today|on-prem|migrat|legacy",
|
||||
"Bookings are kept in a shared spreadsheet today; there is no data to migrate."
|
||||
"Bookings are kept in a shared spreadsheet today; there is no data to migrate.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"depend|integrat|calendar|email|notif|downstream|upstream",
|
||||
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one."
|
||||
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"operat|monitor|logging|alert|team|support|deploy|ci|pipeline",
|
||||
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space."
|
||||
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"network|vpc|private|internet|egress|ingress",
|
||||
"No public database endpoint; the database must use private IP only."
|
||||
"No public database endpoint; the database must use private IP only.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"sustainab|carbon",
|
||||
"No specific requirement."
|
||||
"No specific requirement.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"retention|audit|archive|delete",
|
||||
"Keep booking history 2 years, then delete; keep audit logs 1 year."
|
||||
"Keep booking history 2 years, then delete; keep audit logs 1 year.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
],
|
||||
[
|
||||
".",
|
||||
"Approved. Please proceed.",
|
||||
"^(?!requirements$)"
|
||||
]
|
||||
],
|
||||
"interactive": true
|
||||
"interactive": true,
|
||||
"thresholds": {
|
||||
"revisions": 2,
|
||||
"seconds": 3400,
|
||||
"tokens": 7500000
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user