Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
227 lines
13 KiB
JSON
227 lines
13 KiB
JSON
[
|
|
{
|
|
"id": "case-02-design-recommendations.security.0",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Shift-left security: Are automated security scans integrated into the CI/CD pipeline?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.security.1",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Is Cloud Identity used as a centralized identity provider for managing user lifecycles and federation?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.reliability.2",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Game days or chaos engineering practices are regularly held to validate failure recovery.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.reliability.3",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The system architecture incorporates patterns like circuit breakers, retries with exponential backoff, and rate limiting to support graceful degradation.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.operational-excellence.4",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.operational-excellence.5",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Culture of improvement: A documented strategy is in place for regularly reviewing and adapting cloud operations to industry advancements.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.cost-optimization.6",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Network Egress: Data transfer is minimized by keeping traffic regional, using **Cloud CDN**, and leveraging **Standard Network Tier** where appropriate. High-volume on-premises traffic uses **Direct Peering** or **Cloud Interconnect** (always verify if hybrid connectivity is involved).",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.cost-optimization.7",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.performance-optimization.8",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Automated dashboards and alerts are configured for key performance indicators (KPIs).",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.performance-optimization.9",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: The architecture leverages serverless or managed services to dynamically match capacity to load.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.sustainability.10",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.sustainability.11",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Continuously measure and improve: Applications are instrumented to measure the specific carbon intensity of software features, or automated exports of Carbon Footprint data to BigQuery are configured for deep analysis.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.quality.specific",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.quality.actionable",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.quality.trade_offs",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-02-design-recommendations.quality.traceable",
|
|
"document": "case-02-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.security.16",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Are private connectivity methods (Cloud Interconnect, VPN) used for internal traffic?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.security.17",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Security by design: Is defense-in-depth implemented at the network, host, and application layers?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.reliability.18",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The architecture avoids single points of failure through cross-zone or cross-region redundancy.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.reliability.19",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Application and infrastructure health checks are configured to trigger automated failovers.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.operational-excellence.20",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Resource optimization: Resource utilization is regularly reviewed using recommendations from Active Assist or performance data.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.operational-excellence.21",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.cost-optimization.22",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Rightsizing: Resources are regularly adjusted based on rightsizing suggestions provided by Active Assist Recommender.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.cost-optimization.23",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.performance-optimization.24",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: Auto-scaling rules are configured and validated to handle variable demand.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.performance-optimization.25",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Distributed tracing and profiling tools are used to identify code-level bottlenecks.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.sustainability.26",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.sustainability.27",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "yes_no",
|
|
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Promote a culture of sustainability: Abstract carbon metrics are transformed into tangible progress indicators in annual reports, or sustainability is treated as a first-class technical requirement (NFR) tied to KPIs and performance reviews.",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.quality.specific",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.quality.actionable",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.quality.trade_offs",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
|
|
"label": null
|
|
},
|
|
{
|
|
"id": "case-03-design-recommendations.quality.traceable",
|
|
"document": "case-03-design-recommendations.md",
|
|
"kind": "graded",
|
|
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
|
|
"label": null
|
|
}
|
|
]
|