Files
solution_architecture_agent/eval/datasets/judge/labels.json
Jonathan Boniface c84dbdea20 feat: the solution architecture agent on the shared Execution engine
The reference agent's subject, cloud solution architecture, as an agent in
the operating model agent's pattern: the Execution engine (from
orchestration_engine) and one Workflow in app/solution_architecture,
built for the settings and provider bundle by assembly.workflow_for.

Copied from gcp_solution_architecture_agent at 8b397cd without its own
engine: its role files, Skills, provider bundle, ADRs, vocabulary, scripts
and evaluation datasets. The image takes the engine from a named build
context and installs Terraform. It is called through the Orchestration
engine, and its files are what the operating model agent digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:49:28 +01:00

227 lines
13 KiB
JSON

[
{
"id": "case-02-design-recommendations.security.0",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Shift-left security: Are automated security scans integrated into the CI/CD pipeline?",
"label": null
},
{
"id": "case-02-design-recommendations.security.1",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Is Cloud Identity used as a centralized identity provider for managing user lifecycles and federation?",
"label": null
},
{
"id": "case-02-design-recommendations.reliability.2",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Game days or chaos engineering practices are regularly held to validate failure recovery.",
"label": null
},
{
"id": "case-02-design-recommendations.reliability.3",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The system architecture incorporates patterns like circuit breakers, retries with exponential backoff, and rate limiting to support graceful degradation.",
"label": null
},
{
"id": "case-02-design-recommendations.operational-excellence.4",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
"label": null
},
{
"id": "case-02-design-recommendations.operational-excellence.5",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Culture of improvement: A documented strategy is in place for regularly reviewing and adapting cloud operations to industry advancements.",
"label": null
},
{
"id": "case-02-design-recommendations.cost-optimization.6",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Network Egress: Data transfer is minimized by keeping traffic regional, using **Cloud CDN**, and leveraging **Standard Network Tier** where appropriate. High-volume on-premises traffic uses **Direct Peering** or **Cloud Interconnect** (always verify if hybrid connectivity is involved).",
"label": null
},
{
"id": "case-02-design-recommendations.cost-optimization.7",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
"label": null
},
{
"id": "case-02-design-recommendations.performance-optimization.8",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Automated dashboards and alerts are configured for key performance indicators (KPIs).",
"label": null
},
{
"id": "case-02-design-recommendations.performance-optimization.9",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: The architecture leverages serverless or managed services to dynamically match capacity to load.",
"label": null
},
{
"id": "case-02-design-recommendations.sustainability.10",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
"label": null
},
{
"id": "case-02-design-recommendations.sustainability.11",
"document": "case-02-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Continuously measure and improve: Applications are instrumented to measure the specific carbon intensity of software features, or automated exports of Carbon Footprint data to BigQuery are configured for deep analysis.",
"label": null
},
{
"id": "case-02-design-recommendations.quality.specific",
"document": "case-02-design-recommendations.md",
"kind": "graded",
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
"label": null
},
{
"id": "case-02-design-recommendations.quality.actionable",
"document": "case-02-design-recommendations.md",
"kind": "graded",
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
"label": null
},
{
"id": "case-02-design-recommendations.quality.trade_offs",
"document": "case-02-design-recommendations.md",
"kind": "graded",
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
"label": null
},
{
"id": "case-02-design-recommendations.quality.traceable",
"document": "case-02-design-recommendations.md",
"kind": "graded",
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
"label": null
},
{
"id": "case-03-design-recommendations.security.16",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Are private connectivity methods (Cloud Interconnect, VPN) used for internal traffic?",
"label": null
},
{
"id": "case-03-design-recommendations.security.17",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Security by design: Is defense-in-depth implemented at the network, host, and application layers?",
"label": null
},
{
"id": "case-03-design-recommendations.reliability.18",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The architecture avoids single points of failure through cross-zone or cross-region redundancy.",
"label": null
},
{
"id": "case-03-design-recommendations.reliability.19",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Application and infrastructure health checks are configured to trigger automated failovers.",
"label": null
},
{
"id": "case-03-design-recommendations.operational-excellence.20",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Resource optimization: Resource utilization is regularly reviewed using recommendations from Active Assist or performance data.",
"label": null
},
{
"id": "case-03-design-recommendations.operational-excellence.21",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
"label": null
},
{
"id": "case-03-design-recommendations.cost-optimization.22",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Rightsizing: Resources are regularly adjusted based on rightsizing suggestions provided by Active Assist Recommender.",
"label": null
},
{
"id": "case-03-design-recommendations.cost-optimization.23",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
"label": null
},
{
"id": "case-03-design-recommendations.performance-optimization.24",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: Auto-scaling rules are configured and validated to handle variable demand.",
"label": null
},
{
"id": "case-03-design-recommendations.performance-optimization.25",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Distributed tracing and profiling tools are used to identify code-level bottlenecks.",
"label": null
},
{
"id": "case-03-design-recommendations.sustainability.26",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
"label": null
},
{
"id": "case-03-design-recommendations.sustainability.27",
"document": "case-03-design-recommendations.md",
"kind": "yes_no",
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Promote a culture of sustainability: Abstract carbon metrics are transformed into tangible progress indicators in annual reports, or sustainability is treated as a first-class technical requirement (NFR) tied to KPIs and performance reviews.",
"label": null
},
{
"id": "case-03-design-recommendations.quality.specific",
"document": "case-03-design-recommendations.md",
"kind": "graded",
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
"label": null
},
{
"id": "case-03-design-recommendations.quality.actionable",
"document": "case-03-design-recommendations.md",
"kind": "graded",
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
"label": null
},
{
"id": "case-03-design-recommendations.quality.trade_offs",
"document": "case-03-design-recommendations.md",
"kind": "graded",
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
"label": null
},
{
"id": "case-03-design-recommendations.quality.traceable",
"document": "case-03-design-recommendations.md",
"kind": "graded",
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
"label": null
}
]