docs: publish the runs of 2026-10-21
The evaluation (both cases three times: report, results, spread, each run's Deliverables and lesson candidates) and the hand-off through the Orchestration engine (report, the document of links, both agents' kept files and records). Benchmark cases only, all fictional; no machine paths, and the links carry no signatures. Other runs stay ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"execution_id": "case-01-landing-zone-shared-r1",
|
||||
"customer_case": "bench-landing-zone-shared",
|
||||
"status": "completed",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-4 failed: The finding AF-1 does not stand because the operating model could provide the configuration of alerts itself as a process or using a tool.; OC-5 failed: The finding AF-2 does not stand because the operating model could provide a backup and recovery mechanism itself as a process or a tool.; OC-1 failed: Capability C-10 operates only during business hours, which is insufficient to ensure 99.9% availability as it requires 24x7 cover.; OC-8 failed: Capability C-15 covers measuring SLA achievement but does not cover operational costs, which is handled by C-14.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-4",
|
||||
"evidence": "The finding AF-1 does not stand because the operating model could provide the configuration of alerts itself as a process or using a tool."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-5",
|
||||
"evidence": "The finding AF-2 does not stand because the operating model could provide a backup and recovery mechanism itself as a process or a tool."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-1",
|
||||
"evidence": "Capability C-10 operates only during business hours, which is insufficient to ensure 99.9% availability as it requires 24x7 cover."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-8",
|
||||
"evidence": "Capability C-15 covers measuring SLA achievement but does not cover operational costs, which is handled by C-14."
|
||||
},
|
||||
{
|
||||
"type": "unverifiable_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-2",
|
||||
"evidence": "The condition that audit logs cannot be deleted is a technical constraint that cannot be verified by an operating model capability."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"execution_id": "case-01-landing-zone-shared-r2",
|
||||
"customer_case": "bench-landing-zone-shared",
|
||||
"status": "completed",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-1 failed: Finding AF-1 does not stand. The operating model should provide alert configurations itself using the existing native Google Cloud monitoring tools, rather than raising a finding against the solution.; OC-3 failed: Finding AF-1 does not stand. The operating model should provide alert configurations itself using the existing native Google Cloud monitoring tools, rather than raising a finding against the solution.; SC-budget failed: A month's cost of operating is compared with the budget of 60,000 GBP. The cost is 64,125 GBP, over the budget.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-1",
|
||||
"evidence": "Finding AF-1 does not stand. The operating model should provide alert configurations itself using the existing native Google Cloud monitoring tools, rather than raising a finding against the solution."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-3",
|
||||
"evidence": "Finding AF-1 does not stand. The operating model should provide alert configurations itself using the existing native Google Cloud monitoring tools, rather than raising a finding against the solution."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-budget",
|
||||
"evidence": "A month's cost of operating is compared with the budget of 60,000 GBP. The cost is 64,125 GBP, over the budget."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"execution_id": "case-01-landing-zone-shared-r3",
|
||||
"customer_case": "bench-landing-zone-shared",
|
||||
"status": "completed",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-1 failed: The finding AF-1 does not stand because the operating model can provide alert configuration itself via C-2 (monitoring-and-alerting) using the Cloud provider's native operations tooling. The operating model should configure the alerts rather than expecting the solution to specify them.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-1",
|
||||
"evidence": "The finding AF-1 does not stand because the operating model can provide alert configuration itself via C-2 (monitoring-and-alerting) using the Cloud provider's native operations tooling. The operating model should configure the alerts rather than expecting the solution to specify them."
|
||||
},
|
||||
{
|
||||
"type": "proposed_addition",
|
||||
"attempt": 1,
|
||||
"reason": "Plausibility of effort for continuous 24x7 analysis.",
|
||||
"applies_to": "C-13",
|
||||
"statement": "C-13 (security-monitoring-and-response) has an effort of 40 hours per month for continuous monitoring of security events and continuous analysis of audit logs. If the analysis is manual, 40 hours per month is insufficient for continuous 24x7 coverage. Consider clarifying if this relies on automated SIEM alerting or increasing the effort."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
{
|
||||
"execution_id": "case-02-from-architecture-r1",
|
||||
"customer_case": "case-08-landing-zone",
|
||||
"status": "completed_with_warnings",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-2 failed: The check requires 'immediate failover traffic reception', but Capability C-9 is scoped for 'governing the 4-hour disaster recovery automation logic', meaning it cannot provide immediate failover as designed.; OC-12 failed: The check requires roles to be 'continuously rightsized', but Capability C-11 is designed to execute 'periodic least-privilege adjustments', which cannot meet the continuous requirement.; SC-change-S-2 failed: Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. No Capability for making changes covers it with a responsible Role.; SC-change-S-3 failed: Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. No Capability for making changes covers it with a responsible Role.; SC-change-S-6 failed: Federated Identity and Role Governance Authority (S-6) must be changed. No Capability for making changes covers it with a responsible Role.; SC-change-S-8 failed: Platform Telemetry and Operational Alerting Engine (S-8) must be changed. No Capability for making changes covers it with a responsible Role.; SC-change-S-9 failed: Financial Governance and Budget Enforcement Monitor (S-9) must be changed. No Capability for making changes covers it with a responsible Role.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "capability_model",
|
||||
"reason": "Check OC-2 failed in two validations in a row and may not be implementable as written (The check requires maintaining an active warm standby ready to receive failover traffic, but Capability C-9 is scoped for 'governing the 4-hour disaster recovery automation logic', meaning it cannot provide immediate failover as designed. Capability C-9 should be revised to govern immediate failover traffic reception.). Rewrite it so it can be met, or remove it with a reason. Check SC-change-S-2 failed in two validations in a row and may not be implementable as written (Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. No Capability for making changes covers it with a responsible Role.). Rewrite it so it can be met, or remove it with a reason. Check SC-change-S-3 failed in two validations in a row and may not be implementable as written (Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. No Capability for making changes covers it with a responsible Role.). Rewrite it so it can be met, or remove it with a reason. Check SC-change-S-6 failed in two validations in a row and may not be implementable as written (Federated Identity and Role Governance Authority (S-6) must be changed. No Capability for making changes covers it with a responsible Role.). Rewrite it so it can be met, or remove it with a reason. Check SC-change-S-8 failed in two validations in a row and may not be implementable as written (Platform Telemetry and Operational Alerting Engine (S-8) must be changed. No Capability for making changes covers it with a responsible Role.). Rewrite it so it can be met, or remove it with a reason. Check SC-change-S-9 failed in two validations in a row and may not be implementable as written (Financial Governance and Budget Enforcement Monitor (S-9) must be changed. No Capability for making changes covers it with a responsible Role.). Rewrite it so it can be met, or remove it with a reason.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-11 failed: Capability C-7 focuses on configuration state and data residency alignment, but does not cover optimizing carbon-free energy utilization based on regional scores. The Check should be mapped to a deployment capability like C-5 or C-6, or C-7 should be updated.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "capability_model",
|
||||
"reason": "Check OC-11 failed in two validations in a row and may not be implementable as written (The check requires automated deployments to optimize carbon-free energy utilization, and is mapped to C-6. However, C-6's basis only covers managing automated deployment pipelines to deliver foundational updates rapidly and reliably, without mentioning carbon-free energy optimization. C-6 should be updated to include this optimization.). Rewrite it so it can be met, or remove it with a reason.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-3 failed: Capability C-7 focuses on tracking infrastructure state files and ensuring PCI-DSS v4.0 baseline alignment, but does not mention auditing for UK/EU data residency. Data residency is instead covered by C-21. Capability C-7 should be updated to include auditing data residency state, or the Check should be re-mapped to C-21.; OC-18 failed: Capability C-1 covers triage and managing priority incidents, but does not explicitly mention conducting a written review for every Priority 1 incident. The capability basis should be updated to include post-incident written reviews.; OC-21 failed: Capability C-4 covers fulfilling non-standard operational requests, but does not explicitly state that each kind of foundation service request has a stated way to ask and a stated time to fulfil (e.g., maintaining a service request catalogue). The capability basis should be updated to include maintaining stated request methods and fulfilment times.; OC-23 failed: Capability C-6 covers resolving deployment failures and managing automated pipelines, but does not explicitly guarantee a rollback capability within the 4-hour RTO. The capability should be updated to cover rapid rollback procedures.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision_limit",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-2 failed: The check requires the secondary region to be 'continuously maintained' as an active warm standby, but Capability C-9 is configured for 'business' hours, meaning maintenance cannot be continuous (24x7). C-9 should be 24x7 or another capability should provide continuous maintenance.; OC-4 failed: The check requires validation within 10 weeks, but Capability C-16 covers managing the roadmap without explicitly mentioning the 10-week requirement (which is instead mentioned in C-18). C-16 must mention the 10-week validation or the check mapped to C-18.; OC-9 failed: The check requires that expenditure does not exceed target allocations without 'documented operational sign-off', but Capability C-14 only mentions 'escalating anomalies' and does not explicitly guarantee documented sign-off.; OC-12 failed: The check requires identity permissions to be 'continuously reviewed', but Capability C-11 operates only during 'business' hours, meaning reviews cannot be continuous. C-11 must be 24x7.; OC-17 failed: The check requires verifiable alignment with DORA, PCI-DSS v4.0, and ISO 27001. Capability C-7 only ensures alignment with PCI-DSS v4.0 baseline configurations, completely missing DORA and ISO 27001 controls."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-2",
|
||||
"evidence": "The check requires 'immediate failover traffic reception', but Capability C-9 is scoped for 'governing the 4-hour disaster recovery automation logic', meaning it cannot provide immediate failover as designed."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-12",
|
||||
"evidence": "The check requires roles to be 'continuously rightsized', but Capability C-11 is designed to execute 'periodic least-privilege adjustments', which cannot meet the continuous requirement."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-change-S-2",
|
||||
"evidence": "Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-change-S-3",
|
||||
"evidence": "Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-change-S-6",
|
||||
"evidence": "Federated Identity and Role Governance Authority (S-6) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-change-S-8",
|
||||
"evidence": "Platform Telemetry and Operational Alerting Engine (S-8) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-change-S-9",
|
||||
"evidence": "Financial Governance and Budget Enforcement Monitor (S-9) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "proposed_addition",
|
||||
"attempt": 1,
|
||||
"applies_to": "C-13",
|
||||
"reason": "The stated effort of 30 hours per month is implausible for 'Continuously analyzing egress inspection logs' on a 24x7 basis. Continuous manual analysis would require full-time dedicated shift coverage far exceeding 30 hours a month.",
|
||||
"statement": "Revise the effort for C-13 to reflect the true cost of continuous manual log analysis, or change the basis to rely on automated anomaly alert response rather than continuous manual analysis."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "OC-2",
|
||||
"evidence": "The check requires maintaining an active warm standby ready to receive failover traffic, but Capability C-9 is scoped for 'governing the 4-hour disaster recovery automation logic', meaning it cannot provide immediate failover as designed. Capability C-9 should be revised to govern immediate failover traffic reception."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "SC-change-S-2",
|
||||
"evidence": "Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "SC-change-S-3",
|
||||
"evidence": "Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "SC-change-S-6",
|
||||
"evidence": "Federated Identity and Role Governance Authority (S-6) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "SC-change-S-8",
|
||||
"evidence": "Platform Telemetry and Operational Alerting Engine (S-8) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "SC-change-S-9",
|
||||
"evidence": "Financial Governance and Budget Enforcement Monitor (S-9) must be changed. No Capability for making changes covers it with a responsible Role."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 3,
|
||||
"check": "OC-11",
|
||||
"evidence": "Capability C-7 focuses on configuration state and data residency alignment, but does not cover optimizing carbon-free energy utilization based on regional scores. The Check should be mapped to a deployment capability like C-5 or C-6, or C-7 should be updated."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 4,
|
||||
"check": "OC-4",
|
||||
"evidence": "The check requires handover for workload ingestion within 10 weeks of project start, and is mapped to Capability C-16. However, C-16's basis focuses on roadmap and backlog management and does not mention handover or the 10-week timeline (though C-18 does). The check fails because C-16 does not cover the condition as designed. C-16 should be updated or the check should be mapped to C-18."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 4,
|
||||
"check": "OC-11",
|
||||
"evidence": "The check requires automated deployments to optimize carbon-free energy utilization, and is mapped to C-6. However, C-6's basis only covers managing automated deployment pipelines to deliver foundational updates rapidly and reliably, without mentioning carbon-free energy optimization. C-6 should be updated to include this optimization."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 4,
|
||||
"check": "OC-15",
|
||||
"evidence": "The check requires operations to be delivered entirely under a provider-operated model, and is mapped to C-20 (holding the provider accountable). However, the model assigns C-20 to Provider roles (R-3, R-6) instead of Customer roles. A provider cannot hold itself accountable as the external MSP. Supplier management should be assigned to Customer roles (P-2) to hold the provider accountable."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 4,
|
||||
"check": "OC-17",
|
||||
"evidence": "The check requires Cloud Monitoring alerts to be fully integrated into the provider's event management system, and is mapped to C-2. However, C-2's basis only covers continuous monitoring and alert generation; it does not explicitly mention integrating those alerts into the provider's operational event management system. C-2 should be updated to include this integration."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 5,
|
||||
"check": "OC-3",
|
||||
"evidence": "Capability C-7 focuses on tracking infrastructure state files and ensuring PCI-DSS v4.0 baseline alignment, but does not mention auditing for UK/EU data residency. Data residency is instead covered by C-21. Capability C-7 should be updated to include auditing data residency state, or the Check should be re-mapped to C-21."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 5,
|
||||
"check": "OC-18",
|
||||
"evidence": "Capability C-1 covers triage and managing priority incidents, but does not explicitly mention conducting a written review for every Priority 1 incident. The capability basis should be updated to include post-incident written reviews."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 5,
|
||||
"check": "OC-21",
|
||||
"evidence": "Capability C-4 covers fulfilling non-standard operational requests, but does not explicitly state that each kind of foundation service request has a stated way to ask and a stated time to fulfil (e.g., maintaining a service request catalogue). The capability basis should be updated to include maintaining stated request methods and fulfilment times."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 5,
|
||||
"check": "OC-23",
|
||||
"evidence": "Capability C-6 covers resolving deployment failures and managing automated pipelines, but does not explicitly guarantee a rollback capability within the 4-hour RTO. The capability should be updated to cover rapid rollback procedures."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 6,
|
||||
"check": "OC-2",
|
||||
"evidence": "The check requires the secondary region to be 'continuously maintained' as an active warm standby, but Capability C-9 is configured for 'business' hours, meaning maintenance cannot be continuous (24x7). C-9 should be 24x7 or another capability should provide continuous maintenance."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 6,
|
||||
"check": "OC-4",
|
||||
"evidence": "The check requires validation within 10 weeks, but Capability C-16 covers managing the roadmap without explicitly mentioning the 10-week requirement (which is instead mentioned in C-18). C-16 must mention the 10-week validation or the check mapped to C-18."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 6,
|
||||
"check": "OC-9",
|
||||
"evidence": "The check requires that expenditure does not exceed target allocations without 'documented operational sign-off', but Capability C-14 only mentions 'escalating anomalies' and does not explicitly guarantee documented sign-off."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 6,
|
||||
"check": "OC-12",
|
||||
"evidence": "The check requires identity permissions to be 'continuously reviewed', but Capability C-11 operates only during 'business' hours, meaning reviews cannot be continuous. C-11 must be 24x7."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 6,
|
||||
"check": "OC-17",
|
||||
"evidence": "The check requires verifiable alignment with DORA, PCI-DSS v4.0, and ISO 27001. Capability C-7 only ensures alignment with PCI-DSS v4.0 baseline configurations, completely missing DORA and ISO 27001 controls."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"execution_id": "case-02-from-architecture-r2",
|
||||
"customer_case": "case-08-landing-zone",
|
||||
"status": "completed",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-K6 failed: The Check requires achieving an RTO of no more than 4 hours, meaning disaster recovery might need to be activated at any time. However, Capability C-9 (service-continuity), which is the capability designated to achieve this RTO, is configured for 'business' hours only. It cannot reliably meet a 4-hour RTO outside of business hours as currently designed. The capability should be expanded to 24x7 coverage, or execution should explicitly fall to a 24x7 capability.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-K6",
|
||||
"evidence": "The Check requires achieving an RTO of no more than 4 hours, meaning disaster recovery might need to be activated at any time. However, Capability C-9 (service-continuity), which is the capability designated to achieve this RTO, is configured for 'business' hours only. It cannot reliably meet a 4-hour RTO outside of business hours as currently designed. The capability should be expanded to 24x7 coverage, or execution should explicitly fall to a 24x7 capability."
|
||||
},
|
||||
{
|
||||
"type": "refusal",
|
||||
"role": "operating-model-architect",
|
||||
"deliverable": "target_operating_model",
|
||||
"tool": "submit_deliverable",
|
||||
"message": "Not submitted. The Deliverable is missing: target-operating-model.json: business outcome BO-1 (Continuous, highly available, and low-latency operation of the core payments platform foundation) is shown by no Measure; target-operating-model.json: business outcome BO-2 (Continuous compliance with DORA and PCI-DSS v4.0 standards using customer-managed encryption keys and centralised egress inspection) is shown by no Measure; target-operating-model.json: business outcome BO-3 (Automated, secure infr"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
{
|
||||
"execution_id": "case-02-from-architecture-r3",
|
||||
"customer_case": "case-08-landing-zone",
|
||||
"status": "completed",
|
||||
"role_skills": {
|
||||
"capability-modeller": [
|
||||
"operating-capabilities",
|
||||
"operating-chain"
|
||||
],
|
||||
"operating-model-architect": [
|
||||
"operating-capabilities",
|
||||
"operating-chain",
|
||||
"rate-card"
|
||||
],
|
||||
"operations-analyst": [
|
||||
"operating-scoping",
|
||||
"operating-chain"
|
||||
],
|
||||
"orchestrator": [],
|
||||
"packager": [
|
||||
"operating-guide"
|
||||
],
|
||||
"reply-writer": [],
|
||||
"scope-analyst": [
|
||||
"operating-scoping"
|
||||
],
|
||||
"validator": [
|
||||
"operating-chain"
|
||||
]
|
||||
},
|
||||
"skill_versions": {
|
||||
"operating-capabilities": "49a7e66857a1",
|
||||
"operating-chain": "ec53ad8875e6",
|
||||
"operating-guide": "85b48d5998da",
|
||||
"operating-scoping": "08da33f8cddf",
|
||||
"rate-card": "f86d17a9bcce"
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "target_operating_model",
|
||||
"reason": "OC-9 failed: Capability C-14 is designed only for 'monthly review of cloud consumption against allocated budgets and investigating anomalies.' It lacks any designed process, automation, or mechanism to enforce hard limits that prevent overages. The capability must be redesigned to provide hard limit enforcement.; OC-16 failed: Capability C-21's basis only includes gathering compliance evidence for PCI-DSS v4.0 and DORA auditors, explicitly omitting ISO27001 which is required by the condition. As designed, the model does not cover ISO27001 evidence gathering.; SC-party-P-2 failed: Platform Service Provider cannot be reached. No Role in another Party is responsible or consulted for C-2, C-3, C-4, C-5, C-6, C-7, C-8, C-9, C-10, C-11, C-12, C-13, C-14, C-15, C-17, C-18, C-19, C-20.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "revision",
|
||||
"deliverable": "capability_model",
|
||||
"reason": "Check OC-9 failed in two validations in a row and may not be implementable as written (Capability C-14 is currently designed only for monthly review of cloud consumption and lacks any process or automation to enforce hard limits. The capability must be redesigned to provide hard limit enforcement; the finding AF-1 does not stand as the operating model could provide this mechanism itself.). Rewrite it so it can be met, or remove it with a reason. Check OC-16 failed in two validations in a row and may not be implementable as written (Capability C-21's basis only includes gathering compliance evidence for PCI-DSS v4.0 and DORA auditors, explicitly omitting ISO27001. The finding AF-2 does not stand as the operating model should provide the process and effort to gather this evidence itself.). Rewrite it so it can be met, or remove it with a reason.",
|
||||
"requested_by": "orchestrator"
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-9",
|
||||
"evidence": "Capability C-14 is designed only for 'monthly review of cloud consumption against allocated budgets and investigating anomalies.' It lacks any designed process, automation, or mechanism to enforce hard limits that prevent overages. The capability must be redesigned to provide hard limit enforcement."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-16",
|
||||
"evidence": "Capability C-21's basis only includes gathering compliance evidence for PCI-DSS v4.0 and DORA auditors, explicitly omitting ISO27001 which is required by the condition. As designed, the model does not cover ISO27001 evidence gathering."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 1,
|
||||
"check": "SC-party-P-2",
|
||||
"evidence": "Platform Service Provider cannot be reached. No Role in another Party is responsible or consulted for C-2, C-3, C-4, C-5, C-6, C-7, C-8, C-9, C-10, C-11, C-12, C-13, C-14, C-15, C-17, C-18, C-19, C-20."
|
||||
},
|
||||
{
|
||||
"type": "unverifiable_check",
|
||||
"attempt": 1,
|
||||
"check": "OC-4",
|
||||
"evidence": "The condition specifies a one-time engagement milestone (verification of full provisioning within 10 weeks) which occurs during the build phase. An ongoing operating model governs steady-state operations and cannot continuously verify or guarantee a past delivery event, making this condition not verifiable as worded."
|
||||
},
|
||||
{
|
||||
"type": "proposed_addition",
|
||||
"attempt": 1,
|
||||
"reason": "The stated effort of 80 hours per month is implausible for 'continuous watching' which would require 720+ hours for true 24x7 manual monitoring.",
|
||||
"applies_to": "C-2",
|
||||
"statement": "The basis and effort of C-2 should be aligned to reflect automated alerting, or the effort vastly increased for manual continuous watching."
|
||||
},
|
||||
{
|
||||
"type": "proposed_addition",
|
||||
"attempt": 1,
|
||||
"applies_to": "C-13",
|
||||
"reason": "80 hours per month is insufficient for manual 24x7 security event monitoring. If it relies on automated SIEM alerts, the basis should clarify this.",
|
||||
"statement": "The effort for C-13 should clarify reliance on automated alerts or vastly increase if manual eyes-on-glass monitoring is expected 24x7."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "OC-1",
|
||||
"evidence": "Capability C-7's basis explicitly mentions ensuring compliance with DORA/PCI-DSS baselines but omits ISO27001, meaning the model as designed does not cover enforcing ISO27001 standard baselines."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "OC-9",
|
||||
"evidence": "Capability C-14 is currently designed only for monthly review of cloud consumption and lacks any process or automation to enforce hard limits. The capability must be redesigned to provide hard limit enforcement; the finding AF-1 does not stand as the operating model could provide this mechanism itself."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "OC-16",
|
||||
"evidence": "Capability C-21's basis only includes gathering compliance evidence for PCI-DSS v4.0 and DORA auditors, explicitly omitting ISO27001. The finding AF-2 does not stand as the operating model should provide the process and effort to gather this evidence itself."
|
||||
},
|
||||
{
|
||||
"type": "failed_check",
|
||||
"attempt": 2,
|
||||
"check": "OC-17",
|
||||
"evidence": "Capability C-21's basis specifies gathering evidence for PCI-DSS v4.0 and DORA auditors, omitting the process for gathering evidence of UK and EU data residency guarantees."
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user