From 4c1f1662ded9f87ba497f54e1750330ac01220bc Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Thu, 22 Oct 2026 09:39:02 +0100 Subject: [PATCH] docs: include each run's Deliverables in the published evaluation The general `deliverables/` ignore matched the folder inside each run's Execution too, so the Deliverables as markdown (operating scope, requirements, capability model, operating model, validation, guide) were left out. They are now included for the published run only. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 1 + .../deliverables/capability_model.md | 100 ++++++ .../deliverables/operating_model_guide.md | 229 ++++++++++++++ .../deliverables/operating_scope.md | 39 +++ .../deliverables/operational_requirements.md | 45 +++ .../deliverables/target_operating_model.md | 66 ++++ .../deliverables/validation_results.md | 42 +++ .../deliverables/capability_model.md | 54 ++++ .../deliverables/operating_model_guide.md | 234 ++++++++++++++ .../deliverables/operating_scope.md | 39 +++ .../deliverables/operational_requirements.md | 74 +++++ .../deliverables/target_operating_model.md | 72 +++++ .../deliverables/validation_results.md | 50 +++ .../deliverables/capability_model.md | 109 +++++++ .../deliverables/operating_model_guide.md | 237 ++++++++++++++ .../deliverables/operating_scope.md | 39 +++ .../deliverables/operational_requirements.md | 100 ++++++ .../deliverables/target_operating_model.md | 87 ++++++ .../deliverables/validation_results.md | 47 +++ .../deliverables/capability_model.md | 51 ++++ .../deliverables/operating_scope.md | 42 +++ .../deliverables/operational_requirements.md | 115 +++++++ .../deliverables/target_operating_model.md | 91 ++++++ .../deliverables/validation_results.md | 67 ++++ .../deliverables/capability_model.md | 59 ++++ .../deliverables/operating_model_guide.md | 280 +++++++++++++++++ .../deliverables/operating_scope.md | 42 +++ .../deliverables/operational_requirements.md | 94 ++++++ .../deliverables/target_operating_model.md | 68 +++++ .../deliverables/validation_results.md | 64 ++++ .../deliverables/capability_model.md | 49 +++ .../deliverables/operating_model_guide.md | 288 ++++++++++++++++++ .../deliverables/operating_scope.md | 44 +++ .../deliverables/operational_requirements.md | 87 ++++++ .../deliverables/target_operating_model.md | 83 +++++ .../deliverables/validation_results.md | 66 ++++ 36 files changed, 3254 insertions(+) create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_model_guide.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/validation_results.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_model_guide.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/validation_results.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_model_guide.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/validation_results.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/validation_results.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_model_guide.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/validation_results.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/capability_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_model_guide.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_scope.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operational_requirements.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/target_operating_model.md create mode 100644 eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/validation_results.md diff --git a/.gitignore b/.gitignore index 665d089..ea78f13 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,7 @@ eval/results/* !eval/results/20261021T203624Z/ !eval/results/20261021T203623Z-handoff/ !eval/results/20261021T203624Z/deliverables/ +!eval/results/20261021T203624Z/deliverables/**/deliverables/ # Lesson candidates captured from Executions (may quote user feedback) lessons/candidates/ diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/capability_model.md new file mode 100644 index 0000000..9e0ddd9 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/capability_model.md @@ -0,0 +1,100 @@ +# Capabilities + +The capabilities below define what the operating organisation must be able to do to meet the operational requirements. Note that any cost figures derived from these capabilities will currently use placeholders, as the rate card remains in draft. + +| Capability | Items Covered | Operational Requirements Served | Why it is Needed | +| --- | --- | --- | --- | +| **Incident management** | S-1, S-2, S-3, S-4 | OR-1, OR-2 | To acknowledge and restore service when faults occur across platform components at any hour. | +| **Monitoring and alerting** | S-1, S-2, S-3, S-4 | OR-1 | To provide continuous 24x7 visibility into the state of the landing zone and trigger incident response. | +| **Problem management** | S-1, S-2, S-3, S-4 | OR-2, OR-3 | To analyze root causes of priority 1 incidents and prevent recurrences to maintain 99.9% availability. | +| **Service request fulfilment** | S-1, S-2, S-3, S-4 | OR-8 | To process routine operational tasks and access requests in a shared model. | +| **Change enablement** | S-1, S-2, S-3, S-4 | OR-4, OR-5 | To manage changes and ensure the customer platform team explicitly approves all network modifications. | +| **Release and deployment** | S-3 | OR-4 | To operate the deployment pipeline so that approved changes reach production within a single day. | +| **Configuration management** | S-1, S-2, S-3, S-4 | OR-4, OR-5, OR-7 | To maintain infrastructural consistency and manage the configuration schedule for encryption key rotation. | +| **Backup and recovery** | S-2, S-4 | OR-2, OR-10 | To enforce 7-year audit log retention and ensure recoverability of critical configurations. | +| **Service continuity** | S-1, S-2, S-3, S-4 | OR-3 | To prepare for and test major disruptions and maintain monthly uptime targets. | +| **Capacity and performance management** | S-1 | OR-3 | To track hub network and interconnect throughput, avoiding degradation due to load limits. | +| **Access management** | S-1, S-2, S-3, S-4 | OR-8 | To control who can modify platform items within the shared operational model. | +| **Vulnerability and patch management** | S-1, S-2, S-3, S-4 | OR-3 | To proactively scan for weaknesses and apply security patches to the platform. | +| **Security monitoring and response** | S-1, S-2, S-3, S-4 | OR-10 | To watch for misuse and tampering 24x7, specifically protecting audit log integrity. | +| **Cost management** | S-1, S-2, S-3, S-4 | OR-6 | To compile Google Cloud landing zone run costs for the monthly operational report. | +| **Service level management** | S-1, S-2, S-3, S-4 | OR-6 | To track SLA achievements and compile the monthly service level report. | +| **Platform product management** | S-1, S-2, S-3, S-4 | OR-8 | To manage the foundation as a product, prioritizing features by what slows down development teams. | +| **Self-service and golden paths** | S-1, S-3 | OR-4, OR-8 | To offer automated, friction-free defaults for pipeline and network modifications. | +| **Enablement** | S-1, S-2, S-3, S-4 | OR-8 | To assist and pair with developers adopting the platform in a shared service model. | +| **Delivery and developer experience measurement** | S-3 | OR-4 | To measure delivery speed, lead time, and friction to ensure changes deploy quickly and safely. | +| **Supplier management** | S-1, S-2, S-3, S-4 | OR-8, OR-9 | To monitor external service providers and guarantee operational delivery only from the UK, Poland, and India. | +| **Regulatory evidence and reporting** | S-2 | OR-10 | To retain and produce logs as evidence specified by EU DORA. | + +# Sources + +The capabilities chosen above draw on the following industry frameworks. *Please note that these references are unverified.* The operating model does not claim or certify compliance with any framework. + +* **ITIL 4** (PeopleCert) + * Incident management practice + * Monitoring and event management practice + * Problem management practice + * Service request management practice + * Change enablement practice + * Release management and Deployment management practices + * Service configuration management practice + * Service continuity management practice + * Capacity and performance management practice + * Service level management practice + * Supplier management practice +* **Site Reliability Engineering: How Google Runs Production Systems** (Google / O'Reilly) + * Managing Incidents + * Monitoring Distributed Systems + * Postmortem Culture: Learning from Failure + * Release Engineering + * Service Level Objectives +* **ISO/IEC 20000-1** + * Incident management + * Service request management + * Change management + * Service continuity management + * Capacity management + * Service level management + * Supplier management +* **Regulation (EU) 2022/2554, the Digital Operational Resilience Act** (European Union) + * Article 9, Protection and prevention + * Article 10, Detection + * Article 11, Response and recovery + * Article 12, Backup policies and procedures, restoration and recovery procedures and methods + * Article 17, ICT-related incident management process + * Article 19, Reporting of major ICT-related incidents + * Article 28, General principles on ICT third-party risk +* **NIST Cybersecurity Framework 2.0** + * Detect: Continuous Monitoring + * Identify: Asset Management + * Recover: Incident Recovery Plan Execution + * Protect: Identity Management, Authentication, and Access Control + * Protect: Platform Security + * Detect: Adverse Event Analysis; Respond: Incident Management +* **ISO/IEC 27001** + * Access control + * Management of technical vulnerabilities + * Information security incident management + * Compliance with legal and contractual requirements +* **DevOps Research and Assessment (DORA) software delivery metrics** + * Change failure rate + * Deployment frequency and lead time for changes + * Time to restore service +* **FinOps Framework** (FinOps Foundation) + * Allocation; Budgeting; Forecasting; Anomaly Management +* **Team Topologies: Organizing Business and Technology Teams for Fast Flow** (IT Revolution) + * Platform team; Thinnest viable platform + * X-as-a-Service interaction mode + * Enabling team; Facilitating interaction mode +* **CNCF Platforms White Paper (platform as a product)** (Cloud Native Computing Foundation) + * Platform as a product; Measuring platform success + * Platform capabilities; Self-service +* **The SPACE of Developer Productivity** + * Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +* **DevEx: What Actually Drives Productivity** + * Feedback loops; Cognitive load; Flow state + +# Assumptions + +- **Service Management Tooling**: We assume the use of native Google Cloud operations tooling without an external ITSM integration, as no separate ITSM system was mandated. +- **Change Process**: We assume that standard infrastructure changes proceed via an automated deployment pipeline with peer reviews, while only network changes demand explicit manual approval from the customer platform team. \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_model_guide.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_model_guide.md new file mode 100644 index 0000000..a43dc1f --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_model_guide.md @@ -0,0 +1,229 @@ +# Operating model guide: Google Cloud Landing Zone + +## Summary + +This guide sets out the target operating model for the Google Cloud landing zone platform foundation, which comprises the hub network and interconnect, the audit log store, the deployment pipeline, and encryption keys. The platform is operated under a shared service model between the customer and an external service provider. The service provider is responsible for incident response, continuous monitoring, and day-to-day administrative maintenance during 24x7 support hours, while the customer platform team operates during business hours, retaining architectural leadership, platform enablement, and final approval authority over all network changes. + +Because all operational checks and scenario walk-throughs pass with no open findings and run costs remain within budget, the reader should look first at the assumptions and open questions, specifically the exclusion of hosted application workloads from platform foundation operations and the confirmation of the go-live date. + +### Key figures + +- Operated: 4 Items, with 3 Conditions to keep true. +- Faults responded to: 24x7 hours. +- Capabilities: 21. +- People: 6.8. Cost a month: 46,252 GBP, from rate card version `draft-2026-10-13` (placeholder prices: an illustration). +- Scenario walk-throughs: 13 of 13 pass. +- Business outcomes: 3, 3 shown by a Measure. +- Findings for whoever owns the design: 0. + +## Business outcomes + +**BO-1: New payment products reach customers in weeks rather than months.** Target: A change reaches production within a day of approval.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Lead time for changes | outcome | dora-metrics | Within a day of approval | Delivery and developer experience measurement | Customer Platform Lead | + +**BO-2: Payments are never unavailable for long.** Target: Restored within 4 hours.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Time to restore service | outcome | dora-metrics | Restored within 4 hours | Delivery and developer experience measurement | Customer Platform Lead | + +**BO-3: We keep the platform engineers we hire.** + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Developer satisfaction | guardrail | space-framework | | Delivery and developer experience measurement | Customer Platform Lead | +| Developer cognitive load | leading | devex-framework | | Delivery and developer experience measurement | Customer Platform Lead | + +## What is operated + +| Item | What it is | Kind | Product | From | +| --- | --- | --- | --- | --- | +| S-1 | Hub network and interconnect | network | Cloud Interconnect | | +| S-2 | Audit log store | storage | Cloud Storage | | +| S-3 | Deployment pipeline | pipeline | Cloud Build | | +| S-4 | Encryption keys | keys | Cloud KMS | | + +What must be kept true while it is run: + +| Condition | On | What must hold | From | +| --- | --- | --- | --- | +| K-1 | S-1 | The platform is available at least 99.9% of each month. | | +| K-2 | S-2 | Audit logs cannot be deleted for seven years. | | +| K-3 | S-4 | Encryption keys are rotated every 90 days. | | + +## Operational requirements + +| Requirement | Kind | What must be achieved | From | +| --- | --- | --- | --- | +| OR-1 | service_level | A priority 1 fault on any Item is acknowledged within 15 minutes, at any hour. | service_levels.priority_1_response_minutes, operations.support_hours | +| OR-2 | service_level | Service is restored within 4 hours following a priority 1 incident. | service_levels.recovery_time_hours, BO-2 | +| OR-3 | service_level | The hub network and interconnect maintains at least 99.9% availability during each month. | K-1 | +| OR-4 | service_level | An approved change reaches production within one day of approval. | BO-1 | +| OR-5 | process | Every change to the network requires approval from the customer platform team prior to implementation. | REQ-01 | +| OR-6 | process | A report detailing service levels and operational costs is delivered to the customer monthly. | REQ-02 | +| OR-7 | process | Encryption keys are rotated at least once every 90 days. | K-3 | +| OR-8 | organisation | Platform operations is conducted under a shared service model between the customer and an external service provider. | sourcing.model | +| OR-9 | organisation | Operational delivery is conducted exclusively from the United Kingdom, Poland, and India. | sourcing.delivery_countries | +| OR-10 | compliance | Audit logs cannot be deleted for seven years. | K-2 | + +## Capabilities + +| Capability | What it is | Serves | Covers | Hours | Effort a month | +| --- | --- | --- | --- | --- | --- | +| C-1 | Incident management | OR-1, OR-2 | S-1, S-2, S-3, S-4 | 24x7 | 40 h | +| C-2 | Monitoring and alerting | OR-1 | S-1, S-2, S-3, S-4 | 24x7 | 30 h | +| C-3 | Problem management | OR-2, OR-3 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-4 | Service request fulfilment | OR-8 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-5 | Change enablement | OR-4, OR-5 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-6 | Release and deployment | OR-4 | S-3 | business | 15 h | +| C-7 | Configuration management | OR-4, OR-5, OR-7 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-8 | Backup and recovery | OR-2, OR-10 | S-2, S-4 | business | 10 h | +| C-9 | Service continuity | OR-3 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-10 | Capacity and performance management | OR-3 | S-1 | business | 5 h | +| C-11 | Access management | OR-8 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-12 | Vulnerability and patch management | OR-3 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-13 | Security monitoring and response | OR-10 | S-1, S-2, S-3, S-4 | 24x7 | 40 h | +| C-14 | Cost management | OR-6 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-15 | Service level management | OR-6 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-16 | Platform product management | OR-8 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-17 | Self-service and golden paths | OR-4, OR-8 | S-1, S-3 | business | 25 h | +| C-18 | Enablement | OR-8 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-19 | Delivery and developer experience measurement | OR-4 | S-3 | business | 15 h | +| C-20 | Supplier management | OR-8, OR-9 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-21 | Regulatory evidence and reporting | OR-10 | S-2 | business | 10 h | + +## Who does what + +| Role | Party | Country | Level | Hours | +| --- | --- | --- | --- | --- | +| Customer Platform Lead | Customer (customer) | GB | lead | business | +| Customer Platform Engineer | Customer (customer) | GB | senior | business | +| Provider Delivery Lead | Service Provider (provider) | GB | lead | business | +| Provider 24x7 SRE | Service Provider (provider) | PL | senior | 24x7 | +| Provider Platform Engineer | Service Provider (provider) | IN | engineer | business | + +The teams, as Team Topologies names their kinds: + +| Team | Kind | Party | Roles | Purpose | +| --- | --- | --- | --- | --- | +| Platform team | platform | Customer | Customer Platform Lead, Customer Platform Engineer | Owns the platform roadmap, designs the networking, and provides explicit approvals for structural changes. | +| Provider Operations | platform | Service Provider | Provider Delivery Lead, Provider 24x7 SRE, Provider Platform Engineer | Provides 24x7 incident response, security monitoring, and general infrastructure operational support. | + +How the teams work together: + +| From | To | Mode | Until | Purpose | +| --- | --- | --- | --- | --- | +| Platform team | Provider Operations | x-as-a-service | | The customer platform team consumes infrastructure monitoring and incident response from the provider operations team as a service. | + +| Capability | Customer Platform Lead | Customer Platform Engineer | Provider Delivery Lead | Provider 24x7 SRE | Provider Platform Engineer | +| --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | C | | A | R | | +| Monitoring and alerting (C-2) | C | | A | R | | +| Problem management (C-3) | C | | A | | R | +| Service request fulfilment (C-4) | C | | A | | R | +| Change enablement (C-5) | A | R | | | R | +| Release and deployment (C-6) | A | R | | | | +| Configuration management (C-7) | A | | | | R | +| Backup and recovery (C-8) | C | | A | | R | +| Service continuity (C-9) | C | | A | | R | +| Capacity and performance management (C-10) | A | | | | R | +| Access management (C-11) | C | | A | | R | +| Vulnerability and patch management (C-12) | C | | A | | R | +| Security monitoring and response (C-13) | C | | A | R | | +| Cost management (C-14) | A | | R | | | +| Service level management (C-15) | C | | A/R | | | +| Platform product management (C-16) | A/R | | | | | +| Self-service and golden paths (C-17) | A | R | | | | +| Enablement (C-18) | A | R | | | | +| Delivery and developer experience measurement (C-19) | A | R | | | | +| Supplier management (C-20) | A/R | | | | | +| Regulatory evidence and reporting (C-21) | C | | A | | R | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Customer Platform Lead | Customer | GB | lead | business | 30.0 h | 0.2 | not priced | +| Customer Platform Engineer | Customer | GB | senior | business | 85.0 h | 0.6 | not priced | +| Provider Delivery Lead | Service Provider | GB | lead | business | 20.0 h | 0.1 | 2,212 GBP | +| Provider 24x7 SRE | Service Provider | PL | senior | 24x7 | 110.0 h | 5.0 | 41,640 GBP | +| Provider Platform Engineer | Service Provider | IN | engineer | business | 135.0 h | 0.9 | 2,400 GBP | + +6.8 people in all. Cost a month: 46,252 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. + +## Validation + +| Check | What must hold | Verdict | Evidence | +| --- | --- | --- | --- | +| OC-1 | The hub network and interconnect maintains at least 99.9% availability during each month. | pass | Capability C-1 is responsible for incident management with 24x7 coverage by the Provider 24x7 SRE role (R-4), which is sufficient to maintain highly available operations by responding to faults around the clock. | +| OC-2 | Audit logs are retained for seven years and cannot be deleted during this period. | pass | Capability C-8 manages backup and recovery, validating retention policies for audit logs. It is appropriately staffed during business hours. | +| OC-3 | Encryption keys are rotated at least once every 90 days. | pass | Capability C-7 covers configuration management including tracking key rotation cycles, with the Customer Platform Lead (R-1) accountable and Provider Platform Engineer (R-5) responsible. | +| OC-4 | A priority 1 fault on any Item is acknowledged within 15 minutes, at any hour. | pass | Capability C-1 covers incident management 24x7 with the Provider 24x7 SRE (R-4) responsible, ensuring 15-minute acknowledgement at any hour. | +| OC-5 | Service is restored within 4 hours following a priority 1 incident. | pass | Capability C-1 provides 24x7 incident response, ensuring that service is restored within the 4-hour target following a priority 1 incident. | +| OC-6 | An approved change reaches production within one day of approval. | pass | Capability C-6 manages release and deployment, with the Customer Platform Engineer (R-2) responsible, ensuring approved changes reach production within a day. | +| OC-7 | Every change to the network requires explicit approval from the customer platform team prior to implementation. | pass | Capability C-5 manages change enablement, with the Customer Platform Lead (R-1) accountable and engineers responsible, ensuring network changes require explicit approval. | +| OC-8 | A report detailing service levels is delivered to the customer monthly. | pass | Capability C-15 covers service level management to measure SLA achievement and deliver the monthly report, with the Provider Delivery Lead (R-3) accountable and responsible. | +| OC-9 | Platform operations is conducted under a shared service model between the customer and external provider. | pass | Capability C-20 handles supplier management to ensure shared service model compliance, with the Customer Platform Lead (R-1) accountable and responsible. | +| OC-10 | The Customer Platform Lead role is located in the UK. | pass | The Customer Platform Lead role (R-1) is defined with country 'GB' (UK). | +| OC-11 | The Customer Platform Engineer role is located in the UK. | pass | The Customer Platform Engineer role (R-2) is defined with country 'GB' (UK). | +| OC-12 | The Provider Delivery Lead role is located in the UK. | pass | The Provider Delivery Lead role (R-3) is defined with country 'GB' (UK). | +| OC-13 | The Provider 24x7 SRE role is located in Poland. | pass | The Provider 24x7 SRE role (R-4) is defined with country 'PL' (Poland). | +| OC-14 | The Provider Platform Engineer role is located in India. | pass | The Provider Platform Engineer role (R-5) is defined with country 'IN' (India). | +| OC-15 | A report detailing operational costs is delivered to the customer monthly. | pass | Capability C-14 covers cost management to compile financial reports on monthly landing zone spend, with the Provider Delivery Lead (R-3) responsible. | + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-1 on C-1 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-2 on C-8 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-3 on C-7 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 46,252 GBP, within the budget. | + +## Findings and open questions + +No findings: the solution provides what operating it needs. + +Assumed for the customer, and open to change when they answer: + +- `intent.exclusions`: Application workloads and services hosted on the platform foundation. Assumed based on platform foundation scope: workloads hosted on top of the landing zone are out of scope for this operating model. +- `operations.change_process`: Changes go through a deployment pipeline with peer review, with explicit customer platform team approval required for all network changes.. Pipeline-driven changes with peer review, with customer platform team approval required for network changes per REQ-01. +- `service_management.tooling`: Google Cloud operations suite and native Cloud provider tooling, with no separate external ITSM system.. Default native GCP operations tooling used as no external ITSM was specified. + +## Sources + +The Capabilities are from our own capability catalogue, which draws on these frameworks. It cites them and certifies nothing against them. + +- ITIL 4: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice +- Site Reliability Engineering: How Google Runs Production Systems: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives +- ISO/IEC 20000-1: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management +- Regulation (EU) 2022/2554, the Digital Operational Resilience Act: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents +- NIST Cybersecurity Framework: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management +- DevOps Research and Assessment (DORA) software delivery metrics: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- ISO/IEC 27001: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements +- FinOps Framework: Allocation; Budgeting; Forecasting; Anomaly Management +- CNCF Platforms White Paper (platform as a product): Platform as a product; Measuring platform success; Platform capabilities; Self-service; Platform as a product +- Team Topologies: Organizing Business and Technology Teams for Fast Flow: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode +- The SPACE of Developer Productivity: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- DevEx: What Actually Drives Productivity: Feedback loops; Cognitive load; Flow state + +**These references have not been checked against the frameworks by a person.** Treat each as a pointer to where to look. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_scope.md new file mode 100644 index 0000000..edf9069 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operating_scope.md @@ -0,0 +1,39 @@ +# Scope statement + +Operations covers the Google Cloud landing zone platform foundation, comprising the hub network and interconnect, the audit log store, the deployment pipeline, and encryption keys. The platform is operated under a shared service model between the customer's platform team based in the United Kingdom and a service provider delivering from the United Kingdom, Poland, and India, with the customer platform team retaining approval authority over all network changes. Operational support and system availability are maintained on a 24x7 basis, with priority 1 incident response within 15 minutes and target recovery within 4 hours. All operational costs are stated in British Pounds (GBP). Application workloads and services hosted on the platform foundation are excluded from operational scope. + +## Information register + +20 stated by the customer, 0 answered in conversation, 3 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the scope spec | +| intent.statement | We want our landing zone run as a shared service from go-live. A provider should respond to faults at every hour, and our own platform team keeps the final say over changes to the network. | the scope spec | +| intent.requirements | [{"id": "REQ-01", "statement": "Our platform team approves every change to the network."}, {"id": "REQ-02", "statement": "We receive a monthly report of service levels and cost."}] | the scope spec | +| intent.business_outcomes | [{"id": "BO-1", "outcome": "New payment products reach customers in weeks rather than months.", "target": "A change reaches production within a day of approval."}, {"id": "BO-2", "outcome": "Payments | the scope spec | +| solution.cloud_provider | gcp | the scope spec | +| solution.items | [{"id": "S-1", "name": "Hub network and interconnect", "kind": "network", "product": "Cloud Interconnect"}, {"id": "S-2", "name": "Audit log store", "kind": "storage", "product": "Cloud Storage"}, {"i | the scope spec | +| solution.conditions | [{"id": "K-1", "on": ["S-1"], "condition": "The platform is available at least 99.9% of each month."}, {"id": "K-2", "on": ["S-2"], "condition": "Audit logs cannot be deleted for seven years."}, {"id" | the scope spec | +| solution.build.duration_days | 36 | the scope spec | +| operations.support_hours | 24x7 | the scope spec | +| operations.availability_hours | 24x7 | the scope spec | +| sourcing.model | shared | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| organisation.existing_teams | [{"name": "Platform team", "party": "customer", "hours": "business", "country": "GB"}] | the scope spec | +| service_levels.priority_1_response_minutes | 15 | the scope spec | +| service_levels.recovery_time_hours | 4 | the scope spec | +| compliance.frameworks | ["DORA"] | the scope spec | +| commercial.currency | GBP | the scope spec | +| commercial.monthly_run_budget | 60000 | the scope spec | +| intent.exclusions | ["Application workloads and services hosted on the platform foundation"] | assumed | +| operations.change_process | Changes go through a deployment pipeline with peer review, with explicit customer platform team approval required for all network changes. | assumed | +| service_management.tooling | Google Cloud operations suite and native Cloud provider tooling, with no separate external ITSM system. | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.exclusions is ['Application workloads and services hosted on the platform foundation']. Assumed based on platform foundation scope: workloads hosted on top of the landing zone are out of scope for this operating model. | Changes which Capabilities are designed. | +| ASSUMP-02 | Assumed operations.change_process is Changes go through a deployment pipeline with peer review, with explicit customer platform team approval required for all network changes.. Pipeline-driven changes with peer review, with customer platform team approval required for network changes per REQ-01. | Changes change enablement and release. | +| ASSUMP-03 | Assumed service_management.tooling is Google Cloud operations suite and native Cloud provider tooling, with no separate external ITSM system.. Default native GCP operations tooling used as no external ITSM was specified. | Changes tooling and the processes written for it. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operational_requirements.md new file mode 100644 index 0000000..c7f5cf7 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/operational_requirements.md @@ -0,0 +1,45 @@ +# Operational requirements + +The operational requirements specify what running the Google Cloud platform foundation must achieve. Each requirement is derived directly from customer statements, binding scoping parameters, solution conditions, or stated business outcomes. + +### Service level requirements + +- **OR-1**: A priority 1 fault on any Item is acknowledged within 15 minutes, at any hour (24x7 coverage). +- **OR-2**: Service is restored within 4 hours following a priority 1 incident or critical failure. +- **OR-3**: The hub network and interconnect maintains at least 99.9% availability during each calendar month. +- **OR-4**: An approved change reaches production deployment within one day of approval. + +### Process requirements + +- **OR-5**: Every change to the network requires explicit approval from the customer platform team prior to implementation. +- **OR-6**: A report detailing operational service levels and run costs is delivered to the customer on a monthly basis. +- **OR-7**: Cryptographic encryption keys are rotated at least once every 90 days. + +### Organisation requirements + +- **OR-8**: Platform operations is conducted under a shared service model between the customer and an external service provider. +- **OR-9**: Operational delivery activities are conducted exclusively from the United Kingdom, Poland, and India. + +### Compliance requirements + +- **OR-10**: Audit logs stored in the audit log store cannot be deleted for seven years. + +--- + +# What was not stated + +The operating specification leaves certain operational aspects assumed or unknown. If answered differently by the customer, these would alter the operational requirements. + +## Assumptions + +- **Scope exclusions (`intent.exclusions`)**: It is assumed that application workloads and services hosted on top of the platform foundation are excluded from operations. If customer workloads were included in scope, additional operational requirements would be needed for workload availability, deployment pipeline maintenance, and application incident response. +- **Change process (`operations.change_process`)**: It is assumed that changes proceed through a continuous deployment pipeline with automated checks and peer review, with customer approval reserved specifically for network modifications. If a centralised Change Advisory Board (CAB) or multi-tier manual approval process were required across all components, change velocity requirements would be directly affected. +- **Service management tooling (`service_management.tooling`)**: It is assumed that Google Cloud native operations suite tooling is utilized without a separate enterprise ITSM platform. If an external ITSM or specific ticketing system were mandated, dedicated tooling integration requirements would apply. + +## Unknowns + +- **Go-live date (`transition.go_live_date`)**: The target operational cutover date was not specified, leaving transitional readiness timelines to be determined. +- **Customer-owned build components (`solution.build.customer_owned`)**: The exact division of build ownership between customer and suppliers was not defined, which may influence handoff requirements. +- **Inherited risks (`solution.inherited_risks`)**: No existing technical debt or inherited design risks were declared; if identified, specific operational monitoring or mitigation guardrails would be required. +- **Monitoring integrations (`operations.monitoring`)**: Specific external monitoring, log ingestion, or enterprise SIEM destinations were not stated beyond the native audit log store. +- **Operational headcount limits (`organisation.headcount_limit`)**: No upper bound on staffing was set, with staffing to be determined strictly by service hours and capability requirements. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/target_operating_model.md new file mode 100644 index 0000000..b854264 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/target_operating_model.md @@ -0,0 +1,66 @@ +# Design rationale + +## Sourcing and Division of Work +The operating model establishes a shared service model between the Customer (P-1) and an external Service Provider (P-2). +- The **Customer Platform team** (T-1) retains accountability for the platform's architectural integrity and roadmap, as well as operational decisions surrounding changes to the network. The team handles change enablement, self-service golden paths, delivery and developer experience measurement, and capacity management. +- The **Provider Operations team** (T-2) is tasked with delivering 24x7 visibility and stability. They provide incident management, security monitoring, access management, backup and recovery, and handle routine service requests. This arrangement allows the Customer to focus on enablement and evolving the platform product rather than daily operational toil. + +## Roles, Countries, and Levels +The Roles map directly to the `sourcing.delivery_countries` constraints (GB, PL, IN) and specific support hours: +- **Customer Platform Lead (GB, Lead, Business hours)**: Ensures accountability and oversees the platform's product roadmap, holding final approval over network changes to meet governance requirements. +- **Customer Platform Engineer (GB, Senior, Business hours)**: Focuses on engineering tasks, automation, enablement, and maintaining the deployment pipeline during standard business hours. +- **Provider Delivery Lead (GB, Lead, Business hours)**: Acts as the primary point of contact for the Service Provider. They are accountable for incident management, security monitoring, and compiling monthly reports (detailing service levels and costs) ensuring strict governance. +- **Provider 24x7 SRE (PL, Senior, 24x7)**: Selected in Poland to offer around-the-clock incident response, event monitoring, and security oversight, ensuring 15-minute response times and four-hour restoration for priority incidents. A senior level is appropriate to diagnose unfamiliar faults and autonomously lead shifts. +- **Provider Platform Engineer (IN, Engineer, Business hours)**: Located in India to provide cost-efficient business-hours support for routine service requests, problem management, access requests, patch management, and configuration tasks. + +*(Note: Costs derived from this operating model will use placeholders as the rate card remains in draft.)* + +## Assumptions +- We assume that the customer platform team can review and approve network changes within the pipeline seamlessly, without becoming a bottleneck that prevents the 1-day deployment target. +- We assume that natively available GCP tools (like Cloud Monitoring and Cloud Logging) are used for alert configuration and routing to the 24x7 SRE team, avoiding the need for third-party integrations or findings for specific tools. +- We assume backup and recovery policies will be enforced through configuration mechanisms as opposed to creating distinct tools requiring external findings. + +## Responsibility matrix + +| Capability | Customer Platform Lead | Customer Platform Engineer | Provider Delivery Lead | Provider 24x7 SRE | Provider Platform Engineer | +| --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | C | | A | R | | +| Monitoring and alerting (C-2) | C | | A | R | | +| Problem management (C-3) | C | | A | | R | +| Service request fulfilment (C-4) | C | | A | | R | +| Change enablement (C-5) | A | R | | | R | +| Release and deployment (C-6) | A | R | | | | +| Configuration management (C-7) | A | | | | R | +| Backup and recovery (C-8) | C | | A | | R | +| Service continuity (C-9) | C | | A | | R | +| Capacity and performance management (C-10) | A | | | | R | +| Access management (C-11) | C | | A | | R | +| Vulnerability and patch management (C-12) | C | | A | | R | +| Security monitoring and response (C-13) | C | | A | R | | +| Cost management (C-14) | A | | R | | | +| Service level management (C-15) | C | | A/R | | | +| Platform product management (C-16) | A/R | | | | | +| Self-service and golden paths (C-17) | A | R | | | | +| Enablement (C-18) | A | R | | | | +| Delivery and developer experience measurement (C-19) | A | R | | | | +| Supplier management (C-20) | A/R | | | | | +| Regulatory evidence and reporting (C-21) | C | | A | | R | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Customer Platform Lead | Customer | GB | lead | business | 30.0 h | 0.2 | not priced | +| Customer Platform Engineer | Customer | GB | senior | business | 85.0 h | 0.6 | not priced | +| Provider Delivery Lead | Service Provider | GB | lead | business | 20.0 h | 0.1 | 2,212 GBP | +| Provider 24x7 SRE | Service Provider | PL | senior | 24x7 | 110.0 h | 5.0 | 41,640 GBP | +| Provider Platform Engineer | Service Provider | IN | engineer | business | 135.0 h | 0.9 | 2,400 GBP | + +6.8 people in all. Cost a month: 46,252 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/validation_results.md new file mode 100644 index 0000000..6de8f4e --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r1/deliverables/validation_results.md @@ -0,0 +1,42 @@ +# Check verdicts + +All checks against the target operating model have successfully passed. The operating model is capable of meeting the defined conditions and requirements as designed. + +- **OC-1**: **Pass**. Capability C-1 is responsible for incident management with 24x7 coverage by the Provider 24x7 SRE role (R-4), which is sufficient to maintain highly available operations by responding to faults around the clock. +- **OC-2**: **Pass**. Capability C-8 manages backup and recovery, validating retention policies for audit logs. It is appropriately staffed during business hours. +- **OC-3**: **Pass**. Capability C-7 covers configuration management including tracking key rotation cycles, with the Customer Platform Lead (R-1) accountable and Provider Platform Engineer (R-5) responsible. +- **OC-4**: **Pass**. Capability C-1 covers incident management 24x7 with the Provider 24x7 SRE (R-4) responsible, ensuring 15-minute acknowledgement at any hour. +- **OC-5**: **Pass**. Capability C-1 provides 24x7 incident response, ensuring that service is restored within the 4-hour target following a priority 1 incident. +- **OC-6**: **Pass**. Capability C-6 manages release and deployment, with the Customer Platform Engineer (R-2) responsible, ensuring approved changes reach production within a day. +- **OC-7**: **Pass**. Capability C-5 manages change enablement, with the Customer Platform Lead (R-1) accountable and engineers responsible, ensuring network changes require explicit approval. +- **OC-8**: **Pass**. Capability C-15 covers service level management to measure SLA achievement and deliver the monthly report, with the Provider Delivery Lead (R-3) accountable and responsible. +- **OC-9**: **Pass**. Capability C-20 handles supplier management to ensure shared service model compliance, with the Customer Platform Lead (R-1) accountable and responsible. +- **OC-10**: **Pass**. The Customer Platform Lead role (R-1) is defined with country 'GB' (UK). +- **OC-11**: **Pass**. The Customer Platform Engineer role (R-2) is defined with country 'GB' (UK). +- **OC-12**: **Pass**. The Provider Delivery Lead role (R-3) is defined with country 'GB' (UK). +- **OC-13**: **Pass**. The Provider 24x7 SRE role (R-4) is defined with country 'PL' (Poland). +- **OC-14**: **Pass**. The Provider Platform Engineer role (R-5) is defined with country 'IN' (India). +- **OC-15**: **Pass**. Capability C-14 covers cost management to compile financial reports on monthly landing zone spend, with the Provider Delivery Lead (R-3) responsible. + +**Overall Verdict** +**Pass**. The target operating model successfully satisfies all required operational checks. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Provider 24x7 SRE responds under C-1, and Provider Delivery Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Customer Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-1 on C-1 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-2 on C-8 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-3 on C-7 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 46,252 GBP, within the budget. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/capability_model.md new file mode 100644 index 0000000..2c2e09c --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/capability_model.md @@ -0,0 +1,54 @@ +## Capabilities + +The following capabilities establish what the operating organisation must be able to execute to satisfy the defined operational requirements. + +*(Note: While the attached rate card remains a draft, any costs computed downstream from these capabilities are strictly placeholders.)* + +| ID | Capability | Purpose / Why it is needed | Items covered | Serves | +| :--- | :--- | :--- | :--- | :--- | +| **C-1** | Incident management | To restore normal service after a fault and handle Priority 1 incidents round-the-clock. | S-1, S-2, S-3, S-4 | OR-1, OR-2 | +| **C-2** | Monitoring and alerting | To detect platform faults, track 99.9% availability, and trigger instant alerts across 24x7 hours. | S-1, S-2, S-3, S-4 | OR-1, OR-3 | +| **C-3** | Problem management | To perform root cause analysis and prevent the recurrence of high-impact platform incidents. | S-1, S-2, S-3, S-4 | OR-2, OR-3 | +| **C-4** | Service request fulfilment | To manage standard requests and access requests between the shared sourcing teams. | S-1, S-2, S-3, S-4 | OR-6 | +| **C-5** | Change enablement | To review risks and ensure all hub network changes receive customer platform team approval. | S-1, S-2, S-3, S-4 | OR-4 | +| **C-6** | Release and deployment | To execute automated pipelines smoothly and roll back any failing changes. | S-1, S-3 | OR-4 | +| **C-7** | Configuration management | To maintain infrastructure-as-code state accuracy for all network and key configurations. | S-1, S-2, S-3, S-4 | OR-4 | +| **C-8** | Backup and recovery | To ensure data resilience, key backups, and recovery from catastrophic component loss. | S-2, S-4 | OR-2 | +| **C-9** | Service continuity | To rehearse and ensure the platform foundation recovers within 4 hours in disaster scenarios. | S-1, S-2, S-3, S-4 | OR-2, OR-3 | +| **C-10** | Capacity and performance management | To monitor hub network bandwidth and pipeline usage against limits. | S-1, S-3 | OR-3 | +| **C-11** | Access management | To restrict log and key access tightly, preventing unauthorized deletion of audit logs. | S-1, S-2, S-3, S-4 | OR-8 | +| **C-12** | Vulnerability and patch management | To routinely scan for flaws and apply necessary mitigations, protecting platform availability. | S-1, S-2, S-3, S-4 | OR-3 | +| **C-13** | Security monitoring and response | To monitor security incidents 24x7 and rapidly react to malicious events. | S-1, S-2, S-3, S-4 | OR-1, OR-8 | +| **C-14** | Cost management | To track cloud spend against budget and compile the cost segment of the monthly report. | S-1, S-2, S-3, S-4 | OR-5 | +| **C-15** | Service level management | To measure compliance with stated SLAs and package the findings into the monthly customer report. | S-1, S-2, S-3, S-4 | OR-5 | +| **C-16** | Platform product management | To manage the foundation as a product, prioritizing improvements based on developer feedback. | S-1, S-2, S-3, S-4 | OR-6 | +| **C-17** | Self-service and golden paths | To offer default, streamlined pipeline deployment paths for easier developer adoption. | S-3 | OR-4, OR-6 | +| **C-18** | Enablement | To assist the customer team with using the foundational deployment pipelines via pairing and training. | S-3 | OR-6 | +| **C-19** | Delivery and developer experience measurement | To report on metrics such as deployment speed and team satisfaction, supporting continuous improvement. | S-3 | OR-5 | +| **C-20** | Supplier management | To govern the multi-location external provider and ensure location constraints are enforced. | S-1, S-2, S-3, S-4 | OR-6, OR-7 | +| **C-21** | Regulatory evidence and reporting | To collect audit evidence of 90-day key rotations and 7-year log retention for compliance alignment. | S-2, S-4 | OR-8, OR-9 | + +## Sources + +The capability catalogue draws heavily on prominent industry frameworks. + +*(Note: All sources referenced in this capability model are currently flagged as **unverified** because they are drawn from the catalogue's register, whose status indicates they were drafted from general knowledge. They have not been independently certified. The capability catalogue draws on them and promises nothing regarding compliance.)* + +* **ITIL 4** (PeopleCert, version 4): Incident management practice, Monitoring and event management practice, Problem management practice, Service request management practice, Change enablement practice, Release management and Deployment management practices, Service configuration management practice, Service continuity management practice, Capacity and performance management practice, Supplier management practice, Service level management practice. +* **Site Reliability Engineering: How Google Runs Production Systems** (Google / O'Reilly, 2016): Managing Incidents, Monitoring Distributed Systems, Postmortem Culture: Learning from Failure, Release Engineering, Service Level Objectives. +* **ISO/IEC 20000-1** (ISO/IEC, 2018): Incident management, Service request management, Change management, Service continuity management, Capacity management, Service level management, Supplier management. +* **Regulation (EU) 2022/2554, the Digital Operational Resilience Act** (European Union, 2022): Article 9, Protection and prevention; Article 10, Detection; Article 11, Response and recovery; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 17, ICT-related incident management process; Article 19, Reporting of major ICT-related incidents; Article 28, General principles on ICT third-party risk. +* **NIST Cybersecurity Framework** (National Institute of Standards and Technology, 2.0): Detect: Continuous Monitoring; Detect: Adverse Event Analysis; Respond: Incident Management; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security. +* **ISO/IEC 27001** (ISO/IEC, 2022): Access control, Management of technical vulnerabilities, Information security incident management, Compliance with legal and contractual requirements. +* **FinOps Framework** (FinOps Foundation, 2024): Allocation; Budgeting; Forecasting; Anomaly Management. +* **Team Topologies: Organizing Business and Technology Teams for Fast Flow** (IT Revolution, 2019): Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode. +* **CNCF Platforms White Paper (platform as a product)** (Cloud Native Computing Foundation, 2023): Platform as a product; Measuring platform success; Platform capabilities; Self-service. +* **DevOps Research and Assessment (DORA) software delivery metrics** (Google Cloud, 2023): Change failure rate, Deployment frequency and lead time for changes, Time to restore service. +* **The SPACE of Developer Productivity** (Forsgren et al., ACM Queue, 2021): Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow. +* **DevEx: What Actually Drives Productivity** (Noda et al., ACM Queue, 2023): Feedback loops; Cognitive load; Flow state. + +## Assumptions + +Since pre-approved execution is enabled, the following assumptions were made regarding capability assignments and requirements: +- Single instances of each capability per catalogue entry are sufficient, rather than splitting capabilities based on differing items. The shared sourcing model's division of duties is expected to be managed later at the Role layer. +- Operational capabilities associated with "delivery-and-developer experience measurement" actively apply due to the business intent to streamline delivery and preserve developer retention. These are adopted to provide transparent progress towards these goals without mandating complex underlying toolchains. \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_model_guide.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_model_guide.md new file mode 100644 index 0000000..12665c9 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_model_guide.md @@ -0,0 +1,234 @@ +# Operating model guide: Google Cloud landing zone foundation + +## Summary + +This guide defines the target operating model for running the Google Cloud Platform landing zone foundation, comprising the hub network and interconnect, audit log store, deployment pipeline, and cryptographic keys, while application workloads deployed onto the foundation remain out of operational scope. Operations are shared between the customer platform team and an external service provider across the United Kingdom, Poland, and India. The customer platform team retains platform product ownership and final approval authority over all network changes, while the external service provider delivers continuous monitoring and rapid fault response alongside pipeline maintenance and enablement. + +Operational support and platform availability operate continuously round the clock, with the provider handling fault response across all hours, while change enablement, platform engineering, and governance occur during business hours. Readers should look first at the findings and open questions section to review the unconfirmed operational assumptions and unstated items—including the native operations tooling baseline and workload boundaries—as well as the provisional nature of the draft rate card. + +### Key figures + +- Operated: 4 Items, with 3 Conditions to keep true. +- Faults responded to: 24x7 hours. +- Capabilities: 21. +- People: 11.0. Cost a month: 44,720 GBP, from rate card version `draft-2026-10-13` (placeholder prices: an illustration). +- Scenario walk-throughs: 13 of 13 pass. +- Business outcomes: 3, 3 shown by a Measure. +- Findings for whoever owns the design: 0. + +## Business outcomes + +**BO-1: New payment products reach customers in weeks rather than months.** Target: A change reaches production within a day of approval.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Deployment frequency | outcome | dora-metrics | A change reaches production within a day of approval. | Delivery and developer experience measurement | Operations Lead | + +**BO-2: Payments are never unavailable for long.** Target: Restored within 4 hours.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Time to restore service | outcome | dora-metrics | Restored within 4 hours. | Delivery and developer experience measurement | Operations Lead | + +**BO-3: We keep the platform engineers we hire.** + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Satisfaction and well-being | guardrail | space-framework | | Delivery and developer experience measurement | Operations Lead | +| Friction and cognitive load | leading | devex-framework | | Delivery and developer experience measurement | Operations Lead | + +## What is operated + +| Item | What it is | Kind | Product | From | +| --- | --- | --- | --- | --- | +| S-1 | Hub network and interconnect | network | Cloud Interconnect | | +| S-2 | Audit log store | storage | Cloud Storage | | +| S-3 | Deployment pipeline | pipeline | Cloud Build | | +| S-4 | Encryption keys | keys | Cloud KMS | | + +What must be kept true while it is run: + +| Condition | On | What must hold | From | +| --- | --- | --- | --- | +| K-1 | S-1 | The platform is available at least 99.9% of each month. | | +| K-2 | S-2 | Audit logs cannot be deleted for seven years. | | +| K-3 | S-4 | Encryption keys are rotated every 90 days. | | + +## Operational requirements + +| Requirement | Kind | What must be achieved | From | +| --- | --- | --- | --- | +| OR-1 | service_level | A Priority 1 fault on any landing zone component is acknowledged and responded to within 15 minutes, at any hour (24x7). | service_levels.priority_1_response_minutes, operations.support_hours | +| OR-2 | service_level | Normal platform operations are restored within 4 hours of a Priority 1 incident occurring. | service_levels.recovery_time_hours | +| OR-3 | service_level | The platform foundation maintains an operational availability of at least 99.9% across all hours (24x7) of each calendar month. | K-1, operations.availability_hours | +| OR-4 | process | Every change to the hub network requires customer platform team approval prior to deployment. | REQ-01 | +| OR-5 | process | A reporting package detailing achieved service levels and operational costs is delivered to the customer on a monthly basis. | REQ-02 | +| OR-6 | organisation | Platform foundation operations are delivered under a shared sourcing model between the customer and an external service provider. | sourcing.model | +| OR-7 | organisation | Operational services are delivered exclusively from the United Kingdom, Poland, and India. | sourcing.delivery_countries | +| OR-8 | compliance | Audit logs stored in the audit log store cannot be deleted and are retained for at least seven years. | K-2 | +| OR-9 | compliance | Cryptographic encryption keys are rotated at least once every 90 days. | K-3 | + +## Capabilities + +| Capability | What it is | Serves | Covers | Hours | Effort a month | +| --- | --- | --- | --- | --- | --- | +| C-1 | Incident management | OR-1, OR-2 | S-1, S-2, S-3, S-4 | 24x7 | 20 h | +| C-2 | Monitoring and alerting | OR-1, OR-3 | S-1, S-2, S-3, S-4 | 24x7 | 24 h | +| C-3 | Problem management | OR-2, OR-3 | S-1, S-2, S-3, S-4 | business | 12 h | +| C-4 | Service request fulfilment | OR-6 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-5 | Change enablement | OR-4 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-6 | Release and deployment | OR-4 | S-1, S-3 | business | 12 h | +| C-7 | Configuration management | OR-4 | S-1, S-2, S-3, S-4 | business | 8 h | +| C-8 | Backup and recovery | OR-2 | S-2, S-4 | business | 4 h | +| C-9 | Service continuity | OR-2, OR-3 | S-1, S-2, S-3, S-4 | business | 8 h | +| C-10 | Capacity and performance management | OR-3 | S-1, S-3 | business | 4 h | +| C-11 | Access management | OR-8 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-12 | Vulnerability and patch management | OR-3 | S-1, S-2, S-3, S-4 | business | 16 h | +| C-13 | Security monitoring and response | OR-1, OR-8 | S-1, S-2, S-3, S-4 | 24x7 | 24 h | +| C-14 | Cost management | OR-5 | S-1, S-2, S-3, S-4 | business | 6 h | +| C-15 | Service level management | OR-5 | S-1, S-2, S-3, S-4 | business | 8 h | +| C-16 | Platform product management | OR-6 | S-1, S-2, S-3, S-4 | business | 12 h | +| C-17 | Self-service and golden paths | OR-4, OR-6 | S-3 | business | 16 h | +| C-18 | Enablement | OR-6 | S-3 | business | 10 h | +| C-19 | Delivery and developer experience measurement | OR-5 | S-3 | business | 8 h | +| C-20 | Supplier management | OR-6, OR-7 | S-1, S-2, S-3, S-4 | business | 4 h | +| C-21 | Regulatory evidence and reporting | OR-8, OR-9 | S-2, S-4 | business | 8 h | + +## Who does what + +| Role | Party | Country | Level | Hours | +| --- | --- | --- | --- | --- | +| Platform Lead | Customer platform team (customer) | GB | lead | business | +| Platform Engineer | Customer platform team (customer) | GB | engineer | business | +| Operations Lead | External service provider (provider) | PL | lead | business | +| Operations Engineer | External service provider (provider) | IN | engineer | 24x7 | +| Security Operations Engineer | External service provider (provider) | IN | senior | 24x7 | +| Senior SRE | External service provider (provider) | IN | senior | business | +| Enablement Engineer | External service provider (provider) | IN | senior | business | + +The teams, as Team Topologies names their kinds: + +| Team | Kind | Party | Roles | Purpose | +| --- | --- | --- | --- | --- | +| Platform team | platform | Customer platform team | Platform Lead, Platform Engineer | Customer platform foundation ownership | +| Shared operations | platform | External service provider | Operations Lead, Operations Engineer, Security Operations Engineer, Senior SRE | Shared operational delivery of platform foundations | +| Enablement team | enabling | External service provider | Enablement Engineer | Facilitating adoption of pipelines | + +How the teams work together: + +| From | To | Mode | Until | Purpose | +| --- | --- | --- | --- | --- | +| Platform team | Shared operations | x-as-a-service | | Customer platform team consumes shared operational services and platforms provided by the external provider. | +| Enablement team | Platform team | facilitating | Adoption of foundational pipelines is complete and customer team is self-sufficient. | Provider enablement team assists the customer with adopting foundational deployment pipelines. | + +| Capability | Platform Lead | Platform Engineer | Operations Lead | Operations Engineer | Security Operations Engineer | Senior SRE | Enablement Engineer | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | C | A | R | | | | +| Monitoring and alerting (C-2) | | C | A | R | | | | +| Problem management (C-3) | | C | A | | | R | | +| Service request fulfilment (C-4) | | C | A | R | | | | +| Change enablement (C-5) | A | R | | | | C | | +| Release and deployment (C-6) | | C | A | | | R | | +| Configuration management (C-7) | | C | A | | | R | | +| Backup and recovery (C-8) | | C | A | | | R | | +| Service continuity (C-9) | | C | A | | | R | | +| Capacity and performance management (C-10) | | C | A | | | R | | +| Access management (C-11) | | C | A | | R | | | +| Vulnerability and patch management (C-12) | | C | A | | | R | | +| Security monitoring and response (C-13) | | C | A | | R | | | +| Cost management (C-14) | C | | A | | | R | | +| Service level management (C-15) | C | | A | | | R | | +| Platform product management (C-16) | A/R | | | | | | | +| Self-service and golden paths (C-17) | | C | A | | | R | | +| Enablement (C-18) | | C | A | | | | R | +| Delivery and developer experience measurement (C-19) | C | | A | | | R | | +| Supplier management (C-20) | A/R | | | | | | | +| Regulatory evidence and reporting (C-21) | | C | A | | | R | | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Lead | Customer platform team | GB | lead | business | 16.0 h | 0.1 | not priced | +| Platform Engineer | Customer platform team | GB | engineer | business | 10.0 h | 0.1 | not priced | +| Operations Lead | External service provider | PL | lead | business | 0.0 h | 0.0 | 0 GBP | +| Operations Engineer | External service provider | IN | engineer | 24x7 | 59.0 h | 5.0 | 17,034 GBP | +| Security Operations Engineer | External service provider | IN | senior | 24x7 | 34.0 h | 5.0 | 24,605 GBP | +| Senior SRE | External service provider | IN | senior | business | 110.0 h | 0.7 | 2,824 GBP | +| Enablement Engineer | External service provider | IN | senior | business | 10.0 h | 0.1 | 257 GBP | + +11.0 people in all. Cost a month: 44,720 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. + +## Validation + +| Check | What must hold | Verdict | Evidence | +| --- | --- | --- | --- | +| OC-1 | A Priority 1 fault on the platform is acknowledged and responded to within 15 minutes, at any hour. | pass | Capability C-1 covers incident management and operates 24x7. Role R-4 (Operations Engineer) is responsible and operates 24x7. | +| OC-2 | Normal platform operations are restored within 4 hours of a Priority 1 incident occurring. | pass | Capability C-1 operates 24x7 and responds to incidents. | +| OC-3 | The hub network (S-1) is monitored to verify it maintains at least 99.9% availability. | pass | Capability C-2 covers monitoring and operates 24x7, with Role R-4 responsible. The operating model configures native Google Cloud monitoring tools to fulfill this. | +| OC-4 | Changes to the hub network (S-1) are blocked unless approved by the customer platform team. | pass | Capability C-5 covers change enablement. Role R-1 (Platform Lead from the customer team) is accountable and Role R-2 (Platform Engineer from the customer team) is responsible, giving the customer final approval. | +| OC-5 | A reporting package detailing achieved service levels and operational costs is delivered to the customer monthly. | pass | Capability C-15 covers service level management. Role R-6 (Senior SRE) is responsible for producing the monthly reporting package. | +| OC-6 | Platform foundation operations are governed under a shared sourcing model. | pass | Capability C-20 covers supplier management to govern the shared sourcing model. Role R-1 (Platform Lead) is accountable and responsible. | +| OC-7 | Access controls prevent audit logs in the audit log store (S-2) from being deleted for at least seven years. | pass | Capability C-11 covers access management. Role R-5 (Security Operations Engineer) is responsible. | +| OC-8 | Evidence is collected to demonstrate cryptographic encryption keys (S-4) are rotated at least once every 90 days. | pass | Capability C-21 covers regulatory evidence collection. Role R-6 (Senior SRE) is responsible. | +| OC-9 | The Operations Lead role sits in Poland (PL). | pass | Role R-3 (Operations Lead) is defined with country 'PL' (Poland). | +| OC-10 | The Operations Engineer role sits in India (IN). | pass | Role R-4 (Operations Engineer) is defined with country 'IN' (India). | +| OC-11 | The Security Operations Engineer role sits in India (IN). | pass | Role R-5 (Security Operations Engineer) is defined with country 'IN' (India). | +| OC-12 | The Senior SRE role sits in India (IN). | pass | Role R-6 (Senior SRE) is defined with country 'IN' (India). | +| OC-13 | The Enablement Engineer role sits in India (IN). | pass | Role R-7 (Enablement Engineer) is defined with country 'IN' (India). | +| OC-14 | The Platform Lead role sits in the United Kingdom (GB). | pass | Role R-1 (Platform Lead) is defined with country 'GB' (United Kingdom). | +| OC-15 | The Platform Engineer role sits in the United Kingdom (GB). | pass | Role R-2 (Platform Engineer) is defined with country 'GB' (United Kingdom). | + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-3 on C-2 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-7 on C-11 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-8 on C-21 keeps it true. | +| External service provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 44,720 GBP, within the budget. | + +## Findings and open questions + +No findings: the solution provides what operating it needs. + +Assumed for the customer, and open to change when they answer: + +- `intent.exclusions`: Application workloads deployed onto the platform foundation. Workloads deployed onto the platform foundation are excluded; operations is confined to the landing zone foundation itself. +- `operations.change_process`: Changes go through automated pipelines with peer review; changes affecting the hub network require customer platform team approval prior to release.. Changes follow automated deployment pipelines with peer review, with network changes requiring explicit customer platform team approval per REQ-01. +- `service_management.tooling`: Google Cloud native operations tooling (Cloud Monitoring and Cloud Logging), with no separate service management system.. Defaulting to GCP native operations tooling with no separate ITSM system. + +## Sources + +The Capabilities are from our own capability catalogue, which draws on these frameworks. It cites them and certifies nothing against them. + +- ITIL 4: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice +- Site Reliability Engineering: How Google Runs Production Systems: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives +- ISO/IEC 20000-1: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management +- Regulation (EU) 2022/2554, the Digital Operational Resilience Act: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents +- NIST Cybersecurity Framework: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management +- DevOps Research and Assessment (DORA) software delivery metrics: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- ISO/IEC 27001: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements +- FinOps Framework: Allocation; Budgeting; Forecasting; Anomaly Management +- CNCF Platforms White Paper (platform as a product): Platform as a product; Measuring platform success; Platform capabilities; Self-service; Platform as a product +- Team Topologies: Organizing Business and Technology Teams for Fast Flow: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode +- The SPACE of Developer Productivity: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- DevEx: What Actually Drives Productivity: Feedback loops; Cognitive load; Flow state + +**These references have not been checked against the frameworks by a person.** Treat each as a pointer to where to look. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_scope.md new file mode 100644 index 0000000..ce9bb84 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operating_scope.md @@ -0,0 +1,39 @@ +## Scope statement + +This operating scope covers the ongoing operation of the customer's Google Cloud Platform landing zone platform foundation, comprising the hub network and interconnect, audit log store, deployment pipeline, and encryption keys. Operations are delivered under a shared sourcing model between the customer's platform team and a service provider operating from the United Kingdom, Poland, and India, with the customer platform team retaining approval authority over all network changes. The service is operated continuously with 24x7 availability and 24x7 fault response, targeting a 15-minute response time for Priority 1 incidents and a 4-hour recovery time. All operational costs for this service are stated in British Pounds (GBP). Application workloads and hosted services deployed onto the platform foundation are out of scope for operations. + +## Information register + +20 stated by the customer, 0 answered in conversation, 3 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the scope spec | +| intent.statement | We want our landing zone run as a shared service from go-live. A provider should respond to faults at every hour, and our own platform team keeps the final say over changes to the network. | the scope spec | +| intent.requirements | [{"id": "REQ-01", "statement": "Our platform team approves every change to the network."}, {"id": "REQ-02", "statement": "We receive a monthly report of service levels and cost."}] | the scope spec | +| intent.business_outcomes | [{"id": "BO-1", "outcome": "New payment products reach customers in weeks rather than months.", "target": "A change reaches production within a day of approval."}, {"id": "BO-2", "outcome": "Payments | the scope spec | +| solution.cloud_provider | gcp | the scope spec | +| solution.items | [{"id": "S-1", "name": "Hub network and interconnect", "kind": "network", "product": "Cloud Interconnect"}, {"id": "S-2", "name": "Audit log store", "kind": "storage", "product": "Cloud Storage"}, {"i | the scope spec | +| solution.conditions | [{"id": "K-1", "on": ["S-1"], "condition": "The platform is available at least 99.9% of each month."}, {"id": "K-2", "on": ["S-2"], "condition": "Audit logs cannot be deleted for seven years."}, {"id" | the scope spec | +| solution.build.duration_days | 36 | the scope spec | +| operations.support_hours | 24x7 | the scope spec | +| operations.availability_hours | 24x7 | the scope spec | +| sourcing.model | shared | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| organisation.existing_teams | [{"name": "Platform team", "party": "customer", "hours": "business", "country": "GB"}] | the scope spec | +| service_levels.priority_1_response_minutes | 15 | the scope spec | +| service_levels.recovery_time_hours | 4 | the scope spec | +| compliance.frameworks | ["DORA"] | the scope spec | +| commercial.currency | GBP | the scope spec | +| commercial.monthly_run_budget | 60000 | the scope spec | +| intent.exclusions | ["Application workloads deployed onto the platform foundation"] | assumed | +| operations.change_process | Changes go through automated pipelines with peer review; changes affecting the hub network require customer platform team approval prior to release. | assumed | +| service_management.tooling | Google Cloud native operations tooling (Cloud Monitoring and Cloud Logging), with no separate service management system. | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.exclusions is ['Application workloads deployed onto the platform foundation']. Workloads deployed onto the platform foundation are excluded; operations is confined to the landing zone foundation itself. | Changes which Capabilities are designed. | +| ASSUMP-02 | Assumed operations.change_process is Changes go through automated pipelines with peer review; changes affecting the hub network require customer platform team approval prior to release.. Changes follow automated deployment pipelines with peer review, with network changes requiring explicit customer platform team approval per REQ-01. | Changes change enablement and release. | +| ASSUMP-03 | Assumed service_management.tooling is Google Cloud native operations tooling (Cloud Monitoring and Cloud Logging), with no separate service management system.. Defaulting to GCP native operations tooling with no separate ITSM system. | Changes tooling and the processes written for it. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operational_requirements.md new file mode 100644 index 0000000..e45f189 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/operational_requirements.md @@ -0,0 +1,74 @@ +## Operational requirements + +The operational requirements define what running the landing zone platform foundation must achieve. Every requirement is traced directly to the customer's stated requirements, binding operating fields, or architectural conditions. + +### Service level requirements + +- **OR-1**: A Priority 1 fault on any landing zone component is acknowledged and responded to within 15 minutes, at any hour (24x7). + *Source: `service_levels.priority_1_response_minutes`, `operations.support_hours`.* + This ensures round-the-clock fault intake and immediate response capabilities across the entire platform foundation without interruption. + +- **OR-2**: Normal platform operations are restored within 4 hours of a Priority 1 incident occurring. + *Source: `service_levels.recovery_time_hours`.* + This establishes the mandatory maximum recovery duration for high-impact platform outages to preserve business continuity. + +- **OR-3**: The platform foundation maintains an operational availability of at least 99.9% across all hours (24x7) of each calendar month. + *Source: `K-1`, `operations.availability_hours`.* + This bounds total allowable platform downtime across the hub network and core infrastructure to maintain continuous landing zone operations. + +### Process requirements + +- **OR-4**: Every change to the hub network requires customer platform team approval prior to deployment. + *Source: `REQ-01`.* + This preserves the customer platform team's governance and authoritative control over network modifications while allowing automated deployment pipelines elsewhere. + +- **OR-5**: A reporting package detailing achieved service levels and operational costs is delivered to the customer on a monthly basis. + *Source: `REQ-02`.* + This ensures operational transparency, providing monthly verification of SLA performance and run expenditure. + +### Organisation requirements + +- **OR-6**: Platform foundation operations are delivered under a shared sourcing model between the customer and an external service provider. + *Source: `sourcing.model`.* + This establishes joint delivery boundaries between the customer's internal engineering team and external partner operations. + +- **OR-7**: Operational services are delivered exclusively from the United Kingdom, Poland, and India. + *Source: `sourcing.delivery_countries`.* + This restricts operational execution, personnel locations, and data access to the customer-approved jurisdictions. + +### Compliance requirements + +- **OR-8**: Audit logs stored in the audit log store cannot be deleted and are retained for at least seven years. + *Source: `K-2`.* + This ensures non-repudiation and regulatory compliance for audit evidence by preventing log deletion across the required 7-year retention period. + +- **OR-9**: Cryptographic encryption keys are rotated at least once every 90 days. + *Source: `K-3`.* + This enforces key lifecycle governance to limit cryptographic exposure across the landing zone. + +--- + +## What was not stated + +The operating specification leaves several operational parameters either assumed or unknown. If answered differently by the customer, these would alter the operational requirements and downstream target operating model design. + +### Assumptions + +Because execution is running in pre-approved mode, the following operational assumptions have been adopted from the scoping phase: + +| Assumption ID | Field | Assumed Value | Impact if Different | +| :--- | :--- | :--- | :--- | +| **ASSUMP-01** | `intent.exclusions` | Application workloads deployed onto the platform foundation are excluded from operations. | If application workloads were included in scope, operational requirements would need to expand to cover application-level SLAs, deployment support, database maintenance, and tenant runtime monitoring. | +| **ASSUMP-02** | `operations.change_process` | Changes go through automated pipelines with peer review; changes affecting the hub network require customer platform team approval prior to release. | If the change process required manual change advisory board (CAB) reviews for all platform changes, release cadence requirements would slow and change review mechanisms would expand. | +| **ASSUMP-03** | `service_management.tooling` | Google Cloud native operations tooling (Cloud Monitoring and Cloud Logging) is used, with no separate external service management system. | If an enterprise ITSM platform (such as ServiceNow or Jira Service Management) were mandated, additional tooling integration and ticket-routing requirements would apply. | + +### Unknowns + +The operating spec identifies several unstated fields that remain to be settled: + +1. **`operations.monitoring` (Monitoring and alerting baseline)**: The customer has not specified pre-existing monitoring dashboards, synthetic probes, or alert routing mechanisms. If proprietary third-party monitoring or customer-managed SIEM integration is required, specific telemetry ingestion requirements must be established. +2. **`solution.inherited_risks` (Inherited risks and unverified controls)**: Unverified architectural dependencies or single points of failure in Cloud Interconnect or Cloud KMS were not stated. Known single points of failure would require additional failover verification requirements. +3. **`solution.build.customer_owned` (Asset ownership during build)**: Exact boundaries between provider-built landing zone templates and customer-managed components were not explicitly split, which could influence operational handover requirements. +4. **`organisation.headcount_limit` (Staffing constraints)**: Headcount caps were not provided. Operational requirements currently reflect service outcomes and availability rather than headcount caps. +5. **`transition.go_live_date` (Service commencement timeline)**: The cutover date from build to live operations is unknown. A compressed transition window would necessitate specific operational readiness acceptance criteria. +6. **Regulatory framework certification (`compliance.frameworks`)**: While the customer noted DORA as an applicable framework, operational frameworks certify nothing; operational capabilities align with DORA resilience and incident response requirements without making formal certification claims. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/target_operating_model.md new file mode 100644 index 0000000..a0b9245 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/target_operating_model.md @@ -0,0 +1,72 @@ +# Target Operating Model + +## Design rationale + +The target operating model establishes a shared sourcing arrangement to securely and cost-effectively operate the cloud platform foundation, balancing out-of-hours coverage against budget limits. + +### Division of work +* **Customer platform team (P-1):** Retains full product ownership, accountability for the platform roadmap, and final approval authority for any changes to the hub network. This ensures the operating model explicitly satisfies the requirement that the internal team retains the final say over the network. +* **External service provider (P-2):** Operates a shared "Platform" team and an "Enabling" team. They hold round-the-clock responsibility for incident response, monitoring, and security telemetry, as well as maintaining the pipeline capabilities. The enablement team interacts with the customer through facilitating modes until the foundational pipelines are successfully adopted. + +### Roles and Locations +Due to strict budgetary constraints (£60,000 per month limit), the geographical placement of the provider roles balances coverage, seniority, and cost: +* **Operations Lead (R-3):** Based in Poland (PL) working business hours. This maintains a leadership footprint closer to the customer's UK time zone to facilitate effective governance and escalation. +* **Operations Engineer (R-4) & Security Operations Engineer (R-5):** Based in India (IN) providing 24x7 cover. Operating a 24x7 rota requires a minimum roster of personnel. Placing these roles in India achieves the required round-the-clock incident response (within 15 minutes) while keeping the heavy cost multipliers of 24x7 staffing comfortably within the budget. +* **Senior SRE (R-6) & Enablement Engineer (R-7):** Based in India (IN) during business hours. Initially drafted in Poland, these senior roles were shifted to India to ensure the total monthly run cost remains below the budget. +* **Customer Roles (R-1, R-2):** Based in the UK (GB). Customer headcount does not contribute to the operational run cost. + +### Checks and Native Capabilities +Previous versions of this design incorrectly relied on architectural findings for standard operational tasks (e.g., configuring alerts). This has been rectified. The operating model configures native Google Cloud tooling (Cloud Monitoring and Cloud Logging) to fulfill these checks internally without requiring external architectural modifications. + +## Assumptions +As this model was generated with pre-approved execution, the following assumptions were made: +* **GCP Tooling:** Operational monitoring, dashboarding, and alerting are entirely achievable within the bounds of the existing native GCP solution components and require no special workarounds. +* **Effort Sizing:** The minimum roster size required to staff a 24x7 position provides sufficient aggregate effort to absorb the estimated monthly capability load without needing additional supplementary FTEs. +* **Client approvals:** The customer has verified that shifting several senior provider engineering roles to India does not violate any unstated data residency or collaboration constraints beyond the explicit list of allowed countries. + +## Responsibility matrix + +| Capability | Platform Lead | Platform Engineer | Operations Lead | Operations Engineer | Security Operations Engineer | Senior SRE | Enablement Engineer | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | C | A | R | | | | +| Monitoring and alerting (C-2) | | C | A | R | | | | +| Problem management (C-3) | | C | A | | | R | | +| Service request fulfilment (C-4) | | C | A | R | | | | +| Change enablement (C-5) | A | R | | | | C | | +| Release and deployment (C-6) | | C | A | | | R | | +| Configuration management (C-7) | | C | A | | | R | | +| Backup and recovery (C-8) | | C | A | | | R | | +| Service continuity (C-9) | | C | A | | | R | | +| Capacity and performance management (C-10) | | C | A | | | R | | +| Access management (C-11) | | C | A | | R | | | +| Vulnerability and patch management (C-12) | | C | A | | | R | | +| Security monitoring and response (C-13) | | C | A | | R | | | +| Cost management (C-14) | C | | A | | | R | | +| Service level management (C-15) | C | | A | | | R | | +| Platform product management (C-16) | A/R | | | | | | | +| Self-service and golden paths (C-17) | | C | A | | | R | | +| Enablement (C-18) | | C | A | | | | R | +| Delivery and developer experience measurement (C-19) | C | | A | | | R | | +| Supplier management (C-20) | A/R | | | | | | | +| Regulatory evidence and reporting (C-21) | | C | A | | | R | | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Lead | Customer platform team | GB | lead | business | 16.0 h | 0.1 | not priced | +| Platform Engineer | Customer platform team | GB | engineer | business | 10.0 h | 0.1 | not priced | +| Operations Lead | External service provider | PL | lead | business | 0.0 h | 0.0 | 0 GBP | +| Operations Engineer | External service provider | IN | engineer | 24x7 | 59.0 h | 5.0 | 17,034 GBP | +| Security Operations Engineer | External service provider | IN | senior | 24x7 | 34.0 h | 5.0 | 24,605 GBP | +| Senior SRE | External service provider | IN | senior | business | 110.0 h | 0.7 | 2,824 GBP | +| Enablement Engineer | External service provider | IN | senior | business | 10.0 h | 0.1 | 257 GBP | + +11.0 people in all. Cost a month: 44,720 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/validation_results.md new file mode 100644 index 0000000..7a26ab4 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r2/deliverables/validation_results.md @@ -0,0 +1,50 @@ +# Check verdicts + +## Passed checks +* **OC-1**: Passed. Capability C-1 covers incident management and operates 24x7. Role R-4 (Operations Engineer) is responsible and operates 24x7. +* **OC-2**: Passed. Capability C-1 operates 24x7 and responds to incidents. +* **OC-3**: Passed. Capability C-2 covers monitoring and operates 24x7, with Role R-4 responsible. The operating model configures native Google Cloud monitoring tools to fulfill this internally. +* **OC-4**: Passed. Capability C-5 covers change enablement. Role R-1 (Platform Lead from the customer team) is accountable and Role R-2 (Platform Engineer from the customer team) is responsible, giving the customer final approval. +* **OC-5**: Passed. Capability C-15 covers service level management. Role R-6 (Senior SRE) is responsible for producing the monthly reporting package. +* **OC-6**: Passed. Capability C-20 covers supplier management to govern the shared sourcing model. Role R-1 (Platform Lead) is accountable and responsible. +* **OC-7**: Passed. Capability C-11 covers access management. Role R-5 (Security Operations Engineer) is responsible. +* **OC-8**: Passed. Capability C-21 covers regulatory evidence collection. Role R-6 (Senior SRE) is responsible. +* **OC-9**: Passed. Role R-3 (Operations Lead) sits in Poland (PL). +* **OC-10**: Passed. Role R-4 (Operations Engineer) sits in India (IN). +* **OC-11**: Passed. Role R-5 (Security Operations Engineer) sits in India (IN). +* **OC-12**: Passed. Role R-6 (Senior SRE) sits in India (IN). +* **OC-13**: Passed. Role R-7 (Enablement Engineer) sits in India (IN). +* **OC-14**: Passed. Role R-1 (Platform Lead) sits in the United Kingdom (GB). +* **OC-15**: Passed. Role R-2 (Platform Engineer) sits in the United Kingdom (GB). + +## Failed checks +None. + +## Not verifiable checks +None. + +## Assumptions +* No further assumptions were made during this validation run. + +## Overall verdict +**Pass**: The target operating model successfully meets all operational requirements and conditions. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-3 on C-2 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-7 on C-11 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-8 on C-21 keeps it true. | +| External service provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 44,720 GBP, within the budget. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/capability_model.md new file mode 100644 index 0000000..6b2b8e1 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/capability_model.md @@ -0,0 +1,109 @@ +# Capability Model + +This capability model outlines what the operating organisation must be able to do to meet the operational requirements. It assigns the capabilities from the standard catalogue to specific items of the platform foundation and estimates steady-state effort. Who specifically performs each capability is not decided here. + +## Capabilities + +| ID | Capability | Coverage (Items) | Operational Requirements | Hours | Monthly Effort (hrs) | Purpose / Need | +|---|---|---|---|---|---|---| +| **C-1** | Incident management | S-1, S-2, S-3, S-4 | OR-1, OR-2 | 24x7 | 30 | Restore degraded services. Needed for fast fault handling. | +| **C-2** | Monitoring and alerting | S-1, S-2, S-3, S-4 | OR-1, OR-3 | 24x7 | 20 | Know the state of components. Needed to detect faults and track uptime. | +| **C-3** | Problem management | S-1, S-2, S-3, S-4 | OR-2 | business | 10 | Find why incidents happen. Needed to prevent major failures. | +| **C-4** | Service request fulfilment | S-1, S-2, S-3, S-4 | OR-6 | business | 20 | Handle user requests. Needed for the shared operating model. | +| **C-5** | Change enablement | S-1, S-2, S-3, S-4 | OR-4, OR-10 | business | 10 | Manage changes. Needed for customer approvals on network changes. | +| **C-6** | Release and deployment | S-1, S-2, S-3, S-4 | OR-10 | business | 20 | Put approved changes into service. Needed to hit the 1-day deployment target. | +| **C-7** | Configuration management | S-1, S-2, S-3, S-4 | OR-8, OR-9 | business | 15 | Track configs. Needed for key rotation and audit log policies. | +| **C-8** | Backup and recovery | S-2, S-4 | OR-2, OR-8 | business | 10 | Keep copies of logs/keys. Needed for recovery and log retention. | +| **C-9** | Service continuity | S-1, S-2, S-3, S-4 | OR-2, OR-3 | business | 10 | Restore services after loss. Needed for the 4-hour recovery SLA. | +| **C-10** | Capacity and performance management | S-1, S-2, S-3, S-4 | OR-3 | business | 10 | Manage limits. Needed for 99.9% uptime. | +| **C-11** | Access management | S-1, S-2, S-3, S-4 | OR-6, OR-7 | business | 15 | Give/revoke access. Needed for cross-country shared model boundaries. | +| **C-12** | Vulnerability and patch management | S-1, S-2, S-3, S-4 | OR-9 | business | 15 | Fix weaknesses. Needed to secure components and rotate keys safely. | +| **C-13** | Security monitoring and response | S-1, S-2, S-3, S-4 | OR-1, OR-8 | 24x7 | 40 | Detect misuse. Needed to protect the audit logs and fast response to events. | +| **C-14** | Cost management | S-1, S-2, S-3, S-4 | OR-5 | business | 10 | Track spend. Needed for the monthly cost report. | +| **C-15** | Service level management | S-1, S-2, S-3, S-4 | OR-3, OR-5 | business | 10 | Measure SLAs. Needed for 99.9% tracking and reporting. | +| **C-16** | Platform product management | S-1, S-2, S-3, S-4 | OR-6 | business | 15 | Manage roadmap based on developer friction. Needed to retain engineers. | +| **C-17** | Self-service and golden paths | S-1, S-2, S-3, S-4 | OR-10 | business | 20 | Automate common tasks. Needed for rapid product delivery. | +| **C-18** | Enablement | S-1, S-2, S-3, S-4 | OR-10 | business | 20 | Help teams adopt the platform. Needed for fast flow of value. | +| **C-19** | Delivery and developer experience measurement | S-1, S-2, S-3, S-4 | OR-10 | business | 10 | Measure flow at all layers. Needed to prove fast change delivery. | +| **C-20** | Supplier management | S-1, S-2, S-3, S-4 | OR-6, OR-7 | business | 10 | Hold suppliers to agreements. Needed for the shared model. | +| **C-21** | Regulatory evidence and reporting | S-1, S-2, S-3, S-4 | OR-8 | business | 10 | Retain compliance evidence. Needed to ensure logs remain undeleted for 7 years. | + +*Note: The rate card is currently in draft. Any cost figures or staffing structures derived from these estimations will act as placeholders.* + +## Assumptions +As this execution is pre-approved, unknown values were accommodated using standard capability assumptions: +- The customer platform team retains network change approvals, and applications running on the platform are out of scope. +- Changes traverse a deployment pipeline utilizing peer review. +- The cloud provider's native operations tooling is employed in lieu of a separate ITSM system. +- Unspecified bounds around headcount limits, monitoring integrations, or customer build components do not currently restrict operational coverage. + +## Sources + +The capabilities within this model draw on concepts from several recognized industry frameworks. These are cited as referenced by the catalogue parts below. + +*Please note: The capability catalogue draws on these frameworks but claims no official certification or compliance against them. All references provided below are currently **unverified** as documented by the standard catalogue register.* + +- **ITIL 4** + - Incident management practice + - Monitoring and event management practice + - Problem management practice + - Service request management practice + - Change enablement practice + - Release management and Deployment management practices + - Service configuration management practice + - Service continuity management practice + - Capacity and performance management practice + - Service level management practice + - Supplier management practice +- **Site Reliability Engineering: How Google Runs Production Systems (Google SRE)** + - Managing Incidents + - Monitoring Distributed Systems + - Postmortem Culture: Learning from Failure + - Release Engineering + - Service Level Objectives +- **ISO/IEC 20000-1** + - Incident management + - Service request management + - Change management + - Service continuity management + - Capacity management + - Service level management + - Supplier management +- **Regulation (EU) 2022/2554, the Digital Operational Resilience Act (EU DORA)** + - Article 9, Protection and prevention + - Article 10, Detection + - Article 11, Response and recovery + - Article 12, Backup policies and procedures, restoration and recovery procedures and methods + - Article 17, ICT-related incident management process + - Article 19, Reporting of major ICT-related incidents + - Article 28, General principles on ICT third-party risk +- **NIST Cybersecurity Framework (NIST CSF 2.0)** + - Identify: Asset Management + - Protect: Identity Management, Authentication, and Access Control + - Protect: Platform Security + - Detect: Continuous Monitoring + - Detect: Adverse Event Analysis; Respond: Incident Management + - Recover: Incident Recovery Plan Execution +- **ISO/IEC 27001** + - Access control + - Management of technical vulnerabilities + - Information security incident management + - Compliance with legal and contractual requirements +- **DevOps Research and Assessment (DORA) software delivery metrics** + - Change failure rate + - Deployment frequency and lead time for changes + - Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- **FinOps Framework** + - Allocation; Budgeting; Forecasting; Anomaly Management +- **CNCF Platforms White Paper (platform as a product)** + - Platform as a product; Measuring platform success + - Platform capabilities; Self-service + - Platform as a product +- **Team Topologies: Organizing Business and Technology Teams for Fast Flow** + - Platform team; Thinnest viable platform + - X-as-a-Service interaction mode + - Enabling team; Facilitating interaction mode +- **The SPACE of Developer Productivity** + - Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- **DevEx: What Actually Drives Productivity** + - Feedback loops; Cognitive load; Flow state \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_model_guide.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_model_guide.md new file mode 100644 index 0000000..8702327 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_model_guide.md @@ -0,0 +1,237 @@ +# Operating model guide: Google Cloud landing zone platform foundation + +## Summary + +This guide sets out the target operating model for the Google Cloud landing zone platform foundation, covering the hub network and interconnect, audit log store, deployment pipeline, and encryption keys. Operational responsibility is shared between the customer and an external service provider across the United Kingdom, Poland, and India. The customer platform team steers platform product direction, developer flow measurement, and retains approval authority over all network changes, while the service provider operations team runs day-to-day operations, maintenance, and incident management. Fault response and platform availability are maintained around the clock (24x7), while change governance, service continuity, and lifecycle management are conducted during business hours. + +Readers should look first at the findings and open questions, particularly the catalogue assumptions regarding cloud-native operations tooling, change pipeline procedures, and the exclusion of workload applications, as well as the draft status of the rate card. All scenario walk-throughs and validation checks pass successfully within the allocated operational budget. + +### Key figures + +- Operated: 4 Items, with 3 Conditions to keep true. +- Faults responded to: 24x7 hours. +- Capabilities: 21. +- People: 11.4. Cost a month: 47,603 GBP, from rate card version `draft-2026-10-13` (placeholder prices: an illustration). +- Scenario walk-throughs: 13 of 13 pass. +- Business outcomes: 3, 3 shown by a Measure. +- Findings for whoever owns the design: 0. + +## Business outcomes + +**BO-1: New payment products reach customers in weeks rather than months.** Target: A change reaches production within a day of approval.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Deployment frequency | outcome | dora-metrics | A change reaches production within a day of approval. | Delivery and developer experience measurement | Platform Product Manager | +| Friction and cognitive load | leading | devex-framework | | Delivery and developer experience measurement | Platform Product Manager | + +**BO-2: Payments are never unavailable for long.** Target: Restored within 4 hours.. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Time to restore service | outcome | dora-metrics | Restored within 4 hours. | Incident management | Operations Lead | + +**BO-3: We keep the platform engineers we hire.** + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Developer satisfaction | guardrail | space-framework | | Delivery and developer experience measurement | Platform Product Manager | + +## What is operated + +| Item | What it is | Kind | Product | From | +| --- | --- | --- | --- | --- | +| S-1 | Hub network and interconnect | network | Cloud Interconnect | | +| S-2 | Audit log store | storage | Cloud Storage | | +| S-3 | Deployment pipeline | pipeline | Cloud Build | | +| S-4 | Encryption keys | keys | Cloud KMS | | + +What must be kept true while it is run: + +| Condition | On | What must hold | From | +| --- | --- | --- | --- | +| K-1 | S-1 | The platform is available at least 99.9% of each month. | | +| K-2 | S-2 | Audit logs cannot be deleted for seven years. | | +| K-3 | S-4 | Encryption keys are rotated every 90 days. | | + +## Operational requirements + +| Requirement | Kind | What must be achieved | From | +| --- | --- | --- | --- | +| OR-1 | service_level | A fault on any Item is acknowledged within 15 minutes, at any hour. | service_levels.priority_1_response_minutes, operations.support_hours | +| OR-2 | service_level | Service is restored within 4 hours of a major failure. | service_levels.recovery_time_hours, BO-2 | +| OR-3 | service_level | The platform is available at least 99.9% of each month. | K-1 | +| OR-4 | process | Every change to the network requires customer platform team approval prior to implementation. | REQ-01 | +| OR-5 | process | A report of service levels and operating costs is provided monthly. | REQ-02 | +| OR-6 | organisation | The platform foundation is operated under a shared service model between the customer and an external service provider. | sourcing.model | +| OR-7 | organisation | Operational delivery is performed exclusively from the United Kingdom, Poland, and India. | sourcing.delivery_countries | +| OR-8 | compliance | Audit logs cannot be deleted for seven years. | K-2 | +| OR-9 | process | Encryption keys are rotated every 90 days. | K-3 | +| OR-10 | service_level | An approved change reaches production within one day of approval. | BO-1 | + +## Capabilities + +| Capability | What it is | Serves | Covers | Hours | Effort a month | +| --- | --- | --- | --- | --- | --- | +| C-1 | Incident management | OR-1, OR-2 | S-1, S-2, S-3, S-4 | 24x7 | 30 h | +| C-2 | Monitoring and alerting | OR-1, OR-3 | S-1, S-2, S-3, S-4 | 24x7 | 20 h | +| C-3 | Problem management | OR-2 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-4 | Service request fulfilment | OR-6 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-5 | Change enablement | OR-4, OR-10 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-6 | Release and deployment | OR-10 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-7 | Configuration management | OR-8, OR-9 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-8 | Backup and recovery | OR-2, OR-8 | S-2, S-4 | business | 10 h | +| C-9 | Service continuity | OR-2, OR-3 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-10 | Capacity and performance management | OR-3 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-11 | Access management | OR-6, OR-7 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-12 | Vulnerability and patch management | OR-9 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-13 | Security monitoring and response | OR-1, OR-8 | S-1, S-2, S-3, S-4 | 24x7 | 40 h | +| C-14 | Cost management | OR-5 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-15 | Service level management | OR-3, OR-5 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-16 | Platform product management | OR-6 | S-1, S-2, S-3, S-4 | business | 15 h | +| C-17 | Self-service and golden paths | OR-10 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-18 | Enablement | OR-10 | S-1, S-2, S-3, S-4 | business | 20 h | +| C-19 | Delivery and developer experience measurement | OR-10 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-20 | Supplier management | OR-6, OR-7 | S-1, S-2, S-3, S-4 | business | 10 h | +| C-21 | Regulatory evidence and reporting | OR-8 | S-1, S-2, S-3, S-4 | business | 10 h | + +## Who does what + +| Role | Party | Country | Level | Hours | +| --- | --- | --- | --- | --- | +| Platform Product Manager | Customer (customer) | GB | lead | business | +| Platform Engineer | Customer (customer) | GB | senior | business | +| Operations Engineer | Service Provider (provider) | IN | engineer | 24x7 | +| Operations Lead | Service Provider (provider) | IN | lead | business | +| Security Operations Analyst | Service Provider (provider) | IN | senior | 24x7 | +| Platform Ops Specialist | Service Provider (provider) | PL | senior | business | + +The teams, as Team Topologies names their kinds: + +| Team | Kind | Party | Roles | Purpose | +| --- | --- | --- | --- | --- | +| Customer Platform Team | platform | Customer | Platform Product Manager, Platform Engineer | Owns the platform roadmap, retains network change approvals, and measures developer flow. | +| Provider Operations Team | platform | Service Provider | Operations Engineer, Operations Lead, Security Operations Analyst, Platform Ops Specialist | Provides 24x7 monitoring, incident response, and day-to-day operations of the platform foundation. | + +How the teams work together: + +| From | To | Mode | Until | Purpose | +| --- | --- | --- | --- | --- | +| Provider Operations Team | Customer Platform Team | x-as-a-service | | Provider requests network change approvals from the Customer Platform Team. | +| Customer Platform Team | Provider Operations Team | x-as-a-service | | Customer relies on Provider Operations Team for 24x7 incident response and monitoring. | +| Provider Operations Team | Customer Platform Team | collaboration | Month 3 after go-live, shifting to x-as-a-service. | Handover of runbooks and initial operational processes. | + +| Capability | Platform Product Manager | Platform Engineer | Operations Engineer | Operations Lead | Security Operations Analyst | Platform Ops Specialist | +| --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | C | R | A | | | +| Monitoring and alerting (C-2) | | C | R | A | | | +| Problem management (C-3) | | C | | A | | R | +| Service request fulfilment (C-4) | | C | R | A | | | +| Change enablement (C-5) | | A/R | | | | R | +| Release and deployment (C-6) | | A/R | | | | R | +| Configuration management (C-7) | | C | | A | | R | +| Backup and recovery (C-8) | | C | | A | | R | +| Service continuity (C-9) | | C | | A | | R | +| Capacity and performance management (C-10) | | C | | A/R | | | +| Access management (C-11) | | C | | A/R | | | +| Vulnerability and patch management (C-12) | | C | | | A/R | | +| Security monitoring and response (C-13) | | C | | | A/R | | +| Cost management (C-14) | C | | | A/R | | | +| Service level management (C-15) | C | | | A/R | | | +| Platform product management (C-16) | A/R | | | C | | | +| Self-service and golden paths (C-17) | | A/R | | | | R | +| Enablement (C-18) | | A/R | | | | R | +| Delivery and developer experience measurement (C-19) | A/R | | | C | | | +| Supplier management (C-20) | A/R | | | R | | | +| Regulatory evidence and reporting (C-21) | | C | | A | | R | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Product Manager | Customer | GB | lead | business | 30.0 h | 0.2 | not priced | +| Platform Engineer | Customer | GB | senior | business | 35.0 h | 0.3 | not priced | +| Operations Engineer | Service Provider | IN | engineer | 24x7 | 70.0 h | 5.0 | 17,034 GBP | +| Operations Lead | Service Provider | IN | lead | business | 50.0 h | 0.3 | 1,876 GBP | +| Security Operations Analyst | Service Provider | IN | senior | 24x7 | 55.0 h | 5.0 | 24,605 GBP | +| Platform Ops Specialist | Service Provider | PL | senior | business | 90.0 h | 0.6 | 4,088 GBP | + +11.4 people in all. Cost a month: 47,603 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. + +## Validation + +| Check | What must hold | Verdict | Evidence | +| --- | --- | --- | --- | +| OC-1 | A fault on the hub network or any other item is acknowledged within 15 minutes at any hour. | pass | C-1 handles incident management 24x7, with a 24x7 Operations Engineer responsible, enabling acknowledgement of faults within 15 minutes at any hour. The cloud provider's native tooling (via C-2) supports configuring alerts without requiring a separate finding. | +| OC-2 | Service is restored within 4 hours of a major failure on the platform. | pass | C-1 manages incident resolution with a 24x7 Operations Engineer responsible, allowing restoration of service within 4 hours of a major failure regardless of when the failure occurs. | +| OC-3 | The platform availability is tracked to ensure at least 99.9% uptime each month. | pass | C-15 tracks and gathers metrics on availability to ensure the 99.9% target is tracked, with business-hours roles accountable and responsible. | +| OC-4 | Capacity limits for the platform interconnect are managed to support 99.9% availability. | pass | C-10 reviews capacity trends for the interconnect to assure 99.9% availability, managed by the Operations Lead. | +| OC-5 | Every change to the hub network requires explicit approval from the customer platform team. | pass | C-5 coordinates network change approvals with the customer, and a Customer Platform Engineer is accountable and responsible for it, ensuring explicit approval from the customer platform team. | +| OC-6 | A consolidated monthly report includes service levels and operating costs. | pass | C-15 compiles the monthly operational report, pulling in cost data tracked by C-14, both managed by the Operations Lead. | +| OC-7 | The shared service model responsibilities between customer and external service provider are governed. | pass | C-20 coordinates responsibilities across shared delivery borders, with roles from both the customer and provider teams responsible. | +| OC-8a | Role performs operational delivery from India. | pass | R-3 (Operations Engineer) is based in India, satisfying the geographical requirement. | +| OC-8b | Role performs operational delivery from India. | pass | R-4 (Operations Lead) is based in India, satisfying the geographical requirement. | +| OC-8c | Role performs operational delivery from India. | pass | R-5 (Security Operations Analyst) is based in India, satisfying the geographical requirement. | +| OC-8d | Role performs operational delivery from Poland. | pass | R-6 (Platform Ops Specialist) is based in Poland, satisfying the geographical requirement. | +| OC-8e | Role performs operational delivery from the UK. | pass | R-1 (Platform Product Manager) is based in the UK, satisfying the geographical requirement. | +| OC-8f | Role performs operational delivery from the UK. | pass | R-2 (Platform Engineer) is based in the UK, satisfying the geographical requirement. | +| OC-9 | Audit log storage configuration ensures logs cannot be deleted for seven years. | pass | C-7 maintains audit log retention settings, with the Platform Ops Specialist responsible, ensuring logs cannot be deleted for seven years. | +| OC-10 | Retention policies on Cloud Storage for audit logs are verified to prevent deletion for seven years. | pass | C-7 maintains and verifies retention settings on Cloud Storage for audit logs, satisfying the seven-year retention condition. | +| OC-11 | Encryption key configurations are audited to ensure rotation every 90 days. | pass | C-7 audits configuration drift and manages key rotation schedules, ensuring encryption keys are rotated every 90 days. | +| OC-12 | Cloud KMS encryption keys are verified to rotate every 90 days. | pass | C-7 manages key rotation schedules for the Cloud KMS encryption keys, verifying they rotate every 90 days. | +| OC-13 | Deployment pipelines ensure an approved change reaches production within one day. | pass | C-6 oversees deployment pipelines to ensure approved changes reach production within a day, with customer and provider roles responsible. | +| OC-14 | Backups of audit logs and encryption keys are successfully tested for restoration. | pass | C-8 maintains backups of logs and keys and successfully tests restore procedures, managed by the Platform Ops Specialist. | + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-4 on C-10 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-10 on C-7 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-12 on C-7 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 47,603 GBP, within the budget. | + +## Findings and open questions + +No findings: the solution provides what operating it needs. + +Assumed for the customer, and open to change when they answer: + +- `intent.exclusions`: Workload applications running on the platform foundation, Network change approvals (retained by customer platform team). Assumed from the platform foundation scope: workload applications running on the platform are out of scope, and customer platform team retains network change approvals. +- `operations.change_process`: Changes go through a deployment pipeline with peer review, with network changes requiring customer platform team approval.. Catalogue default adapted to customer requirement REQ-01: changes go through deployment pipelines with peer review, with customer platform team retaining approval over network changes. +- `service_management.tooling`: The Cloud provider's own operations tooling, with no separate system.. Catalogue default: Cloud provider's native operations tooling with no separate system. + +## Sources + +The Capabilities are from our own capability catalogue, which draws on these frameworks. It cites them and certifies nothing against them. + +- ITIL 4: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice +- Site Reliability Engineering: How Google Runs Production Systems: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives +- ISO/IEC 20000-1: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management +- Regulation (EU) 2022/2554, the Digital Operational Resilience Act: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents +- NIST Cybersecurity Framework: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management +- DevOps Research and Assessment (DORA) software delivery metrics: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- ISO/IEC 27001: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements +- FinOps Framework: Allocation; Budgeting; Forecasting; Anomaly Management +- CNCF Platforms White Paper (platform as a product): Platform as a product; Measuring platform success; Platform capabilities; Self-service; Platform as a product +- Team Topologies: Organizing Business and Technology Teams for Fast Flow: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode +- The SPACE of Developer Productivity: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- DevEx: What Actually Drives Productivity: Feedback loops; Cognitive load; Flow state + +**These references have not been checked against the frameworks by a person.** Treat each as a pointer to where to look. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_scope.md new file mode 100644 index 0000000..544764c --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operating_scope.md @@ -0,0 +1,39 @@ +# Scope statement + +This operating scope covers the Google Cloud landing zone platform foundation, comprising the hub network and interconnect, audit log store, deployment pipeline, and encryption keys. The platform is operated under a shared service model jointly by a service provider delivering from the United Kingdom, Poland, and India, and the customer's platform team based in the United Kingdom. Fault response support is provided 24x7, aligning with continuous 24x7 platform availability. All operating costs are stated in GBP. Out of scope for provider operations are workload applications hosted on the platform foundation and final approval authority for network changes, which is retained by the customer platform team. + +## Information register + +20 stated by the customer, 0 answered in conversation, 3 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the scope spec | +| intent.statement | We want our landing zone run as a shared service from go-live. A provider should respond to faults at every hour, and our own platform team keeps the final say over changes to the network. | the scope spec | +| intent.requirements | [{"id": "REQ-01", "statement": "Our platform team approves every change to the network."}, {"id": "REQ-02", "statement": "We receive a monthly report of service levels and cost."}] | the scope spec | +| intent.business_outcomes | [{"id": "BO-1", "outcome": "New payment products reach customers in weeks rather than months.", "target": "A change reaches production within a day of approval."}, {"id": "BO-2", "outcome": "Payments | the scope spec | +| solution.cloud_provider | gcp | the scope spec | +| solution.items | [{"id": "S-1", "name": "Hub network and interconnect", "kind": "network", "product": "Cloud Interconnect"}, {"id": "S-2", "name": "Audit log store", "kind": "storage", "product": "Cloud Storage"}, {"i | the scope spec | +| solution.conditions | [{"id": "K-1", "on": ["S-1"], "condition": "The platform is available at least 99.9% of each month."}, {"id": "K-2", "on": ["S-2"], "condition": "Audit logs cannot be deleted for seven years."}, {"id" | the scope spec | +| solution.build.duration_days | 36 | the scope spec | +| operations.support_hours | 24x7 | the scope spec | +| operations.availability_hours | 24x7 | the scope spec | +| sourcing.model | shared | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| organisation.existing_teams | [{"name": "Platform team", "party": "customer", "hours": "business", "country": "GB"}] | the scope spec | +| service_levels.priority_1_response_minutes | 15 | the scope spec | +| service_levels.recovery_time_hours | 4 | the scope spec | +| compliance.frameworks | ["DORA"] | the scope spec | +| commercial.currency | GBP | the scope spec | +| commercial.monthly_run_budget | 60000 | the scope spec | +| intent.exclusions | ["Workload applications running on the platform foundation", "Network change approvals (retained by customer platform team)"] | assumed | +| operations.change_process | Changes go through a deployment pipeline with peer review, with network changes requiring customer platform team approval. | assumed | +| service_management.tooling | The Cloud provider's own operations tooling, with no separate system. | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.exclusions is ['Workload applications running on the platform foundation', 'Network change approvals (retained by customer platform team)']. Assumed from the platform foundation scope: workload applications running on the platform are out of scope, and customer platform team retains network change approvals. | Changes which Capabilities are designed. | +| ASSUMP-02 | Assumed operations.change_process is Changes go through a deployment pipeline with peer review, with network changes requiring customer platform team approval.. Catalogue default adapted to customer requirement REQ-01: changes go through deployment pipelines with peer review, with customer platform team retaining approval over network changes. | Changes change enablement and release. | +| ASSUMP-03 | Assumed service_management.tooling is The Cloud provider's own operations tooling, with no separate system.. Catalogue default: Cloud provider's native operations tooling with no separate system. | Changes tooling and the processes written for it. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operational_requirements.md new file mode 100644 index 0000000..7fae2fd --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/operational_requirements.md @@ -0,0 +1,100 @@ +# Operational Requirements + +This document defines what running the platform foundation solution must achieve. Each operational requirement is traced directly to customer statements, solution conditions, business outcomes, or binding fields settled within the operating specification. Requirements state what must be achieved without prescribing capabilities, roles, or tools. + +## Operational requirements + +The operational requirements are grouped below by kind: service level, process, organisation, and compliance. + +### Service Level Requirements + +- **OR-1 (Fault acknowledgement response)**: A fault on any Item is acknowledged within 15 minutes, at any hour. + - *Source*: `service_levels.priority_1_response_minutes`, `operations.support_hours` +- **OR-2 (Service restoration time)**: Service is restored within 4 hours of a major failure. + - *Source*: `service_levels.recovery_time_hours`, `BO-2` +- **OR-3 (Platform availability)**: The platform is available at least 99.9% of each month. + - *Source*: `K-1` +- **OR-10 (Change delivery turnaround)**: An approved change reaches production within one day of approval. + - *Source*: `BO-1` + +### Process Requirements + +- **OR-4 (Network change governance)**: Every change to the network requires customer platform team approval prior to implementation. + - *Source*: `REQ-01` +- **OR-5 (Monthly operational reporting)**: A report of service levels and operating costs is provided monthly. + - *Source*: `REQ-02` +- **OR-9 (Cryptographic key lifecycle)**: Encryption keys are rotated every 90 days. + - *Source*: `K-3` + +### Organisation Requirements + +- **OR-6 (Shared operating model)**: The platform foundation is operated under a shared service model between the customer and an external service provider. + - *Source*: `sourcing.model` +- **OR-7 (Delivery geographic boundaries)**: Operational delivery is performed exclusively from the United Kingdom, Poland, and India. + - *Source*: `sourcing.delivery_countries` + +### Compliance Requirements + +- **OR-8 (Audit log retention)**: Audit logs cannot be deleted for seven years. + - *Source*: `K-2` + +### Requirements Traceability Summary + +| Requirement ID | Kind | Operational Requirement Statement | Source References | +| --- | --- | --- | --- | +| **OR-1** | `service_level` | A fault on any Item is acknowledged within 15 minutes, at any hour. | `service_levels.priority_1_response_minutes`, `operations.support_hours` | +| **OR-2** | `service_level` | Service is restored within 4 hours of a major failure. | `service_levels.recovery_time_hours`, `BO-2` | +| **OR-3** | `service_level` | The platform is available at least 99.9% of each month. | `K-1` | +| **OR-4** | `process` | Every change to the network requires customer platform team approval prior to implementation. | `REQ-01` | +| **OR-5** | `process` | A report of service levels and operating costs is provided monthly. | `REQ-02` | +| **OR-6** | `organisation` | The platform foundation is operated under a shared service model between the customer and an external service provider. | `sourcing.model` | +| **OR-7** | `organisation` | Operational delivery is performed exclusively from the United Kingdom, Poland, and India. | `sourcing.delivery_countries` | +| **OR-8** | `compliance` | Audit logs cannot be deleted for seven years. | `K-2` | +| **OR-9** | `process` | Encryption keys are rotated every 90 days. | `K-3` | +| **OR-10** | `service_level` | An approved change reaches production within one day of approval. | `BO-1` | + +--- + +## What was not stated + +The operating specification leaves several areas assumed or unknown. If answered differently, these would alter the operational requirements. + +### Assumptions + +Because execution is pre-approved, the following assumptions were recorded from the operating specification and catalogue defaults: + +1. **Assumed scope exclusions (`intent.exclusions`)**: + - *Assumption*: Workload applications hosted on the platform foundation are out of scope for provider operations, and network change approvals are retained by the customer platform team. + - *Impact if wrong*: If application workloads were in scope, additional operational requirements for application-layer monitoring, incident triage, deployment coordination, and service-level objectives would be required. If the provider had delegated authority for network modifications, OR-4 would need to be re-scoped. + +2. **Assumed change release mechanism (`operations.change_process`)**: + - *Assumption*: Changes proceed through automated deployment pipelines with peer review, with network changes requiring explicit customer platform team approval. + - *Impact if wrong*: If a formal change advisory board (CAB) or manual approval cadence is mandated for non-network components, change lead time requirements (OR-10) would be affected. + +3. **Assumed operations tooling (`service_management.tooling`)**: + - *Assumption*: The cloud provider's native operations tooling is utilized without a separate third-party IT service management (ITSM) system. + - *Impact if wrong*: If an external enterprise ITSM platform (e.g., ServiceNow, Jira Service Desk) or dedicated paging tool is mandated, additional requirements for cross-platform integration, ticket synchronization, and external reporting would emerge. + +### Unknowns and Open Questions + +The operating spec identifies the following unknown fields: + +1. **Existing monitoring and alerting (`operations.monitoring`)**: + - *Current status*: Unknown. + - *Impact if answered*: Existing monitoring configurations and alert thresholds could impose specific operational requirements regarding synthetic checks, telemetry ingestion rates, or incident trigger definitions. + +2. **Customer-owned build components (`solution.build.customer_owned`)**: + - *Current status*: Unknown. + - *Impact if answered*: If specific infrastructure components or pipelines are managed or maintained by customer internal teams, operational boundaries and fault hand-off requirements would need specification. + +3. **Inherited solution risks (`solution.inherited_risks`)**: + - *Current status*: Unknown. + - *Impact if answered*: Known risks in interconnect redundancy or key distribution would require explicit contingency and high-availability operational requirements. + +4. **Headcount constraints (`organisation.headcount_limit`)**: + - *Current status*: Unknown. + - *Impact if answered*: A hard headcount cap would constrain how 24x7 coverage is staffed across the three delivery countries (GB, PL, IN). + +5. **Operational transition go-live date (`transition.go_live_date`)**: + - *Current status*: Unknown. + - *Impact if answered*: A defined target date would introduce operational readiness milestones and timeline-bound acceptance criteria. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/target_operating_model.md new file mode 100644 index 0000000..3545fed --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/target_operating_model.md @@ -0,0 +1,87 @@ +# Target operating model + +## Design rationale + +The operating model is designed around a shared responsibility between the customer and an external service provider, maintaining a clear boundary while enabling fast flow and 24x7 operational coverage. The approach maps operational capabilities to defined roles, ensures proper coverage for the declared service levels, and incorporates measurement feedback loops to achieve the stated business outcomes. + +### Parties and Teams + +Two primary parties are defined in the operating model: the **Customer** and the **Service Provider**. +To align with Team Topologies, we defined two platform-focused teams: +- The **Customer Platform Team** (Customer) focuses on platform evolution, owning the platform roadmap, defining golden paths, and measuring developer flow. This team acts as the platform owner and retains authority over network change approvals. +- The **Provider Operations Team** (Service Provider) is responsible for the day-to-day running of the platform foundation, including 24x7 monitoring, incident response, and executing operational tasks such as backups and patching. + +### Roles, Location, and Capability Coverage + +Roles are situated in the approved delivery countries (UK, Poland, and India): +- The **Platform Product Manager** (Lead) and **Platform Engineer** (Senior) are situated in the UK, closely collaborating with the customer's business and engineering teams during business hours. They steer the platform, ensure adherence to the 1-day deployment targets, and maintain alignment with business goals. +- The **Operations Engineer** and **Security Operations Analyst** provide 24x7 coverage from India, ensuring rapid response to faults and security events at all hours. They form the backbone of the provider's support roster. +- The **Operations Lead** (India) provides business-hours guidance and accountability for the operational capabilities, ensuring adherence to the 99.9% uptime target. +- The **Platform Ops Specialist** sits in Poland, offering advanced business-hours support, handling problem management, configuration management, and disaster recovery preparations. + +For any capabilities where the provider is accountable, a customer role is marked as consulted to ensure there is always a contact point if the provider requires customer guidance. This distribution balances the need for continuous coverage with efficient use of resources across the specified delivery regions. + +*(Note: Costs and exact headcounts are derived from the rate card. Because the rate card is currently in draft, any financial and staffing figures resulting from this model will act as placeholders.)* + +### Interactions + +The Customer Platform Team and Provider Operations Team interact primarily via an **X-as-a-Service** mode. The Provider consumes network change approvals from the Customer, while the Customer relies on the Provider for 24x7 incident response. During the first three months post-go-live, the teams will engage in a **Collaboration** to ensure a smooth handover of runbooks and establish operational processes before shifting entirely to X-as-a-Service. + +### Measurement + +To ensure the model is serving the business effectively, measurements are tied to the required outcomes. Since the model employs delivery and developer experience measurement (C-19), all three core layers of flow are measured: +- **Outcome** is measured by deployment frequency (using DORA metrics) to ensure changes reach production within a day, and time to restore service. +- **Guardrail** metrics measure developer satisfaction (using the SPACE framework) to ensure retention of platform engineers. +- **Leading signals** track friction and cognitive load (using the DevEx framework) to identify early friction that could impede the target of weeks rather than months for new payment products. + +### Assumptions + +As this execution is pre-approved, the following assumptions were made: +- Native operational tooling from the cloud provider is sufficient to configure necessary alerts, monitors, and verify restoration of backups or logs without needing separate systems, manual design-arounds, or findings. +- The customer platform team has sufficient capacity to process network change approvals within the timeframes needed to meet deployment targets. + +## Responsibility matrix + +| Capability | Platform Product Manager | Platform Engineer | Operations Engineer | Operations Lead | Security Operations Analyst | Platform Ops Specialist | +| --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | C | R | A | | | +| Monitoring and alerting (C-2) | | C | R | A | | | +| Problem management (C-3) | | C | | A | | R | +| Service request fulfilment (C-4) | | C | R | A | | | +| Change enablement (C-5) | | A/R | | | | R | +| Release and deployment (C-6) | | A/R | | | | R | +| Configuration management (C-7) | | C | | A | | R | +| Backup and recovery (C-8) | | C | | A | | R | +| Service continuity (C-9) | | C | | A | | R | +| Capacity and performance management (C-10) | | C | | A/R | | | +| Access management (C-11) | | C | | A/R | | | +| Vulnerability and patch management (C-12) | | C | | | A/R | | +| Security monitoring and response (C-13) | | C | | | A/R | | +| Cost management (C-14) | C | | | A/R | | | +| Service level management (C-15) | C | | | A/R | | | +| Platform product management (C-16) | A/R | | | C | | | +| Self-service and golden paths (C-17) | | A/R | | | | R | +| Enablement (C-18) | | A/R | | | | R | +| Delivery and developer experience measurement (C-19) | A/R | | | C | | | +| Supplier management (C-20) | A/R | | | R | | | +| Regulatory evidence and reporting (C-21) | | C | | A | | R | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Product Manager | Customer | GB | lead | business | 30.0 h | 0.2 | not priced | +| Platform Engineer | Customer | GB | senior | business | 35.0 h | 0.3 | not priced | +| Operations Engineer | Service Provider | IN | engineer | 24x7 | 70.0 h | 5.0 | 17,034 GBP | +| Operations Lead | Service Provider | IN | lead | business | 50.0 h | 0.3 | 1,876 GBP | +| Security Operations Analyst | Service Provider | IN | senior | 24x7 | 55.0 h | 5.0 | 24,605 GBP | +| Platform Ops Specialist | Service Provider | PL | senior | business | 90.0 h | 0.6 | 4,088 GBP | + +11.4 people in all. Cost a month: 47,603 GBP. +The monthly budget is 60,000 GBP: this is within it. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/validation_results.md new file mode 100644 index 0000000..6f1049e --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/bench-landing-zone-shared/executions/case-01-landing-zone-shared-r3/deliverables/validation_results.md @@ -0,0 +1,47 @@ +# Validation results + +## Check verdicts + +The target operating model has been evaluated against the operational requirements, conditions, and business outcomes. Below are the verdicts for each check: + +- **OC-1 (pass):** C-1 handles incident management 24x7, with a 24x7 Operations Engineer responsible, enabling acknowledgement of faults within 15 minutes at any hour. The cloud provider's native tooling (via C-2) supports configuring alerts without requiring a separate finding. +- **OC-2 (pass):** C-1 manages incident resolution with a 24x7 Operations Engineer responsible, allowing restoration of service within 4 hours of a major failure regardless of when the failure occurs. +- **OC-3 (pass):** C-15 tracks and gathers metrics on availability to ensure the 99.9% target is tracked, with business-hours roles accountable and responsible. +- **OC-4 (pass):** C-10 reviews capacity trends for the interconnect to assure 99.9% availability, managed by the Operations Lead. +- **OC-5 (pass):** C-5 coordinates network change approvals with the customer, and a Customer Platform Engineer is accountable and responsible for it, ensuring explicit approval from the customer platform team. +- **OC-6 (pass):** C-15 compiles the monthly operational report, pulling in cost data tracked by C-14, both managed by the Operations Lead. +- **OC-7 (pass):** C-20 coordinates responsibilities across shared delivery borders, with roles from both the customer and provider teams responsible. +- **OC-8a (pass):** R-3 (Operations Engineer) is based in India, satisfying the geographical requirement. +- **OC-8b (pass):** R-4 (Operations Lead) is based in India, satisfying the geographical requirement. +- **OC-8c (pass):** R-5 (Security Operations Analyst) is based in India, satisfying the geographical requirement. +- **OC-8d (pass):** R-6 (Platform Ops Specialist) is based in Poland, satisfying the geographical requirement. +- **OC-8e (pass):** R-1 (Platform Product Manager) is based in the UK, satisfying the geographical requirement. +- **OC-8f (pass):** R-2 (Platform Engineer) is based in the UK, satisfying the geographical requirement. +- **OC-9 (pass):** C-7 maintains audit log retention settings, with the Platform Ops Specialist responsible, ensuring logs cannot be deleted for seven years. +- **OC-10 (pass):** C-7 maintains and verifies retention settings on Cloud Storage for audit logs, satisfying the seven-year retention condition. +- **OC-11 (pass):** C-7 audits configuration drift and manages key rotation schedules, ensuring encryption keys are rotated every 90 days. +- **OC-12 (pass):** C-7 manages key rotation schedules for the Cloud KMS encryption keys, verifying they rotate every 90 days. +- **OC-13 (pass):** C-6 oversees deployment pipelines to ensure approved changes reach production within a day, with customer and provider roles responsible. +- **OC-14 (pass):** C-8 maintains backups of logs and keys and successfully tests restore procedures, managed by the Platform Ops Specialist. + +**Overall Verdict:** The target operating model **passes** validation. The roles, coverage hours, and capabilities are fully aligned with the operational requirements and the designated shared service sourcing model. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Hub network and interconnect (S-1) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Hub network and interconnect (S-1) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Audit log store (S-2) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Audit log store (S-2) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Deployment pipeline (S-3) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Deployment pipeline (S-3) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Encryption keys (S-4) fails at an hour faults are responded to (24x7). | pass | Operations Engineer responds under C-1, and Operations Lead is answerable. | +| Encryption keys (S-4) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The platform is available at least 99.9% of each month. | pass | Check OC-4 on C-10 keeps it true. | +| Condition K-2 must be shown to hold: Audit logs cannot be deleted for seven years. | pass | Check OC-10 on C-7 keeps it true. | +| Condition K-3 must be shown to hold: Encryption keys are rotated every 90 days. | pass | Check OC-12 on C-7 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| A month's cost of operating is compared with the budget of 60,000 GBP. | pass | The cost is 47,603 GBP, within the budget. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/capability_model.md new file mode 100644 index 0000000..3d20c54 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/capability_model.md @@ -0,0 +1,51 @@ +# Capabilities + +The capability model defines what the operating organisation must be able to do to meet the operational requirements. These capabilities are drawn directly from the capability catalogue based on the platform foundation's constraints and conditions. + +## Assumptions +Because the execution is running under pre-approved conditions without user input, we have assumed that all proposed catalogue entries are required for a complete foundation operations model. We have also assumed that alerting and incident triage functions operate on a 24x7 basis, mapping directly to the 24x7 operational requirements for priority incidents. + +## Capabilities + +| Id | Capability | Covers Items | Serves | Why it is needed | +|---|---|---|---|---| +| C-1 | Incident management | S-1 to S-9 | OR-1, OR-6 | Required to provide 24x7 rapid triage and response to priority incidents, minimizing downtime for the payments engine. | +| C-2 | Monitoring and alerting | S-1 to S-9 | OR-6, OR-15 | Required to integrate telemetry and ensure alerts immediately trigger operational workflows. | +| C-3 | Problem management | S-1 to S-9 | OR-2 | Needed to investigate root causes of anomalies and ensure the 99.9% platform availability target is maintained long-term. | +| C-4 | Service request fulfilment | S-1 to S-9 | OR-8 | Necessary for handling non-standard provisioning requests to ensure readiness for core payments engine workloads. | +| C-5 | Change enablement | S-1 to S-9 | OR-12 | Enables the secure review and automated pipeline approval of all infrastructure-as-code changes without manual CAB bottlenecks. | +| C-6 | Release and deployment | S-1 to S-9 | OR-10, OR-12 | Manages automated delivery pipelines and optimizes resource deployments to maximize regional carbon-free energy utilization. | +| C-7 | Configuration management | S-1 to S-9 | OR-16 | Maintains version-controlled state and prevents drift from DORA, PCI-DSS v4.0, and ISO27001 baselines. | +| C-8 | Backup and recovery | S-1 to S-9 | OR-4 | Monitors automated backups and tests restores to ensure a 1-hour Recovery Point Objective (RPO) is met. | +| C-9 | Service continuity | S-1 to S-9 | OR-3, OR-7 | Rehearses disaster recovery and maintains the active warm standby in `europe-west1` to receive immediate failover traffic within a 4-hour RTO. | +| C-10 | Capacity and performance | S-1 to S-9 | OR-5 | Needed to oversee 10Gbps interconnect bandwidth and ensure deterministic low-latency hybrid performance. | +| C-11 | Access management | S-1 to S-9 | OR-11, OR-14 | Enforces least-privilege rightsizing via Recommender and restricts administrative access to the UK, Poland, and India. | +| C-12 | Vulnerability and patch management | S-1 to S-9 | OR-16 | Secures the landing zone against emerging threats by scanning for vulnerabilities and maintaining compliance with PCI-DSS v4.0. | +| C-13 | Security monitoring and response | S-1 to S-9 | OR-1, OR-6 | Provides continuous 24x7 detection and response to security events protecting payment processing infrastructure. | +| C-14 | Cost management | S-1 to S-9 | OR-9 | Proactively tracks run-rate expenditure against budget allocations to prevent unexpected cloud foundation cost overruns. | +| C-15 | Service level management | S-1 to S-9 | OR-2 | Compiles and verifies operational reports against the mandatory 99.9% availability service level commitment. | +| C-16 | Platform product management | S-1 to S-9 | OR-8 | Ensures the platform roadmap prioritizes removing friction for payment engine development teams, driving adoption within 10 weeks. | +| C-17 | Self-service and golden paths | S-1 to S-9 | OR-8 | Packages complex foundation patterns into paved self-service paths, reducing cognitive load for application teams. | +| C-18 | Enablement | S-1 to S-9 | OR-8 | Assists application teams through direct pairing and guidance to rapidly onboard them to the Google Cloud landing zone. | +| C-19 | Delivery and developer experience measurement | S-1 to S-9 | OR-12 | Collects DORA metrics for pipeline performance alongside guardrail well-being measures to identify leading signals of friction. | +| C-20 | Supplier management | S-1 to S-9 | OR-13 | Oversees managed service delivery, Google Cloud support tiers, and interconnect carrier relationships. | +| C-21 | Regulatory evidence | S-1 to S-9 | OR-16, OR-17 | Collates continuous audit evidence for DORA/PCI-DSS and ensures UK/EU data residency guarantees are proven. | + +# Sources + +The capability catalogue draws upon the following industry frameworks. + +*Note: References have been mapped from general framework knowledge. As stated in the catalogue's register, these references are currently **unverified**. The operating model does not claim compliance with or certification against any of these frameworks.* + +- **ITIL 4**: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Supplier management practice; Service level management practice. +- **Site Reliability Engineering: How Google Runs Production Systems**: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives. +- **ISO/IEC 20000-1**: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management. +- **NIST Cybersecurity Framework 2.0**: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management. +- **ISO/IEC 27001**: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements. +- **Regulation (EU) 2022/2554, the Digital Operational Resilience Act**: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents. +- **FinOps Framework**: Allocation; Budgeting; Forecasting; Anomaly Management. +- **DevOps Research and Assessment (DORA) software delivery metrics**: Change failure rate; Deployment frequency and lead time for changes; Time to restore service. +- **Team Topologies: Organizing Business and Technology Teams for Fast Flow**: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode. +- **CNCF Platforms White Paper (platform as a product)**: Platform as a product; Measuring platform success; Platform capabilities; Self-service. +- **The SPACE of Developer Productivity**: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow. +- **DevEx: What Actually Drives Productivity**: Feedback loops; Cognitive load; Flow state. \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operating_scope.md new file mode 100644 index 0000000..3e146cd --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operating_scope.md @@ -0,0 +1,42 @@ +# Scope statement + +The scope of operations covers the enterprise Google Cloud platform foundation and landing zone, comprising the resource hierarchy and environment boundaries, dedicated hybrid cloud interconnectivity, centralised outbound inspection appliances, immutable compliance logging, automated deployment orchestration, federated identity governance, cryptographic key lifecycle management, operational telemetry, and financial budget monitoring. Operations are conducted under a provider-operated sourcing model by service provider teams located across the United Kingdom, Poland, and India. Platform operations and fault response are delivered continuously on a 24x7 basis to support 24x7 platform availability. All operational costs for this engagement are stated in British Pounds (GBP). Explicitly excluded from operational scope are core payment engine application code and workload business logic, physical on-premises data centre hosting and hardware, and corporate Entra ID identity lifecycle administration beyond Google Cloud federation. + +## Information register + +3 stated by the customer, 13 digested from the sources supplied, 0 answered in conversation, 7 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the sources supplied | +| intent.statement | We are a UK-regulated payment platform expanding into Google Cloud. We need an enterprise-grade landing zone ready in under 10 weeks to support our core payments engine. All payment processing must co | the sources supplied | +| solution.cloud_provider | gcp | the sources supplied | +| solution.items | [{"id": "S-1", "name": "Resource Hierarchy and Environment Boundary Manager", "kind": "governance", "product": "Resource Manager", "source_ref": "C-1", "elements": ["E-1", "E-2", "E-3", "E-4", "E-5", | the sources supplied | +| solution.conditions | [{"id": "K-1", "on": ["S-1", "S-4", "S-7"], "condition": "The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during extern | the sources supplied | +| solution.build.duration_days | 36 | the sources supplied | +| solution.build.customer_owned | ["Order and provision the Interconnect carrier circuits"] | the sources supplied | +| operations.support_hours | 24x7 | the sources supplied | +| operations.availability_hours | 24x7 | the sources supplied | +| sourcing.model | provider_operated | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| compliance.frameworks | ["DORA", "ISO27001", "PCI_DSS_v4"] | the sources supplied | +| commercial.currency | GBP | the sources supplied | +| intent.business_outcomes | [{"outcome": "Enterprise landing zone ready to ingest core payments engine workloads within 10 weeks", "id": "BO-1", "target": "Workload ready within 10 weeks"}, {"outcome": "Continuous operational re | assumed | +| intent.exclusions | ["Application workload code, payment engine business logic, and database-level transaction data management", "Physical data centre hosting, hardware maintenance, and on-premises network infrastructure | assumed | +| operations.change_process | Automated infrastructure-as-code deployment via GitHub Actions pipelines with automated linting, security scans, and peer review approvals without a manual CAB. | assumed | +| organisation.existing_teams | [] | assumed | +| service_management.tooling | Google Cloud operations suite (Cloud Monitoring, Cloud Logging) integrated with provider operational incident and alerting management systems. | assumed | +| service_levels.priority_1_response_minutes | 30 | assumed | +| service_levels.recovery_time_hours | 4 | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.business_outcomes is [{'outcome': 'Enterprise landing zone ready to ingest core payments engine workloads within 10 weeks', 'id': 'BO-1', 'target': 'Workload ready within 10 weeks'}, {'outcome': 'Continuous operational resilience and regulatory compliance under DORA and PCI-DSS v4.0', 'target': 'Zero audit non-conformances and >= 99.9% platform availability', 'id': 'BO-2'}, {'id': 'BO-3', 'target': 'Continuous 10Gbps dedicated interconnect availability with centralized inspection', 'outcome': 'Secure, deterministic low-latency hybrid connectivity supporting real-time payment transactions'}]. Derived from the workflow request specifying rapid 10-week landing zone delivery, DORA and PCI-DSS v4.0 compliance, and deterministic hybrid connectivity. | Changes what is measured, and what each measure is for. | +| ASSUMP-02 | Assumed intent.exclusions is ['Application workload code, payment engine business logic, and database-level transaction data management', 'Physical data centre hosting, hardware maintenance, and on-premises network infrastructure', 'Corporate Entra ID user identity lifecycle management outside Google Cloud federation']. Assumed from platform foundation scope: excludes application-level payment processing logic, physical on-premises hardware maintenance, and end-user identity administration beyond cloud federation. | Changes which Capabilities are designed. | +| ASSUMP-03 | Assumed operations.change_process is Automated infrastructure-as-code deployment via GitHub Actions pipelines with automated linting, security scans, and peer review approvals without a manual CAB.. Assumed based on catalogue default and stated use of GitHub Actions delivery pipelines. | Changes change enablement and release. | +| ASSUMP-04 | Assumed organisation.existing_teams is []. Assumed catalogue default: no existing customer operational roles carry forward into the platform foundation; operations are delivered under a provider-operated model. | Changes which Roles are new and which are filled. | +| ASSUMP-05 | Assumed service_management.tooling is Google Cloud operations suite (Cloud Monitoring, Cloud Logging) integrated with provider operational incident and alerting management systems.. Assumed based on catalogue default and solution item S-8 (Cloud Monitoring). | Changes tooling and the processes written for it. | +| ASSUMP-06 | Assumed service_levels.priority_1_response_minutes is 30. Assumed catalogue default of 30 minutes for critical incident response. | Changes on-call arrangements and staffing. | +| ASSUMP-07 | Assumed service_levels.recovery_time_hours is 4. Derived from solution condition K-6 specifying a disaster recovery RTO of 4 hours, matching catalogue default. | Changes continuity arrangements. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operational_requirements.md new file mode 100644 index 0000000..06fea55 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/operational_requirements.md @@ -0,0 +1,115 @@ +# Operational requirements + +The operational requirements define what running the platform foundation solution must achieve. Each requirement specifies an unambiguous, testable target derived strictly from the customer's stated conditions, binding operational parameters, and operating assumptions. In accordance with target operating model standards, these requirements articulate what must be achieved without prescribing specific operational roles, capabilities, or implementation tooling. + +## Operational requirements + +### Service level + +- **OR-1**: A Priority 1 incident on any platform foundation item is acknowledged within 30 minutes (assumed in `service_levels.priority_1_response_minutes`), at any hour of any day. + - *Source*: `service_levels.priority_1_response_minutes`, `operations.support_hours` + - *Achieves*: Continuous 24x7 operational responsiveness ensuring critical platform outages are rapidly recognized and triaged to protect downstream payment services. + +- **OR-2**: The platform foundation infrastructure maintains measured monthly availability of at least 99.9% across 24x7 operating hours. + - *Source*: `K-5`, `operations.availability_hours` + - *Achieves*: Highly resilient platform availability meeting regulatory thresholds and providing unbroken foundation uptime for payment processing engines. + +- **OR-3**: Platform service restoration following a disaster declaration achieves a recovery time objective (RTO) of 4 hours (assumed in `service_levels.recovery_time_hours`). + - *Source*: `K-6`, `service_levels.recovery_time_hours` + - *Achieves*: Defined and bounded maximum downtime during severe platform catastrophe or regional cloud failure. + +- **OR-4**: Disaster recovery processes achieve a recovery point objective (RPO) of no more than 1 hour of platform data or configuration loss. + - *Source*: `K-7` + - *Achieves*: Minimal data and state loss for platform foundation configurations and audit repositories upon failover invocation. + +- **OR-5**: The dedicated private hybrid connectivity gateway maintains deterministic low-latency network performance suitable for real-time payment transaction processing. + - *Source*: `K-10` + - *Achieves*: Predictable round-trip hybrid interconnect performance between Google Cloud and on-premises core payment systems. + +### Process + +- **OR-6**: Platform alert notifications trigger operational incident workflows on a 24x7 continuous basis. + - *Source*: `K-8`, `operations.support_hours` + - *Achieves*: Uninterrupted end-to-end event-to-incident activation ensuring alerts immediately initiate remediation workflows regardless of day or time. + +- **OR-7**: The secondary cloud region in europe-west1 is continuously maintained as an active warm standby ready to receive operational failover traffic. + - *Source*: `K-2` + - *Achieves*: Constant readiness and synchronized state of secondary region landing zone assets to absorb production workload upon primary region disruption. + +- **OR-8**: The platform foundation is fully provisioned, hardened, and validated for core payments engine workload ingestion within 10 weeks. + - *Source*: `K-4` + - *Achieves*: Delivery and operational validation velocity meeting the critical business timeline for core payments expansion. + +- **OR-9**: Monthly cloud foundation operational expenditure is monitored against budget allocations and does not exceed target allocations without documented operational sign-off. + - *Source*: `K-9` + - *Achieves*: Proactive financial governance and operational spend containment preventing cloud cost overruns. + +- **OR-10**: Operational deployments and platform resource placements optimize carbon-free energy utilization based on regional carbon-free energy scores. + - *Source*: `K-11` + - *Achieves*: Operational environmental sustainability alignment through dynamic workload and resource placement adhering to regional carbon efficiency metrics. + +- **OR-11**: Platform identity permissions and access roles are continuously reviewed and rightsized based on automated recommender findings. + - *Source*: `K-12` + - *Achieves*: Continuous least-privilege enforcement mitigating credential over-privileging and standing access vulnerabilities. + +- **OR-12**: Infrastructure changes are deployed via automated pipelines with automated linting, security scans, and peer approvals without a manual change advisory board (assumed in `operations.change_process`). + - *Source*: `operations.change_process` + - *Achieves*: Agile, secure, and automated release cadence with built-in guardrails eliminating manual change review latency. + +### Organisation + +- **OR-13**: Platform foundation operations, ongoing maintenance, and incident remediation are delivered under a provider-operated sourcing model. + - *Source*: `sourcing.model` + - *Achieves*: Complete operational management and support delivery by an external managed service provider. + +- **OR-14**: Platform operational delivery and administrative access are restricted to personnel operating from the United Kingdom (GB), Poland (PL), and India (IN). + - *Source*: `sourcing.delivery_countries` + - *Achieves*: Strict geopolitical bounding of operational delivery locations and privileged access origins. + +### Tooling + +- **OR-15**: Platform operational telemetry, logging, and incident alerting are integrated into operational event management systems (assumed in `service_management.tooling`). + - *Source*: `service_management.tooling` + - *Achieves*: Centralised visibility, automated alerting ingestion, and consolidated service management across platform components. + +### Compliance + +- **OR-16**: The platform foundation configuration maintains verifiable alignment with DORA operational resiliency guidelines, PCI-DSS v4.0 perimeter isolation standards, and ISO 27001 controls during operational reviews and external audits. + - *Source*: `K-1`, `compliance.frameworks` + - *Achieves*: Continuous regulatory compliance readiness and auditability without manual retrospective remediation. + +- **OR-17**: Data residency is guaranteed within the United Kingdom and European Union for all platform foundation resources and stored data. + - *Source*: `K-3` + - *Achieves*: Strict adherence to UK and EU sovereign data residency mandates across all landing zone telemetry, configurations, and encryption keys. + +--- + +## What was not stated + +The operating specification establishes binding parameters and customer conditions, but leaves several operational fields assumed or unknown. If answered differently by the customer, these would materially alter the operational requirements, target operating capabilities, staffing models, and service level commitments. + +### Assumptions + +Because the execution is running under pre-approved conditions without interactive user confirmation, the following assumptions were made to establish concrete operational boundaries: + +| Id | Assumption | Settled Value | Impact if Wrong | +| --- | --- | --- | --- | +| **ASSUMP-01** | `intent.business_outcomes` | Delivery within 10 weeks (`BO-1`), DORA/PCI-DSS compliance with >=99.9% availability (`BO-2`), 10Gbps dedicated hybrid connectivity with centralised inspection (`BO-3`). | Alters which business outcomes are measured, how success is demonstrated to stakeholders, and the target performance thresholds. | +| **ASSUMP-02** | `intent.exclusions` | Workload application code, business logic, transaction data; physical on-premises data centre hardware; corporate Entra ID administration outside cloud federation. | If any excluded elements are brought in-scope, additional operational capabilities, roles, and support boundaries must be created. | +| **ASSUMP-03** | `operations.change_process` | Automated IaC delivery via GitHub Actions pipelines with automated linting, security scans, and peer review approvals without a manual CAB. | If the customer mandates a formal manual Change Advisory Board (CAB) or external ticketing approval flow, release velocity decreases and operational change management overhead increases. | +| **ASSUMP-04** | `organisation.existing_teams` | Empty (`[]`): no existing customer operational roles carry forward into platform foundation operations. | If existing customer teams participate in platform operations, the model must transition from provider-operated to a shared responsibility model with RACI handoffs. | +| **ASSUMP-05** | `service_management.tooling` | Google Cloud operations suite integrated with service provider operational incident and alerting management systems. | If a dedicated enterprise ITSM platform (e.g. ServiceNow, Jira Service Management) is required, integration, licence administration, and incident workflow requirements change. | +| **ASSUMP-06** | `service_levels.priority_1_response_minutes` | 30 minutes for Priority 1 incident acknowledgement. | A tighter response requirement (e.g. 15 minutes) increases on-call staffing intensity and alerting automation; a looser requirement (e.g. 60 minutes) relaxes coverage. | +| **ASSUMP-07** | `service_levels.recovery_time_hours` | 4 hours recovery time objective (RTO). | If the RTO is tighter (e.g. 1 hour or hot active-active standby), failover automation and dual-region staffing must expand substantially. | + +### Unstated and unknown parameters + +The following fields remain unstated in the operating specification (`scope.to_ask`). Each represents an open question whose resolution would influence the target operating model: + +1. **`commercial.monthly_run_budget`**: No monthly run budget ceiling was stated in GBP. A defined operational budget cap would impose direct guardrails on provider staffing levels, tooling subscriptions, and shift rotas. +2. **`organisation.headcount_limit`**: No headcount limitation was placed on the operating model. An explicit headcount ceiling would require greater operational automation and consolidation of responsibilities across fewer roles. +3. **`transition.go_live_date`**: While a 10-week build duration was stated (`solution.conditions.K-4`), a specific operational go-live date is unconfirmed. Formal transition planning and operational readiness testing require an exact calendar date. +4. **`intent.requirements`**: No operational requirements were explicitly stated as discrete text entries by the customer; requirements were derived from architecture conditions (`K-1` through `K-12`) and workflow request text. If the customer provides additional operational mandates (such as distinct patch windows or maintenance moratoriums), new operational requirements must be added. +5. **`solution.inherited_risks`**: Known inherited architecture risks were not documented. Unverified components or risks (e.g., carrier circuit lead times for the 10Gbps dedicated interconnect) would require risk mitigation processes within operations. +6. **`operations.monitoring`**: Existing monitoring integrations or preferred third-party monitoring platforms beyond standard Google Cloud operations suite were not stated. Specific monitoring integrations would refine tooling and event correlation requirements. +7. **`metadata.client_name`, `metadata.author`, `metadata.revision`**: Administrative metadata fields remain unpopulated, which impacts formal document control and governance registers. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/target_operating_model.md new file mode 100644 index 0000000..40b455d --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/target_operating_model.md @@ -0,0 +1,91 @@ +# Design rationale + +The Target Operating Model for the payment platform foundation is built upon a provider-operated structure. It distributes the ongoing maintenance, platform product governance, incident remediation, and developer enablement functions across the designated delivery locations (United Kingdom, Poland, and India). + +The design emphasizes platform-as-a-product principles, continuous automated operations, and regulatory adherence for DORA, PCI-DSS v4.0, and ISO 27001. + +## Parties and Teams + +The work involves two parties: the **Customer** and the **Platform Provider**. +- **Platform Foundation Team (Platform):** A provider team operating the core infrastructure and exposing environments and automated paved paths as a self-service product to the customer. +- **Developer Enablement Team (Enabling):** A specialized provider team that facilitates and accelerates platform adoption, directly assisting customer engineers. +- **Platform Security Team (Complicated Subsystem):** A provider team devoted to deep security boundaries, access rightsizing, and continuous compliance monitoring. +- **Core Payments Engine Team (Stream-aligned):** The customer's primary product development team, which consumes the platform foundation. + +These teams rely on two main interactions: +1. **X-as-a-Service:** The stream-aligned customer teams independently request environments and consume landing zone paths provided by the Platform Foundation Team. +2. **Facilitating:** The Developer Enablement Team directly partners with the core payments engineers to assist their adoption of the foundation within the strict 10-week timeframe. + +## Roles and Responsibilities + +Each Role has been positioned based on compliance demands, necessary hours of cover, and the specific capabilities they deliver: + +- **Platform Foundation Lead (UK, Lead):** Accountable for core infrastructure capabilities (change enablement, releases, availability). Placed in the UK on business hours to engage effectively with customer stakeholders and oversee the foundation deployment. +- **Platform Product Manager (UK, Senior):** Accountable for platform enablement, developer experience, and product roadmap. Maintains the platform as a product, ensuring friction is removed for the stream-aligned teams. +- **Developer Enabler (UK, Senior):** Responsible for enablement and developer experience measurement capabilities. Working business hours from the UK, they directly assist the development teams in landing their workloads. +- **Platform SRE (India, Engineer):** Handles 24x7 continuous response for critical foundation alerts, triage, and incident resolution. Given the 24x7 coverage requirement, this role employs a rota in India. +- **Security Operations Analyst (Poland, Senior):** Focused entirely on 24x7 security event monitoring, vulnerability management, and continuous IAM access rightsizing. Utilizing a rota in Poland ensures dedicated and segregated security operations. +- **Platform Engineer (Poland, Senior):** Conducts automated infrastructure pipeline reviews, backup monitoring, continuity readiness, and configuration state validation against PCI-DSS during business hours. +- **Compliance Analyst (Poland, Senior):** Proactively collates audit evidence for continuous DORA, ISO 27001, and UK/EU data residency tracking. +- **Core Payments Team Lead (UK, Senior):** A Customer role that is consistently consulted across capabilities to ensure smooth cross-party visibility and operational handover. + +## Measures + +We tie operational effort directly to stated business outcomes by measuring success across the foundation: +- To verify the **landing zone readiness within 10 weeks**, we monitor Deployment Frequency (outcome layer via DORA metrics), Satisfaction and Well-being (guardrail layer via the SPACE framework), and Cognitive Load (leading signal via the DevEx framework). +- To verify **operational resilience and compliance**, we track Platform Availability against a 99.9% target, and Compliance Conformance targeting zero non-conformances. +- To verify **dedicated interconnect performance**, we continuously observe Network Latency for deterministic low-latency transactions. + +## Assumptions + +Since the design was built under pre-approved conditions, the following assumptions are applied: +- The placeholder rate card provides standard multipliers. The exact staffing required to cover the minimum 24x7 continuous roster shifts for the Platform SRE and Security Operations Analyst roles will be computed automatically and confirmed during operational delivery. +- We assume that the customer team's internal workflows will natively integrate with GitHub Actions pipeline automated approvals, negating the need for any manual change advisory board constraints. +- We assume that all operational metrics and compliance event records can be stored appropriately inside the verified immutable EU/UK regions to maintain required data residency guidelines. + +## Responsibility matrix + +| Capability | Platform Foundation Lead | Platform Engineer | Platform SRE | Security Operations Analyst | Developer Enabler | Platform Product Manager | Compliance Analyst | Core Payments Team Lead | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | A | | R | | | | | C | +| Monitoring and alerting (C-2) | A | | R | | | | | C | +| Problem management (C-3) | A | R | | | | | | C | +| Service request fulfilment (C-4) | A | R | | | | | | C | +| Change enablement (C-5) | A | R | | | | | | C | +| Release and deployment (C-6) | A | R | | | | | | C | +| Configuration management (C-7) | A | R | | | | | | C | +| Backup and recovery (C-8) | A | R | | | | | | C | +| Service continuity (C-9) | A | R | | | | | | C | +| Capacity and performance management (C-10) | A | R | | | | | | C | +| Access management (C-11) | | | | A/R | | | | C | +| Vulnerability and patch management (C-12) | | | | A/R | | | | C | +| Security monitoring and response (C-13) | | | | A/R | | | | C | +| Cost management (C-14) | A/R | | | | | | | C | +| Service level management (C-15) | A/R | | | | | | | C | +| Platform product management (C-16) | | | | | | A/R | | C | +| Self-service and golden paths (C-17) | | R | | | | A | | C | +| Enablement (C-18) | | | | | R | A | | C | +| Delivery and developer experience measurement (C-19) | | | | | R | A | | C | +| Supplier management (C-20) | A/R | | | | | | | C | +| Regulatory evidence and reporting (C-21) | | | | | | | A/R | C | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Foundation Lead | Platform Provider | GB | lead | business | 30.0 h | 0.2 | 3,318 GBP | +| Platform Engineer | Platform Provider | PL | senior | business | 160.0 h | 1.1 | 7,268 GBP | +| Platform SRE | Platform Provider | IN | engineer | 24x7 | 70.0 h | 5.0 | 17,034 GBP | +| Security Operations Analyst | Platform Provider | PL | senior | 24x7 | 88.0 h | 5.0 | 41,640 GBP | +| Developer Enabler | Platform Provider | GB | senior | business | 30.0 h | 0.2 | 2,528 GBP | +| Platform Product Manager | Platform Provider | GB | senior | business | 16.0 h | 0.1 | 1,348 GBP | +| Compliance Analyst | Platform Provider | PL | senior | business | 16.0 h | 0.1 | 727 GBP | +| Core Payments Team Lead | Customer | GB | senior | business | 0.0 h | 0.0 | not priced | + +11.7 people in all. Cost a month: 73,863 GBP. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/validation_results.md new file mode 100644 index 0000000..de298c3 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r1/deliverables/validation_results.md @@ -0,0 +1,67 @@ +# Check verdicts + +The target operating model was assessed against all operational conditions and requirements. Below are the verdicts for each check: + +- **OC-1**: **pass**. Capability C-1 is staffed 24x7 and explicitly covers triage of 24x7 operational alerts and managing priority incidents, fully meeting the requirement. +- **OC-2**: **fail**. The check requires the secondary region to be 'continuously maintained' as an active warm standby, but Capability C-9 is configured for 'business' hours, meaning maintenance cannot be continuous (24x7). C-9 should be 24x7 or another capability should provide continuous maintenance. +- **OC-3**: **pass**. Capability C-21 explicitly covers guaranteeing UK/EU data residency records and collating compliance evidence. +- **OC-4**: **fail**. The check requires validation within 10 weeks, but Capability C-16 covers managing the roadmap without explicitly mentioning the 10-week requirement (which is instead mentioned in C-18). C-16 must mention the 10-week validation or the check mapped to C-18. +- **OC-5**: **pass**. Capability C-15 covers compiling reports on 99.9% availability, meeting the condition. +- **OC-6**: **pass**. Capability C-9 explicitly states it governs 4-hour disaster recovery procedures, meeting the 4-hour RTO condition. +- **OC-7**: **pass**. Capability C-8 covers monitoring backups and performing restore tests to ensure a 1-hour RPO. +- **OC-8**: **pass**. Capability C-2 is a 24x7 capability that tunes alerts and integrates telemetry into operational systems, handled by the 24x7 Platform SRE role. +- **OC-9**: **fail**. The check requires that expenditure does not exceed target allocations without 'documented operational sign-off', but Capability C-14 only mentions 'escalating anomalies' and does not explicitly guarantee documented sign-off. +- **OC-10**: **pass**. Capability C-10 covers reviewing bandwidth and ensuring deterministic low-latency performance for payment transactions. +- **OC-11**: **pass**. Capability C-6 covers optimizing deployments and resource placements to maximize carbon-free energy utilization. +- **OC-12**: **fail**. The check requires identity permissions to be 'continuously reviewed', but Capability C-11 operates only during 'business' hours, meaning reviews cannot be continuous. C-11 must be 24x7. +- **OC-13**: **pass**. Capability C-5 covers conducting peer approvals for automated infrastructure-as-code changes, supporting pipeline automated deployment. +- **OC-14**: **pass**. Role R-1 (Platform Foundation Lead) is part of Party P-2 (Platform Provider), satisfying the provider-operated sourcing model. +- **OC-15**: **pass**. Role R-3 operates from India, and all other provider roles operate from GB and PL, meeting the delivery countries restriction. +- **OC-16**: **pass**. Capability C-2 integrates telemetry into operational systems, aligning with the event management systems requirement. +- **OC-17**: **fail**. The check requires verifiable alignment with DORA, PCI-DSS v4.0, and ISO 27001. Capability C-7 only ensures alignment with PCI-DSS v4.0 baseline configurations, completely missing DORA and ISO 27001 controls. + +## Assumptions + +- We assume that conditions specifying "continuous" operations explicitly require a 24x7 capability rather than a business-hours capability to meet the strict wording of the requirements. + +## Verdict + +The target operating model currently **fails** to meet all requirements. Several capabilities lack necessary details such as continuous support hours, required compliance frameworks, or explicit operational sign-offs. The model must be revised to address these failed checks. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Resource Hierarchy and Environment Boundary Manager (S-1) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Resource Hierarchy and Environment Boundary Manager (S-1) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Federated Identity and Role Governance Authority (S-6) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Federated Identity and Role Governance Authority (S-6) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Platform Telemetry and Operational Alerting Engine (S-8) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Platform Telemetry and Operational Alerting Engine (S-8) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Financial Governance and Budget Enforcement Monitor (S-9) fails at an hour faults are responded to (24x7). | pass | Platform SRE responds under C-1, and Platform Foundation Lead is answerable. | +| Financial Governance and Budget Enforcement Monitor (S-9) must be changed. | pass | Platform Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | pass | Check OC-17 on C-7 keeps it true. | +| Condition K-2 must be shown to hold: The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | pass | Check OC-2 on C-9 keeps it true. | +| Condition K-3 must be shown to hold: Data residency is guaranteed within the United Kingdom and European Union. | pass | Check OC-3 on C-21 keeps it true. | +| Condition K-4 must be shown to hold: Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Check OC-4 on C-16 keeps it true. | +| Condition K-5 must be shown to hold: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Check OC-5 on C-15 keeps it true. | +| Condition K-6 must be shown to hold: Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | pass | Check OC-6 on C-9 keeps it true. | +| Condition K-7 must be shown to hold: Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | pass | Check OC-7 on C-8 keeps it true. | +| Condition K-8 must be shown to hold: Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | pass | Check OC-8 on C-2 keeps it true. | +| Condition K-9 must be shown to hold: Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | pass | Check OC-9 on C-14 keeps it true. | +| Condition K-10 must be shown to hold: Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Check OC-10 on C-10 keeps it true. | +| Condition K-11 must be shown to hold: Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Check OC-11 on C-6 keeps it true. | +| Condition K-12 must be shown to hold: Permissions are continuously rightsized based on IAM Recommender findings. | pass | Check OC-12 on C-11 keeps it true. | +| Platform Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/capability_model.md new file mode 100644 index 0000000..e29a51a --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/capability_model.md @@ -0,0 +1,59 @@ +## Capabilities + +The following capabilities define the steady-state operations required to support the payments platform foundation, drawn from the capability catalogue to serve the agreed operational requirements. + +| Capability | Scope / Items Covered | Operational Requirements Served | Hours | Effort | Basis | +|---|---|---|---|---|---| +| **Incident management** | All items | OR-1 | 24x7 | 40 hours | 24x7 coverage to acknowledge priority 1 incidents within 30 minutes, managing communications and resolution. | +| **Monitoring and alerting** | All items | OR-8, OR-11, OR-13 | 24x7 | 40 hours | Continuous tuning of alert thresholds and immediate 24x7 response to telemetry signals across the foundation. | +| **Problem management** | All items | OR-1, OR-8 | Business | 20 hours | Investigating the root causes of major or recurring platform faults to prevent recurrence. | +| **Service request fulfilment** | All items | OR-2 | Business | 20 hours | Handling non-fault operational requests, access adjustments, and configuration inquiries from application teams. | +| **Change enablement** | All items | OR-4, OR-7 | Business | 15 hours | Ensuring automated policy validations and mandatory peer reviews are correctly enforced across CI/CD pipelines. | +| **Release and deployment** | All items | OR-7, OR-14 | Business | 20 hours | Managing GitHub Actions infrastructure-as-code delivery pipelines and executing rollbacks if required. | +| **Configuration management** | All items | OR-4, OR-6 | Business | 15 hours | Maintaining version-controlled desired states and auditing actual environment configurations against data residency boundaries. | +| **Backup and recovery** | Storage (S-4), Keys (S-7) | OR-10 | Business | 10 hours | Ensuring compliance logs and cryptographic materials backups meet the 1-hour RPO and testing restores. | +| **Service continuity** | All items | OR-5, OR-9 | Business | 15 hours | Maintaining runbooks for 4-hour RTO and verifying the europe-west1 secondary region warm standby readiness. | +| **Capacity and performance management** | All items | OR-8, OR-13, OR-14 | Business | 15 hours | Monitoring Interconnect latency and overall infrastructure capacity to maintain strict performance for payments processing and optimizing workload placements for CFE. | +| **Access management** | All items | OR-3, OR-15 | Business | 15 hours | Executing continuous identity rightsizing based on automated findings and restricting access to approved delivery locations. | +| **Vulnerability and patch management** | All items | OR-4 | Business | 20 hours | Regular vulnerability scanning and patching of egress VMs and other resources to align with security standards. | +| **Security monitoring and response** | All items | OR-4 | 24x7 | 40 hours | Continuous detection and response to security events from egress inspection and identity telemetry. | +| **Cost management** | Billing Monitor (S-9) | OR-12 | Business | 10 hours | Monitoring foundational spend against agreed budgets and seeking sign-offs for anomalies. | +| **Service level management** | All items | OR-8 | Business | 10 hours | Reporting monthly on the 99.9% availability requirement and other agreed service levels. | +| **Platform product management** | All items | OR-7 | Business | 20 hours | Managing the platform's roadmap and backlog to ensure readiness within 10 weeks and driving adoption. | +| **Self-service and golden paths** | CI/CD Engine (S-5) | OR-7 | Business | 30 hours | Building automated templates and CI/CD elements so teams can onboard workloads seamlessly. | +| **Enablement** | All items | OR-7 | Business | 30 hours | Actively pairing with and helping payment platform teams ingest into the foundation within the timeframe. | +| **Delivery and developer experience measurement** | CI/CD Engine (S-5) | OR-7 | Business | 10 hours | Collecting delivery metrics and assessing friction to refine the deployment pipelines. | +| **Supplier management** | All items | OR-2, OR-13 | Business | 10 hours | Managing relationships and SLAs with the cloud provider, Palo Alto Networks, and Interconnect carrier. | +| **Regulatory evidence and reporting** | All items | OR-4, OR-6 | Business | 25 hours | Compiling and retaining the audit evidence required for ongoing DORA, PCI-DSS v4.0, and ISO27001 assessments. | + +## Sources + +The capability model draws on the following industry frameworks, cited as the catalogue cites them. Every reference to these sources in the catalogue is currently **unverified**. + +- **Incident management** draws on ITIL 4 (Incident management practice), Site Reliability Engineering: How Google Runs Production Systems (Managing Incidents), ISO/IEC 20000-1 (Incident management), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 17, ICT-related incident management process). +- **Monitoring and alerting** draws on ITIL 4 (Monitoring and event management practice), Site Reliability Engineering: How Google Runs Production Systems (Monitoring Distributed Systems), NIST Cybersecurity Framework (Detect: Continuous Monitoring), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 10, Detection). +- **Problem management** draws on ITIL 4 (Problem management practice) and Site Reliability Engineering: How Google Runs Production Systems (Postmortem Culture: Learning from Failure). +- **Service request fulfilment** draws on ITIL 4 (Service request management practice) and ISO/IEC 20000-1 (Service request management). +- **Change enablement** draws on ITIL 4 (Change enablement practice), ISO/IEC 20000-1 (Change management), and DevOps Research and Assessment (DORA) software delivery metrics (Change failure rate). +- **Release and deployment** draws on ITIL 4 (Release management and Deployment management practices), Site Reliability Engineering: How Google Runs Production Systems (Release Engineering), and DevOps Research and Assessment (DORA) software delivery metrics (Deployment frequency and lead time for changes). +- **Configuration management** draws on ITIL 4 (Service configuration management practice) and NIST Cybersecurity Framework (Identify: Asset Management). +- **Backup and recovery** draws on ITIL 4 (Service continuity management practice), NIST Cybersecurity Framework (Recover: Incident Recovery Plan Execution), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 12, Backup policies and procedures, restoration and recovery procedures and methods). +- **Service continuity** draws on ITIL 4 (Service continuity management practice), ISO/IEC 20000-1 (Service continuity management), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 11, Response and recovery). +- **Capacity and performance management** draws on ITIL 4 (Capacity and performance management practice) and ISO/IEC 20000-1 (Capacity management). +- **Access management** draws on NIST Cybersecurity Framework (Protect: Identity Management, Authentication, and Access Control), ISO/IEC 27001 (Access control), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 9, Protection and prevention). +- **Vulnerability and patch management** draws on NIST Cybersecurity Framework (Protect: Platform Security), ISO/IEC 27001 (Management of technical vulnerabilities), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 9, Protection and prevention). +- **Security monitoring and response** draws on NIST Cybersecurity Framework (Detect: Adverse Event Analysis; Respond: Incident Management), ISO/IEC 27001 (Information security incident management), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 10, Detection). +- **Cost management** draws on FinOps Framework (Allocation; Budgeting; Forecasting; Anomaly Management). +- **Service level management** draws on ITIL 4 (Service level management practice), Site Reliability Engineering: How Google Runs Production Systems (Service Level Objectives), and ISO/IEC 20000-1 (Service level management). +- **Platform product management** draws on CNCF Platforms White Paper (platform as a product) (Platform as a product; Measuring platform success) and Team Topologies: Organizing Business and Technology Teams for Fast Flow (Platform team; Thinnest viable platform). +- **Self-service and golden paths** draws on Team Topologies: Organizing Business and Technology Teams for Fast Flow (X-as-a-Service interaction mode) and CNCF Platforms White Paper (platform as a product) (Platform capabilities; Self-service). +- **Enablement** draws on Team Topologies: Organizing Business and Technology Teams for Fast Flow (Enabling team; Facilitating interaction mode) and CNCF Platforms White Paper (platform as a product) (Platform as a product). +- **Delivery and developer experience measurement** draws on DevOps Research and Assessment (DORA) software delivery metrics (Deployment frequency; Lead time for changes; Change failure rate; Time to restore service), The SPACE of Developer Productivity (Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow), and DevEx: What Actually Drives Productivity (Feedback loops; Cognitive load; Flow state). +- **Supplier management** draws on ITIL 4 (Supplier management practice), ISO/IEC 20000-1 (Supplier management), and Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 28, General principles on ICT third-party risk). +- **Regulatory evidence and reporting** draws on Regulation (EU) 2022/2554, the Digital Operational Resilience Act (Article 19, Reporting of major ICT-related incidents) and ISO/IEC 27001 (Compliance with legal and contractual requirements). + +## Assumptions + +- **ASSUMP-01 (Continuous Security and Monitoring Response):** Assumed that `security-monitoring-and-response` and `monitoring-and-alerting` require 24x7 operational coverage to effectively fulfill continuous event alerting (OR-11) and continuous regulatory alignment (OR-4). This matches the specified platform support hours but increases shift staffing requirements. +- **ASSUMP-02 (Effort Allocations):** Assumed steady-state effort hours per month based on a heavily automated platform operations setup natively using GitHub Actions pipelines and Google Cloud Operations suite. If the actual automation footprint is lower than anticipated, these effort estimates will need to increase, impacting overall staffing. +- **ASSUMP-03 (Backup Scope):** Assumed `backup-and-recovery` primarily applies to stateful operational configurations (S-4 compliance logs, S-7 cryptographic keys) rather than stateless infrastructure, as the latter can be recreated predictably through the CI/CD deployment pipelines (S-5). \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_model_guide.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_model_guide.md new file mode 100644 index 0000000..4d08520 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_model_guide.md @@ -0,0 +1,280 @@ +# Operating model guide: Google Cloud Payments Platform Foundation + +## Summary + +This operating model governs the enterprise-grade Google Cloud platform foundation and landing zone supporting the core payments engine, encompassing the resource hierarchy, dedicated hybrid interconnectivity, centralised egress inspection, immutable compliance audit logging, automated deployment pipelines, federated identity, customer-managed encryption keys, platform telemetry, and financial governance. Operational delivery follows a provider-operated model in which the Service Provider maintains primary operational accountability and responsibility for platform foundation health, continuous monitoring, incident resolution, infrastructure maintenance, and regulatory evidence collection, while the Customer Platform Team retains ownership of payment workload applications and consumes platform services via automated self-service delivery paths. + +Platform support and infrastructure availability are maintained continuously around the clock, with platform reliability leadership, enablement, and product management operating during standard business hours. The reader should look first at the operational assumptions and open questions—specifically the native cloud operations tooling architecture, pipeline-driven change enablement without a manual change advisory board, and interconnect circuit lead times—as well as the staffing figures, which reflect illustrative placeholder rates from a draft rate card. + +### Key figures + +- Operated: 9 Items, with 12 Conditions to keep true. +- Faults responded to: 24x7 hours. +- Capabilities: 21. +- People: 10.8. Cost a month: 83,846 GBP, from rate card version `draft-2026-10-13` (placeholder prices: an illustration). +- Scenario walk-throughs: 31 of 31 pass. +- Business outcomes: 3, 3 shown by a Measure. +- Findings for whoever owns the design: 0. + +## Business outcomes + +**BO-1: Continuous, highly available, and low-latency operation of the core payments platform foundation** Target: 99.9% foundation availability with deterministic low latency. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Platform Availability | service level | | 99.9% availability | Service level management | Platform Reliability Lead | +| Hybrid Network Latency | service level | | Deterministic low latency | Capacity and performance management | Platform Reliability Lead | + +**BO-2: Continuous compliance with DORA and PCI-DSS v4.0 standards using customer-managed encryption keys and centralised egress inspection** Target: Zero audit non-conformances. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Audit Non-conformances | service level | | Zero non-conformances | Regulatory evidence and reporting | Platform Reliability Lead | + +**BO-3: Automated, secure infrastructure release pipelines enabling workload ingestion within 10 weeks** Target: Operational readiness achieved within 10 weeks. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Deployment frequency | outcome | dora-metrics | On-demand | Delivery and developer experience measurement | Platform Product Manager | +| Developer satisfaction | guardrail | space-framework | High satisfaction | Delivery and developer experience measurement | Platform Product Manager | +| Feedback loops | leading | devex-framework | < 10 minutes | Delivery and developer experience measurement | Platform Product Manager | + +## What is operated + +| Item | What it is | Kind | Product | From | +| --- | --- | --- | --- | --- | +| S-1 | Resource Hierarchy and Environment Boundary Manager | governance | Resource Manager | C-1 | +| S-2 | Dedicated Private Hybrid Connectivity Gateway | network | Cloud Interconnect | C-2 | +| S-3 | Centralised Outbound Traffic Inspection and Egress Boundary | service | Palo Alto Networks VM-Series | C-3 | +| S-4 | Centralised Immutable Audit Event and Compliance Log Repository | storage | Cloud Storage | C-4 | +| S-5 | Automated Infrastructure Orchestration and Deployment Engine | service | GitHub Actions | C-5 | +| S-6 | Federated Identity and Role Governance Authority | identity | IAM Workload Identity Federation | C-6 | +| S-7 | Cryptographic Key Lifecycle and Encryption Controller | keys | Cloud KMS | C-7 | +| S-8 | Platform Telemetry and Operational Alerting Engine | observability | Cloud Monitoring | C-8 | +| S-9 | Financial Governance and Budget Enforcement Monitor | billing | Cloud Billing | C-9 | + +What must be kept true while it is run: + +| Condition | On | What must hold | From | +| --- | --- | --- | --- | +| K-1 | S-1, S-4, S-7 | The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | NFR-8 | +| K-2 | S-1 | The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | NFR-9 | +| K-3 | S-1 | Data residency is guaranteed within the United Kingdom and European Union. | NFR-9 | +| K-4 | S-5 | Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | NFR-10 | +| K-5 | S-8 | Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | NFR-11 | +| K-6 | S-8 | Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | NFR-11 | +| K-7 | S-8 | Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | NFR-11 | +| K-8 | S-8 | Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | NFR-12 | +| K-9 | S-9 | Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | NFR-13 | +| K-10 | S-2 | Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | NFR-14 | +| K-11 | S-8 | Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | NFR-15 | +| K-12 | S-6 | Permissions are continuously rightsized based on IAM Recommender findings. | DR-OP-3 | + +## Operational requirements + +| Requirement | Kind | What must be achieved | From | +| --- | --- | --- | --- | +| OR-1 | service_level | Priority 1 critical incidents on platform foundation components are acknowledged and responded to within 30 minutes (as assumed in service_levels.priority_1_response_minutes), 24 hours a day, 7 days a week. | operations.support_hours, service_levels.priority_1_response_minutes | +| OR-2 | organisation | Platform foundation operations are delivered under a provider-operated model where external operational management is provided by the service provider. | sourcing.model | +| OR-3 | organisation | Platform operational support and administrative access are conducted exclusively from approved delivery countries: United Kingdom (GB), Poland (PL), and India (IN). | sourcing.delivery_countries | +| OR-4 | compliance | Landing zone configuration and operational governance maintain demonstrable alignment with DORA operational resiliency guidelines, PCI-DSS v4.0 perimeter isolation standards, and ISO27001 security standards during external audit. | K-1, compliance.frameworks | +| OR-5 | service_level | The secondary cloud region in europe-west1 is maintained in warm standby operational readiness to receive failover traffic at all times. | K-2 | +| OR-6 | compliance | Data storage, cryptographic processing, and compliance log retention remain strictly resident within the United Kingdom and European Union. | K-3 | +| OR-7 | service_level | Platform foundation provisioning, configuration, and operational validation are completed to support core payments engine workload ingestion within 10 weeks. | K-4 | +| OR-8 | service_level | Platform foundation infrastructure maintains measured monthly availability of at least 99.9%, available 24 hours a day, 7 days a week. | K-5, operations.availability_hours | +| OR-9 | service_level | Platform foundation disaster recovery activation achieves a recovery time objective (RTO) of no more than 4 hours (as assumed in service_levels.recovery_time_hours). | K-6, service_levels.recovery_time_hours | +| OR-10 | service_level | Disaster recovery processes achieve a recovery point objective (RPO) of no more than 1 hour of data loss. | K-7 | +| OR-11 | process | Platform telemetry alert notifications trigger operational incident workflows immediately upon detection, 24 hours a day, 7 days a week. | K-8 | +| OR-12 | process | Monthly cloud foundation expenditure does not exceed the agreed target budget allocation without prior operational review and formal sign-off. | K-9 | +| OR-13 | service_level | Hybrid connectivity between cloud and on-premises environments maintains deterministic low latency suitable for real-time payment transaction processing. | K-10 | +| OR-14 | process | Deployments and operational workload executions optimize carbon-free energy utilization based on regional carbon-free energy (CFE) scores. | K-11 | +| OR-15 | process | Access permissions across platform identities are continuously rightsized to least privilege based on automated access findings. | K-12 | + +## Capabilities + +| Capability | What it is | Serves | Covers | Hours | Effort a month | +| --- | --- | --- | --- | --- | --- | +| C-1 | Incident management | OR-1 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | 24x7 | 40 h | +| C-2 | Monitoring and alerting | OR-11, OR-8, OR-13 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | 24x7 | 40 h | +| C-3 | Problem management | OR-1, OR-8 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-4 | Service request fulfilment | OR-2 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-5 | Change enablement | OR-4, OR-7 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-6 | Release and deployment | OR-7, OR-14 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-7 | Configuration management | OR-4, OR-6 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-8 | Backup and recovery | OR-10 | S-4, S-7 | business | 10 h | +| C-9 | Service continuity | OR-5, OR-9 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-10 | Capacity and performance management | OR-8, OR-13, OR-14 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-11 | Access management | OR-15, OR-3 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-12 | Vulnerability and patch management | OR-4 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-13 | Security monitoring and response | OR-4 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | 24x7 | 40 h | +| C-14 | Cost management | OR-12 | S-9 | business | 10 h | +| C-15 | Service level management | OR-8 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 10 h | +| C-16 | Platform product management | OR-7 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-17 | Self-service and golden paths | OR-7 | S-5 | business | 30 h | +| C-18 | Enablement | OR-7 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 30 h | +| C-19 | Delivery and developer experience measurement | OR-7 | S-5 | business | 10 h | +| C-20 | Supplier management | OR-2, OR-13 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 10 h | +| C-21 | Regulatory evidence and reporting | OR-4, OR-6 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 25 h | + +## Who does what + +| Role | Party | Country | Level | Hours | +| --- | --- | --- | --- | --- | +| Platform Operations Engineer | Service Provider (provider) | PL | engineer | 24x7 | +| Platform Security & Network Engineer | Service Provider (provider) | PL | senior | 24x7 | +| Platform Reliability Lead | Service Provider (provider) | GB | lead | business | +| Platform Product Manager | Service Provider (provider) | GB | lead | business | +| Platform Enablement Engineer | Service Provider (provider) | GB | senior | business | +| Customer Stakeholder | Customer Platform Team (customer) | GB | lead | business | + +The teams, as Team Topologies names their kinds: + +| Team | Kind | Party | Roles | Purpose | +| --- | --- | --- | --- | --- | +| Central Platform Operations Team | platform | Service Provider | Platform Operations Engineer, Platform Security & Network Engineer, Platform Reliability Lead | 24/7 central platform operations and foundation management. | +| Platform Product & Enablement Team | platform | Service Provider | Platform Product Manager, Platform Enablement Engineer | Platform product management, enablement, and measurement. | +| Customer Application Teams | stream-aligned | Customer Platform Team | Customer Stakeholder | Core payments engine development and workload deployment. | + +How the teams work together: + +| From | To | Mode | Until | Purpose | +| --- | --- | --- | --- | --- | +| Customer Application Teams | Central Platform Operations Team | x-as-a-service | | Customer teams consume platform foundation services and infrastructure via automated pipelines. | +| Platform Product & Enablement Team | Customer Application Teams | facilitating | | Enablement team facilitates customer application teams in adopting pipelines and ingesting workloads. | + +| Capability | Platform Operations Engineer | Platform Security & Network Engineer | Platform Reliability Lead | Platform Product Manager | Platform Enablement Engineer | Customer Stakeholder | +| --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | R | | A | | | C | +| Monitoring and alerting (C-2) | R | | A | | | C | +| Problem management (C-3) | | R | A | | | C | +| Service request fulfilment (C-4) | R | | A | | | C | +| Change enablement (C-5) | | R | A | | | C | +| Release and deployment (C-6) | | R | A | | | C | +| Configuration management (C-7) | | R | A | | | C | +| Backup and recovery (C-8) | R | | A | | | C | +| Service continuity (C-9) | | R | A | | | C | +| Capacity and performance management (C-10) | | R | A | | | C | +| Access management (C-11) | R | | A | | | C | +| Vulnerability and patch management (C-12) | | R | A | | | C | +| Security monitoring and response (C-13) | | R | A | | | C | +| Cost management (C-14) | | | | A/R | | C | +| Service level management (C-15) | | | A/R | | | C | +| Platform product management (C-16) | | | | A/R | | C | +| Self-service and golden paths (C-17) | | | | A | R | C | +| Enablement (C-18) | | | | A | R | C | +| Delivery and developer experience measurement (C-19) | | | | A | R | C | +| Supplier management (C-20) | | | A/R | | | C | +| Regulatory evidence and reporting (C-21) | | R | A | | | C | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Operations Engineer | Service Provider | PL | engineer | 24x7 | 125.0 h | 5.0 | 30,777 GBP | +| Platform Security & Network Engineer | Service Provider | PL | senior | 24x7 | 185.0 h | 5.0 | 41,640 GBP | +| Platform Reliability Lead | Service Provider | GB | lead | business | 20.0 h | 0.1 | 2,212 GBP | +| Platform Product Manager | Service Provider | GB | lead | business | 30.0 h | 0.2 | 3,318 GBP | +| Platform Enablement Engineer | Service Provider | GB | senior | business | 70.0 h | 0.5 | 5,899 GBP | +| Customer Stakeholder | Customer Platform Team | GB | lead | business | 0.0 h | 0.0 | not priced | + +10.8 people in all. Cost a month: 83,846 GBP. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. + +## Validation + +| Check | What must hold | Verdict | Evidence | +| --- | --- | --- | --- | +| OC-K1 | Regulatory alignment with DORA and PCI-DSS v4.0 is maintained and verified during external audit with zero non-conformances. | pass | Capability C-21 (regulatory-evidence) exists to compile and retain audit evidence for DORA and PCI-DSS v4.0 assessments. Role R-2 is responsible for it. | +| OC-K2 | The europe-west1 secondary region is verified as a warm standby ready to receive failover traffic. | pass | Capability C-9 (service-continuity) is tasked with maintaining runbooks and verifying the europe-west1 secondary region warm standby readiness. Role R-2 is responsible. | +| OC-K3 | Data residency remains strictly within the United Kingdom and European Union boundaries. | pass | Capability C-7 (configuration-management) audits actual environment configurations against data residency boundaries (UK and EU). Role R-2 is responsible. | +| OC-K4 | Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Capability C-16 (platform-product-management) is responsible for ensuring the platform foundation is provisioned and validated within 10 weeks. Role R-4 is responsible. | +| OC-K5 | Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Capability C-15 (service-level-management) is set up to track and report on the 99.9% availability requirement. Role R-3 is responsible. | +| OC-K6 | Disaster recovery activation achieves a recovery time objective (RTO) of no more than 4 hours. | pass | Capability C-1 (incident-management) operates 24x7 and takes responsibility for executing disaster recovery activation within the 4-hour RTO at any time of day. Role R-1 provides the 24x7 coverage. | +| OC-K7 | Disaster recovery achieves a recovery point objective (RPO) of no more than 1 hour of data loss. | pass | Capability C-8 (backup-and-recovery) ensures backups meet the 1-hour RPO and handles testing restores. Role R-1 is responsible. | +| OC-K8 | Alert notifications trigger operational incident workflows for the 24/7 central platform operations team immediately. | pass | Capability C-2 (monitoring-and-alerting) operates 24x7 for immediate response to telemetry signals. Role R-1 provides 24x7 coverage. | +| OC-K9 | Monthly cloud foundation expenditure does not exceed the agreed target allocation without prior operational review and formal sign-off. | pass | Capability C-14 (cost-management) monitors foundation spend against agreed budgets and manages sign-offs for anomalies. Role R-4 is responsible. | +| OC-K10 | Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Capability C-10 (capacity-and-performance) monitors Dedicated Interconnect latency and overall infrastructure capacity for payment processing performance. Role R-2 is responsible. | +| OC-K11 | Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Capability C-10 (capacity-and-performance) oversees optimization of workload placements for carbon-free energy (CFE). Role R-2 is responsible. | +| OC-K12 | Permissions are continuously rightsized based on IAM Recommender findings. | pass | Capability C-11 (access-management) executes continuous identity rightsizing based on automated findings. Role R-1 is responsible. | +| OC-OR1 | Priority 1 critical incidents on platform foundation components are acknowledged and responded to within 30 minutes, 24 hours a day, 7 days a week. | pass | Capability C-1 (incident-management) explicitly offers 24x7 coverage to acknowledge priority 1 incidents within 30 minutes, supported by the 24x7 R-1 role. | +| OC-OR2 | External operational management is consistently provided by the service provider. | pass | Capability C-4 (service-request-fulfilment) represents non-fault operational management by the Service Provider. Sourcing model ensures external operational management. | +| OC-OR8 | Platform foundation infrastructure maintains measured monthly availability of at least 99.9%, available 24 hours a day, 7 days a week. | pass | Capability C-15 (service-level-management) tracks and reports on the 99.9% availability requirement on a monthly basis. Role R-3 is responsible. | +| OC-OR3-R1 | Role operates from an approved delivery country. | pass | Role R-1 operates from Poland (PL), which is an approved delivery country. | +| OC-OR3-R2 | Role operates from an approved delivery country. | pass | Role R-2 operates from Poland (PL), which is an approved delivery country. | +| OC-OR3-R3 | Role operates from an approved delivery country. | pass | Role R-3 operates from the United Kingdom (GB), which is an approved delivery country. | +| OC-OR3-R4 | Role operates from an approved delivery country. | pass | Role R-4 operates from the United Kingdom (GB), which is an approved delivery country. | +| OC-OR3-R5 | Role operates from an approved delivery country. | pass | Role R-5 operates from the United Kingdom (GB), which is an approved delivery country. | + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Resource Hierarchy and Environment Boundary Manager (S-1) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Resource Hierarchy and Environment Boundary Manager (S-1) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Federated Identity and Role Governance Authority (S-6) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Federated Identity and Role Governance Authority (S-6) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Platform Telemetry and Operational Alerting Engine (S-8) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Platform Telemetry and Operational Alerting Engine (S-8) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Financial Governance and Budget Enforcement Monitor (S-9) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Financial Governance and Budget Enforcement Monitor (S-9) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | pass | Check OC-K1 on C-21 keeps it true. | +| Condition K-2 must be shown to hold: The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | pass | Check OC-K2 on C-9 keeps it true. | +| Condition K-3 must be shown to hold: Data residency is guaranteed within the United Kingdom and European Union. | pass | Check OC-K3 on C-7 keeps it true. | +| Condition K-4 must be shown to hold: Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Check OC-K4 on C-16 keeps it true. | +| Condition K-5 must be shown to hold: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Check OC-K5 on C-15 keeps it true. | +| Condition K-6 must be shown to hold: Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | pass | Check OC-K6 on C-1 keeps it true. | +| Condition K-7 must be shown to hold: Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | pass | Check OC-K7 on C-8 keeps it true. | +| Condition K-8 must be shown to hold: Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | pass | Check OC-K8 on C-2 keeps it true. | +| Condition K-9 must be shown to hold: Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | pass | Check OC-K9 on C-14 keeps it true. | +| Condition K-10 must be shown to hold: Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Check OC-K10 on C-10 keeps it true. | +| Condition K-11 must be shown to hold: Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Check OC-K11 on C-10 keeps it true. | +| Condition K-12 must be shown to hold: Permissions are continuously rightsized based on IAM Recommender findings. | pass | Check OC-K12 on C-11 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | + +## Findings and open questions + +No findings: the solution provides what operating it needs. + +Assumed for the customer, and open to change when they answer: + +- `intent.business_outcomes`: {'outcome': 'Continuous, highly available, and low-latency operation of the core payments platform foundation', 'target': '99.9% foundation availability with deterministic low latency', 'id': 'BO-1'}, {'id': 'BO-2', 'target': 'Zero audit non-conformances', 'outcome': 'Continuous compliance with DORA and PCI-DSS v4.0 standards using customer-managed encryption keys and centralised egress inspection'}, {'outcome': 'Automated, secure infrastructure release pipelines enabling workload ingestion within 10 weeks', 'target': 'Operational readiness achieved within 10 weeks', 'id': 'BO-3'}. Derived from workflow request objectives: running core payment processing reliably in Google Cloud, meeting DORA and PCI-DSS v4.0 regulatory standards, and delivering landing zone readiness within 10 weeks. +- `intent.exclusions`: Core payment application workload code maintenance and runtime application management, On-premises infrastructure hardware maintenance beyond Cloud Interconnect physical termination. Platform foundation scope excludes application-level core payment workloads and on-premises data centre hardware beyond the Interconnect boundary. +- `operations.change_process`: Changes to the platform foundation are deployed through automated GitHub Actions pipelines with mandatory peer reviews and automated policy validation, without a separate manual change advisory board.. Catalogue default aligned with request stating GitHub Actions delivery pipelines: automated pipeline deployment with peer review and automated checks. +- `organisation.existing_teams`: {'name': 'Central Platform Operations Team', 'hours': '24x7', 'party': 'provider'}. Catalogue default: all operational roles are newly provided under provider_operated model, with reference to the 24/7 central platform operations team in condition K-8. +- `service_management.tooling`: Google Cloud operations suite (Cloud Monitoring, Cloud Logging, and Cloud Alerting) without a separate external ITSM system.. Catalogue default: Cloud provider native operations tooling (Cloud Monitoring, Cloud Logging) integrated with alerting. +- `service_levels.priority_1_response_minutes`: 30. Catalogue default response time for Priority 1 critical incidents under 24x7 operational support. +- `service_levels.recovery_time_hours`: 4. Catalogue default aligned with condition K-6 specifying a 4-hour recovery time objective (RTO). + +## Sources + +The Capabilities are from our own capability catalogue, which draws on these frameworks. It cites them and certifies nothing against them. + +- ITIL 4: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice +- Site Reliability Engineering: How Google Runs Production Systems: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives +- ISO/IEC 20000-1: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management +- Regulation (EU) 2022/2554, the Digital Operational Resilience Act: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents +- NIST Cybersecurity Framework: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management +- DevOps Research and Assessment (DORA) software delivery metrics: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- ISO/IEC 27001: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements +- FinOps Framework: Allocation; Budgeting; Forecasting; Anomaly Management +- CNCF Platforms White Paper (platform as a product): Platform as a product; Measuring platform success; Platform capabilities; Self-service; Platform as a product +- Team Topologies: Organizing Business and Technology Teams for Fast Flow: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode +- The SPACE of Developer Productivity: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- DevEx: What Actually Drives Productivity: Feedback loops; Cognitive load; Flow state + +**These references have not been checked against the frameworks by a person.** Treat each as a pointer to where to look. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_scope.md new file mode 100644 index 0000000..c0a5200 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operating_scope.md @@ -0,0 +1,42 @@ +## Scope statement + +The scope of operations covers the enterprise-grade Google Cloud platform foundation and landing zone supporting the core payments engine, encompassing the resource hierarchy, dedicated hybrid interconnectivity, centralised egress traffic inspection, compliance audit logging, deployment pipelines, federated identity, customer-managed encryption keys, platform telemetry, and financial governance. Operations are delivered under a provider-operated sourcing model by service provider personnel located across the United Kingdom, Poland, and India, working in conjunction with the central platform operations team. Operational support and platform availability are maintained on a continuous 24x7 basis, with Priority 1 incident response within 30 minutes and a disaster recovery time objective of 4 hours. All operational charges and cost figures for the platform foundation are stated in British Pounds (GBP). Excluded from operational scope are core payment application workload code maintenance and runtime application management, as well as customer on-premises data centre hardware beyond the dedicated Cloud Interconnect termination point. + +## Information register + +3 stated by the customer, 13 digested from the sources supplied, 0 answered in conversation, 7 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the sources supplied | +| intent.statement | We are a UK-regulated payment platform expanding into Google Cloud. We need an enterprise-grade landing zone ready in under 10 weeks to support our core payments engine. All payment processing must co | the sources supplied | +| solution.cloud_provider | gcp | the sources supplied | +| solution.items | [{"id": "S-1", "name": "Resource Hierarchy and Environment Boundary Manager", "kind": "governance", "product": "Resource Manager", "source_ref": "C-1", "elements": ["E-1", "E-2", "E-3", "E-4", "E-5", | the sources supplied | +| solution.conditions | [{"id": "K-1", "on": ["S-1", "S-4", "S-7"], "condition": "The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during extern | the sources supplied | +| solution.build.duration_days | 36 | the sources supplied | +| solution.build.customer_owned | ["Order and provision the Interconnect carrier circuits"] | the sources supplied | +| operations.support_hours | 24x7 | the sources supplied | +| operations.availability_hours | 24x7 | the sources supplied | +| sourcing.model | provider_operated | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| compliance.frameworks | ["DORA", "ISO27001", "PCI_DSS_v4"] | the sources supplied | +| commercial.currency | GBP | the sources supplied | +| intent.business_outcomes | [{"outcome": "Continuous, highly available, and low-latency operation of the core payments platform foundation", "target": "99.9% foundation availability with deterministic low latency", "id": "BO-1"} | assumed | +| intent.exclusions | ["Core payment application workload code maintenance and runtime application management", "On-premises infrastructure hardware maintenance beyond Cloud Interconnect physical termination"] | assumed | +| operations.change_process | Changes to the platform foundation are deployed through automated GitHub Actions pipelines with mandatory peer reviews and automated policy validation, without a separate manual change advisory board. | assumed | +| organisation.existing_teams | [{"name": "Central Platform Operations Team", "hours": "24x7", "party": "provider"}] | assumed | +| service_management.tooling | Google Cloud operations suite (Cloud Monitoring, Cloud Logging, and Cloud Alerting) without a separate external ITSM system. | assumed | +| service_levels.priority_1_response_minutes | 30 | assumed | +| service_levels.recovery_time_hours | 4 | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.business_outcomes is [{'outcome': 'Continuous, highly available, and low-latency operation of the core payments platform foundation', 'target': '99.9% foundation availability with deterministic low latency', 'id': 'BO-1'}, {'id': 'BO-2', 'target': 'Zero audit non-conformances', 'outcome': 'Continuous compliance with DORA and PCI-DSS v4.0 standards using customer-managed encryption keys and centralised egress inspection'}, {'outcome': 'Automated, secure infrastructure release pipelines enabling workload ingestion within 10 weeks', 'target': 'Operational readiness achieved within 10 weeks', 'id': 'BO-3'}]. Derived from workflow request objectives: running core payment processing reliably in Google Cloud, meeting DORA and PCI-DSS v4.0 regulatory standards, and delivering landing zone readiness within 10 weeks. | Changes what is measured, and what each measure is for. | +| ASSUMP-02 | Assumed intent.exclusions is ['Core payment application workload code maintenance and runtime application management', 'On-premises infrastructure hardware maintenance beyond Cloud Interconnect physical termination']. Platform foundation scope excludes application-level core payment workloads and on-premises data centre hardware beyond the Interconnect boundary. | Changes which Capabilities are designed. | +| ASSUMP-03 | Assumed operations.change_process is Changes to the platform foundation are deployed through automated GitHub Actions pipelines with mandatory peer reviews and automated policy validation, without a separate manual change advisory board.. Catalogue default aligned with request stating GitHub Actions delivery pipelines: automated pipeline deployment with peer review and automated checks. | Changes change enablement and release. | +| ASSUMP-04 | Assumed organisation.existing_teams is [{'name': 'Central Platform Operations Team', 'hours': '24x7', 'party': 'provider'}]. Catalogue default: all operational roles are newly provided under provider_operated model, with reference to the 24/7 central platform operations team in condition K-8. | Changes which Roles are new and which are filled. | +| ASSUMP-05 | Assumed service_management.tooling is Google Cloud operations suite (Cloud Monitoring, Cloud Logging, and Cloud Alerting) without a separate external ITSM system.. Catalogue default: Cloud provider native operations tooling (Cloud Monitoring, Cloud Logging) integrated with alerting. | Changes tooling and the processes written for it. | +| ASSUMP-06 | Assumed service_levels.priority_1_response_minutes is 30. Catalogue default response time for Priority 1 critical incidents under 24x7 operational support. | Changes on-call arrangements and staffing. | +| ASSUMP-07 | Assumed service_levels.recovery_time_hours is 4. Catalogue default aligned with condition K-6 specifying a 4-hour recovery time objective (RTO). | Changes continuity arrangements. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operational_requirements.md new file mode 100644 index 0000000..e83c36b --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/operational_requirements.md @@ -0,0 +1,94 @@ +## Operational requirements + +Operational requirements define what running the Google Cloud payments platform foundation must achieve, traced directly to customer statements, solution conditions, and agreed operating parameters. + +### Service level + +- **OR-1 (Critical incident response time)**: Priority 1 critical incidents on platform foundation components are acknowledged and responded to within 30 minutes (as assumed in `service_levels.priority_1_response_minutes`), 24 hours a day, 7 days a week. + *Achievement*: Ensures uninterrupted operational readiness and rapid containment for any critical fault affecting the payment foundation at any hour. + *Source*: `operations.support_hours`, `service_levels.priority_1_response_minutes`. + +- **OR-5 (Secondary region warm standby readiness)**: The secondary cloud region in europe-west1 is maintained in warm standby operational readiness to receive failover traffic at all times. + *Achievement*: Guarantees that regional infrastructure capacity and configurations are kept continuously primed for deterministic traffic cutover during primary region disruption. + *Source*: `K-2`. + +- **OR-7 (Operational readiness within delivery timeline)**: Platform foundation provisioning, configuration, and operational validation are completed to support core payments engine workload ingestion within 10 weeks. + *Achievement*: Requires platform operational governance, pipelines, and controls to be fully operationalised before application ingestion commences. + *Source*: `K-4`. + +- **OR-8 (Platform foundation monthly availability)**: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%, available 24 hours a day, 7 days a week. + *Achievement*: Establishes minimum allowable uptime for core platform services supporting payment transaction routing and processing. + *Source*: `K-5`, `operations.availability_hours`. + +- **OR-9 (Disaster recovery time objective)**: Platform foundation disaster recovery activation achieves a recovery time objective (RTO) of no more than 4 hours (as assumed in `service_levels.recovery_time_hours`). + *Achievement*: Ensures that severe disruption or primary region loss results in service restoration within an agreed maximum outage duration. + *Source*: `K-6`, `service_levels.recovery_time_hours`. + +- **OR-10 (Disaster recovery point objective)**: Disaster recovery processes achieve a recovery point objective (RPO) of no more than 1 hour of data loss. + *Achievement*: Constrains data loss across platform configuration and stateful foundation metadata during any disaster recovery invocation to under one hour. + *Source*: `K-7`. + +- **OR-13 (Hybrid connectivity latency)**: Hybrid connectivity between cloud and on-premises environments maintains deterministic low latency suitable for real-time payment transaction processing. + *Achievement*: Ensures that the dedicated private interconnect consistently delivers the transmission performance required for latency-critical payment workflows. + *Source*: `K-10`. + +### Compliance + +- **OR-4 (Regulatory and security audit alignment)**: Landing zone configuration and operational governance maintain demonstrable alignment with DORA operational resiliency guidelines, PCI-DSS v4.0 perimeter isolation standards, and ISO27001 security standards during external audit. + *Achievement*: Mandates that platform controls, egress inspection, and key lifecycles remain continuously auditable without findings across applicable regulatory frameworks. + *Source*: `K-1`, `compliance.frameworks`. + +- **OR-6 (Data residency boundaries)**: Data storage, cryptographic processing, and compliance log retention remain strictly resident within the United Kingdom and European Union. + *Achievement*: Enforces strict geographical boundaries on customer payment data, encryption keys, and immutable audit logs to satisfy regulatory sovereignty requirements. + *Source*: `K-3`. + +### Process + +- **OR-11 (Continuous event alerting and incident dispatch)**: Platform telemetry alert notifications trigger operational incident workflows immediately upon detection, 24 hours a day, 7 days a week. + *Achievement*: Ensures that system faults, anomalies, and threshold breaches automatically invoke incident response workflows without human polling delays. + *Source*: `K-8`. + +- **OR-12 (Financial governance and cost sign-off)**: Monthly cloud foundation expenditure does not exceed the agreed target budget allocation without prior operational review and formal sign-off. + *Achievement*: Enforces disciplined expenditure controls, budget threshold alerts, and management pre-approval for platform capacity expansions. + *Source*: `K-9`. + +- **OR-14 (Carbon-free energy deployment optimization)**: Deployments and operational workload executions optimize carbon-free energy utilization based on regional carbon-free energy (CFE) scores. + *Achievement*: Incorporates environmental sustainability into operational scheduling and deployment execution across Google Cloud regions. + *Source*: `K-11`. + +- **OR-15 (Continuous identity rightsizing)**: Access permissions across platform identities are continuously rightsized to least privilege based on automated access findings. + *Achievement*: Prevents privilege creep and removes unutilised administrative permissions systematically across all federated identities and service accounts. + *Source*: `K-12`. + +### Organisation + +- **OR-2 (Provider-operated operational delivery model)**: Platform foundation operations are delivered under a provider-operated model where external operational management is provided by the service provider. + *Achievement*: Defines operational ownership such that platform run, monitoring, and administrative duties are fulfilled by provider operational personnel. + *Source*: `sourcing.model`. + +- **OR-3 (Approved operational delivery locations)**: Platform operational support and administrative access are conducted exclusively from approved delivery countries: United Kingdom (GB), Poland (PL), and India (IN). + *Achievement*: Restricts operational access and personnel location strictly to vetted, approved jurisdictions for compliance and risk assurance. + *Source*: `sourcing.delivery_countries`. + +--- + +## What was not stated + +The customer requirements and operating specification leave several operational details unstated or assumed. If answered differently in future iterations, these items would directly alter the operational requirements, capability models, and staffing profiles. + +### Assumptions + +- **ASSUMP-01 (Business Outcomes)**: Assumed `intent.business_outcomes` comprises BO-1 (continuous, 99.9% availability with deterministic low latency), BO-2 (continuous compliance with DORA and PCI-DSS v4.0 with zero audit non-conformances), and BO-3 (automated release pipelines achieving operational readiness within 10 weeks). If wrong, the operational measures and the business objectives justifying the operational model will change. +- **ASSUMP-02 (Scope Exclusions)**: Assumed `intent.exclusions` excludes core payment application workload code maintenance and runtime application management, as well as customer on-premises data centre hardware beyond the dedicated Cloud Interconnect termination point. If wrong, the boundary of operational responsibilities and capabilities must be expanded to encompass application support and physical data centre operations. +- **ASSUMP-03 (Change Enablement Process)**: Assumed `operations.change_process` allows changes to be deployed through automated delivery pipelines with mandatory peer reviews and automated policy validation without a manual change advisory board. If the customer requires a formal, manual CAB approval process, deployment velocity and release lead times will increase significantly. +- **ASSUMP-04 (Existing Organisational Teams)**: Assumed `organisation.existing_teams` provides a central platform operations team functioning 24x7 under the provider model. If customer internal teams retain platform administration responsibilities, the sourcing model would become shared, altering role allocation and RACI boundaries. +- **ASSUMP-05 (Tooling Architecture)**: Assumed `service_management.tooling` relies natively on the Google Cloud operations suite (Cloud Monitoring, Cloud Logging, Cloud Alerting) without integrating an external enterprise ITSM platform. If an external ITSM system (e.g., ServiceNow, Jira Service Management) is required, integration engineering, alerting bridges, and dual-system processes must be incorporated. +- **ASSUMP-06 (Priority 1 Incident Response Time)**: Assumed `service_levels.priority_1_response_minutes` is 30 minutes under 24x7 operational support. If the customer mandates a tighter 15-minute response target, staffing depth, rota arrangements, and dedicated on-call allocations will increase. +- **ASSUMP-07 (Disaster Recovery RTO)**: Assumed `service_levels.recovery_time_hours` is 4 hours, matching condition K-6. If business impact analysis requires a lower RTO (e.g., 1 hour), hot standby automation, active-active topologies, and continuous data synchronisation mechanisms will be required. + +### Unstated items to decide + +1. **Monthly Operating Budget (`commercial.monthly_run_budget`)**: The customer has not provided a defined monthly operational expenditure ceiling for running the platform foundation. An explicit budget constraint would govern staffing tiers and tooling investments. +2. **Transition Go-Live Date (`transition.go_live_date`)**: While a 10-week foundation readiness duration is specified (`K-4`), the absolute calendar cutover date for production workload ingestion is unstated. This dictates the operational transition timeline and readiness gates. +3. **Operational Headcount Ceiling (`organisation.headcount_limit`)**: No limit was placed on the operational headcount required to deliver 24x7 coverage across the UK, Poland, and India. +4. **Inherited Solution Risks (`solution.inherited_risks`)**: The client has not identified pre-existing unverified architecture choices, carrier lead-time risks for the 10Gbps dedicated interconnect, or dependencies on on-premises firewall updates. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/target_operating_model.md new file mode 100644 index 0000000..bdedf36 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/target_operating_model.md @@ -0,0 +1,68 @@ +# Design rationale + +## Work Division +The operating model is built around a `provider_operated` sourcing model, as stated by the customer. The Service Provider takes on full operational responsibility for the enterprise-grade landing zone, managing all underlying capabilities such as incident response, monitoring, and platform engineering. The Customer team handles the core payments engine workload development and deployment, consuming the platform as a service and participating as consulted stakeholders for broader platform foundation configurations. + +## Role Placements and Levels +Operational support and administrative access are strictly confined to approved delivery countries (United Kingdom, Poland, and India). + +- **Platform Operations Engineer (R-1)**: Placed in Poland (PL) to provide 24x7 shift coverage. The 'engineer' level is chosen as they will primarily run established procedures, manage known faults, and handle routine foundational tasks. +- **Platform Security & Network Engineer (R-2)**: Placed in Poland (PL) for 24x7 coverage. Given the strict PCI-DSS v4.0 perimeter isolation, centralized egress inspection, and complex Dedicated Interconnect components, this role requires a 'senior' level to diagnose unfamiliar faults and lead the operational shift. +- **Platform Reliability Lead (R-3)**: Sited in the United Kingdom (GB) to align tightly with the customer's business hours and core teams. Set at the 'lead' level, this role is accountable for designing the ways of working and managing overall service reliability. +- **Platform Product Manager (R-4)**: Placed in the United Kingdom (GB) at the 'lead' level. This role acts as the primary interface for the platform's roadmap and is accountable for ensuring the foundation is fully provisioned and validated within the tight 10-week deadline. +- **Platform Enablement Engineer (R-5)**: Placed in the United Kingdom (GB) at the 'senior' level to actively pair with and facilitate the customer's application teams in adopting the GitHub Actions deployment pipelines. +- **Customer Stakeholder (R-6)**: Operates from the United Kingdom (GB) representing the customer organization for consultations and architectural sign-offs. + +## Revisions and Adjustments +- **Disaster Recovery Activation (OC-K6)**: To ensure a 4-hour RTO can be met at any time of day, the validation check for executing disaster recovery activation has been placed on the Incident Management capability (C-1), which operates 24x7, rather than the business-hours Service Continuity capability. +- **Measurements and Business Outcomes**: While business outcomes were not stated initially, we assumed outcomes based on the customer request (BO-1, BO-2, BO-3). The target operating model ties measures directly to these assumed outcomes to prove alignment: Developer experience and delivery metrics (DORA and SPACE) measure the ingestion pipelines (C-19), service-level metrics trace the core 99.9% availability requirement (C-15) and strict network latency (C-10), and regulatory audit non-conformances track the compliance requirement (C-21). + +## Assumptions +- **Effort Allocations**: We assume the foundation heavily utilizes GitHub Actions and Google Cloud native operations suite, lowering manual effort. If the actual footprint of automation is lower than anticipated, staffing and effort hours will need to be increased. +- **Backup Scope**: We assume backups primarily apply to stateful configurations (compliance logs, cryptographic materials), while stateless infrastructure can be rebuilt reliably via CI/CD. +- **Continuous Security and Monitoring**: We assume that continuous detection and response for security events and telemetry alerts require 24x7 operational coverage to satisfy DORA and PCI-DSS compliance, inherently matching the stated 24x7 support hours and impacting shift staffing needs. Since the figures in the drafted rate card are placeholders, actual costs will be verified once final provider rates are confirmed. + +## Responsibility matrix + +| Capability | Platform Operations Engineer | Platform Security & Network Engineer | Platform Reliability Lead | Platform Product Manager | Platform Enablement Engineer | Customer Stakeholder | +| --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | R | | A | | | C | +| Monitoring and alerting (C-2) | R | | A | | | C | +| Problem management (C-3) | | R | A | | | C | +| Service request fulfilment (C-4) | R | | A | | | C | +| Change enablement (C-5) | | R | A | | | C | +| Release and deployment (C-6) | | R | A | | | C | +| Configuration management (C-7) | | R | A | | | C | +| Backup and recovery (C-8) | R | | A | | | C | +| Service continuity (C-9) | | R | A | | | C | +| Capacity and performance management (C-10) | | R | A | | | C | +| Access management (C-11) | R | | A | | | C | +| Vulnerability and patch management (C-12) | | R | A | | | C | +| Security monitoring and response (C-13) | | R | A | | | C | +| Cost management (C-14) | | | | A/R | | C | +| Service level management (C-15) | | | A/R | | | C | +| Platform product management (C-16) | | | | A/R | | C | +| Self-service and golden paths (C-17) | | | | A | R | C | +| Enablement (C-18) | | | | A | R | C | +| Delivery and developer experience measurement (C-19) | | | | A | R | C | +| Supplier management (C-20) | | | A/R | | | C | +| Regulatory evidence and reporting (C-21) | | R | A | | | C | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Operations Engineer | Service Provider | PL | engineer | 24x7 | 125.0 h | 5.0 | 30,777 GBP | +| Platform Security & Network Engineer | Service Provider | PL | senior | 24x7 | 185.0 h | 5.0 | 41,640 GBP | +| Platform Reliability Lead | Service Provider | GB | lead | business | 20.0 h | 0.1 | 2,212 GBP | +| Platform Product Manager | Service Provider | GB | lead | business | 30.0 h | 0.2 | 3,318 GBP | +| Platform Enablement Engineer | Service Provider | GB | senior | business | 70.0 h | 0.5 | 5,899 GBP | +| Customer Stakeholder | Customer Platform Team | GB | lead | business | 0.0 h | 0.0 | not priced | + +10.8 people in all. Cost a month: 83,846 GBP. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/validation_results.md new file mode 100644 index 0000000..d711412 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r2/deliverables/validation_results.md @@ -0,0 +1,64 @@ +# Check verdicts + +All checks have been verified against the target operating model and capability model. + +- **OC-K1 (pass):** Capability C-21 (regulatory-evidence) exists to compile and retain audit evidence for DORA and PCI-DSS v4.0 assessments. Role R-2 is responsible for it. +- **OC-K2 (pass):** Capability C-9 (service-continuity) is tasked with maintaining runbooks and verifying the europe-west1 secondary region warm standby readiness. Role R-2 is responsible. +- **OC-K3 (pass):** Capability C-7 (configuration-management) audits actual environment configurations against data residency boundaries (UK and EU). Role R-2 is responsible. +- **OC-K4 (pass):** Capability C-16 (platform-product-management) is responsible for ensuring the platform foundation is provisioned and validated within 10 weeks. Role R-4 is responsible. +- **OC-K5 (pass):** Capability C-15 (service-level-management) is set up to track and report on the 99.9% availability requirement. Role R-3 is responsible. +- **OC-K6 (pass):** Capability C-1 (incident-management) operates 24x7 and takes responsibility for executing disaster recovery activation within the 4-hour RTO at any time of day. Role R-1 provides the 24x7 coverage. +- **OC-K7 (pass):** Capability C-8 (backup-and-recovery) ensures backups meet the 1-hour RPO and handles testing restores. Role R-1 is responsible. +- **OC-K8 (pass):** Capability C-2 (monitoring-and-alerting) operates 24x7 for immediate response to telemetry signals. Role R-1 provides 24x7 coverage. +- **OC-K9 (pass):** Capability C-14 (cost-management) monitors foundation spend against agreed budgets and manages sign-offs for anomalies. Role R-4 is responsible. +- **OC-K10 (pass):** Capability C-10 (capacity-and-performance) monitors Dedicated Interconnect latency and overall infrastructure capacity for payment processing performance. Role R-2 is responsible. +- **OC-K11 (pass):** Capability C-10 (capacity-and-performance) oversees optimization of workload placements for carbon-free energy (CFE). Role R-2 is responsible. +- **OC-K12 (pass):** Capability C-11 (access-management) executes continuous identity rightsizing based on automated findings. Role R-1 is responsible. +- **OC-OR1 (pass):** Capability C-1 (incident-management) explicitly offers 24x7 coverage to acknowledge priority 1 incidents within 30 minutes, supported by the 24x7 R-1 role. +- **OC-OR2 (pass):** Capability C-4 (service-request-fulfilment) represents non-fault operational management by the Service Provider. The Provider Operated sourcing model ensures external operational management. +- **OC-OR8 (pass):** Capability C-15 (service-level-management) tracks and reports on the 99.9% availability requirement on a monthly basis. Role R-3 is responsible. +- **OC-OR3-R1 (pass):** Role R-1 operates from Poland (PL), which is an approved delivery country. +- **OC-OR3-R2 (pass):** Role R-2 operates from Poland (PL), which is an approved delivery country. +- **OC-OR3-R3 (pass):** Role R-3 operates from the United Kingdom (GB), which is an approved delivery country. +- **OC-OR3-R4 (pass):** Role R-4 operates from the United Kingdom (GB), which is an approved delivery country. +- **OC-OR3-R5 (pass):** Role R-5 operates from the United Kingdom (GB), which is an approved delivery country. + +**Overall Verdict:** The target operating model as a whole passes. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Resource Hierarchy and Environment Boundary Manager (S-1) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Resource Hierarchy and Environment Boundary Manager (S-1) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Federated Identity and Role Governance Authority (S-6) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Federated Identity and Role Governance Authority (S-6) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Platform Telemetry and Operational Alerting Engine (S-8) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Platform Telemetry and Operational Alerting Engine (S-8) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Financial Governance and Budget Enforcement Monitor (S-9) fails at an hour faults are responded to (24x7). | pass | Platform Operations Engineer responds under C-1, and Platform Reliability Lead is answerable. | +| Financial Governance and Budget Enforcement Monitor (S-9) must be changed. | pass | Platform Security & Network Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | pass | Check OC-K1 on C-21 keeps it true. | +| Condition K-2 must be shown to hold: The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | pass | Check OC-K2 on C-9 keeps it true. | +| Condition K-3 must be shown to hold: Data residency is guaranteed within the United Kingdom and European Union. | pass | Check OC-K3 on C-7 keeps it true. | +| Condition K-4 must be shown to hold: Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Check OC-K4 on C-16 keeps it true. | +| Condition K-5 must be shown to hold: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Check OC-K5 on C-15 keeps it true. | +| Condition K-6 must be shown to hold: Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | pass | Check OC-K6 on C-1 keeps it true. | +| Condition K-7 must be shown to hold: Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | pass | Check OC-K7 on C-8 keeps it true. | +| Condition K-8 must be shown to hold: Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | pass | Check OC-K8 on C-2 keeps it true. | +| Condition K-9 must be shown to hold: Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | pass | Check OC-K9 on C-14 keeps it true. | +| Condition K-10 must be shown to hold: Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Check OC-K10 on C-10 keeps it true. | +| Condition K-11 must be shown to hold: Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Check OC-K11 on C-10 keeps it true. | +| Condition K-12 must be shown to hold: Permissions are continuously rightsized based on IAM Recommender findings. | pass | Check OC-K12 on C-11 keeps it true. | +| Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/capability_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/capability_model.md new file mode 100644 index 0000000..b6874e0 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/capability_model.md @@ -0,0 +1,49 @@ +# Assumptions + +- Operational efforts are estimated based on managing an enterprise-grade Google Cloud foundation requiring rigorous compliance alignments (DORA, PCI-DSS v4.0, ISO27001). +- Non-fault-responsive capabilities will operate within standard business hours, allowing the critical 24x7 support window to be fulfilled specifically by monitoring, security response, and incident management capabilities. + +# Capabilities + +| ID | Capability | Items Covered | Hours | Effort (hrs/mo) | Reason / Basis | +|---|---|---|---|---|---| +| **C-1** | Incident management | All foundation items | 24x7 | 40 | Handling service disruptions across all platform items. Provides 30-minute response capability for priority 1 issues 24x7. | +| **C-2** | Monitoring and alerting | All foundation items | 24x7 | 30 | Configuring, tuning, and maintaining telemetry pipelines and automated alerting rules on a continuous basis. | +| **C-3** | Problem management | All foundation items | business | 20 | Post-incident analysis and root cause remediation to protect the 99.9% availability target. | +| **C-4** | Service request fulfilment | All foundation items | business | 20 | Managing non-incident requests from customer teams regarding the foundation environment. | +| **C-5** | Change enablement | All foundation items | business | 15 | Reviewing infrastructure changes to ensure DORA/PCI-DSS/ISO27001 regulatory compliance. | +| **C-6** | Release and deployment | S-5 | business | 25 | Operating GitHub Actions CI/CD pipelines to ensure timely deployments and carbon-optimised workload placement. | +| **C-7** | Configuration management | All foundation items | business | 15 | Enforcing state definition parity across the active and warm standby regions, and auditing configurations. | +| **C-8** | Backup and recovery | S-1, S-4, S-7 | business | 15 | Configuring backup policies and regularly rehearsing restoration processes for Cloud Storage, KMS, and foundation state to meet RPO/RTO targets. | +| **C-9** | Service continuity | All foundation items | business | 10 | Planning and periodic testing of regional failovers to europe-west1 to validate readiness and timing. | +| **C-10** | Capacity and performance management | S-1, S-2, S-3 | business | 15 | Monitoring and optimising Interconnect bandwidth and firewall VM capacity to guarantee deterministic latency and scaling. | +| **C-11** | Access management | S-1, S-6 | business | 20 | Continuously applying automated IAM rightsizing recommendations and managing operational access from approved countries. | +| **C-12** | Vulnerability and patch management | S-1, S-3, S-6 | business | 25 | Identifying and remediating CVEs in the cloud environment and Palo Alto VM-Series firewalls to meet compliance standards. | +| **C-13** | Security monitoring and response | S-1, S-3, S-4, S-6, S-7 | 24x7 | 30 | Watching for and responding to security misconfigurations, anomalies, and attempts to breach residency or perimeter isolation boundaries around the clock. | +| **C-14** | Cost management | S-1, S-9 | business | 15 | Monitoring cloud expenditure against allocated budgets, and designing and operating automation to enforce hard financial limits and execution stops. | +| **C-15** | Service level management | S-1, S-2, S-8 | business | 10 | Aggregating and publishing reporting on 99.9% uptime, 30-minute response times, and network latency commitments. | +| **C-16** | Platform product management | S-1, S-5 | business | 20 | Prioritising features and technical debt to ensure the platform foundation supports the core payments engine developers effectively. | +| **C-17** | Self-service and golden paths | S-1, S-5 | business | 30 | Building reusable Terraform modules and GitHub Actions templates to enable developers to deploy securely by default. | +| **C-18** | Enablement | S-1, S-5 | business | 25 | Working directly with the payments engine team to coach them on the new GCP landing zone and CI/CD pipelines. | +| **C-19** | Delivery and developer experience measurement | S-1, S-5, S-8 | business | 10 | Surveying cognitive load and measuring flow metrics across platform deployment activities. | +| **C-20** | Supplier management | S-2, S-3 | business | 10 | Managing ongoing relationship and escalations with Interconnect carriers and Palo Alto Networks. | +| **C-21** | Regulatory evidence and reporting | S-1, S-4, S-7 | business | 20 | Continually capturing, preserving, and providing the documented audit evidence required to prove compliance with DORA, PCI-DSS v4.0, and ISO27001 to regulators. | + +# Sources + +The capabilities structured in this operational model draw on practices and principles from the following industry frameworks. + +*Note: All references and alignments to these frameworks are currently **unverified**.* + +- **ITIL 4**: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice. +- **Site Reliability Engineering: How Google Runs Production Systems**: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives. +- **ISO/IEC 20000-1**: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management. +- **Regulation (EU) 2022/2554, the Digital Operational Resilience Act (DORA)**: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents. +- **NIST Cybersecurity Framework**: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management. +- **DevOps Research and Assessment (DORA) software delivery metrics**: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Time to restore service. +- **ISO/IEC 27001**: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements. +- **FinOps Framework**: Allocation; Budgeting; Forecasting; Anomaly Management. +- **CNCF Platforms White Paper (platform as a product)**: Platform as a product; Measuring platform success; Platform capabilities; Self-service. +- **Team Topologies: Organizing Business and Technology Teams for Fast Flow**: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode. +- **The SPACE of Developer Productivity**: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow. +- **DevEx: What Actually Drives Productivity**: Feedback loops; Cognitive load; Flow state. \ No newline at end of file diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_model_guide.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_model_guide.md new file mode 100644 index 0000000..b983467 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_model_guide.md @@ -0,0 +1,288 @@ +# Operating model guide: Google Cloud Foundation for Core Payments Engine + +## Summary + +This operating model guide governs the operation of the Google Cloud platform foundation supporting the core payments engine, encompassing the landing zone resource hierarchy, dedicated hybrid Cloud Interconnect connectivity, centralised egress traffic inspection appliances, immutable compliance audit logging, automated deployment pipelines, federated identity authority, customer-managed encryption keys, operational telemetry, and financial budget monitors. The platform operates under a provider-operated sourcing model. The Platform Service Provider delivers platform engineering, telemetry, compliance governance, and security operations from delivery locations in the United Kingdom, Poland, and India. The Customer directs platform strategy and product direction through its platform owner, maintaining internal operations liaison and escalation pathways. External suppliers provide the underlying cloud infrastructure, telecommunications carrier circuits, and inspection appliances under provider supplier management. + +Operational support and infrastructure availability are maintained continuously twenty-four hours a day, seven days a week (24x7) for incident triage, telemetry alerting, and perimeter security response, alongside standard business-hours coverage for release deployment, configuration management, and platform enablement. With all validation checks and scenario walk-throughs passing successfully, the reader should look first at the operational assumptions and regulatory baselines documented in this guide—specifically the critical incident acknowledgement target, disaster recovery recovery objectives, and the rate card pricing status—to ensure formal alignment before operational onboarding. + +### Key figures + +- Operated: 9 Items, with 12 Conditions to keep true. +- Faults responded to: 24x7 hours. +- Capabilities: 21. +- People: 16.9. Cost a month: 116,409 GBP, from rate card version `draft-2026-10-13` (placeholder prices: an illustration). +- Scenario walk-throughs: 34 of 34 pass. +- Business outcomes: 3, 3 shown by a Measure. +- Findings for whoever owns the design: 0. + +## Business outcomes + +**BO-1: Deliver an enterprise-grade Google Cloud landing zone within 10 weeks to enable ingestion and operation of the core payments engine.** Target: Foundation ready in under 10 weeks. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Deployment Lead Time | outcome | dora-metrics | Under 1 hour | Delivery and developer experience measurement | Platform Engineering Lead | +| Developer Friction | leading | devex-framework | Low friction reported in monthly surveys | Delivery and developer experience measurement | Platform Engineering Lead | +| Team Satisfaction | guardrail | space-framework | Positive quarterly trend | Delivery and developer experience measurement | Platform Engineering Lead | + +**BO-2: Maintain continuous operational resilience and regulatory compliance with DORA and PCI-DSS v4.0 with platform availability of at least 99.9%.** Target: >=99.9% monthly availability and full regulatory audit alignment. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Platform Availability | service level | | >=99.9% | Service level management | Platform Engineering Lead | +| Time to Restore Service | outcome | dora-metrics | Under 4 hours | Service level management | Platform Engineering Lead | + +**BO-3: Ensure secure, low-latency hybrid transaction transport and zero-trust data protection using customer-managed encryption keys.** Target: Deterministic low-latency 10Gbps interconnect and 100% CMEK lifecycle coverage. + +| Measure | Layer | Framework | Target | Measured by | Answerable | +| --- | --- | --- | --- | --- | --- | +| Interconnect Latency | service level | | Deterministic low latency (<5ms) | Capacity and performance management | Platform Engineering Lead | +| CMEK Lifecycle Coverage | guardrail | | 100% coverage | Regulatory evidence and reporting | Compliance & Governance Analyst | + +## What is operated + +| Item | What it is | Kind | Product | From | +| --- | --- | --- | --- | --- | +| S-1 | Resource Hierarchy and Environment Boundary Manager | governance | Resource Manager | C-1 | +| S-2 | Dedicated Private Hybrid Connectivity Gateway | network | Cloud Interconnect | C-2 | +| S-3 | Centralised Outbound Traffic Inspection and Egress Boundary | service | Palo Alto Networks VM-Series | C-3 | +| S-4 | Centralised Immutable Audit Event and Compliance Log Repository | storage | Cloud Storage | C-4 | +| S-5 | Automated Infrastructure Orchestration and Deployment Engine | service | GitHub Actions | C-5 | +| S-6 | Federated Identity and Role Governance Authority | identity | IAM Workload Identity Federation | C-6 | +| S-7 | Cryptographic Key Lifecycle and Encryption Controller | keys | Cloud KMS | C-7 | +| S-8 | Platform Telemetry and Operational Alerting Engine | observability | Cloud Monitoring | C-8 | +| S-9 | Financial Governance and Budget Enforcement Monitor | billing | Cloud Billing | C-9 | + +What must be kept true while it is run: + +| Condition | On | What must hold | From | +| --- | --- | --- | --- | +| K-1 | S-1, S-4, S-7 | The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | NFR-8 | +| K-2 | S-1 | The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | NFR-9 | +| K-3 | S-1 | Data residency is guaranteed within the United Kingdom and European Union. | NFR-9 | +| K-4 | S-5 | Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | NFR-10 | +| K-5 | S-8 | Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | NFR-11 | +| K-6 | S-8 | Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | NFR-11 | +| K-7 | S-8 | Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | NFR-11 | +| K-8 | S-8 | Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | NFR-12 | +| K-9 | S-9 | Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | NFR-13 | +| K-10 | S-2 | Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | NFR-14 | +| K-11 | S-8 | Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | NFR-15 | +| K-12 | S-6 | Permissions are continuously rightsized based on IAM Recommender findings. | DR-OP-3 | + +## Operational requirements + +| Requirement | Kind | What must be achieved | From | +| --- | --- | --- | --- | +| OR-1 | service_level | A Priority 1 fault on any platform foundation Item is acknowledged within 30 minutes (as assumed in service_levels.priority_1_response_minutes), 24 hours a day, 7 days a week. | operations.support_hours, service_levels.priority_1_response_minutes | +| OR-2 | service_level | The platform foundation infrastructure maintains measured monthly service availability of at least 99.9%, evaluated over continuous 24x7 operating hours. | K-5, operations.availability_hours | +| OR-3 | service_level | The dedicated hybrid interconnect maintains deterministic low network latency suitable for real-time payment transaction processing. | K-10 | +| OR-4 | service_level | Disaster recovery activation achieves full operational restoration within a recovery time objective (RTO) of 4 hours (as assumed in service_levels.recovery_time_hours). | K-6, service_levels.recovery_time_hours | +| OR-5 | service_level | Disaster recovery failover achieves a recovery point objective (RPO) of no more than 1 hour of operational state or configuration loss. | K-7 | +| OR-6 | process | The secondary cloud region in europe-west1 is continuously maintained as an active warm standby ready to receive failover traffic immediately upon activation. | K-2 | +| OR-7 | process | The platform foundation is fully provisioned, validated, and ready for core payments engine workload ingestion within 10 weeks of initiation. | K-4 | +| OR-8 | process | Platform telemetry alert notifications automatically trigger operational incident response workflows without human delay, maintaining active 24x7 coverage. | K-8 | +| OR-9 | process | Monthly cloud foundation consumption expenditure is tracked against allocated budgets and prevented from exceeding limits without documented operational sign-off. | K-9 | +| OR-10 | process | Platform foundation deployments and workload placements optimize carbon-free energy utilization based on regional carbon-free energy scores. | K-11 | +| OR-11 | process | Access permissions across all foundation environments are continuously audited and rightsized based on automated access recommendations. | K-12 | +| OR-12 | organisation | Platform foundation operations and infrastructure maintenance are conducted under a provider-operated delivery model. | sourcing.model | +| OR-13 | organisation | All operational management and administrative access to the platform foundation are executed exclusively from delivery locations within the United Kingdom, Poland, and India. | sourcing.delivery_countries | +| OR-14 | compliance | Platform foundation configurations and audit mechanisms maintain continuous alignment with DORA operational resiliency guidelines, PCI-DSS v4.0 perimeter isolation standards, and ISO27001 information security controls during external audit. | K-1, compliance.frameworks | +| OR-15 | compliance | Data residency for all platform foundation storage, audit logs, and cryptographic keys is guaranteed within the United Kingdom and European Union. | K-3 | + +## Capabilities + +| Capability | What it is | Serves | Covers | Hours | Effort a month | +| --- | --- | --- | --- | --- | --- | +| C-1 | Incident management | OR-1, OR-2, OR-8 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | 24x7 | 40 h | +| C-2 | Monitoring and alerting | OR-1, OR-2, OR-3, OR-8 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | 24x7 | 30 h | +| C-3 | Problem management | OR-2 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-4 | Service request fulfilment | OR-12 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 20 h | +| C-5 | Change enablement | OR-7, OR-14 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-6 | Release and deployment | OR-7, OR-10 | S-5 | business | 25 h | +| C-7 | Configuration management | OR-6, OR-14 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 15 h | +| C-8 | Backup and recovery | OR-4, OR-5 | S-1, S-4, S-7 | business | 15 h | +| C-9 | Service continuity | OR-4, OR-5, OR-6 | S-1, S-2, S-3, S-4, S-5, S-6, S-7, S-8, S-9 | business | 10 h | +| C-10 | Capacity and performance management | OR-2, OR-3 | S-1, S-2, S-3 | business | 15 h | +| C-11 | Access management | OR-11, OR-13 | S-1, S-6 | business | 20 h | +| C-12 | Vulnerability and patch management | OR-14 | S-1, S-3, S-6 | business | 25 h | +| C-13 | Security monitoring and response | OR-14, OR-15 | S-1, S-3, S-4, S-6, S-7 | 24x7 | 30 h | +| C-14 | Cost management | OR-9 | S-1, S-9 | business | 15 h | +| C-15 | Service level management | OR-1, OR-2, OR-3 | S-1, S-2, S-8 | business | 10 h | +| C-16 | Platform product management | OR-7, OR-12 | S-1, S-5 | business | 20 h | +| C-17 | Self-service and golden paths | OR-7, OR-12 | S-1, S-5 | business | 30 h | +| C-18 | Enablement | OR-7, OR-12 | S-1, S-5 | business | 25 h | +| C-19 | Delivery and developer experience measurement | OR-7, OR-12 | S-1, S-5, S-8 | business | 10 h | +| C-20 | Supplier management | OR-12, OR-13 | S-2, S-3 | business | 10 h | +| C-21 | Regulatory evidence and reporting | OR-14, OR-15 | S-1, S-4, S-7 | business | 20 h | + +## Who does what + +| Role | Party | Country | Level | Hours | +| --- | --- | --- | --- | --- | +| Platform Operations Engineer | Platform Service Provider (provider) | IN | engineer | 24x7 | +| Platform Reliability Engineer | Platform Service Provider (provider) | PL | senior | 24x7 | +| Security Operations Analyst | Platform Service Provider (provider) | PL | senior | 24x7 | +| Platform Automation Engineer | Platform Service Provider (provider) | PL | senior | business | +| Platform Engineering Lead | Platform Service Provider (provider) | GB | lead | business | +| Compliance & Governance Analyst | Platform Service Provider (provider) | GB | senior | business | +| Customer Platform Owner | Customer (customer) | GB | lead | business | +| Customer Operations Liaison | Customer (customer) | GB | senior | 24x7 | + +The teams, as Team Topologies names their kinds: + +| Team | Kind | Party | Roles | Purpose | +| --- | --- | --- | --- | --- | +| Cloud Platform Foundation Team | platform | Platform Service Provider | Platform Operations Engineer, Platform Reliability Engineer, Platform Automation Engineer, Platform Engineering Lead | Builds, maintains, and reliably operates the enterprise Google Cloud landing zone. | +| Platform Security Team | complicated-subsystem | Platform Service Provider | Security Operations Analyst | Provides 24x7 security monitoring, egress inspection analysis, and vulnerability management. | +| Customer Central Platform Operations Team | stream-aligned | Customer | Customer Operations Liaison | Receives incident workflows and serves as the internal operations escalation point. | +| Customer Platform Owner Team | stream-aligned | Customer | Customer Platform Owner | Provides business context and sets the strategic direction for the platform. | +| Platform Governance Team | enabling | Platform Service Provider | Compliance & Governance Analyst | Ensures continuous regulatory compliance with DORA, PCI-DSS v4.0, and ISO27001. | + +How the teams work together: + +| From | To | Mode | Until | Purpose | +| --- | --- | --- | --- | --- | +| Customer Central Platform Operations Team | Cloud Platform Foundation Team | x-as-a-service | | Customer Central Platform Operations consumes incident workflows and alert notifications from the Provider Cloud Platform Foundation. | +| Cloud Platform Foundation Team | Platform Security Team | collaboration | PCI-DSS baseline controls are fully automated and verified by external audit, after which they revert to x_as_a_service. | Jointly establish automated security response runbooks and egress inspection rules for PCI-DSS compliance. | +| Platform Governance Team | Cloud Platform Foundation Team | facilitating | | Assisting the platform team in integrating regulatory evidence collection into the automated deployment engine. | + +| Capability | Platform Operations Engineer | Platform Reliability Engineer | Security Operations Analyst | Platform Automation Engineer | Platform Engineering Lead | Compliance & Governance Analyst | Customer Platform Owner | Customer Operations Liaison | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | R | | | A | | | C | +| Monitoring and alerting (C-2) | R | | | | A | | | C | +| Problem management (C-3) | | | | R | A | | | C | +| Service request fulfilment (C-4) | | | | R | A | | | C | +| Change enablement (C-5) | | | | R | A | R | | C | +| Release and deployment (C-6) | | | | R | A | | | C | +| Configuration management (C-7) | | | | R | A | | | C | +| Backup and recovery (C-8) | | | | R | A | | | C | +| Service continuity (C-9) | | R | | | A | | | C | +| Capacity and performance management (C-10) | | | | R | A | | | C | +| Access management (C-11) | | | A/R | | | | | C | +| Vulnerability and patch management (C-12) | | | A/R | | | | | C | +| Security monitoring and response (C-13) | | | A/R | | | | | C | +| Cost management (C-14) | | | | | A | R | | C | +| Service level management (C-15) | | | | | A | R | | C | +| Platform product management (C-16) | | | | | A/R | | C | | +| Self-service and golden paths (C-17) | | | | R | A | | | C | +| Enablement (C-18) | | | | R | A | | | C | +| Delivery and developer experience measurement (C-19) | | | | R | A/R | | | C | +| Supplier management (C-20) | | | | | A | R | | C | +| Regulatory evidence and reporting (C-21) | | | | | | A/R | C | | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Operations Engineer | Platform Service Provider | IN | engineer | 24x7 | 30.0 h | 5.0 | 17,034 GBP | +| Platform Reliability Engineer | Platform Service Provider | PL | senior | 24x7 | 50.0 h | 5.0 | 41,640 GBP | +| Security Operations Analyst | Platform Service Provider | PL | senior | 24x7 | 75.0 h | 5.0 | 41,640 GBP | +| Platform Automation Engineer | Platform Service Provider | PL | senior | business | 177.5 h | 1.2 | 8,063 GBP | +| Platform Engineering Lead | Platform Service Provider | GB | lead | business | 25.0 h | 0.2 | 2,765 GBP | +| Compliance & Governance Analyst | Platform Service Provider | GB | senior | business | 62.5 h | 0.5 | 5,267 GBP | +| Customer Platform Owner | Customer | GB | lead | business | 0.0 h | 0.0 | not priced | +| Customer Operations Liaison | Customer | GB | senior | business | 0.0 h | 0.0 | not priced | + +16.9 people in all. Cost a month: 116,409 GBP. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. + +## Validation + +| Check | What must hold | Verdict | Evidence | +| --- | --- | --- | --- | +| OC-1 | The landing zone configurations are continuously enforced to align with DORA, PCI-DSS v4.0, and ISO27001 standard baselines. | pass | Capability C-7 provides configuration management, enforcing state definition parity and auditing configurations, and supports ensuring compliance standards like DORA, PCI-DSS v4.0, and ISO27001 are continuously enforced. Role R-4 is responsible for it. | +| OC-2 | The europe-west1 secondary region is continuously maintained as an active warm standby. | pass | Capability C-9 handles service continuity, which includes planning and periodic testing of regional failovers, ensuring the secondary region is maintained as an active warm standby. Role R-2 is responsible for it. | +| OC-3 | Configuration enforces that all data residency, audit logs, and keys remain strictly within the UK and EU. | pass | Capability C-7 enforces configuration state parity and audits configurations, satisfying the condition that data residency, logs, and keys remain in the UK and EU. | +| OC-4 | The foundation platform is verified as fully provisioned and ready for core payments engine workload ingestion within 10 weeks. | pass | Capability C-16 prioritises features and technical debt to ensure the platform foundation supports the payments engine developers effectively, meeting the 10-week provisioning timeframe. R-5 is responsible. | +| OC-5 | Continuous telemetry verifies that the platform foundation infrastructure maintains at least 99.9% monthly availability. | pass | Capability C-2 provides continuous monitoring and alerting 24x7, maintaining telemetry pipelines to verify the infrastructure maintains 99.9% availability. | +| OC-6 | Disaster recovery failover tests verify that the 4-hour RTO can be met. | pass | Capability C-9 handles disaster recovery failover tests to validate readiness and timing to ensure the 4-hour RTO can be met. | +| OC-7 | Backup and restore tests verify that the 1-hour RPO for compliance logs and KMS keys is met. | pass | Capability C-8 rehearses restoration processes for Cloud Storage and KMS to verify the 1-hour RPO is met. | +| OC-8 | Alert notifications from platform telemetry automatically trigger incident workflows for the 24x7 central platform operations team without human delay. | pass | Capability C-2 configures and maintains automated alerting rules that trigger workflows to the 24x7 customer central platform operations team (R-8 is consulted). | +| OC-9 | Cloud consumption expenditure is tracked against budgets, and automation enforces hard limits that prevent overages without operational sign-off. | pass | Capability C-14 designs and operates automation to enforce hard financial limits and execution stops, tracking cloud consumption against budgets. | +| OC-10 | The dedicated hybrid interconnect bandwidth and latency are monitored to ensure deterministic low latency for real-time payments. | pass | Capability C-10 monitors and optimises the dedicated hybrid interconnect bandwidth to guarantee deterministic low latency for real-time payments. | +| OC-11 | Workload placements in the deployment pipeline are optimized for carbon-free energy utilization. | pass | Capability C-6 operates the deployment pipelines to ensure timely deployments and carbon-optimised workload placement. | +| OC-12 | Access permissions are continuously audited and rightsized based on IAM recommender alerts. | pass | Capability C-11 continuously applies automated IAM rightsizing recommendations and manages access from approved countries. | +| OC-13 | A Priority 1 fault on the platform foundation is acknowledged within 30 minutes, at any time of day or night. | pass | Capability C-1 provides 24x7 incident management and 30-minute response capability for Priority 1 faults. | +| OC-14 | The platform foundation is led and managed by a provider-operated delivery model. | pass | Role R-5 (Platform Engineering Lead) sits within the Provider (P-2) party, managing and directing the platform strategy and operations. | +| OC-15 | Operational management is performed exclusively by staff located in the UK, Poland, or India. | pass | All provider operational staff (R-1, R-2, R-3, R-4, R-5, R-6) are based exclusively in the UK, Poland, or India, fulfilling the sourcing requirement. | +| OC-16 | Evidence of continuous alignment with DORA, PCI-DSS v4.0, and ISO27001 is gathered for external audit. | pass | Capability C-21 gathers and preserves documented audit evidence to prove compliance with DORA, PCI-DSS v4.0, and ISO27001 to external regulators. | +| OC-17 | The UK and EU data residency guarantee is evidenced for all storage, audit logs, and keys. | pass | Capability C-21 captures, preserves, and provides documented audit evidence to external regulators, which includes proving data residency constraints. | + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Resource Hierarchy and Environment Boundary Manager (S-1) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Resource Hierarchy and Environment Boundary Manager (S-1) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Federated Identity and Role Governance Authority (S-6) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Federated Identity and Role Governance Authority (S-6) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Platform Telemetry and Operational Alerting Engine (S-8) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Platform Telemetry and Operational Alerting Engine (S-8) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Financial Governance and Budget Enforcement Monitor (S-9) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Financial Governance and Budget Enforcement Monitor (S-9) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | pass | Check OC-1 on C-7 keeps it true. | +| Condition K-2 must be shown to hold: The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | pass | Check OC-2 on C-9 keeps it true. | +| Condition K-3 must be shown to hold: Data residency is guaranteed within the United Kingdom and European Union. | pass | Check OC-3 on C-7 keeps it true. | +| Condition K-4 must be shown to hold: Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Check OC-4 on C-16 keeps it true. | +| Condition K-5 must be shown to hold: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Check OC-5 on C-2 keeps it true. | +| Condition K-6 must be shown to hold: Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | pass | Check OC-6 on C-9 keeps it true. | +| Condition K-7 must be shown to hold: Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | pass | Check OC-7 on C-8 keeps it true. | +| Condition K-8 must be shown to hold: Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | pass | Check OC-8 on C-2 keeps it true. | +| Condition K-9 must be shown to hold: Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | pass | Check OC-9 on C-14 keeps it true. | +| Condition K-10 must be shown to hold: Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Check OC-10 on C-10 keeps it true. | +| Condition K-11 must be shown to hold: Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Check OC-11 on C-6 keeps it true. | +| Condition K-12 must be shown to hold: Permissions are continuously rightsized based on IAM Recommender findings. | pass | Check OC-12 on C-11 keeps it true. | +| Platform Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| Google Cloud cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| Network Carrier cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| External Firewall Vendor cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | + +## Findings and open questions + +No findings: the solution provides what operating it needs. + +Assumed for the customer, and open to change when they answer: + +- `intent.business_outcomes`: {'target': 'Foundation ready in under 10 weeks', 'id': 'BO-1', 'outcome': 'Deliver an enterprise-grade Google Cloud landing zone within 10 weeks to enable ingestion and operation of the core payments engine.'}, {'outcome': 'Maintain continuous operational resilience and regulatory compliance with DORA and PCI-DSS v4.0 with platform availability of at least 99.9%.', 'target': '>=99.9% monthly availability and full regulatory audit alignment', 'id': 'BO-2'}, {'target': 'Deterministic low-latency 10Gbps interconnect and 100% CMEK lifecycle coverage', 'id': 'BO-3', 'outcome': 'Ensure secure, low-latency hybrid transaction transport and zero-trust data protection using customer-managed encryption keys.'}. Derived from the workflow request stating the need for an enterprise-grade landing zone ready in under 10 weeks to support the core payments engine, complying with DORA and PCI-DSS v4.0, with secure hybrid connectivity and customer-controlled encryption. +- `intent.exclusions`: Core payments engine application-level development, maintenance, and transaction processing logic, On-premises data centre infrastructure and systems beyond the Cloud Interconnect demarcation, Physical telecom carrier circuit ordering and provisioning (customer-owned build item). Derived from engagement kind (platform foundation) and workflow request: application-level workloads, on-prem systems, and physical circuit provisioning are excluded from platform operations. +- `operations.change_process`: Automated CI/CD pipelines via GitHub Actions with peer review and automated policy checks; standard operational changes deployed without a separate manual CAB.. Derived from request stating GitHub Actions delivery pipelines and catalogue default guidance. +- `organisation.existing_teams`: {'party': 'customer', 'name': 'Central Platform Operations Team', 'hours': '24x7'}. Derived from condition K-8 noting alerting to the 24/7 central platform operations team, alongside provider-operated landing zone model. +- `service_management.tooling`: Google Cloud operations suite (Cloud Monitoring and Cloud Logging) integrated with native alerting workflows and GitHub Actions.. Catalogue default guidance leveraging Google Cloud operations suite and GitHub Actions specified in request. +- `service_levels.priority_1_response_minutes`: 30. Standard catalogue default for 24x7 critical infrastructure response. +- `service_levels.recovery_time_hours`: 4. Matches condition K-6 disaster recovery RTO requirement of 4 hours and catalogue default. + +## Sources + +The Capabilities are from our own capability catalogue, which draws on these frameworks. It cites them and certifies nothing against them. + +- ITIL 4: Incident management practice; Monitoring and event management practice; Problem management practice; Service request management practice; Change enablement practice; Release management and Deployment management practices; Service configuration management practice; Service continuity management practice; Capacity and performance management practice; Service level management practice; Supplier management practice +- Site Reliability Engineering: How Google Runs Production Systems: Managing Incidents; Monitoring Distributed Systems; Postmortem Culture: Learning from Failure; Release Engineering; Service Level Objectives +- ISO/IEC 20000-1: Incident management; Service request management; Change management; Service continuity management; Capacity management; Service level management; Supplier management +- Regulation (EU) 2022/2554, the Digital Operational Resilience Act: Article 17, ICT-related incident management process; Article 10, Detection; Article 12, Backup policies and procedures, restoration and recovery procedures and methods; Article 11, Response and recovery; Article 9, Protection and prevention; Article 28, General principles on ICT third-party risk; Article 19, Reporting of major ICT-related incidents +- NIST Cybersecurity Framework: Detect: Continuous Monitoring; Identify: Asset Management; Recover: Incident Recovery Plan Execution; Protect: Identity Management, Authentication, and Access Control; Protect: Platform Security; Detect: Adverse Event Analysis; Respond: Incident Management +- DevOps Research and Assessment (DORA) software delivery metrics: Change failure rate; Deployment frequency and lead time for changes; Deployment frequency; Lead time for changes; Change failure rate; Time to restore service +- ISO/IEC 27001: Access control; Management of technical vulnerabilities; Information security incident management; Compliance with legal and contractual requirements +- FinOps Framework: Allocation; Budgeting; Forecasting; Anomaly Management +- CNCF Platforms White Paper (platform as a product): Platform as a product; Measuring platform success; Platform capabilities; Self-service; Platform as a product +- Team Topologies: Organizing Business and Technology Teams for Fast Flow: Platform team; Thinnest viable platform; X-as-a-Service interaction mode; Enabling team; Facilitating interaction mode +- The SPACE of Developer Productivity: Satisfaction and well-being; Performance; Activity; Communication and collaboration; Efficiency and flow +- DevEx: What Actually Drives Productivity: Feedback loops; Cognitive load; Flow state + +**These references have not been checked against the frameworks by a person.** Treat each as a pointer to where to look. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_scope.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_scope.md new file mode 100644 index 0000000..f5f54ca --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operating_scope.md @@ -0,0 +1,44 @@ +# Operating Scope + +## Scope statement + +This operating scope covers the 24x7 operation of the Google Cloud platform foundation supporting the customer's core payments engine, encompassing the resource hierarchy, dedicated hybrid Cloud Interconnect, centralised egress inspection appliances, compliance audit logging, automated deployment pipelines, federated identity authority, customer-managed encryption key controllers, operational telemetry, and cloud billing monitors. The platform is operated under a provider-operated sourcing model from delivery locations in the United Kingdom, Poland, and India, maintaining continuous 24x7 operational support and 24x7 system availability. Operational workflows adhere to DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter standards, targeting a 30-minute priority-1 incident response and a 4-hour disaster recovery restoration objective. All operating costs and commercial accounting for this engagement are stated in British Pounds (GBP). Explicitly excluded from platform operations are core payments engine application-level development and transaction processing logic, existing on-premises infrastructure beyond the Cloud Interconnect demarcation, and physical telecommunications carrier circuit procurement. + +## Information register + +3 stated by the customer, 13 digested from the sources supplied, 0 answered in conversation, 7 assumed, 10 unknown, 0 not applicable. + +| Field | Value | From | +| --- | --- | --- | +| engagement.kind | platform_foundation | the sources supplied | +| intent.statement | We are a UK-regulated payment platform expanding into Google Cloud. We need an enterprise-grade landing zone ready in under 10 weeks to support our core payments engine. All payment processing must co | the sources supplied | +| solution.cloud_provider | gcp | the sources supplied | +| solution.items | [{"id": "S-1", "name": "Resource Hierarchy and Environment Boundary Manager", "kind": "governance", "product": "Resource Manager", "source_ref": "C-1", "elements": ["E-1", "E-2", "E-3", "E-4", "E-5", | the sources supplied | +| solution.conditions | [{"id": "K-1", "on": ["S-1", "S-4", "S-7"], "condition": "The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during extern | the sources supplied | +| solution.build.duration_days | 36 | the sources supplied | +| solution.build.customer_owned | ["Order and provision the Interconnect carrier circuits"] | the sources supplied | +| operations.support_hours | 24x7 | the sources supplied | +| operations.availability_hours | 24x7 | the sources supplied | +| sourcing.model | provider_operated | the scope spec | +| sourcing.delivery_countries | ["GB", "PL", "IN"] | the scope spec | +| compliance.frameworks | ["DORA", "ISO27001", "PCI_DSS_v4"] | the sources supplied | +| commercial.currency | GBP | the sources supplied | +| intent.business_outcomes | [{"target": "Foundation ready in under 10 weeks", "id": "BO-1", "outcome": "Deliver an enterprise-grade Google Cloud landing zone within 10 weeks to enable ingestion and operation of the core payments | assumed | +| intent.exclusions | ["Core payments engine application-level development, maintenance, and transaction processing logic", "On-premises data centre infrastructure and systems beyond the Cloud Interconnect demarcation", "P | assumed | +| operations.change_process | Automated CI/CD pipelines via GitHub Actions with peer review and automated policy checks; standard operational changes deployed without a separate manual CAB. | assumed | +| organisation.existing_teams | [{"party": "customer", "name": "Central Platform Operations Team", "hours": "24x7"}] | assumed | +| service_management.tooling | Google Cloud operations suite (Cloud Monitoring and Cloud Logging) integrated with native alerting workflows and GitHub Actions. | assumed | +| service_levels.priority_1_response_minutes | 30 | assumed | +| service_levels.recovery_time_hours | 4 | assumed | + +### Assumptions + +| Id | Assumption | If it is wrong | +| --- | --- | --- | +| ASSUMP-01 | Assumed intent.business_outcomes is [{'target': 'Foundation ready in under 10 weeks', 'id': 'BO-1', 'outcome': 'Deliver an enterprise-grade Google Cloud landing zone within 10 weeks to enable ingestion and operation of the core payments engine.'}, {'outcome': 'Maintain continuous operational resilience and regulatory compliance with DORA and PCI-DSS v4.0 with platform availability of at least 99.9%.', 'target': '>=99.9% monthly availability and full regulatory audit alignment', 'id': 'BO-2'}, {'target': 'Deterministic low-latency 10Gbps interconnect and 100% CMEK lifecycle coverage', 'id': 'BO-3', 'outcome': 'Ensure secure, low-latency hybrid transaction transport and zero-trust data protection using customer-managed encryption keys.'}]. Derived from the workflow request stating the need for an enterprise-grade landing zone ready in under 10 weeks to support the core payments engine, complying with DORA and PCI-DSS v4.0, with secure hybrid connectivity and customer-controlled encryption. | Changes what is measured, and what each measure is for. | +| ASSUMP-02 | Assumed intent.exclusions is ['Core payments engine application-level development, maintenance, and transaction processing logic', 'On-premises data centre infrastructure and systems beyond the Cloud Interconnect demarcation', 'Physical telecom carrier circuit ordering and provisioning (customer-owned build item)']. Derived from engagement kind (platform foundation) and workflow request: application-level workloads, on-prem systems, and physical circuit provisioning are excluded from platform operations. | Changes which Capabilities are designed. | +| ASSUMP-03 | Assumed operations.change_process is Automated CI/CD pipelines via GitHub Actions with peer review and automated policy checks; standard operational changes deployed without a separate manual CAB.. Derived from request stating GitHub Actions delivery pipelines and catalogue default guidance. | Changes change enablement and release. | +| ASSUMP-04 | Assumed organisation.existing_teams is [{'party': 'customer', 'name': 'Central Platform Operations Team', 'hours': '24x7'}]. Derived from condition K-8 noting alerting to the 24/7 central platform operations team, alongside provider-operated landing zone model. | Changes which Roles are new and which are filled. | +| ASSUMP-05 | Assumed service_management.tooling is Google Cloud operations suite (Cloud Monitoring and Cloud Logging) integrated with native alerting workflows and GitHub Actions.. Catalogue default guidance leveraging Google Cloud operations suite and GitHub Actions specified in request. | Changes tooling and the processes written for it. | +| ASSUMP-06 | Assumed service_levels.priority_1_response_minutes is 30. Standard catalogue default for 24x7 critical infrastructure response. | Changes on-call arrangements and staffing. | +| ASSUMP-07 | Assumed service_levels.recovery_time_hours is 4. Matches condition K-6 disaster recovery RTO requirement of 4 hours and catalogue default. | Changes continuity arrangements. | diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operational_requirements.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operational_requirements.md new file mode 100644 index 0000000..5e6df84 --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/operational_requirements.md @@ -0,0 +1,87 @@ +# Operational Requirements + +This document establishes the operational requirements for running the Google Cloud platform foundation supporting the core payments engine. Each requirement defines an outcome that platform operations must achieve, traced directly to the customer's stated architecture conditions, regulatory mandates, and agreed operating parameters. + +## Operational requirements + +The operational requirements are organised below by kind, detailing the verifiable target state required of operations without prescribing implementation capabilities, staffing roles, or specific software tooling. + +### Service levels + +- **OR-1**: A Priority 1 fault on any platform foundation Item is acknowledged within 30 minutes (as assumed in service_levels.priority_1_response_minutes), 24 hours a day, 7 days a week. + *This ensures critical operational incidents impacting the foundation infrastructure receive immediate operational attention at any hour of the day or night.* +- **OR-2**: The platform foundation infrastructure maintains measured monthly service availability of at least 99.9%, evaluated over continuous 24x7 operating hours. + *This establishes the minimum continuous uptime threshold for core foundation landing zone components to support payments processing.* +- **OR-3**: The dedicated hybrid interconnect maintains deterministic low network latency suitable for real-time payment transaction processing. + *This ensures that hybrid connectivity between cloud infrastructure and on-premises systems provides bounded latency suitable for transaction settlement.* +- **OR-4**: Disaster recovery activation achieves full operational restoration within a recovery time objective (RTO) of 4 hours (as assumed in service_levels.recovery_time_hours). + *This defines the maximum allowable elapsed time between disaster invocation and full recovery of foundation services in the secondary region.* +- **OR-5**: Disaster recovery failover achieves a recovery point objective (RPO) of no more than 1 hour of operational state or configuration loss. + *This sets the upper limit on data or configuration drift permitted during an unplanned failover event.* + +### Processes + +- **OR-6**: The secondary cloud region in europe-west1 is continuously maintained as an active warm standby ready to receive failover traffic immediately upon activation. + *This requires ongoing configuration parity and health synchronization so that standby infrastructure can accept live traffic without pre-activation provisioning delays.* +- **OR-7**: The platform foundation is fully provisioned, validated, and ready for core payments engine workload ingestion within 10 weeks of initiation. + *This bounds the operational transition and readiness verification timeline to enable dependent payment engine deployment milestones.* +- **OR-8**: Platform telemetry alert notifications automatically trigger operational incident response workflows without human delay, maintaining active 24x7 coverage. + *This mandates end-to-end telemetry event ingestion and instant incident workflow triggering at all times.* +- **OR-9**: Monthly cloud foundation consumption expenditure is tracked against allocated budgets and prevented from exceeding limits without documented operational sign-off. + *This enforces proactive financial governance and budget alerting across platform foundation infrastructure usage.* +- **OR-10**: Platform foundation deployments and workload placements optimize carbon-free energy utilization based on regional carbon-free energy scores. + *This operationalizes sustainability targets by aligning non-real-time compute deployment and operational jobs with regional carbon-free energy availability.* +- **OR-11**: Access permissions across all foundation environments are continuously audited and rightsized based on automated access recommendations. + *This mandates continuous privilege hygiene and automated reduction of excessive IAM permissions across all platform boundaries.* + +### Organisation + +- **OR-12**: Platform foundation operations and infrastructure maintenance are conducted under a provider-operated delivery model. + *This sets the boundary of operational accountability, requiring day-to-day platform foundation operations to be delivered by the external service provider.* +- **OR-13**: All operational management and administrative access to the platform foundation are executed exclusively from delivery locations within the United Kingdom, Poland, and India. + *This establishes geographic boundaries for operational access, ensuring adherence to sovereign and client-approved delivery locations.* + +### Compliance + +- **OR-14**: Platform foundation configurations and audit mechanisms maintain continuous alignment with DORA operational resiliency guidelines, PCI-DSS v4.0 perimeter isolation standards, and ISO27001 information security controls during external audit. + *This requires configuration state, logging immutability, and boundary controls to remain auditable and compliant with external financial and security regulations.* +- **OR-15**: Data residency for all platform foundation storage, audit logs, and cryptographic keys is guaranteed within the United Kingdom and European Union. + *This ensures that all operational log archives, key hierarchies, and platform state configurations never leave approved UK and EU geographical perimeters.* + +--- + +## What was not stated + +The operational scope register contains several fields that were assumed or remain unstated in customer inputs. These assumptions directly influence the operational requirements defined above; if answered differently by the customer, they would reshape the operational model. + +### Assumptions + +- **service_levels.priority_1_response_minutes (Assumed: 30 minutes)**: + Assumed based on catalogue standard defaults for 24x7 critical financial platform operations. If the customer requires a faster response (e.g., 15 minutes) or accepts a slower response (e.g., 60 minutes), OR-1 would tighten or relax accordingly, altering on-call escalation paging and shift handoff rigor. +- **service_levels.recovery_time_hours (Assumed: 4 hours)**: + Assumed in alignment with architecture condition K-6. If external regulators or business resilience plans require a shorter RTO (e.g., 2 hours) or allow a longer RTO (e.g., 8 hours), OR-4 would change, modifying warm standby synchronization mechanisms and failover automation requirements. +- **intent.business_outcomes (Assumed: BO-1, BO-2, BO-3)**: + Derived from the workflow request regarding 10-week foundation availability, DORA and PCI-DSS v4.0 regulatory compliance with >=99.9% uptime, and deterministic low-latency interconnect with customer-managed keys. If the customer defines alternate business metrics (such as developer deployment velocity or operational cost per transaction), requirement targets and operational measurement frameworks would adjust. +- **intent.exclusions (Assumed: Core payments engine application logic, on-premises systems beyond Interconnect demarcation, and physical telecom circuit ordering)**: + Assumed based on the platform foundation engagement kind. If operations were required to support application-level transaction handling or maintain on-premises network equipment, operational requirements would need to expand to encompass workload runtimes and physical datacentre operations. +- **operations.change_process (Assumed: Automated CI/CD pipelines via GitHub Actions with peer review and automated policy checks; standard changes without manual CAB)**: + Derived from workflow delivery parameters. If the customer mandates formal Change Advisory Board (CAB) review for routine foundation changes, deployment lead times and change governance requirements would become significantly more restrictive. +- **organisation.existing_teams (Assumed: Customer Central Platform Operations Team on 24x7)**: + Derived from condition K-8 noting alerting to a 24/7 central operations team. If the customer does not have a 24x7 internal platform team, operational escalation paths and tier-1 triage boundaries in OR-8 and OR-12 would need to be absorbed entirely by the provider. +- **service_management.tooling (Assumed: Google Cloud operations suite integrated with native alerting and GitHub Actions)**: + Catalogue default guidance assuming native cloud tooling without an external ITSM tool. If the customer mandates integration with enterprise ITSM platforms (e.g., ServiceNow, Jira Service Management, or PagerDuty), operational event handling and ticket workflow requirements would be added. + +### Unknowns and unstated areas + +- **intent.requirements (Unstated)**: + The customer stated architectural conditions and technical parameters, but did not provide direct statements of operational process requirements. Operational requirements were therefore synthesized from technical conditions (K-1 through K-12) and settled operating spec fields. +- **operations.monitoring (Unstated)**: + Specific monitoring requirements, third-party SIEM integrations, and existing synthetic probe frameworks were unstated. Defining these could yield explicit requirements for synthetic transaction probing and external audit log streaming. +- **solution.inherited_risks (Unstated)**: + Known solution risks or architectural constraints (such as carrier circuit delivery lead times or third-party firewall failover convergence times) were unstated. If documented, additional operational mitigation requirements would be necessary. +- **commercial.monthly_run_budget (Unstated)**: + A target operational run budget in GBP was not specified. While OR-9 enforces allocation governance, establishing a fixed financial envelope would introduce cost-efficiency thresholds. +- **organisation.headcount_limit (Unstated)**: + Headcount constraints for provider operations were unstated. If a strict cap exists, automation requirements across incident triage and change validation would be prioritized. +- **transition.go_live_date (Unstated)**: + The exact production cutover date for live payment traffic was unstated beyond the 10-week landing zone readiness target in OR-7. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/target_operating_model.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/target_operating_model.md new file mode 100644 index 0000000..5a77bbd --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/target_operating_model.md @@ -0,0 +1,83 @@ +# Design rationale + +The target operating model defines how the enterprise Google Cloud landing zone will be run by a service provider alongside the customer's own teams. The model focuses on delivering rapid foundation provisioning while guaranteeing strict adherence to DORA, PCI-DSS v4.0, and ISO27001 compliance standards, deterministic low latency for hybrid traffic, and robust 24x7 resilience. + +### Division of Work Among Parties + +- **Customer**: The customer retains strategic direction over the platform through the Customer Platform Owner role, prioritizing the delivery roadmap to ensure the core payments engine is unblocked and can go live within the 10-week target. They also maintain a Customer Operations Liaison team working 24x7 to receive incident alerts and manage the internal business escalation process. +- **Platform Service Provider**: As the core operator, this party holds accountability for foundational capabilities, encompassing 24x7 incident management, continuous telemetry, security monitoring, infrastructure deployment, financial governance, and compliance reporting. The provider ensures all infrastructure aligns with regulatory baselines and is optimized for low latency and high availability. +- **Cloud Provider**, **Network Carrier**, and **External Firewall Vendor**: These entities supply the underlying GCP infrastructure, the dedicated 10Gbps hybrid circuits, and the VM-Series firewall appliances. The provider’s teams manage these relationships under the Supplier Management capability. + +### Team Topologies + +To guarantee safe and rapid delivery alongside operational resilience, roles are organized according to Team Topologies principles: + +- **Cloud Platform Foundation Team** (`platform`): The primary operational group, building and maintaining the foundational infrastructure as a service for the payments engine developers. They construct golden paths, enforce configuration parity across regions, and manage standard operations. +- **Platform Security Team** (`complicated_subsystem`): Handles continuous 24x7 monitoring of egress boundaries and investigates security anomalies, effectively offloading deep compliance and perimeter defense cognitive load from the primary platform team. +- **Platform Governance Team** (`enabling`): A specialized team that acts as a facilitating layer. They assist the platform engineers with integrating automated checks for ISO27001, DORA, and PCI-DSS compliance and capturing evidential artifacts directly from CI/CD pipelines. + +The platform and security teams will engage in a focused **collaboration** to establish the initial PCI-DSS egress rules and incident runbooks, reverting to an `x_as_a_service` interaction once controls are automated and audited. + +### Role Placement and Sizing + +All roles provided by the service provider are strictly placed in **GB, PL, and IN** to comply with the operational requirement. Levels are determined by the capability's technical demands, and cost totals (currently calculated based on draft rates) reflect the required coverages. + +- **Platform Engineering Lead (`GB`, `lead`)**: Sits in the UK during business hours to direct the overarching platform strategy, maintaining tight alignment with the Customer Platform Owner and holding primary accountability for service availability, budget enforcement, and infrastructure design. +- **Compliance & Governance Analyst (`GB`, `senior`)**: Ensures DORA and PCI-DSS data residency constraints are met while sitting within the primary regulatory jurisdiction. +- **Security Operations Analyst (`PL`, `senior`)** and **Platform Reliability Engineer (`PL`, `senior`)**: Situated in Poland, these senior engineers support 24x7 coverage through shift rotations, dealing with complex failover configurations and security anomaly detection. +- **Platform Operations Engineer (`IN`, `engineer`)**: Based in India to provide continuous 24x7 alert response, ensuring the strict 30-minute acknowledgement SLA for Priority 1 faults is achieved efficiently. +- **Platform Automation Engineer (`PL`, `senior`)**: Works business hours to automate the delivery pipelines via GitHub Actions and optimize resource placements for carbon-free energy. + +### Assumptions + +As this was pre-approved without a live review, the following assumptions were made: +- It is assumed that the budget allocations permit the specific staffing levels modeled to meet the 24x7 DORA/PCI-DSS requirements and the 10-week delivery timeline. +- It is assumed that "hard limits" on cloud consumption can be functionally enforced via automation (e.g., using Cloud Functions triggered by billing alerts to halt non-critical deployments), rather than just alerting, to satisfy the financial governance constraint. +- It is assumed that the customer's operations liaison team has sufficient capacity to consult on all key foundational capabilities continuously. + +## Responsibility matrix + +| Capability | Platform Operations Engineer | Platform Reliability Engineer | Security Operations Analyst | Platform Automation Engineer | Platform Engineering Lead | Compliance & Governance Analyst | Customer Platform Owner | Customer Operations Liaison | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Incident management (C-1) | | R | | | A | | | C | +| Monitoring and alerting (C-2) | R | | | | A | | | C | +| Problem management (C-3) | | | | R | A | | | C | +| Service request fulfilment (C-4) | | | | R | A | | | C | +| Change enablement (C-5) | | | | R | A | R | | C | +| Release and deployment (C-6) | | | | R | A | | | C | +| Configuration management (C-7) | | | | R | A | | | C | +| Backup and recovery (C-8) | | | | R | A | | | C | +| Service continuity (C-9) | | R | | | A | | | C | +| Capacity and performance management (C-10) | | | | R | A | | | C | +| Access management (C-11) | | | A/R | | | | | C | +| Vulnerability and patch management (C-12) | | | A/R | | | | | C | +| Security monitoring and response (C-13) | | | A/R | | | | | C | +| Cost management (C-14) | | | | | A | R | | C | +| Service level management (C-15) | | | | | A | R | | C | +| Platform product management (C-16) | | | | | A/R | | C | | +| Self-service and golden paths (C-17) | | | | R | A | | | C | +| Enablement (C-18) | | | | R | A | | | C | +| Delivery and developer experience measurement (C-19) | | | | R | A/R | | | C | +| Supplier management (C-20) | | | | | A | R | | C | +| Regulatory evidence and reporting (C-21) | | | | | | A/R | C | | + +A accountable, R responsible, C consulted, I informed. + +## Staffing and cost + +| Role | Party | Country | Level | Hours | Effort a month | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform Operations Engineer | Platform Service Provider | IN | engineer | 24x7 | 30.0 h | 5.0 | 17,034 GBP | +| Platform Reliability Engineer | Platform Service Provider | PL | senior | 24x7 | 50.0 h | 5.0 | 41,640 GBP | +| Security Operations Analyst | Platform Service Provider | PL | senior | 24x7 | 75.0 h | 5.0 | 41,640 GBP | +| Platform Automation Engineer | Platform Service Provider | PL | senior | business | 177.5 h | 1.2 | 8,063 GBP | +| Platform Engineering Lead | Platform Service Provider | GB | lead | business | 25.0 h | 0.2 | 2,765 GBP | +| Compliance & Governance Analyst | Platform Service Provider | GB | senior | business | 62.5 h | 0.5 | 5,267 GBP | +| Customer Platform Owner | Customer | GB | lead | business | 0.0 h | 0.0 | not priced | +| Customer Operations Liaison | Customer | GB | senior | business | 0.0 h | 0.0 | not priced | + +16.9 people in all. Cost a month: 116,409 GBP. + +Worked from rate card `rate-card`, version `draft-2026-10-13`. **Its figures are placeholders: no price has been set, and this cost is an illustration.** +Converted from USD at 0.79, a rate set on 2026-10-13. +The customer's own staff are counted and not priced. A Role that covers hours wider than a working day is never smaller than the team that can cover them. diff --git a/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/validation_results.md b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/validation_results.md new file mode 100644 index 0000000..e540baa --- /dev/null +++ b/eval/results/20261021T203624Z/deliverables/cases/case-08-landing-zone/executions/case-02-from-architecture-r3/deliverables/validation_results.md @@ -0,0 +1,66 @@ +# Check verdicts + +The target operating model was evaluated against the operational requirements and architectural conditions to verify its viability and alignment with the customer's constraints. + +All checks passed successfully. + +- **OC-1 (Compliance Baseline Enforcement):** **Pass**. Capability C-7 provides configuration management, enforcing state definition parity and auditing configurations, supporting continuous enforcement of DORA, PCI-DSS v4.0, and ISO27001 baselines. Role R-4 is responsible for this capability. +- **OC-2 (Warm Standby Maintenance):** **Pass**. Capability C-9 handles service continuity, which includes planning and periodic testing of regional failovers, ensuring the secondary region is maintained as an active warm standby. Role R-2 is responsible for it. +- **OC-3 (Data Residency Configuration Enforcement):** **Pass**. Capability C-7 enforces configuration state parity and audits configurations, satisfying the condition that data residency, logs, and keys remain in the UK and EU. +- **OC-4 (Timely Platform Provisioning):** **Pass**. Capability C-16 prioritises features and technical debt to ensure the platform foundation supports the payments engine developers effectively, meeting the 10-week provisioning timeframe. +- **OC-5 (Continuous Telemetry and Availability):** **Pass**. Capability C-2 provides continuous monitoring and alerting 24x7, maintaining telemetry pipelines to verify the infrastructure maintains 99.9% availability. +- **OC-6 (Disaster Recovery RTO Verification):** **Pass**. Capability C-9 handles disaster recovery failover tests to validate readiness and timing to ensure the 4-hour RTO can be met. +- **OC-7 (Backup and Restore RPO Verification):** **Pass**. Capability C-8 rehearses restoration processes for Cloud Storage and KMS to verify the 1-hour RPO is met. +- **OC-8 (Automated Alert Triggering):** **Pass**. Capability C-2 configures and maintains automated alerting rules that trigger workflows to the 24x7 customer central platform operations team (R-8 is consulted). +- **OC-9 (Budget Tracking and Hard Limits Enforcement):** **Pass**. Capability C-14 designs and operates automation to enforce hard financial limits and execution stops, tracking cloud consumption against budgets. The capability was revised to address the prior feedback. +- **OC-10 (Hybrid Interconnect Monitoring):** **Pass**. Capability C-10 monitors and optimises the dedicated hybrid interconnect bandwidth to guarantee deterministic low latency for real-time payments. +- **OC-11 (Carbon-Optimised Workload Placement):** **Pass**. Capability C-6 operates the deployment pipelines to ensure timely deployments and carbon-optimised workload placement. +- **OC-12 (IAM Permissions Auditing):** **Pass**. Capability C-11 continuously applies automated IAM rightsizing recommendations and manages access from approved countries. +- **OC-13 (Priority 1 Fault Acknowledgement):** **Pass**. Capability C-1 provides 24x7 incident management and 30-minute response capability for Priority 1 faults. +- **OC-14 (Provider-Operated Delivery Model):** **Pass**. Role R-5 (Platform Engineering Lead) sits within the Provider (P-2) party, managing and directing the platform strategy and operations. +- **OC-15 (Operational Management Locations):** **Pass**. All provider operational staff (such as Role R-2 in Poland) are based exclusively in the UK, Poland, or India, fulfilling the sourcing requirement. +- **OC-16 (Compliance Audit Evidence Gathering):** **Pass**. Capability C-21 gathers and preserves documented audit evidence to prove compliance with DORA, PCI-DSS v4.0, and ISO27001 to external regulators. The capability basis was updated to include ISO27001 explicitly. +- **OC-17 (Data Residency Guarantee Evidence):** **Pass**. Capability C-21 captures, preserves, and provides documented audit evidence to external regulators, which includes proving data residency constraints. + +**Model Verdict:** The target operating model as a whole receives a **pass** verdict. All findings and constraints have been resolved, and the model is fully equipped to meet the operational requirements as designed. + +## Scenario walk-throughs + +Each scenario is generated from the operating model and traced through it by code. + +| Scenario | Result | Path | +| --- | --- | --- | +| Resource Hierarchy and Environment Boundary Manager (S-1) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Resource Hierarchy and Environment Boundary Manager (S-1) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Dedicated Private Hybrid Connectivity Gateway (S-2) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Centralised Outbound Traffic Inspection and Egress Boundary (S-3) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Centralised Immutable Audit Event and Compliance Log Repository (S-4) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Automated Infrastructure Orchestration and Deployment Engine (S-5) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Federated Identity and Role Governance Authority (S-6) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Federated Identity and Role Governance Authority (S-6) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Cryptographic Key Lifecycle and Encryption Controller (S-7) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Platform Telemetry and Operational Alerting Engine (S-8) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Platform Telemetry and Operational Alerting Engine (S-8) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Financial Governance and Budget Enforcement Monitor (S-9) fails at an hour faults are responded to (24x7). | pass | Platform Reliability Engineer responds under C-1, and Platform Engineering Lead is answerable. | +| Financial Governance and Budget Enforcement Monitor (S-9) must be changed. | pass | Platform Automation Engineer makes the change under C-5. | +| Condition K-1 must be shown to hold: The landing zone configuration aligns with DORA operational resiliency guidelines and PCI-DSS v4.0 perimeter isolation standards during external audit. | pass | Check OC-1 on C-7 keeps it true. | +| Condition K-2 must be shown to hold: The secondary region in europe-west1 operates as a warm standby ready to receive failover traffic. | pass | Check OC-2 on C-9 keeps it true. | +| Condition K-3 must be shown to hold: Data residency is guaranteed within the United Kingdom and European Union. | pass | Check OC-3 on C-7 keeps it true. | +| Condition K-4 must be shown to hold: Platform foundation is fully provisioned and validated for core payments engine workload ingestion within 10 weeks. | pass | Check OC-4 on C-16 keeps it true. | +| Condition K-5 must be shown to hold: Platform foundation infrastructure maintains measured monthly availability of at least 99.9%. | pass | Check OC-5 on C-2 keeps it true. | +| Condition K-6 must be shown to hold: Disaster recovery activation achieves a recovery time objective (RTO) of 4 hours. | pass | Check OC-6 on C-9 keeps it true. | +| Condition K-7 must be shown to hold: Disaster recovery processes achieve a recovery point objective (RPO) of 1 hour. | pass | Check OC-7 on C-8 keeps it true. | +| Condition K-8 must be shown to hold: Alert notifications trigger operational incident workflows for the 24/7 central platform operations team. | pass | Check OC-8 on C-2 keeps it true. | +| Condition K-9 must be shown to hold: Monthly cloud foundation expenditure does not exceed the target allocation without operational sign-off. | pass | Check OC-9 on C-14 keeps it true. | +| Condition K-10 must be shown to hold: Hybrid connection provides deterministic low latency suitable for real-time payment transaction processing. | pass | Check OC-10 on C-10 keeps it true. | +| Condition K-11 must be shown to hold: Deployments optimize carbon-free energy utilization based on Google Cloud regional CFE scores. | pass | Check OC-11 on C-6 keeps it true. | +| Condition K-12 must be shown to hold: Permissions are continuously rightsized based on IAM Recommender findings. | pass | Check OC-12 on C-11 keeps it true. | +| Platform Service Provider cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| Google Cloud cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| Network Carrier cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. | +| External Firewall Vendor cannot be reached. | pass | Every Capability it is answerable for has a Role in another Party to turn to. |