docs: publish the runs of 2026-10-21
The evaluation (both cases three times: report, results, spread, each run's Deliverables and lesson candidates) and the hand-off through the Orchestration engine (report, the document of links, both agents' kept files and records). Benchmark cases only, all fictional; no machine paths, and the links carry no signatures. Other runs stay ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,10 @@
|
||||
# Intent 1: bench-landing-zone-shared
|
||||
|
||||
- Recorded: 2026-10-21T20:36:25+00:00
|
||||
- Execution: `case-01-landing-zone-shared-r1`
|
||||
|
||||
## What the customer wants
|
||||
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
@@ -0,0 +1,198 @@
|
||||
schema_version: 1
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
client_name: ask
|
||||
author: ask
|
||||
revision: ask
|
||||
source: ask
|
||||
status: BASELINE_LOCKED
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: 'We want our landing zone run as a shared service from go-live. A provider
|
||||
should
|
||||
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
|
||||
over changes to the network.
|
||||
|
||||
'
|
||||
requirements:
|
||||
- id: REQ-01
|
||||
statement: Our platform team approves every change to the network.
|
||||
- id: REQ-02
|
||||
statement: We receive a monthly report of service levels and cost.
|
||||
business_outcomes:
|
||||
- id: BO-1
|
||||
outcome: New payment products reach customers in weeks rather than months.
|
||||
target: A change reaches production within a day of approval.
|
||||
- id: BO-2
|
||||
outcome: Payments are never unavailable for long.
|
||||
target: Restored within 4 hours.
|
||||
- id: BO-3
|
||||
outcome: We keep the platform engineers we hire.
|
||||
exclusions:
|
||||
- Application workloads and services hosted on the platform foundation
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- id: S-1
|
||||
name: Hub network and interconnect
|
||||
kind: network
|
||||
product: Cloud Interconnect
|
||||
- id: S-2
|
||||
name: Audit log store
|
||||
kind: storage
|
||||
product: Cloud Storage
|
||||
- id: S-3
|
||||
name: Deployment pipeline
|
||||
kind: pipeline
|
||||
product: Cloud Build
|
||||
- id: S-4
|
||||
name: Encryption keys
|
||||
kind: keys
|
||||
product: Cloud KMS
|
||||
conditions:
|
||||
- id: K-1
|
||||
'on':
|
||||
- S-1
|
||||
condition: The platform is available at least 99.9% of each month.
|
||||
- id: K-2
|
||||
'on':
|
||||
- S-2
|
||||
condition: Audit logs cannot be deleted for seven years.
|
||||
- id: K-3
|
||||
'on':
|
||||
- S-4
|
||||
condition: Encryption keys are rotated every 90 days.
|
||||
build:
|
||||
duration_days: 36
|
||||
customer_owned: ask
|
||||
inherited_risks: ask
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: Changes go through a deployment pipeline with peer review, with
|
||||
explicit customer platform team approval required for all network changes.
|
||||
monitoring: ask
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries:
|
||||
- GB
|
||||
- PL
|
||||
- IN
|
||||
organisation:
|
||||
existing_teams:
|
||||
- name: Platform team
|
||||
party: customer
|
||||
hours: business
|
||||
country: GB
|
||||
headcount_limit: ask
|
||||
service_management:
|
||||
tooling: Google Cloud operations suite and native Cloud provider tooling, with no
|
||||
separate external ITSM system.
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks:
|
||||
- DORA
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
transition:
|
||||
go_live_date: ask
|
||||
run:
|
||||
pre_approved: true
|
||||
overlay_skills: ask
|
||||
inferred_assumptions:
|
||||
- id: ASSUMP-01
|
||||
field: intent.exclusions
|
||||
statement: 'Assumed intent.exclusions is [''Application workloads and services hosted
|
||||
on the platform foundation'']. Assumed based on platform foundation scope: workloads
|
||||
hosted on top of the landing zone are out of scope for this operating model.'
|
||||
impact_if_invalid: Changes which Capabilities are designed.
|
||||
- id: ASSUMP-02
|
||||
field: operations.change_process
|
||||
statement: Assumed operations.change_process is Changes go through a deployment
|
||||
pipeline with peer review, with explicit customer platform team approval required
|
||||
for all network changes.. Pipeline-driven changes with peer review, with customer
|
||||
platform team approval required for network changes per REQ-01.
|
||||
impact_if_invalid: Changes change enablement and release.
|
||||
- id: ASSUMP-03
|
||||
field: service_management.tooling
|
||||
statement: Assumed service_management.tooling is Google Cloud operations suite and
|
||||
native Cloud provider tooling, with no separate external ITSM system.. Default
|
||||
native GCP operations tooling used as no external ITSM was specified.
|
||||
impact_if_invalid: Changes tooling and the processes written for it.
|
||||
discovery_gaps:
|
||||
- field: metadata.client_name
|
||||
prompt: What is metadata.client_name?
|
||||
severity: LOW
|
||||
- field: metadata.author
|
||||
prompt: What is metadata.author?
|
||||
severity: LOW
|
||||
- field: metadata.revision
|
||||
prompt: What is metadata.revision?
|
||||
severity: LOW
|
||||
- field: metadata.source
|
||||
prompt: What is metadata.source?
|
||||
severity: LOW
|
||||
- field: solution.build.customer_owned
|
||||
prompt: Which parts of the build does the customer or their supplier own?
|
||||
severity: LOW
|
||||
- field: solution.inherited_risks
|
||||
prompt: What is already known to be unverified or at risk in the solution?
|
||||
severity: LOW
|
||||
- field: operations.monitoring
|
||||
prompt: Which monitoring and alerting is already in place or required?
|
||||
severity: LOW
|
||||
- field: organisation.headcount_limit
|
||||
prompt: Is there a limit on the number of people operating this?
|
||||
severity: LOW
|
||||
- field: transition.go_live_date
|
||||
prompt: When must the solution be in operation?
|
||||
severity: LOW
|
||||
- field: run.overlay_skills
|
||||
prompt: Which overlay Skills should this Execution load?
|
||||
severity: LOW
|
||||
provenance:
|
||||
commercial.currency: supplied
|
||||
commercial.monthly_run_budget: supplied
|
||||
compliance.frameworks: supplied
|
||||
engagement.kind: supplied
|
||||
intent.business_outcomes: supplied
|
||||
intent.exclusions: assumed
|
||||
intent.requirements: supplied
|
||||
intent.statement: supplied
|
||||
metadata.author: unknown
|
||||
metadata.client_name: unknown
|
||||
metadata.customer_case: supplied
|
||||
metadata.revision: unknown
|
||||
metadata.source: unknown
|
||||
operations.availability_hours: supplied
|
||||
operations.change_process: assumed
|
||||
operations.monitoring: unknown
|
||||
operations.support_hours: supplied
|
||||
organisation.existing_teams: supplied
|
||||
organisation.headcount_limit: unknown
|
||||
run.overlay_skills: unknown
|
||||
run.pre_approved: supplied
|
||||
service_levels.priority_1_response_minutes: supplied
|
||||
service_levels.recovery_time_hours: supplied
|
||||
service_management.tooling: assumed
|
||||
solution.build.customer_owned: unknown
|
||||
solution.build.duration_days: supplied
|
||||
solution.cloud_provider: supplied
|
||||
solution.conditions: supplied
|
||||
solution.inherited_risks: unknown
|
||||
solution.items: supplied
|
||||
sourcing.delivery_countries: supplied
|
||||
sourcing.model: supplied
|
||||
transition.go_live_date: unknown
|
||||
readiness:
|
||||
supplied: 20
|
||||
answered: 0
|
||||
assumed: 3
|
||||
unknown: 10
|
||||
not_applicable: 0
|
||||
@@ -0,0 +1,53 @@
|
||||
# A fictional benchmark: a small landing zone run as a shared service. No customer's
|
||||
# data is in it. Complete enough to run with nobody to ask.
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: |
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
business_outcomes:
|
||||
- {id: BO-1, outcome: New payment products reach customers in weeks rather than months., target: "A change reaches production within a day of approval."}
|
||||
- {id: BO-2, outcome: Payments are never unavailable for long., target: "Restored within 4 hours."}
|
||||
- {id: BO-3, outcome: We keep the platform engineers we hire.}
|
||||
requirements:
|
||||
- {id: REQ-01, statement: Our platform team approves every change to the network.}
|
||||
- {id: REQ-02, statement: We receive a monthly report of service levels and cost.}
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- {id: S-1, name: Hub network and interconnect, kind: network, product: Cloud Interconnect}
|
||||
- {id: S-2, name: Audit log store, kind: storage, product: Cloud Storage}
|
||||
- {id: S-3, name: Deployment pipeline, kind: pipeline, product: Cloud Build}
|
||||
- {id: S-4, name: Encryption keys, kind: keys, product: Cloud KMS}
|
||||
conditions:
|
||||
- {id: K-1, "on": [S-1], condition: The platform is available at least 99.9% of each month.}
|
||||
- {id: K-2, "on": [S-2], condition: Audit logs cannot be deleted for seven years.}
|
||||
- {id: K-3, "on": [S-4], condition: Encryption keys are rotated every 90 days.}
|
||||
build:
|
||||
duration_days: 36
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: decide
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries: [GB, PL, IN]
|
||||
organisation:
|
||||
existing_teams:
|
||||
- {name: Platform team, party: customer, hours: business, country: GB}
|
||||
service_management:
|
||||
tooling: decide
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks: [DORA]
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
run:
|
||||
pre_approved: true
|
||||
@@ -0,0 +1,10 @@
|
||||
# Intent 2: bench-landing-zone-shared
|
||||
|
||||
- Recorded: 2026-10-21T20:54:36+00:00
|
||||
- Execution: `case-01-landing-zone-shared-r2`
|
||||
|
||||
## What the customer wants
|
||||
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
@@ -0,0 +1,199 @@
|
||||
schema_version: 1
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
client_name: ask
|
||||
author: ask
|
||||
revision: ask
|
||||
source: ask
|
||||
status: BASELINE_LOCKED
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: 'We want our landing zone run as a shared service from go-live. A provider
|
||||
should
|
||||
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
|
||||
over changes to the network.
|
||||
|
||||
'
|
||||
requirements:
|
||||
- id: REQ-01
|
||||
statement: Our platform team approves every change to the network.
|
||||
- id: REQ-02
|
||||
statement: We receive a monthly report of service levels and cost.
|
||||
business_outcomes:
|
||||
- id: BO-1
|
||||
outcome: New payment products reach customers in weeks rather than months.
|
||||
target: A change reaches production within a day of approval.
|
||||
- id: BO-2
|
||||
outcome: Payments are never unavailable for long.
|
||||
target: Restored within 4 hours.
|
||||
- id: BO-3
|
||||
outcome: We keep the platform engineers we hire.
|
||||
exclusions:
|
||||
- Application workloads deployed onto the platform foundation
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- id: S-1
|
||||
name: Hub network and interconnect
|
||||
kind: network
|
||||
product: Cloud Interconnect
|
||||
- id: S-2
|
||||
name: Audit log store
|
||||
kind: storage
|
||||
product: Cloud Storage
|
||||
- id: S-3
|
||||
name: Deployment pipeline
|
||||
kind: pipeline
|
||||
product: Cloud Build
|
||||
- id: S-4
|
||||
name: Encryption keys
|
||||
kind: keys
|
||||
product: Cloud KMS
|
||||
conditions:
|
||||
- id: K-1
|
||||
'on':
|
||||
- S-1
|
||||
condition: The platform is available at least 99.9% of each month.
|
||||
- id: K-2
|
||||
'on':
|
||||
- S-2
|
||||
condition: Audit logs cannot be deleted for seven years.
|
||||
- id: K-3
|
||||
'on':
|
||||
- S-4
|
||||
condition: Encryption keys are rotated every 90 days.
|
||||
build:
|
||||
duration_days: 36
|
||||
customer_owned: ask
|
||||
inherited_risks: ask
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: Changes go through automated pipelines with peer review; changes
|
||||
affecting the hub network require customer platform team approval prior to release.
|
||||
monitoring: ask
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries:
|
||||
- GB
|
||||
- PL
|
||||
- IN
|
||||
organisation:
|
||||
existing_teams:
|
||||
- name: Platform team
|
||||
party: customer
|
||||
hours: business
|
||||
country: GB
|
||||
headcount_limit: ask
|
||||
service_management:
|
||||
tooling: Google Cloud native operations tooling (Cloud Monitoring and Cloud Logging),
|
||||
with no separate service management system.
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks:
|
||||
- DORA
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
transition:
|
||||
go_live_date: ask
|
||||
run:
|
||||
pre_approved: true
|
||||
overlay_skills: ask
|
||||
inferred_assumptions:
|
||||
- id: ASSUMP-01
|
||||
field: intent.exclusions
|
||||
statement: Assumed intent.exclusions is ['Application workloads deployed onto the
|
||||
platform foundation']. Workloads deployed onto the platform foundation are excluded;
|
||||
operations is confined to the landing zone foundation itself.
|
||||
impact_if_invalid: Changes which Capabilities are designed.
|
||||
- id: ASSUMP-02
|
||||
field: operations.change_process
|
||||
statement: Assumed operations.change_process is Changes go through automated pipelines
|
||||
with peer review; changes affecting the hub network require customer platform
|
||||
team approval prior to release.. Changes follow automated deployment pipelines
|
||||
with peer review, with network changes requiring explicit customer platform team
|
||||
approval per REQ-01.
|
||||
impact_if_invalid: Changes change enablement and release.
|
||||
- id: ASSUMP-03
|
||||
field: service_management.tooling
|
||||
statement: Assumed service_management.tooling is Google Cloud native operations
|
||||
tooling (Cloud Monitoring and Cloud Logging), with no separate service management
|
||||
system.. Defaulting to GCP native operations tooling with no separate ITSM system.
|
||||
impact_if_invalid: Changes tooling and the processes written for it.
|
||||
discovery_gaps:
|
||||
- field: metadata.client_name
|
||||
prompt: What is metadata.client_name?
|
||||
severity: LOW
|
||||
- field: metadata.author
|
||||
prompt: What is metadata.author?
|
||||
severity: LOW
|
||||
- field: metadata.revision
|
||||
prompt: What is metadata.revision?
|
||||
severity: LOW
|
||||
- field: metadata.source
|
||||
prompt: What is metadata.source?
|
||||
severity: LOW
|
||||
- field: solution.build.customer_owned
|
||||
prompt: Which parts of the build does the customer or their supplier own?
|
||||
severity: LOW
|
||||
- field: solution.inherited_risks
|
||||
prompt: What is already known to be unverified or at risk in the solution?
|
||||
severity: LOW
|
||||
- field: operations.monitoring
|
||||
prompt: Which monitoring and alerting is already in place or required?
|
||||
severity: LOW
|
||||
- field: organisation.headcount_limit
|
||||
prompt: Is there a limit on the number of people operating this?
|
||||
severity: LOW
|
||||
- field: transition.go_live_date
|
||||
prompt: When must the solution be in operation?
|
||||
severity: LOW
|
||||
- field: run.overlay_skills
|
||||
prompt: Which overlay Skills should this Execution load?
|
||||
severity: LOW
|
||||
provenance:
|
||||
commercial.currency: supplied
|
||||
commercial.monthly_run_budget: supplied
|
||||
compliance.frameworks: supplied
|
||||
engagement.kind: supplied
|
||||
intent.business_outcomes: supplied
|
||||
intent.exclusions: assumed
|
||||
intent.requirements: supplied
|
||||
intent.statement: supplied
|
||||
metadata.author: unknown
|
||||
metadata.client_name: unknown
|
||||
metadata.customer_case: supplied
|
||||
metadata.revision: unknown
|
||||
metadata.source: unknown
|
||||
operations.availability_hours: supplied
|
||||
operations.change_process: assumed
|
||||
operations.monitoring: unknown
|
||||
operations.support_hours: supplied
|
||||
organisation.existing_teams: supplied
|
||||
organisation.headcount_limit: unknown
|
||||
run.overlay_skills: unknown
|
||||
run.pre_approved: supplied
|
||||
service_levels.priority_1_response_minutes: supplied
|
||||
service_levels.recovery_time_hours: supplied
|
||||
service_management.tooling: assumed
|
||||
solution.build.customer_owned: unknown
|
||||
solution.build.duration_days: supplied
|
||||
solution.cloud_provider: supplied
|
||||
solution.conditions: supplied
|
||||
solution.inherited_risks: unknown
|
||||
solution.items: supplied
|
||||
sourcing.delivery_countries: supplied
|
||||
sourcing.model: supplied
|
||||
transition.go_live_date: unknown
|
||||
readiness:
|
||||
supplied: 20
|
||||
answered: 0
|
||||
assumed: 3
|
||||
unknown: 10
|
||||
not_applicable: 0
|
||||
@@ -0,0 +1,53 @@
|
||||
# A fictional benchmark: a small landing zone run as a shared service. No customer's
|
||||
# data is in it. Complete enough to run with nobody to ask.
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: |
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
business_outcomes:
|
||||
- {id: BO-1, outcome: New payment products reach customers in weeks rather than months., target: "A change reaches production within a day of approval."}
|
||||
- {id: BO-2, outcome: Payments are never unavailable for long., target: "Restored within 4 hours."}
|
||||
- {id: BO-3, outcome: We keep the platform engineers we hire.}
|
||||
requirements:
|
||||
- {id: REQ-01, statement: Our platform team approves every change to the network.}
|
||||
- {id: REQ-02, statement: We receive a monthly report of service levels and cost.}
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- {id: S-1, name: Hub network and interconnect, kind: network, product: Cloud Interconnect}
|
||||
- {id: S-2, name: Audit log store, kind: storage, product: Cloud Storage}
|
||||
- {id: S-3, name: Deployment pipeline, kind: pipeline, product: Cloud Build}
|
||||
- {id: S-4, name: Encryption keys, kind: keys, product: Cloud KMS}
|
||||
conditions:
|
||||
- {id: K-1, "on": [S-1], condition: The platform is available at least 99.9% of each month.}
|
||||
- {id: K-2, "on": [S-2], condition: Audit logs cannot be deleted for seven years.}
|
||||
- {id: K-3, "on": [S-4], condition: Encryption keys are rotated every 90 days.}
|
||||
build:
|
||||
duration_days: 36
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: decide
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries: [GB, PL, IN]
|
||||
organisation:
|
||||
existing_teams:
|
||||
- {name: Platform team, party: customer, hours: business, country: GB}
|
||||
service_management:
|
||||
tooling: decide
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks: [DORA]
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
run:
|
||||
pre_approved: true
|
||||
@@ -0,0 +1,10 @@
|
||||
# Intent 3: bench-landing-zone-shared
|
||||
|
||||
- Recorded: 2026-10-21T21:11:10+00:00
|
||||
- Execution: `case-01-landing-zone-shared-r3`
|
||||
|
||||
## What the customer wants
|
||||
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
@@ -0,0 +1,201 @@
|
||||
schema_version: 1
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
client_name: ask
|
||||
author: ask
|
||||
revision: ask
|
||||
source: ask
|
||||
status: BASELINE_LOCKED
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: 'We want our landing zone run as a shared service from go-live. A provider
|
||||
should
|
||||
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
|
||||
over changes to the network.
|
||||
|
||||
'
|
||||
requirements:
|
||||
- id: REQ-01
|
||||
statement: Our platform team approves every change to the network.
|
||||
- id: REQ-02
|
||||
statement: We receive a monthly report of service levels and cost.
|
||||
business_outcomes:
|
||||
- id: BO-1
|
||||
outcome: New payment products reach customers in weeks rather than months.
|
||||
target: A change reaches production within a day of approval.
|
||||
- id: BO-2
|
||||
outcome: Payments are never unavailable for long.
|
||||
target: Restored within 4 hours.
|
||||
- id: BO-3
|
||||
outcome: We keep the platform engineers we hire.
|
||||
exclusions:
|
||||
- Workload applications running on the platform foundation
|
||||
- Network change approvals (retained by customer platform team)
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- id: S-1
|
||||
name: Hub network and interconnect
|
||||
kind: network
|
||||
product: Cloud Interconnect
|
||||
- id: S-2
|
||||
name: Audit log store
|
||||
kind: storage
|
||||
product: Cloud Storage
|
||||
- id: S-3
|
||||
name: Deployment pipeline
|
||||
kind: pipeline
|
||||
product: Cloud Build
|
||||
- id: S-4
|
||||
name: Encryption keys
|
||||
kind: keys
|
||||
product: Cloud KMS
|
||||
conditions:
|
||||
- id: K-1
|
||||
'on':
|
||||
- S-1
|
||||
condition: The platform is available at least 99.9% of each month.
|
||||
- id: K-2
|
||||
'on':
|
||||
- S-2
|
||||
condition: Audit logs cannot be deleted for seven years.
|
||||
- id: K-3
|
||||
'on':
|
||||
- S-4
|
||||
condition: Encryption keys are rotated every 90 days.
|
||||
build:
|
||||
duration_days: 36
|
||||
customer_owned: ask
|
||||
inherited_risks: ask
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: Changes go through a deployment pipeline with peer review, with
|
||||
network changes requiring customer platform team approval.
|
||||
monitoring: ask
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries:
|
||||
- GB
|
||||
- PL
|
||||
- IN
|
||||
organisation:
|
||||
existing_teams:
|
||||
- name: Platform team
|
||||
party: customer
|
||||
hours: business
|
||||
country: GB
|
||||
headcount_limit: ask
|
||||
service_management:
|
||||
tooling: The Cloud provider's own operations tooling, with no separate system.
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks:
|
||||
- DORA
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
transition:
|
||||
go_live_date: ask
|
||||
run:
|
||||
pre_approved: true
|
||||
overlay_skills: ask
|
||||
inferred_assumptions:
|
||||
- id: ASSUMP-01
|
||||
field: intent.exclusions
|
||||
statement: 'Assumed intent.exclusions is [''Workload applications running on the
|
||||
platform foundation'', ''Network change approvals (retained by customer platform
|
||||
team)'']. Assumed from the platform foundation scope: workload applications running
|
||||
on the platform are out of scope, and customer platform team retains network change
|
||||
approvals.'
|
||||
impact_if_invalid: Changes which Capabilities are designed.
|
||||
- id: ASSUMP-02
|
||||
field: operations.change_process
|
||||
statement: 'Assumed operations.change_process is Changes go through a deployment
|
||||
pipeline with peer review, with network changes requiring customer platform team
|
||||
approval.. Catalogue default adapted to customer requirement REQ-01: changes go
|
||||
through deployment pipelines with peer review, with customer platform team retaining
|
||||
approval over network changes.'
|
||||
impact_if_invalid: Changes change enablement and release.
|
||||
- id: ASSUMP-03
|
||||
field: service_management.tooling
|
||||
statement: 'Assumed service_management.tooling is The Cloud provider''s own operations
|
||||
tooling, with no separate system.. Catalogue default: Cloud provider''s native
|
||||
operations tooling with no separate system.'
|
||||
impact_if_invalid: Changes tooling and the processes written for it.
|
||||
discovery_gaps:
|
||||
- field: metadata.client_name
|
||||
prompt: What is metadata.client_name?
|
||||
severity: LOW
|
||||
- field: metadata.author
|
||||
prompt: What is metadata.author?
|
||||
severity: LOW
|
||||
- field: metadata.revision
|
||||
prompt: What is metadata.revision?
|
||||
severity: LOW
|
||||
- field: metadata.source
|
||||
prompt: What is metadata.source?
|
||||
severity: LOW
|
||||
- field: solution.build.customer_owned
|
||||
prompt: Which parts of the build does the customer or their supplier own?
|
||||
severity: LOW
|
||||
- field: solution.inherited_risks
|
||||
prompt: What is already known to be unverified or at risk in the solution?
|
||||
severity: LOW
|
||||
- field: operations.monitoring
|
||||
prompt: Which monitoring and alerting is already in place or required?
|
||||
severity: LOW
|
||||
- field: organisation.headcount_limit
|
||||
prompt: Is there a limit on the number of people operating this?
|
||||
severity: LOW
|
||||
- field: transition.go_live_date
|
||||
prompt: When must the solution be in operation?
|
||||
severity: LOW
|
||||
- field: run.overlay_skills
|
||||
prompt: Which overlay Skills should this Execution load?
|
||||
severity: LOW
|
||||
provenance:
|
||||
commercial.currency: supplied
|
||||
commercial.monthly_run_budget: supplied
|
||||
compliance.frameworks: supplied
|
||||
engagement.kind: supplied
|
||||
intent.business_outcomes: supplied
|
||||
intent.exclusions: assumed
|
||||
intent.requirements: supplied
|
||||
intent.statement: supplied
|
||||
metadata.author: unknown
|
||||
metadata.client_name: unknown
|
||||
metadata.customer_case: supplied
|
||||
metadata.revision: unknown
|
||||
metadata.source: unknown
|
||||
operations.availability_hours: supplied
|
||||
operations.change_process: assumed
|
||||
operations.monitoring: unknown
|
||||
operations.support_hours: supplied
|
||||
organisation.existing_teams: supplied
|
||||
organisation.headcount_limit: unknown
|
||||
run.overlay_skills: unknown
|
||||
run.pre_approved: supplied
|
||||
service_levels.priority_1_response_minutes: supplied
|
||||
service_levels.recovery_time_hours: supplied
|
||||
service_management.tooling: assumed
|
||||
solution.build.customer_owned: unknown
|
||||
solution.build.duration_days: supplied
|
||||
solution.cloud_provider: supplied
|
||||
solution.conditions: supplied
|
||||
solution.inherited_risks: unknown
|
||||
solution.items: supplied
|
||||
sourcing.delivery_countries: supplied
|
||||
sourcing.model: supplied
|
||||
transition.go_live_date: unknown
|
||||
readiness:
|
||||
supplied: 20
|
||||
answered: 0
|
||||
assumed: 3
|
||||
unknown: 10
|
||||
not_applicable: 0
|
||||
@@ -0,0 +1,53 @@
|
||||
# A fictional benchmark: a small landing zone run as a shared service. No customer's
|
||||
# data is in it. Complete enough to run with nobody to ask.
|
||||
metadata:
|
||||
customer_case: bench-landing-zone-shared
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
intent:
|
||||
statement: |
|
||||
We want our landing zone run as a shared service from go-live. A provider should
|
||||
respond to faults at every hour, and our own platform team keeps the final say
|
||||
over changes to the network.
|
||||
business_outcomes:
|
||||
- {id: BO-1, outcome: New payment products reach customers in weeks rather than months., target: "A change reaches production within a day of approval."}
|
||||
- {id: BO-2, outcome: Payments are never unavailable for long., target: "Restored within 4 hours."}
|
||||
- {id: BO-3, outcome: We keep the platform engineers we hire.}
|
||||
requirements:
|
||||
- {id: REQ-01, statement: Our platform team approves every change to the network.}
|
||||
- {id: REQ-02, statement: We receive a monthly report of service levels and cost.}
|
||||
solution:
|
||||
cloud_provider: gcp
|
||||
items:
|
||||
- {id: S-1, name: Hub network and interconnect, kind: network, product: Cloud Interconnect}
|
||||
- {id: S-2, name: Audit log store, kind: storage, product: Cloud Storage}
|
||||
- {id: S-3, name: Deployment pipeline, kind: pipeline, product: Cloud Build}
|
||||
- {id: S-4, name: Encryption keys, kind: keys, product: Cloud KMS}
|
||||
conditions:
|
||||
- {id: K-1, "on": [S-1], condition: The platform is available at least 99.9% of each month.}
|
||||
- {id: K-2, "on": [S-2], condition: Audit logs cannot be deleted for seven years.}
|
||||
- {id: K-3, "on": [S-4], condition: Encryption keys are rotated every 90 days.}
|
||||
build:
|
||||
duration_days: 36
|
||||
operations:
|
||||
support_hours: 24x7
|
||||
availability_hours: 24x7
|
||||
change_process: decide
|
||||
sourcing:
|
||||
model: shared
|
||||
delivery_countries: [GB, PL, IN]
|
||||
organisation:
|
||||
existing_teams:
|
||||
- {name: Platform team, party: customer, hours: business, country: GB}
|
||||
service_management:
|
||||
tooling: decide
|
||||
service_levels:
|
||||
priority_1_response_minutes: 15
|
||||
recovery_time_hours: 4
|
||||
compliance:
|
||||
frameworks: [DORA]
|
||||
commercial:
|
||||
currency: GBP
|
||||
monthly_run_budget: 60000
|
||||
run:
|
||||
pre_approved: true
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"customer_case": "bench-landing-zone-shared",
|
||||
"intents": [
|
||||
{
|
||||
"version": 1,
|
||||
"recorded_at": "2026-10-21T20:36:25+00:00",
|
||||
"statement": "intent/001.md",
|
||||
"workflow_request": "We want our landing zone run as a shared service from go-live. A provider should\nrespond to faults at every hour, and our own platform team keeps the final say\nover changes to the network.",
|
||||
"execution_id": "case-01-landing-zone-shared-r1",
|
||||
"session_id": "9fe31b7a-c779-4d34-bcee-cc2e36533a1d",
|
||||
"status": "completed",
|
||||
"execution": "executions/case-01-landing-zone-shared-r1",
|
||||
"baseline": null,
|
||||
"scope": {
|
||||
"completed": "intent/001.scope.completed.yaml",
|
||||
"supplied": "intent/001.scope.yaml"
|
||||
},
|
||||
"sources": null
|
||||
},
|
||||
{
|
||||
"version": 2,
|
||||
"recorded_at": "2026-10-21T20:54:36+00:00",
|
||||
"statement": "intent/002.md",
|
||||
"workflow_request": "We want our landing zone run as a shared service from go-live. A provider should\nrespond to faults at every hour, and our own platform team keeps the final say\nover changes to the network.",
|
||||
"execution_id": "case-01-landing-zone-shared-r2",
|
||||
"session_id": "c7cf78e8-2f3b-4f26-bd2f-850de41fe1d6",
|
||||
"status": "completed",
|
||||
"execution": "executions/case-01-landing-zone-shared-r2",
|
||||
"baseline": null,
|
||||
"scope": {
|
||||
"completed": "intent/002.scope.completed.yaml",
|
||||
"supplied": "intent/002.scope.yaml"
|
||||
},
|
||||
"sources": null
|
||||
},
|
||||
{
|
||||
"version": 3,
|
||||
"recorded_at": "2026-10-21T21:11:10+00:00",
|
||||
"statement": "intent/003.md",
|
||||
"workflow_request": "We want our landing zone run as a shared service from go-live. A provider should\nrespond to faults at every hour, and our own platform team keeps the final say\nover changes to the network.",
|
||||
"execution_id": "case-01-landing-zone-shared-r3",
|
||||
"session_id": "d0d953a6-5254-4646-8d70-7fd6139f1afc",
|
||||
"status": "completed",
|
||||
"execution": "executions/case-01-landing-zone-shared-r3",
|
||||
"baseline": null,
|
||||
"scope": {
|
||||
"completed": "intent/003.scope.completed.yaml",
|
||||
"supplied": "intent/003.scope.yaml"
|
||||
},
|
||||
"sources": null
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user