Files
tom_analysis_agent/.agents/plans/iter-01-first-slice/steps/step-3.md

6.9 KiB
Raw Blame History

Socio-technical and organisational frameworks

Added 2026-10-14, at the owner's request.

What was added

  • Six frameworks in the register (skills/operating-capabilities/assets/sources.yaml): Team Topologies; platform as a product, cited from the CNCF Platforms White Paper; the SPACE framework; and the DevEx framework. The DORA delivery metrics were already there. Like every other entry, their references are unverified.
  • Four Capabilities in the catalogue: platform product management, self-service and golden paths, and enablement, proposed for a platform foundation; and delivery and developer experience measurement, proposed for a platform foundation or a workload. The catalogue can now propose an entry by the kind of engagement.
  • Measurement at three layers. The measurement Capability cites DORA, SPACE and DevEx together, each for a different layer: delivery speed and stability as the outcome, sustainable productivity as the guardrail, friction as the leading signal. The Skill says no one layer stands in for another.
  • Teams and interactions in the graph, both optional. A Team is of one of Team Topologies' four kinds, in one Party; a Role may be in a Team of its own Party. An Interaction between two Teams is in one of the three modes, and a collaboration must say when it ends. The organisation chart draws teams inside their Parties and interactions as labelled edges; the guide lists both.
  • Two new terms in CONTEXT.md: Team and Interaction.
  • The architect's instructions say how to use teams and interaction modes, and that a Check goes on the Capability that does the work, not the one that reports it. The second was the cause of most Revisions in the runs of 2026-10-14.

Not done

  • No run with real models since. The four new Capabilities add to what the capability modeller takes, which was already all 17 entries for a platform foundation; it will now be 21.
  • Teams are not used by staffing, cost or the scenario walk-throughs.

Verified

159 tests pass; ruff and mypy clean.

The run with real models after this change, 2026-10-14

Case Minutes Revisions Capabilities Teams Interactions Checks Scenarios People Cost a month Model cost
01, written spec 20.9 (was 25.8) operating model ×1 (was 3 in all) 21 2 2 13 13 of 13 6.6 32,234 GBP $1.87 (was $2.27)
02, digested architecture 16.6 (was 16.8) operating model ×1 (was 1) 21 3 2 31 35 of 35 12.5 77,550 GBP $1.61 (was $1.55)

Both passed every check. The one Revision in each was still a Check whose hours or Capability did not fit, but the capability model was not reopened in either. Both models used teams: platform and stream-aligned teams, an X-as-a-Service interaction in each, and in case 01 a collaboration that ends three months after go-live, in case 02 a facilitating interaction that ends with the build. Case 02 named six Parties.

Business outcomes

Added 2026-10-14, at the owner's request: every measure is tied back to an outcome the business stated.

  • intent.business_outcomes in the operating spec: each with an id, the customer's words, and a target where given. The digest carries the architecture's stated success criteria across as business outcomes.
  • Measures in the graph: what is measured, its layer (outcome, guardrail, leading, service level or cost), its framework, its target, the outcome it shows and the Capability that takes it. Refused: a stated outcome shown by no Measure, a Measure for an outcome that was not stated, a Measure taken by a Capability nobody is responsible for, a framework not in the register, and, where the model measures how work flows, any of the three developer layers left unmeasured.
  • An Operational requirement may come from a business outcome. It need not: the Measures are what show each outcome is served.
  • The guide's second section is Business outcomes, written by code: each outcome with the Measures that show it and the Role answerable for each.
  • Two new terms in CONTEXT.md: Business outcome and Measure. The benchmark spec now states three outcomes.

Not yet run with real models. 173 tests pass; ruff and mypy clean.

The run with business outcomes, 2026-10-14

  • Case 01 passed in 29.2 minutes, for $2.73. It tied four Measures to the spec's three business outcomes: deployment lead time (DORA, outcome layer) to faster releases, time to restore service (DORA) to short outages, and satisfaction and wellbeing (SPACE, guardrail) and feedback loops and cognitive load (DevEx, leading) both to keeping engineers. Two Revisions of the operating model and one of the capability model.
  • Case 02 looped and was stopped after about two and a half hours. The architecture states no business outcome, so every Measure was refused as serving an outcome "none were stated", while the measurement Capability's rule asked for Measures at three layers. The two rules could not both be met. The architect asked a question five times in a pre-approved run, each answered automatically, each run taking 10 to 28 minutes, and the last model request hung. About 173 Pro requests; an estimated $7 to $10.

Fixed after:

  • intent.business_outcomes is essential. With someone present it is asked for; a pre-approved start must state them or leave them to the agent with decide. Case 02 now does the latter.
  • The three-layer rule applies only when there is an outcome to tie Measures to.
  • The evaluation harness stops a case at one and a half times its time limit and records it as stopped.

Not fixed: the engine lets a pre-approved Specialist end without submitting again and again, each time answered automatically, until the Orchestrator's turn limit. A guard there would have stopped case 02 within minutes.

Case 02 rerun, and findings kept to the solution, 2026-10-14

Case 02 passed with its business outcomes left to the agent: 31.6 minutes, $2.71, two Revisions. The agent assumed three outcomes from the request (the landing zone ready within 10 weeks, continuous DORA and PCI DSS compliance, high availability with low-latency hybrid connectivity) and tied six Measures to them, all three developer layers on the first.

It also filed a gap in its own model as a finding: service continuity staffed only in business hours against a 4-hour recovery target. That drafted a change request to the architecture agent for a fault that was the operating model's. Fixed:

  • A finding names the Items of the solution it is about. One with none is refused.
  • The validator judges each finding. A Check waiting on a finding the operating model could meet itself (staffing, hours, a Role, a process, a tool) is given fail, which sends the model back to the architect. Only a Check left not_verifiable is recorded as waiting on its finding.
  • The architect is told what a finding is and is not.

177 tests pass. Not yet run with real models.