Files
tom_analysis_agent/.agents/plans/iter-01-first-slice/steps/step-2.md

3.2 KiB
Raw Blame History

Steps T3, T4, T11 and T14: the digest, the guide, and the first runs with real models

Built 2026-10-13 and 14, with the engine's digest step (its steps/step-6.md).

What was built

  • The architecture adapter (app/operating_model/digest.py, python -m app.operating_model.digest <folder>). An Item is a component of the architecture, with the Elements that realise it; a Condition is what its requirements and recommendations say can only be shown in operation (outside_code); a Check that did not pass is an inherited risk. Only what the customer stated in the architecture's scope is carried across. The provider bundle's operating.item_kinds says what kind of thing each product is. Tested against a real completed landing-zone Execution, copied into tests/fixtures/architecture/.
  • Documents code cannot read go to the scope analyst, who records what it takes from them with the passage quoted (T4, as instructions; no model has run it).
  • The guide (app/operating_model/guide.py, skills/operating-guide/). The packager writes the summary; code writes every other section and the key figures from the data.
  • The evaluation harness (eval/): two cases, structural scoring, tokens per role.

Differed from the plan

  • An Item is a component, not an Element. The real architecture had 60 Elements in 9 components; an organisation operates the components.
  • The architecture's 104 Checks on infrastructure settings are not Conditions. Keeping them true is configuration management. Its 12 statements of what can only be shown in operation are.

The runs with real models, 2026-10-14

Case Result Minutes Revisions Capabilities Roles Checks Scenarios People Cost a month
01, written spec (4 Items) pass 25.8 operating model ×2, capability model ×1 17 4 8 13 of 13 5.2 25,355 GBP
02, digested architecture (9 Items) pass 16.8 operating model ×1 17 5 36 31 of 31 11.6 67,298 GBP

Costs are from the placeholder rate card. An earlier run of case 01 passed every structural check in 30.6 minutes, over its limit then of 30.

What they showed:

  • Most time goes on the architect placing a Check on the wrong Capability, such as a restore-time Check on service-level reporting. Case 01 revised the same way in both runs. Not yet acted on.
  • The router's reasons still speak of code. A Check that could not be verified reopens the capability model with "could not be verified from the code", and a Check that failed twice is said to be possibly "not implementable". These are the engine's words from the architecture's subject.
  • The capability modeller takes all 17 catalogue entries whatever the size of the solution, as its instructions say.
  • The architect is most of the cost: about 380,000 to 490,000 of each case's 780,000 to 940,000 prompt tokens.
  • One request was retried after Vertex AI's tokens-per-minute quota was reached; it did not affect the result.

Nobody has yet read the outputs for quality: these checks are structural only.

Verified

The engine's 529 tests and this repository's 140 pass, with ruff and mypy clean in both.