feat: working TOM agent
This commit is contained in:
44
.agents/plans/iter-01-first-slice/steps/step-1.md
Normal file
44
.agents/plans/iter-01-first-slice/steps/step-1.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Steps T1, T2, T5 to T13: the Workflow, from a written operating spec
|
||||
|
||||
Built 2026-10-13, on the Execution engine as it stood that day (not yet committed there).
|
||||
|
||||
## What was built
|
||||
|
||||
- **The Workflow** (`app/operating_model/workflow.py`): five Phases, six Deliverables, each with its Specialist, sections and data file; the validation policy; the views; the account of a Change request; the card.
|
||||
- **The operating spec's catalogue** (`skills/operating-scoping/`): 32 fields, read by the engine's spec reader. The provider bundle supplies the Cloud provider and the currencies.
|
||||
- **The capability catalogue** (`skills/operating-capabilities/`): 17 entries in five domains, each citing frameworks from a register of nine. `catalogue.py` refuses an entry that cites nothing or cites an unregistered framework, and proposes the entries that apply to a spec.
|
||||
- **The chain** (`chain.py`): the checks on operational requirements and on capabilities.
|
||||
- **The graph** (`graph.py`): its shape for the engine's graph tools, its refusals, findings, and the rendered responsibility matrix, organisation chart, and staffing and cost.
|
||||
- **Staffing and cost** (`staffing.py`): the rate card's loader and checks, and the arithmetic.
|
||||
- **Scenarios** (`scenarios.py`): five kinds, generated and traced by code, their verdicts written into `verdicts.json`.
|
||||
- **The agent's folder**: eight role files and a shared protocol, `providers/gcp.yaml`, the `operating-chain` Skill, `app/main.py`.
|
||||
- **106 tests**, among them seven whole Executions on the scripted runtime.
|
||||
|
||||
## Differed from the plan and the spec
|
||||
|
||||
- **The spec's files keep the engine's names,** `scope.yaml` and `scope.json`. The plan said `operating-spec.yaml`. The engine's spec support does not yet let a Workflow name them.
|
||||
- **The inventory travels with the rest of the spec** to every Specialist, not as a separate `solution.json`. The engine's digest step (E6) is where that changes.
|
||||
- **The graph holds no list of Capabilities.** They are the capability model's, named by id, so a failed Check can reopen who does the work without reopening what the work is.
|
||||
- **The currencies a spec may state are listed in the provider bundle,** so the engine refuses another at the start. A test keeps the list the same as the rate card's conversion rates.
|
||||
- **A Role on business hours is as small as its work.** The spec's formula applied the minimum roster to every Role; the plan, which the owner approved, applies it only to hours wider than a working day. The plan was followed.
|
||||
- **A Check that depends on a finding is recorded as `waits_on_finding`.** The validator gives it `not_verifiable`; code rewrites the verdict when the validation is submitted. Left as `not_verifiable`, the engine's router reopened the capability model for it, again and again: a finding is not a gap in the model.
|
||||
- **A scenario's verdicts are written into `verdicts.json` by code,** as Checks whose ids begin `SC-`, and the walk-through as a whole is the file's `dry_run`. A failed scenario is therefore a failed Check to the router, which reopens the graph.
|
||||
- **The matrix shows both parts** when one Role is accountable and responsible for a Capability (`A/R`).
|
||||
|
||||
## Not built
|
||||
|
||||
- **The digest** (T3, T4). An Execution starts from an operating spec that is already written. Nothing reads the solution architecture agent's files yet.
|
||||
- **A guide template** (T11). The packager has its sections and its instructions.
|
||||
- **Evaluation** (T14). Nothing has run with a real model: no role file, Skill or catalogue entry has been read by one.
|
||||
- An interactive Execution and a Change request are not tested here; the engine's own tests cover both on other Workflows.
|
||||
- The capability catalogue's references are drafted from general knowledge and marked `unverified`. Nobody has checked them against the frameworks.
|
||||
- The rate card's figures are placeholders.
|
||||
|
||||
## What it found in the engine
|
||||
|
||||
- The engine was not typed as a package (`py.typed` added there), so this repository could not be type-checked against it.
|
||||
- An editable install of the engine is invisible to the type checker unless made with `editable_mode=compat`.
|
||||
|
||||
## Verified
|
||||
|
||||
`pytest -q`: 106 passed. `ruff check .`, `ruff format --check .`, `mypy .` (20 files): clean. The engine's suite still passes (521). `git status` in the solution architecture agent shows the same nine modified files as before any of this work began.
|
||||
38
.agents/plans/iter-01-first-slice/steps/step-2.md
Normal file
38
.agents/plans/iter-01-first-slice/steps/step-2.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Steps T3, T4, T11 and T14: the digest, the guide, and the first runs with real models
|
||||
|
||||
Built 2026-10-13 and 14, with the engine's digest step (its `steps/step-6.md`).
|
||||
|
||||
## What was built
|
||||
|
||||
- **The architecture adapter** (`app/operating_model/digest.py`, `python -m app.operating_model.digest <folder>`). An Item is a component of the architecture, with the Elements that realise it; a Condition is what its requirements and recommendations say can only be shown in operation (`outside_code`); a Check that did not pass is an inherited risk. Only what the customer stated in the architecture's scope is carried across. The provider bundle's `operating.item_kinds` says what kind of thing each product is. Tested against a real completed landing-zone Execution, copied into `tests/fixtures/architecture/`.
|
||||
- **Documents code cannot read** go to the scope analyst, who records what it takes from them with the passage quoted (T4, as instructions; no model has run it).
|
||||
- **The guide** (`app/operating_model/guide.py`, `skills/operating-guide/`). The packager writes the summary; code writes every other section and the key figures from the data.
|
||||
- **The evaluation harness** (`eval/`): two cases, structural scoring, tokens per role.
|
||||
|
||||
## Differed from the plan
|
||||
|
||||
- An Item is a component, not an Element. The real architecture had 60 Elements in 9 components; an organisation operates the components.
|
||||
- The architecture's 104 Checks on infrastructure settings are not Conditions. Keeping them true is configuration management. Its 12 statements of what can only be shown in operation are.
|
||||
|
||||
## The runs with real models, 2026-10-14
|
||||
|
||||
| Case | Result | Minutes | Revisions | Capabilities | Roles | Checks | Scenarios | People | Cost a month |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 01, written spec (4 Items) | pass | 25.8 | operating model ×2, capability model ×1 | 17 | 4 | 8 | 13 of 13 | 5.2 | 25,355 GBP |
|
||||
| 02, digested architecture (9 Items) | pass | 16.8 | operating model ×1 | 17 | 5 | 36 | 31 of 31 | 11.6 | 67,298 GBP |
|
||||
|
||||
Costs are from the placeholder rate card. An earlier run of case 01 passed every structural check in 30.6 minutes, over its limit then of 30.
|
||||
|
||||
What they showed:
|
||||
|
||||
- **Most time goes on the architect placing a Check on the wrong Capability**, such as a restore-time Check on service-level reporting. Case 01 revised the same way in both runs. Not yet acted on.
|
||||
- **The router's reasons still speak of code.** A Check that could not be verified reopens the capability model with "could not be verified from the code", and a Check that failed twice is said to be possibly "not implementable". These are the engine's words from the architecture's subject.
|
||||
- **The capability modeller takes all 17 catalogue entries** whatever the size of the solution, as its instructions say.
|
||||
- **The architect is most of the cost:** about 380,000 to 490,000 of each case's 780,000 to 940,000 prompt tokens.
|
||||
- One request was retried after Vertex AI's tokens-per-minute quota was reached; it did not affect the result.
|
||||
|
||||
Nobody has yet read the outputs for quality: these checks are structural only.
|
||||
|
||||
## Verified
|
||||
|
||||
The engine's 529 tests and this repository's 140 pass, with `ruff` and `mypy` clean in both.
|
||||
67
.agents/plans/iter-01-first-slice/steps/step-3.md
Normal file
67
.agents/plans/iter-01-first-slice/steps/step-3.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Socio-technical and organisational frameworks
|
||||
|
||||
Added 2026-10-14, at the owner's request.
|
||||
|
||||
## What was added
|
||||
|
||||
- **Six frameworks in the register** (`skills/operating-capabilities/assets/sources.yaml`): Team Topologies; platform as a product, cited from the CNCF Platforms White Paper; the SPACE framework; and the DevEx framework. The DORA delivery metrics were already there. Like every other entry, their references are unverified.
|
||||
- **Four Capabilities in the catalogue:** platform product management, self-service and golden paths, and enablement, proposed for a platform foundation; and delivery and developer experience measurement, proposed for a platform foundation or a workload. The catalogue can now propose an entry by the kind of engagement.
|
||||
- **Measurement at three layers.** The measurement Capability cites DORA, SPACE and DevEx together, each for a different layer: delivery speed and stability as the outcome, sustainable productivity as the guardrail, friction as the leading signal. The Skill says no one layer stands in for another.
|
||||
- **Teams and interactions in the graph,** both optional. A Team is of one of Team Topologies' four kinds, in one Party; a Role may be in a Team of its own Party. An Interaction between two Teams is in one of the three modes, and a collaboration must say when it ends. The organisation chart draws teams inside their Parties and interactions as labelled edges; the guide lists both.
|
||||
- **Two new terms** in `CONTEXT.md`: Team and Interaction.
|
||||
- **The architect's instructions** say how to use teams and interaction modes, and that a Check goes on the Capability that does the work, not the one that reports it. The second was the cause of most Revisions in the runs of 2026-10-14.
|
||||
|
||||
## Not done
|
||||
|
||||
- No run with real models since. The four new Capabilities add to what the capability modeller takes, which was already all 17 entries for a platform foundation; it will now be 21.
|
||||
- Teams are not used by staffing, cost or the scenario walk-throughs.
|
||||
|
||||
## Verified
|
||||
|
||||
159 tests pass; `ruff` and `mypy` clean.
|
||||
|
||||
## The run with real models after this change, 2026-10-14
|
||||
|
||||
| Case | Minutes | Revisions | Capabilities | Teams | Interactions | Checks | Scenarios | People | Cost a month | Model cost |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 01, written spec | 20.9 (was 25.8) | operating model ×1 (was 3 in all) | 21 | 2 | 2 | 13 | 13 of 13 | 6.6 | 32,234 GBP | $1.87 (was $2.27) |
|
||||
| 02, digested architecture | 16.6 (was 16.8) | operating model ×1 (was 1) | 21 | 3 | 2 | 31 | 35 of 35 | 12.5 | 77,550 GBP | $1.61 (was $1.55) |
|
||||
|
||||
Both passed every check. The one Revision in each was still a Check whose hours or Capability did not fit, but the capability model was not reopened in either. Both models used teams: platform and stream-aligned teams, an X-as-a-Service interaction in each, and in case 01 a collaboration that ends three months after go-live, in case 02 a facilitating interaction that ends with the build. Case 02 named six Parties.
|
||||
|
||||
# Business outcomes
|
||||
|
||||
Added 2026-10-14, at the owner's request: every measure is tied back to an outcome the business stated.
|
||||
|
||||
- **`intent.business_outcomes`** in the operating spec: each with an id, the customer's words, and a target where given. The digest carries the architecture's stated success criteria across as business outcomes.
|
||||
- **Measures** in the graph: what is measured, its layer (outcome, guardrail, leading, service level or cost), its framework, its target, the outcome it shows and the Capability that takes it. Refused: a stated outcome shown by no Measure, a Measure for an outcome that was not stated, a Measure taken by a Capability nobody is responsible for, a framework not in the register, and, where the model measures how work flows, any of the three developer layers left unmeasured.
|
||||
- **An Operational requirement may come from a business outcome.** It need not: the Measures are what show each outcome is served.
|
||||
- **The guide's second section is Business outcomes,** written by code: each outcome with the Measures that show it and the Role answerable for each.
|
||||
- **Two new terms** in `CONTEXT.md`: Business outcome and Measure. The benchmark spec now states three outcomes.
|
||||
|
||||
Not yet run with real models. 173 tests pass; `ruff` and `mypy` clean.
|
||||
|
||||
## The run with business outcomes, 2026-10-14
|
||||
|
||||
- **Case 01 passed** in 29.2 minutes, for $2.73. It tied four Measures to the spec's three business outcomes: deployment lead time (DORA, outcome layer) to faster releases, time to restore service (DORA) to short outages, and satisfaction and wellbeing (SPACE, guardrail) and feedback loops and cognitive load (DevEx, leading) both to keeping engineers. Two Revisions of the operating model and one of the capability model.
|
||||
- **Case 02 looped and was stopped** after about two and a half hours. The architecture states no business outcome, so every Measure was refused as serving an outcome "none were stated", while the measurement Capability's rule asked for Measures at three layers. The two rules could not both be met. The architect asked a question five times in a pre-approved run, each answered automatically, each run taking 10 to 28 minutes, and the last model request hung. About 173 Pro requests; an estimated $7 to $10.
|
||||
|
||||
Fixed after:
|
||||
|
||||
- `intent.business_outcomes` is essential. With someone present it is asked for; a pre-approved start must state them or leave them to the agent with `decide`. Case 02 now does the latter.
|
||||
- The three-layer rule applies only when there is an outcome to tie Measures to.
|
||||
- The evaluation harness stops a case at one and a half times its time limit and records it as stopped.
|
||||
|
||||
Not fixed: the engine lets a pre-approved Specialist end without submitting again and again, each time answered automatically, until the Orchestrator's turn limit. A guard there would have stopped case 02 within minutes.
|
||||
|
||||
## Case 02 rerun, and findings kept to the solution, 2026-10-14
|
||||
|
||||
Case 02 passed with its business outcomes left to the agent: 31.6 minutes, $2.71, two Revisions. The agent assumed three outcomes from the request (the landing zone ready within 10 weeks, continuous DORA and PCI DSS compliance, high availability with low-latency hybrid connectivity) and tied six Measures to them, all three developer layers on the first.
|
||||
|
||||
It also filed a gap in its own model as a finding: service continuity staffed only in business hours against a 4-hour recovery target. That drafted a change request to the architecture agent for a fault that was the operating model's. Fixed:
|
||||
|
||||
- **A finding names the Items of the solution it is about.** One with none is refused.
|
||||
- **The validator judges each finding.** A Check waiting on a finding the operating model could meet itself (staffing, hours, a Role, a process, a tool) is given `fail`, which sends the model back to the architect. Only a Check left `not_verifiable` is recorded as waiting on its finding.
|
||||
- **The architect is told** what a finding is and is not.
|
||||
|
||||
177 tests pass. Not yet run with real models.
|
||||
Reference in New Issue
Block a user