Merge pull request 'docs: record the first run with real models, through the Orchestration engine' (#1) from docs/first-real-run into main
Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
@@ -4,7 +4,7 @@ One directory per iteration.
|
|||||||
|
|
||||||
| # | Directory | Status | What it is |
|
| # | Directory | Status | What it is |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| 1 | `iter-01-on-the-engine/` | step 1 built 2026-10-20 | The reference agent's subject as a Workflow on the shared Execution engine, in the operating model agent's pattern, behind the Orchestration engine |
|
| 1 | `iter-01-on-the-engine/` | step 1 built 2026-10-20; step 2 run 2026-10-21 | The reference agent's subject as a Workflow on the shared Execution engine, in the operating model agent's pattern, behind the Orchestration engine |
|
||||||
|
|
||||||
Each iteration has a `MAP.md` (intent, each step's status, sources of truth) and `steps/step-N.md`, written when a step is finished. A step with no file is not done.
|
Each iteration has a `MAP.md` (intent, each step's status, sources of truth) and `steps/step-N.md`, written when a step is finished. A step with no file is not done.
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
**Iteration:** 1, on the engine
|
**Iteration:** 1, on the engine
|
||||||
**Intent:** The reference agent's subject, cloud solution architecture, as an agent in the operating model agent's pattern: the shared Execution engine and one Workflow, called through the Orchestration engine, its files passed on to the operating model agent.
|
**Intent:** The reference agent's subject, cloud solution architecture, as an agent in the operating model agent's pattern: the shared Execution engine and one Workflow, called through the Orchestration engine, its files passed on to the operating model agent.
|
||||||
**Status:** step 1 built 2026-10-20
|
**Status:** step 1 built 2026-10-20; step 2 run 2026-10-21
|
||||||
|
|
||||||
## Decided by the owner, 2026-10-20
|
## Decided by the owner, 2026-10-20
|
||||||
|
|
||||||
@@ -21,5 +21,5 @@
|
|||||||
| Step | What | Done when | Status |
|
| Step | What | Done when | Status |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| 1 | The agent on the engine: its folder, `app/` as the Workflow, the image, on the local platform behind the Orchestration engine | A pre-approved Execution on the agent's own roles, Skills and provider bundle delivers every file the operating model agent digests; the container answers on the platform | built (`steps/step-1.md`) |
|
| 1 | The agent on the engine: its folder, `app/` as the Workflow, the image, on the local platform behind the Orchestration engine | A pre-approved Execution on the agent's own roles, Skills and provider bundle delivers every file the operating model agent digests; the container answers on the platform | built (`steps/step-1.md`) |
|
||||||
| 2 | A run with real models through the Orchestration engine, then the sequence into the operating model agent | One request for `architecture_and_operating_model` yields both sets of Deliverables under one Customer case | planned |
|
| 2 | A run with real models through the Orchestration engine, then the sequence into the operating model agent | One request for `architecture_and_operating_model` yields both sets of Deliverables under one Customer case | built (`steps/step-2.md`): completed 2026-10-21 in 74.9 minutes |
|
||||||
| 3 | The evaluation harness on the engine, from the reference's `eval/` | The reference's cases run against this agent | planned |
|
| 3 | The evaluation harness on the engine, from the reference's `eval/` | The reference's cases run against this agent | planned |
|
||||||
|
|||||||
17
.agents/plans/iter-01-on-the-engine/steps/step-2.md
Normal file
17
.agents/plans/iter-01-on-the-engine/steps/step-2.md
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
# Step 2: a run with real models through the Orchestration engine, into the operating model agent
|
||||||
|
|
||||||
|
Run 2026-10-21.
|
||||||
|
|
||||||
|
One pre-approved request to the Orchestration engine for `architecture_and_operating_model`, Customer case `case-08-landing-zone`, from the benchmark spec `eval/datasets/specs/landing-zone.yaml`, the operating model agent's own fields under `operating_model`. Models: Gemini on Vertex AI; Google Cloud's remote MCP servers on.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
- **This agent's Execution** (`6c316f05…`) completed in about 50 minutes on the shared Execution engine: its first run with real models since it was built from the reference. Fifteen files kept: the solution guide, the scope, requirements, components, products, recommendations, the Target architecture and its diagram, implementation, verdicts, the work breakdown structure, and Terraform. The reference took 55 to 60 minutes on the same spec on its own engine.
|
||||||
|
- **The operating model agent** was given those files as `sources` with the Orchestration engine's record that the work completed, digested them, and completed: 24 Operational requirements, 21 Capabilities, 8 Roles, 26 Checks, 44 of 44 scenarios, 12.2 people at 87,959 GBP a month.
|
||||||
|
- 74.9 minutes end to end. Both Executions are kept in the Orchestration engine's store and were fetched through their links.
|
||||||
|
|
||||||
|
The record of the run, its report and every file: `../tom_analysis/eval/results/20261021T203623Z-handoff/` (not in a repository).
|
||||||
|
|
||||||
|
## Not done
|
||||||
|
|
||||||
|
- The evaluation harness (step 3): this agent's own checks on what it delivered were not run; the run was judged by the operating model agent completing from it.
|
||||||
Reference in New Issue
Block a user