The reference agent's subject, cloud solution architecture, as an agent in the operating model agent's pattern: the Execution engine (from orchestration_engine) and one Workflow in app/solution_architecture, built for the settings and provider bundle by assembly.workflow_for. Copied from gcp_solution_architecture_agent at 8b397cd without its own engine: its role files, Skills, provider bundle, ADRs, vocabulary, scripts and evaluation datasets. The image takes the engine from a named build context and installs Terraform. It is called through the Orchestration engine, and its files are what the operating model agent digests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
91 lines
9.9 KiB
Markdown
91 lines
9.9 KiB
Markdown
---
|
|
status: accepted
|
|
---
|
|
|
|
# Jev is the Orchestrator's decision point, and the Orchestrator is a router
|
|
|
|
[ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev ([Jev in AI agent loops](https://vercel.com/i/jev-agent-control)): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.
|
|
|
|
## Who owns what
|
|
|
|
| Responsibility | Owner in the pattern | Here |
|
|
|---|---|---|
|
|
| Interpret the request | A model with supplied context | The requirements analyst and the Specialists after it |
|
|
| Propose the next action | The decision model | Jev, asked typed questions about the Execution's state |
|
|
| Check permission | Application code | The Ledger: Dependencies, Approvals, the Revision limit, the cloud-read gate |
|
|
| Execute the action | The tool runner | The Orchestrator's existing tools, called by code |
|
|
| Update state | Application code | The Execution graph and its checkpoint |
|
|
| Explain the result | A generative model | One small model call that writes an Approval summary or a question, from data it is given |
|
|
|
|
## The Orchestrator is a router
|
|
|
|
With Jev as its decision point, the Orchestrator's job is routing: it reads what has arrived and the state of the Execution, asks Jev which route applies, and dispatches. It does no work of its own. There are four routes to choose.
|
|
|
|
| What is routed | The routes Jev chooses between | Who is dispatched |
|
|
|---|---|---|
|
|
| A message that starts something | a new Workflow request; a Change request against a Baseline; a question about work already delivered; not something this agent does | a full Execution; an Execution from the Baseline; the reply writer, from the delivered files; a polite refusal |
|
|
| A message inside an Execution | an answer to a waiting question; a request to change submitted work, and which Deliverable; anything else | the Specialist that asked; a Revision of that Deliverable; a question back to the user |
|
|
| A validation outcome | revise the code; revise the Target architecture; proceed | the IaC author; the target architect; the Approval |
|
|
| Ready work, in a Change request | which Deliverables the change reaches, and which carry over from the Baseline unchanged | only the Specialists whose Deliverables are reached |
|
|
|
|
Which Specialist produces a given Deliverable is still fixed by the dependency graph: Jev chooses the route, never the worker for it. And the Ledger still refuses a route whose Dependencies or Approvals are not met.
|
|
|
|
## How a turn is decided: readings, disposition, playbook
|
|
|
|
The owner's model for this is a four-stage pipeline: readings of what arrived, a disposition worked out in code, further readings of the state, and a playbook that takes the first group that applies and, within it, the strongest action whose conditions hold, escalating when no group applies.
|
|
|
|
1. **Readings.** Facts about the turn. Most are read from the Ledger and the Execution's data by code: are questions waiting, is a Phase complete, is work ready, which Checks failed, did a Check fail twice running. The ones that need judgement are typed answers from Jev: is the user asking for a change (yes or no), which Deliverable would have to change first (a choice), how specific are they (a score), does a gap in the Target architecture leave a requirement unmet (yes or no). Jev reads; it never picks the action.
|
|
2. **Disposition.** Code turns a judged reading into act, queue or close by threshold: a probability of 0.75 or more is acted on, 0.15 or less is dropped, and anything between is put to the user. A choice with confidence under 0.6 is not relied on. The thresholds are settings.
|
|
3. **Playbook.** Ordered groups; the first that applies is taken, and within it the strongest action whose conditions hold:
|
|
|
|
| Group | Actions, strongest first |
|
|
|---|---|
|
|
| The user asked for a change | revise the Deliverable it affects; else ask them to confirm which and what |
|
|
| Questions are waiting | relay them |
|
|
| Validation decided | revise the Target architecture (a Check failed twice running, or a gap that matters); else revise the code (a Check failed); else ask about the gaps; else proceed |
|
|
| A Phase is complete | request its Approval |
|
|
| Work is ready | delegate all of it |
|
|
| Packaging is approved | finish |
|
|
| No group applies | escalate: ask the user, or end with warnings when there is none |
|
|
|
|
4. **Permission.** Every action goes through the Orchestrator's tools, so the Ledger refuses what is not allowed whatever was decided, and the playbook then takes the next action that is. A typed answer can still be wrong.
|
|
5. **Record.** Each decision is kept in the Execution's state: the turn, the group, the action and what Jev answered with its confidence.
|
|
6. **Reply.** A generative model writes only what the user reads, the summary they approve a Phase from, out of data it is given. It runs after the decision and cannot change it. When it fails, the summary is assembled from the same data.
|
|
|
|
A failed call to Jev is not an answer. The client returns nothing, and the reading falls back to a rule that acts only on what is certain: a change is acted on when the Deliverable is named outright, and a gap is revised for when it names a real requirement or cannot be verified.
|
|
|
|
## Decisions
|
|
|
|
Accepted by the owner on 2026-10-07, with each of these:
|
|
|
|
1. **No language model decides what the Orchestrator does.** Jev reads, and the playbook in code decides. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
|
|
2. **Low confidence goes to the user, not to another model.** Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
|
|
3. **Jev being unreachable does not stop an Execution.** After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
|
|
4. **What is sent to Jev is the Execution's state, not its content:** Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
|
|
5. **The Orchestrator routes; Jev chooses the route.** That covers what kind of work a message starts, where a message inside an Execution goes, where a validation outcome goes, and, in a Change request, which Deliverables are reached. Jev never chooses which Specialist produces a Deliverable: the dependency graph fixes that.
|
|
6. **In a new Execution every Deliverable is still produced.** Routing does not skip a Specialist for a new Workflow request, because each later Deliverable is checked against the ones before it. Skipping happens only in a Change request, where the Baseline supplies what is carried over.
|
|
|
|
## Considered Options
|
|
|
|
- **A smaller language model as Orchestrator:** rejected by the owner. It keeps the slow, open-ended turn and the stalls, at lower cost.
|
|
- **No generative model at all, with fixed wording for summaries and questions:** this ADR's first draft. Rejected to follow the pattern, which pairs Jev with a generative model for replies: an Approval summary is what a person decides from and reads better written than assembled.
|
|
- **Code only, with fixed rules:** rejected. It cannot tell what a user's free-text reply is asking for.
|
|
- **Code for the mechanical steps and the existing model for judgement:** rejected in favour of one decision point. It keeps two ways of deciding and the large model in the loop.
|
|
|
|
## Consequences
|
|
|
|
- Jev is reached through TypeSafe AI's Python SDK, `typesafe-sdk`, pinned exactly (owner, 2026-10-07). `app/tools/jev.py` is the only module that imports it.
|
|
- The Orchestrator as a language-model agent is kept behind the setting `ORCHESTRATOR=model`, for the tests that drive the Ledger's rules in arbitrary order and for comparing the two in an evaluation. The default is `router`. Remove the setting once the router is proven.
|
|
|
|
- The Orchestrator's instructions in `agents/orchestrator.md` become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.
|
|
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
|
|
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.
|
|
- An unusual request that fits none of the questions is handled by asking the user, where a language model might have improvised.
|
|
|
|
## As built (2026-10-09)
|
|
|
|
- **Jev is used beyond the Orchestrator.** It also chooses the pillars a change touches and the Skill references a Specialist reads first, checks remote MCP replies and the user's text for instructions aimed at the agent, tells a rewording from a change, and scores each Execution's design recommendations. Each has a fallback in code and is a setting.
|
|
- **Decision 4 covers Orchestrator decisions only.** These other uses do send content: design recommendations, Check and Element text, remote MCP replies and the Workflow request. Jev's data retention is therefore a question about Deliverable content, and it is still unanswered by the vendor.
|
|
- **Jev did not become a hard dependency.** Without a key, or when a call fails, every decision takes its fallback and the Execution continues.
|
|
- **The fourth route, which Deliverables a change reaches, is decided in code** from the changed requirements and the Requirements chain, not by Jev.
|