Files
solution_architecture_agent/docs/adr/0007-jev-is-the-orchestrators-decision-point.md
Jonathan Boniface c84dbdea20 feat: the solution architecture agent on the shared Execution engine
The reference agent's subject, cloud solution architecture, as an agent in
the operating model agent's pattern: the Execution engine (from
orchestration_engine) and one Workflow in app/solution_architecture,
built for the settings and provider bundle by assembly.workflow_for.

Copied from gcp_solution_architecture_agent at 8b397cd without its own
engine: its role files, Skills, provider bundle, ADRs, vocabulary, scripts
and evaluation datasets. The image takes the engine from a named build
context and installs Terraform. It is called through the Orchestration
engine, and its files are what the operating model agent digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:49:28 +01:00

9.9 KiB

status
status
accepted

Jev is the Orchestrator's decision point, and the Orchestrator is a router

ADR-0003 made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev (Jev in AI agent loops): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.

Who owns what

Responsibility Owner in the pattern Here
Interpret the request A model with supplied context The requirements analyst and the Specialists after it
Propose the next action The decision model Jev, asked typed questions about the Execution's state
Check permission Application code The Ledger: Dependencies, Approvals, the Revision limit, the cloud-read gate
Execute the action The tool runner The Orchestrator's existing tools, called by code
Update state Application code The Execution graph and its checkpoint
Explain the result A generative model One small model call that writes an Approval summary or a question, from data it is given

The Orchestrator is a router

With Jev as its decision point, the Orchestrator's job is routing: it reads what has arrived and the state of the Execution, asks Jev which route applies, and dispatches. It does no work of its own. There are four routes to choose.

What is routed The routes Jev chooses between Who is dispatched
A message that starts something a new Workflow request; a Change request against a Baseline; a question about work already delivered; not something this agent does a full Execution; an Execution from the Baseline; the reply writer, from the delivered files; a polite refusal
A message inside an Execution an answer to a waiting question; a request to change submitted work, and which Deliverable; anything else the Specialist that asked; a Revision of that Deliverable; a question back to the user
A validation outcome revise the code; revise the Target architecture; proceed the IaC author; the target architect; the Approval
Ready work, in a Change request which Deliverables the change reaches, and which carry over from the Baseline unchanged only the Specialists whose Deliverables are reached

Which Specialist produces a given Deliverable is still fixed by the dependency graph: Jev chooses the route, never the worker for it. And the Ledger still refuses a route whose Dependencies or Approvals are not met.

How a turn is decided: readings, disposition, playbook

The owner's model for this is a four-stage pipeline: readings of what arrived, a disposition worked out in code, further readings of the state, and a playbook that takes the first group that applies and, within it, the strongest action whose conditions hold, escalating when no group applies.

  1. Readings. Facts about the turn. Most are read from the Ledger and the Execution's data by code: are questions waiting, is a Phase complete, is work ready, which Checks failed, did a Check fail twice running. The ones that need judgement are typed answers from Jev: is the user asking for a change (yes or no), which Deliverable would have to change first (a choice), how specific are they (a score), does a gap in the Target architecture leave a requirement unmet (yes or no). Jev reads; it never picks the action.

  2. Disposition. Code turns a judged reading into act, queue or close by threshold: a probability of 0.75 or more is acted on, 0.15 or less is dropped, and anything between is put to the user. A choice with confidence under 0.6 is not relied on. The thresholds are settings.

  3. Playbook. Ordered groups; the first that applies is taken, and within it the strongest action whose conditions hold:

    Group Actions, strongest first
    The user asked for a change revise the Deliverable it affects; else ask them to confirm which and what
    Questions are waiting relay them
    Validation decided revise the Target architecture (a Check failed twice running, or a gap that matters); else revise the code (a Check failed); else ask about the gaps; else proceed
    A Phase is complete request its Approval
    Work is ready delegate all of it
    Packaging is approved finish
    No group applies escalate: ask the user, or end with warnings when there is none
  4. Permission. Every action goes through the Orchestrator's tools, so the Ledger refuses what is not allowed whatever was decided, and the playbook then takes the next action that is. A typed answer can still be wrong.

  5. Record. Each decision is kept in the Execution's state: the turn, the group, the action and what Jev answered with its confidence.

  6. Reply. A generative model writes only what the user reads, the summary they approve a Phase from, out of data it is given. It runs after the decision and cannot change it. When it fails, the summary is assembled from the same data.

A failed call to Jev is not an answer. The client returns nothing, and the reading falls back to a rule that acts only on what is certain: a change is acted on when the Deliverable is named outright, and a gap is revised for when it names a real requirement or cannot be verified.

Decisions

Accepted by the owner on 2026-10-07, with each of these:

  1. No language model decides what the Orchestrator does. Jev reads, and the playbook in code decides. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
  2. Low confidence goes to the user, not to another model. Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
  3. Jev being unreachable does not stop an Execution. After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
  4. What is sent to Jev is the Execution's state, not its content: Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
  5. The Orchestrator routes; Jev chooses the route. That covers what kind of work a message starts, where a message inside an Execution goes, where a validation outcome goes, and, in a Change request, which Deliverables are reached. Jev never chooses which Specialist produces a Deliverable: the dependency graph fixes that.
  6. In a new Execution every Deliverable is still produced. Routing does not skip a Specialist for a new Workflow request, because each later Deliverable is checked against the ones before it. Skipping happens only in a Change request, where the Baseline supplies what is carried over.

Considered Options

  • A smaller language model as Orchestrator: rejected by the owner. It keeps the slow, open-ended turn and the stalls, at lower cost.
  • No generative model at all, with fixed wording for summaries and questions: this ADR's first draft. Rejected to follow the pattern, which pairs Jev with a generative model for replies: an Approval summary is what a person decides from and reads better written than assembled.
  • Code only, with fixed rules: rejected. It cannot tell what a user's free-text reply is asking for.
  • Code for the mechanical steps and the existing model for judgement: rejected in favour of one decision point. It keeps two ways of deciding and the large model in the loop.

Consequences

  • Jev is reached through TypeSafe AI's Python SDK, typesafe-sdk, pinned exactly (owner, 2026-10-07). app/tools/jev.py is the only module that imports it.

  • The Orchestrator as a language-model agent is kept behind the setting ORCHESTRATOR=model, for the tests that drive the Ledger's rules in arbitrary order and for comparing the two in an evaluation. The default is router. Remove the setting once the router is proven.

  • The Orchestrator's instructions in agents/orchestrator.md become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.

  • Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.

  • Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.

  • An unusual request that fits none of the questions is handled by asking the user, where a language model might have improvised.

As built (2026-10-09)

  • Jev is used beyond the Orchestrator. It also chooses the pillars a change touches and the Skill references a Specialist reads first, checks remote MCP replies and the user's text for instructions aimed at the agent, tells a rewording from a change, and scores each Execution's design recommendations. Each has a fallback in code and is a setting.
  • Decision 4 covers Orchestrator decisions only. These other uses do send content: design recommendations, Check and Element text, remote MCP replies and the Workflow request. Jev's data retention is therefore a question about Deliverable content, and it is still unanswered by the vendor.
  • Jev did not become a hard dependency. Without a key, or when a call fails, every decision takes its fallback and the Execution continues.
  • The fourth route, which Deliverables a change reaches, is decided in code from the changed requirements and the Requirements chain, not by Jev.