feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005). A new target-architect Specialist produces one graph of Elements, Relationships and Checks; the diagram is rendered from it, deployment guidance must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations carry ids in data files, and a submission with a gap in the Requirements chain is refused. Also in this commit: - Customer cases group Executions, with an intent history per case - lesson candidates are captured when an Execution ends - evaluation gains chain, verdict, judged and interactive checks, and per-case thresholds - Terraform providers are cached between checks - the requirements analyst proposes defaults instead of asking - the legacy-baseline case and the checks that only read it are removed - ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed - plans for iterations 4 to 7 under .agents/plans/ Unproven with a real model: no evaluation run has completed on this design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -114,21 +114,31 @@ Rules for every step from 3 onward:
|
||||
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
|
||||
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
|
||||
|
||||
## Step 11: ADR: Orchestrator routing
|
||||
## Step 11: ADR: Jev is the Orchestrator's decision point
|
||||
|
||||
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
|
||||
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
|
||||
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
|
||||
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
|
||||
- **Capability:** A recorded decision that Jev proposes the Orchestrator's next action, code checks and carries it out, and a generative model only writes replies.
|
||||
- **Rationale:** The owner's direction on 2026-10-07: use Jev as the decision point, not a different model. It replaces part of ADR-0003, which made the Orchestrator an ADK agent and rejected an Orchestrator without a model.
|
||||
- **Where:** `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, with four points to confirm; a note on ADR-0003.
|
||||
- **Acceptance criteria:** ADR-0007 is `accepted`, or amended with step 12 updated to match.
|
||||
|
||||
## Step 12: Orchestrator routing
|
||||
## Step 12: The Orchestrator decides with Jev, with no language model
|
||||
|
||||
- **Report ref:** J1
|
||||
- **Capability:** The Orchestrator model runs only when judgement is needed.
|
||||
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
|
||||
- **Fallback:** the Orchestrator model, as today.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
|
||||
- **Report ref:** J1, widened by ADR-0007
|
||||
- **Capability:** Each Orchestrator turn is decided by the Ledger's rules and, where there is a choice or the user has spoken, by Jev.
|
||||
- **Rationale:** In the stopped run of 2026-10-06 the Orchestrator took about 7 of 35 minutes and stalled four times. In case 03 it made 36 requests on the larger model.
|
||||
- **Design:**
|
||||
1. A pure function lists the allowed next actions from the Ledger and the Execution's state.
|
||||
2. One allowed action is taken without a decision.
|
||||
3. Otherwise Jev is asked closed questions through the `Decider` from step 2: whether the user asked for a change and to which Deliverable; what follows a validation outcome; which allowed action is next.
|
||||
4. Code carries the decision out through `OrchestratorToolbox`, so the Ledger's rules are unchanged.
|
||||
5. Each decision is recorded in the Execution's state with its questions, answers and probabilities, and reused when a resumed Execution has unchanged input.
|
||||
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
|
||||
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
|
||||
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
|
||||
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
|
||||
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
|
||||
- **Acceptance criteria:** The engine tests pass with a scripted `Decider` in place of the scripted Orchestrator model, covering every Ledger rule they cover today. No language model call chooses an action; the only Orchestrator model calls write user-facing text. A decision Jev proposes that the Ledger does not allow is refused and recorded. With no Jev key configured, an Execution still completes on the fixed order. On the evaluation cases, quality checks hold and Orchestrator seconds fall against the baseline.
|
||||
|
||||
## Step 13: Cloud-read call guard
|
||||
|
||||
@@ -149,3 +159,13 @@ Rules for every step from 3 onward:
|
||||
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
|
||||
- **Depends on step 1:** MCP results hold discovered cloud state.
|
||||
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.
|
||||
|
||||
## Step 15: Changes of meaning in a Change request
|
||||
|
||||
- **Report ref:** none; from the "UI copy translation review" use case the owner named on 2026-10-07
|
||||
- **Capability:** A Change request follows through only the things whose meaning changed.
|
||||
- **Rationale:** A Specialist revising from a Baseline rewords text it was not asked to change. Comparing text exactly would treat every rewording as a change; ignoring text would miss a dropped restriction, such as "only" removed from a requirement.
|
||||
- **Design:** For each requirement, Element and Check whose text differs from the Baseline's, one Boolean question with the old and new text and its id: "Does the new text change what must be true?" A yes marks it changed, so the Requirements chain is followed from it; a no carries the Baseline's verdict forward.
|
||||
- **Fallback:** Treat it as changed.
|
||||
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
|
||||
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1
|
||||
**This directory covers:** steps 1 to 14
|
||||
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 15
|
||||
|
||||
## What Jev is
|
||||
|
||||
@@ -26,10 +26,29 @@ Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
|
||||
## Sources of truth
|
||||
|
||||
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
|
||||
- https://vercel.com/i/jev-agent-control: the agent-control pattern step 12 follows (decision model proposes, code checks and executes, a generative model explains).
|
||||
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
|
||||
- `iter-04-learning-and-skills/`: several steps here build on steps there.
|
||||
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
|
||||
|
||||
## The capabilities the owner wants from Jev
|
||||
|
||||
On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use-cases as the capabilities to use. Each maps to this agent as follows.
|
||||
|
||||
| Jev use case | Question type | What it is here | Step |
|
||||
|---|---|---|---|
|
||||
| Form routing | Choice | Read the user's message: an answer, a request for changes, or a new statement of Intent, and which Deliverable a change affects | 12 |
|
||||
| Tool-call review (`clear` or `caution`) | Choice | Review each cloud-read call and `terraform_plan` before it runs; `caution` pauses for the user | 13 |
|
||||
| Response model selection | Choice | Pick the model tier for each delegation from the bundle's list | 6 |
|
||||
| Response evaluation | Boolean, Score | Judge a Deliverable at submission (does it address its task; is each Check verifiable as written), and score the evaluation's rubric | 4, 10 |
|
||||
| Content moderation | Boolean | Flag text that breaks a written policy: requirements that name products; request or tool text that tries to instruct the agent | 4, 14 |
|
||||
| UI copy translation review (did an edit change the meaning?) | Boolean | In a Change request, flag a requirement, Element or Check whose meaning changed against the Baseline, so only real changes are followed through | 15 |
|
||||
| Ticket prioritization | Score | Rate how much a proposed addition or a failed Check matters, to order Revisions and lesson candidates | 10 |
|
||||
| Document categorization | Choice | Sort lesson candidates by Deliverable for the curator | 10 |
|
||||
| CSV column mapping | Choice | Suggest which requirement or recommendation a proposed addition serves, for confirmation; code still checks the link exists | 10 |
|
||||
|
||||
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
@@ -48,10 +67,11 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Orchestrator routing | none | planned, owner's decision |
|
||||
| 12 | Orchestrator routing | J1 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
|
||||
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -65,9 +85,10 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
- Step 8: depends on step 2, iteration 4 step 12
|
||||
- Step 9: depends on step 2
|
||||
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
|
||||
- Step 11: depends on step 4 having evaluation results
|
||||
- Step 12: depends on step 2, step 11
|
||||
- Step 11: no dependencies
|
||||
- Step 12: depends on step 2, step 11. Build it straight after step 2, ahead of steps 3 to 10.
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
- Step 15: depends on step 2, iteration 7 step 4
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
|
||||
Reference in New Issue
Block a user