feat: establish the Target architecture as a graph before code

Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 13:59:09 +01:00
parent 7ee49ab352
commit e9246828b3
586 changed files with 5166 additions and 22432 deletions

View File

@@ -114,21 +114,31 @@ Rules for every step from 3 onward:
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
## Step 11: ADR: Orchestrator routing
## Step 11: ADR: Jev is the Orchestrator's decision point
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
- **Capability:** A recorded decision that Jev proposes the Orchestrator's next action, code checks and carries it out, and a generative model only writes replies.
- **Rationale:** The owner's direction on 2026-10-07: use Jev as the decision point, not a different model. It replaces part of ADR-0003, which made the Orchestrator an ADK agent and rejected an Orchestrator without a model.
- **Where:** `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, with four points to confirm; a note on ADR-0003.
- **Acceptance criteria:** ADR-0007 is `accepted`, or amended with step 12 updated to match.
## Step 12: Orchestrator routing
## Step 12: The Orchestrator decides with Jev, with no language model
- **Report ref:** J1
- **Capability:** The Orchestrator model runs only when judgement is needed.
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
- **Fallback:** the Orchestrator model, as today.
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
- **Report ref:** J1, widened by ADR-0007
- **Capability:** Each Orchestrator turn is decided by the Ledger's rules and, where there is a choice or the user has spoken, by Jev.
- **Rationale:** In the stopped run of 2026-10-06 the Orchestrator took about 7 of 35 minutes and stalled four times. In case 03 it made 36 requests on the larger model.
- **Design:**
1. A pure function lists the allowed next actions from the Ledger and the Execution's state.
2. One allowed action is taken without a decision.
3. Otherwise Jev is asked closed questions through the `Decider` from step 2: whether the user asked for a change and to which Deliverable; what follows a validation outcome; which allowed action is next.
4. Code carries the decision out through `OrchestratorToolbox`, so the Ledger's rules are unchanged.
5. Each decision is recorded in the Execution's state with its questions, answers and probabilities, and reused when a resumed Execution has unchanged input.
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
- **Acceptance criteria:** The engine tests pass with a scripted `Decider` in place of the scripted Orchestrator model, covering every Ledger rule they cover today. No language model call chooses an action; the only Orchestrator model calls write user-facing text. A decision Jev proposes that the Ledger does not allow is refused and recorded. With no Jev key configured, an Execution still completes on the fixed order. On the evaluation cases, quality checks hold and Orchestrator seconds fall against the baseline.
## Step 13: Cloud-read call guard
@@ -149,3 +159,13 @@ Rules for every step from 3 onward:
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
- **Depends on step 1:** MCP results hold discovered cloud state.
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.
## Step 15: Changes of meaning in a Change request
- **Report ref:** none; from the "UI copy translation review" use case the owner named on 2026-10-07
- **Capability:** A Change request follows through only the things whose meaning changed.
- **Rationale:** A Specialist revising from a Baseline rewords text it was not asked to change. Comparing text exactly would treat every rewording as a change; ignoring text would miss a dropped restriction, such as "only" removed from a requirement.
- **Design:** For each requirement, Element and Check whose text differs from the Baseline's, one Boolean question with the old and new text and its id: "Does the new text change what must be true?" A yes marks it changed, so the Requirements chain is followed from it; a no carries the Baseline's verdict forward.
- **Fallback:** Treat it as changed.
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.