- An Execution's files are written as each Deliverable is submitted, marked in_progress, so a killed process or one waiting for its user leaves them on disk. A write removes files an earlier write left that it no longer has. - A question about delivered work is answered from that Execution's solution guide by the reply writer; a question about the service is answered as before. Nothing is started for either. - Lessons: prove writes only for lessons the Skill holds; the Skill records the vendored Skill versions the lessons were read against, status names any that changed, and a test fails until they are reviewed. - Four more code patterns: Firestore, BigQuery, Eventarc and GKE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9.8 KiB
Step 12 — Implementation Detail
Built against a scripted decider. No call has been made to the live Jev service, and no evaluation has run on it: the owner holds evaluation runs until the Jev work is in.
app/nodes/router.py.Routerdecides an Orchestrator turn in three stages: readings (_read_ledgerfrom the Ledger and data;_read_messageand_read_gapfrom the decider), a disposition byThresholds(act at 0.75, close at 0.15, choice confidence 0.6), and the playbook in_choose: ordered groups, the strongest action whose conditions hold, escalate when none applies.- Permission stays with the tools. The router calls the same
OrchestratorToolboxtools the model did. A refused action is skipped for the rest of the turn and the next one that holds is taken. Past the Revision limit the user is asked. - What Jev is asked (
Questionobjects, batched when they rest on the same evidence):- of the user's message:
asks_change(noul),change_target(choice over submitted Deliverables),change_clarity(score: vague, partial, specific); - of validation gaps:
gap_material(noul); - of the user's reply about gaps:
wants_revision(noul). What is sent is Deliverable names, the count of waiting questions, the user's message, and for gaps the proposed Check statements and unverifiable Check ids.
- of the user's message:
- Rules when Jev gives no answer. A requested change is acted on only when one Deliverable is named outright, otherwise the user is asked which. A free-text reply is treated as an answer. A gap is revised for when a Check cannot be verified or an addition names a real requirement or recommendation id.
- A new reading from the run of 2026-10-07: a Check that fails in two validations running is sent to the Target architecture, not the code. That run retried one unimplementable Check as code three times.
- Decisions are recorded in the Execution's state under
decisions: turn, group, action and what was judged, with confidence. - The reply writer. A new role,
agents/reply-writer.md, writes the Approval summary from facts (approval_facts) through one tool. When it fails or is absent,assembled_summarybuilds the summary from the same facts. - Wiring.
ORCHESTRATOR=router(the default) ormodel;open_assemblybuilds the router with the decider and the thresholds (JEV_ACT_THRESHOLD,JEV_CLOSE_THRESHOLD,JEV_CONFIDENCE_THRESHOLD). State gainedlast_reply,pending_change,awaiting_gap_decision,validation_settledanddecisions.
Changed from the plan and the ADR's first wording:
- Jev does not propose the next action. It supplies readings, and the playbook in code picks the action. This follows the owner's triage model and keeps every decision explainable by a rule and a threshold.
- The model Orchestrator is kept behind
ORCHESTRATOR=model. The 40 or so engine tests drive the Ledger's rules by scripting its tool calls in arbitrary order, which the router cannot do.agents/orchestrator.mdis therefore unchanged and unused in router mode. - An unclear change is confirmed once. If the reply still does not settle it, a named Deliverable is acted on and anything else is dropped.
- A pre-approved Execution's automatic replies are not routed. There is no user message to read.
Not built from the ADR's routes:
- A message that starts something (new request, Change request, question about delivered work, out of scope). It needs an entry point that can answer from a case's files, and Change requests (iteration 7).
- Which Deliverables a Change request reaches. Iteration 7.
Tests: tests/execution/test_router.py (18), tests/test_assembly.py.
Unverified:
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
Faults found by evaluation case 04, fixed on 2026-10-08
Case 04 is the first interactive run. It stopped at 100 Orchestrator turns (../../iter-06-target-architecture-graph/steps/step-11.md). Three changes:
- Only the user's own words are their message. A reply to questions now reaches the router with the questions it answered (
asked) and with any quoted question andQ:/A:labels removed from its text (_own_wordsinapp/nodes/execution_nodes.py)._read_messagegives the questions to the decider as background and the cleaned text as the message. Before, a describer's "please confirm my draft" question, quoted back by the scripted user, was judged a request to change the Target architecture, twice. - A Revision limit is told to the user once. When a reopen is refused at the limit, the router records the Deliverable in
revision_limit_askedand setsawaiting_limit_decision. The reply is not read as a request for changes, every action that would reopen that Deliverable is barred from then on, and the Execution carries on with it as it stands. Before, the reply was read as another change request for the same Deliverable and the refusal repeated until the turn limit. - A run that only asks is stopped. The engine counts questions put to the user with no change in any Deliverable's status or attempt, or any Phase's status, in between. At the fourth it ends with a warning (
MAX_IDLE_ASKS). This holds for any Orchestrator, whatever the cause.
Also: agents/validator.md now tells the validator to confirm with the user before submitting in an interactive Execution, which the case's specialists_ask_before_submitting check found it did not.
Tests: tests/execution/test_router.py, "the two faults of evaluation case 04" (5).
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change requirements. See the run record in ../../iter-06-target-architecture-graph/steps/step-11.md.
Finished on 2026-10-08: the third case 04 fault, and intake
A reply that restates what is agreed is an answer. _read_message now shows the decider the agreed requirements.json and asks one more noul, already_agreed: does the message only confirm or repeat what those requirements say? At 0.75 or more the reply is closed as an answer, whatever asks_change says. Measured against live Jev with requirements holding the facts being restated:
| Reply | asks_change | already_agreed | Disposition |
|---|---|---|---|
| Restates the GDPR and secrets requirement | 0.12 | 0.78 | close |
| Restates the availability requirement | 0.26 | 0.48 | queue (asked once, then let go) |
| Restates the retention requirement | 0.52 | 0.81 | close |
| "Approved. Please proceed." | 0.04 | 0.70 | close |
| Raises availability to 99.99% | 0.85 | 0.01 | act |
| Changes retention from 2 years to 7 | 0.89 | 0.01 | act |
| Adds SMS notifications and a mobile app | 0.85 | 0.01 | act |
| Changes secret rotation to 30 days | 0.87 | 0.02 | act |
Intake: a message that starts something is read first. app/nodes/intake.py asks one choice of a message that arrives with no Execution in progress: new request, change request, or question. The A2A server (app/a2a_server.py) acts on it:
- a question, at 0.75 confidence or more, is answered with a short statement of what the agent does and starts nothing;
- a change request, when the Customer case has a completed Execution, is started as a new request with a note saying Change requests are not available yet (iteration 7 builds them);
- anything else, any reading that is not confident, and every pre-approved task starts an Execution as before.
Measured against live Jev on ten messages: two new requests, two questions, two change requests with an earlier Execution, a question about earlier work, a different new request for the same case, a change with nothing delivered (taken as new), and "hello" (0.53, taken as new). All ten as intended.
Not built: acting on a change request (iteration 7 step 2), and answering a question about earlier work from that work.
Tests: tests/execution/test_intake.py (4), tests/a2a/test_server.py "intake" (4), tests/execution/test_router.py (1).
A question about delivered work is answered from it (2026-10-10)
Intake answered every question with a description of the service. A question read as about work already delivered in the caller's Customer case is now answered from that Execution's solution guide.
- When intake reads a message as a question and the case has a completed Execution, a second yes or no is asked: is this about an architecture already delivered, or about the service in general. At 0.6 or above it is about delivered work.
DeliveredAnswersinapp/nodes/intake.pyloads the latest completed Execution and gives its solution guide and the question to the reply writer.agents/reply-writer.mdnow covers this second task: answer from the guide alone, say so when it does not say, name the section, propose no change.- The A2A reply carries
about, the Execution it was answered from. Nothing is started. When no answer could be written the caller is told which Execution's documents to read.
Measured against live Jev on twelve messages: six about delivered work scored 0.37 to 0.96, five of them 0.73 or more ("what does the load balancer in my architecture do?" scored 0.37 and would get the general answer). Five about the service scored 0.07 to 0.16. "Is my data stored anywhere?" scored 0.74: it is ambiguous, and would get an answer from the guide. No real model has written such an answer yet.
Tests: tests/execution/test_intake.py (3 more), tests/a2a/test_server.py (3 more).