Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/MAP.md
Jonathan Boniface a48468d477
Some checks failed
validation / verify (push) Failing after 11s
feat: what a request does not say is a question for the customer
A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.

Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:30:12 +01:00

10 KiB

Plan Map

Iteration: 5, Jev decisions Intent: Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching. Status: steps 2 to 12, 14, 15 and 19 are built or settled (2026-10-09); 16 awaits the owner's labels; 13 and 18 are planned; 1 and 17 await the owner This directory covers: steps 1 to 19

What Jev is

Jev is TypeSafe AI's "System One" model. It takes a state (text) and a map of typed questions, and returns calibrated probabilities instead of prose:

  • Choice: pick one option. Returns choice, confidence, probabilities.
  • Score: rate against ordered levels. Returns score, probabilities, legend, confidence.
  • Noul: yes or no. Returns noul (0 to 1) and confidence.

Facts from the vendor's documentation, read 2026-10-06:

  • POST /api/v1/decide with Authorization: Bearer <key>; body model (optional), state, questions; response answers plus usage (input_tokens, cost_usd, credits_remaining_usd).
  • Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
  • Price: $0.42 per million input tokens; output free.
  • Undocumented: rate limits, latency, batching, data retention, self-hosting.
  • The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.

Jev will be deployed separately and reached with an API key (owner, 2026-10-06).

Sources of truth

The capabilities the owner wants from Jev

On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use-cases as the capabilities to use. Each maps to this agent as follows.

Jev use case Question type What it is here Step
Form routing Choice Read the user's message: an answer, a request for changes, or a new statement of Intent, and which Deliverable a change affects 12
Tool-call review (clear or caution) Choice Review each cloud-read call and terraform_plan before it runs; caution pauses for the user 13
Response model selection Choice Pick the model tier for each delegation from the bundle's list 6
Response evaluation Boolean, Score Judge a Deliverable at submission (does it address its task; is each Check verifiable as written), and score the evaluation's rubric 4, 10
Content moderation Boolean Flag text that breaks a written policy: requirements that name products; request or tool text that tries to instruct the agent 4, 14
UI copy translation review (did an edit change the meaning?) Boolean In a Change request, flag a requirement, Element or Check whose meaning changed against the Baseline, so only real changes are followed through 15
Ticket prioritization Score Rate how much a proposed addition or a failed Check matters, to order Revisions and lesson candidates 10
Document categorization Choice Sort lesson candidates by Deliverable for the curator 10
CSV column mapping Choice Suggest which requirement or recommendation a proposed addition serves, for confirmation; code still checks the link exists 10

The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.

Alignment with LangChain's published guidance

Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.

Their guidance Here Step
Code owns the workflow; Jev makes narrow, typed decisions The Ledger and the playbook are code; Jev supplies readings 12
Encode domain knowledge in the graph's topology, not in prompts The dependency graph, Phases and playbook are data and code 12
Several atomic questions against one state, in one call Readings are batched per turn 2, 12
Language models for open-ended work Specialists write Deliverables; a model writes replies 12
Humans in the loop through interrupts Approvals and every user question are interrupts done
Checkpointing, so a failed run resumes with the decisions it made The Execution is checkpointed; a decision is saved only when its turn ends 18
Low confidence falls back to a language model (their example: under 0.7) Low confidence goes to the user, by the owner's choice in ADR-0007 differs on purpose
Trace every decision, with its inputs and calibrated outputs Decisions are kept in Execution state and the evaluation report; nothing is traced 17
Jev as the judge: typed questions, no grading instructions in the state DeciderJudge in eval/judge.py, used when a key is set 16
Check a judge against human labels, and for repeatability Not done; there is no labelled set 16
Score every live run and alert on risk flags Evaluation is offline only 19

Boundaries

Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.

Status

Step What Report ref Status
1 Resolve the open questions none the vendor's answers on retention, training, context and rate limits recorded on 2026-10-10, with the owner's decisions; retention for client work is held (steps/step-1.md)
2 Decider interface and Jev client with limits none built on the TypeSafe SDK; unverified against the live service
3 Pillars a Revision touches J3 built, as a graded question; calibrated live; ran in cases 05 and 06 on 2026-10-09
4 Semantic checks at submission J4 two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (steps/step-4.md)
5 Finding-to-Deliverable routing J2 superseded: iteration 6 gives a verdict per Check, and the router sends a failed Check to the code and a gap to the Target architecture
6 Model tier per delegation J5 built and measured: Flash for the IaC author's first attempt saved five cents and took twice the requests, so it stays off (steps/step-6.md)
7 Context selection J6 decided against as planned: earlier Deliverables are a tenth of the describer's prompt (steps/step-7.md)
8 Skill routing J9 built and on: the references a task needs are named in the Specialist's context; calibrated live; ran in cases 05 and 06 on 2026-10-09 (steps/step-8.md)
9 In-run context compaction J10 decided against as planned: it would lose the prompt cache and cost more (steps/step-9.md)
10 Evaluation rubric and lesson triage J8 built: the judged rubric (step 16) and triage of evidence a lesson already covers (iteration 4 step 10)
11 ADR: Jev is the Orchestrator's decision point none complete: ADR-0007 accepted 2026-10-07
12 The Orchestrator routes by readings and a playbook; a model only writes replies J1 built, including intake of a message that starts something (steps/step-12.md); the three faults case 04 found are fixed; a completed interactive run is outstanding
13 Cloud-read call guard J7 planned; cloud reads are off in every case, so it could not be exercised
14 Input guardrail J11 built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; ran in cases 05 and 06 on 2026-10-09
15 Changes of meaning in a Change request none built: reworded items are listed apart in changes.json; calibrated live; ran in case 05, where nothing was reworded
16 Jev as the evaluation judge, calibrated none judge and calibration built; the owner labelled the set on 2026-10-10, which showed the question was wrong, not the judges; the set is to be rebuilt and relabelled after iteration 4 step 29 (steps/step-16.md)
17 Trace every decision none open, owner's decision on LangSmith
18 Readings as their own graph step none planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash
19 Online evaluation of live Executions none built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (steps/step-19.md)

Dependency graph

  • Step 1: no dependencies
  • Step 2: depends on step 1
  • Step 3: depends on step 2, iteration 4 step 3
  • Step 4: depends on step 2
  • Step 5: depends on step 2, iteration 4 step 1
  • Step 6: depends on step 2
  • Step 7: depends on step 2, iteration 4 step 13
  • Step 8: depends on step 2, iteration 4 step 12
  • Step 9: depends on step 2
  • Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
  • Step 11: no dependencies
  • Step 12: depends on step 2, step 11. Build it straight after step 2, ahead of steps 3 to 10.
  • Step 13: depends on step 2
  • Step 14: depends on step 2
  • Step 15: depends on step 2, iteration 7 step 4
  • Step 16: depends on step 2
  • Step 17: no dependencies
  • Step 18: depends on step 12
  • Step 19: depends on step 16, step 17

Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.