Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
121 lines
10 KiB
Markdown
121 lines
10 KiB
Markdown
# Plan Map
|
|
|
|
**Iteration:** 5, Jev decisions
|
|
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
|
**Status:** steps 2 to 12, 14, 15 and 19 are built or settled (2026-10-09); 16 awaits the owner's labels; 13 and 18 are planned; 1 and 17 await the owner
|
|
**This directory covers:** steps 1 to 19
|
|
|
|
## What Jev is
|
|
|
|
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
|
|
|
|
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
|
|
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
|
|
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
|
|
|
|
Facts from the vendor's documentation, read 2026-10-06:
|
|
|
|
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
|
|
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
|
|
- Price: $0.42 per million input tokens; output free.
|
|
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
|
|
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
|
|
|
|
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
|
|
|
|
## Sources of truth
|
|
|
|
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
|
|
- https://vercel.com/i/jev-agent-control: the agent-control pattern step 12 follows (decision model proposes, code checks and executes, a generative model explains).
|
|
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
|
|
- `iter-04-learning-and-skills/`: several steps here build on steps there.
|
|
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
|
|
|
|
## The capabilities the owner wants from Jev
|
|
|
|
On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use-cases as the capabilities to use. Each maps to this agent as follows.
|
|
|
|
| Jev use case | Question type | What it is here | Step |
|
|
|---|---|---|---|
|
|
| Form routing | Choice | Read the user's message: an answer, a request for changes, or a new statement of Intent, and which Deliverable a change affects | 12 |
|
|
| Tool-call review (`clear` or `caution`) | Choice | Review each cloud-read call and `terraform_plan` before it runs; `caution` pauses for the user | 13 |
|
|
| Response model selection | Choice | Pick the model tier for each delegation from the bundle's list | 6 |
|
|
| Response evaluation | Boolean, Score | Judge a Deliverable at submission (does it address its task; is each Check verifiable as written), and score the evaluation's rubric | 4, 10 |
|
|
| Content moderation | Boolean | Flag text that breaks a written policy: requirements that name products; request or tool text that tries to instruct the agent | 4, 14 |
|
|
| UI copy translation review (did an edit change the meaning?) | Boolean | In a Change request, flag a requirement, Element or Check whose meaning changed against the Baseline, so only real changes are followed through | 15 |
|
|
| Ticket prioritization | Score | Rate how much a proposed addition or a failed Check matters, to order Revisions and lesson candidates | 10 |
|
|
| Document categorization | Choice | Sort lesson candidates by Deliverable for the curator | 10 |
|
|
| CSV column mapping | Choice | Suggest which requirement or recommendation a proposed addition serves, for confirmation; code still checks the link exists | 10 |
|
|
|
|
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
|
|
|
## Alignment with LangChain's published guidance
|
|
|
|
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
|
|
|
| Their guidance | Here | Step |
|
|
|---|---|---|
|
|
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
|
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
|
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
|
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
|
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
|
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
|
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
|
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
|
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
|
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
|
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
|
|
|
## Boundaries
|
|
|
|
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
|
|
|
## Status
|
|
|
|
| Step | What | Report ref | Status |
|
|
|---|---|---|---|
|
|
| 1 | Resolve the open questions | none | the vendor's answers on retention, training, context and rate limits recorded on 2026-10-10, with the owner's decisions; retention for client work is held (`steps/step-1.md`) |
|
|
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
|
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
|
|
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
|
|
| 5 | Finding-to-Deliverable routing | J2 | superseded: iteration 6 gives a verdict per Check, and the router sends a failed Check to the code and a gap to the Target architecture |
|
|
| 6 | Model tier per delegation | J5 | built and measured: Flash for the IaC author's first attempt saved five cents and took twice the requests, so it stays off (`steps/step-6.md`) |
|
|
| 7 | Context selection | J6 | decided against as planned: earlier Deliverables are a tenth of the describer's prompt (`steps/step-7.md`) |
|
|
| 8 | Skill routing | J9 | built and on: the references a task needs are named in the Specialist's context; calibrated live; ran in cases 05 and 06 on 2026-10-09 (`steps/step-8.md`) |
|
|
| 9 | In-run context compaction | J10 | decided against as planned: it would lose the prompt cache and cost more (`steps/step-9.md`) |
|
|
| 10 | Evaluation rubric and lesson triage | J8 | built: the judged rubric (step 16) and triage of evidence a lesson already covers (iteration 4 step 10) |
|
|
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
|
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
|
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
|
|
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
|
|
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; ran in case 05, where nothing was reworded |
|
|
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; the owner labelled the set on 2026-10-10, which showed the question was wrong, not the judges; the set is to be rebuilt and relabelled after iteration 4 step 29 (`steps/step-16.md`) |
|
|
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
|
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
|
|
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |
|
|
|
|
## Dependency graph
|
|
|
|
- Step 1: no dependencies
|
|
- Step 2: depends on step 1
|
|
- Step 3: depends on step 2, iteration 4 step 3
|
|
- Step 4: depends on step 2
|
|
- Step 5: depends on step 2, iteration 4 step 1
|
|
- Step 6: depends on step 2
|
|
- Step 7: depends on step 2, iteration 4 step 13
|
|
- Step 8: depends on step 2, iteration 4 step 12
|
|
- Step 9: depends on step 2
|
|
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
|
|
- Step 11: no dependencies
|
|
- Step 12: depends on step 2, step 11. Build it straight after step 2, ahead of steps 3 to 10.
|
|
- Step 13: depends on step 2
|
|
- Step 14: depends on step 2
|
|
- Step 15: depends on step 2, iteration 7 step 4
|
|
- Step 16: depends on step 2
|
|
- Step 17: no dependencies
|
|
- Step 18: depends on step 12
|
|
- Step 19: depends on step 16, step 17
|
|
|
|
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|