121 lines
8.5 KiB
Markdown
121 lines
8.5 KiB
Markdown
# Plan Map
|
|
|
|
**Iteration:** 5, Jev decisions
|
|
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
|
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
|
**This directory covers:** steps 1 to 19
|
|
|
|
## What Jev is
|
|
|
|
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
|
|
|
|
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
|
|
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
|
|
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
|
|
|
|
Facts from the vendor's documentation, read 2026-10-06:
|
|
|
|
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
|
|
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
|
|
- Price: $0.42 per million input tokens; output free.
|
|
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
|
|
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
|
|
|
|
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
|
|
|
|
## Sources of truth
|
|
|
|
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
|
|
- https://vercel.com/i/jev-agent-control: the agent-control pattern step 12 follows (decision model proposes, code checks and executes, a generative model explains).
|
|
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
|
|
- `iter-04-learning-and-skills/`: several steps here build on steps there.
|
|
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
|
|
|
|
## The capabilities the owner wants from Jev
|
|
|
|
On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use-cases as the capabilities to use. Each maps to this agent as follows.
|
|
|
|
| Jev use case | Question type | What it is here | Step |
|
|
|---|---|---|---|
|
|
| Form routing | Choice | Read the user's message: an answer, a request for changes, or a new statement of Intent, and which Deliverable a change affects | 12 |
|
|
| Tool-call review (`clear` or `caution`) | Choice | Review each cloud-read call and `terraform_plan` before it runs; `caution` pauses for the user | 13 |
|
|
| Response model selection | Choice | Pick the model tier for each delegation from the bundle's list | 6 |
|
|
| Response evaluation | Boolean, Score | Judge a Deliverable at submission (does it address its task; is each Check verifiable as written), and score the evaluation's rubric | 4, 10 |
|
|
| Content moderation | Boolean | Flag text that breaks a written policy: requirements that name products; request or tool text that tries to instruct the agent | 4, 14 |
|
|
| UI copy translation review (did an edit change the meaning?) | Boolean | In a Change request, flag a requirement, Element or Check whose meaning changed against the Baseline, so only real changes are followed through | 15 |
|
|
| Ticket prioritization | Score | Rate how much a proposed addition or a failed Check matters, to order Revisions and lesson candidates | 10 |
|
|
| Document categorization | Choice | Sort lesson candidates by Deliverable for the curator | 10 |
|
|
| CSV column mapping | Choice | Suggest which requirement or recommendation a proposed addition serves, for confirmation; code still checks the link exists | 10 |
|
|
|
|
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
|
|
|
## Alignment with LangChain's published guidance
|
|
|
|
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
|
|
|
| Their guidance | Here | Step |
|
|
|---|---|---|
|
|
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
|
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
|
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
|
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
|
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
|
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
|
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
|
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
|
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
|
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
|
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
|
|
|
## Boundaries
|
|
|
|
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
|
|
|
## Status
|
|
|
|
| Step | What | Report ref | Status |
|
|
|---|---|---|---|
|
|
| 1 | Resolve the open questions | none | open, owner's input |
|
|
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
|
| 3 | Pillars a Revision touches | J3 | planned |
|
|
| 4 | Semantic checks at submission | J4 | planned |
|
|
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
|
| 6 | Model tier per delegation | J5 | planned |
|
|
| 7 | Context selection | J6 | planned |
|
|
| 8 | Skill routing | J9 | planned |
|
|
| 9 | In-run context compaction | J10 | planned |
|
|
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
|
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
|
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
|
| 13 | Cloud-read call guard | J7 | planned |
|
|
| 14 | Input guardrail | J11 | planned |
|
|
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
|
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
|
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
|
| 18 | Readings as their own graph step | none | planned |
|
|
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
|
|
|
## Dependency graph
|
|
|
|
- Step 1: no dependencies
|
|
- Step 2: depends on step 1
|
|
- Step 3: depends on step 2, iteration 4 step 3
|
|
- Step 4: depends on step 2
|
|
- Step 5: depends on step 2, iteration 4 step 1
|
|
- Step 6: depends on step 2
|
|
- Step 7: depends on step 2, iteration 4 step 13
|
|
- Step 8: depends on step 2, iteration 4 step 12
|
|
- Step 9: depends on step 2
|
|
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
|
|
- Step 11: no dependencies
|
|
- Step 12: depends on step 2, step 11. Build it straight after step 2, ahead of steps 3 to 10.
|
|
- Step 13: depends on step 2
|
|
- Step 14: depends on step 2
|
|
- Step 15: depends on step 2, iteration 7 step 4
|
|
- Step 16: depends on step 2
|
|
- Step 17: no dependencies
|
|
- Step 18: depends on step 12
|
|
- Step 19: depends on step 16, step 17
|
|
|
|
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|