Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/MAP.md
Jonathan Boniface 567aede5ca feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:45:56 +01:00

121 lines
9.9 KiB
Markdown

# Plan Map
**Iteration:** 5, Jev decisions
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
**Status:** steps 2 to 5, 8, 10 to 12, 14, 15 and 19 are built or settled (2026-10-09); 16 awaits the owner's labels; 6, 7, 9, 13 and 18 are planned; 1 and 17 await the owner
**This directory covers:** steps 1 to 19
## What Jev is
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
Facts from the vendor's documentation, read 2026-10-06:
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
- Price: $0.42 per million input tokens; output free.
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
## Sources of truth
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
- https://vercel.com/i/jev-agent-control: the agent-control pattern step 12 follows (decision model proposes, code checks and executes, a generative model explains).
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
- `iter-04-learning-and-skills/`: several steps here build on steps there.
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
## The capabilities the owner wants from Jev
On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use-cases as the capabilities to use. Each maps to this agent as follows.
| Jev use case | Question type | What it is here | Step |
|---|---|---|---|
| Form routing | Choice | Read the user's message: an answer, a request for changes, or a new statement of Intent, and which Deliverable a change affects | 12 |
| Tool-call review (`clear` or `caution`) | Choice | Review each cloud-read call and `terraform_plan` before it runs; `caution` pauses for the user | 13 |
| Response model selection | Choice | Pick the model tier for each delegation from the bundle's list | 6 |
| Response evaluation | Boolean, Score | Judge a Deliverable at submission (does it address its task; is each Check verifiable as written), and score the evaluation's rubric | 4, 10 |
| Content moderation | Boolean | Flag text that breaks a written policy: requirements that name products; request or tool text that tries to instruct the agent | 4, 14 |
| UI copy translation review (did an edit change the meaning?) | Boolean | In a Change request, flag a requirement, Element or Check whose meaning changed against the Baseline, so only real changes are followed through | 15 |
| Ticket prioritization | Score | Rate how much a proposed addition or a failed Check matters, to order Revisions and lesson candidates | 10 |
| Document categorization | Choice | Sort lesson candidates by Deliverable for the curator | 10 |
| CSV column mapping | Choice | Suggest which requirement or recommendation a proposed addition serves, for confirmation; code still checks the link exists | 10 |
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
## Alignment with LangChain's published guidance
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
| Their guidance | Here | Step |
|---|---|---|
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
## Boundaries
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
## Status
| Step | What | Report ref | Status |
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; not run in an Execution |
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
| 5 | Finding-to-Deliverable routing | J2 | superseded: iteration 6 gives a verdict per Check, and the router sends a failed Check to the code and a gap to the Target architecture |
| 6 | Model tier per delegation | J5 | planned; held until the cases have run on this tranche, so its effect on quality can be told apart |
| 7 | Context selection | J6 | planned; held for the same reason as step 6 |
| 8 | Skill routing | J9 | built and on: the references a task needs are named in the Specialist's context; calibrated live; not run in an Execution (`steps/step-8.md`) |
| 9 | In-run context compaction | J10 | planned; held for the same reason as step 6 |
| 10 | Evaluation rubric and lesson triage | J8 | built: the judged rubric (step 16) and triage of evidence a lesson already covers (iteration 4 step 10) |
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; not run in an Execution |
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; not run in a Change request |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |
## Dependency graph
- Step 1: no dependencies
- Step 2: depends on step 1
- Step 3: depends on step 2, iteration 4 step 3
- Step 4: depends on step 2
- Step 5: depends on step 2, iteration 4 step 1
- Step 6: depends on step 2
- Step 7: depends on step 2, iteration 4 step 13
- Step 8: depends on step 2, iteration 4 step 12
- Step 9: depends on step 2
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
- Step 11: no dependencies
- Step 12: depends on step 2, step 11. Build it straight after step 2, ahead of steps 3 to 10.
- Step 13: depends on step 2
- Step 14: depends on step 2
- Step 15: depends on step 2, iteration 7 step 4
- Step 16: depends on step 2
- Step 17: no dependencies
- Step 18: depends on step 12
- Step 19: depends on step 16, step 17
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.