This commit is contained in:
@@ -2,8 +2,8 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 15
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 19
|
||||
|
||||
## What Jev is
|
||||
|
||||
@@ -49,6 +49,24 @@ On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use
|
||||
|
||||
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
||||
|
||||
## Alignment with LangChain's published guidance
|
||||
|
||||
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
||||
|
||||
| Their guidance | Here | Step |
|
||||
|---|---|---|
|
||||
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
||||
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
||||
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
||||
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
||||
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
||||
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
||||
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
||||
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
||||
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
||||
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
||||
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
@@ -58,7 +76,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
@@ -67,11 +85,15 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
|
||||
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -90,5 +112,9 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
- Step 15: depends on step 2, iteration 7 step 4
|
||||
- Step 16: depends on step 2
|
||||
- Step 17: no dependencies
|
||||
- Step 18: depends on step 12
|
||||
- Step 19: depends on step 16, step 17
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
|
||||
Reference in New Issue
Block a user