This commit is contained in:
@@ -135,6 +135,12 @@ Rules for every step from 3 onward:
|
||||
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
|
||||
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
|
||||
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
|
||||
- **Routing (owner's direction, 2026-10-07):** the Orchestrator acts as a router with Jev choosing the route. Four things are routed:
|
||||
- a message that starts something: new Workflow request, Change request against a Baseline, question about delivered work, or out of scope (Choice);
|
||||
- a message inside an Execution: an answer to a waiting question, a request to change a named Deliverable, or anything else (Choice, then a Choice over the Deliverables);
|
||||
- a validation outcome: revise the code, revise the Target architecture, or proceed (Choice);
|
||||
- in a Change request, which Deliverables the change reaches (a Boolean per Deliverable, with the Requirements chain as the deterministic check).
|
||||
The route decides what is dispatched. Which Specialist produces a Deliverable stays fixed by the dependency graph, and every Deliverable is still produced in a new Execution. The first route needs an entry point that can start a Change request or answer from a case's files (`app/main.py`, `app/a2a_server.py`); the last depends on iteration 7.
|
||||
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
|
||||
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
|
||||
@@ -169,3 +175,33 @@ Rules for every step from 3 onward:
|
||||
- **Fallback:** Treat it as changed.
|
||||
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
|
||||
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
|
||||
|
||||
## Step 16: Jev as the evaluation judge, calibrated
|
||||
|
||||
- **Capability:** The evaluation's judged checks are answered by Jev, and the judge is shown to agree with a person.
|
||||
- **Rationale:** LangChain's evaluation posts report Jev agreeing with a human reviewer on every pass-or-fail decision in a small experiment, with far lower variance than language-model judges, at $0.00035 a call. They also warn that low cost can amplify mistakes and that human review and judge alignment are still needed.
|
||||
- **Built:** `DeciderJudge` in `eval/judge.py` asks every question in one call, with the document as the state and no grading instructions in it. `eval/solution_eval.py --judge` uses it when `TYPESAFE_API_KEY` is set, and the language-model judge otherwise or with `EVAL_JUDGE=gemini`. A judge that gives no answer leaves its check unanswered, not passed.
|
||||
- **Open:** a small set of design recommendations labelled by a person against the pillar checklists and the quality rubric; agreement of each judge with those labels; variance over repeated runs on the same output. This replaces the stability criterion in step 10.
|
||||
- **Acceptance criteria:** On the labelled set, the Jev judge's pass-or-fail agreement and its variance are recorded beside the language-model judge's, and the evaluation uses the one that agrees better.
|
||||
|
||||
## Step 17: Trace every decision
|
||||
|
||||
- **Capability:** For any Execution, a person can see each decision, what it was given and what came back.
|
||||
- **Rationale:** LangChain's production post treats tracing as required for model-driven steps and describes LangSmith views for decision models. Here decisions are kept in the Execution's state and the evaluation report, and nothing is traced.
|
||||
- **Owner's decision needed:** whether to send traces to LangSmith. Traces carry the Workflow request and Deliverable text, so the same retention question as Jev applies. `langsmith` is already installed as a dependency of LangChain; LangGraph traces when its environment variables are set.
|
||||
- **Where:** environment settings for tracing; a traced span around `JevDecider.decide` in `app/tools/jev.py` carrying the questions, answers and probabilities; `docs/operations/`.
|
||||
- **Acceptance criteria:** One Execution's trace shows every Orchestrator turn, each Jev call with its inputs and outputs, and each Specialist run. With tracing off, nothing leaves the process.
|
||||
|
||||
## Step 18: Readings as their own graph step
|
||||
|
||||
- **Capability:** A resumed Execution reuses the decisions it already made.
|
||||
- **Rationale:** LangGraph saves state after each node. The router takes its readings and acts inside one node, so a failure in that node asks Jev again on retry. Jev is designed to return the same answer for the same input, and the client caches answers in memory, but neither survives a restart.
|
||||
- **Where:** `app/workflows/execution_graph.py` and `app/nodes/`: a node that takes the readings and saves them, before the node that applies the playbook.
|
||||
- **Acceptance criteria:** A test that fails the playbook node after the readings node shows the retry using the saved readings, with no second call to the decider.
|
||||
|
||||
## Step 19: Online evaluation of live Executions
|
||||
|
||||
- **Capability:** Every Execution is scored as it runs, not only benchmark cases.
|
||||
- **Rationale:** At Jev's cost, LangChain recommends scoring every run and alerting on risk flags such as leaked personal data or injected instructions.
|
||||
- **Where:** the judged checks from `eval/scoring.py`, run when an Execution ends and stored with its record; with step 17, as feedback on its trace.
|
||||
- **Acceptance criteria:** A completed Execution carries its judged scores. A flagged risk is recorded and surfaced without stopping the Execution.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 15
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 19
|
||||
|
||||
## What Jev is
|
||||
|
||||
@@ -49,6 +49,24 @@ On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use
|
||||
|
||||
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
||||
|
||||
## Alignment with LangChain's published guidance
|
||||
|
||||
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
||||
|
||||
| Their guidance | Here | Step |
|
||||
|---|---|---|
|
||||
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
||||
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
||||
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
||||
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
||||
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
||||
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
||||
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
||||
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
||||
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
||||
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
||||
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
@@ -58,7 +76,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
@@ -67,11 +85,15 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
|
||||
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -90,5 +112,9 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
- Step 15: depends on step 2, iteration 7 step 4
|
||||
- Step 16: depends on step 2
|
||||
- Step 17: no dependencies
|
||||
- Step 18: depends on step 12
|
||||
- Step 19: depends on step 16, step 17
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
|
||||
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 11 — Implementation Detail
|
||||
|
||||
ADR-0007 was accepted by the owner on 2026-10-07 ("this is exactly what I want, I approve ADR-0007"), after three refinements made the same day and recorded in it:
|
||||
|
||||
1. Jev is the decision point, in place of a different language model.
|
||||
2. The agent-control pattern at https://vercel.com/i/jev-agent-control is followed: a decision model reads, application code checks permission and executes, a generative model writes replies.
|
||||
3. The Orchestrator is a router, and decides by readings, a disposition and a playbook, on the model of a triage pipeline the owner supplied: the first group that applies, the strongest action whose conditions hold, escalate when none applies.
|
||||
|
||||
Jev is reached through TypeSafe AI's Python SDK and API endpoint (owner, 2026-10-07). The API key is still to be supplied.
|
||||
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Step 12 — Implementation Detail
|
||||
|
||||
Built against a scripted decider. No call has been made to the live Jev service, and no evaluation has run on it: the owner holds evaluation runs until the Jev work is in.
|
||||
|
||||
1. **`app/nodes/router.py`.** `Router` decides an Orchestrator turn in three stages: readings (`_read_ledger` from the Ledger and data; `_read_message` and `_read_gap` from the decider), a disposition by `Thresholds` (act at 0.75, close at 0.15, choice confidence 0.6), and the playbook in `_choose`: ordered groups, the strongest action whose conditions hold, escalate when none applies.
|
||||
2. **Permission stays with the tools.** The router calls the same `OrchestratorToolbox` tools the model did. A refused action is skipped for the rest of the turn and the next one that holds is taken. Past the Revision limit the user is asked.
|
||||
3. **What Jev is asked** (`Question` objects, batched when they rest on the same evidence):
|
||||
- of the user's message: `asks_change` (noul), `change_target` (choice over submitted Deliverables), `change_clarity` (score: vague, partial, specific);
|
||||
- of validation gaps: `gap_material` (noul);
|
||||
- of the user's reply about gaps: `wants_revision` (noul).
|
||||
What is sent is Deliverable names, the count of waiting questions, the user's message, and for gaps the proposed Check statements and unverifiable Check ids.
|
||||
4. **Rules when Jev gives no answer.** A requested change is acted on only when one Deliverable is named outright, otherwise the user is asked which. A free-text reply is treated as an answer. A gap is revised for when a Check cannot be verified or an addition names a real requirement or recommendation id.
|
||||
5. **A new reading from the run of 2026-10-07:** a Check that fails in two validations running is sent to the Target architecture, not the code. That run retried one unimplementable Check as code three times.
|
||||
6. **Decisions are recorded** in the Execution's state under `decisions`: turn, group, action and what was judged, with confidence.
|
||||
7. **The reply writer.** A new role, `agents/reply-writer.md`, writes the Approval summary from facts (`approval_facts`) through one tool. When it fails or is absent, `assembled_summary` builds the summary from the same facts.
|
||||
8. **Wiring.** `ORCHESTRATOR=router` (the default) or `model`; `open_assembly` builds the router with the decider and the thresholds (`JEV_ACT_THRESHOLD`, `JEV_CLOSE_THRESHOLD`, `JEV_CONFIDENCE_THRESHOLD`). State gained `last_reply`, `pending_change`, `awaiting_gap_decision`, `validation_settled` and `decisions`.
|
||||
|
||||
Changed from the plan and the ADR's first wording:
|
||||
|
||||
- **Jev does not propose the next action.** It supplies readings, and the playbook in code picks the action. This follows the owner's triage model and keeps every decision explainable by a rule and a threshold.
|
||||
- **The model Orchestrator is kept** behind `ORCHESTRATOR=model`. The 40 or so engine tests drive the Ledger's rules by scripting its tool calls in arbitrary order, which the router cannot do. `agents/orchestrator.md` is therefore unchanged and unused in router mode.
|
||||
- **An unclear change is confirmed once.** If the reply still does not settle it, a named Deliverable is acted on and anything else is dropped.
|
||||
- **A pre-approved Execution's automatic replies are not routed.** There is no user message to read.
|
||||
|
||||
Not built from the ADR's routes:
|
||||
|
||||
- **A message that starts something** (new request, Change request, question about delivered work, out of scope). It needs an entry point that can answer from a case's files, and Change requests (iteration 7).
|
||||
- **Which Deliverables a Change request reaches.** Iteration 7.
|
||||
|
||||
Tests: `tests/execution/test_router.py` (18), `tests/test_assembly.py`.
|
||||
|
||||
Unverified:
|
||||
|
||||
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
|
||||
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
|
||||
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
|
||||
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
Built ahead of step 1's answers, on the owner's instruction of 2026-10-07 that no evaluation be rerun until the Jev work is in. Nothing here has called the live service.
|
||||
|
||||
1. **The seam** (`app/adk/runtime.py`): `Question` (kind choice, score or noul, with instructions and options), `Answer` (value, confidence, probabilities) and the `Decider` protocol. `decide` returns the answers, or `None` when no decision could be made, and the caller then takes its fallback. `NoDecider` always returns `None`.
|
||||
2. **The client** (`app/tools/jev.py`): `JevDecider` calls Jev through TypeSafe AI's official Python SDK, `typesafe-sdk` 0.7.2 (`AsyncTypeSafeClient.system_one`). The owner chose the SDK and its API endpoint on 2026-10-07. It is the only module that imports the SDK.
|
||||
3. **Wiring:** `open_decider` in `app/agent.py` returns the client when `TYPESAFE_API_KEY` is set and `NoDecider` otherwise; the `Assembly` carries it.
|
||||
4. **A scripted decider** for tests, `FakeDecider` in `app/adk/fake_runtime.py`.
|
||||
|
||||
Which package: three names exist on the index. `typesafe-sdk` is the vendor's (author TypeSafe AI, repository `typesafe-ai/typesafe-sdk-python`). `typesafe` is an unrelated type-checking library, and `typesafe-ai` is a third party's redirect shim; neither is used.
|
||||
|
||||
What the SDK settled, read from the installed package and a stand-in transport:
|
||||
|
||||
- The request is `POST /v1/systemone` with a bearer key; default endpoint `https://api.typesafe.ai`, default model `jev-latest`.
|
||||
- Questions are `Noul(instructions)`, `Choice(instructions, criteria={option: description})` and `Score(instructions, criteria=[levels])`.
|
||||
- Answers are `noul` (a probability, with no confidence of its own), `choice` with `confidence` and `probabilities`, and `score` with `confidence`. Usage reports `input_tokens` only: no cost and no remaining credit.
|
||||
|
||||
Limits kept, each with a test in `tests/tools/test_jev.py`:
|
||||
|
||||
| Practice | Built as |
|
||||
|---|---|
|
||||
| Size | Refused locally above 48,000 estimated tokens in all, or 24,000 for the state plus the longest question. Never truncated. |
|
||||
| Batching | Every question about one state goes in one request. |
|
||||
| Timeout | 5 seconds a request, 15 for a decision with its retries (the SDK's). |
|
||||
| Retries | 2, by the SDK: on 408, 429, 5xx, connection and timeout errors, with jittered backoff, honouring `Retry-After`. |
|
||||
| Rate | A token bucket, 5 a second with a burst of 10; the rate halves when a rate limit outlasts the retries. |
|
||||
| Concurrency | At most 8 requests in flight. |
|
||||
| Circuit breaker | Opens after 5 failures in 30 seconds for 60 seconds, then lets one probe through. |
|
||||
| Cache | Answers kept by a hash of the state and questions, up to 1,024. |
|
||||
| Budget | 500 calls or 1,000,000 input tokens per scope (the Execution), then fallbacks with one warning. |
|
||||
| Records | Each decision's outcome, seconds, rate limiting, input tokens and confidence; `summary()` totals them. |
|
||||
| Key | From `TYPESAFE_API_KEY` only; never logged. |
|
||||
|
||||
Settings: `TYPESAFE_API_KEY`, `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL` (the SDK's own names), and `JEV_TIMEOUT_SECONDS`, `JEV_REQUESTS_PER_SECOND`, `JEV_MAX_IN_FLIGHT`, `JEV_BUDGET_CALLS`, `JEV_BUDGET_TOKENS`.
|
||||
|
||||
Changed from the plan:
|
||||
|
||||
- The cache is per client, not per Execution. A decision depends only on its state and questions, so two Executions asking the same thing can share the answer.
|
||||
- A reply that leaves any question unanswered, or picks a choice that was not offered, counts as no decision for the whole request.
|
||||
- The budget is in calls and input tokens, not dollars, and there is no low-credit warning: the SDK reports neither cost nor balance.
|
||||
- A yes-or-no answer's confidence is derived as its distance from an even probability, because Jev returns none.
|
||||
- Retries and their backoff are the SDK's, configured here, not reimplemented.
|
||||
|
||||
Unverified: the client has run only against a stand-in transport, with the real SDK in the loop. The first call with a real key is still to be made, and the service's actual rate limits, latency and data retention are still unknown.
|
||||
Reference in New Issue
Block a user