208 lines
20 KiB
Markdown
208 lines
20 KiB
Markdown
# Plan Detail
|
|
|
|
Rules for every step from 3 onward:
|
|
|
|
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
|
|
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
|
|
- **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14).
|
|
- With no API key configured, the agent behaves exactly as it does without this iteration.
|
|
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
|
|
|
|
## Step 1: Resolve the open questions
|
|
|
|
- **Capability:** The facts needed to send client data to Jev.
|
|
- **Questions for the owner or vendor:**
|
|
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
|
|
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
|
|
- Rate limits and typical latency.
|
|
- **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
|
|
|
|
## Step 2: Decider interface and Jev client with limits
|
|
|
|
- **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake.
|
|
- **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly.
|
|
- **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`.
|
|
- **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
|
|
|
|
| Practice | Default |
|
|
|---|---|
|
|
| Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question |
|
|
| Send every question about one state in one call | one call per state |
|
|
| Timeout | 2 seconds connect, 5 seconds total |
|
|
| Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap |
|
|
| Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 |
|
|
| Semaphore on calls in flight | 8 |
|
|
| Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down |
|
|
| Cache answers by a hash of model, state and questions | in memory, per Execution |
|
|
| Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 |
|
|
| Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 |
|
|
| Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always |
|
|
| API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` |
|
|
|
|
- **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback".
|
|
|
|
## Step 3: Pillars a Revision touches
|
|
|
|
- **Report ref:** J3
|
|
- **Capability:** The feedback text selects which pillar advisors re-run.
|
|
- **Design:** One call with a Noul per pillar: "Does this feedback require changes to the <pillar> recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`.
|
|
- **Fallback:** all pillars.
|
|
- **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`.
|
|
- **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
|
|
|
|
## Step 4: Semantic checks at submission
|
|
|
|
- **Report ref:** J4, and Jev's "task-completion check" use case
|
|
- **Capability:** Submission contracts that check meaning, not only headings.
|
|
- **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
|
|
- **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
|
|
- **Fallback:** accept on the structural contract alone.
|
|
- **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle.
|
|
- **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
|
|
|
|
## Step 5: Finding-to-Deliverable routing
|
|
|
|
- **Report ref:** J2
|
|
- **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity.
|
|
- **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`.
|
|
- **Fallback:** the validator's own stated Deliverable.
|
|
- **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`.
|
|
- **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
|
|
|
|
## Step 6: Model tier per delegation
|
|
|
|
- **Report ref:** J5, and Jev's "model routing" use case
|
|
- **Capability:** The cheaper model tier where the task allows.
|
|
- **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
|
|
- **Fallback:** the role's configured model.
|
|
- **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`.
|
|
- **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
|
|
|
|
## Step 7: Context selection
|
|
|
|
- **Report ref:** J6, and Jev's "repo context selection" use case
|
|
- **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full.
|
|
- **Design:** A Noul per section: "Is this section needed to produce the <kind> Deliverable?" Sections below threshold are passed as one-line summaries.
|
|
- **Fallback:** the rule from iteration 4 step 13.
|
|
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`.
|
|
- **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged.
|
|
|
|
## Step 8: Skill routing
|
|
|
|
- **Report ref:** J9, and Jev's "agent skill routing" use case
|
|
- **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
|
|
- **Design:** A Choice over the role's Skills and their references, with a `none` option.
|
|
- **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today.
|
|
- **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`.
|
|
- **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged.
|
|
|
|
## Step 9: In-run context compaction
|
|
|
|
- **Report ref:** J10, and Jev's "context compaction" use case
|
|
- **Capability:** Old tool results inside one agent run are kept, truncated or dropped.
|
|
- **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each.
|
|
- **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result.
|
|
- **Fallback:** keep everything.
|
|
- **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook.
|
|
- **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
|
|
|
|
## Step 10: Evaluation rubric and lesson triage
|
|
|
|
- **Report ref:** J8
|
|
- **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates.
|
|
- **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
|
|
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
|
|
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
|
|
|
|
## Step 11: ADR: Jev is the Orchestrator's decision point
|
|
|
|
- **Capability:** A recorded decision that Jev proposes the Orchestrator's next action, code checks and carries it out, and a generative model only writes replies.
|
|
- **Rationale:** The owner's direction on 2026-10-07: use Jev as the decision point, not a different model. It replaces part of ADR-0003, which made the Orchestrator an ADK agent and rejected an Orchestrator without a model.
|
|
- **Where:** `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, with four points to confirm; a note on ADR-0003.
|
|
- **Acceptance criteria:** ADR-0007 is `accepted`, or amended with step 12 updated to match.
|
|
|
|
## Step 12: The Orchestrator decides with Jev, with no language model
|
|
|
|
- **Report ref:** J1, widened by ADR-0007
|
|
- **Capability:** Each Orchestrator turn is decided by the Ledger's rules and, where there is a choice or the user has spoken, by Jev.
|
|
- **Rationale:** In the stopped run of 2026-10-06 the Orchestrator took about 7 of 35 minutes and stalled four times. In case 03 it made 36 requests on the larger model.
|
|
- **Design:**
|
|
1. A pure function lists the allowed next actions from the Ledger and the Execution's state.
|
|
2. One allowed action is taken without a decision.
|
|
3. Otherwise Jev is asked closed questions through the `Decider` from step 2: whether the user asked for a change and to which Deliverable; what follows a validation outcome; which allowed action is next.
|
|
4. Code carries the decision out through `OrchestratorToolbox`, so the Ledger's rules are unchanged.
|
|
5. Each decision is recorded in the Execution's state with its questions, answers and probabilities, and reused when a resumed Execution has unchanged input.
|
|
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
|
|
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
|
|
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
|
|
- **Routing (owner's direction, 2026-10-07):** the Orchestrator acts as a router with Jev choosing the route. Four things are routed:
|
|
- a message that starts something: new Workflow request, Change request against a Baseline, question about delivered work, or out of scope (Choice);
|
|
- a message inside an Execution: an answer to a waiting question, a request to change a named Deliverable, or anything else (Choice, then a Choice over the Deliverables);
|
|
- a validation outcome: revise the code, revise the Target architecture, or proceed (Choice);
|
|
- in a Change request, which Deliverables the change reaches (a Boolean per Deliverable, with the Requirements chain as the deterministic check).
|
|
The route decides what is dispatched. Which Specialist produces a Deliverable stays fixed by the dependency graph, and every Deliverable is still produced in a new Execution. The first route needs an entry point that can start a Change request or answer from a case's files (`app/main.py`, `app/a2a_server.py`); the last depends on iteration 7.
|
|
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
|
|
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
|
|
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
|
|
- **Acceptance criteria:** The engine tests pass with a scripted `Decider` in place of the scripted Orchestrator model, covering every Ledger rule they cover today. No language model call chooses an action; the only Orchestrator model calls write user-facing text. A decision Jev proposes that the Ledger does not allow is refused and recorded. With no Jev key configured, an Execution still completes on the fixed order. On the evaluation cases, quality checks hold and Orchestrator seconds fall against the baseline.
|
|
|
|
## Step 13: Cloud-read call guard
|
|
|
|
- **Report ref:** J7, and Jev's "tool-call risk gate" use case
|
|
- **Capability:** A check before each cloud-read call that it stays within the scope the user named.
|
|
- **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
|
|
- **Fallback:** fail closed: ask the user.
|
|
- **Where:** `gate_cloud_read` in `app/tools/execution.py`.
|
|
- **Depends on step 1:** tool arguments name the user's cloud resources.
|
|
- **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user.
|
|
|
|
## Step 14: Input guardrail
|
|
|
|
- **Report ref:** J11, and Jev's "LLM guardrail" use case
|
|
- **Capability:** Injected instructions are caught before a Specialist reads them.
|
|
- **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
|
|
- **Fallback:** fail closed: withhold the text and tell the Orchestrator.
|
|
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
|
|
- **Depends on step 1:** MCP results hold discovered cloud state.
|
|
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.
|
|
|
|
## Step 15: Changes of meaning in a Change request
|
|
|
|
- **Report ref:** none; from the "UI copy translation review" use case the owner named on 2026-10-07
|
|
- **Capability:** A Change request follows through only the things whose meaning changed.
|
|
- **Rationale:** A Specialist revising from a Baseline rewords text it was not asked to change. Comparing text exactly would treat every rewording as a change; ignoring text would miss a dropped restriction, such as "only" removed from a requirement.
|
|
- **Design:** For each requirement, Element and Check whose text differs from the Baseline's, one Boolean question with the old and new text and its id: "Does the new text change what must be true?" A yes marks it changed, so the Requirements chain is followed from it; a no carries the Baseline's verdict forward.
|
|
- **Fallback:** Treat it as changed.
|
|
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
|
|
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
|
|
|
|
## Step 16: Jev as the evaluation judge, calibrated
|
|
|
|
- **Capability:** The evaluation's judged checks are answered by Jev, and the judge is shown to agree with a person.
|
|
- **Rationale:** LangChain's evaluation posts report Jev agreeing with a human reviewer on every pass-or-fail decision in a small experiment, with far lower variance than language-model judges, at $0.00035 a call. They also warn that low cost can amplify mistakes and that human review and judge alignment are still needed.
|
|
- **Built:** `DeciderJudge` in `eval/judge.py` asks every question in one call, with the document as the state and no grading instructions in it. `eval/solution_eval.py --judge` uses it when `TYPESAFE_API_KEY` is set, and the language-model judge otherwise or with `EVAL_JUDGE=gemini`. A judge that gives no answer leaves its check unanswered, not passed.
|
|
- **Open:** a small set of design recommendations labelled by a person against the pillar checklists and the quality rubric; agreement of each judge with those labels; variance over repeated runs on the same output. This replaces the stability criterion in step 10.
|
|
- **Acceptance criteria:** On the labelled set, the Jev judge's pass-or-fail agreement and its variance are recorded beside the language-model judge's, and the evaluation uses the one that agrees better.
|
|
|
|
## Step 17: Trace every decision
|
|
|
|
- **Capability:** For any Execution, a person can see each decision, what it was given and what came back.
|
|
- **Rationale:** LangChain's production post treats tracing as required for model-driven steps and describes LangSmith views for decision models. Here decisions are kept in the Execution's state and the evaluation report, and nothing is traced.
|
|
- **Owner's decision needed:** whether to send traces to LangSmith. Traces carry the Workflow request and Deliverable text, so the same retention question as Jev applies. `langsmith` is already installed as a dependency of LangChain; LangGraph traces when its environment variables are set.
|
|
- **Where:** environment settings for tracing; a traced span around `JevDecider.decide` in `app/tools/jev.py` carrying the questions, answers and probabilities; `docs/operations/`.
|
|
- **Acceptance criteria:** One Execution's trace shows every Orchestrator turn, each Jev call with its inputs and outputs, and each Specialist run. With tracing off, nothing leaves the process.
|
|
|
|
## Step 18: Readings as their own graph step
|
|
|
|
- **Capability:** A resumed Execution reuses the decisions it already made.
|
|
- **Rationale:** LangGraph saves state after each node. The router takes its readings and acts inside one node, so a failure in that node asks Jev again on retry. Jev is designed to return the same answer for the same input, and the client caches answers in memory, but neither survives a restart.
|
|
- **Where:** `app/workflows/execution_graph.py` and `app/nodes/`: a node that takes the readings and saves them, before the node that applies the playbook.
|
|
- **Acceptance criteria:** A test that fails the playbook node after the readings node shows the retry using the saved readings, with no second call to the decider.
|
|
|
|
## Step 19: Online evaluation of live Executions
|
|
|
|
- **Capability:** Every Execution is scored as it runs, not only benchmark cases.
|
|
- **Rationale:** At Jev's cost, LangChain recommends scoring every run and alerting on risk flags such as leaked personal data or injected instructions.
|
|
- **Where:** the judged checks from `eval/scoring.py`, run when an Execution ends and stored with its record; with step 17, as feedback on its trace.
|
|
- **Acceptance criteria:** A completed Execution carries its judged scores. A flagged risk is recorded and surfaced without stopping the Execution.
|