This commit is contained in:
151
.agents/plans/iter-05-jev-decisions/DETAIL.md
Normal file
151
.agents/plans/iter-05-jev-decisions/DETAIL.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# Plan Detail
|
||||
|
||||
Rules for every step from 3 onward:
|
||||
|
||||
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
|
||||
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
|
||||
- **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14).
|
||||
- With no API key configured, the agent behaves exactly as it does without this iteration.
|
||||
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
|
||||
|
||||
## Step 1: Resolve the open questions
|
||||
|
||||
- **Capability:** The facts needed to send client data to Jev.
|
||||
- **Questions for the owner or vendor:**
|
||||
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
|
||||
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
|
||||
- Rate limits and typical latency.
|
||||
- **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
|
||||
|
||||
## Step 2: Decider interface and Jev client with limits
|
||||
|
||||
- **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake.
|
||||
- **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly.
|
||||
- **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`.
|
||||
- **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
|
||||
|
||||
| Practice | Default |
|
||||
|---|---|
|
||||
| Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question |
|
||||
| Send every question about one state in one call | one call per state |
|
||||
| Timeout | 2 seconds connect, 5 seconds total |
|
||||
| Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap |
|
||||
| Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 |
|
||||
| Semaphore on calls in flight | 8 |
|
||||
| Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down |
|
||||
| Cache answers by a hash of model, state and questions | in memory, per Execution |
|
||||
| Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 |
|
||||
| Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 |
|
||||
| Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always |
|
||||
| API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` |
|
||||
|
||||
- **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback".
|
||||
|
||||
## Step 3: Pillars a Revision touches
|
||||
|
||||
- **Report ref:** J3
|
||||
- **Capability:** The feedback text selects which pillar advisors re-run.
|
||||
- **Design:** One call with a Noul per pillar: "Does this feedback require changes to the <pillar> recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`.
|
||||
- **Fallback:** all pillars.
|
||||
- **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`.
|
||||
- **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
|
||||
|
||||
## Step 4: Semantic checks at submission
|
||||
|
||||
- **Report ref:** J4, and Jev's "task-completion check" use case
|
||||
- **Capability:** Submission contracts that check meaning, not only headings.
|
||||
- **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
|
||||
- **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
|
||||
- **Fallback:** accept on the structural contract alone.
|
||||
- **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle.
|
||||
- **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
|
||||
|
||||
## Step 5: Finding-to-Deliverable routing
|
||||
|
||||
- **Report ref:** J2
|
||||
- **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity.
|
||||
- **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`.
|
||||
- **Fallback:** the validator's own stated Deliverable.
|
||||
- **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
|
||||
|
||||
## Step 6: Model tier per delegation
|
||||
|
||||
- **Report ref:** J5, and Jev's "model routing" use case
|
||||
- **Capability:** The cheaper model tier where the task allows.
|
||||
- **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
|
||||
- **Fallback:** the role's configured model.
|
||||
- **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`.
|
||||
- **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
|
||||
|
||||
## Step 7: Context selection
|
||||
|
||||
- **Report ref:** J6, and Jev's "repo context selection" use case
|
||||
- **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full.
|
||||
- **Design:** A Noul per section: "Is this section needed to produce the <kind> Deliverable?" Sections below threshold are passed as one-line summaries.
|
||||
- **Fallback:** the rule from iteration 4 step 13.
|
||||
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged.
|
||||
|
||||
## Step 8: Skill routing
|
||||
|
||||
- **Report ref:** J9, and Jev's "agent skill routing" use case
|
||||
- **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
|
||||
- **Design:** A Choice over the role's Skills and their references, with a `none` option.
|
||||
- **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today.
|
||||
- **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`.
|
||||
- **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged.
|
||||
|
||||
## Step 9: In-run context compaction
|
||||
|
||||
- **Report ref:** J10, and Jev's "context compaction" use case
|
||||
- **Capability:** Old tool results inside one agent run are kept, truncated or dropped.
|
||||
- **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each.
|
||||
- **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result.
|
||||
- **Fallback:** keep everything.
|
||||
- **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook.
|
||||
- **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
|
||||
|
||||
## Step 10: Evaluation rubric and lesson triage
|
||||
|
||||
- **Report ref:** J8
|
||||
- **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates.
|
||||
- **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
|
||||
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
|
||||
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
|
||||
|
||||
## Step 11: ADR: Orchestrator routing
|
||||
|
||||
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
|
||||
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
|
||||
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
|
||||
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
|
||||
|
||||
## Step 12: Orchestrator routing
|
||||
|
||||
- **Report ref:** J1
|
||||
- **Capability:** The Orchestrator model runs only when judgement is needed.
|
||||
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
|
||||
- **Fallback:** the Orchestrator model, as today.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
|
||||
|
||||
## Step 13: Cloud-read call guard
|
||||
|
||||
- **Report ref:** J7, and Jev's "tool-call risk gate" use case
|
||||
- **Capability:** A check before each cloud-read call that it stays within the scope the user named.
|
||||
- **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
|
||||
- **Fallback:** fail closed: ask the user.
|
||||
- **Where:** `gate_cloud_read` in `app/tools/execution.py`.
|
||||
- **Depends on step 1:** tool arguments name the user's cloud resources.
|
||||
- **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user.
|
||||
|
||||
## Step 14: Input guardrail
|
||||
|
||||
- **Report ref:** J11, and Jev's "LLM guardrail" use case
|
||||
- **Capability:** Injected instructions are caught before a Specialist reads them.
|
||||
- **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
|
||||
- **Fallback:** fail closed: withhold the text and tell the Orchestrator.
|
||||
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
|
||||
- **Depends on step 1:** MCP results hold discovered cloud state.
|
||||
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.
|
||||
73
.agents/plans/iter-05-jev-decisions/MAP.md
Normal file
73
.agents/plans/iter-05-jev-decisions/MAP.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1
|
||||
**This directory covers:** steps 1 to 14
|
||||
|
||||
## What Jev is
|
||||
|
||||
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
|
||||
|
||||
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
|
||||
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
|
||||
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
|
||||
|
||||
Facts from the vendor's documentation, read 2026-10-06:
|
||||
|
||||
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
|
||||
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
|
||||
- Price: $0.42 per million input tokens; output free.
|
||||
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
|
||||
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
|
||||
|
||||
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
|
||||
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
|
||||
- `iter-04-learning-and-skills/`: several steps here build on steps there.
|
||||
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
| 6 | Model tier per delegation | J5 | planned |
|
||||
| 7 | Context selection | J6 | planned |
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Orchestrator routing | none | planned, owner's decision |
|
||||
| 12 | Orchestrator routing | J1 | planned |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: depends on step 1
|
||||
- Step 3: depends on step 2, iteration 4 step 3
|
||||
- Step 4: depends on step 2
|
||||
- Step 5: depends on step 2, iteration 4 step 1
|
||||
- Step 6: depends on step 2
|
||||
- Step 7: depends on step 2, iteration 4 step 13
|
||||
- Step 8: depends on step 2, iteration 4 step 12
|
||||
- Step 9: depends on step 2
|
||||
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
|
||||
- Step 11: depends on step 4 having evaluation results
|
||||
- Step 12: depends on step 2, step 11
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
Reference in New Issue
Block a user