# Plan Detail Rules for every step from 3 onward: - The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable. - Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs. - **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14). - With no API key configured, the agent behaves exactly as it does without this iteration. - The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline. ## Step 1: Resolve the open questions - **Capability:** The facts needed to send client data to Jev. - **Questions for the owner or vendor:** - The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option. - How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent. - Rate limits and typical latency. - **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so. ## Step 2: Decider interface and Jev client with limits - **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake. - **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly. - **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`. - **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement. | Practice | Default | |---|---| | Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question | | Send every question about one state in one call | one call per state | | Timeout | 2 seconds connect, 5 seconds total | | Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap | | Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 | | Semaphore on calls in flight | 8 | | Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down | | Cache answers by a hash of model, state and questions | in memory, per Execution | | Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 | | Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 | | Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always | | API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` | - **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback". ## Step 3: Pillars a Revision touches - **Report ref:** J3 - **Capability:** The feedback text selects which pillar advisors re-run. - **Design:** One call with a Noul per pillar: "Does this feedback require changes to the recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`. - **Fallback:** all pillars. - **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`. - **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged. ## Step 4: Semantic checks at submission - **Report ref:** J4, and Jev's "task-completion check" use case - **Capability:** Submission contracts that check meaning, not only headings. - **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens. - **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today. - **Fallback:** accept on the structural contract alone. - **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle. - **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall. ## Step 5: Finding-to-Deliverable routing - **Report ref:** J2 - **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity. - **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`. - **Fallback:** the validator's own stated Deliverable. - **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`. - **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate. ## Step 6: Model tier per delegation - **Report ref:** J5, and Jev's "model routing" use case - **Capability:** The cheaper model tier where the task allows. - **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier. - **Fallback:** the role's configured model. - **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`. - **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted. ## Step 7: Context selection - **Report ref:** J6, and Jev's "repo context selection" use case - **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full. - **Design:** A Noul per section: "Is this section needed to produce the Deliverable?" Sections below threshold are passed as one-line summaries. - **Fallback:** the rule from iteration 4 step 13. - **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`. - **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged. ## Step 8: Skill routing - **Report ref:** J9, and Jev's "agent skill routing" use case - **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution. - **Design:** A Choice over the role's Skills and their references, with a `none` option. - **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today. - **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`. - **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged. ## Step 9: In-run context compaction - **Report ref:** J10, and Jev's "context compaction" use case - **Capability:** Old tool results inside one agent run are kept, truncated or dropped. - **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each. - **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result. - **Fallback:** keep everything. - **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook. - **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits. ## Step 10: Evaluation rubric and lesson triage - **Report ref:** J8 - **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates. - **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator. - **Where:** `eval/scoring.py`, the curator from iteration 4 step 10. - **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated. ## Step 11: ADR: Orchestrator routing - **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps. - **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code. - **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03. - **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing. ## Step 12: Orchestrator routing - **Report ref:** J1 - **Capability:** The Orchestrator model runs only when judgement is needed. - **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen. - **Fallback:** the Orchestrator model, as today. - **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`. - **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged. ## Step 13: Cloud-read call guard - **Report ref:** J7, and Jev's "tool-call risk gate" use case - **Capability:** A check before each cloud-read call that it stays within the scope the user named. - **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither. - **Fallback:** fail closed: ask the user. - **Where:** `gate_cloud_read` in `app/tools/execution.py`. - **Depends on step 1:** tool arguments name the user's cloud resources. - **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user. ## Step 14: Input guardrail - **Report ref:** J11, and Jev's "LLM guardrail" use case - **Capability:** Injected instructions are caught before a Specialist reads them. - **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?" - **Fallback:** fail closed: withhold the text and tell the Orchestrator. - **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`. - **Depends on step 1:** MCP results hold discovered cloud state. - **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.