Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/DETAIL.md
Jonathan Boniface 7ee49ab352
Some checks failed
validation / verify (push) Failing after 10s
latest refactoring
2026-10-06 20:22:37 +01:00

152 lines
12 KiB
Markdown

# Plan Detail
Rules for every step from 3 onward:
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
- **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14).
- With no API key configured, the agent behaves exactly as it does without this iteration.
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
## Step 1: Resolve the open questions
- **Capability:** The facts needed to send client data to Jev.
- **Questions for the owner or vendor:**
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
- Rate limits and typical latency.
- **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
## Step 2: Decider interface and Jev client with limits
- **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake.
- **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly.
- **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`.
- **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
| Practice | Default |
|---|---|
| Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question |
| Send every question about one state in one call | one call per state |
| Timeout | 2 seconds connect, 5 seconds total |
| Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap |
| Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 |
| Semaphore on calls in flight | 8 |
| Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down |
| Cache answers by a hash of model, state and questions | in memory, per Execution |
| Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 |
| Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 |
| Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always |
| API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` |
- **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback".
## Step 3: Pillars a Revision touches
- **Report ref:** J3
- **Capability:** The feedback text selects which pillar advisors re-run.
- **Design:** One call with a Noul per pillar: "Does this feedback require changes to the <pillar> recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`.
- **Fallback:** all pillars.
- **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`.
- **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
## Step 4: Semantic checks at submission
- **Report ref:** J4, and Jev's "task-completion check" use case
- **Capability:** Submission contracts that check meaning, not only headings.
- **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
- **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
- **Fallback:** accept on the structural contract alone.
- **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle.
- **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
## Step 5: Finding-to-Deliverable routing
- **Report ref:** J2
- **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity.
- **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`.
- **Fallback:** the validator's own stated Deliverable.
- **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
## Step 6: Model tier per delegation
- **Report ref:** J5, and Jev's "model routing" use case
- **Capability:** The cheaper model tier where the task allows.
- **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
- **Fallback:** the role's configured model.
- **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`.
- **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
## Step 7: Context selection
- **Report ref:** J6, and Jev's "repo context selection" use case
- **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full.
- **Design:** A Noul per section: "Is this section needed to produce the <kind> Deliverable?" Sections below threshold are passed as one-line summaries.
- **Fallback:** the rule from iteration 4 step 13.
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged.
## Step 8: Skill routing
- **Report ref:** J9, and Jev's "agent skill routing" use case
- **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
- **Design:** A Choice over the role's Skills and their references, with a `none` option.
- **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today.
- **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`.
- **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged.
## Step 9: In-run context compaction
- **Report ref:** J10, and Jev's "context compaction" use case
- **Capability:** Old tool results inside one agent run are kept, truncated or dropped.
- **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each.
- **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result.
- **Fallback:** keep everything.
- **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook.
- **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
## Step 10: Evaluation rubric and lesson triage
- **Report ref:** J8
- **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates.
- **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
## Step 11: ADR: Orchestrator routing
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
## Step 12: Orchestrator routing
- **Report ref:** J1
- **Capability:** The Orchestrator model runs only when judgement is needed.
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
- **Fallback:** the Orchestrator model, as today.
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
## Step 13: Cloud-read call guard
- **Report ref:** J7, and Jev's "tool-call risk gate" use case
- **Capability:** A check before each cloud-read call that it stays within the scope the user named.
- **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
- **Fallback:** fail closed: ask the user.
- **Where:** `gate_cloud_read` in `app/tools/execution.py`.
- **Depends on step 1:** tool arguments name the user's cloud resources.
- **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user.
## Step 14: Input guardrail
- **Report ref:** J11, and Jev's "LLM guardrail" use case
- **Capability:** Injected instructions are caught before a Specialist reads them.
- **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
- **Fallback:** fail closed: withhold the text and tell the Orchestrator.
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
- **Depends on step 1:** MCP results hold discovered cloud state.
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.