12 KiB
Plan Detail
Rules for every step from 3 onward:
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
- Optimisations fail open to today's behaviour (steps 3, 6, 7, 8, 9). Guards fail closed and ask the user (steps 13, 14).
- With no API key configured, the agent behaves exactly as it does without this iteration.
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
Step 1: Resolve the open questions
- Capability: The facts needed to send client data to Jev.
- Questions for the owner or vendor:
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
- Rate limits and typical latency.
- Acceptance criteria: Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
Step 2: Decider interface and Jev client with limits
-
Capability: One seam through which the engine asks closed questions, with a real client and a scripted fake.
-
Rationale: The same reason
AgentRuntimeexists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targetscreate_agentand does not plug into the ADK runtime, so the client calls the REST API directly. -
Where: a
Deciderprotocol besideAgentRuntimeinapp/adk/runtime.py; a client and fake underapp/tools/; settings inapp/config.py(JEV_API_KEY,JEV_BASE_URL, and the limits below); wiring inopen_assemblyinapp/agent.py. -
Limits the client enforces. Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
Practice Default Refuse locally above a token count; split by Deliverable or section, never truncate 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question Send every question about one state in one call one call per state Timeout 2 seconds connect, 5 seconds total Retry 429, 5xx and network errors with exponential backoff and full jitter; honour Retry-After; never retry other 4xx2 retries, 0.5 second base, 8 second cap Process-wide token bucket, lowered automatically after a 429 5 requests per second, burst 10 Semaphore on calls in flight 8 Circuit breaker, with one probe call before closing open after 5 failures in 30 seconds; 60 second cool-down Cache answers by a hash of model, state and questions in memory, per Execution Budget per Execution; on reaching it use fallbacks and record a warning 500 calls or $0.50 Read usage.credits_remaining_usdand warn before credits run outwarn below $5 Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report always API key from the secret store or environment, redacted from logs, never in the bundle or repository JEV_API_KEY -
Acceptance criteria: Tests with a stub HTTP server cover each row of the table. The fake
Decideris used by every other test. WithJEV_API_KEYunset,open_assemblywires aDeciderthat always returns "fallback".
Step 3: Pillars a Revision touches
- Report ref: J3
- Capability: The feedback text selects which pillar advisors re-run.
- Design: One call with a Noul per pillar: "Does this feedback require changes to the recommendations?" The pillars above threshold become the
pillarsargument ofconsult_pillar_advisors. - Fallback: all pillars.
- Where:
app/tools/pillars.py; questions besidepillar_skillsinproviders/gcp.yaml. - Acceptance criteria: A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
Step 4: Semantic checks at submission
- Report ref: J4, and Jev's "task-completion check" use case
- Capability: Submission contracts that check meaning, not only headings.
- Rationale:
check_submissionchecks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens. - Design: Noul questions per Deliverable kind at
submit_deliverable, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today. - Fallback: accept on the structural contract alone.
- Where:
submit_deliverableinapp/tools/execution.py; questions per kind in a Skill or the bundle. - Acceptance criteria: A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
Step 5: Finding-to-Deliverable routing
- Report ref: J2
- Capability: Each validator finding is routed to the Deliverable to reopen, with a severity.
- Design: A Choice over the Deliverable kinds and a Score for severity, per finding from
findings.json. The result is offered to the Orchestrator in its context; the Orchestrator still callsreopen. - Fallback: the validator's own stated Deliverable.
- Where:
_orchestrator_contextinapp/nodes/execution_nodes.py. - Acceptance criteria: A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
Step 6: Model tier per delegation
- Report ref: J5, and Jev's "model routing" use case
- Capability: The cheaper model tier where the task allows.
- Design: A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
- Fallback: the role's configured model.
- Where:
default_model_factoryinapp/adk/adk_runtime.py; tiers and criteria inmodelsinproviders/gcp.yaml. - Acceptance criteria: Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
Step 7: Context selection
- Report ref: J6, and Jev's "repo context selection" use case
- Capability: Only relevant sections of earlier Deliverables reach a Specialist in full.
- Design: A Noul per section: "Is this section needed to produce the Deliverable?" Sections below threshold are passed as one-line summaries.
- Fallback: the rule from iteration 4 step 13.
- Where:
_specialist_inputinapp/nodes/execution_nodes.py. - Acceptance criteria: Describer and packager tokens fall with the quality checks unchanged.
Step 8: Skill routing
- Report ref: J9, and Jev's "agent skill routing" use case
- Capability: A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
- Design: A Choice over the role's Skills and their references, with a
noneoption. - Fallback: on
noneor low confidence, the Specialist searches the Skill itself, as today. - Where:
_render_taskinapp/adk/adk_runtime.py,app/adk/roles.py. - Acceptance criteria: Fewer
load_skill_resourcecalls per Specialist run with the quality checks unchanged.
Step 9: In-run context compaction
- Report ref: J10, and Jev's "context compaction" use case
- Capability: Old tool results inside one agent run are kept, truncated or dropped.
- Rationale: The IaC author makes 23 requests and resends full Terraform files and validate output on each.
- Design: Before each model request, a Noul and a Choice (
keep,truncate,drop) per old tool result. - Fallback: keep everything.
- Where:
app/adk/adk_runtime.py, through ADK's before-model hook. - Acceptance criteria: IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
Step 10: Evaluation rubric and lesson triage
- Report ref: J8
- Capability: Cheap graded scoring in the evaluation, and sorted lesson candidates.
- Design: Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
- Where:
eval/scoring.py, the curator from iteration 4 step 10. - Acceptance criteria: Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
Step 11: ADR: Orchestrator routing
- Capability: A recorded decision on whether code and Jev may take Orchestrator steps.
- Rationale: ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
- Evidence to bring: results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
- Acceptance criteria: An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
Step 12: Orchestrator routing
- Report ref: J1
- Capability: The Orchestrator model runs only when judgement is needed.
- Design: Four of the seven steps in
agents/orchestrator.mdare decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen. - Fallback: the Orchestrator model, as today.
- Where:
orchestrator_turninapp/nodes/execution_nodes.py. - Acceptance criteria: Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
Step 13: Cloud-read call guard
- Report ref: J7, and Jev's "tool-call risk gate" use case
- Capability: A check before each cloud-read call that it stays within the scope the user named.
- Design: A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
- Fallback: fail closed: ask the user.
- Where:
gate_cloud_readinapp/tools/execution.py. - Depends on step 1: tool arguments name the user's cloud resources.
- Acceptance criteria: A test shows an out-of-scope call becoming a question to the user.
Step 14: Input guardrail
- Report ref: J11, and Jev's "LLM guardrail" use case
- Capability: Injected instructions are caught before a Specialist reads them.
- Design: A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
- Fallback: fail closed: withhold the text and tell the Orchestrator.
- Where:
app/tools/mcp.py,await_userinapp/nodes/execution_nodes.py. - Depends on step 1: MCP results hold discovered cloud state.
- Acceptance criteria: A test with a planted instruction in an MCP result shows it withheld and noted.