Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/DETAIL.md
Jonathan Boniface 7ee49ab352
Some checks failed
validation / verify (push) Failing after 10s
latest refactoring
2026-10-06 20:22:37 +01:00

12 KiB

Plan Detail

Rules for every step from 3 onward:

  • The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
  • Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
  • Optimisations fail open to today's behaviour (steps 3, 6, 7, 8, 9). Guards fail closed and ask the user (steps 13, 14).
  • With no API key configured, the agent behaves exactly as it does without this iteration.
  • The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.

Step 1: Resolve the open questions

  • Capability: The facts needed to send client data to Jev.
  • Questions for the owner or vendor:
    • The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
    • How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
    • Rate limits and typical latency.
  • Acceptance criteria: Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.

Step 2: Decider interface and Jev client with limits

  • Capability: One seam through which the engine asks closed questions, with a real client and a scripted fake.

  • Rationale: The same reason AgentRuntime exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets create_agent and does not plug into the ADK runtime, so the client calls the REST API directly.

  • Where: a Decider protocol beside AgentRuntime in app/adk/runtime.py; a client and fake under app/tools/; settings in app/config.py (JEV_API_KEY, JEV_BASE_URL, and the limits below); wiring in open_assembly in app/agent.py.

  • Limits the client enforces. Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.

    Practice Default
    Refuse locally above a token count; split by Deliverable or section, never truncate 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question
    Send every question about one state in one call one call per state
    Timeout 2 seconds connect, 5 seconds total
    Retry 429, 5xx and network errors with exponential backoff and full jitter; honour Retry-After; never retry other 4xx 2 retries, 0.5 second base, 8 second cap
    Process-wide token bucket, lowered automatically after a 429 5 requests per second, burst 10
    Semaphore on calls in flight 8
    Circuit breaker, with one probe call before closing open after 5 failures in 30 seconds; 60 second cool-down
    Cache answers by a hash of model, state and questions in memory, per Execution
    Budget per Execution; on reaching it use fallbacks and record a warning 500 calls or $0.50
    Read usage.credits_remaining_usd and warn before credits run out warn below $5
    Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report always
    API key from the secret store or environment, redacted from logs, never in the bundle or repository JEV_API_KEY
  • Acceptance criteria: Tests with a stub HTTP server cover each row of the table. The fake Decider is used by every other test. With JEV_API_KEY unset, open_assembly wires a Decider that always returns "fallback".

Step 3: Pillars a Revision touches

  • Report ref: J3
  • Capability: The feedback text selects which pillar advisors re-run.
  • Design: One call with a Noul per pillar: "Does this feedback require changes to the recommendations?" The pillars above threshold become the pillars argument of consult_pillar_advisors.
  • Fallback: all pillars.
  • Where: app/tools/pillars.py; questions beside pillar_skills in providers/gcp.yaml.
  • Acceptance criteria: A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.

Step 4: Semantic checks at submission

  • Report ref: J4, and Jev's "task-completion check" use case
  • Capability: Submission contracts that check meaning, not only headings.
  • Rationale: check_submission checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
  • Design: Noul questions per Deliverable kind at submit_deliverable, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
  • Fallback: accept on the structural contract alone.
  • Where: submit_deliverable in app/tools/execution.py; questions per kind in a Skill or the bundle.
  • Acceptance criteria: A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.

Step 5: Finding-to-Deliverable routing

  • Report ref: J2
  • Capability: Each validator finding is routed to the Deliverable to reopen, with a severity.
  • Design: A Choice over the Deliverable kinds and a Score for severity, per finding from findings.json. The result is offered to the Orchestrator in its context; the Orchestrator still calls reopen.
  • Fallback: the validator's own stated Deliverable.
  • Where: _orchestrator_context in app/nodes/execution_nodes.py.
  • Acceptance criteria: A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.

Step 6: Model tier per delegation

  • Report ref: J5, and Jev's "model routing" use case
  • Capability: The cheaper model tier where the task allows.
  • Design: A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
  • Fallback: the role's configured model.
  • Where: default_model_factory in app/adk/adk_runtime.py; tiers and criteria in models in providers/gcp.yaml.
  • Acceptance criteria: Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.

Step 7: Context selection

  • Report ref: J6, and Jev's "repo context selection" use case
  • Capability: Only relevant sections of earlier Deliverables reach a Specialist in full.
  • Design: A Noul per section: "Is this section needed to produce the Deliverable?" Sections below threshold are passed as one-line summaries.
  • Fallback: the rule from iteration 4 step 13.
  • Where: _specialist_input in app/nodes/execution_nodes.py.
  • Acceptance criteria: Describer and packager tokens fall with the quality checks unchanged.

Step 8: Skill routing

  • Report ref: J9, and Jev's "agent skill routing" use case
  • Capability: A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
  • Design: A Choice over the role's Skills and their references, with a none option.
  • Fallback: on none or low confidence, the Specialist searches the Skill itself, as today.
  • Where: _render_task in app/adk/adk_runtime.py, app/adk/roles.py.
  • Acceptance criteria: Fewer load_skill_resource calls per Specialist run with the quality checks unchanged.

Step 9: In-run context compaction

  • Report ref: J10, and Jev's "context compaction" use case
  • Capability: Old tool results inside one agent run are kept, truncated or dropped.
  • Rationale: The IaC author makes 23 requests and resends full Terraform files and validate output on each.
  • Design: Before each model request, a Noul and a Choice (keep, truncate, drop) per old tool result.
  • Fallback: keep everything.
  • Where: app/adk/adk_runtime.py, through ADK's before-model hook.
  • Acceptance criteria: IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.

Step 10: Evaluation rubric and lesson triage

  • Report ref: J8
  • Capability: Cheap graded scoring in the evaluation, and sorted lesson candidates.
  • Design: Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
  • Where: eval/scoring.py, the curator from iteration 4 step 10.
  • Acceptance criteria: Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.

Step 11: ADR: Orchestrator routing

  • Capability: A recorded decision on whether code and Jev may take Orchestrator steps.
  • Rationale: ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
  • Evidence to bring: results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
  • Acceptance criteria: An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.

Step 12: Orchestrator routing

  • Report ref: J1
  • Capability: The Orchestrator model runs only when judgement is needed.
  • Design: Four of the seven steps in agents/orchestrator.md are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
  • Fallback: the Orchestrator model, as today.
  • Where: orchestrator_turn in app/nodes/execution_nodes.py.
  • Acceptance criteria: Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.

Step 13: Cloud-read call guard

  • Report ref: J7, and Jev's "tool-call risk gate" use case
  • Capability: A check before each cloud-read call that it stays within the scope the user named.
  • Design: A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
  • Fallback: fail closed: ask the user.
  • Where: gate_cloud_read in app/tools/execution.py.
  • Depends on step 1: tool arguments name the user's cloud resources.
  • Acceptance criteria: A test shows an out-of-scope call becoming a question to the user.

Step 14: Input guardrail

  • Report ref: J11, and Jev's "LLM guardrail" use case
  • Capability: Injected instructions are caught before a Specialist reads them.
  • Design: A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
  • Fallback: fail closed: withhold the text and tell the Orchestrator.
  • Where: app/tools/mcp.py, await_user in app/nodes/execution_nodes.py.
  • Depends on step 1: MCP results hold discovered cloud state.
  • Acceptance criteria: A test with a planted instruction in an MCP result shows it withheld and noted.