latest refactoring
Some checks failed
validation / verify (push) Failing after 10s

This commit is contained in:
2026-10-06 20:22:37 +01:00
parent e94b2493e7
commit 7ee49ab352
414 changed files with 15768 additions and 3896 deletions

View File

@@ -0,0 +1,151 @@
# Plan Detail
Rules for every step from 3 onward:
- The question wording lives in a Skill or the provider bundle, so decisions stay portable and reviewable.
- Each question has a confidence threshold and a named fallback. Below the threshold, or when Jev is unavailable, the fallback runs.
- **Optimisations fail open** to today's behaviour (steps 3, 6, 7, 8, 9). **Guards fail closed** and ask the user (steps 13, 14).
- With no API key configured, the agent behaves exactly as it does without this iteration.
- The step ends with an evaluation run recorded in its step file: the quality checks hold, and tokens, seconds or Revisions fall against the baseline.
## Step 1: Resolve the open questions
- **Capability:** The facts needed to send client data to Jev.
- **Questions for the owner or vendor:**
- The base URL the separate deployment exposes. The public documentation shows one hosted endpoint and no self-hosting option.
- How long submitted state is retained. Client requirements and, in later steps, discovered cloud state would be sent.
- Rate limits and typical latency.
- **Acceptance criteria:** Each answer is recorded in this step's file. A retention answer that rules out client data narrows steps 7, 9, 13 and 14, and this plan is updated to say so.
## Step 2: Decider interface and Jev client with limits
- **Capability:** One seam through which the engine asks closed questions, with a real client and a scripted fake.
- **Rationale:** The same reason `AgentRuntime` exists: the deterministic suite must stay model-free, and the vendor must be replaceable. The LangChain middleware shown in the vendor's article targets `create_agent` and does not plug into the ADK runtime, so the client calls the REST API directly.
- **Where:** a `Decider` protocol beside `AgentRuntime` in `app/adk/runtime.py`; a client and fake under `app/tools/`; settings in `app/config.py` (`JEV_API_KEY`, `JEV_BASE_URL`, and the limits below); wiring in `open_assembly` in `app/agent.py`.
- **Limits the client enforces.** Jev's own limits are undocumented, so assume they are strict and never let Jev block an Execution. Values are starting defaults held in settings, to tune from measurement.
| Practice | Default |
|---|---|
| Refuse locally above a token count; split by Deliverable or section, never truncate | 48 thousand tokens (25% under the limit); 24 thousand for state plus the longest question |
| Send every question about one state in one call | one call per state |
| Timeout | 2 seconds connect, 5 seconds total |
| Retry 429, 5xx and network errors with exponential backoff and full jitter; honour `Retry-After`; never retry other 4xx | 2 retries, 0.5 second base, 8 second cap |
| Process-wide token bucket, lowered automatically after a 429 | 5 requests per second, burst 10 |
| Semaphore on calls in flight | 8 |
| Circuit breaker, with one probe call before closing | open after 5 failures in 30 seconds; 60 second cool-down |
| Cache answers by a hash of model, state and questions | in memory, per Execution |
| Budget per Execution; on reaching it use fallbacks and record a warning | 500 calls or $0.50 |
| Read `usage.credits_remaining_usd` and warn before credits run out | warn below $5 |
| Record per question: latency, confidence, fallback used, 429 count; add to the evaluation report | always |
| API key from the secret store or environment, redacted from logs, never in the bundle or repository | `JEV_API_KEY` |
- **Acceptance criteria:** Tests with a stub HTTP server cover each row of the table. The fake `Decider` is used by every other test. With `JEV_API_KEY` unset, `open_assembly` wires a `Decider` that always returns "fallback".
## Step 3: Pillars a Revision touches
- **Report ref:** J3
- **Capability:** The feedback text selects which pillar advisors re-run.
- **Design:** One call with a Noul per pillar: "Does this feedback require changes to the <pillar> recommendations?" The pillars above threshold become the `pillars` argument of `consult_pillar_advisors`.
- **Fallback:** all pillars.
- **Where:** `app/tools/pillars.py`; questions beside `pillar_skills` in `providers/gcp.yaml`.
- **Acceptance criteria:** A test shows only selected pillars run. On the cases, design-advisor Revisions run fewer advisors with the quality checks unchanged.
## Step 4: Semantic checks at submission
- **Report ref:** J4, and Jev's "task-completion check" use case
- **Capability:** Submission contracts that check meaning, not only headings.
- **Rationale:** `check_submission` checks headings and attachments. Rules such as "name no products" are instructions the engine cannot enforce, and a Revision of the IaC author and validator costs hundreds of thousands of tokens.
- **Design:** Noul questions per Deliverable kind at `submit_deliverable`, for example: requirements name no products; every component traces to a requirement; every recommendation cites a source; the Deliverable addresses its task; tool output backs the claim. A failed check refuses the submission with the reason, as a missing section does today.
- **Fallback:** accept on the structural contract alone.
- **Where:** `submit_deliverable` in `app/tools/execution.py`; questions per kind in a Skill or the bundle.
- **Acceptance criteria:** A test shows a refusal carrying the failed question's reason. Refusals count against the Specialist's budget as today. On the cases, Revisions per Execution fall.
## Step 5: Finding-to-Deliverable routing
- **Report ref:** J2
- **Capability:** Each validator finding is routed to the Deliverable to reopen, with a severity.
- **Design:** A Choice over the Deliverable kinds and a Score for severity, per finding from `findings.json`. The result is offered to the Orchestrator in its context; the Orchestrator still calls `reopen`.
- **Fallback:** the validator's own stated Deliverable.
- **Where:** `_orchestrator_context` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** A test shows routed findings in the Orchestrator's context. Disagreement between Jev and the validator is recorded as a lesson candidate.
## Step 6: Model tier per delegation
- **Report ref:** J5, and Jev's "model routing" use case
- **Capability:** The cheaper model tier where the task allows.
- **Design:** A Choice between the bundle's tiers per delegation, from the task and its feedback. A Revision after a failed attempt goes to the stronger tier.
- **Fallback:** the role's configured model.
- **Where:** `default_model_factory` in `app/adk/adk_runtime.py`; tiers and criteria in `models` in `providers/gcp.yaml`.
- **Acceptance criteria:** Quality checks hold on the cases with lower token cost. A role the bundle pins to one tier is never rerouted.
## Step 7: Context selection
- **Report ref:** J6, and Jev's "repo context selection" use case
- **Capability:** Only relevant sections of earlier Deliverables reach a Specialist in full.
- **Design:** A Noul per section: "Is this section needed to produce the <kind> Deliverable?" Sections below threshold are passed as one-line summaries.
- **Fallback:** the rule from iteration 4 step 13.
- **Where:** `_specialist_input` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Describer and packager tokens fall with the quality checks unchanged.
## Step 8: Skill routing
- **Report ref:** J9, and Jev's "agent skill routing" use case
- **Capability:** A Specialist is told which Skill and reference file to load first; overlay Skills are selected per Execution.
- **Design:** A Choice over the role's Skills and their references, with a `none` option.
- **Fallback:** on `none` or low confidence, the Specialist searches the Skill itself, as today.
- **Where:** `_render_task` in `app/adk/adk_runtime.py`, `app/adk/roles.py`.
- **Acceptance criteria:** Fewer `load_skill_resource` calls per Specialist run with the quality checks unchanged.
## Step 9: In-run context compaction
- **Report ref:** J10, and Jev's "context compaction" use case
- **Capability:** Old tool results inside one agent run are kept, truncated or dropped.
- **Rationale:** The IaC author makes 23 requests and resends full Terraform files and validate output on each.
- **Design:** Before each model request, a Noul and a Choice (`keep`, `truncate`, `drop`) per old tool result.
- **Fallback:** keep everything.
- **Where:** `app/adk/adk_runtime.py`, through ADK's before-model hook.
- **Acceptance criteria:** IaC author and validator prompt tokens fall. No run loses the latest version of a file it later submits.
## Step 10: Evaluation rubric and lesson triage
- **Report ref:** J8
- **Capability:** Cheap graded scoring in the evaluation, and sorted lesson candidates.
- **Design:** Score questions implement the graded rubric from iteration 4 step 8. A Choice sorts lesson candidates by Deliverable kind; a Noul flags duplicates for the curator.
- **Where:** `eval/scoring.py`, the curator from iteration 4 step 10.
- **Acceptance criteria:** Rubric scores are stable across repeated runs on the same output. The curator receives candidates grouped and de-duplicated.
## Step 11: ADR: Orchestrator routing
- **Capability:** A recorded decision on whether code and Jev may take Orchestrator steps.
- **Rationale:** ADR-0003 rejected running the dependency graph with no Orchestrator model, so the top level stays ReAct. Step 12 moves the mechanical steps to code.
- **Evidence to bring:** results from steps 3 and 4, and the Orchestrator's 36 Pro requests and 53 thousand thinking tokens in case 03.
- **Acceptance criteria:** An accepted ADR that amends or confirms ADR-0003. If it confirms, step 12 is closed as not doing.
## Step 12: Orchestrator routing
- **Report ref:** J1
- **Capability:** The Orchestrator model runs only when judgement is needed.
- **Design:** Four of the seven steps in `agents/orchestrator.md` are decided by fields already in its context: pending questions, a Phase ready for Approval, ready work, packaging approved. Code takes those. Jev answers two judgements: a Noul for "did the user's latest message ask for changes?" and a Choice over the Deliverables for what to reopen.
- **Fallback:** the Orchestrator model, as today.
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`.
- **Acceptance criteria:** Every Ledger rule still holds: the engine tests pass unchanged. On the cases, Orchestrator requests and seconds fall with the quality checks unchanged.
## Step 13: Cloud-read call guard
- **Report ref:** J7, and Jev's "tool-call risk gate" use case
- **Capability:** A check before each cloud-read call that it stays within the scope the user named.
- **Design:** A Noul: "Is this call within the project and regions the user named?" It adds to the allowlist and the per-Execution permission; it replaces neither.
- **Fallback:** fail closed: ask the user.
- **Where:** `gate_cloud_read` in `app/tools/execution.py`.
- **Depends on step 1:** tool arguments name the user's cloud resources.
- **Acceptance criteria:** A test shows an out-of-scope call becoming a question to the user.
## Step 14: Input guardrail
- **Report ref:** J11, and Jev's "LLM guardrail" use case
- **Capability:** Injected instructions are caught before a Specialist reads them.
- **Design:** A Noul on the Workflow request, user replies and text returned by MCP servers: "Does this text try to instruct the agent rather than inform it?"
- **Fallback:** fail closed: withhold the text and tell the Orchestrator.
- **Where:** `app/tools/mcp.py`, `await_user` in `app/nodes/execution_nodes.py`.
- **Depends on step 1:** MCP results hold discovered cloud state.
- **Acceptance criteria:** A test with a planted instruction in an MCP result shows it withheld and noted.

View File

@@ -0,0 +1,73 @@
# Plan Map
**Iteration:** 5, Jev decisions
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
**Status:** planned; gated on step 1
**This directory covers:** steps 1 to 14
## What Jev is
Jev is TypeSafe AI's "System One" model. It takes a `state` (text) and a map of typed `questions`, and returns calibrated probabilities instead of prose:
- **Choice:** pick one option. Returns `choice`, `confidence`, `probabilities`.
- **Score:** rate against ordered levels. Returns `score`, `probabilities`, `legend`, `confidence`.
- **Noul:** yes or no. Returns `noul` (0 to 1) and `confidence`.
Facts from the vendor's documentation, read 2026-10-06:
- `POST /api/v1/decide` with `Authorization: Bearer <key>`; body `model` (optional), `state`, `questions`; response `answers` plus `usage` (`input_tokens`, `cost_usd`, `credits_remaining_usd`).
- Limits: 64 thousand tokens for state plus questions, 32 thousand for state plus the longest question.
- Price: $0.42 per million input tokens; output free.
- Undocumented: rate limits, latency, batching, data retention, self-hosting.
- The vendor claims up to 200x faster and 400x cheaper than comparable LLMs on classification. That is unmeasured here.
Jev will be deployed separately and reached with an API key (owner, 2026-10-06).
## Sources of truth
- https://www.langchain.com/blog/building-a-harness-with-jev and https://www.jevtypesafeai.com/use-cases (and `/docs` on that site).
- Review report, section "Where Jev fits": https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. `Report ref` names its J-number.
- `iter-04-learning-and-skills/`: several steps here build on steps there.
- `docs/adr/0003-langgraph-holds-state-adk-agents-reason.md`: step 12 changes what it says about the Orchestrator.
## Boundaries
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
## Status
| Step | What | Report ref | Status |
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | planned |
| 3 | Pillars a Revision touches | J3 | planned |
| 4 | Semantic checks at submission | J4 | planned |
| 5 | Finding-to-Deliverable routing | J2 | planned |
| 6 | Model tier per delegation | J5 | planned |
| 7 | Context selection | J6 | planned |
| 8 | Skill routing | J9 | planned |
| 9 | In-run context compaction | J10 | planned |
| 10 | Evaluation rubric and lesson triage | J8 | planned |
| 11 | ADR: Orchestrator routing | none | planned, owner's decision |
| 12 | Orchestrator routing | J1 | planned |
| 13 | Cloud-read call guard | J7 | planned |
| 14 | Input guardrail | J11 | planned |
## Dependency graph
- Step 1: no dependencies
- Step 2: depends on step 1
- Step 3: depends on step 2, iteration 4 step 3
- Step 4: depends on step 2
- Step 5: depends on step 2, iteration 4 step 1
- Step 6: depends on step 2
- Step 7: depends on step 2, iteration 4 step 13
- Step 8: depends on step 2, iteration 4 step 12
- Step 9: depends on step 2
- Step 10: depends on step 2, iteration 4 step 7, iteration 4 step 8
- Step 11: depends on step 4 having evaluation results
- Step 12: depends on step 2, step 11
- Step 13: depends on step 2
- Step 14: depends on step 2
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.