fixed multile issues
Some checks failed
validation / verify (push) Failing after 8s

This commit is contained in:
2026-10-07 17:33:36 +01:00
parent 4c4d44ef87
commit 5bb22d984e
76 changed files with 5799 additions and 179 deletions

View File

@@ -9,8 +9,8 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8 and 15 built; no evaluation run has completed on them; 9 awaits the owner; 10–14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. |
| 5 | `iter-05-jev-decisions/` | planned, gated on open questions | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–20 built; no evaluation run has completed on them, so their effect on time and tokens is unmeasured; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; step 11's evaluation run is outstanding, so unproven with real models | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |

View File

@@ -137,3 +137,35 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Rationale:** Each check runs `terraform init` in a new scratch directory, so every call downloaded its providers again. The setting `TF_PLUGIN_CACHE_DIR` existed and was unset by default.
- **Where:** `TerraformRunner` in `app/tools/local.py`, `app/config.py`, `.env.example`.
- **Acceptance criteria:** The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.
## Step 17: A working copy for files and for the graph
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
- **Capability:** A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
- **Rationale:** In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every `terraform_fmt` and `terraform_validate` call carried every file, `terraform_fmt` returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
- **Where:** `app/tools/workspace.py` (new), `specialist_tools` in `app/tools/execution.py`, `app/tools/local.py`, `app/tools/toolkit.py`, `DeliverableSpec.working_copy` in `app/workflows/deliverables.py`, `render_tables` in `app/workflows/target_architecture.py`, `agents/target-architect.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
- **Acceptance criteria:** The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.
## Step 18: Provider schema lookup and one pinned provider version
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** An agent can ask what a resource type accepts, and get the provider's own answer.
- **Rationale:** All 13 Terraform refusals in the 2026-10-07 run were one mistake: `startup_cpu_boost` and `cpu_idle` in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
- **Where:** `TerraformRunner.schema` in `app/tools/local.py`, `terraform.required_providers` in `providers/gcp.yaml`, `app/providers.py`, the role files of the target architect, the IaC author and the validator.
- **Acceptance criteria:** `terraform_schema` lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.
## Step 19: A Check the code cannot meet goes back to the Target architecture
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
- **Rationale:** Check K-30 of the 2026-10-07 run asked for `startup_cpu_boost` on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
- **Where:** `check_implementation` and `unimplementable_checks` in `app/workflows/chain.py`, the playbook in `app/nodes/router.py`, `_orchestrator_context` in `app/nodes/execution_nodes.py`, `agents/orchestrator.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** `implementation.json` may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.
## Step 20: A model timeout per role
- **Report ref:** none; requested by the owner on 2026-10-07
- **Capability:** A role whose replies are long is given longer for one model request than the rest.
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.

View File

@@ -2,9 +2,9 @@
**Iteration:** 4, learning and Skills
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
**Status:** steps 1 to 8 and 15 built; no evaluation run has completed on them (see `steps/step-8.md`); step 9 drafted and awaiting the owner; steps 10 to 14 planned. Steps 1 to 6 are in commit `7ee49ab`; 7, 8 and 15 are uncommitted.
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; no evaluation run has completed on any of them (see `steps/step-8.md`), so every "tokens fall" or "Revisions fall" criterion is unmeasured; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
**This directory covers:** steps 1 to 16
**This directory covers:** steps 1 to 20
## Sources of truth
@@ -27,12 +27,16 @@
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
| 9 | ADR: the lessons loop and overlay Skills | B | S | drafted as `docs/adr/0004-lessons-and-overlays-arrive-as-skills.md`, status proposed; awaits the owner |
| 10 | Lessons Skill and lesson curator | B | L | planned |
| 11 | IaC patterns Skill | B | M | planned |
| 11 | IaC patterns Skill | B | M | built; every pattern validates; effect on Revisions and tokens unmeasured |
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
| 13 | Trim Specialist context | C | M | planned |
| 13 | Trim Specialist context | C | M | built; effect on tokens unmeasured |
| 14 | Provider-owned contracts | C | M | planned |
| 15 | Group Executions by Customer case | added | S | complete |
| 16 | Cache Terraform providers between checks | added | S | complete |
| 17 | A working copy for files and for the graph | added | M | built; effect on time and tokens unmeasured |
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
| 20 | A model timeout per role | added | S | built |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
@@ -49,5 +53,7 @@ Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that
- Step 14: depends on step 4
- Step 15: no dependencies (added on 2026-10-06 at the owner's request)
- Step 16: no dependencies (added on 2026-10-07 at the owner's request)
- Steps 17 to 20: no dependencies (added on 2026-10-07 at the owner's request, from the causes of the 60 minute run recorded in `../iter-06-target-architecture-graph/steps/`)
- Step 11 uses step 18's pinned provider version; step 19 uses step 18's lookup to find the Checks it reports
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.

View File

@@ -0,0 +1,13 @@
# Step 11 — Implementation Detail
Built on 2026-10-07 as `skills/google-cloud-iac-patterns/`. The name carries the Cloud provider because the content does; the provider bundle assigns it to the IaC author (`role_skills.iac-author`), so another provider brings its own patterns Skill with no code change.
1. `SKILL.md` says how to use the patterns and lists one reference per product. `assets/versions.tf` pins `hashicorp/google` 8.6.0, the version `providers/gcp.yaml` pins (step 18).
2. Thirteen references, each a short list of what goes wrong and one self-contained pattern: enabling APIs, IAM, Cloud Run service, Cloud Run job, Pub/Sub, Cloud Storage, Secret Manager, Cloud KMS, Cloud SQL, VPC networking, Cloud Scheduler, Artifact Registry, Monitoring and Logging.
3. Seeded from the 2026-10-07 run's lesson candidates. All 13 Terraform refusals were `startup_cpu_boost` or `cpu_idle` in the wrong block or on a Cloud Run job. The service reference says which block each setting sits in; the job reference lists what a job does not have.
Differs from the plan: the plan's acceptance is a fall in IaC author Revisions or tokens on the three cases. **No evaluation has run, so that is unmeasured.** What is verified is that the patterns are correct.
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.

View File

@@ -0,0 +1,14 @@
# Step 13 — Implementation Detail
Built on 2026-10-07.
1. `DeliverableSpec.reads_in_full` names the earlier Deliverables a Specialist reads whole. Left unset it is the Deliverables it depends on.
2. `dependencies_for(kind, deliverables)` gives every other input as its data file alone: `requirements.json`, `components.json` and so on. Those ids are all the Requirements chain checks need, so submission checks are unchanged.
3. Two overrides. The IaC author reads nothing whole: the code is written from `target-architecture.json`. The packager reads everything whole, since the guide is assembled from every Deliverable's prose.
4. A Deliverable given as data can be read whole with `read_deliverable(kind)`, offered only when something was trimmed.
Differs from the plan: the plan said "summaries". The data file is used instead. It already exists, is exact, and costs no model call to produce.
**Unmeasured:** the plan's acceptance is that describer tokens fall against the step 8 baseline. No evaluation has run.
Tests: `tests/execution/test_working_copy.py`, "each Specialist reads what it works from".

View File

@@ -0,0 +1,21 @@
# Step 17 — Implementation Detail
Built on 2026-10-07. `app/tools/workspace.py` keeps a working copy in the Deliverable's working memory, which the Execution already checkpoints and returns on every attempt.
Files (`working["files"]`), for the Deliverable whose `working_copy` is its own kind (deployment guidance):
1. `write_file`, `edit_file` (replaces the one place `old` matches, and refuses when it matches none or several), `read_file` (one file, or the list) and `delete_file`.
2. `terraform_fmt`, `terraform_validate` and `terraform_plan` run on the Terraform files in the working copy and take no `files` argument. `terraform_fmt` formats in place and returns the names it changed. Files passed anyway are written to the working copy first, so a model that ignores the instruction still works.
3. `submit_deliverable` attaches the working copy; a file passed as an attachment replaces its copy. The submitted files become the working copy, so a Revision starts from them, and the task context lists them under `working_copy`.
The validator's `working_copy` is deployment guidance: its tools are given a fresh copy of the submitted files on every attempt, it has no write tools, and nothing from the copy is attached to its Deliverable.
Graph (`working["graph"]`), for the Target architecture:
4. `graph_put` adds items or replaces those with the same id, `graph_remove` removes by id, `graph_read` returns the graph. The submitted graph becomes the working graph.
5. `submit_deliverable` attaches the working graph when no `target-architecture.json` is passed.
6. `render_tables` writes the Elements, Relationships, Checks and not-applicable tables from the graph, and the engine appends them to the content on submission. The contract's sections are now `Design rationale` alone. This closes the duplicate output iteration 6 left: the architect no longer writes the graph twice.
**Unmeasured:** no evaluation has run, so the fall in requests, tokens and time is expected, not shown. The mechanism is what is tested.
Tests: `tests/execution/test_working_copy.py` (21), including an Execution in which the validator validates the submitted code with no files passed.

View File

@@ -0,0 +1,17 @@
# Step 18 — Implementation Detail
Built on 2026-10-07.
1. `providers/gcp.yaml` pins `terraform.required_providers.google` to `hashicorp/google` 8.6.0. `load_provider` requires a `source` and an exact `version`.
2. `TerraformRunner.schema(resource_type, block, find)` writes a versions file for the pinned providers in a scratch directory, runs `terraform init -backend=false` and `terraform providers schema -json` once per process, and keeps the resource schemas in memory. Measured here: 0.6 seconds and a 9.7 MB reply for the Google provider, from the plugin cache.
3. With `resource_type` alone it lists the arguments code may set (read-only ones are left out) and outlines every block. With `block` it gives one block's arguments with descriptions. With `find` it says where a setting sits, or that the resource type has none. An unknown resource type gets close matches.
4. The tool is `terraform_schema`, tier general, given to the target architect, the IaC author and the validator.
Checked against the real provider on 2026-10-07:
- `find="startup_cpu_boost"` on `google_cloud_run_v2_job`: "has no argument or block named like 'startup_cpu_boost'. It cannot be set on this resource type."
- the same on `google_cloud_run_v2_service`: `template.containers.resources.startup_cpu_boost (bool, optional)`.
Not done: generated code is not forced onto the pinned version. An agent that writes its own `required_providers` with another version is validated against that version. The patterns Skill gives it the pinned `versions.tf` to use.
Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `tests/test_providers.py` for the pin.

View File

@@ -0,0 +1,12 @@
# Step 19 — Implementation Detail
Built on 2026-10-07.
1. An `implementation.json` entry may be `{"check": "K-7", "unimplementable": "<what the resource type lacks>"}` in place of `file` and `address`. It accounts for the Check; an empty reason does not.
2. `unimplementable_checks` reads those entries. The router reads them as soon as deployment guidance is submitted, and a new playbook group, "the code cannot meet a Check", sits before "validation decided" and "a Phase is complete". Its action reopens the Target architecture with each Check's id and reason. That returns the code to draft, so the group stops applying.
3. A model Orchestrator is given the same list as `unimplementable_checks` in its context, and `agents/orchestrator.md` has the step.
4. The validator is told to confirm with `terraform_schema` before failing a Check for a missing setting, and to give not verifiable when the resource type has no such setting. Not verifiable already routes to the Target architecture.
Past the Revision limit the reopen is refused as any other is: the user decides, or a pre-approved Execution ends with warnings.
Tests: `tests/execution/test_requirements_chain.py` (two) and `tests/execution/test_router.py`, "a Check the code cannot meet goes back to the graph unvalidated".

View File

@@ -0,0 +1,7 @@
# Step 20 — Implementation Detail
Built on 2026-10-07. `models.timeout_seconds` in the provider bundle maps a role to the seconds one model request may take; `RoleSpec.model_timeout_seconds` carries it and `AdkAgentRuntime` uses it in place of `MODEL_TIMEOUT_SECONDS`.
`providers/gcp.yaml` sets the target architect to 420 and the IaC author and the packager to 300. These are starting values, not measurements. With step 17 the architect's replies are short, so its 420 should rarely be reached.
Tests: `tests/agents/test_adk_runtime.py`, `tests/agents/test_roles.py`, `tests/test_providers.py`.

View File

@@ -135,6 +135,12 @@ Rules for every step from 3 onward:
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
- **Routing (owner's direction, 2026-10-07):** the Orchestrator acts as a router with Jev choosing the route. Four things are routed:
- a message that starts something: new Workflow request, Change request against a Baseline, question about delivered work, or out of scope (Choice);
- a message inside an Execution: an answer to a waiting question, a request to change a named Deliverable, or anything else (Choice, then a Choice over the Deliverables);
- a validation outcome: revise the code, revise the Target architecture, or proceed (Choice);
- in a Change request, which Deliverables the change reaches (a Boolean per Deliverable, with the Requirements chain as the deterministic check).
The route decides what is dispatched. Which Specialist produces a Deliverable stays fixed by the dependency graph, and every Deliverable is still produced in a new Execution. The first route needs an entry point that can start a Change request or answer from a case's files (`app/main.py`, `app/a2a_server.py`); the last depends on iteration 7.
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
@@ -169,3 +175,33 @@ Rules for every step from 3 onward:
- **Fallback:** Treat it as changed.
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
## Step 16: Jev as the evaluation judge, calibrated
- **Capability:** The evaluation's judged checks are answered by Jev, and the judge is shown to agree with a person.
- **Rationale:** LangChain's evaluation posts report Jev agreeing with a human reviewer on every pass-or-fail decision in a small experiment, with far lower variance than language-model judges, at $0.00035 a call. They also warn that low cost can amplify mistakes and that human review and judge alignment are still needed.
- **Built:** `DeciderJudge` in `eval/judge.py` asks every question in one call, with the document as the state and no grading instructions in it. `eval/solution_eval.py --judge` uses it when `TYPESAFE_API_KEY` is set, and the language-model judge otherwise or with `EVAL_JUDGE=gemini`. A judge that gives no answer leaves its check unanswered, not passed.
- **Open:** a small set of design recommendations labelled by a person against the pillar checklists and the quality rubric; agreement of each judge with those labels; variance over repeated runs on the same output. This replaces the stability criterion in step 10.
- **Acceptance criteria:** On the labelled set, the Jev judge's pass-or-fail agreement and its variance are recorded beside the language-model judge's, and the evaluation uses the one that agrees better.
## Step 17: Trace every decision
- **Capability:** For any Execution, a person can see each decision, what it was given and what came back.
- **Rationale:** LangChain's production post treats tracing as required for model-driven steps and describes LangSmith views for decision models. Here decisions are kept in the Execution's state and the evaluation report, and nothing is traced.
- **Owner's decision needed:** whether to send traces to LangSmith. Traces carry the Workflow request and Deliverable text, so the same retention question as Jev applies. `langsmith` is already installed as a dependency of LangChain; LangGraph traces when its environment variables are set.
- **Where:** environment settings for tracing; a traced span around `JevDecider.decide` in `app/tools/jev.py` carrying the questions, answers and probabilities; `docs/operations/`.
- **Acceptance criteria:** One Execution's trace shows every Orchestrator turn, each Jev call with its inputs and outputs, and each Specialist run. With tracing off, nothing leaves the process.
## Step 18: Readings as their own graph step
- **Capability:** A resumed Execution reuses the decisions it already made.
- **Rationale:** LangGraph saves state after each node. The router takes its readings and acts inside one node, so a failure in that node asks Jev again on retry. Jev is designed to return the same answer for the same input, and the client caches answers in memory, but neither survives a restart.
- **Where:** `app/workflows/execution_graph.py` and `app/nodes/`: a node that takes the readings and saves them, before the node that applies the playbook.
- **Acceptance criteria:** A test that fails the playbook node after the readings node shows the retry using the saved readings, with no second call to the decider.
## Step 19: Online evaluation of live Executions
- **Capability:** Every Execution is scored as it runs, not only benchmark cases.
- **Rationale:** At Jev's cost, LangChain recommends scoring every run and alerting on risk flags such as leaked personal data or injected instructions.
- **Where:** the judged checks from `eval/scoring.py`, run when an Execution ends and stored with its record; with step 17, as feedback on its trace.
- **Acceptance criteria:** A completed Execution carries its judged scores. A flagged risk is recorded and surfaced without stopping the Execution.

View File

@@ -2,8 +2,8 @@
**Iteration:** 5, Jev decisions
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
**This directory covers:** steps 1 to 15
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
**This directory covers:** steps 1 to 19
## What Jev is
@@ -49,6 +49,24 @@ On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
## Alignment with LangChain's published guidance
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
| Their guidance | Here | Step |
|---|---|---|
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
## Boundaries
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
@@ -58,7 +76,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| Step | What | Report ref | Status |
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | planned |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | planned |
| 4 | Semantic checks at submission | J4 | planned |
| 5 | Finding-to-Deliverable routing | J2 | planned |
@@ -67,11 +85,15 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 8 | Skill routing | J9 | planned |
| 9 | In-run context compaction | J10 | planned |
| 10 | Evaluation rubric and lesson triage | J8 | planned |
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
| 13 | Cloud-read call guard | J7 | planned |
| 14 | Input guardrail | J11 | planned |
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned |
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
## Dependency graph
@@ -90,5 +112,9 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
- Step 13: depends on step 2
- Step 14: depends on step 2
- Step 15: depends on step 2, iteration 7 step 4
- Step 16: depends on step 2
- Step 17: no dependencies
- Step 18: depends on step 12
- Step 19: depends on step 16, step 17
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.

View File

@@ -0,0 +1,9 @@
# Step 11 — Implementation Detail
ADR-0007 was accepted by the owner on 2026-10-07 ("this is exactly what I want, I approve ADR-0007"), after three refinements made the same day and recorded in it:
1. Jev is the decision point, in place of a different language model.
2. The agent-control pattern at https://vercel.com/i/jev-agent-control is followed: a decision model reads, application code checks permission and executes, a generative model writes replies.
3. The Orchestrator is a router, and decides by readings, a disposition and a playbook, on the model of a triage pipeline the owner supplied: the first group that applies, the strongest action whose conditions hold, escalate when none applies.
Jev is reached through TypeSafe AI's Python SDK and API endpoint (owner, 2026-10-07). The API key is still to be supplied.

View File

@@ -0,0 +1,36 @@
# Step 12 — Implementation Detail
Built against a scripted decider. No call has been made to the live Jev service, and no evaluation has run on it: the owner holds evaluation runs until the Jev work is in.
1. **`app/nodes/router.py`.** `Router` decides an Orchestrator turn in three stages: readings (`_read_ledger` from the Ledger and data; `_read_message` and `_read_gap` from the decider), a disposition by `Thresholds` (act at 0.75, close at 0.15, choice confidence 0.6), and the playbook in `_choose`: ordered groups, the strongest action whose conditions hold, escalate when none applies.
2. **Permission stays with the tools.** The router calls the same `OrchestratorToolbox` tools the model did. A refused action is skipped for the rest of the turn and the next one that holds is taken. Past the Revision limit the user is asked.
3. **What Jev is asked** (`Question` objects, batched when they rest on the same evidence):
- of the user's message: `asks_change` (noul), `change_target` (choice over submitted Deliverables), `change_clarity` (score: vague, partial, specific);
- of validation gaps: `gap_material` (noul);
- of the user's reply about gaps: `wants_revision` (noul).
What is sent is Deliverable names, the count of waiting questions, the user's message, and for gaps the proposed Check statements and unverifiable Check ids.
4. **Rules when Jev gives no answer.** A requested change is acted on only when one Deliverable is named outright, otherwise the user is asked which. A free-text reply is treated as an answer. A gap is revised for when a Check cannot be verified or an addition names a real requirement or recommendation id.
5. **A new reading from the run of 2026-10-07:** a Check that fails in two validations running is sent to the Target architecture, not the code. That run retried one unimplementable Check as code three times.
6. **Decisions are recorded** in the Execution's state under `decisions`: turn, group, action and what was judged, with confidence.
7. **The reply writer.** A new role, `agents/reply-writer.md`, writes the Approval summary from facts (`approval_facts`) through one tool. When it fails or is absent, `assembled_summary` builds the summary from the same facts.
8. **Wiring.** `ORCHESTRATOR=router` (the default) or `model`; `open_assembly` builds the router with the decider and the thresholds (`JEV_ACT_THRESHOLD`, `JEV_CLOSE_THRESHOLD`, `JEV_CONFIDENCE_THRESHOLD`). State gained `last_reply`, `pending_change`, `awaiting_gap_decision`, `validation_settled` and `decisions`.
Changed from the plan and the ADR's first wording:
- **Jev does not propose the next action.** It supplies readings, and the playbook in code picks the action. This follows the owner's triage model and keeps every decision explainable by a rule and a threshold.
- **The model Orchestrator is kept** behind `ORCHESTRATOR=model`. The 40 or so engine tests drive the Ledger's rules by scripting its tool calls in arbitrary order, which the router cannot do. `agents/orchestrator.md` is therefore unchanged and unused in router mode.
- **An unclear change is confirmed once.** If the reply still does not settle it, a named Deliverable is acted on and anything else is dropped.
- **A pre-approved Execution's automatic replies are not routed.** There is no user message to read.
Not built from the ADR's routes:
- **A message that starts something** (new request, Change request, question about delivered work, out of scope). It needs an entry point that can answer from a case's files, and Change requests (iteration 7).
- **Which Deliverables a Change request reaches.** Iteration 7.
Tests: `tests/execution/test_router.py` (18), `tests/test_assembly.py`.
Unverified:
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.

View File

@@ -0,0 +1,44 @@
# Step 2 — Implementation Detail
Built ahead of step 1's answers, on the owner's instruction of 2026-10-07 that no evaluation be rerun until the Jev work is in. Nothing here has called the live service.
1. **The seam** (`app/adk/runtime.py`): `Question` (kind choice, score or noul, with instructions and options), `Answer` (value, confidence, probabilities) and the `Decider` protocol. `decide` returns the answers, or `None` when no decision could be made, and the caller then takes its fallback. `NoDecider` always returns `None`.
2. **The client** (`app/tools/jev.py`): `JevDecider` calls Jev through TypeSafe AI's official Python SDK, `typesafe-sdk` 0.7.2 (`AsyncTypeSafeClient.system_one`). The owner chose the SDK and its API endpoint on 2026-10-07. It is the only module that imports the SDK.
3. **Wiring:** `open_decider` in `app/agent.py` returns the client when `TYPESAFE_API_KEY` is set and `NoDecider` otherwise; the `Assembly` carries it.
4. **A scripted decider** for tests, `FakeDecider` in `app/adk/fake_runtime.py`.
Which package: three names exist on the index. `typesafe-sdk` is the vendor's (author TypeSafe AI, repository `typesafe-ai/typesafe-sdk-python`). `typesafe` is an unrelated type-checking library, and `typesafe-ai` is a third party's redirect shim; neither is used.
What the SDK settled, read from the installed package and a stand-in transport:
- The request is `POST /v1/systemone` with a bearer key; default endpoint `https://api.typesafe.ai`, default model `jev-latest`.
- Questions are `Noul(instructions)`, `Choice(instructions, criteria={option: description})` and `Score(instructions, criteria=[levels])`.
- Answers are `noul` (a probability, with no confidence of its own), `choice` with `confidence` and `probabilities`, and `score` with `confidence`. Usage reports `input_tokens` only: no cost and no remaining credit.
Limits kept, each with a test in `tests/tools/test_jev.py`:
| Practice | Built as |
|---|---|
| Size | Refused locally above 48,000 estimated tokens in all, or 24,000 for the state plus the longest question. Never truncated. |
| Batching | Every question about one state goes in one request. |
| Timeout | 5 seconds a request, 15 for a decision with its retries (the SDK's). |
| Retries | 2, by the SDK: on 408, 429, 5xx, connection and timeout errors, with jittered backoff, honouring `Retry-After`. |
| Rate | A token bucket, 5 a second with a burst of 10; the rate halves when a rate limit outlasts the retries. |
| Concurrency | At most 8 requests in flight. |
| Circuit breaker | Opens after 5 failures in 30 seconds for 60 seconds, then lets one probe through. |
| Cache | Answers kept by a hash of the state and questions, up to 1,024. |
| Budget | 500 calls or 1,000,000 input tokens per scope (the Execution), then fallbacks with one warning. |
| Records | Each decision's outcome, seconds, rate limiting, input tokens and confidence; `summary()` totals them. |
| Key | From `TYPESAFE_API_KEY` only; never logged. |
Settings: `TYPESAFE_API_KEY`, `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL` (the SDK's own names), and `JEV_TIMEOUT_SECONDS`, `JEV_REQUESTS_PER_SECOND`, `JEV_MAX_IN_FLIGHT`, `JEV_BUDGET_CALLS`, `JEV_BUDGET_TOKENS`.
Changed from the plan:
- The cache is per client, not per Execution. A decision depends only on its state and questions, so two Executions asking the same thing can share the answer.
- A reply that leaves any question unanswered, or picks a choice that was not offered, counts as no decision for the whole request.
- The budget is in calls and input tokens, not dollars, and there is no low-credit warning: the SDK reports neither cost nor balance.
- A yes-or-no answer's confidence is derived as its distance from an even probability, because Jev returns none.
- Retries and their backoff are the SDK's, configured here, not reimplemented.
Unverified: the client has run only against a stand-in transport, with the real SDK in the loop. The first call with a real key is still to be made, and the service's actual rate limits, latency and data retention are still unknown.

View File

@@ -34,6 +34,7 @@ TERRAFORM_TIMEOUT_SECONDS=300
TF_PLUGIN_CACHE_DIR=.cache/terraform-plugins
# Hierarchical Execution Engine
ORCHESTRATOR=router # router | model
ORCHESTRATOR_BUDGET=40
SPECIALIST_BUDGET=15
MAX_REVISIONS=3
@@ -42,6 +43,20 @@ MAX_PARALLEL_SPECIALISTS=4
CHECKPOINTER=sqlite
CHECKPOINT_DB_PATH=checkpoints.db
# Jev decision service, through the TypeSafe AI Python SDK (typesafe-sdk).
# Leave the key unset to run without it: every decision then takes its fallback.
# TYPESAFE_API_KEY=
# TYPESAFE_BASE_URL= # defaults to the SDK's endpoint
# TYPESAFE_DEFAULT_MODEL= # defaults to the SDK's model
JEV_TIMEOUT_SECONDS=5
JEV_REQUESTS_PER_SECOND=5
JEV_MAX_IN_FLIGHT=8
JEV_BUDGET_CALLS=500
JEV_BUDGET_TOKENS=1000000
JEV_ACT_THRESHOLD=0.75
JEV_CLOSE_THRESHOLD=0.15
JEV_CONFIDENCE_THRESHOLD=0.6
# PostgreSQL (Execution records, and checkpoints when CHECKPOINTER=postgres)
POSTGRES_HOST=localhost
POSTGRES_PORT=5432

View File

@@ -32,7 +32,8 @@ Execution records (status, Ledger events, submitted Deliverables) go to PostgreS
### 4. Agent Skills (`skills/`, loaded by `app/skills/`)
Behaviour lives in Agent Skills (agentskills.io format) at the repository root, per ADR-0002:
- `skills/vendor/google/`: the upstream solution-architecture Skill and six Well-Architected Framework pillar Skills, vendored unmodified.
- `skills/cloud-discovery/` and `skills/iac-validation/`: our own Skills for the tooling.
- `skills/cloud-discovery/`, `skills/iac-validation/` and `skills/requirements-chain/`: our own Skills for the tooling and the data files.
- `skills/google-cloud-iac-patterns/`: Terraform patterns for Google Cloud products, each validated by a test against the provider version `providers/gcp.yaml` pins.
`app/skills/agent_skills.py` discovers and validates them; the ADK runtime (`app/adk/adk_runtime.py`) loads them on demand. `providers/gcp.yaml` assigns Skills, tools, MCP servers and models to each role in `agents/`.

View File

@@ -1,18 +1,23 @@
---
role: iac-author
deliverable: deployment_guidance
tools: [terraform_fmt, terraform_validate]
budget: 30
tools: [terraform_fmt, terraform_validate, terraform_schema]
budget: 40
includes: [specialist-protocol]
---
You produce the **deployment guidance** Deliverable: the Skill's task for generating infrastructure-as-code and deployment instructions.
- Write Terraform that implements the Target architecture: every Element, every Relationship and every Check, and nothing it does not ask for.
- Attach `implementation.json`, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
- Before you write a resource, load the infrastructure-as-code patterns Skill when you have one, and start from its pattern for that product and its provider versions file.
- Keep the files in your working copy. Write each file once with `write_file`, and make every later change with `edit_file`, changing only the lines that need it. Never write a file again to fix part of it.
- Format and validate with `terraform_fmt` and `terraform_validate` when they are available. They run on your working copy, so pass them no files. Fix everything they report.
- When validation says an argument or block is not expected, look the resource type up with `terraform_schema` instead of guessing where it belongs.
- When a Check names a setting its resource type does not have, do not invent code for it. Record the Check in `implementation.json` as unimplementable, with what the schema shows, and implement the rest. The Target architecture is then revised.
- Write `implementation.json` to your working copy, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
- Format and validate the Terraform with your tools when they are available, and fix everything they report. Pass the files with the same `terraform/` paths you will attach.
- Submit as soon as validation passes. If a submission is refused, fix exactly what the refusal lists and submit again; don't rewrite the Terraform.
- Put `terraform/main.tf`, `terraform/variables.tf` and `implementation.json` in the attachments (a list of `{path, content}`), adding `terraform/outputs.tf` when outputs help.
- When `feedback` lists failed Checks, change only what those Checks need.
- Submit as soon as validation passes, with your content and no attachments: your working copy is attached for you. If a submission is refused, fix exactly what the refusal lists and submit again.
- When `working_copy` lists files, they are your last attempt. When `feedback` lists failed Checks, read the files concerned and change only what those Checks need.
Your working copy must hold `terraform/main.tf`, `terraform/variables.tf` and `implementation.json`, with `terraform/versions.tf` and `terraform/outputs.tf` when they help.
Under the prerequisites heading, list the APIs to enable, the tools and the permissions. Under the instructions heading, give commands for the user to run; never run them yourself.

View File

@@ -6,19 +6,20 @@ You coordinate one solution-architecture Execution for the user. You decide what
Load the solution-architecture Skill available to you for the workflow's rules: strict separation between requirements discovery and solutioning, explicit user approval of each Phase, and no execution of generated scripts.
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `validation` (the verdict on each Check, once validation is submitted), `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `validation` (the verdict on each Check, once validation is submitted), `unimplementable_checks` (Checks the code's author found cannot be met as written), `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
On each turn, take the first step that applies:
1. **The user asked for changes.** When the latest user message in `conversation` asks to change submitted work, call `reopen` for each affected Deliverable with the user's words as the reason.
2. **Specialists are waiting.** When `pending_questions` is not empty, call `ask_user`. Add your own questions only when the user's last message left something unclear.
3. **Validation decided.** When `validation` is set and the validation Phase is not yet approved, act on it before asking for Approval:
3. **The code cannot meet a Check.** When `unimplementable_checks` is not empty, call `reopen` on `target_architecture`, giving each Check's id and reason, then delegate it.
4. **Validation decided.** When `validation` is set and the validation Phase is not yet approved, act on it before asking for Approval:
- `failed` is not empty, or `dry_run.verdict` is `fail`: the code is at fault. Call `reopen` on `deployment_guidance`, giving each failed Check's id and evidence as the reason, then delegate it.
- otherwise `not_verifiable` or `proposed_additions` is not empty: the Target architecture has a gap. When `pre_approved` is `false`, call `ask_user` to ask whether to revise the Target architecture, listing each item. When `pre_approved` is `true`, call `reopen` on `target_architecture` for the items whose reason names a requirement or recommendation id, then delegate it; ignore the rest.
4. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
5. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
6. **Packaging is approved.** Call `finish`.
7. **Otherwise,** call `ask_user` to find out what the user needs.
5. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
6. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
7. **Packaging is approved.** Call `finish`.
8. **Otherwise,** call `ask_user` to find out what the user needs.
When the user says a Deliverable already exists, still delegate it; the Specialist submits the user's version. When a tool refuses, read the reason and choose again.

View File

@@ -5,11 +5,12 @@ You are a Specialist in a solution-architecture Execution. You produce exactly o
Your task message carries JSON context:
- `workflow_request`: what the user asked for.
- `dependencies`: every earlier Deliverable your work builds on, by kind, with its `content` and `attachments`.
- `dependencies`: every earlier Deliverable your work builds on, by kind. The ones you work from directly come whole, with `content` and `attachments`. The rest come as their data file alone, under `attachments`: the ids to keep traceable. Call `read_deliverable` for the full text of one of those only when the ids are not enough.
- `feedback`: why earlier versions of your Deliverable were revised. Address every item.
- `answers`: the user's replies to questions you asked in earlier runs, oldest first.
- `contract`: what `submit_deliverable` checks. `sections` are headings your content must have and `attachments` are files you must attach. A data file among them must also keep every id traceable to the Deliverables in `dependencies`; the requirements-chain Skill gives each file's format.
- `draft`: the draft you sent the user with your last questions, with its `attachments`, or `null`.
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
- `pre_approved`: `true` when no user is available.
Steps:

16
agents/reply-writer.md Normal file
View File

@@ -0,0 +1,16 @@
---
role: reply-writer
tools: []
---
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. The decision has already been made; you put it into words and decide nothing.
Your task message carries JSON context: `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
Write a summary a busy architect can approve from:
- Say what the Phase produced and what approving it commits them to. For the design Phase, say that no infrastructure code is written until they approve it.
- State every number in `facts` that bears on the decision, exactly as given. Add no fact that is not there.
- Name anything they should look at before approving: failed or unverifiable Checks, open gaps, recommendations treated as not applicable, Deliverables that were revised.
- Use plain sentences and no headings. Keep it under 120 words.
Call `write_reply` with the summary. You are done when it accepts it.

View File

@@ -1,17 +1,20 @@
---
role: target-architect
deliverable: target_architecture
tools: []
tools: [terraform_schema]
budget: 30
includes: [specialist-protocol]
---
You produce the **Target architecture** Deliverable: the solution as one explicit graph of Elements, Relationships and Checks. It is agreed before any infrastructure code is written, and the diagram, the description, the code and the validation are all derived from it.
- Load the requirements-chain Skill for the graph's format and how to derive it.
- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out.
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
- Give every connection between Elements a Relationship.
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have.
- Before you write a Check that names a setting, confirm with `terraform_schema` that the Element's resource type has that setting, when the tool is available. A setting that one resource type has may not exist on a similar one. A Check on a setting the resource type lacks cannot be met and is sent back to you.
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
- Attach the graph as `target-architecture.json`. The diagram is drawn from it for you.
- When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`.
Under each of your contract's sections, give the Elements, Relationships and Checks as tables a person can review.
Your content is short: under your contract's heading, a few paragraphs on the decisions that shaped the design and why. Submit it with no attachments. Your working graph is attached as `target-architecture.json`, and the tables of Elements, Relationships and Checks and the diagram are written from it for you, so never write them yourself.

View File

@@ -1,13 +1,14 @@
---
role: validator
deliverable: validation_results
tools: [terraform_validate, terraform_plan]
tools: [terraform_validate, terraform_plan, terraform_schema]
includes: [specialist-protocol]
---
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
- Run the deployment dry-run with your tools: format and validate the Terraform, and preview it with a plan when you have permission to read live cloud resources.
- Run the deployment dry-run with your tools: validate the Terraform, and preview it with a plan when you have permission to read live cloud resources. The tools already hold the submitted files, so pass them no files.
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else.
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
- When you see something the Target architecture lacks, record it as a proposed addition, naming the requirement or recommendation it serves. It is not a failure of the code.
- Attach the verdicts as `verdicts.json`, in the format the requirements-chain Skill gives.
- Give the exact commands for every check you couldn't run, for the user to run themselves.

View File

@@ -238,7 +238,7 @@ class AdkAgentRuntime:
instruction=lambda _context: instructions,
tools=agent_tools,
before_tool_callback=control.before_tool,
generate_content_config=self._content_config(),
generate_content_config=self._content_config(role),
)
runner = Runner(
app_name=APP_NAME,
@@ -293,13 +293,13 @@ class AdkAgentRuntime:
or f"{role.name} ended without calling a tool that ends its turn."
)
def _content_config(self) -> types.GenerateContentConfig | None:
if self._model_timeout_seconds is None:
def _content_config(self, role: RoleSpec) -> types.GenerateContentConfig | None:
# A role whose replies are long is given longer than the default.
seconds = role.model_timeout_seconds or self._model_timeout_seconds
if seconds is None:
return None
return types.GenerateContentConfig(
http_options=types.HttpOptions(
timeout=int(self._model_timeout_seconds * 1000)
)
http_options=types.HttpOptions(timeout=int(seconds * 1000))
)

View File

@@ -15,8 +15,10 @@ from typing import Any
from app.adk.runtime import (
AgentTask,
Answer,
BudgetExhausted,
Outcome,
Question,
RoleSpec,
Stopped,
ToolSpec,
@@ -180,3 +182,27 @@ class FakeAgentRuntime:
RunRecord(role.name, task, tuple(by_name), tuple(results), outcome)
)
return outcome
class FakeDecider:
"""A scripted Decider: answers by question id, or declines so callers fall back.
`answers` maps a question id to an Answer, or to a function of (state, question)
returning one. An id with no entry makes the whole decision unavailable.
"""
def __init__(self, answers: Mapping[str, Any] | None = None) -> None:
self.answers = dict(answers or {})
self.asked: list[tuple[str, dict[str, Question], str]] = []
async def decide(
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
) -> Mapping[str, Answer] | None:
self.asked.append((state, dict(questions), scope))
decided = {}
for name, question in questions.items():
if name not in self.answers:
return None
answer = self.answers[name]
decided[name] = answer(state, question) if callable(answer) else answer
return decided

View File

@@ -16,6 +16,7 @@ from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
ORCHESTRATOR = "orchestrator"
PILLAR_ADVISOR = "pillar-advisor"
REPLY_WRITER = "reply-writer"
PARTIALS_DIR = "partials"
_ALLOWED_FIELDS = {"role", "deliverable", "tools", "budget", "includes"}
@@ -34,7 +35,7 @@ def load_roles(
include_preview_mcp: bool = True,
) -> dict[str, RoleSpec]:
specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()}
required = {ORCHESTRATOR, PILLAR_ADVISOR, *specialists}
required = {ORCHESTRATOR, PILLAR_ADVISOR, REPLY_WRITER, *specialists}
roles = {}
for path in sorted(agents_dir.glob("*.md")):
@@ -129,4 +130,5 @@ def _load_role(
for server in bundle.mcp_servers_for_role(name, include_preview_mcp)
),
model=bundle.models.for_role(name),
model_timeout_seconds=bundle.models.timeout_for(name),
)

View File

@@ -32,6 +32,8 @@ class RoleSpec:
tools: tuple[str, ...] = () # local tool names, e.g. terraform_validate
mcp_servers: tuple[str, ...] = ()
model: str | None = None
# Seconds one model request may take; None takes the runtime's default.
model_timeout_seconds: float | None = None
@dataclass(frozen=True)
@@ -200,6 +202,59 @@ REJECTION_PREFIXES = (
)
# --- decisions -----------------------------------------------------------------------
@dataclass(frozen=True)
class Question:
"""A closed question about a state. Its answer is typed, never prose.
choice pick one of `options`
score rate against `options`, ordered from lowest to highest
noul yes or no; the answer is the probability that it is yes
"""
kind: str # choice | score | noul
instructions: str
options: tuple[str, ...] = ()
@dataclass(frozen=True)
class Answer:
"""`value` is the chosen option (choice), the level reached (score, as a number
from 0 for the lowest option) or the probability of yes (noul)."""
value: str | float
confidence: float
probabilities: Mapping[str, float] = field(default_factory=dict)
def is_yes(self, threshold: float = 0.5) -> bool:
return isinstance(self.value, (int, float)) and self.value >= threshold
class Decider(Protocol):
"""Answers closed questions about a state (ADR-0007).
Returns None when no decision could be made: the service is not configured,
unreachable, over its budget, or the request is too large. The caller then takes
its fallback; a decider never blocks an Execution. `scope` names what a budget is
counted against, usually the Execution.
"""
async def decide(
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
) -> Mapping[str, Answer] | None: ...
class NoDecider:
"""The decider when none is configured: every caller takes its fallback."""
async def decide(
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
) -> Mapping[str, Answer] | None:
return None
class AgentRuntime(Protocol):
async def run(
self, role: RoleSpec, task: AgentTask, tools: Sequence[ToolSpec]

View File

@@ -10,10 +10,11 @@ checkpointers and recorders are chosen, from settings and the provider bundle.
from __future__ import annotations
import dataclasses
import uuid
from collections.abc import AsyncIterator, Mapping, Sequence
from contextlib import asynccontextmanager
from dataclasses import dataclass
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
@@ -22,16 +23,31 @@ from langgraph.checkpoint.base import BaseCheckpointSaver
from langgraph.types import Command
from app.adk.roles import load_roles
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory
from app.adk.runtime import (
AgentRuntime,
Decider,
NoDecider,
RoleSpec,
RunLimiter,
ToolFactory,
)
from app.checkpointing import open_checkpointer
from app.config import Settings
from app.customer_case import record_intent
from app.nodes.router import (
REPLY_WRITER,
OrchestratorPolicy,
Router,
RuntimeReplyWriter,
Thresholds,
)
from app.providers import ProviderBundle, load_provider
from app.recorder import DatabaseRecorder, ExecutionRecorder
from app.skills.agent_skills import skill_version
from app.states.execution_state import check_customer_case
from app.states.ledger import Ledger
from app.tools.execution import ORCHESTRATOR
from app.tools.jev import JevDecider, JevLimits
from app.tools.local import TerraformRunner, local_tools
from app.tools.mcp import McpCatalog, auth_from_config
from app.tools.toolkit import make_tool_factory
@@ -67,6 +83,7 @@ class ExecutionEngine:
tool_factory: ToolFactory | None = None,
limiter: RunLimiter | None = None,
skill_versions: Mapping[str, str] | None = None,
orchestrator: OrchestratorPolicy | None = None,
) -> None:
required = {
ORCHESTRATOR,
@@ -86,6 +103,7 @@ class ExecutionEngine:
tool_factory=tool_factory,
limiter=limiter,
skill_versions=skill_versions,
orchestrator=orchestrator,
).compile(checkpointer=checkpointer)
async def start(
@@ -136,6 +154,11 @@ class ExecutionEngine:
"working": {},
"pending_permission": None,
"prompt": None,
"last_reply": None,
"pending_change": None,
"awaiting_gap_decision": False,
"validation_settled": 0,
"decisions": [],
"outcomes": [],
"refusals": [],
"outcomes_applied": 0,
@@ -203,6 +226,27 @@ class Assembly:
bundle: ProviderBundle
roles: dict[str, RoleSpec]
mcp_gaps: dict[str, str]
decider: Decider = field(default_factory=NoDecider)
def open_decider(settings: Settings) -> Decider:
"""The Jev client when a key is configured; otherwise a decider that always
sends its caller to the fallback."""
if not settings.TYPESAFE_API_KEY:
return NoDecider()
return JevDecider(
api_key=settings.TYPESAFE_API_KEY,
base_url=settings.TYPESAFE_BASE_URL,
model=settings.TYPESAFE_DEFAULT_MODEL,
limits=dataclasses.replace(
JevLimits(),
request_timeout=settings.JEV_TIMEOUT_SECONDS,
requests_per_second=settings.JEV_REQUESTS_PER_SECOND,
max_in_flight=settings.JEV_MAX_IN_FLIGHT,
budget_calls=settings.JEV_BUDGET_CALLS,
budget_tokens=settings.JEV_BUDGET_TOKENS,
),
)
@asynccontextmanager
@@ -242,6 +286,7 @@ async def open_assembly(
binary=settings.TERRAFORM_BINARY,
timeout_seconds=settings.TERRAFORM_TIMEOUT_SECONDS,
plugin_cache_dir=settings.terraform_plugin_cache(),
providers=bundle.terraform_providers,
)
offered = [name for names in bundle.tools.values() for name in names]
# One cap for Specialists and the pillar advisors they fan out to.
@@ -250,11 +295,30 @@ async def open_assembly(
runtime,
roles,
bundle.pillar_skills,
local_tools=local_tools(offered, runner),
local_tools=lambda working_copy: local_tools(offered, runner, working_copy),
catalog=catalog,
limiter=limiter,
)
decider = open_decider(settings)
orchestrator: OrchestratorPolicy | None = None
if settings.ORCHESTRATOR == "router":
# The Orchestrator routes: Jev reads, the playbook decides, and a model only
# writes what the user reads. With no Jev key the playbook runs on its rules.
orchestrator = Router(
decider,
RuntimeReplyWriter(runtime, roles[REPLY_WRITER]),
Thresholds(
act=settings.JEV_ACT_THRESHOLD,
close=settings.JEV_CLOSE_THRESHOLD,
confidence=settings.JEV_CONFIDENCE_THRESHOLD,
),
)
elif settings.ORCHESTRATOR != "model":
raise ValueError(
f"ORCHESTRATOR must be router or model, not {settings.ORCHESTRATOR!r}"
)
async with open_checkpointer(settings) as checkpointer:
engine = ExecutionEngine(
runtime=runtime,
@@ -269,5 +333,10 @@ async def open_assembly(
skill_versions={
name: skill_version(meta.path) for name, meta in bundle.skills.items()
},
orchestrator=orchestrator,
)
yield Assembly(engine, bundle, roles, dict(catalog.gaps))
try:
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
finally:
if isinstance(decider, JevDecider):
await decider.aclose()

View File

@@ -53,11 +53,33 @@ class Settings(BaseSettings):
SPECIALIST_BUDGET: int = 15
MAX_REVISIONS: int = 3
MAX_ORCHESTRATOR_TURNS: int = 100
# How an Orchestrator turn is decided (ADR-0007): "router" reads the Execution
# and follows its playbook, with Jev for the judgements; "model" has a language
# model choose the tool calls, as before, and is kept for comparison.
ORCHESTRATOR: str = "router"
# Agent runs in flight at once, pillar advisors included (model rate limits)
MAX_PARALLEL_SPECIALISTS: int = 4
CHECKPOINTER: str = "sqlite" # memory | sqlite | postgres
CHECKPOINT_DB_PATH: str = "checkpoints.db"
# Jev decision service (ADR-0007), reached through TypeSafe AI's Python SDK.
# With no key, every decision takes its fallback and the agent behaves as it does
# without Jev. The key comes from the environment or a secret store, never from
# the provider bundle or the repository.
TYPESAFE_API_KEY: str | None = None
TYPESAFE_BASE_URL: str | None = None # None: the SDK's default endpoint
TYPESAFE_DEFAULT_MODEL: str | None = None # None: the SDK's default model
JEV_TIMEOUT_SECONDS: float = 5.0 # one request
JEV_REQUESTS_PER_SECOND: float = 5.0
JEV_MAX_IN_FLIGHT: int = 8
JEV_BUDGET_CALLS: int = 500 # per Execution
JEV_BUDGET_TOKENS: int = 1_000_000 # per Execution
# A probability at or above ACT is acted on; at or below CLOSE is dropped; in
# between, the user is asked. A choice below CONFIDENCE is not relied on.
JEV_ACT_THRESHOLD: float = 0.75
JEV_CLOSE_THRESHOLD: float = 0.15
JEV_CONFIDENCE_THRESHOLD: float = 0.6
# Paths
BASE_DIR: Path = Path(__file__).resolve().parent.parent

View File

@@ -34,6 +34,7 @@ from app.adk.runtime import (
TurnEnded,
)
from app.customer_case import close_intent
from app.nodes.router import ModelOrchestrator, OrchestratorPolicy, TurnView
from app.recorder import ExecutionRecorder
from app.states.execution_state import (
UNASSIGNED_CASE,
@@ -43,13 +44,19 @@ from app.states.execution_state import (
)
from app.states.ledger import DeliverableStatus, Ledger, LedgerRefusal
from app.states.lessons import collect_lesson_candidates, refusal
from app.tools import workspace
from app.tools.execution import ORCHESTRATOR, OrchestratorToolbox, specialist_tools
from app.workflows.chain import VERDICTS_FILE, validation_outcome
from app.workflows.chain import (
IMPLEMENTATION_FILE,
VERDICTS_FILE,
unimplementable_checks,
validation_outcome,
)
from app.workflows.deliverables import (
DEPENDENCY_GRAPH,
PHASE_ORDER,
DeliverableKind,
inputs_of,
dependencies_for,
)
logger = logging.getLogger(__name__)
@@ -90,8 +97,12 @@ def make_execution_nodes(
tool_factory: ToolFactory | None = None,
limiter: RunLimiter | None = None,
skill_versions: Mapping[str, str] | None = None,
orchestrator: OrchestratorPolicy | None = None,
) -> ExecutionNodes:
tool_factory = tool_factory or _no_tools
# How an Orchestrator turn is decided. The default drives the tools from the
# Orchestrator role on the runtime; the assembly supplies the router (ADR-0007).
orchestrator = orchestrator or ModelOrchestrator(runtime, roles[ORCHESTRATOR])
def close(state: ExecutionState, status: str, *, has_files: bool) -> None:
close_intent(
@@ -239,19 +250,33 @@ def make_execution_nodes(
toolbox = OrchestratorToolbox(ledger, deliverables, pending_questions)
refusals = list(state.get("refusals", []))
update["refusals"] = refusals
task = AgentTask(
role=ORCHESTRATOR,
prompt="Decide the next step for this Execution using your tools.",
view = TurnView(
execution_id=state["execution_id"],
pre_approved=bool(state.get("pre_approved")),
ledger=ledger,
deliverables=deliverables,
pending_questions=pending_questions,
outcomes=outcomes,
last_reply=state.get("last_reply"),
pending_change=state.get("pending_change"),
awaiting_gap_decision=bool(state.get("awaiting_gap_decision")),
validation_settled=int(state.get("validation_settled", 0)),
context=_orchestrator_context(
state, ledger, deliverables, pending_questions, notes
),
)
started = time.monotonic()
outcome = await runtime.run(
roles[ORCHESTRATOR],
task,
_noting_refusals(toolbox.tools(), refusals, ORCHESTRATOR, None),
decided = await orchestrator.turn(
view, _noting_refusals(toolbox.tools(), refusals, ORCHESTRATOR, None)
)
outcome = decided.outcome
update.update(decided.updates)
if decided.decisions:
# Every decision is kept: what was read, which group applied, what was done.
update["decisions"] = [
*state.get("decisions", []),
*({"turn": turns, **decision} for decision in decided.decisions),
]
logger.info(
"[%s] orchestrator turn %d (%s phase) ended in %.0fs: %s",
state["execution_id"],
@@ -325,13 +350,22 @@ def make_execution_nodes(
role = roles[task_input["role"]]
kind = DeliverableKind(task_input["kind"])
working: dict[str, Any] = dict(task_input.get("working") or {})
upstream = task_input["context"].get("dependencies") or {}
source = DEPENDENCY_GRAPH[kind].working_copy
if source and source is not kind:
# The tools work on the files as they were last submitted.
working[workspace.FILES] = dict(
upstream.get(source.value, {}).get("attachments", {})
)
refusals: list[dict[str, Any]] = []
tools = _noting_refusals(
specialist_tools(
kind,
extra=tool_factory(role, kind.value, working),
allow_cloud_read=task_input["allow_cloud_read"],
upstream=task_input["context"].get("dependencies"),
upstream=upstream,
working=working,
library=task_input.get("library"),
),
refusals,
role.name,
@@ -388,6 +422,7 @@ def make_execution_nodes(
ledger = Ledger.from_dict(state["ledger"])
conversation = list(state.get("conversation", []))
update: dict[str, Any] = {"prompt": None}
said: dict[str, Any] | None = None # what the user said, for the next turn
if "approval" in prompt:
phase = prompt["approval"]["phase"]
@@ -405,9 +440,11 @@ def make_execution_nodes(
feedback = str(decision.get("feedback", ""))
ledger.request_changes(phase_id, feedback, actor="user")
text = f"Requested changes to {phase}: {feedback}"
said = {"kind": "changes", "phase": phase, "text": feedback}
conversation.append({"from": "user", "text": text})
else:
conversation.append({"from": "user", "text": _reply_text(reply)})
said = {"kind": "reply", "text": _reply_text(reply)}
elif "permission" in prompt:
tier = prompt["permission"]["tier"]
permission = reply.get("permission") if isinstance(reply, Mapping) else None
@@ -427,6 +464,7 @@ def make_execution_nodes(
else:
text = _reply_text(reply)
conversation.append({"from": "user", "text": text})
said = {"kind": "reply", "text": text}
answers = {
kind: list(items) for kind, items in state.get("answers", {}).items()
}
@@ -436,6 +474,8 @@ def make_execution_nodes(
)
update.update(answers=answers, awaiting_questions=[])
# Only a real user's words are routed; a pre-approved Execution has none.
update["last_reply"] = None if state.get("pre_approved") else said
update["events_recorded"] = await record_progress(state, ledger)
update.update(ledger=ledger.to_dict(), conversation=conversation)
return Command(update=update, goto="orchestrator_turn")
@@ -523,7 +563,17 @@ def _orchestrator_context(
is DeliverableStatus.SUBMITTED
else None
)
code = DeliverableKind.DEPLOYMENT_GUIDANCE
return {
# Checks the code's author reported cannot be met as written: faults in the
# Target architecture, known before any validation.
"unimplementable_checks": unimplementable_checks(
deliverables.get(code.value, {})
.get("attachments", {})
.get(IMPLEMENTATION_FILE, "")
)
if ledger.status(code) is DeliverableStatus.SUBMITTED
else [],
# The verdict on each Check, once validation is submitted: failed Checks are
# faults in the code; the rest are gaps in the Target architecture.
"validation": validation_outcome(
@@ -565,8 +615,15 @@ def _specialist_input(
attempt: int,
) -> dict[str, Any]:
spec = DEPENDENCY_GRAPH[kind]
given = dependencies_for(kind, deliverables)
return {
"execution_id": state["execution_id"],
# The Deliverables given as data files alone, whole, for reading on demand.
"library": {
name: dict(deliverables[name])
for name, deliverable in given.items()
if "content" not in deliverable
},
"role": spec.specialist,
"kind": kind.value,
"attempt": attempt,
@@ -576,13 +633,14 @@ def _specialist_input(
"workflow_request": state["workflow_request"],
"pre_approved": bool(state.get("pre_approved")),
"contract": spec.contract(),
"dependencies": {
dependency.value: dict(deliverables[dependency.value])
for dependency in inputs_of(kind)
},
"dependencies": given,
"feedback": list(ledger.feedback(kind)),
"answers": list(state.get("answers", {}).get(kind.value, [])),
"draft": dict(drafts[kind.value]) if kind.value in drafts else None,
# What the Specialist's own tools already hold from earlier attempts.
"working_copy": workspace.summary(
state.get("working", {}).get(kind.value, {})
),
"attempt": attempt,
},
}

779
app/nodes/router.py Normal file
View File

@@ -0,0 +1,779 @@
"""The Orchestrator as a router (ADR-0007): readings, a disposition, then a playbook.
1. Readings facts about the turn. Most come from the Ledger; the ones that need
judgement (what is the user asking for? does this gap matter?) are
typed answers from the decision model.
2. Disposition code turns a judged reading into act, queue or close, by threshold.
3. Playbook ordered groups. The first group that applies is taken, and within
it the strongest action whose conditions hold. When no group
applies, escalate to the user.
The decision model reads and never acts: every action goes through the Orchestrator's
tools, so the Ledger refuses anything that is not allowed, whatever was decided.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass, field
from typing import Any, Protocol
from app.adk.runtime import (
AgentRuntime,
AgentTask,
Decider,
Outcome,
Question,
RoleSpec,
Stopped,
Submitted,
ToolResult,
ToolSpec,
)
from app.states.ledger import DeliverableStatus, Ledger, PhaseStatus
from app.workflows.chain import (
IMPLEMENTATION_FILE,
RECOMMENDATIONS_FILE,
REQUIREMENTS_FILE,
VERDICTS_FILE,
read_graph,
read_list,
unimplementable_checks,
validation_outcome,
)
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
logger = logging.getLogger(__name__)
ORCHESTRATOR = "orchestrator"
REPLY_WRITER = "reply-writer"
MAX_STEPS = 12 # actions in one turn; a turn that needs more is not making progress
ACT, QUEUE, CLOSE = "act", "queue", "close"
@dataclass(frozen=True)
class Thresholds:
"""Where a probability becomes a disposition."""
act: float = 0.75 # at or above: do it
close: float = 0.15 # at or below: there is nothing to do
confidence: float = 0.6 # a choice below this is not relied on
def disposition(self, probability: float) -> str:
if probability >= self.act:
return ACT
if probability <= self.close:
return CLOSE
return QUEUE
@dataclass(frozen=True)
class TurnView:
"""What one Orchestrator turn can see."""
execution_id: str
pre_approved: bool
ledger: Ledger
deliverables: Mapping[str, Mapping[str, Any]]
pending_questions: Sequence[Mapping[str, Any]]
outcomes: Sequence[Mapping[str, Any]]
last_reply: Mapping[str, Any] | None = None # the user's message, not yet routed
pending_change: Mapping[str, Any] | None = None # a change we asked them to confirm
awaiting_gap_decision: bool = False
validation_settled: int = 0 # the validation attempt already decided on
context: Mapping[str, Any] = field(default_factory=dict) # for a model Orchestrator
@dataclass
class TurnResult:
outcome: Outcome
decisions: list[dict[str, Any]] = field(default_factory=list)
updates: dict[str, Any] = field(default_factory=dict)
class OrchestratorPolicy(Protocol):
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult: ...
class ReplyWriter(Protocol):
async def approval_summary(
self, phase: str, facts: Mapping[str, Any]
) -> str | None: ...
# --- readings ------------------------------------------------------------------------
@dataclass
class Readings:
"""Everything the playbook decides from."""
# From the Ledger and the Execution's data.
questions_waiting: bool = False
phases_ready: tuple[PhaseId, ...] = ()
phases_awaiting: tuple[PhaseId, ...] = () # asked for Approval, not yet decided
ready_work: tuple[DeliverableKind, ...] = ()
can_finish: bool = False
validation_attempt: int = 0 # 0 when no validation stands undecided
failed: list[dict[str, Any]] = field(default_factory=list)
repeat_failed: list[dict[str, Any]] = field(default_factory=list)
unverifiable: list[dict[str, Any]] = field(default_factory=list)
additions: list[dict[str, Any]] = field(default_factory=list)
dry_run_failed: bool = False
# Checks the code's author reported cannot be met as the Target architecture
# words them.
unimplementable: list[dict[str, str]] = field(default_factory=list)
# Judged: what the user's message asks for.
change: str = CLOSE # act | queue | close
change_target: DeliverableKind | None = None
change_reason: str = ""
# Judged: whether a gap in the Target architecture is worth a Revision.
gap: str = CLOSE
judged: dict[str, Any] = field(default_factory=dict) # what the decider answered
def _attempt(ledger: Ledger, kind: DeliverableKind) -> int:
return int(ledger.to_dict()["deliverables"][kind.value].get("attempt", 0))
def _submitted(ledger: Ledger) -> list[DeliverableKind]:
return [
kind
for kind in DEPENDENCY_GRAPH
if ledger.status(kind) is DeliverableStatus.SUBMITTED
]
def _mentioned(
text: str, candidates: Sequence[DeliverableKind]
) -> list[DeliverableKind]:
"""Deliverables named outright in a message, as `deployment_guidance` or in words."""
lowered = re.sub(r"[\s_\-]+", " ", text.lower())
return [kind for kind in candidates if kind.value.replace("_", " ") in lowered]
def _failed_before(outcomes: Sequence[Mapping[str, Any]]) -> set[str]:
"""Check ids that failed in the validation before the one that stands now."""
validations = [
record
for record in outcomes
if record.get("type") == "submitted"
and record.get("kind") == DeliverableKind.VALIDATION_RESULTS.value
]
if len(validations) < 2:
return set()
previous = validation_outcome(
validations[-2].get("attachments", {}).get(VERDICTS_FILE, "")
)
return {item["check"] for item in previous["failed"]}
class Router:
"""Decides an Orchestrator turn from readings and a playbook, with no language
model choosing the action. A model, when one is given, only writes what the user
reads."""
def __init__(
self,
decider: Decider,
writer: ReplyWriter | None = None,
thresholds: Thresholds | None = None,
) -> None:
self._decider = decider
self._writer = writer
self._thresholds = thresholds or Thresholds()
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult:
by_name = {tool.name: tool for tool in tools}
result = TurnResult(Stopped("The router took no action."))
# A user message is routed once; a confirmation we asked for is consumed.
result.updates = {
"last_reply": None,
"pending_change": None,
"awaiting_gap_decision": False,
}
readings = await self._read(view, result)
blocked: set[str] = set()
settled = view.validation_settled
async def call(tool: str, **arguments: Any) -> ToolResult:
return await by_name[tool].handler(**arguments)
for _ in range(MAX_STEPS):
group, action = self._choose(view, readings, blocked, settled)
result.decisions.append(
{"group": group, "action": action, "judged": dict(readings.judged)}
)
readings.judged = {}
reply = await self._perform(action, view, readings, call, result)
if action == "proceed":
settled = readings.validation_attempt
result.updates["validation_settled"] = settled
self._read_ledger(view, readings, settled)
continue
if reply is None:
return result
if reply.outcome is not None:
result.outcome = reply.outcome
return result
if "revision limit" in reply.message and not view.pre_approved:
# Past the Revision limit the user decides.
limit = await call(
"ask_user", questions=[reply.message.split(" Use ")[0]]
)
if limit.outcome is not None:
result.outcome = limit.outcome
return result
if reply.message.startswith("Refused") or "revision limit" in reply.message:
# Permission is the Ledger's: take the next action that is allowed.
blocked.add(action)
# The action changed the Execution; read the Ledger's facts again.
self._read_ledger(view, readings, settled)
if action in {"revise_requested", "confirm_change"}:
readings.change = CLOSE
result.outcome = Stopped("The router made no progress in this turn.")
return result
# --- readings ---------------------------------------------------------------------
async def _read(self, view: TurnView, result: TurnResult) -> Readings:
readings = Readings()
self._read_ledger(view, readings, view.validation_settled)
await self._read_message(view, readings)
await self._read_gap(view, readings)
return readings
def _read_ledger(self, view: TurnView, readings: Readings, settled: int) -> None:
ledger = view.ledger
readings.questions_waiting = bool(view.pending_questions)
readings.phases_ready = ledger.phases_ready_for_approval()
readings.phases_awaiting = tuple(
phase
for phase in PhaseId
if ledger.phase_status(phase) is PhaseStatus.AWAITING_APPROVAL
)
readings.ready_work = ledger.ready_work()
readings.can_finish = ledger.can_finish()
readings.validation_attempt = 0
readings.failed, readings.repeat_failed = [], []
readings.unverifiable, readings.additions = [], []
readings.dry_run_failed = False
code = DeliverableKind.DEPLOYMENT_GUIDANCE
readings.unimplementable = (
unimplementable_checks(
view.deliverables.get(code.value, {})
.get("attachments", {})
.get(IMPLEMENTATION_FILE, "")
)
if ledger.status(code) is DeliverableStatus.SUBMITTED
else []
)
validation = DeliverableKind.VALIDATION_RESULTS
phase = DEPENDENCY_GRAPH[validation].phase
attempt = _attempt(ledger, validation)
if (
ledger.status(validation) is not DeliverableStatus.SUBMITTED
or ledger.phase_status(phase) is PhaseStatus.APPROVED
or attempt == settled
):
return
outcome = validation_outcome(
view.deliverables.get(validation.value, {})
.get("attachments", {})
.get(VERDICTS_FILE, "")
)
before = _failed_before(view.outcomes)
readings.validation_attempt = attempt
readings.failed = outcome["failed"]
readings.repeat_failed = [f for f in outcome["failed"] if f["check"] in before]
readings.unverifiable = outcome["not_verifiable"]
readings.additions = outcome["proposed_additions"]
readings.dry_run_failed = outcome["dry_run"].get("verdict") == "fail"
async def _read_message(self, view: TurnView, readings: Readings) -> None:
"""Triage of the user's message: is it asking for a change, to what, and how
clearly?"""
reply = view.last_reply
if not reply or view.awaiting_gap_decision:
return
text = str(reply.get("text", "")).strip()
pending = view.pending_change
if pending:
text = f"{pending.get('text', '')}\nThe user then said: {text}"
candidates = _submitted(view.ledger)
if not text or not candidates:
return
readings.change_reason = text
# A structured request for changes is known to be one, and so is the reply to
# our own question about a change they already asked for.
structured = reply.get("kind") == "changes" or bool(pending)
questions = {
"change_target": Question(
"choice",
"Which submitted Deliverable would have to change first to do what "
"the user asks?",
tuple(kind.value for kind in candidates),
),
"change_clarity": Question(
"score",
"How specific is the user about what should change?",
("vague", "partial", "specific"),
),
}
if not structured:
questions["asks_change"] = Question(
"noul",
"Is the user asking for a change to work that has already been "
"submitted, as opposed to answering a question or adding information?",
)
state = (
f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n"
f"Questions waiting for the user: {len(view.pending_questions)}.\n"
f"The user's message: {text}"
)
answers = await self._decider.decide(state, questions, scope=view.execution_id)
named = _mentioned(text, candidates)
if answers is None:
# No judgement available: act only on what is certain.
if structured:
target = (
candidates[0]
if len(candidates) == 1
else named[0]
if len(named) == 1
else None
)
readings.change = ACT if target else QUEUE
readings.change_target = target
return
readings.judged = {
name: {"value": answer.value, "confidence": answer.confidence}
for name, answer in answers.items()
}
probability = 1.0 if structured else float(answers["asks_change"].value)
disposition = self._thresholds.disposition(probability)
target_answer = answers["change_target"]
target = DeliverableKind(str(target_answer.value))
readings.change_target = target
if disposition == ACT and (
target_answer.confidence < self._thresholds.confidence
or float(answers["change_clarity"].value) < 0.5
):
# They want a change, but which, or what, is not clear enough to act on.
disposition = QUEUE
if disposition == QUEUE and pending:
# Asked once already: act if they named it, otherwise let it go.
disposition = ACT if len(named) == 1 else CLOSE
readings.change_target = named[0] if named else target
readings.change = disposition
async def _read_gap(self, view: TurnView, readings: Readings) -> None:
"""Whether a gap in the Target architecture is worth revising it for."""
if (
readings.failed
or readings.dry_run_failed
or not readings.validation_attempt
):
return
if not (readings.additions or readings.unverifiable):
return
items = "\n".join(
[
f"- proposed addition on {item.get('applies_to')}: "
f"{item.get('statement')} (serves {item.get('reason')})"
for item in readings.additions
]
+ [
f"- Check {item.get('check')} could not be verified: "
f"{item.get('evidence')}"
for item in readings.unverifiable
]
)
reply = view.last_reply if view.awaiting_gap_decision else None
if reply:
question = Question(
"noul",
"Is the user asking for the Target architecture to be revised to "
"close these gaps?",
)
state = f"Gaps found by validation:\n{items}\nThe user's reply: {reply.get('text', '')}"
name = "wants_revision"
else:
question = Question(
"noul",
"Would leaving these gaps open mean a stated requirement or design "
"recommendation is not met?",
)
state = f"Gaps found by validation:\n{items}"
name = "gap_material"
answers = await self._decider.decide(
state, {name: question}, scope=view.execution_id
)
if answers is None:
readings.gap = self._gap_by_rule(view, readings) if not reply else CLOSE
return
readings.judged[name] = {
"value": answers[name].value,
"confidence": answers[name].confidence,
}
disposition = self._thresholds.disposition(float(answers[name].value))
if reply and disposition == QUEUE:
disposition = CLOSE # asked once already; an unclear answer means go on
readings.gap = disposition
@staticmethod
def _gap_by_rule(view: TurnView, readings: Readings) -> str:
"""Without a judgement: a Check that cannot be verified must be fixed, and an
addition counts when it names a real requirement or recommendation."""
if readings.unverifiable:
return ACT
known = {
item["id"]
for name, kind in (
(REQUIREMENTS_FILE, "requirements"),
(RECOMMENDATIONS_FILE, "design_recommendations"),
)
for item in read_list(view.deliverables, kind, name)
if "id" in item
}
for item in readings.additions:
words = set(re.findall(r"[A-Za-z]+-\d+", str(item.get("reason", ""))))
if words & known:
return ACT
return CLOSE
# --- the playbook -----------------------------------------------------------------
def _choose(
self, view: TurnView, r: Readings, blocked: set[str], settled: int
) -> tuple[str, str]:
"""The first group that applies, and within it the strongest action whose
conditions hold. No group applies: escalate."""
interactive = not view.pre_approved
playbook: list[tuple[str, bool, list[tuple[str, bool]]]] = [
(
"the user asked for a change",
r.change in {ACT, QUEUE},
[
(
"revise_requested",
r.change == ACT and r.change_target is not None,
),
("confirm_change", r.change == QUEUE and interactive),
],
),
(
"questions are waiting",
r.questions_waiting,
[("relay_questions", True)],
),
(
# Known as soon as the code is submitted, so no validation is spent
# on a Check that cannot pass.
"the code cannot meet a Check",
bool(r.unimplementable),
[("revise_unimplementable", True)],
),
(
"validation decided",
bool(r.validation_attempt),
[
(
"revise_target_architecture",
bool(r.repeat_failed) or (not r.failed and r.gap == ACT),
),
("revise_code", bool(r.failed) or r.dry_run_failed),
("ask_about_gaps", r.gap == QUEUE and interactive),
("proceed", True),
],
),
(
"a Phase is complete",
bool(r.phases_ready or r.phases_awaiting),
# An Approval is granted only by a structured decision, so a Phase
# still awaiting one is asked for again.
[("request_approval", True)],
),
("work is ready", bool(r.ready_work), [("delegate", True)]),
("packaging is approved", r.can_finish, [("finish", True)]),
]
for group, applies, actions in playbook:
if not applies:
continue
for action, holds in actions:
if holds and action not in blocked:
return group, action
return "no group applies", "escalate"
async def _perform(
self,
action: str,
view: TurnView,
r: Readings,
call: Callable[..., Awaitable[ToolResult]],
result: TurnResult,
) -> ToolResult | None:
if action == "proceed":
return ToolResult("Proceeding.")
if action == "revise_requested":
assert r.change_target is not None
return await call(
"reopen", kind=r.change_target.value, reason=r.change_reason
)
if action == "confirm_change":
guess = r.change_target.value if r.change_target else None
result.updates["pending_change"] = {"text": r.change_reason, "guess": guess}
options = ", ".join(kind.value for kind in _submitted(view.ledger))
question = (
f"Do you want {guess} changed? Reply naming the Deliverable to change "
f"and what should be different. Submitted so far: {options}."
if guess
else "Which Deliverable should change, and what should be different? "
f"Submitted so far: {options}."
)
return await call("ask_user", questions=[question])
if action == "relay_questions":
return await call("ask_user", questions=[])
if action == "revise_code":
items = r.failed or [
{"check": "dry-run", "evidence": "the deployment dry-run failed"}
]
reason = "; ".join(f"{i['check']} failed: {i['evidence']}" for i in items)
return await call(
"reopen", kind=DeliverableKind.DEPLOYMENT_GUIDANCE.value, reason=reason
)
if action == "revise_unimplementable":
reason = " ".join(
f"Check {item['check']} cannot be implemented as written "
f"({item['reason']}). Rewrite it so it can be met, or remove it with "
"a reason."
for item in r.unimplementable
)
return await call(
"reopen", kind=DeliverableKind.TARGET_ARCHITECTURE.value, reason=reason
)
if action == "revise_target_architecture":
return await call(
"reopen",
kind=DeliverableKind.TARGET_ARCHITECTURE.value,
reason=_graph_reason(r),
)
if action == "ask_about_gaps":
result.updates["awaiting_gap_decision"] = True
listed = "; ".join(
[str(i.get("statement")) for i in r.additions]
+ [
f"Check {i.get('check')} could not be verified"
for i in r.unverifiable
]
)
question = (
"Validation passed every Check it could, and found gaps in the "
f"Target architecture: {listed}. Should the Target architecture "
"be revised to close them, or shall I proceed as it stands?"
)
return await call("ask_user", questions=[question])
if action == "request_approval":
phase = (r.phases_ready or r.phases_awaiting)[0]
return await call(
"request_approval",
phase=phase.value,
summary=await self._summary(phase, view, r),
)
if action == "delegate":
return await call("delegate", kinds=[kind.value for kind in r.ready_work])
if action == "finish":
return await call("finish")
# escalate: no group applies.
if view.pre_approved:
result.outcome = Stopped(
"No next step applies and no user is available to decide."
)
return None
return await call(
"ask_user",
questions=["I have no next step to take. What would you like me to do?"],
)
async def _summary(self, phase: PhaseId, view: TurnView, r: Readings) -> str:
facts = approval_facts(phase, view, r)
if self._writer is not None:
try:
written = await self._writer.approval_summary(phase.value, facts)
except Exception: # a summary must never stop an Approval
logger.exception("The reply writer failed; using the assembled summary")
written = None
if written and written.strip():
return written.strip()
return assembled_summary(phase, facts)
def _graph_reason(r: Readings) -> str:
parts = [
f"Check {item['check']} failed in two validations in a row and may not be "
f"implementable as written ({item['evidence']}). Rewrite it so it can be met, "
"or remove it with a reason."
for item in r.repeat_failed
]
if not r.repeat_failed:
parts += [
f"Check {item.get('check')} could not be verified from the code "
f"({item.get('evidence')}). Rewrite it so it can be."
for item in r.unverifiable
]
parts += [
f"Add a Check on {item.get('applies_to')}: {item.get('statement')} "
f"(serves {item.get('reason')})."
for item in r.additions
]
return " ".join(parts)
def approval_facts(phase: PhaseId, view: TurnView, r: Readings) -> dict[str, Any]:
"""What an Approval summary is written from: data, never Deliverable prose."""
kinds = [kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase]
facts: dict[str, Any] = {
"phase": phase.value,
"deliverables": [kind.value for kind in kinds],
"revisions": {
kind.value: view.ledger.revisions(kind)
for kind in kinds
if view.ledger.revisions(kind)
},
}
counts = {
"requirements": len(
read_list(view.deliverables, "requirements", REQUIREMENTS_FILE)
),
"recommendations": len(
read_list(view.deliverables, "design_recommendations", RECOMMENDATIONS_FILE)
),
}
facts["counts"] = {name: count for name, count in counts.items() if count}
graph = read_graph(view.deliverables)
if graph is not None:
facts["target_architecture"] = graph.summary()
validation = view.deliverables.get(DeliverableKind.VALIDATION_RESULTS.value)
if (
validation
and phase is DEPENDENCY_GRAPH[DeliverableKind.VALIDATION_RESULTS].phase
):
outcome = validation_outcome(
validation.get("attachments", {}).get(VERDICTS_FILE, "")
)
facts["validation"] = {
"passed": len(outcome["passed"]),
"failed": [item["check"] for item in outcome["failed"]],
"not_verifiable": [item["check"] for item in outcome["not_verifiable"]],
"open_gaps": [
item.get("statement") for item in outcome["proposed_additions"]
],
}
return facts
def assembled_summary(phase: PhaseId, facts: Mapping[str, Any]) -> str:
"""An Approval summary from the facts alone, used when no writer is available."""
lines = [
(
f"The {phase.value} Phase is ready: "
f"{', '.join(facts['deliverables'])} submitted."
)
]
counts = facts.get("counts") or {}
if counts:
lines.append(
", ".join(f"{count} {name}" for name, count in counts.items()) + "."
)
graph = facts.get("target_architecture")
if graph:
lines.append(
f"Target architecture: {graph['elements']} Elements, "
f"{graph['relationships']} Relationships, {graph['checks']} Checks."
)
validation = facts.get("validation")
if validation:
lines.append(
f"Validation: {validation['passed']} Checks passed, "
f"{len(validation['failed'])} failed, "
f"{len(validation['not_verifiable'])} not verifiable."
)
if validation["open_gaps"]:
lines.append(
"Open gaps: " + "; ".join(map(str, validation["open_gaps"])) + "."
)
if facts.get("revisions"):
lines.append(
"Revised: "
+ ", ".join(
f"{kind} × {count}" for kind, count in facts["revisions"].items()
)
+ "."
)
return " ".join(lines)
# --- the two ways an Orchestrator turn can be driven ---------------------------------
class ModelOrchestrator:
"""The Orchestrator as a reasoning agent: a language model chooses the tool calls.
Kept so the Ledger's rules can be driven in any order by a scripted runtime in
tests, and so an evaluation can compare it with the router.
"""
def __init__(self, runtime: AgentRuntime, role: RoleSpec) -> None:
self._runtime = runtime
self._role = role
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult:
task = AgentTask(
role=ORCHESTRATOR,
prompt="Decide the next step for this Execution using your tools.",
context=view.context,
)
return TurnResult(await self._runtime.run(self._role, task, tools))
class RuntimeReplyWriter:
"""Writes what the user reads, on a language model. It is given a decision that
has already been made and the facts, and it decides nothing."""
def __init__(self, runtime: AgentRuntime, role: RoleSpec) -> None:
self._runtime = runtime
self._role = role
async def approval_summary(
self, phase: str, facts: Mapping[str, Any]
) -> str | None:
async def write_reply(text: str) -> ToolResult:
if not text.strip():
return ToolResult("Not submitted: the reply is empty.")
return ToolResult("Written.", Submitted(text))
tool = ToolSpec(
"write_reply",
"Give the text the user will read. Ends your turn.",
{
"type": "object",
"properties": {"text": {"type": "string"}},
"required": ["text"],
},
write_reply,
tier="execution",
)
task = AgentTask(
role=REPLY_WRITER,
prompt=f"Write the summary the user will approve the {phase} Phase from.",
context={"phase": phase, "facts": dict(facts)},
)
outcome = await self._runtime.run(self._role, task, [tool])
return outcome.content if isinstance(outcome, Submitted) else None

View File

@@ -42,10 +42,16 @@ class McpServerSpec:
class ModelConfig:
default: str
roles: Mapping[str, str] = field(default_factory=dict)
# Seconds one model request may take, for the roles whose replies are long. A
# role not named here gets the runtime's default.
timeouts: Mapping[str, float] = field(default_factory=dict)
def for_role(self, role: str) -> str:
return self.roles.get(role, self.default)
def timeout_for(self, role: str) -> float | None:
return self.timeouts.get(role)
@dataclass(frozen=True)
class ProviderBundle:
@@ -60,6 +66,9 @@ class ProviderBundle:
role_mcp_servers: Mapping[str, tuple[str, ...]]
role_skills: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
pillar_skills: Mapping[str, str] = field(default_factory=dict)
# The Terraform providers generated code is written for, pinned so the schema
# lookup, the patterns and validation agree: {name: {source, version}}.
terraform_providers: Mapping[str, Mapping[str, str]] = field(default_factory=dict)
def mcp_servers_for_role(
self, role: str, include_preview: bool = True
@@ -108,10 +117,34 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
models = config.get("models") or {}
if not isinstance(models.get("default"), str) or not models["default"]:
raise fail("'models.default' is required")
timeouts = {}
for role, seconds in (models.get("timeout_seconds") or {}).items():
if isinstance(seconds, bool) or not isinstance(seconds, (int, float)):
raise fail(f"'models.timeout_seconds' for {role!r} must be a number")
if seconds <= 0:
raise fail(f"'models.timeout_seconds' for {role!r} must be positive")
timeouts[str(role)] = float(seconds)
model_config = ModelConfig(
default=models["default"], roles=dict(models.get("roles") or {})
default=models["default"],
roles=dict(models.get("roles") or {}),
timeouts=timeouts,
)
terraform_providers = {}
pins = (config.get("terraform") or {}).get("required_providers") or {}
for provider, pin in pins.items():
if not isinstance(pin, dict) or not all(
isinstance(pin.get(key), str) and pin[key] for key in ("source", "version")
):
raise fail(
f"terraform provider {provider!r} needs a 'source' and an exact "
"'version', both strings"
)
terraform_providers[str(provider)] = {
"source": pin["source"],
"version": pin["version"],
}
tools = {}
for tier, tool_names in (config.get("tools") or {}).items():
if tier not in TIERS:
@@ -173,4 +206,5 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
role_mcp_servers=role_mcp_servers,
role_skills=role_skills,
pillar_skills=pillar_skills,
terraform_providers=terraform_providers,
)

View File

@@ -82,6 +82,13 @@ class ExecutionState(TypedDict, total=False):
refusals: list[dict[str, Any]] # the Orchestrator's refused tool calls
events_recorded: int
# Routing (ADR-0007).
last_reply: dict[str, Any] | None # the user's message, until a turn routes it
pending_change: dict[str, Any] | None # a change the user was asked to confirm
awaiting_gap_decision: bool # the user was asked whether to close gaps
validation_settled: int # the validation attempt already decided on
decisions: list[dict[str, Any]] # every routing decision, in order
turns: int
status: str # running | completed | completed_with_warnings
warnings: list[str]

View File

@@ -7,8 +7,7 @@ outcome.
from __future__ import annotations
from collections.abc import Mapping, Sequence
from pathlib import PurePosixPath
from collections.abc import Mapping, MutableMapping, Sequence
from typing import Any
from app.adk.runtime import (
@@ -21,6 +20,14 @@ from app.adk.runtime import (
files_from_argument,
)
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
from app.tools.workspace import (
file_tools,
graph_text,
graph_tools,
keep_graph,
safe_relative,
working_files,
)
from app.workflows.deliverables import (
DEPENDENCY_GRAPH,
DeliverableKind,
@@ -35,6 +42,7 @@ from app.workflows.target_architecture import (
TargetArchitecture,
parse_graph,
render_mermaid,
render_tables,
)
ORCHESTRATOR = "orchestrator"
@@ -244,29 +252,37 @@ class OrchestratorToolbox:
return ToolResult("Finishing the Execution.", TurnEnded("finish"))
def _safe_relative(name: str) -> bool:
path = PurePosixPath(name)
return bool(name) and not path.is_absolute() and ".." not in path.parts
def specialist_tools(
kind: DeliverableKind,
extra: Sequence[ToolSpec],
allow_cloud_read: bool | None,
upstream: Mapping[str, Mapping[str, Any]] | None = None,
working: MutableMapping[str, Any] | None = None,
library: Mapping[str, Mapping[str, Any]] | None = None,
) -> list[ToolSpec]:
"""A Specialist's tools: submit, ask, and its role's tools with the cloud-read gate.
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
the cloud-read gate.
`upstream` is the Deliverables this one builds on; with it, a submission's links
along the Requirements chain are checked.
along the Requirements chain are checked. `working` is the Deliverable's working
memory, where its working copy is kept. `library` is the earlier Deliverables
given to the Specialist as data files alone, to read whole when it needs one.
"""
working = {} if working is None else working
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
own_files = DEPENDENCY_GRAPH[kind].working_copy is kind
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
try:
attachments = files_from_argument(attachments)
except TypeError as exc:
return _refused(f"attachments: {exc}")
unsafe = [name for name in attachments if not _safe_relative(name)]
# The working copy is the submission; a file passed here replaces its copy.
if own_files:
attachments = {**working_files(working), **attachments}
if is_graph and GRAPH_ATTACHMENT not in attachments and graph_text(working):
attachments[GRAPH_ATTACHMENT] = graph_text(working)
unsafe = [name for name in attachments if not safe_relative(name)]
if unsafe:
return _refused(
"attachment names must be relative paths without '..': "
@@ -277,13 +293,27 @@ def specialist_tools(
return ToolResult(
"Not submitted. The Deliverable is missing: " + "; ".join(missing)
)
if kind is DeliverableKind.TARGET_ARCHITECTURE:
# The diagram is a view of the graph, so the engine draws it.
attachments[DIAGRAM_ATTACHMENT] = render_mermaid(
parse_graph(attachments[GRAPH_ATTACHMENT])
)
if own_files:
working_files(working).update(attachments)
if is_graph:
# The tables and the diagram are views of the graph, so the engine
# writes them: they cannot disagree with it.
graph = parse_graph(attachments[GRAPH_ATTACHMENT])
keep_graph(working, attachments[GRAPH_ATTACHMENT])
content = content.rstrip() + "\n\n" + render_tables(graph)
attachments[DIAGRAM_ATTACHMENT] = render_mermaid(graph)
return ToolResult(f"Submitted {kind}.", Submitted(content, attachments))
async def read_deliverable(kind: str) -> ToolResult:
deliverable = (library or {}).get(kind)
if deliverable is None:
available = ", ".join(library or {}) or "none"
return _refused(f"{kind} is not one you can read here: {available}")
attached = ", ".join(deliverable.get("attachments", {})) or "none"
return ToolResult(
f"{deliverable.get('content', '')}\n\nAttachments: {attached}"
)
async def needs_input(
questions: Sequence[str], draft: str | None = None, attachments: Any = None
) -> ToolResult:
@@ -331,6 +361,23 @@ def specialist_tools(
needs_input,
tier="execution",
),
*(
[
ToolSpec(
"read_deliverable",
"Read the full text of an earlier Deliverable that `dependencies` "
"gives you as its data file alone: "
f"{', '.join(library)}. Use it only when the ids are not enough.",
_schema({"kind": {"type": "string"}}, ["kind"]),
read_deliverable,
tier="execution",
)
]
if library
else ()
),
*(graph_tools(working) if is_graph else ()),
*(file_tools(working) if own_files else ()),
*(gate_cloud_read(tool, allow_cloud_read) for tool in extra),
]

327
app/tools/jev.py Normal file
View File

@@ -0,0 +1,327 @@
"""The Jev decision client: typed answers to closed questions, within strict limits.
Jev is TypeSafe AI's decision model. This module is the only one that speaks to it,
through the vendor's Python SDK (`typesafe-sdk`), which owns the request shape, the
timeout and the retries. Everything else here is ours: the service's limits are not
published, so the client assumes they are strict and never lets Jev hold up an
Execution. Any refusal, failure or timeout returns None and the caller takes its
fallback.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import logging
import time
from collections import OrderedDict, deque
from collections.abc import Awaitable, Callable, Mapping
from dataclasses import dataclass, field
from typing import Any
from typesafe_sdk import (
AsyncTypeSafeClient,
Choice,
ChoiceAnswer,
Noul,
NoulAnswer,
RetryPolicy,
Score,
ScoreAnswer,
TypeSafeError,
TypeSafeRateLimitError,
)
from app.adk.runtime import Answer, Question
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class JevLimits:
"""Every limit the client keeps. Starting values, to tune from measurement."""
max_tokens: int = 48_000 # state plus questions; the service allows 64,000
max_question_tokens: int = 24_000 # state plus the longest question; allows 32,000
request_timeout: float = 5.0 # one HTTP attempt
total_timeout: float = 15.0 # a decision, retries included
retries: int = 2
backoff_base: float = 0.5
backoff_cap: float = 8.0
requests_per_second: float = 5.0
burst: int = 10
max_in_flight: int = 8
breaker_failures: int = 5
breaker_window: float = 30.0
breaker_cooldown: float = 60.0
cache_size: int = 1024
budget_calls: int = 500 # per scope, usually one Execution
budget_tokens: int = 1_000_000 # per scope; about $0.42 at the published price
@dataclass
class DecisionRecord:
"""One attempt to decide, kept for the evaluation report and for tuning."""
scope: str
questions: tuple[str, ...]
# answered | cached | too_large | over_budget | breaker_open | failed
outcome: str = "failed"
seconds: float = 0.0
rate_limited: bool = False
input_tokens: int = 0
confidence: Mapping[str, float] = field(default_factory=dict)
def estimate_tokens(text: str) -> int:
"""A deliberately high estimate: refusing locally is cheaper than a rejected call."""
return len(text) // 3 + 1
def encode_question(question: Question) -> Noul | Choice | Score:
if question.kind == "choice":
return Choice(
instructions=question.instructions,
criteria=dict.fromkeys(question.options),
)
if question.kind == "score":
return Score(
instructions=question.instructions, criteria=list(question.options)
)
return Noul(instructions=question.instructions)
def decode_answer(question: Question, raw: object) -> Answer | None:
"""The SDK's answer as ours, or None when it does not answer the question asked."""
if question.kind == "choice" and isinstance(raw, ChoiceAnswer):
# A choice outside the options asked for is not an answer.
if raw.choice not in question.options:
return None
return Answer(raw.choice, raw.confidence, dict(raw.probabilities))
if question.kind == "score" and isinstance(raw, ScoreAnswer):
probabilities = {str(level): p for level, p in raw.probabilities.items()}
return Answer(raw.score, raw.confidence, probabilities)
if question.kind == "noul" and isinstance(raw, NoulAnswer):
# Jev gives a probability for a yes-or-no question and no separate
# confidence, so confidence is how far the probability is from even.
return Answer(raw.noul, abs(2 * raw.noul - 1))
return None
class JevDecider:
def __init__(
self,
*,
api_key: str,
base_url: str | None = None,
model: str | None = None,
limits: JevLimits | None = None,
transport: Any = None,
clock: Callable[[], float] = time.monotonic,
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
) -> None:
self.limits = limits or JevLimits()
self._clock = clock
self._sleep = sleep
self._client = AsyncTypeSafeClient(
api_key=api_key,
base_url=base_url,
model=model,
timeout=self.limits.request_timeout,
# The SDK retries 408, 429 and 5xx, and connection and timeout errors,
# with jittered backoff, and honours Retry-After.
retry=RetryPolicy(
max_retries=self.limits.retries,
backoff_initial=self.limits.backoff_base,
backoff_max=self.limits.backoff_cap,
timeout=self.limits.total_timeout,
),
transport=transport,
)
self._in_flight = asyncio.Semaphore(self.limits.max_in_flight)
# Token bucket: starts full, refills at `rate`, which drops when the service
# says we are going too fast.
self._rate = self.limits.requests_per_second
self._tokens = float(self.limits.burst)
self._refilled = clock()
self._bucket = asyncio.Lock()
self._failures: deque[float] = deque()
self._open_until = 0.0
self._probing = False
self._cache: OrderedDict[str, dict[str, Answer]] = OrderedDict()
self._calls: dict[str, int] = {}
self._tokens_used: dict[str, int] = {}
self._over_budget: set[str] = set()
self.records: list[DecisionRecord] = []
async def aclose(self) -> None:
await self._client.aclose()
async def decide(
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
) -> Mapping[str, Answer] | None:
record = DecisionRecord(scope=scope, questions=tuple(questions))
self.records.append(record)
started = self._clock()
try:
return await self._decide(state, questions, scope, record)
finally:
record.seconds = self._clock() - started
async def _decide(
self,
state: str,
questions: Mapping[str, Question],
scope: str,
record: DecisionRecord,
) -> Mapping[str, Answer] | None:
if not questions:
record.outcome = "answered"
return {}
asked = {
name: {
"kind": question.kind,
"instructions": question.instructions,
"options": list(question.options),
}
for name, question in questions.items()
}
sizes = [estimate_tokens(json.dumps(item)) for item in asked.values()]
state_size = estimate_tokens(state)
if (
state_size + sum(sizes) > self.limits.max_tokens
or state_size + max(sizes) > self.limits.max_question_tokens
):
# Never truncate: the caller must split by Deliverable or section.
record.outcome = "too_large"
return None
key = hashlib.sha256(
json.dumps({"state": state, "questions": asked}, sort_keys=True).encode()
).hexdigest()
if key in self._cache:
self._cache.move_to_end(key)
record.outcome = "cached"
return dict(self._cache[key])
if (
self._calls.get(scope, 0) >= self.limits.budget_calls
or self._tokens_used.get(scope, 0) >= self.limits.budget_tokens
):
if scope not in self._over_budget:
self._over_budget.add(scope)
logger.warning("Jev budget reached for %s; using fallbacks", scope)
record.outcome = "over_budget"
return None
if not self._breaker_allows():
record.outcome = "breaker_open"
return None
self._calls[scope] = self._calls.get(scope, 0) + 1
await self._take_token()
try:
async with self._in_flight:
response = await self._client.system_one(
state=state,
questions={
name: encode_question(question)
for name, question in questions.items()
},
)
except TypeSafeRateLimitError:
# Still rate limited after the SDK's retries: halve our own rate.
record.rate_limited = True
self._rate = max(self._rate / 2, 0.25)
self._note_failure()
return None
except TypeSafeError as exc:
logger.info("Jev request failed: %s", type(exc).__name__)
self._note_failure()
return None
self._note_success()
used = response.usage.input_tokens or 0
record.input_tokens = used
self._tokens_used[scope] = self._tokens_used.get(scope, 0) + used
answers: dict[str, Answer] = {}
for name, question in questions.items():
answer = decode_answer(question, response.answers.get(name))
if answer is None:
# A reply that does not answer every question answers none.
return None
answers[name] = answer
record.outcome = "answered"
record.confidence = {name: a.confidence for name, a in answers.items()}
self._cache[key] = answers
while len(self._cache) > self.limits.cache_size:
self._cache.popitem(last=False)
return dict(answers)
async def _take_token(self) -> None:
async with self._bucket:
while True:
now = self._clock()
self._tokens = min(
float(self.limits.burst),
self._tokens + (now - self._refilled) * self._rate,
)
self._refilled = now
if self._tokens >= 1:
self._tokens -= 1
return
await self._sleep((1 - self._tokens) / self._rate)
def _breaker_allows(self) -> bool:
now = self._clock()
if now < self._open_until:
return False
if self._open_until and not self._probing:
# Cool-down over: let exactly one call through to test the service.
self._probing = True
return True
def _note_failure(self) -> None:
now = self._clock()
if self._probing:
self._probing = False
self._open_until = now + self.limits.breaker_cooldown
return
self._failures.append(now)
while self._failures and now - self._failures[0] > self.limits.breaker_window:
self._failures.popleft()
if len(self._failures) >= self.limits.breaker_failures:
self._failures.clear()
self._open_until = now + self.limits.breaker_cooldown
logger.warning(
"Jev circuit open for %.0fs after repeated failures",
self.limits.breaker_cooldown,
)
def _note_success(self) -> None:
self._failures.clear()
self._open_until = 0.0
self._probing = False
def summary(self) -> dict[str, Any]:
"""Totals for the evaluation report."""
outcomes: dict[str, int] = {}
for record in self.records:
outcomes[record.outcome] = outcomes.get(record.outcome, 0) + 1
answered = [r for r in self.records if r.outcome == "answered"]
return {
"decisions": len(self.records),
"outcomes": outcomes,
"fallbacks": sum(
count
for name, count in outcomes.items()
if name not in {"answered", "cached"}
),
"rate_limited": sum(1 for r in self.records if r.rate_limited),
"seconds": round(sum(r.seconds for r in answered), 3),
"input_tokens": sum(r.input_tokens for r in self.records),
}

View File

@@ -1,16 +1,23 @@
"""Local tools: a Mermaid syntax check and Terraform checks in a scratch directory.
Terraform runs only fmt, init -backend=false, validate and plan. No command that
changes cloud resources is ever constructed.
Terraform runs only fmt, init -backend=false, validate, plan and providers schema. No
command that changes cloud resources is ever constructed.
A Specialist with a working copy (app/tools/workspace.py) runs the checks on it and
passes no files; `terraform_fmt` formats the working copy in place. That is what
keeps a fix cheap: the model changes a line and validates, without sending every file
again on each call.
"""
from __future__ import annotations
import asyncio
import difflib
import json
import os
import re
import tempfile
from collections.abc import Awaitable, Callable, Iterable, Mapping
from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping
from dataclasses import dataclass, field
from pathlib import Path, PurePosixPath
from typing import Any
@@ -79,10 +86,19 @@ class TerraformRunner:
# Providers are downloaded here once and reused by every later check. Without it,
# each validate or plan downloads its providers again into a scratch directory.
plugin_cache_dir: str | None = None
# The providers the schema lookup answers for, as the provider bundle pins them:
# {local name: {"source": ..., "version": ...}}.
providers: Mapping[str, Mapping[str, str]] = field(default_factory=dict)
# Terraform's plugin cache is not safe for concurrent installs.
_init_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False)
_schema_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False)
# resource type -> (provider label, the resource's schema block)
_schemas: dict[str, tuple[str, dict[str, Any]]] | None = field(
default=None, repr=False
)
async def fmt(self, files: Mapping[str, str]) -> ToolResult:
async def formatted(self, files: Mapping[str, str]) -> dict[str, str] | ToolResult:
"""The files formatting changes, with their new content; or why it failed."""
refusal = _check_files(files)
if refusal:
return refusal
@@ -93,19 +109,24 @@ class TerraformRunner:
)
if code != 0:
return ToolResult(f"terraform fmt failed:\n{output}")
changed = [
name for name in files if (root / name).read_text() != files[name]
]
if not changed:
return ToolResult("terraform fmt: all files are already formatted.")
formatted = "\n\n".join(
f"--- {name}\n{(root / name).read_text()}" for name in changed
)
return ToolResult(
_clip(
f"terraform fmt reformatted {len(changed)} file(s):\n\n{formatted}"
)
)
return {
name: (root / name).read_text()
for name in files
if (root / name).read_text() != files[name]
}
async def fmt(self, files: Mapping[str, str]) -> ToolResult:
changed = await self.formatted(files)
if isinstance(changed, ToolResult):
return changed
if not changed:
return ToolResult("terraform fmt: all files are already formatted.")
formatted = "\n\n".join(
f"--- {name}\n{content}" for name, content in changed.items()
)
return ToolResult(
_clip(f"terraform fmt reformatted {len(changed)} file(s):\n\n{formatted}")
)
async def validate(self, files: Mapping[str, str]) -> ToolResult:
refusal = _check_files(files)
@@ -150,6 +171,87 @@ class TerraformRunner:
_clip(f"terraform plan {verdict} (nothing was applied):\n{output}")
)
async def schema(
self, resource_type: str, block: str = "", find: str = ""
) -> ToolResult:
"""What a resource type accepts, from the pinned provider's own schema."""
loaded = await self._load_schemas()
if isinstance(loaded, ToolResult):
return loaded
if resource_type not in loaded:
close = difflib.get_close_matches(resource_type, loaded, n=5, cutoff=0.6)
hint = f" Close matches: {', '.join(close)}." if close else ""
return ToolResult(
f"No resource type {resource_type} in {self._provider_labels()}.{hint}"
)
label, root = loaded[resource_type]
title = f"{resource_type} ({label})"
if find:
return ToolResult(_clip(_find_in_schema(title, root, find)))
node = root
for step in [part for part in block.split(".") if part]:
nested = node.get("block_types", {}).get(step)
if nested is None:
inside = ", ".join(sorted(node.get("block_types", {}))) or "none"
return ToolResult(
f"{title} has no block {block!r}: {step!r} is not a block there. "
f"Blocks at that level: {inside}."
)
node = nested["block"]
heading = f"{title}, block {block}" if block else title
return ToolResult(_clip(_render_schema(heading, node)))
def _provider_labels(self) -> str:
return (
", ".join(
f"{pin['source']} {pin['version']}" for pin in self.providers.values()
)
or "no pinned provider"
)
async def _load_schemas(self) -> dict[str, tuple[str, dict[str, Any]]] | ToolResult:
async with self._schema_lock:
if self._schemas is not None:
return self._schemas
if not self.providers:
return ToolResult(
"Schema lookup is unavailable: the Cloud provider bundle pins no "
"Terraform provider."
)
required = "\n".join(
f' {name} = {{ source = "{pin["source"]}", '
f'version = "{pin["version"]}" }}'
for name, pin in self.providers.items()
)
with tempfile.TemporaryDirectory(prefix="tf-schema-") as tmp:
root = Path(tmp)
(root / "versions.tf").write_text(
f"terraform {{\n required_providers {{\n{required}\n }}\n}}\n"
)
failure = await self._init(root)
if failure:
return failure
code, output = await self._run(
[self.binary, "providers", "schema", "-json"], root
)
if code != 0:
return ToolResult(
_clip(f"terraform providers schema failed:\n{output}")
)
try:
data = json.loads(output)
except ValueError:
return ToolResult("terraform providers schema returned no JSON.")
labels = {pin["source"]: pin["version"] for pin in self.providers.values()}
schemas: dict[str, tuple[str, dict[str, Any]]] = {}
for address, provider in (data.get("provider_schemas") or {}).items():
source = address.split("/", 1)[-1] # drop the registry host
label = f"{source} {labels.get(source, '')}".strip()
for name, resource in (provider.get("resource_schemas") or {}).items():
schemas[name] = (label, resource.get("block") or {})
self._schemas = schemas
return schemas
async def _init(self, root: Path) -> ToolResult | None:
command = [self.binary, "init", "-backend=false", "-input=false", "-no-color"]
if self.plugin_cache_dir:
@@ -236,6 +338,100 @@ def _clip(text: str) -> str:
return text if len(text) <= _MAX_OUTPUT else text[:_MAX_OUTPUT] + "\n[truncated]"
def _type(value: Any) -> str:
if isinstance(value, str):
return value
if isinstance(value, list) and len(value) == 2 and value[0] != "object":
return f"{value[0]} of {_type(value[1])}"
return "object"
def _use(attribute: Mapping[str, Any]) -> str:
if attribute.get("required"):
return "required"
return "optional" if attribute.get("optional") else "read-only"
def _arguments(node: Mapping[str, Any]) -> dict[str, Mapping[str, Any]]:
"""The arguments code may set: everything but the read-only ones."""
return {
name: attribute
for name, attribute in sorted((node.get("attributes") or {}).items())
if _use(attribute) != "read-only"
}
def _block_note(nested: Mapping[str, Any]) -> str:
notes = [str(nested.get("nesting_mode", "block"))]
if nested.get("min_items"):
notes.append(f"at least {nested['min_items']}")
if nested.get("max_items"):
notes.append(f"at most {nested['max_items']}")
return ", ".join(notes)
def _render_schema(heading: str, node: Mapping[str, Any]) -> str:
"""One level in full, and the blocks beneath it as an outline of names."""
lines = [heading, "", "Arguments:"]
arguments = _arguments(node)
for name, attribute in arguments.items():
described = " ".join(str(attribute.get("description") or "").split())[:160]
deprecated = " DEPRECATED" if attribute.get("deprecated") else ""
lines.append(
f" {name} ({_type(attribute.get('type'))}, {_use(attribute)})"
f"{deprecated}" + (f": {described}" if described else "")
)
if not arguments:
lines.append(" none")
def outline(parent: Mapping[str, Any], depth: int) -> None:
for name, nested in sorted((parent.get("block_types") or {}).items()):
inner = nested.get("block") or {}
names = ", ".join(
f"{argument}*" if attribute.get("required") else argument
for argument, attribute in _arguments(inner).items()
)
lines.append(
f"{' ' * depth}{name} {{ }} ({_block_note(nested)}): {names or '-'}"
)
outline(inner, depth + 1)
lines += ["", "Blocks (* marks a required argument):"]
before = len(lines)
outline(node, 1)
if len(lines) == before:
lines.append(" none")
lines += ["", "Pass `block`, such as a.b, for one block's arguments in full."]
return "\n".join(lines)
def _find_in_schema(title: str, root: Mapping[str, Any], wanted: str) -> str:
"""Every argument or block whose name contains `wanted`, with where it sits."""
needle = wanted.lower()
found: list[str] = []
def search(node: Mapping[str, Any], path: str) -> None:
for name, attribute in sorted((node.get("attributes") or {}).items()):
if needle in name.lower():
where = f"{path}.{name}" if path else name
found.append(
f" {where} ({_type(attribute.get('type'))}, {_use(attribute)})"
)
for name, nested in sorted((node.get("block_types") or {}).items()):
where = f"{path}.{name}" if path else name
if needle in name.lower():
found.append(f" {where} {{ }} ({_block_note(nested)})")
search(nested.get("block") or {}, where)
search(root, "")
if not found:
return (
f"{title} has no argument or block named like {wanted!r}. It cannot be "
"set on this resource type."
)
return f"{title}: {wanted!r} is found at\n" + "\n".join(found)
_FILES_SCHEMA: dict[str, Any] = {
**FILES_SCHEMA,
"description": "Terraform files in one directory, e.g. terraform/main.tf",
@@ -244,19 +440,53 @@ _FILES_SCHEMA: dict[str, Any] = {
def _with_files(
method: Callable[..., Awaitable[ToolResult]],
working_copy: MutableMapping[str, str] | None = None,
) -> Callable[..., Awaitable[ToolResult]]:
"""Adapt a runner method to take the FILES_SCHEMA argument."""
"""Adapt a runner method to take the FILES_SCHEMA argument.
async def handler(files: Any, **kwargs: Any) -> ToolResult:
With a working copy, the files are the Terraform files in it and the argument is
optional: files passed anyway are written to the working copy first.
"""
async def handler(files: Any = None, **kwargs: Any) -> ToolResult:
try:
converted = files_from_argument(files)
except TypeError as exc:
return ToolResult(f"Refused: files: {exc}")
return await method(converted, **kwargs)
if working_copy is None:
return await method(converted, **kwargs)
working_copy.update(converted)
return await method(_terraform_files(working_copy), **kwargs)
return handler
def _terraform_files(files: Mapping[str, str]) -> dict[str, str]:
return {
name: content
for name, content in files.items()
if name.endswith(_TERRAFORM_SUFFIXES)
}
def _fmt_in_place(
runner: TerraformRunner, working_copy: MutableMapping[str, str]
) -> Callable[..., Awaitable[ToolResult]]:
async def format_working_copy(files: Mapping[str, str]) -> ToolResult:
changed = await runner.formatted(files)
if isinstance(changed, ToolResult):
return changed
working_copy.update(changed)
if not changed:
return ToolResult("terraform fmt: all files are already formatted.")
return ToolResult(
f"terraform fmt reformatted {len(changed)} file(s) in your working "
f"copy: {', '.join(sorted(changed))}."
)
return _with_files(format_working_copy, working_copy)
LOCAL_TOOLS: dict[str, tuple[str, str]] = {
"mermaid_check": ("general", "Check a Mermaid diagram's basic structure."),
"terraform_fmt": ("general", "Format Terraform files and return any that changed."),
@@ -268,10 +498,48 @@ LOCAL_TOOLS: dict[str, tuple[str, str]] = {
"cloud_read",
"Preview Terraform against the live cloud environment with a plan. Never applies.",
),
"terraform_schema": (
"general",
(
"Look up what a Terraform resource type accepts, from the provider's own "
"schema: its arguments and blocks, one block in full with `block`, or "
"whether an argument exists anywhere in it with `find`."
),
),
}
_ON_WORKING_COPY = " Runs on your working copy; pass no files."
def local_tools(
names: Iterable[str],
runner: TerraformRunner,
working_copy: MutableMapping[str, str] | None = None,
) -> list[ToolSpec]:
"""The named tools. With a `working_copy`, the Terraform checks run on it."""
files_required = [] if working_copy is not None else ["files"]
suffix = _ON_WORKING_COPY if working_copy is not None else ""
def checks(name: str, method: Callable[..., Awaitable[ToolResult]]) -> ToolSpec:
properties: dict[str, Any] = {"files": _FILES_SCHEMA}
if name == "terraform_plan":
properties["variables"] = {
"type": "object",
"additionalProperties": {"type": "string"},
}
tier, description = LOCAL_TOOLS[name]
return ToolSpec(
name,
description + suffix,
{
"type": "object",
"properties": properties,
"required": files_required,
},
method,
tier=tier,
)
def local_tools(names: Iterable[str], runner: TerraformRunner) -> list[ToolSpec]:
builders = {
"mermaid_check": lambda: ToolSpec(
"mermaid_check",
@@ -284,44 +552,41 @@ def local_tools(names: Iterable[str], runner: TerraformRunner) -> list[ToolSpec]
mermaid_check,
tier="general",
),
"terraform_fmt": lambda: ToolSpec(
"terraform_fmt": lambda: checks(
"terraform_fmt",
LOCAL_TOOLS["terraform_fmt"][1],
{
"type": "object",
"properties": {"files": _FILES_SCHEMA},
"required": ["files"],
},
_with_files(runner.fmt),
tier="general",
_fmt_in_place(runner, working_copy)
if working_copy is not None
else _with_files(runner.fmt),
),
"terraform_validate": lambda: ToolSpec(
"terraform_validate",
LOCAL_TOOLS["terraform_validate"][1],
{
"type": "object",
"properties": {"files": _FILES_SCHEMA},
"required": ["files"],
},
_with_files(runner.validate),
tier="general",
"terraform_validate": lambda: checks(
"terraform_validate", _with_files(runner.validate, working_copy)
),
"terraform_plan": lambda: ToolSpec(
"terraform_plan",
LOCAL_TOOLS["terraform_plan"][1],
"terraform_plan": lambda: checks(
"terraform_plan", _with_files(runner.plan, working_copy)
),
"terraform_schema": lambda: ToolSpec(
"terraform_schema",
LOCAL_TOOLS["terraform_schema"][1],
{
"type": "object",
"properties": {
"files": _FILES_SCHEMA,
"variables": {
"type": "object",
"additionalProperties": {"type": "string"},
"resource_type": {
"type": "string",
"description": "For example google_cloud_run_v2_service",
},
"block": {
"type": "string",
"description": "A nested block, such as template.containers",
},
"find": {
"type": "string",
"description": "An argument or block name to look for",
},
},
"required": ["files"],
"required": ["resource_type"],
},
_with_files(runner.plan),
tier="cloud_read",
runner.schema,
tier="general",
),
}
return [builders[name]() for name in names]

View File

@@ -2,19 +2,24 @@
A role gets the local tools its role file names, the tools of the remote MCP servers
the provider bundle assigns it, and, for the design advisor, the pillar consultation.
When the Deliverable has a working copy, the local tools are built on it.
Cloud-read gating is applied later, per Execution, by `specialist_tools`.
"""
from __future__ import annotations
from collections.abc import Mapping, MutableMapping, Sequence
from collections.abc import Callable, Mapping, MutableMapping, Sequence
from typing import Any
from app.adk.roles import PILLAR_ADVISOR
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory, ToolSpec
from app.tools.mcp import McpCatalog
from app.tools.pillars import pillar_consultation_tool
from app.workflows.deliverables import DeliverableKind
from app.tools.workspace import working_files
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
# The local tools, built on a working copy of files when the Deliverable has one.
LocalToolBuilder = Callable[[MutableMapping[str, str] | None], Sequence[ToolSpec]]
def make_tool_factory(
@@ -22,11 +27,14 @@ def make_tool_factory(
roles: Mapping[str, RoleSpec],
pillar_skills: Mapping[str, str],
*,
local_tools: Sequence[ToolSpec] = (),
local_tools: Sequence[ToolSpec] | LocalToolBuilder = (),
catalog: McpCatalog | None = None,
limiter: RunLimiter | None = None,
) -> ToolFactory:
local_by_name = {tool.name: tool for tool in local_tools}
def build_local(
working_copy: MutableMapping[str, str] | None,
) -> Sequence[ToolSpec]:
return local_tools(working_copy) if callable(local_tools) else local_tools
def mcp_tools(role: RoleSpec) -> list[ToolSpec]:
return catalog.tools_for(role.mcp_servers) if catalog else []
@@ -42,6 +50,16 @@ def make_tool_factory(
def tools_for(
role: RoleSpec, deliverable: str | None, working: MutableMapping[str, Any]
) -> Sequence[ToolSpec]:
has_working_copy = any(
kind.value == deliverable and spec.working_copy is not None
for kind, spec in DEPENDENCY_GRAPH.items()
)
local_by_name = {
tool.name: tool
for tool in build_local(
working_files(working) if has_working_copy else None
)
}
tools = [local_by_name[name] for name in role.tools if name in local_by_name]
tools += mcp_tools(role)
if (

289
app/tools/workspace.py Normal file
View File

@@ -0,0 +1,289 @@
"""A Specialist's working copy: files and a graph built up over several tool calls.
A model that must return a whole Deliverable in one reply spends most of its output
repeating itself, and a long reply is what times out. With a working copy it writes a
file once, changes only the lines that need changing, and adds to a graph a few items
at a time. The working copy is kept with the Execution, so a Revision starts from the
last attempt instead of from nothing.
It lives in the Deliverable's working memory:
working["files"] {path: content}
working["graph"] the Target architecture, as target-architecture.json holds it
"""
from __future__ import annotations
import json
from collections.abc import Mapping, MutableMapping, Sequence
from pathlib import PurePosixPath
from typing import Any
from app.adk.runtime import ToolResult, ToolSpec
FILES = "files"
GRAPH = "graph"
_GRAPH_LISTS = ("elements", "relationships", "checks", "not_applicable")
_MAX_READ = 40_000
def safe_relative(name: str) -> bool:
path = PurePosixPath(name)
return bool(name) and not path.is_absolute() and ".." not in path.parts
def working_files(working: MutableMapping[str, Any]) -> MutableMapping[str, str]:
files = working.get(FILES)
if not isinstance(files, MutableMapping):
files = working[FILES] = {}
return files
def _schema(properties: Mapping[str, Any], required: Sequence[str]) -> dict[str, Any]:
return {
"type": "object",
"properties": dict(properties),
"required": list(required),
}
def _lines(content: str) -> int:
return content.count("\n") + (1 if content and not content.endswith("\n") else 0)
def file_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
"""Tools that write, change and read the files of a working copy."""
files = working_files(working)
async def write_file(path: str, content: str) -> ToolResult:
if not safe_relative(path):
return ToolResult(
f"Refused: {path!r} must be a relative path without '..'."
)
files[path] = content
return ToolResult(f"Wrote {path} ({_lines(content)} lines).")
async def edit_file(path: str, old: str, new: str) -> ToolResult:
if path not in files:
return ToolResult(f"Refused: {path} is not in your working copy.")
found = files[path].count(old) if old else 0
if found != 1:
return ToolResult(
f"Refused: `old` must match exactly one place in {path}; it matches "
f"{found}. Include more of the surrounding lines."
)
files[path] = files[path].replace(old, new)
return ToolResult(f"Edited {path}.")
async def read_file(path: str = "") -> ToolResult:
if not path:
if not files:
return ToolResult("Your working copy is empty.")
return ToolResult(
"Your working copy:\n"
+ "\n".join(
f"- {name} ({_lines(files[name])} lines)" for name in sorted(files)
)
)
if path not in files:
return ToolResult(f"Refused: {path} is not in your working copy.")
content = files[path]
if len(content) > _MAX_READ:
content = content[:_MAX_READ] + "\n[truncated]"
return ToolResult(content)
async def delete_file(path: str) -> ToolResult:
if files.pop(path, None) is None:
return ToolResult(f"Refused: {path} is not in your working copy.")
return ToolResult(f"Deleted {path}.")
text = {"type": "string"}
return [
ToolSpec(
"write_file",
"Write one file of your working copy, replacing it if it exists. Your "
"working copy is attached for you when you submit.",
_schema({"path": text, "content": text}, ["path", "content"]),
write_file,
),
ToolSpec(
"edit_file",
"Change part of one file: replace the single place `old` matches with "
"`new`. Use this for a fix instead of writing the file again.",
_schema({"path": text, "old": text, "new": text}, ["path", "old", "new"]),
edit_file,
),
ToolSpec(
"read_file",
"Read one file of your working copy, or list its files when `path` is "
"left out.",
_schema({"path": text}, []),
read_file,
),
ToolSpec(
"delete_file",
"Remove one file from your working copy.",
_schema({"path": text}, ["path"]),
delete_file,
),
]
# --- the graph -----------------------------------------------------------------------
def working_graph(working: MutableMapping[str, Any]) -> dict[str, list[dict[str, Any]]]:
graph = working.get(GRAPH)
if not isinstance(graph, dict):
graph = working[GRAPH] = {}
for name in _GRAPH_LISTS:
if not isinstance(graph.get(name), list):
graph[name] = []
return graph
def graph_text(working: MutableMapping[str, Any]) -> str:
"""The working graph as target-architecture.json, or "" when nothing is in it."""
graph = working.get(GRAPH)
if not isinstance(graph, dict) or not any(graph.get(n) for n in _GRAPH_LISTS):
return ""
return json.dumps(
{name: graph.get(name, []) for name in _GRAPH_LISTS},
indent=2,
ensure_ascii=False,
)
def keep_graph(working: MutableMapping[str, Any], text: str) -> None:
"""Make a submitted graph the working graph, so a Revision amends it."""
try:
data = json.loads(text)
except ValueError:
return
if isinstance(data, dict):
working[GRAPH] = {
name: [item for item in data.get(name, []) if isinstance(item, dict)]
for name in _GRAPH_LISTS
}
def _key(name: str) -> str:
return "recommendation" if name == "not_applicable" else "id"
def graph_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
"""Tools that build the Target architecture a few items at a time."""
async def graph_put(
elements: Sequence[Any] = (),
relationships: Sequence[Any] = (),
checks: Sequence[Any] = (),
not_applicable: Sequence[Any] = (),
) -> ToolResult:
graph = working_graph(working)
given = dict(
zip(
_GRAPH_LISTS,
(elements, relationships, checks, not_applicable),
strict=True,
)
)
bad = [
f"{name}[{index}]"
for name, items in given.items()
for index, item in enumerate(items or ())
if not isinstance(item, Mapping)
or not isinstance(item.get(_key(name)), str)
or not item[_key(name)].strip()
]
if bad:
return ToolResult(
"Refused: each item must be an object with an `id` (a "
f"`recommendation` under not_applicable): {', '.join(bad)}"
)
added = replaced = 0
for name, items in given.items():
key = _key(name)
for item in items or ():
for index, existing in enumerate(graph[name]):
if existing.get(key) == item[key]:
graph[name][index] = dict(item)
replaced += 1
break
else:
graph[name].append(dict(item))
added += 1
return ToolResult(f"Added {added}, replaced {replaced}. {_counts(graph)}")
async def graph_remove(ids: Sequence[str]) -> ToolResult:
graph = working_graph(working)
wanted = set(ids)
removed = 0
for name in _GRAPH_LISTS:
kept = [item for item in graph[name] if item.get(_key(name)) not in wanted]
removed += len(graph[name]) - len(kept)
graph[name] = kept
return ToolResult(f"Removed {removed}. {_counts(graph)}")
async def graph_read() -> ToolResult:
return ToolResult(graph_text(working) or "Your working graph is empty.")
def items(*fields: str, optional: tuple[str, ...] = ()) -> dict[str, Any]:
return {
"type": "array",
"items": _schema(
{name: {"type": "string"} for name in (*fields, *optional)}, fields
),
}
lists = {
"elements": items(
"id", "name", "product", "resource_type", optional=("component",)
),
"relationships": items("id", "from", "to", "kind"),
"checks": items("id", "applies_to", "statement", "source"),
"not_applicable": items("recommendation", "reason"),
}
return [
ToolSpec(
"graph_put",
"Add items to your working graph, or replace the ones with the same id. "
"Call it several times with a few items each. Your working graph is "
"attached as target-architecture.json when you submit.",
_schema(lists, []),
graph_put,
),
ToolSpec(
"graph_remove",
"Remove items from your working graph by id (a not-applicable entry by "
"its recommendation id).",
_schema({"ids": {"type": "array", "items": {"type": "string"}}}, ["ids"]),
graph_remove,
),
ToolSpec(
"graph_read",
"Read your working graph.",
{"type": "object", "properties": {}},
graph_read,
),
]
def _counts(graph: Mapping[str, Sequence[Any]]) -> str:
return (
f"The graph has {len(graph['elements'])} Elements, "
f"{len(graph['relationships'])} Relationships and "
f"{len(graph['checks'])} Checks."
)
def summary(working: Mapping[str, Any]) -> dict[str, Any] | None:
"""What a Specialist is told it already has, at the start of a run."""
told: dict[str, Any] = {}
files = working.get(FILES)
if isinstance(files, Mapping) and files:
told["files"] = sorted(files)
graph = working.get(GRAPH)
if isinstance(graph, Mapping) and any(graph.get(name) for name in _GRAPH_LISTS):
told["graph"] = {name: len(graph.get(name) or ()) for name in _GRAPH_LISTS}
return told or None

View File

@@ -242,20 +242,51 @@ def _covers_every_check(
return problems
def _split_implementation(text: str, name: str) -> tuple[str, str]:
"""The entries that say where a Check is met, and those that say it cannot be."""
try:
data = json.loads(text)
except ValueError:
return text, "[]"
if not isinstance(data, list):
return text, "[]"
cannot = [
item for item in data if isinstance(item, dict) and "unimplementable" in item
]
met = [item for item in data if item not in cannot]
return json.dumps(met), json.dumps(cannot)
def check_implementation(
attachments: Mapping[str, str], upstream: Upstream | None
) -> list[str]:
name = IMPLEMENTATION_FILE
items, problems = _objects(attachments[name], name, ("check", "file", "address"))
problems += _covers_every_check(items, name, read_graph(upstream))
met_text, cannot_text = _split_implementation(attachments[name], name)
met, problems = _objects(met_text, name, ("check", "file", "address"))
cannot, faults = _objects(cannot_text, name, ("check", "unimplementable"))
problems += faults
problems += _covers_every_check([*met, *cannot], name, read_graph(upstream))
problems += [
f"{name}: {item['check']} names {item['file']}, which is not attached"
for item in items
for item in met
if item["file"] not in attachments
]
return problems
def unimplementable_checks(implementation_text: str) -> list[dict[str, str]]:
"""The Checks the code's author found cannot be met as written, with the reason.
Each is a fault in the Target architecture, not in the code: the Check names
something the resource does not have.
"""
_, cannot_text = _split_implementation(implementation_text, IMPLEMENTATION_FILE)
cannot, _ = _objects(cannot_text, IMPLEMENTATION_FILE, ("check", "unimplementable"))
return [
{"check": item["check"], "reason": item["unimplementable"]} for item in cannot
]
def check_verdicts(
attachments: Mapping[str, str], upstream: Upstream | None
) -> list[str]:

View File

@@ -64,6 +64,14 @@ class DeliverableSpec:
depends_on: tuple[DeliverableKind, ...] = ()
sections: tuple[str, ...] = ()
attachments: tuple[str, ...] = ()
# Which earlier Deliverables the Specialist reads whole. None means the ones it
# depends on. Every other input arrives as its data file alone: the ids the
# Requirements chain runs through, without the prose written around them.
reads_in_full: tuple[DeliverableKind, ...] | None = None
# The Deliverable whose files this Specialist's tools work on. Its own kind: the
# files are its working copy, kept across attempts and attached on submission.
# Another kind: a fresh copy of that Deliverable's files on every attempt.
working_copy: DeliverableKind | None = None
def contract(self) -> dict[str, list[str]]:
"""What a submission must contain, as plain data for the Specialist's context."""
@@ -126,8 +134,10 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
PhaseId.SOLUTION_ARCHITECTURE,
"target-architect",
depends_on=(_K.PRODUCT_SELECTION, _K.DESIGN_RECOMMENDATIONS),
sections=("Elements", "Relationships", "Checks"),
# The engine adds architecture.mmd, rendered from the graph, on submission.
# The Specialist writes the reasoning and the graph. On submission the
# engine adds the Elements, Relationships and Checks tables to the content
# and architecture.mmd to the attachments, both rendered from the graph.
sections=("Design rationale",),
attachments=(GRAPH_ATTACHMENT,),
),
DeliverableSpec(
@@ -151,6 +161,9 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
"terraform/variables.tf",
IMPLEMENTATION_FILE,
),
# The graph is all the code is written from.
reads_in_full=(),
working_copy=_K.DEPLOYMENT_GUIDANCE,
),
DeliverableSpec(
_K.VALIDATION_RESULTS,
@@ -159,6 +172,7 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
depends_on=(_K.DEPLOYMENT_GUIDANCE,),
sections=("Deployment dry-run", "Architecture and policy analysis"),
attachments=(VERDICTS_FILE,),
working_copy=_K.DEPLOYMENT_GUIDANCE,
),
DeliverableSpec(
_K.SOLUTION_GUIDE,
@@ -175,6 +189,8 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
"Validation plan",
"References",
),
# The guide is assembled from every Deliverable's prose.
reads_in_full=tuple(_K)[:-1],
),
)
}
@@ -218,6 +234,31 @@ def inputs_of(kind: DeliverableKind) -> tuple[DeliverableKind, ...]:
)
def dependencies_for(
kind: DeliverableKind, deliverables: Mapping[str, Mapping[str, object]]
) -> dict[str, dict[str, object]]:
"""What a Specialist is given of each Deliverable it works from.
A Deliverable read in full arrives as submitted. Any other arrives as its data
file alone, which is all the Requirements chain needs of it.
"""
spec = DEPENDENCY_GRAPH[kind]
in_full = spec.depends_on if spec.reads_in_full is None else spec.reads_in_full
given: dict[str, dict[str, object]] = {}
for dependency in inputs_of(kind):
deliverable = deliverables[dependency.value]
if dependency in in_full or dependency not in DATA_CHECKS:
given[dependency.value] = dict(deliverable)
continue
name = DATA_CHECKS[dependency][0]
attachments = deliverable.get("attachments")
files = attachments if isinstance(attachments, Mapping) else {}
given[dependency.value] = {
"attachments": {name: files[name]} if name in files else {}
}
return given
def deliverables_in(phase: PhaseId) -> tuple[DeliverableKind, ...]:
return tuple(kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase)

View File

@@ -20,6 +20,7 @@ from langgraph.graph import END, START, StateGraph
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory
from app.nodes.execution_nodes import make_execution_nodes
from app.nodes.router import OrchestratorPolicy
from app.recorder import ExecutionRecorder
from app.states.execution_state import ExecutionState
@@ -34,6 +35,7 @@ def build_execution_graph(
tool_factory: ToolFactory | None = None,
limiter: RunLimiter | None = None,
skill_versions: Mapping[str, str] | None = None,
orchestrator: OrchestratorPolicy | None = None,
) -> StateGraph:
nodes = make_execution_nodes(
runtime=runtime,
@@ -44,6 +46,7 @@ def build_execution_graph(
tool_factory=tool_factory,
limiter=limiter,
skill_versions=skill_versions,
orchestrator=orchestrator,
)
graph = StateGraph(ExecutionState)

View File

@@ -6,7 +6,8 @@
to the requirement or design recommendation it comes from
This module is pure: it parses the graph, states what is wrong with it, renders it as
a diagram, and says whether the Requirements chain through it is complete.
a diagram and as tables a person can review, and says whether the Requirements chain
through it is complete.
"""
from __future__ import annotations
@@ -281,3 +282,64 @@ def render_mermaid(graph: TargetArchitecture) -> str:
f"{nodes[relationship.target]}"
)
return "\n".join(lines) + "\n"
# --- so are the tables a person reviews ----------------------------------------------
def _cell(text: str | None) -> str:
return " ".join((text or "").replace("|", "\\|").split())
def _table(
heading: str, columns: Sequence[str], rows: Iterable[Sequence[str | None]]
) -> str:
lines = [
f"## {heading}",
"",
"| " + " | ".join(columns) + " |",
"|" + "|".join(" --- " for _ in columns) + "|",
]
lines += ["| " + " | ".join(_cell(value) for value in row) + " |" for row in rows]
return "\n".join(lines)
def render_tables(graph: TargetArchitecture) -> str:
"""The graph as markdown tables: what a person approving the design reads.
Written by code, so the tables cannot disagree with the graph and no model spends
output repeating it.
"""
names = {element.id: element.name for element in graph.elements}
parts = [
_table(
"Elements",
("Id", "Name", "Product", "Resource type", "Component"),
(
(e.id, e.name, e.product, e.resource_type, e.component)
for e in graph.elements
),
),
_table(
"Relationships",
("Id", "From", "Kind", "To"),
(
(r.id, names[r.source], r.kind, names[r.target])
for r in graph.relationships
),
),
_table(
"Checks",
("Id", "Applies to", "Statement", "Source"),
((c.id, c.applies_to, c.statement, c.source) for c in graph.checks),
),
]
if graph.not_applicable:
parts.append(
_table(
"Not applicable",
("Recommendation", "Reason"),
((n.recommendation, n.reason) for n in graph.not_applicable),
)
)
return "\n\n".join(parts) + "\n"

View File

@@ -13,7 +13,7 @@ An Execution is a hierarchy of ReAct agents shaped by the dependencies between D
The dependency graph is declared as data in code. The Orchestrator picks what to do next through tools that refuse work whose dependencies aren't met. Dependencies are guaranteed by code, while ordering and parallelism are the model's choice.
> **Under review (2026-10-07).** [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), proposed, would have the Orchestrator decide with Jev and run no language model. If accepted, it replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected.
> **Amended (2026-10-07) by [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), accepted.** The Orchestrator is now a router: it decides by readings and a playbook in code, with Jev for the judgements, and runs no language model to choose an action. That replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected and still reason on language models.
## Considered Options

View File

@@ -1,8 +1,8 @@
---
status: proposed
status: accepted
---
# Jev is the Orchestrator's decision point
# Jev is the Orchestrator's decision point, and the Orchestrator is a router
[ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev ([Jev in AI agent loops](https://vercel.com/i/jev-agent-control)): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.
@@ -17,28 +17,53 @@ status: proposed
| Update state | Application code | The Execution graph and its checkpoint |
| Explain the result | A generative model | One small model call that writes an Approval summary or a question, from data it is given |
## How a turn is decided
## The Orchestrator is a router
1. **Code lists the allowed actions** from the Ledger and the Execution's state: relay waiting questions, act on a validation outcome, request an Approval, delegate ready work, finish. This is the permission check, and it holds whatever Jev answers, because a typed decision can still be wrong.
2. **Jev proposes the next action** from that list, and answers the other closed questions the turn needs, in one request where they rest on the same evidence:
- Did the user's last message ask for a change to submitted work? (yes or no)
- Which Deliverable does that change affect? (one of the Deliverables)
- What should happen after this validation: revise the code, revise the Target architecture, or proceed? (one of three)
A question whose options depend on another answer, or on evidence not yet fetched, is a second request.
3. **Code carries the decision out** through the existing tools and records it: the questions, the answers, their probabilities and what was done. A resumed Execution reuses a recorded decision when its input is unchanged.
4. **A generative model writes the reply**, only where the user reads prose: the summary they approve a Phase from, and any question the Orchestrator itself asks. It is given the decision and the data and decides nothing.
With Jev as its decision point, the Orchestrator's job is routing: it reads what has arrived and the state of the Execution, asks Jev which route applies, and dispatches. It does no work of its own. There are four routes to choose.
Two guards from the pattern are kept in code. Repeated actions are limited, so the Orchestrator cannot loop on the same step: the turn limit and the Revision limit already do this. A failed call to Jev is recorded as a failure, distinct from an answer, so "Jev was unreachable" is never read as "nothing to do".
| What is routed | The routes Jev chooses between | Who is dispatched |
|---|---|---|
| A message that starts something | a new Workflow request; a Change request against a Baseline; a question about work already delivered; not something this agent does | a full Execution; an Execution from the Baseline; the reply writer, from the delivered files; a polite refusal |
| A message inside an Execution | an answer to a waiting question; a request to change submitted work, and which Deliverable; anything else | the Specialist that asked; a Revision of that Deliverable; a question back to the user |
| A validation outcome | revise the code; revise the Target architecture; proceed | the IaC author; the target architect; the Approval |
| Ready work, in a Change request | which Deliverables the change reaches, and which carry over from the Baseline unchanged | only the Specialists whose Deliverables are reached |
## Decisions to confirm
Which Specialist produces a given Deliverable is still fixed by the dependency graph: Jev chooses the route, never the worker for it. And the Ledger still refuses a route whose Dependencies or Approvals are not met.
This ADR is proposed. Accepting it means agreeing to each of these:
## How a turn is decided: readings, disposition, playbook
1. **No language model decides what the Orchestrator does.** Jev proposes and code disposes. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
The owner's model for this is a four-stage pipeline: readings of what arrived, a disposition worked out in code, further readings of the state, and a playbook that takes the first group that applies and, within it, the strongest action whose conditions hold, escalating when no group applies.
1. **Readings.** Facts about the turn. Most are read from the Ledger and the Execution's data by code: are questions waiting, is a Phase complete, is work ready, which Checks failed, did a Check fail twice running. The ones that need judgement are typed answers from Jev: is the user asking for a change (yes or no), which Deliverable would have to change first (a choice), how specific are they (a score), does a gap in the Target architecture leave a requirement unmet (yes or no). Jev reads; it never picks the action.
2. **Disposition.** Code turns a judged reading into act, queue or close by threshold: a probability of 0.75 or more is acted on, 0.15 or less is dropped, and anything between is put to the user. A choice with confidence under 0.6 is not relied on. The thresholds are settings.
3. **Playbook.** Ordered groups; the first that applies is taken, and within it the strongest action whose conditions hold:
| Group | Actions, strongest first |
|---|---|
| The user asked for a change | revise the Deliverable it affects; else ask them to confirm which and what |
| Questions are waiting | relay them |
| Validation decided | revise the Target architecture (a Check failed twice running, or a gap that matters); else revise the code (a Check failed); else ask about the gaps; else proceed |
| A Phase is complete | request its Approval |
| Work is ready | delegate all of it |
| Packaging is approved | finish |
| No group applies | escalate: ask the user, or end with warnings when there is none |
4. **Permission.** Every action goes through the Orchestrator's tools, so the Ledger refuses what is not allowed whatever was decided, and the playbook then takes the next action that is. A typed answer can still be wrong.
5. **Record.** Each decision is kept in the Execution's state: the turn, the group, the action and what Jev answered with its confidence.
6. **Reply.** A generative model writes only what the user reads, the summary they approve a Phase from, out of data it is given. It runs after the decision and cannot change it. When it fails, the summary is assembled from the same data.
A failed call to Jev is not an answer. The client returns nothing, and the reading falls back to a rule that acts only on what is certain: a change is acted on when the Deliverable is named outright, and a gap is revised for when it names a real requirement or cannot be verified.
## Decisions
Accepted by the owner on 2026-10-07, with each of these:
1. **No language model decides what the Orchestrator does.** Jev reads, and the playbook in code decides. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
2. **Low confidence goes to the user, not to another model.** Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
3. **Jev being unreachable does not stop an Execution.** After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
4. **What is sent to Jev is the Execution's state, not its content:** Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
5. **Jev does not choose Specialists.** Which Specialist produces a Deliverable is fixed by the dependency graph, and all ready work is delegated together. The pattern's specialist routing has nothing to route here.
5. **The Orchestrator routes; Jev chooses the route.** That covers what kind of work a message starts, where a message inside an Execution goes, where a validation outcome goes, and, in a Change request, which Deliverables are reached. Jev never chooses which Specialist produces a Deliverable: the dependency graph fixes that.
6. **In a new Execution every Deliverable is still produced.** Routing does not skip a Specialist for a new Workflow request, because each later Deliverable is checked against the ones before it. Skipping happens only in a Change request, where the Baseline supplies what is carried over.
## Considered Options
@@ -49,6 +74,9 @@ This ADR is proposed. Accepting it means agreeing to each of these:
## Consequences
- Jev is reached through TypeSafe AI's Python SDK, `typesafe-sdk`, pinned exactly (owner, 2026-10-07). `app/tools/jev.py` is the only module that imports it.
- The Orchestrator as a language-model agent is kept behind the setting `ORCHESTRATOR=model`, for the tests that drive the Ledger's rules in arbitrary order and for comparing the two in an evaluation. The default is `router`. Remove the setting once the router is proven.
- The Orchestrator's instructions in `agents/orchestrator.md` become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.

View File

@@ -1,8 +1,11 @@
"""Judges: graded answers about a Deliverable that string matching cannot give.
A judge takes a document and a set of questions and returns a score from 0 to 1 per
question. The shape (one state, many typed questions) is deliberately the one a
decision model takes, so the Gemini judge here can be swapped for one later.
question. The shape is the one a decision model takes: one state, many typed
questions, answered in a single call. `DeciderJudge` puts Jev in that seat, which is
the judge to prefer for narrow, typed criteria at volume: it is cheap enough to ask
every question of every run and returns the same answer for the same input.
`GeminiJudge` is a language-model judge for comparison, and for when no Jev key is set.
"""
from __future__ import annotations
@@ -11,6 +14,8 @@ import json
from collections.abc import Mapping
from typing import Any, Protocol
from app.adk.runtime import Decider, Question
INSTRUCTIONS = (
"You grade a document from a cloud solution architecture engagement. Answer every "
"question about the document only, strictly and without benefit of the doubt. "
@@ -31,6 +36,48 @@ class Judge(Protocol):
...
GRADES = ("not at all", "weakly", "mostly", "fully")
class JudgeUnavailable(RuntimeError):
"""The judge gave no answer; the check it serves is unanswered, not failed."""
class DeciderJudge:
"""Jev as the judge. The document is the state and carries no grading
instructions; each question is asked on its own terms, all in one call."""
def __init__(self, decider: Decider, scope: str = "evaluation") -> None:
self._decider = decider
self._scope = scope
async def answer(
self, document: str, questions: Mapping[str, Mapping[str, str]]
) -> Mapping[str, float]:
asked = {
key: Question("score", item["text"], GRADES)
if item["kind"] == "graded"
else Question("noul", item["text"])
for key, item in questions.items()
}
answers = await self._decider.decide(document, asked, scope=self._scope)
if answers is None:
raise JudgeUnavailable("Jev gave no answer")
scores = {}
for key, question in asked.items():
value = float(answers[key].value)
# A graded answer is a level from 0 to 3; a yes-or-no answer is the
# probability of yes, counted as a yes at an even chance or better.
scores[key] = (
min(1.0, max(0.0, value / (len(GRADES) - 1)))
if question.kind == "score"
else 1.0
if value >= 0.5
else 0.0
)
return scores
class GeminiJudge:
"""A judge on the Gemini API or Vertex AI, chosen by the environment as ADK does."""

View File

@@ -40,7 +40,7 @@ from app.workflows.target_architecture import (
parse_graph,
render_mermaid,
)
from eval.judge import Judge
from eval.judge import Judge, JudgeUnavailable
PASS_THRESHOLD = 0.8
CHECKLIST_COVERAGE = 0.7 # share of a pillar Skill's checklist the design must address
@@ -403,7 +403,10 @@ async def _checklist_coverage(
}
if not recommendations or not questions:
return Check(name, False, "no design recommendations or no checklists")
scores = await judge.answer(recommendations, questions)
try:
scores = await judge.answer(recommendations, questions)
except JudgeUnavailable as exc:
return Check(name, False, f"judge unavailable: {exc}")
per_pillar = {
pillar: sum(scores.get(f"{pillar}.{i}", 0) for i in range(len(items)))
/ len(items)
@@ -419,10 +422,16 @@ async def _quality(judge: Judge, recommendations: str) -> Check:
name = "recommendations_quality"
if not recommendations:
return Check(name, False, "no design recommendations")
scores = await judge.answer(
recommendations,
{key: {"kind": "graded", "text": text} for key, text in QUALITY_RUBRIC.items()},
)
try:
scores = await judge.answer(
recommendations,
{
key: {"kind": "graded", "text": text}
for key, text in QUALITY_RUBRIC.items()
},
)
except JudgeUnavailable as exc:
return Check(name, False, f"judge unavailable: {exc}")
mean = sum(scores.values()) / len(QUALITY_RUBRIC)
detail = ", ".join(f"{key} {scores.get(key, 0):.2f}" for key in QUALITY_RUBRIC)
return Check(name, mean >= QUALITY_BAR, f"mean {mean:.2f}: {detail}")

View File

@@ -8,8 +8,9 @@ Usage:
GOOGLE_GENAI_USE_VERTEXAI=true GOOGLE_CLOUD_PROJECT=<project> GOOGLE_CLOUD_LOCATION=global \
.venv/bin/python -m eval.solution_eval [--judge] [case-id ...]
--judge adds the judged checks (pillar checklist coverage, recommendation quality),
using the provider bundle's default model unless EVAL_JUDGE_MODEL names another.
--judge adds the judged checks (pillar checklist coverage, recommendation quality).
The judge is Jev when TYPESAFE_API_KEY is set; otherwise, or with EVAL_JUDGE=gemini,
a language model: the provider bundle's default unless EVAL_JUDGE_MODEL names another.
A case marked "interactive" runs with a scripted user in the loop. The exit code is
non-zero when a case fails or exceeds one of its thresholds.
"""
@@ -36,7 +37,7 @@ from app.nodes.execution_nodes import automatic_reply
from app.recorder import InMemoryRecorder
from app.states.execution_state import execution_dir
from app.tools.local import TerraformRunner
from eval.judge import GeminiJudge, Judge
from eval.judge import DeciderJudge, GeminiJudge, Judge
from eval.scoring import (
CaseResult,
CaseSpec,
@@ -209,6 +210,8 @@ async def run_case(
"agent_runs": dict(runtime.runs),
"outcomes": dict(runtime.outcomes),
"revisions": revisions,
# Every routing decision: the group that applied, the action, what was judged.
"decisions": interaction.state.get("decisions", []),
"warnings": interaction.state.get("warnings", []),
"conversation": interaction.state.get("conversation", []),
"output_dir": str(execution_dir(run_dir, interaction.execution_id, case.id)),
@@ -217,8 +220,13 @@ async def run_case(
def make_judge(settings: Settings) -> Judge:
"""Jev when a TypeSafe key is set, unless EVAL_JUDGE=gemini asks for the
language-model judge; otherwise the language-model judge."""
from app.agent import open_decider
from app.providers import load_provider
if settings.TYPESAFE_API_KEY and os.environ.get("EVAL_JUDGE") != "gemini":
return DeciderJudge(open_decider(settings))
model = os.environ.get("EVAL_JUDGE_MODEL") or load_provider(
settings.CLOUD_PROVIDER
).models.default.removeprefix(GEMINI_PREFIX)

View File

@@ -21,6 +21,13 @@ models:
target-architect: gemini/gemini-3.1-pro-preview
iac-author: gemini/gemini-3.1-pro-preview
validator: gemini/gemini-3.1-pro-preview
# Seconds one model request may take, for roles whose replies are long. Other
# roles get MODEL_TIMEOUT_SECONDS. The target architect timed out at 180 s in both
# attempts of the 2026-10-07 run.
timeout_seconds:
target-architect: 420
iac-author: 300
packager: 300
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
role_skills:
@@ -29,9 +36,9 @@ role_skills:
decomposer: [google-cloud-solution-architecture, requirements-chain]
product-selector: [google-cloud-solution-architecture, requirements-chain]
design-advisor: [google-cloud-solution-architecture, requirements-chain]
target-architect: [google-cloud-solution-architecture, requirements-chain]
describer: [google-cloud-solution-architecture]
iac-author: [google-cloud-solution-architecture, requirements-chain, iac-validation]
target-architect: [google-cloud-solution-architecture, requirements-chain]
iac-author: [google-cloud-solution-architecture, requirements-chain, iac-validation, google-cloud-iac-patterns]
validator: [google-cloud-solution-architecture, requirements-chain, iac-validation]
packager: [google-cloud-solution-architecture]
@@ -45,9 +52,18 @@ pillar_skills:
sustainability: google-cloud-waf-sustainability
tools:
general: [mermaid_check, terraform_fmt, terraform_validate]
general: [mermaid_check, terraform_fmt, terraform_validate, terraform_schema]
cloud_read: [terraform_plan]
# The Terraform providers generated code is written for. One exact version, so the
# schema lookup, the patterns Skill and validation all describe the same provider.
# The 2026-10-07 run left five different versions in the plugin cache.
terraform:
required_providers:
google:
source: hashicorp/google
version: 8.6.0
mcp_auth:
kind: google_adc # Application Default Credentials bearer token; sends X-Goog-User-Project

View File

@@ -14,3 +14,4 @@ psycopg2-binary>=2.9.9
google-adk[extensions]==2.9.0 # Skills are Experimental: pin exactly; extensions brings LiteLLM
mcp>=1.24,<2
a2a-sdk[http-server]>=1.1,<2
typesafe-sdk==0.7.2 # Jev decisions (ADR-0007); a young SDK, so pinned exactly

View File

@@ -0,0 +1,49 @@
---
name: google-cloud-iac-patterns
description: >-
Terraform patterns for Google Cloud products that validate against one pinned
version of the hashicorp/google provider: Cloud Run services and jobs,
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
networking, Cloud Scheduler, Artifact Registry, Monitoring and API
enablement. Use when writing Terraform for a Google Cloud product, before
writing the resource, to start from a pattern whose arguments are known to be
in the right place.
metadata:
scope: deployment guidance
provider: hashicorp/google 8.6.0
---
# Google Cloud IaC patterns
Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0. A test runs that on each of them, so an argument you copy from a pattern is in the right block.
## How to use them
1. Write `assets/versions.tf` to your working copy as `terraform/versions.tf`, unchanged. It pins the provider version these patterns and the `terraform_schema` tool describe. Declare the `project_id` and `region` variables it uses in `terraform/variables.tf`.
2. For each Element in the Target architecture, read the reference for its product with `load_skill_resource` and start from its pattern. Read only the references for products the Target architecture uses.
3. Change the pattern to meet the Element's Checks. For a setting the pattern does not show, call `terraform_schema` with `find` set to its name before you write it.
| Product | Reference |
| --- | --- |
| Enabling APIs | `references/project-services.md` |
| Service accounts and IAM grants | `references/iam.md` |
| Cloud Run service | `references/cloud-run-service.md` |
| Cloud Run job | `references/cloud-run-job.md` |
| Pub/Sub | `references/pubsub.md` |
| Cloud Storage | `references/cloud-storage.md` |
| Secret Manager | `references/secret-manager.md` |
| Cloud KMS | `references/kms.md` |
| Cloud SQL | `references/cloud-sql.md` |
| VPC networking | `references/networking.md` |
| Cloud Scheduler | `references/cloud-scheduler.md` |
| Artifact Registry | `references/artifact-registry.md` |
| Cloud Monitoring and Logging | `references/monitoring.md` |
A product with no reference has no pattern yet: write it from `terraform_schema` and the product documentation.
## Rules that hold for every product
- A Cloud Run service and a Cloud Run job are different resource types with different arguments. A setting of one is not a setting of the other. `references/cloud-run-job.md` lists what a job lacks.
- Grant a role with an `_iam_member` resource. An `_iam_binding` or `_iam_policy` resource removes grants made elsewhere.
- Give a Google-managed service agent its role before the resource that needs it, with `depends_on`. A key, a dead-letter topic and a push subscription all need one.
- Set `deletion_protection` from a variable on the resources that have it, so a test deployment can be removed.

View File

@@ -0,0 +1,15 @@
terraform {
required_version = ">= 1.9"
required_providers {
google = {
source = "hashicorp/google"
version = "8.6.0"
}
}
}
provider "google" {
project = var.project_id
region = var.region
}

View File

@@ -0,0 +1,65 @@
# Artifact Registry
A Docker repository with a cleanup policy, and the grant that lets a runtime identity pull from it.
- `format = "DOCKER"` for container images. The image path is `REGION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE`.
- Cleanup policies delete nothing while `cleanup_policy_dry_run` is `true`.
- `docker_config.immutable_tags = true` stops a tag being moved to a different image.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
resource "google_service_account" "runtime" {
project = var.project_id
account_id = "image-puller"
display_name = "Pulls workload images"
}
resource "google_artifact_registry_repository" "images" {
project = var.project_id
location = var.region
repository_id = "workload-images"
format = "DOCKER"
description = "Container images for the workload."
cleanup_policy_dry_run = false
docker_config {
immutable_tags = true
}
cleanup_policies {
id = "keep-recent"
action = "KEEP"
most_recent_versions {
keep_count = 10
}
}
cleanup_policies {
id = "delete-old-untagged"
action = "DELETE"
condition {
tag_state = "UNTAGGED"
older_than = "2592000s"
}
}
}
resource "google_artifact_registry_repository_iam_member" "runtime_pulls" {
project = var.project_id
location = google_artifact_registry_repository.images.location
repository = google_artifact_registry_repository.images.name
role = "roles/artifactregistry.reader"
member = google_service_account.runtime.member
}
```

View File

@@ -0,0 +1,95 @@
# Cloud Run job
`google_cloud_run_v2_job`: a task that runs to completion. It is a different resource type from a service, and its template is nested one level deeper.
What a job does not have. A Check that asks for one of these on a job cannot be met:
- `startup_cpu_boost` and `cpu_idle`. A job's `resources` block has `limits` only.
- `ingress`, `traffic`, `scaling`, `min_instance_count`, `max_instance_request_concurrency`. A job answers no requests.
- `liveness_probe`. A job's container has a `startup_probe` only.
Where a job's settings sit:
- `task_count` and `parallelism` are in `template`.
- `service_account`, `timeout`, `max_retries`, `execution_environment`, `vpc_access` and `encryption_key` are in `template.template`.
- `image`, `env`, `args`, `command` and `resources` are in `template.template.containers`.
- Whoever starts the job needs `roles/run.invoker` on it; Cloud Scheduler starts it with an HTTP call to the `:run` method.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "image" {
type = string
description = "Container image reference, including the tag or digest."
}
variable "deletion_protection" {
type = bool
description = "Whether Terraform is prevented from deleting the job."
default = true
}
resource "google_service_account" "replay" {
project = var.project_id
account_id = "replay-job"
display_name = "Replay job"
}
resource "google_cloud_run_v2_job" "replay" {
project = var.project_id
name = "replay"
location = var.region
deletion_protection = var.deletion_protection
template {
task_count = 1
parallelism = 1
template {
service_account = google_service_account.replay.email
timeout = "3600s"
max_retries = 3
execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
containers {
image = var.image
args = ["--mode=replay"]
resources {
limits = {
cpu = "1"
memory = "1Gi"
}
}
env {
name = "BATCH_SIZE"
value = "100"
}
}
}
}
}
resource "google_service_account" "operator" {
project = var.project_id
account_id = "replay-operator"
display_name = "Starts the replay job"
}
resource "google_cloud_run_v2_job_iam_member" "operator_runs_replay" {
project = var.project_id
location = google_cloud_run_v2_job.replay.location
name = google_cloud_run_v2_job.replay.name
role = "roles/run.invoker"
member = google_service_account.operator.member
}
```

View File

@@ -0,0 +1,144 @@
# Cloud Run service
`google_cloud_run_v2_service`: a long-running service that answers requests.
Where the settings sit, which is what goes wrong most often:
- `ingress`, `deletion_protection` and `traffic` are on the resource.
- `service_account`, `scaling`, `timeout`, `max_instance_request_concurrency`, `execution_environment`, `vpc_access` and `encryption_key` are in `template`.
- `image`, `env`, `ports`, `resources` and the probes are in `template.containers`.
- `cpu_idle` and `startup_cpu_boost` are in `template.containers.resources`, beside `limits`. They exist on a service only.
- `min_instance_count` and `max_instance_count` are in `template.scaling`.
- A secret is read through `env.value_source.secret_key_ref`; the service account needs `roles/secretmanager.secretAccessor` on that secret.
- Direct VPC egress is `template.vpc_access.network_interfaces`. A connector is `template.vpc_access.connector`. Use one of them.
- Public access is a grant of `roles/run.invoker` to `allUsers` together with `ingress = "INGRESS_TRAFFIC_ALL"`. A private service has neither.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "image" {
type = string
description = "Container image reference, including the tag or digest."
}
variable "deletion_protection" {
type = bool
description = "Whether Terraform is prevented from deleting the service."
default = true
}
resource "google_service_account" "ingest" {
project = var.project_id
account_id = "ingest-service"
display_name = "Ingest service"
}
resource "google_secret_manager_secret" "signing_key" {
project = var.project_id
secret_id = "webhook-signing-key"
replication {
auto {}
}
}
resource "google_secret_manager_secret_iam_member" "ingest_reads_signing_key" {
project = var.project_id
secret_id = google_secret_manager_secret.signing_key.secret_id
role = "roles/secretmanager.secretAccessor"
member = google_service_account.ingest.member
}
resource "google_cloud_run_v2_service" "ingest" {
project = var.project_id
name = "ingest"
location = var.region
ingress = "INGRESS_TRAFFIC_ALL"
deletion_protection = var.deletion_protection
template {
service_account = google_service_account.ingest.email
timeout = "30s"
max_instance_request_concurrency = 80
execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
scaling {
min_instance_count = 1
max_instance_count = 10
}
containers {
image = var.image
ports {
container_port = 8080
}
resources {
limits = {
cpu = "1"
memory = "512Mi"
}
cpu_idle = true
startup_cpu_boost = true
}
env {
name = "LOG_LEVEL"
value = "info"
}
env {
name = "SIGNING_KEY"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.signing_key.secret_id
version = "latest"
}
}
}
startup_probe {
initial_delay_seconds = 0
period_seconds = 5
failure_threshold = 6
http_get {
path = "/healthz"
}
}
liveness_probe {
period_seconds = 30
http_get {
path = "/healthz"
}
}
}
}
traffic {
type = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
percent = 100
}
depends_on = [google_secret_manager_secret_iam_member.ingest_reads_signing_key]
}
# Public access. Leave this out, and set ingress to INGRESS_TRAFFIC_INTERNAL_ONLY,
# for a service only other workloads call.
resource "google_cloud_run_v2_service_iam_member" "public" {
project = var.project_id
location = google_cloud_run_v2_service.ingest.location
name = google_cloud_run_v2_service.ingest.name
role = "roles/run.invoker"
member = "allUsers"
}
```

View File

@@ -0,0 +1,75 @@
# Cloud Scheduler
A schedule that starts a Cloud Run job, and one that calls a private Cloud Run service.
- A job is started by an HTTP POST to its `:run` method with an OAuth token (`oauth_token`). A service is called with an OIDC token (`oidc_token`). Using the wrong one is refused at run time, not by `terraform validate`.
- The schedule's service account needs `roles/run.invoker` on the job or service it starts.
- `schedule` is unix-cron; `time_zone` is an IANA name.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "job_name" {
type = string
description = "Name of the Cloud Run job to start."
}
variable "service_url" {
type = string
description = "HTTPS URL of the private Cloud Run service to call."
}
resource "google_service_account" "scheduler" {
project = var.project_id
account_id = "nightly-scheduler"
display_name = "Identity of the nightly schedules"
}
resource "google_cloud_scheduler_job" "start_job" {
project = var.project_id
region = var.region
name = "start-nightly-job"
schedule = "0 2 * * *"
time_zone = "Etc/UTC"
attempt_deadline = "180s"
retry_config {
retry_count = 3
}
http_target {
http_method = "POST"
uri = "https://run.googleapis.com/v2/projects/${var.project_id}/locations/${var.region}/jobs/${var.job_name}:run"
oauth_token {
service_account_email = google_service_account.scheduler.email
}
}
}
resource "google_cloud_scheduler_job" "call_service" {
project = var.project_id
region = var.region
name = "call-nightly-service"
schedule = "30 2 * * *"
time_zone = "Etc/UTC"
http_target {
http_method = "POST"
uri = "${var.service_url}/tasks/nightly"
oidc_token {
service_account_email = google_service_account.scheduler.email
audience = var.service_url
}
}
}
```

View File

@@ -0,0 +1,124 @@
# Cloud SQL
A PostgreSQL instance with a private address, backups and IAM database authentication.
- A private address needs a reserved range and a service networking connection on the VPC first; the instance must `depends_on` the connection.
- `availability_type = "REGIONAL"` is high availability; `"ZONAL"` is a single zone.
- Backups, point-in-time recovery, flags and the address are all in `settings`.
- `deletion_protection` on the resource guards Terraform; `settings.deletion_protection_enabled` guards the API.
- Take the database password from Secret Manager at run time, or use IAM database authentication as here. Never put a password in the configuration.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "deletion_protection" {
type = bool
description = "Whether Terraform is prevented from deleting the instance."
default = true
}
resource "google_compute_network" "main" {
project = var.project_id
name = "main"
auto_create_subnetworks = false
}
resource "google_compute_global_address" "private_services" {
project = var.project_id
name = "private-services"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
network = google_compute_network.main.id
}
resource "google_service_networking_connection" "private_services" {
network = google_compute_network.main.id
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [google_compute_global_address.private_services.name]
}
resource "google_service_account" "app" {
project = var.project_id
account_id = "orders-app"
display_name = "Orders application"
}
resource "google_sql_database_instance" "orders" {
project = var.project_id
name = "orders"
region = var.region
database_version = "POSTGRES_16"
deletion_protection = var.deletion_protection
settings {
tier = "db-custom-2-7680"
edition = "ENTERPRISE"
availability_type = "REGIONAL"
disk_type = "PD_SSD"
disk_autoresize = true
deletion_protection_enabled = var.deletion_protection
ip_configuration {
ipv4_enabled = false
private_network = google_compute_network.main.id
ssl_mode = "ENCRYPTED_ONLY"
}
backup_configuration {
enabled = true
point_in_time_recovery_enabled = true
start_time = "02:00"
backup_retention_settings {
retained_backups = 14
}
}
database_flags {
name = "cloudsql.iam_authentication"
value = "on"
}
maintenance_window {
day = 7
hour = 3
}
}
depends_on = [google_service_networking_connection.private_services]
}
resource "google_sql_database" "orders" {
project = var.project_id
name = "orders"
instance = google_sql_database_instance.orders.name
}
resource "google_sql_user" "app" {
project = var.project_id
name = trimsuffix(google_service_account.app.email, ".gserviceaccount.com")
instance = google_sql_database_instance.orders.name
type = "CLOUD_IAM_SERVICE_ACCOUNT"
}
resource "google_project_iam_member" "app_connects" {
project = var.project_id
role = "roles/cloudsql.client"
member = google_service_account.app.member
}
resource "google_project_iam_member" "app_logs_in" {
project = var.project_id
role = "roles/cloudsql.instanceUser"
member = google_service_account.app.member
}
```

View File

@@ -0,0 +1,67 @@
# Cloud Storage
A private bucket with a retention period and a lifecycle.
- Bucket names are global. Build the name from the project id.
- `uniform_bucket_level_access = true` and `public_access_prevention = "enforced"` keep the bucket private.
- `retention_policy.retention_period` is in seconds and stops objects being deleted before then. A lifecycle `Delete` rule with `age` in days removes them after. Retaining for 30 days and then deleting needs both.
- Versioning and a retention policy cannot be used on the same bucket.
- Customer-managed encryption is `encryption.default_kms_key_name`; the Cloud Storage service agent needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key first.
- `force_destroy = false` stops Terraform deleting a bucket that holds objects.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
resource "google_service_account" "writer" {
project = var.project_id
account_id = "raw-events-writer"
display_name = "Writes raw events"
}
resource "google_storage_bucket" "raw_events" {
project = var.project_id
name = "${var.project_id}-raw-events"
location = var.region
storage_class = "STANDARD"
uniform_bucket_level_access = true
public_access_prevention = "enforced"
force_destroy = false
retention_policy {
retention_period = 2592000
}
lifecycle_rule {
condition {
age = 31
}
action {
type = "Delete"
}
}
lifecycle_rule {
condition {
age = 7
}
action {
type = "SetStorageClass"
storage_class = "NEARLINE"
}
}
}
resource "google_storage_bucket_iam_member" "writer_creates_objects" {
bucket = google_storage_bucket.raw_events.name
role = "roles/storage.objectCreator"
member = google_service_account.writer.member
}
```

View File

@@ -0,0 +1,40 @@
# Service accounts and IAM grants
One service account per workload identity, and one `_iam_member` resource per grant.
- `account_id` is 6 to 30 characters: lower-case letters, digits and hyphens.
- Use the narrowest resource the role can be granted on: the bucket, topic, secret or service, in preference to the project.
- A grant to a service account is `member = "serviceAccount:${...email}"`. The same value is the `member` attribute of the service account resource.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
resource "google_service_account" "worker" {
project = var.project_id
account_id = "event-worker"
display_name = "Event worker"
description = "Identity of the worker service."
}
# A project-level role, only when no narrower resource can take the grant.
resource "google_project_iam_member" "worker_metrics" {
project = var.project_id
role = "roles/monitoring.metricWriter"
member = google_service_account.worker.member
}
# Who may act as the service account, for example a deployment pipeline.
resource "google_service_account_iam_member" "deployer_acts_as_worker" {
service_account_id = google_service_account.worker.name
role = "roles/iam.serviceAccountUser"
member = "serviceAccount:deployer@${var.project_id}.iam.gserviceaccount.com"
}
```

View File

@@ -0,0 +1,79 @@
# Cloud KMS
A key ring and a key for customer-managed encryption, and the grant that lets a product use it.
- The key ring must be in the same location as the resource it protects.
- Key rings and keys cannot be deleted. `prevent_destroy` stops Terraform trying.
- `rotation_period` is in seconds, such as `"7776000s"` for 90 days.
- The product's service agent, not the workload's service account, needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key. The resource that uses the key must `depends_on` that grant.
- Service agents: Pub/Sub is `service-NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com`, Cloud Run is `service-NUMBER@serverless-robot-prod.iam.gserviceaccount.com`, Cloud Storage comes from the `google_storage_project_service_account` data source.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
data "google_project" "this" {
project_id = var.project_id
}
data "google_storage_project_service_account" "this" {
project = var.project_id
}
resource "google_kms_key_ring" "workload" {
project = var.project_id
name = "workload"
location = var.region
}
resource "google_kms_crypto_key" "data" {
name = "data"
key_ring = google_kms_key_ring.workload.id
purpose = "ENCRYPT_DECRYPT"
rotation_period = "7776000s"
lifecycle {
prevent_destroy = true
}
}
resource "google_kms_crypto_key_iam_member" "pubsub_uses_key" {
crypto_key_id = google_kms_crypto_key.data.id
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
member = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
resource "google_kms_crypto_key_iam_member" "storage_uses_key" {
crypto_key_id = google_kms_crypto_key.data.id
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"
}
resource "google_pubsub_topic" "encrypted" {
project = var.project_id
name = "encrypted-events"
kms_key_name = google_kms_crypto_key.data.id
depends_on = [google_kms_crypto_key_iam_member.pubsub_uses_key]
}
resource "google_storage_bucket" "encrypted" {
project = var.project_id
name = "${var.project_id}-encrypted"
location = var.region
uniform_bucket_level_access = true
encryption {
default_kms_key_name = google_kms_crypto_key.data.id
}
depends_on = [google_kms_crypto_key_iam_member.storage_uses_key]
}
```

View File

@@ -0,0 +1,107 @@
# Cloud Monitoring and Logging
A notification channel, an alert on a metric, a log-based metric and a log sink.
- An alert policy needs `combiner` and at least one `conditions` block. A threshold condition's `filter` must name both a `metric.type` and a `resource.type`.
- Every `aggregations` block needs an `alignment_period` of at least `"60s"` and a `per_series_aligner` that fits the metric kind: `ALIGN_RATE` or `ALIGN_DELTA` for a delta or cumulative metric, `ALIGN_MEAN` or `ALIGN_MAX` for a gauge.
- A log-based metric's name is used in an alert filter as `logging.googleapis.com/user/NAME`.
- A sink's writer identity needs a grant on the destination, or nothing arrives.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "alert_email" {
type = string
description = "Address alerts are sent to."
}
resource "google_monitoring_notification_channel" "email" {
project = var.project_id
display_name = "Operations email"
type = "email"
labels = {
email_address = var.alert_email
}
}
resource "google_monitoring_alert_policy" "server_errors" {
project = var.project_id
display_name = "Cloud Run 5xx responses"
combiner = "OR"
conditions {
display_name = "5xx rate above 1 per second for 5 minutes"
condition_threshold {
filter = "resource.type = \"cloud_run_revision\" AND metric.type = \"run.googleapis.com/request_count\" AND metric.labels.response_code_class = \"5xx\""
comparison = "COMPARISON_GT"
threshold_value = 1
duration = "300s"
aggregations {
alignment_period = "60s"
per_series_aligner = "ALIGN_RATE"
cross_series_reducer = "REDUCE_SUM"
group_by_fields = ["resource.labels.service_name"]
}
trigger {
count = 1
}
}
}
notification_channels = [google_monitoring_notification_channel.email.name]
alert_strategy {
auto_close = "1800s"
}
documentation {
content = "The service is returning server errors. Check the latest revision's logs."
mime_type = "text/markdown"
}
}
resource "google_logging_metric" "rejected_signatures" {
project = var.project_id
name = "rejected_signatures"
filter = "resource.type = \"cloud_run_revision\" AND jsonPayload.event = \"signature_rejected\""
metric_descriptor {
metric_kind = "DELTA"
value_type = "INT64"
unit = "1"
}
}
resource "google_storage_bucket" "audit_logs" {
project = var.project_id
name = "${var.project_id}-audit-logs"
location = var.region
uniform_bucket_level_access = true
}
resource "google_logging_project_sink" "audit" {
project = var.project_id
name = "audit-to-storage"
destination = "storage.googleapis.com/${google_storage_bucket.audit_logs.name}"
filter = "logName:\"cloudaudit.googleapis.com\""
unique_writer_identity = true
}
resource "google_storage_bucket_iam_member" "sink_writes" {
bucket = google_storage_bucket.audit_logs.name
role = "roles/storage.objectCreator"
member = google_logging_project_sink.audit.writer_identity
}
```

View File

@@ -0,0 +1,102 @@
# VPC networking
A custom-mode VPC, a subnet, outbound internet through Cloud NAT, and a firewall rule.
- `auto_create_subnetworks = false` gives a custom-mode network: only the subnets you declare.
- `private_ip_google_access = true` lets workloads without external addresses reach Google APIs.
- Cloud Run reaches the VPC with Direct VPC egress: `template.vpc_access.network_interfaces` naming this network and subnet. It needs no connector resource.
- Firewall rules match by `target_service_accounts` or `target_tags`. A rule with neither applies to the whole network.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
resource "google_compute_network" "main" {
project = var.project_id
name = "main"
auto_create_subnetworks = false
routing_mode = "REGIONAL"
}
resource "google_compute_subnetwork" "workloads" {
project = var.project_id
name = "workloads"
region = var.region
network = google_compute_network.main.id
ip_cidr_range = "10.10.0.0/24"
private_ip_google_access = true
log_config {
aggregation_interval = "INTERVAL_5_SEC"
flow_sampling = 0.5
metadata = "INCLUDE_ALL_METADATA"
}
}
resource "google_compute_router" "egress" {
project = var.project_id
name = "egress"
region = var.region
network = google_compute_network.main.id
}
resource "google_compute_router_nat" "egress" {
project = var.project_id
name = "egress"
region = var.region
router = google_compute_router.egress.name
nat_ip_allocate_option = "AUTO_ONLY"
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
log_config {
enable = true
filter = "ERRORS_ONLY"
}
}
resource "google_compute_firewall" "allow_health_checks" {
project = var.project_id
name = "allow-health-checks"
network = google_compute_network.main.id
direction = "INGRESS"
priority = 1000
# Google Cloud load balancer health check ranges.
source_ranges = ["130.211.0.0/22", "35.191.0.0/16"]
target_tags = ["web"]
allow {
protocol = "tcp"
ports = ["8080"]
}
}
resource "google_cloud_run_v2_service" "internal" {
project = var.project_id
name = "internal"
location = var.region
ingress = "INGRESS_TRAFFIC_INTERNAL_ONLY"
template {
containers {
image = "us-docker.pkg.dev/cloudrun/container/hello"
}
vpc_access {
egress = "PRIVATE_RANGES_ONLY"
network_interfaces {
network = google_compute_network.main.id
subnetwork = google_compute_subnetwork.workloads.id
}
}
}
}
```

View File

@@ -0,0 +1,33 @@
# Enabling APIs
Enable each API the Elements need, once, and make the first resource of each product depend on it.
- `disable_on_destroy = false` keeps the API on when the configuration is destroyed; other workloads in the project may use it.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
locals {
services = [
"run.googleapis.com",
"pubsub.googleapis.com",
"secretmanager.googleapis.com",
]
}
resource "google_project_service" "enabled" {
for_each = toset(local.services)
project = var.project_id
service = each.value
disable_on_destroy = false
}
```

View File

@@ -0,0 +1,111 @@
# Pub/Sub
A topic, a push subscription that calls a private Cloud Run service, and a dead-letter topic.
- A push subscription authenticates with `push_config.oidc_token`; its service account needs `roles/run.invoker` on the service it calls.
- Dead-lettering needs two grants to the Pub/Sub service agent: `roles/pubsub.publisher` on the dead-letter topic and `roles/pubsub.subscriber` on the source subscription. Without them messages are never moved.
- Give the dead-letter topic a subscription of its own, or the messages moved there are lost.
- `max_delivery_attempts` is 5 to 100. `ack_deadline_seconds` is 10 to 600.
- `message_retention_duration` is a duration in seconds, such as `"604800s"`.
- Customer-managed encryption is `kms_key_name` on the topic; the Pub/Sub service agent needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key first.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "worker_url" {
type = string
description = "HTTPS URL of the service the subscription pushes to."
}
data "google_project" "this" {
project_id = var.project_id
}
locals {
pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
resource "google_service_account" "push" {
project = var.project_id
account_id = "events-push"
display_name = "Identity Pub/Sub pushes with"
}
resource "google_pubsub_topic" "events" {
project = var.project_id
name = "events"
message_retention_duration = "604800s"
message_storage_policy {
allowed_persistence_regions = [var.region]
}
}
resource "google_pubsub_topic" "dead_letter" {
project = var.project_id
name = "events-dead-letter"
}
resource "google_pubsub_subscription" "dead_letter" {
project = var.project_id
name = "events-dead-letter"
topic = google_pubsub_topic.dead_letter.id
message_retention_duration = "1209600s"
expiration_policy {
ttl = ""
}
}
resource "google_pubsub_subscription" "worker" {
project = var.project_id
name = "events-worker"
topic = google_pubsub_topic.events.id
ack_deadline_seconds = 60
push_config {
push_endpoint = var.worker_url
oidc_token {
service_account_email = google_service_account.push.email
audience = var.worker_url
}
}
retry_policy {
minimum_backoff = "10s"
maximum_backoff = "600s"
}
dead_letter_policy {
dead_letter_topic = google_pubsub_topic.dead_letter.id
max_delivery_attempts = 5
}
expiration_policy {
ttl = ""
}
}
resource "google_pubsub_topic_iam_member" "agent_publishes_dead_letters" {
project = var.project_id
topic = google_pubsub_topic.dead_letter.name
role = "roles/pubsub.publisher"
member = local.pubsub_agent
}
resource "google_pubsub_subscription_iam_member" "agent_forwards_from_worker" {
project = var.project_id
subscription = google_pubsub_subscription.worker.name
role = "roles/pubsub.subscriber"
member = local.pubsub_agent
}
```

View File

@@ -0,0 +1,45 @@
# Secret Manager
Terraform creates the secret and who may read it. The value is added outside Terraform, so it never enters the state or the repository.
- `replication` is required: `auto {}` for Google-managed locations, or `user_managed` with one `replicas` block per location.
- Grant `roles/secretmanager.secretAccessor` on the secret, to the one identity that reads it.
- Never write a `google_secret_manager_secret_version` with the value in the configuration.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
resource "google_service_account" "reader" {
project = var.project_id
account_id = "api-key-reader"
display_name = "Reads the API key"
}
resource "google_secret_manager_secret" "api_key" {
project = var.project_id
secret_id = "partner-api-key"
replication {
user_managed {
replicas {
location = var.region
}
}
}
}
resource "google_secret_manager_secret_iam_member" "reader_reads_api_key" {
project = var.project_id
secret_id = google_secret_manager_secret.api_key.secret_id
role = "roles/secretmanager.secretAccessor"
member = google_service_account.reader.member
}
```

View File

@@ -2,31 +2,35 @@
name: iac-validation
description: >-
Checks Terraform infrastructure-as-code without provisioning anything, using
the local terraform_fmt, terraform_validate and terraform_plan tools. Use when
writing deployment guidance or validating it before deployment.
the local terraform_fmt, terraform_validate, terraform_plan and
terraform_schema tools. Use when writing deployment guidance or validating it
before deployment.
metadata:
scope: deployment-guidance validation
---
# IaC validation
Every check runs on a scratch copy of the files you pass. Nothing is applied, and no state is written.
Every check runs on a scratch copy of your Terraform files. Nothing is applied, and no state is written.
The files are your working copy: the files you wrote with `write_file`, or, when you are validating, the files that were submitted. Pass the tools no files.
## Tools
- `terraform_fmt(files)`: returns the files formatting changed. Replace your copies with the returned content.
- `terraform_validate(files)`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments.
- `terraform_plan(files, variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires.
- `terraform_fmt()`: formats your working copy in place and names the files it changed.
- `terraform_validate()`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments.
- `terraform_plan(variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires.
- `terraform_schema(resource_type, block, find)`: what a resource type accepts, from the provider's own schema. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none.
## When writing deployment guidance
1. Run `terraform_fmt`, then `terraform_validate`, on the complete set of files you will attach.
2. Fix every error and run both again, until `terraform_validate` passes.
3. Attach the formatted files.
1. Run `terraform_fmt`, then `terraform_validate`.
2. Fix each error with `edit_file`, changing only the lines concerned, and run `terraform_validate` again until it passes.
3. On "An argument named X is not expected here" or "Unsupported block type", call `terraform_schema` with `find` set to the name. It is either in a different block, which the reply names, or it does not exist on that resource type. Never move it around by trial.
## When validating deployment guidance
1. Run `terraform_validate` on the attached files, and report its verdict and output.
1. Run `terraform_validate`, and report its verdict and output.
2. When permission to read the live environment is granted, run `terraform_plan` with the variable values from the requirements, or clearly marked placeholders. Summarise the resources to add, change and destroy, and flag any destroy or replace.
3. When a check can't run (a missing CLI, a refused permission or a provider download failure), give the user the exact commands to run themselves:
@@ -40,4 +44,6 @@ Every check runs on a scratch copy of the files you pass. Nothing is applied, an
Give a verdict on every Check in the Target architecture and attach them as `verdicts.json`, in the format the requirements-chain Skill gives. The dry-run's result goes in its `dry_run` entry: `pass` when `terraform_validate` passes, `fail` with the error when it does not, and `not_run` when the tool could not run.
A Check fails when the code could meet it and does not. When `terraform_schema` shows the resource type has no such setting, no code could meet it: the verdict is `not_verifiable`, with the schema's answer as evidence.
Never suggest `terraform apply` as part of validation.

View File

@@ -92,14 +92,23 @@ A Check is a statement that is true or false of the infrastructure code. Name th
Split a broad recommendation into several Checks, one per setting. A Check that the validator cannot verify from the code is sent back to you.
A Check must name a setting its Element's resource type really has. Two similar resource types often differ: a setting of a long-running service may not exist on a batch job. A Check on a setting the resource type lacks is sent back to you.
The tables of Elements, Relationships and Checks, and the diagram, are written from this file when you submit. Write the graph once, here.
### `implementation.json` (deployment guidance)
One entry per Check in the Target architecture, saying where the code satisfies it. `file` is an attached file; `address` is the resource address and argument.
```json
[{"check": "K-1", "file": "terraform/main.tf", "address": "google_cloud_run_v2_service.ingest.ingress"}]
[
{"check": "K-1", "file": "terraform/main.tf", "address": "google_cloud_run_v2_service.ingest.ingress"},
{"check": "K-7", "unimplementable": "google_cloud_run_v2_job has no startup_cpu_boost argument; it exists only on a service."}
]
```
When a Check names a setting its resource type does not have, give it an `unimplementable` entry in place of `file` and `address`, saying what the resource type lacks. Write no code for it. The Target architecture is revised before anything is validated.
Implement every Element, Relationship and Check, and nothing the Target architecture does not ask for. When you believe the Target architecture is wrong or incomplete, say so with `needs_input`; do not quietly build something else.
### `verdicts.json` (validation results)

View File

@@ -271,6 +271,21 @@ async def test_model_timeout_is_set_on_every_request():
assert model.requests[0].config.http_options.timeout == 90_000
async def test_a_role_with_its_own_timeout_is_given_it_over_the_default():
model = ScriptedLlm(script=[[call("delegate")]])
runtime = AdkAgentRuntime(
skill_dirs=SKILLS, model_factory=lambda role: model, model_timeout_seconds=90
)
await runtime.run(
RoleSpec("orchestrator", "Coordinate.", budget=5, model_timeout_seconds=420),
TASK,
[Recorder().tool("delegate", TurnEnded("delegate"))],
)
assert model.requests[0].config.http_options.timeout == 420_000
class MeteredLlm(ScriptedLlm):
async def generate_content_async(
self, llm_request: LlmRequest, stream: bool = False

View File

@@ -19,7 +19,7 @@ def bundle():
def test_every_engine_role_and_the_pillar_advisor_is_defined(bundle):
roles = load_roles(AGENTS_DIR, bundle)
expected = {"orchestrator", PILLAR_ADVISOR}
expected = {"orchestrator", "reply-writer", PILLAR_ADVISOR}
expected |= {spec.specialist for spec in DEPENDENCY_GRAPH.values()}
assert set(roles) == expected
for role in roles.values():
@@ -46,12 +46,35 @@ def test_roles_take_skills_mcp_servers_and_models_from_the_bundle(bundle):
def test_roles_name_only_the_local_tools_they_need(bundle):
roles = load_roles(AGENTS_DIR, bundle)
assert roles["iac-author"].tools == ("terraform_fmt", "terraform_validate")
assert roles["target-architect"].tools == ()
assert roles["validator"].tools == ("terraform_validate", "terraform_plan")
assert roles["iac-author"].tools == (
"terraform_fmt",
"terraform_validate",
"terraform_schema",
)
# The architect only looks settings up; it writes and runs no code.
assert roles["target-architect"].tools == ("terraform_schema",)
assert roles["validator"].tools == (
"terraform_validate",
"terraform_plan",
"terraform_schema",
)
assert roles["requirements-analyst"].tools == ()
def test_roles_with_long_replies_are_given_longer_for_a_model_request(bundle):
roles = load_roles(AGENTS_DIR, bundle)
assert roles["target-architect"].model_timeout_seconds == 420
assert roles["decomposer"].model_timeout_seconds is None
def test_the_code_author_is_given_the_patterns_for_its_cloud_provider(bundle):
roles = load_roles(AGENTS_DIR, bundle)
assert "google-cloud-iac-patterns" in roles["iac-author"].skills
assert "google-cloud-iac-patterns" not in roles["validator"].skills
def test_preview_mcp_servers_can_be_excluded(bundle):
roles = load_roles(AGENTS_DIR, bundle, include_preview_mcp=False)

View File

@@ -475,3 +475,56 @@ async def test_a_scripted_user_drives_an_interactive_execution(tmp_path):
assert any("Approved." in reply for reply in replies)
assert any("europe-west2." in reply for reply in replies)
assert final.state["deliverables"]["requirements"]["content"]
# --- Jev as the judge -------------------------------------------------------------------
async def test_jev_judges_every_question_in_one_call_with_no_grading_in_the_state():
from app.adk.fake_runtime import FakeDecider
from app.adk.runtime import Answer
from eval.judge import DeciderJudge
decider = FakeDecider(
{
"covered": Answer(0.91, 0.82),
"missed": Answer(0.2, 0.6),
"specific": Answer(2.0, 0.7), # "mostly", on a scale of four
}
)
judge = DeciderJudge(decider)
scores = await judge.answer(
"## Reliability\nUse two zones.",
{
"covered": {"kind": "yes_no", "text": "Is redundancy addressed?"},
"missed": {"kind": "yes_no", "text": "Are backups tested?"},
"specific": {"kind": "graded", "text": "Is it specific?"},
},
)
assert scores == {"covered": 1.0, "missed": 0.0, "specific": 2 / 3}
[(state, questions, scope)] = decider.asked
assert state == "## Reliability\nUse two zones." # the document, nothing else
assert questions["covered"].kind == "noul"
assert questions["specific"].options == ("not at all", "weakly", "mostly", "fully")
assert scope == "evaluation"
async def test_a_judge_that_gives_no_answer_leaves_the_check_unanswered_not_passed():
from app.adk.runtime import NoDecider
from eval.judge import DeciderJudge
result = await score_execution(
CASE,
execution(),
HEADINGS,
passing_terraform,
judge=DeciderJudge(NoDecider()),
pillar_checklists=CHECKLISTS,
)
assert check(result, "pillar_checklists_covered").detail == (
"judge unavailable: Jev gave no answer"
)
assert not check(result, "recommendations_quality").passed

View File

@@ -15,7 +15,7 @@ from app.agent import ExecutionEngine
from app.recorder import InMemoryRecorder
from app.tools.execution import specialist_tools
from app.tools.local import mermaid_check
from app.workflows.chain import validation_outcome
from app.workflows.chain import unimplementable_checks, validation_outcome
from app.workflows.deliverables import (
DEPENDENCY_GRAPH,
DeliverableKind,
@@ -327,6 +327,54 @@ def test_deployment_guidance_must_account_for_every_check():
)
def test_a_check_that_cannot_be_implemented_is_accounted_for_with_its_reason():
args = sample_submission(K.DEPLOYMENT_GUIDANCE).args
implementation = json.dumps(
[
{"check": "K-1", "file": "terraform/main.tf", "address": "a"},
{
"check": "K-2",
"unimplementable": "example_service_account has no such argument",
},
]
)
missing = check_submission(
K.DEPLOYMENT_GUIDANCE,
args["content"],
{**args["attachments"], "implementation.json": implementation},
upstream(*EVERYTHING_BEFORE_CODE),
)
assert missing == ()
assert unimplementable_checks(implementation) == [
{"check": "K-2", "reason": "example_service_account has no such argument"}
]
assert unimplementable_checks(SAMPLE_DATA["implementation.json"]) == []
def test_an_unimplementable_entry_needs_its_reason():
args = sample_submission(K.DEPLOYMENT_GUIDANCE).args
implementation = json.dumps(
[
{"check": "K-1", "file": "terraform/main.tf", "address": "a"},
{"check": "K-2", "unimplementable": ""},
]
)
missing = check_submission(
K.DEPLOYMENT_GUIDANCE,
args["content"],
{**args["attachments"], "implementation.json": implementation},
upstream(*EVERYTHING_BEFORE_CODE),
)
assert missing == (
"implementation.json[0]: needs unimplementable",
"implementation.json: Check K-2 is not accounted for",
)
def verdicts(**changes) -> str:
data = json.loads(SAMPLE_DATA["verdicts.json"])
data.update(changes)

View File

@@ -0,0 +1,598 @@
"""The Orchestrator as a router (ADR-0007): readings, a disposition, then a playbook.
No language model chooses an action here. Specialists are scripted; the Orchestrator
is the Router, with a scripted decider standing in for Jev, or none at all.
"""
import json
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import (
SAMPLE_DATA,
Call,
FakeAgentRuntime,
FakeDecider,
sample_submission,
)
from app.adk.runtime import Answer, NoDecider, RoleSpec
from app.agent import ExecutionEngine
from app.nodes.router import (
REPLY_WRITER,
Router,
RuntimeReplyWriter,
Thresholds,
)
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
K = DeliverableKind
ROLES = {
"orchestrator": RoleSpec("orchestrator", "Unused by the router.", budget=40),
REPLY_WRITER: RoleSpec(REPLY_WRITER, "Write the reply.", budget=3),
**{
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
for spec in DEPENDENCY_GRAPH.values()
},
}
def specialist(kind: DeliverableKind) -> str:
return DEPENDENCY_GRAPH[kind].specialist
def once() -> dict:
return {
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
}
def validation(**changes) -> Call:
"""A validation submission whose verdicts differ from the all-pass sample."""
call = sample_submission(K.VALIDATION_RESULTS)
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
verdicts.update(changes)
return Call(
call.tool,
{**call.args, "attachments": {"verdicts.json": json.dumps(verdicts)}},
)
def failing(check: str = "K-1") -> Call:
checks = json.loads(SAMPLE_DATA["verdicts.json"])["checks"]
for item in checks:
if item["check"] == check:
item.update(verdict="fail", evidence="min instances is 1")
return validation(checks=checks)
def addition(reason: str) -> Call:
return validation(
proposed_additions=[
{"statement": "A timeout is set.", "applies_to": "E-1", "reason": reason}
]
)
def engine_for(tmp_path, turns: dict, decider=None, writer: bool = False, **options):
runtime = FakeAgentRuntime(turns)
router = Router(
decider or NoDecider(),
RuntimeReplyWriter(runtime, ROLES[REPLY_WRITER]) if writer else None,
Thresholds(),
)
engine = ExecutionEngine(
runtime=runtime,
roles=ROLES,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=router,
**options,
)
return engine, runtime
def actions(result) -> list[str]:
return [decision["action"] for decision in result.state["decisions"]]
def reopened(result) -> list[tuple[str, str]]:
return [
(event["subject"], event["detail"])
for event in result.state["ledger"]["events"]
if event["type"] == "reopened"
]
def approve(phase: str) -> dict:
return {"approval": {"phase": phase, "decision": "approve"}}
# --- the playbook on rules alone -------------------------------------------------------
async def test_an_execution_completes_with_no_model_choosing_an_action(tmp_path):
engine, runtime = engine_for(tmp_path, once())
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
assert "orchestrator" not in {run.role for run in runtime.runs}
assert actions(result) == [
"delegate", # requirements
"delegate", # technical decomposition
"request_approval",
"delegate", # product selection
"delegate", # design recommendations
"delegate", # Target architecture
"request_approval",
"delegate", # description and deployment guidance, together
"request_approval",
"delegate", # validation
"proceed",
"request_approval",
"delegate", # solution guide
"request_approval",
"finish",
]
groups = {decision["group"] for decision in result.state["decisions"]}
assert groups == {
"work is ready",
"a Phase is complete",
"validation decided",
"packaging is approved",
}
async def test_every_decision_is_recorded_with_its_turn_group_and_action(tmp_path):
engine, _ = engine_for(tmp_path, once())
result = await engine.start("Ingest webhooks", pre_approved=True)
first = result.state["decisions"][0]
assert first == {
"turn": 1,
"group": "work is ready",
"action": "delegate",
"judged": {},
}
assert [d["turn"] for d in result.state["decisions"]] == sorted(
d["turn"] for d in result.state["decisions"]
)
async def test_when_no_group_applies_a_pre_approved_execution_ends_with_warnings(
tmp_path,
):
turns = {specialist(K.REQUIREMENTS): [[]]} # stops without submitting
engine, _ = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
# The Specialist's question is relayed, answered automatically, and it stops
# again; the router never invents work.
assert result.status == "completed_with_warnings"
# --- Approvals: the summary is written from data, and only data approves ----------------
async def test_the_approval_summary_is_assembled_from_the_data(tmp_path):
engine, _ = engine_for(tmp_path, once())
first = await engine.start("Ingest webhooks")
design = await engine.resume(first.execution_id, approve("requirements_discovery"))
assert first.prompt["approval"]["summary"] == (
"The requirements_discovery Phase is ready: requirements, "
"technical_decomposition submitted. 2 requirements."
)
summary = design.prompt["approval"]["summary"]
assert "Target architecture: 2 Elements, 1 Relationships, 2 Checks." in summary
assert design.prompt["approval"]["target_architecture"]["checks"] == 2
async def test_a_model_writes_the_summary_but_decides_nothing(tmp_path):
turns = {
**once(),
REPLY_WRITER: [[Call("write_reply", {"text": "Two requirements, ready."})]],
}
engine, runtime = engine_for(tmp_path, turns, writer=True)
first = await engine.start("Ingest webhooks")
assert first.prompt["approval"]["summary"] == "Two requirements, ready."
writer_run = next(run for run in runtime.runs if run.role == REPLY_WRITER)
assert writer_run.tools == ("write_reply",)
assert writer_run.task.context["facts"]["deliverables"] == [
"requirements",
"technical_decomposition",
]
async def test_a_writer_that_fails_does_not_stop_the_approval(tmp_path):
engine, _ = engine_for(tmp_path, once(), writer=True) # no writer turn scripted
first = await engine.start("Ingest webhooks")
assert first.prompt["approval"]["summary"].startswith(
"The requirements_discovery Phase is ready"
)
async def test_a_text_reply_never_approves_and_the_approval_is_asked_again(tmp_path):
decider = FakeDecider(
{
"asks_change": Answer(0.03, 0.94),
"change_target": Answer("requirements", 0.5),
"change_clarity": Answer(0.0, 0.9),
}
)
engine, _ = engine_for(tmp_path, once(), decider)
first = await engine.start("Ingest webhooks")
again = await engine.resume(first.execution_id, {"reply": "Looks good to me."})
assert again.prompt["approval"]["phase"] == "requirements_discovery"
assert again.state["ledger"]["phases"]["requirements_discovery"] != "approved"
# --- validation: the code, the graph, or proceed ----------------------------------------
async def test_a_failed_check_sends_the_code_back_and_leaves_the_design_approved(
tmp_path,
):
turns = once()
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [
[failing("K-1")],
[sample_submission(K.VALIDATION_RESULTS)],
]
engine, runtime = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
assert reopened(result) == [
("deployment_guidance", "K-1 failed: min instances is 1")
]
assert "revise_code" in actions(result)
architect_runs = [
run for run in runtime.runs if run.role == specialist(K.TARGET_ARCHITECTURE)
]
assert len(architect_runs) == 1
async def test_a_check_the_code_cannot_meet_goes_back_to_the_graph_unvalidated(
tmp_path,
):
"""The code's author found the Check names a setting the resource lacks. No
validation is spent on it: the Target architecture is revised first."""
cannot = sample_submission(K.DEPLOYMENT_GUIDANCE)
implementation = json.loads(SAMPLE_DATA["implementation.json"])
implementation[1] = {"check": "K-2", "unimplementable": "no such argument"}
turns = once()
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
turns[specialist(K.ARCHITECTURE_DESCRIPTION)] *= 2
turns[specialist(K.DEPLOYMENT_GUIDANCE)] = [
[
Call(
cannot.tool,
{
**cannot.args,
"attachments": {
**cannot.args["attachments"],
"implementation.json": json.dumps(implementation),
},
},
)
],
[sample_submission(K.DEPLOYMENT_GUIDANCE)],
]
engine, runtime = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
assert reopened(result) == [
(
"target_architecture",
(
"Check K-2 cannot be implemented as written (no such argument). "
"Rewrite it so it can be met, or remove it with a reason."
),
)
]
assert "the code cannot meet a Check" in {
decision["group"] for decision in result.state["decisions"]
}
validations = [
run for run in runtime.runs if run.role == specialist(K.VALIDATION_RESULTS)
]
assert len(validations) == 1
async def test_a_check_that_fails_twice_running_is_sent_back_to_the_graph(tmp_path):
"""The code was revised and the same Check failed again: the Check is suspect."""
turns = once()
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 3
turns[specialist(K.ARCHITECTURE_DESCRIPTION)] *= 2
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [
[failing("K-1")],
[failing("K-1")],
[sample_submission(K.VALIDATION_RESULTS)],
]
engine, _ = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
[first, second] = reopened(result)
assert first[0] == "deployment_guidance"
assert second[0] == "target_architecture"
assert "K-1 failed in two validations in a row" in second[1]
assert actions(result).count("revise_code") == 1
async def test_without_jev_a_gap_naming_a_real_requirement_revises_the_graph(tmp_path):
turns = once()
for kind in (
K.TARGET_ARCHITECTURE,
K.ARCHITECTURE_DESCRIPTION,
K.DEPLOYMENT_GUIDANCE,
):
turns[specialist(kind)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [
[addition("NFR-1 has no Check on timeouts")],
[sample_submission(K.VALIDATION_RESULTS)],
]
engine, _ = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
[(kind, reason)] = reopened(result)
assert kind == "target_architecture"
assert reason == (
"Add a Check on E-1: A timeout is set. (serves NFR-1 has no Check on timeouts)."
)
async def test_without_jev_a_gap_that_names_nothing_real_is_left(tmp_path):
turns = once()
turns[specialist(K.VALIDATION_RESULTS)] = [[addition("it would be nice")]]
engine, _ = engine_for(tmp_path, turns)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
assert reopened(result) == []
async def test_jev_judges_whether_a_gap_is_worth_a_revision(tmp_path):
turns = once()
turns[specialist(K.VALIDATION_RESULTS)] = [[addition("NFR-1")]]
immaterial = FakeDecider({"gap_material": Answer(0.04, 0.92)})
engine, _ = engine_for(tmp_path, turns, immaterial)
result = await engine.start("Ingest webhooks", pre_approved=True)
# The rule alone would have revised the graph; the reading says it is not needed.
assert reopened(result) == []
judged = next(d for d in result.state["decisions"] if d["action"] == "proceed")
assert judged["judged"] == {"gap_material": {"value": 0.04, "confidence": 0.92}}
[(state, questions, scope)] = immaterial.asked
assert "A timeout is set." in state and scope == result.execution_id
assert questions["gap_material"].kind == "noul"
async def test_an_uncertain_gap_is_put_to_the_user_who_decides(tmp_path):
turns = once()
for kind in (
K.TARGET_ARCHITECTURE,
K.ARCHITECTURE_DESCRIPTION,
K.DEPLOYMENT_GUIDANCE,
):
turns[specialist(kind)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [
[addition("NFR-1")],
[sample_submission(K.VALIDATION_RESULTS)],
]
decider = FakeDecider(
{
"gap_material": Answer(0.5, 0.0), # the grey zone
"wants_revision": Answer(0.96, 0.92),
}
)
engine, _ = engine_for(tmp_path, turns, decider)
result = await engine.start("Ingest webhooks")
for phase in (
"requirements_discovery",
"solution_architecture",
"solution_implementation",
):
result = await engine.resume(result.execution_id, approve(phase))
[question] = result.prompt["questions"]
assert (
"found gaps in the Target architecture: A timeout is set."
in (question["question"])
)
result = await engine.resume(result.execution_id, {"reply": "Yes, revise it."})
assert reopened(result)[0][0] == "target_architecture"
# Reopening the graph withdrew the design Approval, so it is asked for again.
assert result.prompt["approval"]["phase"] == "solution_architecture"
# --- the user's message: act, queue or close --------------------------------------------
def to_design_approval(engine):
async def run():
first = await engine.start("Ingest webhooks")
return await engine.resume(
first.execution_id, approve("requirements_discovery")
)
return run()
def changes(phase: str, feedback: str) -> dict:
return {"approval": {"phase": phase, "decision": "changes", "feedback": feedback}}
async def test_a_requested_change_is_routed_to_the_deliverable_it_affects(tmp_path):
turns = once()
turns[specialist(K.DESIGN_RECOMMENDATIONS)] *= 2
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
decider = FakeDecider(
{
"change_target": Answer("design_recommendations", 0.88),
"change_clarity": Answer(1.8, 0.8),
}
)
engine, _ = engine_for(tmp_path, turns, decider)
design = await to_design_approval(engine)
result = await engine.resume(
design.execution_id,
changes("solution_architecture", "Use customer-managed keys on the topic."),
)
assert reopened(result) == [
("design_recommendations", "Use customer-managed keys on the topic.")
]
# Product selection was not touched; the design is asked for again once redone.
assert result.state["ledger"]["deliverables"]["product_selection"]["revisions"] == 0
assert result.prompt["approval"]["phase"] == "solution_architecture"
[(state, questions, _)] = decider.asked
assert "The user's message: Use customer-managed keys on the topic." in state
assert questions["change_target"].options == (
"requirements",
"technical_decomposition",
"product_selection",
"design_recommendations",
"target_architecture",
)
assert "asks_change" not in questions # a structured request is known to be one
async def test_an_unclear_change_is_confirmed_with_the_user_before_anything_moves(
tmp_path,
):
turns = once()
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
decider = FakeDecider(
{
"change_target": Answer("product_selection", 0.31), # not relied on
"change_clarity": Answer(1.5, 0.7),
}
)
engine, _ = engine_for(tmp_path, turns, decider)
design = await to_design_approval(engine)
asked = await engine.resume(
design.execution_id, changes("solution_architecture", "Tighten it up.")
)
assert reopened(asked) == []
[question] = asked.prompt["questions"]
assert question["question"].startswith("Do you want product_selection changed?")
result = await engine.resume(
asked.execution_id,
{"reply": "No, change the target architecture: add a Check on timeouts."},
)
[(kind, reason)] = reopened(result)
assert kind == "target_architecture"
assert "Tighten it up." in reason and "add a Check on timeouts" in reason
async def test_a_free_text_reply_that_asks_for_a_change_is_acted_on(tmp_path):
turns = once()
turns[specialist(K.REQUIREMENTS)] = [
[sample_submission(K.REQUIREMENTS)],
[sample_submission(K.REQUIREMENTS)],
]
turns[specialist(K.TECHNICAL_DECOMPOSITION)] *= 2
decider = FakeDecider(
{
"asks_change": Answer(0.93, 0.86),
"change_target": Answer("requirements", 0.9),
"change_clarity": Answer(1.9, 0.8),
}
)
engine, _ = engine_for(tmp_path, turns, decider)
first = await engine.start("Ingest webhooks")
result = await engine.resume(
first.execution_id, {"reply": "Retention must be 7 years, not 30 days."}
)
assert reopened(result) == [
("requirements", "Retention must be 7 years, not 30 days.")
]
async def test_without_jev_a_change_is_acted_on_only_when_the_deliverable_is_named(
tmp_path,
):
turns = once()
turns[specialist(K.DESIGN_RECOMMENDATIONS)] *= 2
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
engine, _ = engine_for(tmp_path, turns)
design = await to_design_approval(engine)
unnamed = await engine.resume(
design.execution_id, changes("solution_architecture", "Make it cheaper.")
)
assert reopened(unnamed) == []
assert unnamed.prompt["questions"][0]["question"].startswith(
"Which Deliverable should change"
)
named = await engine.resume(
unnamed.execution_id, {"reply": "The design recommendations."}
)
[(kind, reason)] = reopened(named)
assert kind == "design_recommendations"
assert reason == (
"Make it cheaper.\nThe user then said: The design recommendations."
)
# --- permission is the Ledger's, whatever was decided ------------------------------------
async def test_past_the_revision_limit_the_user_decides(tmp_path):
turns = once()
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [[failing("K-2")], [failing("K-1")]]
engine, _ = engine_for(tmp_path, turns, max_revisions=1)
result = await engine.start("Ingest webhooks")
for phase in (
"requirements_discovery",
"solution_architecture",
"solution_implementation",
"solution_implementation",
):
result = await engine.resume(result.execution_id, approve(phase))
assert len(reopened(result)) == 1
[question] = result.prompt["questions"]
assert (
"deployment_guidance has reached its revision limit (1)"
in (question["question"])
)

View File

@@ -0,0 +1,480 @@
"""A Specialist's working copy, and what each Specialist is given to read.
Three things made an Execution slow: a Specialist rewrote a whole Deliverable in one
reply, sent every file again on each tool call, and read every earlier Deliverable in
full. A working copy and trimmed reading remove each of them.
"""
import json
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import (
SAMPLE_DATA,
Call,
FakeAgentRuntime,
sample_submission,
)
from app.adk.runtime import NoDecider, RoleSpec, Submitted, ToolResult
from app.agent import ExecutionEngine
from app.nodes.router import Router, Thresholds
from app.recorder import InMemoryRecorder
from app.tools.execution import specialist_tools
from app.tools.local import TerraformRunner, local_tools
from app.tools.toolkit import make_tool_factory
from app.tools.workspace import file_tools, graph_tools
from app.workflows.deliverables import (
DEPENDENCY_GRAPH,
DeliverableKind,
dependencies_for,
)
from app.workflows.target_architecture import parse_graph, render_tables
K = DeliverableKind
GRAPH = json.loads(SAMPLE_DATA["target-architecture.json"])
UPSTREAM = {
"requirements": {
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
},
"technical_decomposition": {
"attachments": {"components.json": SAMPLE_DATA["components.json"]}
},
"product_selection": {
"attachments": {"products.json": SAMPLE_DATA["products.json"]}
},
"design_recommendations": {
"attachments": {"recommendations.json": SAMPLE_DATA["recommendations.json"]}
},
"target_architecture": {
"attachments": {
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
}
},
}
def tools_of(kind, working, upstream=UPSTREAM, extra=()):
return {
tool.name: tool
for tool in specialist_tools(
kind, extra, allow_cloud_read=False, upstream=upstream, working=working
)
}
# --- the Target architecture is written once -------------------------------------------
def test_tables_are_rendered_from_the_graph():
tables = render_tables(parse_graph(SAMPLE_DATA["target-architecture.json"]))
assert "## Elements" in tables and "## Relationships" in tables
assert (
"| E-1 | Ingress service | Example Run | example_run_service | C-1 |" in tables
)
assert "| R-1 | Ingress service | runs as | Ingress identity |" in tables
assert "| K-1 | E-1 | The service runs at least two instances. | NFR-1 |" in tables
def test_a_table_cell_cannot_break_its_row():
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
graph["checks"][0]["statement"] = "a | b\nc"
assert "| a \\| b c |" in render_tables(parse_graph(json.dumps(graph)))
async def test_the_engine_writes_the_tables_and_the_diagram_on_submission():
tools = tools_of(K.TARGET_ARCHITECTURE, {})
result = await tools["submit_deliverable"].handler(
content="## Design rationale\nOne service, one identity.\n",
attachments={
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
},
)
assert isinstance(result.outcome, Submitted)
assert result.outcome.content.startswith("## Design rationale")
assert "## Checks" in result.outcome.content
assert result.outcome.attachments["architecture.mmd"].startswith("flowchart")
async def test_the_graph_is_built_a_few_items_at_a_time_and_attached_on_submission():
working: dict = {}
tools = tools_of(K.TARGET_ARCHITECTURE, working)
await tools["graph_put"].handler(elements=GRAPH["elements"])
await tools["graph_put"].handler(relationships=GRAPH["relationships"])
put = await tools["graph_put"].handler(checks=GRAPH["checks"])
assert put.message == (
"Added 2, replaced 0. The graph has 2 Elements, 1 Relationships and 2 Checks."
)
result = await tools["submit_deliverable"].handler(
content="## Design rationale\nReasons.\n"
)
assert isinstance(result.outcome, Submitted)
submitted = json.loads(result.outcome.attachments["target-architecture.json"])
assert [item["id"] for item in submitted["checks"]] == ["K-1", "K-2"]
async def test_a_revision_amends_the_graph_that_was_submitted():
working: dict = {}
tools = tools_of(K.TARGET_ARCHITECTURE, working)
await tools["submit_deliverable"].handler(
content="## Design rationale\nReasons.\n",
attachments={
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
},
)
# A later attempt, with the working memory the Execution kept.
tools = tools_of(K.TARGET_ARCHITECTURE, working)
changed = {**GRAPH["checks"][0], "statement": "min_instance_count is at least 2"}
put = await tools["graph_put"].handler(checks=[changed])
read = await tools["graph_read"].handler()
assert put.message.startswith("Added 0, replaced 1.")
assert "min_instance_count is at least 2" in read.message
assert len(json.loads(read.message)["checks"]) == 2
async def test_graph_items_can_be_removed_and_an_item_without_an_id_is_refused():
working: dict = {}
tools = {tool.name: tool for tool in graph_tools(working)}
await tools["graph_put"].handler(
elements=GRAPH["elements"],
not_applicable=[{"recommendation": "DR-9", "reason": "Single region."}],
)
removed = await tools["graph_remove"].handler(ids=["E-2", "DR-9"])
refused = await tools["graph_put"].handler(checks=[{"statement": "no id"}])
assert removed.message.startswith("Removed 2.")
assert refused.message.startswith("Refused") and "checks[0]" in refused.message
assert [e["id"] for e in working["graph"]["elements"]] == ["E-1"]
# --- files are written once and changed in place ----------------------------------------
async def test_a_file_is_written_once_and_changed_in_place():
working: dict = {}
tools = {tool.name: tool for tool in file_tools(working)}
await tools["write_file"].handler(
path="terraform/main.tf", content="a = 1\nb = 1\n"
)
edited = await tools["edit_file"].handler(
path="terraform/main.tf", old="b = 1", new="b = 2"
)
listing = await tools["read_file"].handler()
assert edited.message == "Edited terraform/main.tf."
assert working["files"] == {"terraform/main.tf": "a = 1\nb = 2\n"}
assert "- terraform/main.tf (2 lines)" in listing.message
async def test_an_edit_that_does_not_match_exactly_once_changes_nothing():
working: dict = {"files": {"main.tf": "x = 1\nx = 1\n"}}
tools = {tool.name: tool for tool in file_tools(working)}
twice = await tools["edit_file"].handler(path="main.tf", old="x = 1", new="x = 2")
never = await tools["edit_file"].handler(path="main.tf", old="y", new="z")
absent = await tools["edit_file"].handler(path="other.tf", old="x", new="y")
unsafe = await tools["write_file"].handler(path="../main.tf", content="x")
assert "it matches 2" in twice.message and "it matches 0" in never.message
assert absent.message.startswith("Refused") and unsafe.message.startswith("Refused")
assert working["files"] == {"main.tf": "x = 1\nx = 1\n"}
async def test_the_working_copy_is_what_is_submitted():
working: dict = {}
tools = tools_of(K.DEPLOYMENT_GUIDANCE, working)
for path in ("terraform/main.tf", "terraform/variables.tf"):
await tools["write_file"].handler(path=path, content=f"# {path}\n")
await tools["write_file"].handler(
path="implementation.json", content=SAMPLE_DATA["implementation.json"]
)
content = sample_submission(K.DEPLOYMENT_GUIDANCE).args["content"]
result = await tools["submit_deliverable"].handler(content=content)
assert isinstance(result.outcome, Submitted)
assert set(result.outcome.attachments) == {
"terraform/main.tf",
"terraform/variables.tf",
"implementation.json",
}
async def test_a_file_passed_on_submission_replaces_its_working_copy():
working: dict = {"files": {"terraform/main.tf": "old\n"}}
tools = tools_of(K.DEPLOYMENT_GUIDANCE, working)
call = sample_submission(K.DEPLOYMENT_GUIDANCE)
result = await tools["submit_deliverable"].handler(**call.args)
assert isinstance(result.outcome, Submitted)
assert result.outcome.attachments["terraform/main.tf"] != "old\n"
assert working["files"]["terraform/main.tf"] != "old\n"
def test_only_a_specialist_that_owns_files_can_write_them():
assert "write_file" in tools_of(K.DEPLOYMENT_GUIDANCE, {})
assert "write_file" not in tools_of(K.VALIDATION_RESULTS, {})
assert "graph_put" in tools_of(K.TARGET_ARCHITECTURE, {})
assert "graph_put" not in tools_of(K.DEPLOYMENT_GUIDANCE, {})
# --- Terraform runs on the working copy -------------------------------------------------
class Runner(TerraformRunner):
def __init__(self) -> None:
super().__init__()
self.validated: list[dict[str, str]] = []
async def validate(self, files):
self.validated.append(dict(files))
return ToolResult("terraform validate passed:\n")
async def formatted(self, files):
return {name: content.strip() + "\n" for name, content in files.items()}
async def test_terraform_checks_take_no_files_when_there_is_a_working_copy():
runner = Runner()
files = {"terraform/main.tf": "x = 1\n", "implementation.json": "[]"}
(validate,) = local_tools(["terraform_validate"], runner, files)
await validate.handler()
assert validate.parameters["required"] == []
assert "pass no files" in validate.description
# Only the Terraform files are checked.
assert runner.validated == [{"terraform/main.tf": "x = 1\n"}]
async def test_formatting_changes_the_working_copy_and_returns_only_names():
files = {"terraform/main.tf": " x = 1 \n"}
(fmt,) = local_tools(["terraform_fmt"], Runner(), files)
result = await fmt.handler()
assert files == {"terraform/main.tf": "x = 1\n"}
assert result.message == (
"terraform fmt reformatted 1 file(s) in your working copy: terraform/main.tf."
)
async def test_files_passed_anyway_are_written_to_the_working_copy_first():
runner = Runner()
files: dict[str, str] = {}
(validate,) = local_tools(["terraform_validate"], runner, files)
await validate.handler(files=[{"path": "terraform/main.tf", "content": "y"}])
assert files == {"terraform/main.tf": "y"}
assert runner.validated == [files]
def roles_with(tools_by_role: dict[str, tuple[str, ...]]) -> dict[str, RoleSpec]:
return {
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
**{
spec.specialist: RoleSpec(
spec.specialist,
"Produce.",
budget=15,
tools=tools_by_role.get(spec.specialist, ()),
)
for spec in DEPENDENCY_GRAPH.values()
},
}
async def test_the_validator_checks_the_submitted_code_without_being_sent_it(tmp_path):
runner = Runner()
roles = roles_with({"validator": ("terraform_validate",)})
turns = {
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
}
turns["validator"] = [
[Call("terraform_validate", {}), sample_submission(K.VALIDATION_RESULTS)]
]
runtime = FakeAgentRuntime(turns)
engine = ExecutionEngine(
runtime=runtime,
roles=roles,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
tool_factory=make_tool_factory(
runtime,
roles,
{},
local_tools=lambda working_copy: local_tools(
["terraform_validate"], runner, working_copy
),
),
)
result = await engine.start("Ingest webhooks", pre_approved=True)
assert result.status == "completed"
assert runner.validated == [
{
"terraform/main.tf": "# terraform/main.tf\n",
"terraform/variables.tf": "# terraform/variables.tf\n",
}
]
# The code is the code author's: validation attaches its verdicts and nothing else.
validation = result.state["deliverables"]["validation_results"]
assert set(validation["attachments"]) == {"verdicts.json"}
# --- each Specialist reads what it works from -------------------------------------------
def submitted() -> dict[str, dict]:
return {
kind.value: {
"content": f"# {kind}\nProse.\n",
"attachments": {
name: SAMPLE_DATA.get(name, f"# {name}\n") for name in spec.attachments
},
}
for kind, spec in DEPENDENCY_GRAPH.items()
}
def test_the_code_is_written_from_the_graph_and_the_ids_alone():
given = dependencies_for(K.DEPLOYMENT_GUIDANCE, submitted())
assert given["target_architecture"] == {
"attachments": {
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
}
}
assert all("content" not in deliverable for deliverable in given.values())
assert set(given["requirements"]["attachments"]) == {"requirements.json"}
def test_a_direct_dependency_is_read_whole_and_the_rest_as_data():
given = dependencies_for(K.TARGET_ARCHITECTURE, submitted())
assert "content" in given["product_selection"]
assert "content" in given["design_recommendations"]
assert "content" not in given["requirements"]
assert "content" not in given["technical_decomposition"]
def test_the_validator_reads_the_code_whole_and_the_graph_as_data():
given = dependencies_for(K.VALIDATION_RESULTS, submitted())
assert "terraform/main.tf" in given["deployment_guidance"]["attachments"]
assert "content" not in given["target_architecture"]
# A Deliverable with no data file has nothing to trim to, so it comes whole.
assert "content" in given["architecture_description"]
def test_the_solution_guide_is_assembled_from_every_deliverable_whole():
given = dependencies_for(K.SOLUTION_GUIDE, submitted())
assert len(given) == len(DEPENDENCY_GRAPH) - 1
assert all("content" in deliverable for deliverable in given.values())
async def test_a_specialist_is_told_what_its_working_copy_already_holds(tmp_path):
turns = {
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
}
failing = sample_submission(K.VALIDATION_RESULTS)
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
verdicts["checks"][0].update(verdict="fail", evidence="min instances is 1")
turns["validator"] = [
[
Call(
failing.tool,
{
**failing.args,
"attachments": {"verdicts.json": json.dumps(verdicts)},
},
)
],
[sample_submission(K.VALIDATION_RESULTS)],
]
turns["iac-author"] = [[sample_submission(K.DEPLOYMENT_GUIDANCE)]] * 2
runtime = FakeAgentRuntime(turns)
engine = ExecutionEngine(
runtime=runtime,
roles=roles_with({}),
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
)
await engine.start("Ingest webhooks", pre_approved=True)
first, second = (run for run in runtime.runs if run.role == "iac-author")
assert first.task.context["working_copy"] is None
assert second.task.context["working_copy"] == {
"files": ["implementation.json", "terraform/main.tf", "terraform/variables.tf"]
}
async def test_a_deliverable_given_as_data_can_be_read_whole_on_demand():
library = {
"requirements": {
"content": "# requirements\nProse.\n",
"attachments": {"requirements.json": "[]"},
}
}
tools = {
tool.name: tool
for tool in specialist_tools(
K.DEPLOYMENT_GUIDANCE, (), allow_cloud_read=False, library=library
)
}
read = await tools["read_deliverable"].handler(kind="requirements")
refused = await tools["read_deliverable"].handler(kind="solution_guide")
assert (
read.message.startswith("# requirements")
and "requirements.json" in read.message
)
assert refused.message.startswith("Refused")
# With nothing trimmed there is nothing to read back.
assert "read_deliverable" not in tools_of(K.REQUIREMENTS, {})
async def test_the_engine_offers_the_trimmed_deliverables_for_reading(tmp_path):
turns = {
spec.specialist: [[sample_submission(kind)]]
for kind, spec in DEPENDENCY_GRAPH.items()
}
runtime = FakeAgentRuntime(turns)
engine = ExecutionEngine(
runtime=runtime,
roles=roles_with({}),
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
)
await engine.start("Ingest webhooks", pre_approved=True)
author = next(run for run in runtime.runs if run.role == "iac-author")
packager = next(run for run in runtime.runs if run.role == "packager")
assert "read_deliverable" in author.tools
assert "content" not in author.task.context["dependencies"]["requirements"]
assert "read_deliverable" not in packager.tools

View File

@@ -12,6 +12,7 @@ def settings(tmp_path, **overrides) -> Settings:
values: dict[str, Any] = {
"CHECKPOINTER": "memory",
"MCP_ENABLED": False,
"ORCHESTRATOR": "model",
"ORCHESTRATOR_BUDGET": 9,
"SPECIALIST_BUDGET": 4,
}
@@ -50,3 +51,42 @@ async def test_preview_mcp_servers_follow_the_setting(tmp_path):
assert "asset_inventory" not in servers
assert "compute" in servers
async def test_the_assembly_routes_by_default_and_runs_without_a_jev_key(tmp_path):
"""With no Jev key the router follows its playbook on rules alone."""
from app.adk.fake_runtime import sample_submission
from app.adk.runtime import NoDecider
from app.workflows.deliverables import DeliverableKind
runtime = FakeAgentRuntime(
{"requirements-analyst": [[sample_submission(DeliverableKind.REQUIREMENTS)]]}
)
async with open_assembly(
settings(tmp_path, ORCHESTRATOR="router", TYPESAFE_API_KEY=None),
runtime=runtime,
recorder=InMemoryRecorder(),
output_root=tmp_path,
) as assembly:
interaction = await assembly.engine.start("Design ingestion")
assert isinstance(assembly.decider, NoDecider)
# The router delegated requirements with no Orchestrator model, then the
# decomposer (unscripted here) stopped and its question reached the user.
assert [run.role for run in runtime.runs] == ["requirements-analyst"]
assert interaction.status == "awaiting_user"
assert "reply-writer" in assembly.roles
async def test_an_unknown_orchestrator_setting_is_refused(tmp_path):
import pytest
with pytest.raises(ValueError, match="ORCHESTRATOR must be router or model"):
async with open_assembly(
settings(tmp_path, ORCHESTRATOR="llm"),
runtime=FakeAgentRuntime({}),
recorder=InMemoryRecorder(),
output_root=tmp_path,
):
pass

View File

@@ -0,0 +1,62 @@
"""The IaC patterns Skill: every pattern validates against the pinned provider.
A pattern that does not validate teaches the mistake it was written to prevent, so
each reference's Terraform is run through `terraform validate` with the Skill's own
versions file. The provider comes from the plugin cache; with no cache and no
network the test is skipped, not failed.
"""
import re
import shutil
from pathlib import Path
import pytest
from app.config import Settings
from app.providers import load_provider
from app.tools.local import TerraformRunner
SKILL = "google-cloud-iac-patterns"
ROOT = Path(__file__).resolve().parent.parent / "skills" / SKILL
REFERENCES = sorted((ROOT / "references").glob("*.md"))
_HCL = re.compile(r"^```hcl\n(.*?)^```", re.MULTILINE | re.DOTALL)
needs_terraform = pytest.mark.skipif(
shutil.which("terraform") is None, reason="terraform CLI not installed"
)
def _runner() -> TerraformRunner:
return TerraformRunner(plugin_cache_dir=Settings().terraform_plugin_cache())
def test_every_reference_is_listed_in_the_skill_and_holds_a_pattern():
skill = (ROOT / "SKILL.md").read_text(encoding="utf-8")
assert REFERENCES
for reference in REFERENCES:
assert f"references/{reference.name}" in skill
assert _HCL.findall(reference.read_text(encoding="utf-8")), reference.name
listed = set(re.findall(r"references/([\w-]+\.md)", skill))
assert listed == {reference.name for reference in REFERENCES}
def test_the_skill_pins_the_version_the_provider_bundle_pins():
pin = load_provider("gcp").terraform_providers["google"]
versions = (ROOT / "assets" / "versions.tf").read_text(encoding="utf-8")
skill = (ROOT / "SKILL.md").read_text(encoding="utf-8")
assert f'source = "{pin["source"]}"' in versions
assert f'version = "{pin["version"]}"' in versions
assert f"{pin['source']} {pin['version']}" in skill
@needs_terraform
@pytest.mark.parametrize("reference", REFERENCES, ids=lambda path: path.stem)
async def test_pattern_validates(reference: Path):
files = {
"terraform/versions.tf": (ROOT / "assets" / "versions.tf").read_text(),
"terraform/main.tf": "\n".join(_HCL.findall(reference.read_text())),
}
result = await _runner().validate(files)
if result.message.startswith("terraform init failed"):
pytest.skip("the pinned provider could not be installed")
assert result.message.startswith("terraform validate passed"), result.message

View File

@@ -218,3 +218,55 @@ def test_bundle_places_each_local_tool_in_its_own_tier():
for tier, names in bundle.tools.items():
for name in names:
assert LOCAL_TOOLS[name][0] == tier, name
def test_bundle_pins_terraform_providers_and_role_timeouts(tmp_path):
config = minimal_bundle(
tmp_path,
models={"default": "gemini/m", "timeout_seconds": {"target-architect": 420}},
terraform={
"required_providers": {
"cloud": {"source": "example/cloud", "version": "1.2.3"}
}
},
)
write_bundle(tmp_path, config)
bundle = load_provider("test", root=tmp_path)
assert bundle.terraform_providers == {
"cloud": {"source": "example/cloud", "version": "1.2.3"}
}
assert bundle.models.timeout_for("target-architect") == 420
assert bundle.models.timeout_for("decomposer") is None
@pytest.mark.parametrize(
"overrides, message",
[
(
{"terraform": {"required_providers": {"cloud": {"source": "x/cloud"}}}},
"needs a 'source' and an exact 'version'",
),
(
{"models": {"default": "gemini/m", "timeout_seconds": {"x": 0}}},
"must be positive",
),
(
{"models": {"default": "gemini/m", "timeout_seconds": {"x": "slow"}}},
"must be a number",
),
],
)
def test_bad_terraform_pins_and_timeouts_are_rejected(tmp_path, overrides, message):
write_bundle(tmp_path, minimal_bundle(tmp_path, **overrides))
with pytest.raises(ProviderConfigError, match=message):
load_provider("test", root=tmp_path)
def test_gcp_bundle_pins_one_exact_google_provider_version():
pin = load_provider("gcp").terraform_providers["google"]
assert pin["source"] == "hashicorp/google"
assert pin["version"].replace(".", "").isdigit() # exact, not a range

417
tests/tools/test_jev.py Normal file
View File

@@ -0,0 +1,417 @@
"""The Jev decision client: typed answers within strict limits, and a fallback always.
Every test runs the real TypeSafe SDK against a stand-in transport; nothing here calls
the live service.
"""
import asyncio
import json
import logging
import httpx2
from app.adk.runtime import Answer, NoDecider, Question
from app.agent import open_decider
from app.config import Settings
from app.tools.jev import JevDecider, JevLimits, estimate_tokens
URGENT = Question("noul", "Does this need attention right now?")
ROUTE = Question(
"choice", "What comes next?", ("revise_code", "revise_graph", "proceed")
)
SEVERITY = Question("score", "How much does it matter?", ("low", "medium", "high"))
ALL = {"urgent": URGENT, "route": ROUTE, "severity": SEVERITY}
ANSWERS = {
"urgent": {"type": "noul", "noul": 0.97},
"route": {
"type": "choice",
"choice": "revise_code",
"confidence": 0.81,
"probabilities": {"revise_code": 0.81, "revise_graph": 0.12, "proceed": 0.07},
},
"severity": {
"type": "score",
"score": 1.6,
"confidence": 0.7,
"legend": {"0": "low", "1": "medium", "2": "high"},
"probabilities": {"0": 0.1, "1": 0.2, "2": 0.7},
},
}
class Clock:
"""Time that moves only when something sleeps."""
def __init__(self) -> None:
self.now = 1000.0
self.slept: list[float] = []
def __call__(self) -> float:
return self.now
async def sleep(self, seconds: float) -> None:
self.slept.append(seconds)
self.now += seconds
def ok(*names: str, tokens: int = 40) -> httpx2.Response:
answers = {name: ANSWERS[name] for name in names} if names else ANSWERS
return httpx2.Response(
200,
json={
"model": "jev-latest",
"answers": answers,
"usage": {"input_tokens": tokens, "output_tokens": 0},
},
)
class Service:
"""A stand-in Jev: replies from a queue, then with `default`; records requests.
A reply answers only the questions the request asked, as the service does.
"""
def __init__(self, *replies, default=None) -> None:
self.replies = list(replies)
self.default = default
self.requests: list[httpx2.Request] = []
def __call__(self, request: httpx2.Request) -> httpx2.Response:
self.requests.append(request)
reply = self.replies.pop(0) if self.replies else self.default
if reply is None:
return ok(*json.loads(request.content)["questions"])
if isinstance(reply, Exception):
raise reply
return reply
@property
def bodies(self) -> list[dict]:
return [json.loads(request.content) for request in self.requests]
def decider(service: Service, clock: Clock | None = None, **limits) -> JevDecider:
clock = clock or Clock()
limits.setdefault("backoff_base", 0.0) # no real waiting between SDK retries
return JevDecider(
api_key="tsk_secret",
base_url="https://jev.example",
limits=JevLimits(**limits),
transport=httpx2.MockTransport(service),
clock=clock,
sleep=clock.sleep,
)
# --- the request and the typed answers -------------------------------------------------
async def test_one_request_carries_the_state_and_every_question():
service = Service()
jev = decider(service)
answers = await jev.decide("K-30 failed twice.", ALL, scope="exec-1")
[request] = service.requests
assert request.method == "POST"
assert str(request.url) == "https://jev.example/v1/systemone"
assert request.headers["Authorization"] == "Bearer tsk_secret"
body = service.bodies[0]
assert body["state"] == "K-30 failed twice."
assert body["questions"] == {
"urgent": {"type": "noul", "instructions": URGENT.instructions},
"route": {
"type": "choice",
"instructions": ROUTE.instructions,
"criteria": {"revise_code": None, "revise_graph": None, "proceed": None},
},
"severity": {
"type": "score",
"instructions": SEVERITY.instructions,
"criteria": ["low", "medium", "high"],
},
}
assert answers["route"] == Answer(
"revise_code",
0.81,
{"revise_code": 0.81, "revise_graph": 0.12, "proceed": 0.07},
)
assert answers["severity"] == Answer(1.6, 0.7, {"0": 0.1, "1": 0.2, "2": 0.7})
# A yes-or-no answer is a probability; its confidence is its distance from even.
assert answers["urgent"].value == 0.97
assert round(answers["urgent"].confidence, 2) == 0.94
assert answers["urgent"].is_yes() and not Answer(0.2, 0.6).is_yes()
async def test_a_choice_outside_the_options_or_a_missing_answer_is_no_decision():
invented = httpx2.Response(
200,
json={
"model": "jev-latest",
"usage": {"input_tokens": 1, "output_tokens": 0},
"answers": {
"route": {
"type": "choice",
"choice": "delete_everything",
"confidence": 0.99,
"probabilities": {},
}
},
},
)
assert await decider(Service(invented)).decide("s", {"route": ROUTE}) is None
assert await decider(Service(ok("urgent"))).decide("s", ALL) is None
assert (
await decider(Service(httpx2.Response(200, text="not json"))).decide(
"s", {"urgent": URGENT}
)
is None
)
async def test_no_questions_needs_no_request():
service = Service()
assert await decider(service).decide("state", {}) == {}
assert service.requests == []
# --- size --------------------------------------------------------------------------------
async def test_a_request_over_the_size_limit_is_refused_locally_never_truncated():
service = Service()
jev = decider(service, max_tokens=100, max_question_tokens=100)
state = "x" * 600
assert estimate_tokens(state) > 100
assert await jev.decide(state, {"urgent": URGENT}) is None
assert service.requests == []
assert jev.records[-1].outcome == "too_large"
# --- timeouts and retries (the SDK's, configured here) ------------------------------------
async def test_temporary_failures_are_retried_then_answered():
service = Service(httpx2.Response(503), httpx2.ConnectTimeout("slow"))
jev = decider(service)
assert await jev.decide("s", {"urgent": URGENT}) is not None
assert len(service.requests) == 3
async def test_a_rate_limit_that_outlasts_the_retries_lowers_the_clients_own_rate():
service = Service(default=httpx2.Response(429))
jev = decider(service, retries=1)
assert await jev.decide("s", {"urgent": URGENT}) is None
assert len(service.requests) == 2
assert jev.records[-1].rate_limited
assert jev._rate == JevLimits().requests_per_second / 2
async def test_a_rejected_request_is_not_retried():
service = Service(default=httpx2.Response(400, json={"error": "bad question"}))
jev = decider(service)
assert await jev.decide("s", {"urgent": URGENT}) is None
assert len(service.requests) == 1
async def test_retries_stop_at_the_budget_and_the_caller_falls_back():
service = Service(default=httpx2.Response(503))
jev = decider(service, retries=2)
assert await jev.decide("s", {"urgent": URGENT}) is None
assert len(service.requests) == 3
assert jev.records[-1].outcome == "failed"
# --- rate and concurrency ----------------------------------------------------------------
async def test_calls_beyond_the_burst_wait_for_the_bucket_to_refill():
clock = Clock()
service = Service()
jev = decider(service, clock, burst=2, requests_per_second=4)
for index in range(3):
await jev.decide(f"state {index}", {"urgent": URGENT})
assert len(service.requests) == 3
assert clock.slept == [0.25] # one token at four a second
async def test_calls_in_flight_never_exceed_the_cap():
in_flight = peak = 0
async def slow(request: httpx2.Request) -> httpx2.Response:
nonlocal in_flight, peak
in_flight += 1
peak = max(peak, in_flight)
await asyncio.sleep(0.01)
in_flight -= 1
return ok("urgent")
jev = JevDecider(
api_key="k",
base_url="https://jev.example",
limits=JevLimits(max_in_flight=2, burst=50),
transport=httpx2.MockTransport(slow),
)
await asyncio.gather(
*(jev.decide(f"state {index}", {"urgent": URGENT}) for index in range(8))
)
assert peak == 2
# --- circuit breaker ---------------------------------------------------------------------
async def test_repeated_failures_open_the_circuit_and_one_probe_closes_it():
clock = Clock()
service = Service(default=httpx2.Response(503))
jev = decider(
service, clock, retries=0, breaker_failures=3, breaker_cooldown=60, burst=50
)
for index in range(3):
assert await jev.decide(f"s{index}", {"urgent": URGENT}) is None
sent = len(service.requests)
# Open: the service is not called at all.
assert await jev.decide("s-open", {"urgent": URGENT}) is None
assert len(service.requests) == sent
assert jev.records[-1].outcome == "breaker_open"
# After the cool-down a single call probes; its success closes the circuit.
clock.now += 61
service.default = None
assert await jev.decide("s-probe", {"urgent": URGENT}) is not None
assert await jev.decide("s-after", {"urgent": URGENT}) is not None
async def test_a_failed_probe_reopens_the_circuit():
clock = Clock()
service = Service(default=httpx2.Response(503))
jev = decider(
service, clock, retries=0, breaker_failures=2, breaker_cooldown=60, burst=50
)
for index in range(2):
await jev.decide(f"s{index}", {"urgent": URGENT})
clock.now += 61
await jev.decide("probe", {"urgent": URGENT})
sent = len(service.requests)
assert await jev.decide("again", {"urgent": URGENT}) is None
assert len(service.requests) == sent
# --- cache and budget --------------------------------------------------------------------
async def test_the_same_question_about_the_same_state_is_answered_from_memory():
service = Service()
jev = decider(service)
first = await jev.decide("unchanged", {"route": ROUTE})
second = await jev.decide("unchanged", {"route": ROUTE})
await jev.decide("changed", {"route": ROUTE})
assert first == second
assert len(service.requests) == 2
assert [r.outcome for r in jev.records] == ["answered", "cached", "answered"]
async def test_each_execution_has_a_budget_of_calls_and_then_falls_back(caplog):
service = Service()
jev = decider(service, budget_calls=2, burst=50)
with caplog.at_level(logging.WARNING):
results = [
await jev.decide(f"state {index}", {"urgent": URGENT}, scope="exec-1")
for index in range(4)
]
other = await jev.decide("state", {"urgent": URGENT}, scope="exec-2")
assert [result is not None for result in results] == [True, True, False, False]
assert other is not None # another Execution has its own budget
assert len(service.requests) == 3
assert caplog.text.count("budget reached for exec-1") == 1
async def test_each_execution_has_a_budget_of_tokens():
service = Service(default=ok("urgent", tokens=600))
jev = decider(service, budget_tokens=1000, burst=50)
results = [
await jev.decide(f"state {index}", {"urgent": URGENT}, scope="exec-1")
for index in range(3)
]
assert [result is not None for result in results] == [True, True, False]
async def test_the_key_is_never_logged(caplog):
service = Service(httpx2.Response(500), httpx2.Response(401))
jev = decider(service, retries=0)
with caplog.at_level(logging.DEBUG):
await jev.decide("a", {"urgent": URGENT})
await jev.decide("b", {"urgent": URGENT})
assert "tsk_secret" not in caplog.text
# --- what is recorded, and running without Jev -------------------------------------------
async def test_every_decision_is_recorded_for_the_evaluation_report():
service = Service(ok("urgent"), default=httpx2.Response(400))
jev = decider(service)
await jev.decide("first", {"urgent": URGENT}, scope="exec-1")
await jev.decide("first", {"urgent": URGENT}, scope="exec-1")
await jev.decide("second", {"urgent": URGENT}, scope="exec-1")
assert round(jev.records[0].confidence["urgent"], 2) == 0.94
assert jev.summary() == {
"decisions": 3,
"outcomes": {"answered": 1, "cached": 1, "failed": 1},
"fallbacks": 1,
"rate_limited": 0,
"seconds": round(jev.records[0].seconds, 3),
"input_tokens": 40,
}
async def test_without_a_key_every_caller_takes_its_fallback():
unset = open_decider(Settings(_env_file=None, TYPESAFE_API_KEY=None))
assert isinstance(unset, NoDecider)
assert await unset.decide("state", {"urgent": URGENT}) is None
async def test_with_a_key_the_client_takes_its_limits_from_settings():
configured = open_decider(
Settings(
_env_file=None,
TYPESAFE_API_KEY="tsk_x",
TYPESAFE_BASE_URL="https://jev.internal",
JEV_REQUESTS_PER_SECOND=2,
JEV_BUDGET_CALLS=40,
)
)
assert isinstance(configured, JevDecider)
assert configured.limits.requests_per_second == 2
assert configured.limits.budget_calls == 40
assert configured.limits.max_tokens == 48_000
await configured.aclose()

View File

@@ -1,5 +1,6 @@
"""Local tools: Mermaid check and Terraform fmt/validate/plan in a scratch directory."""
import json
import shutil
from pathlib import Path
@@ -140,6 +141,110 @@ async def test_plan_never_runs_apply_and_refuses_unsafe_variables(monkeypatch):
assert "-var=region=europe-west2" in plan
# --- schema lookup -----------------------------------------------------------------------
SCHEMA = {
"provider_schemas": {
"registry.terraform.io/example/cloud": {
"resource_schemas": {
"cloud_service": {
"block": {
"attributes": {
"name": {"type": "string", "required": True},
"labels": {"type": ["map", "string"], "optional": True},
"uri": {"type": "string", "computed": True},
},
"block_types": {
"template": {
"nesting_mode": "list",
"max_items": 1,
"block": {
"attributes": {
"cpu_boost": {
"type": "bool",
"optional": True,
"description": "Boost CPU at start.",
}
}
},
}
},
}
},
"cloud_job": {"block": {"attributes": {"name": {"type": "string"}}}},
}
}
}
}
PIN = {"cloud": {"source": "example/cloud", "version": "1.2.3"}}
def schema_runner(monkeypatch) -> tuple[TerraformRunner, list[list[str]]]:
commands: list[list[str]] = []
async def fake_run(self, args, cwd, env=None):
commands.append(list(args))
if args[1] == "init":
assert (
'cloud = { source = "example/cloud", version = "1.2.3" }'
in (Path(cwd) / "versions.tf").read_text()
)
return 0, "ok"
return 0, json.dumps(SCHEMA)
monkeypatch.setattr(TerraformRunner, "_run", fake_run)
return TerraformRunner(providers=PIN), commands
async def test_schema_lists_what_code_may_set_and_outlines_the_blocks(monkeypatch):
runner, _ = schema_runner(monkeypatch)
result = await runner.schema("cloud_service")
assert result.message.startswith("cloud_service (example/cloud 1.2.3)")
assert " name (string, required)" in result.message
assert " labels (map of string, optional)" in result.message
assert "uri" not in result.message # read-only: code cannot set it
assert " template { } (list, at most 1): cpu_boost" in result.message
async def test_schema_gives_one_block_in_full(monkeypatch):
runner, _ = schema_runner(monkeypatch)
result = await runner.schema("cloud_service", block="template")
missing = await runner.schema("cloud_service", block="template.containers")
assert " cpu_boost (bool, optional): Boost CPU at start." in result.message
assert "'containers' is not a block there" in missing.message
async def test_schema_says_where_a_setting_sits_or_that_it_cannot_be_set(monkeypatch):
runner, _ = schema_runner(monkeypatch)
found = await runner.schema("cloud_service", find="cpu_boost")
absent = await runner.schema("cloud_job", find="cpu_boost")
assert "template.cpu_boost (bool, optional)" in found.message
assert "It cannot be set on this resource type." in absent.message
async def test_schema_suggests_close_resource_types_and_is_loaded_once(monkeypatch):
runner, commands = schema_runner(monkeypatch)
unknown = await runner.schema("cloud_servce")
await runner.schema("cloud_job")
assert "Close matches: cloud_service" in unknown.message
assert [command[1] for command in commands] == ["init", "providers"]
assert "apply" not in [arg for command in commands for arg in command]
async def test_schema_is_unavailable_without_a_pinned_provider():
result = await TerraformRunner().schema("cloud_service")
assert "pins no Terraform provider" in result.message
# --- registry ------------------------------------------------------------------------------
@@ -155,6 +260,7 @@ def test_local_tools_are_built_by_name_with_their_tier():
"terraform_fmt",
"terraform_validate",
"terraform_plan",
"terraform_schema",
}
with pytest.raises(KeyError, match="terraform_apply"):
local_tools(["terraform_apply"], TerraformRunner())