This commit is contained in:
@@ -9,8 +9,8 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8 and 15 built; no evaluation run has completed on them; 9 awaits the owner; 10–14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. |
|
||||
| 5 | `iter-05-jev-decisions/` | planned, gated on open questions | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–20 built; no evaluation run has completed on them, so their effect on time and tokens is unmeasured; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; step 11's evaluation run is outstanding, so unproven with real models | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
|
||||
@@ -137,3 +137,35 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Rationale:** Each check runs `terraform init` in a new scratch directory, so every call downloaded its providers again. The setting `TF_PLUGIN_CACHE_DIR` existed and was unset by default.
|
||||
- **Where:** `TerraformRunner` in `app/tools/local.py`, `app/config.py`, `.env.example`.
|
||||
- **Acceptance criteria:** The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.
|
||||
|
||||
## Step 17: A working copy for files and for the graph
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
|
||||
- **Capability:** A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
|
||||
- **Rationale:** In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every `terraform_fmt` and `terraform_validate` call carried every file, `terraform_fmt` returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
|
||||
- **Where:** `app/tools/workspace.py` (new), `specialist_tools` in `app/tools/execution.py`, `app/tools/local.py`, `app/tools/toolkit.py`, `DeliverableSpec.working_copy` in `app/workflows/deliverables.py`, `render_tables` in `app/workflows/target_architecture.py`, `agents/target-architect.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
|
||||
- **Acceptance criteria:** The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.
|
||||
|
||||
## Step 18: Provider schema lookup and one pinned provider version
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** An agent can ask what a resource type accepts, and get the provider's own answer.
|
||||
- **Rationale:** All 13 Terraform refusals in the 2026-10-07 run were one mistake: `startup_cpu_boost` and `cpu_idle` in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
|
||||
- **Where:** `TerraformRunner.schema` in `app/tools/local.py`, `terraform.required_providers` in `providers/gcp.yaml`, `app/providers.py`, the role files of the target architect, the IaC author and the validator.
|
||||
- **Acceptance criteria:** `terraform_schema` lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.
|
||||
|
||||
## Step 19: A Check the code cannot meet goes back to the Target architecture
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
|
||||
- **Rationale:** Check K-30 of the 2026-10-07 run asked for `startup_cpu_boost` on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
|
||||
- **Where:** `check_implementation` and `unimplementable_checks` in `app/workflows/chain.py`, the playbook in `app/nodes/router.py`, `_orchestrator_context` in `app/nodes/execution_nodes.py`, `agents/orchestrator.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** `implementation.json` may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.
|
||||
|
||||
## Step 20: A model timeout per role
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** A role whose replies are long is given longer for one model request than the rest.
|
||||
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
|
||||
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
|
||||
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Iteration:** 4, learning and Skills
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8 and 15 built; no evaluation run has completed on them (see `steps/step-8.md`); step 9 drafted and awaiting the owner; steps 10 to 14 planned. Steps 1 to 6 are in commit `7ee49ab`; 7, 8 and 15 are uncommitted.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; no evaluation run has completed on any of them (see `steps/step-8.md`), so every "tokens fall" or "Revisions fall" criterion is unmeasured; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 16
|
||||
**This directory covers:** steps 1 to 20
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -27,12 +27,16 @@
|
||||
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
|
||||
| 9 | ADR: the lessons loop and overlay Skills | B | S | drafted as `docs/adr/0004-lessons-and-overlays-arrive-as-skills.md`, status proposed; awaits the owner |
|
||||
| 10 | Lessons Skill and lesson curator | B | L | planned |
|
||||
| 11 | IaC patterns Skill | B | M | planned |
|
||||
| 11 | IaC patterns Skill | B | M | built; every pattern validates; effect on Revisions and tokens unmeasured |
|
||||
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
|
||||
| 13 | Trim Specialist context | C | M | planned |
|
||||
| 13 | Trim Specialist context | C | M | built; effect on tokens unmeasured |
|
||||
| 14 | Provider-owned contracts | C | M | planned |
|
||||
| 15 | Group Executions by Customer case | added | S | complete |
|
||||
| 16 | Cache Terraform providers between checks | added | S | complete |
|
||||
| 17 | A working copy for files and for the graph | added | M | built; effect on time and tokens unmeasured |
|
||||
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
@@ -49,5 +53,7 @@ Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that
|
||||
- Step 14: depends on step 4
|
||||
- Step 15: no dependencies (added on 2026-10-06 at the owner's request)
|
||||
- Step 16: no dependencies (added on 2026-10-07 at the owner's request)
|
||||
- Steps 17 to 20: no dependencies (added on 2026-10-07 at the owner's request, from the causes of the 60 minute run recorded in `../iter-06-target-architecture-graph/steps/`)
|
||||
- Step 11 uses step 18's pinned provider version; step 19 uses step 18's lookup to find the Checks it reports
|
||||
|
||||
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.
|
||||
|
||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-11.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-11.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 11 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07 as `skills/google-cloud-iac-patterns/`. The name carries the Cloud provider because the content does; the provider bundle assigns it to the IaC author (`role_skills.iac-author`), so another provider brings its own patterns Skill with no code change.
|
||||
|
||||
1. `SKILL.md` says how to use the patterns and lists one reference per product. `assets/versions.tf` pins `hashicorp/google` 8.6.0, the version `providers/gcp.yaml` pins (step 18).
|
||||
2. Thirteen references, each a short list of what goes wrong and one self-contained pattern: enabling APIs, IAM, Cloud Run service, Cloud Run job, Pub/Sub, Cloud Storage, Secret Manager, Cloud KMS, Cloud SQL, VPC networking, Cloud Scheduler, Artifact Registry, Monitoring and Logging.
|
||||
3. Seeded from the 2026-10-07 run's lesson candidates. All 13 Terraform refusals were `startup_cpu_boost` or `cpu_idle` in the wrong block or on a Cloud Run job. The service reference says which block each setting sits in; the job reference lists what a job does not have.
|
||||
|
||||
Differs from the plan: the plan's acceptance is a fall in IaC author Revisions or tokens on the three cases. **No evaluation has run, so that is unmeasured.** What is verified is that the patterns are correct.
|
||||
|
||||
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
|
||||
|
||||
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-13.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-13.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 13 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `DeliverableSpec.reads_in_full` names the earlier Deliverables a Specialist reads whole. Left unset it is the Deliverables it depends on.
|
||||
2. `dependencies_for(kind, deliverables)` gives every other input as its data file alone: `requirements.json`, `components.json` and so on. Those ids are all the Requirements chain checks need, so submission checks are unchanged.
|
||||
3. Two overrides. The IaC author reads nothing whole: the code is written from `target-architecture.json`. The packager reads everything whole, since the guide is assembled from every Deliverable's prose.
|
||||
4. A Deliverable given as data can be read whole with `read_deliverable(kind)`, offered only when something was trimmed.
|
||||
|
||||
Differs from the plan: the plan said "summaries". The data file is used instead. It already exists, is exact, and costs no model call to produce.
|
||||
|
||||
**Unmeasured:** the plan's acceptance is that describer tokens fall against the step 8 baseline. No evaluation has run.
|
||||
|
||||
Tests: `tests/execution/test_working_copy.py`, "each Specialist reads what it works from".
|
||||
21
.agents/plans/iter-04-learning-and-skills/steps/step-17.md
Normal file
21
.agents/plans/iter-04-learning-and-skills/steps/step-17.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Step 17 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07. `app/tools/workspace.py` keeps a working copy in the Deliverable's working memory, which the Execution already checkpoints and returns on every attempt.
|
||||
|
||||
Files (`working["files"]`), for the Deliverable whose `working_copy` is its own kind (deployment guidance):
|
||||
|
||||
1. `write_file`, `edit_file` (replaces the one place `old` matches, and refuses when it matches none or several), `read_file` (one file, or the list) and `delete_file`.
|
||||
2. `terraform_fmt`, `terraform_validate` and `terraform_plan` run on the Terraform files in the working copy and take no `files` argument. `terraform_fmt` formats in place and returns the names it changed. Files passed anyway are written to the working copy first, so a model that ignores the instruction still works.
|
||||
3. `submit_deliverable` attaches the working copy; a file passed as an attachment replaces its copy. The submitted files become the working copy, so a Revision starts from them, and the task context lists them under `working_copy`.
|
||||
|
||||
The validator's `working_copy` is deployment guidance: its tools are given a fresh copy of the submitted files on every attempt, it has no write tools, and nothing from the copy is attached to its Deliverable.
|
||||
|
||||
Graph (`working["graph"]`), for the Target architecture:
|
||||
|
||||
4. `graph_put` adds items or replaces those with the same id, `graph_remove` removes by id, `graph_read` returns the graph. The submitted graph becomes the working graph.
|
||||
5. `submit_deliverable` attaches the working graph when no `target-architecture.json` is passed.
|
||||
6. `render_tables` writes the Elements, Relationships, Checks and not-applicable tables from the graph, and the engine appends them to the content on submission. The contract's sections are now `Design rationale` alone. This closes the duplicate output iteration 6 left: the architect no longer writes the graph twice.
|
||||
|
||||
**Unmeasured:** no evaluation has run, so the fall in requests, tokens and time is expected, not shown. The mechanism is what is tested.
|
||||
|
||||
Tests: `tests/execution/test_working_copy.py` (21), including an Execution in which the validator validates the submitted code with no files passed.
|
||||
17
.agents/plans/iter-04-learning-and-skills/steps/step-18.md
Normal file
17
.agents/plans/iter-04-learning-and-skills/steps/step-18.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Step 18 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `providers/gcp.yaml` pins `terraform.required_providers.google` to `hashicorp/google` 8.6.0. `load_provider` requires a `source` and an exact `version`.
|
||||
2. `TerraformRunner.schema(resource_type, block, find)` writes a versions file for the pinned providers in a scratch directory, runs `terraform init -backend=false` and `terraform providers schema -json` once per process, and keeps the resource schemas in memory. Measured here: 0.6 seconds and a 9.7 MB reply for the Google provider, from the plugin cache.
|
||||
3. With `resource_type` alone it lists the arguments code may set (read-only ones are left out) and outlines every block. With `block` it gives one block's arguments with descriptions. With `find` it says where a setting sits, or that the resource type has none. An unknown resource type gets close matches.
|
||||
4. The tool is `terraform_schema`, tier general, given to the target architect, the IaC author and the validator.
|
||||
|
||||
Checked against the real provider on 2026-10-07:
|
||||
|
||||
- `find="startup_cpu_boost"` on `google_cloud_run_v2_job`: "has no argument or block named like 'startup_cpu_boost'. It cannot be set on this resource type."
|
||||
- the same on `google_cloud_run_v2_service`: `template.containers.resources.startup_cpu_boost (bool, optional)`.
|
||||
|
||||
Not done: generated code is not forced onto the pinned version. An agent that writes its own `required_providers` with another version is validated against that version. The patterns Skill gives it the pinned `versions.tf` to use.
|
||||
|
||||
Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `tests/test_providers.py` for the pin.
|
||||
12
.agents/plans/iter-04-learning-and-skills/steps/step-19.md
Normal file
12
.agents/plans/iter-04-learning-and-skills/steps/step-19.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Step 19 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. An `implementation.json` entry may be `{"check": "K-7", "unimplementable": "<what the resource type lacks>"}` in place of `file` and `address`. It accounts for the Check; an empty reason does not.
|
||||
2. `unimplementable_checks` reads those entries. The router reads them as soon as deployment guidance is submitted, and a new playbook group, "the code cannot meet a Check", sits before "validation decided" and "a Phase is complete". Its action reopens the Target architecture with each Check's id and reason. That returns the code to draft, so the group stops applying.
|
||||
3. A model Orchestrator is given the same list as `unimplementable_checks` in its context, and `agents/orchestrator.md` has the step.
|
||||
4. The validator is told to confirm with `terraform_schema` before failing a Check for a missing setting, and to give not verifiable when the resource type has no such setting. Not verifiable already routes to the Target architecture.
|
||||
|
||||
Past the Revision limit the reopen is refused as any other is: the user decides, or a pre-approved Execution ends with warnings.
|
||||
|
||||
Tests: `tests/execution/test_requirements_chain.py` (two) and `tests/execution/test_router.py`, "a Check the code cannot meet goes back to the graph unvalidated".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 20 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07. `models.timeout_seconds` in the provider bundle maps a role to the seconds one model request may take; `RoleSpec.model_timeout_seconds` carries it and `AdkAgentRuntime` uses it in place of `MODEL_TIMEOUT_SECONDS`.
|
||||
|
||||
`providers/gcp.yaml` sets the target architect to 420 and the IaC author and the packager to 300. These are starting values, not measurements. With step 17 the architect's replies are short, so its 420 should rarely be reached.
|
||||
|
||||
Tests: `tests/agents/test_adk_runtime.py`, `tests/agents/test_roles.py`, `tests/test_providers.py`.
|
||||
@@ -135,6 +135,12 @@ Rules for every step from 3 onward:
|
||||
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
|
||||
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
|
||||
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
|
||||
- **Routing (owner's direction, 2026-10-07):** the Orchestrator acts as a router with Jev choosing the route. Four things are routed:
|
||||
- a message that starts something: new Workflow request, Change request against a Baseline, question about delivered work, or out of scope (Choice);
|
||||
- a message inside an Execution: an answer to a waiting question, a request to change a named Deliverable, or anything else (Choice, then a Choice over the Deliverables);
|
||||
- a validation outcome: revise the code, revise the Target architecture, or proceed (Choice);
|
||||
- in a Change request, which Deliverables the change reaches (a Boolean per Deliverable, with the Requirements chain as the deterministic check).
|
||||
The route decides what is dispatched. Which Specialist produces a Deliverable stays fixed by the dependency graph, and every Deliverable is still produced in a new Execution. The first route needs an entry point that can start a Change request or answer from a case's files (`app/main.py`, `app/a2a_server.py`); the last depends on iteration 7.
|
||||
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
|
||||
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
|
||||
@@ -169,3 +175,33 @@ Rules for every step from 3 onward:
|
||||
- **Fallback:** Treat it as changed.
|
||||
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
|
||||
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
|
||||
|
||||
## Step 16: Jev as the evaluation judge, calibrated
|
||||
|
||||
- **Capability:** The evaluation's judged checks are answered by Jev, and the judge is shown to agree with a person.
|
||||
- **Rationale:** LangChain's evaluation posts report Jev agreeing with a human reviewer on every pass-or-fail decision in a small experiment, with far lower variance than language-model judges, at $0.00035 a call. They also warn that low cost can amplify mistakes and that human review and judge alignment are still needed.
|
||||
- **Built:** `DeciderJudge` in `eval/judge.py` asks every question in one call, with the document as the state and no grading instructions in it. `eval/solution_eval.py --judge` uses it when `TYPESAFE_API_KEY` is set, and the language-model judge otherwise or with `EVAL_JUDGE=gemini`. A judge that gives no answer leaves its check unanswered, not passed.
|
||||
- **Open:** a small set of design recommendations labelled by a person against the pillar checklists and the quality rubric; agreement of each judge with those labels; variance over repeated runs on the same output. This replaces the stability criterion in step 10.
|
||||
- **Acceptance criteria:** On the labelled set, the Jev judge's pass-or-fail agreement and its variance are recorded beside the language-model judge's, and the evaluation uses the one that agrees better.
|
||||
|
||||
## Step 17: Trace every decision
|
||||
|
||||
- **Capability:** For any Execution, a person can see each decision, what it was given and what came back.
|
||||
- **Rationale:** LangChain's production post treats tracing as required for model-driven steps and describes LangSmith views for decision models. Here decisions are kept in the Execution's state and the evaluation report, and nothing is traced.
|
||||
- **Owner's decision needed:** whether to send traces to LangSmith. Traces carry the Workflow request and Deliverable text, so the same retention question as Jev applies. `langsmith` is already installed as a dependency of LangChain; LangGraph traces when its environment variables are set.
|
||||
- **Where:** environment settings for tracing; a traced span around `JevDecider.decide` in `app/tools/jev.py` carrying the questions, answers and probabilities; `docs/operations/`.
|
||||
- **Acceptance criteria:** One Execution's trace shows every Orchestrator turn, each Jev call with its inputs and outputs, and each Specialist run. With tracing off, nothing leaves the process.
|
||||
|
||||
## Step 18: Readings as their own graph step
|
||||
|
||||
- **Capability:** A resumed Execution reuses the decisions it already made.
|
||||
- **Rationale:** LangGraph saves state after each node. The router takes its readings and acts inside one node, so a failure in that node asks Jev again on retry. Jev is designed to return the same answer for the same input, and the client caches answers in memory, but neither survives a restart.
|
||||
- **Where:** `app/workflows/execution_graph.py` and `app/nodes/`: a node that takes the readings and saves them, before the node that applies the playbook.
|
||||
- **Acceptance criteria:** A test that fails the playbook node after the readings node shows the retry using the saved readings, with no second call to the decider.
|
||||
|
||||
## Step 19: Online evaluation of live Executions
|
||||
|
||||
- **Capability:** Every Execution is scored as it runs, not only benchmark cases.
|
||||
- **Rationale:** At Jev's cost, LangChain recommends scoring every run and alerting on risk flags such as leaked personal data or injected instructions.
|
||||
- **Where:** the judged checks from `eval/scoring.py`, run when an Execution ends and stored with its record; with step 17, as feedback on its trace.
|
||||
- **Acceptance criteria:** A completed Execution carries its judged scores. A flagged risk is recorded and surfaced without stopping the Execution.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 15
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 19
|
||||
|
||||
## What Jev is
|
||||
|
||||
@@ -49,6 +49,24 @@ On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use
|
||||
|
||||
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
||||
|
||||
## Alignment with LangChain's published guidance
|
||||
|
||||
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
||||
|
||||
| Their guidance | Here | Step |
|
||||
|---|---|---|
|
||||
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
||||
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
||||
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
||||
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
||||
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
||||
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
||||
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
||||
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
||||
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
||||
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
||||
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
@@ -58,7 +76,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
@@ -67,11 +85,15 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
|
||||
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -90,5 +112,9 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
- Step 15: depends on step 2, iteration 7 step 4
|
||||
- Step 16: depends on step 2
|
||||
- Step 17: no dependencies
|
||||
- Step 18: depends on step 12
|
||||
- Step 19: depends on step 16, step 17
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
|
||||
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 11 — Implementation Detail
|
||||
|
||||
ADR-0007 was accepted by the owner on 2026-10-07 ("this is exactly what I want, I approve ADR-0007"), after three refinements made the same day and recorded in it:
|
||||
|
||||
1. Jev is the decision point, in place of a different language model.
|
||||
2. The agent-control pattern at https://vercel.com/i/jev-agent-control is followed: a decision model reads, application code checks permission and executes, a generative model writes replies.
|
||||
3. The Orchestrator is a router, and decides by readings, a disposition and a playbook, on the model of a triage pipeline the owner supplied: the first group that applies, the strongest action whose conditions hold, escalate when none applies.
|
||||
|
||||
Jev is reached through TypeSafe AI's Python SDK and API endpoint (owner, 2026-10-07). The API key is still to be supplied.
|
||||
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Step 12 — Implementation Detail
|
||||
|
||||
Built against a scripted decider. No call has been made to the live Jev service, and no evaluation has run on it: the owner holds evaluation runs until the Jev work is in.
|
||||
|
||||
1. **`app/nodes/router.py`.** `Router` decides an Orchestrator turn in three stages: readings (`_read_ledger` from the Ledger and data; `_read_message` and `_read_gap` from the decider), a disposition by `Thresholds` (act at 0.75, close at 0.15, choice confidence 0.6), and the playbook in `_choose`: ordered groups, the strongest action whose conditions hold, escalate when none applies.
|
||||
2. **Permission stays with the tools.** The router calls the same `OrchestratorToolbox` tools the model did. A refused action is skipped for the rest of the turn and the next one that holds is taken. Past the Revision limit the user is asked.
|
||||
3. **What Jev is asked** (`Question` objects, batched when they rest on the same evidence):
|
||||
- of the user's message: `asks_change` (noul), `change_target` (choice over submitted Deliverables), `change_clarity` (score: vague, partial, specific);
|
||||
- of validation gaps: `gap_material` (noul);
|
||||
- of the user's reply about gaps: `wants_revision` (noul).
|
||||
What is sent is Deliverable names, the count of waiting questions, the user's message, and for gaps the proposed Check statements and unverifiable Check ids.
|
||||
4. **Rules when Jev gives no answer.** A requested change is acted on only when one Deliverable is named outright, otherwise the user is asked which. A free-text reply is treated as an answer. A gap is revised for when a Check cannot be verified or an addition names a real requirement or recommendation id.
|
||||
5. **A new reading from the run of 2026-10-07:** a Check that fails in two validations running is sent to the Target architecture, not the code. That run retried one unimplementable Check as code three times.
|
||||
6. **Decisions are recorded** in the Execution's state under `decisions`: turn, group, action and what was judged, with confidence.
|
||||
7. **The reply writer.** A new role, `agents/reply-writer.md`, writes the Approval summary from facts (`approval_facts`) through one tool. When it fails or is absent, `assembled_summary` builds the summary from the same facts.
|
||||
8. **Wiring.** `ORCHESTRATOR=router` (the default) or `model`; `open_assembly` builds the router with the decider and the thresholds (`JEV_ACT_THRESHOLD`, `JEV_CLOSE_THRESHOLD`, `JEV_CONFIDENCE_THRESHOLD`). State gained `last_reply`, `pending_change`, `awaiting_gap_decision`, `validation_settled` and `decisions`.
|
||||
|
||||
Changed from the plan and the ADR's first wording:
|
||||
|
||||
- **Jev does not propose the next action.** It supplies readings, and the playbook in code picks the action. This follows the owner's triage model and keeps every decision explainable by a rule and a threshold.
|
||||
- **The model Orchestrator is kept** behind `ORCHESTRATOR=model`. The 40 or so engine tests drive the Ledger's rules by scripting its tool calls in arbitrary order, which the router cannot do. `agents/orchestrator.md` is therefore unchanged and unused in router mode.
|
||||
- **An unclear change is confirmed once.** If the reply still does not settle it, a named Deliverable is acted on and anything else is dropped.
|
||||
- **A pre-approved Execution's automatic replies are not routed.** There is no user message to read.
|
||||
|
||||
Not built from the ADR's routes:
|
||||
|
||||
- **A message that starts something** (new request, Change request, question about delivered work, out of scope). It needs an entry point that can answer from a case's files, and Change requests (iteration 7).
|
||||
- **Which Deliverables a Change request reaches.** Iteration 7.
|
||||
|
||||
Tests: `tests/execution/test_router.py` (18), `tests/test_assembly.py`.
|
||||
|
||||
Unverified:
|
||||
|
||||
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
|
||||
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
|
||||
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
|
||||
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
Built ahead of step 1's answers, on the owner's instruction of 2026-10-07 that no evaluation be rerun until the Jev work is in. Nothing here has called the live service.
|
||||
|
||||
1. **The seam** (`app/adk/runtime.py`): `Question` (kind choice, score or noul, with instructions and options), `Answer` (value, confidence, probabilities) and the `Decider` protocol. `decide` returns the answers, or `None` when no decision could be made, and the caller then takes its fallback. `NoDecider` always returns `None`.
|
||||
2. **The client** (`app/tools/jev.py`): `JevDecider` calls Jev through TypeSafe AI's official Python SDK, `typesafe-sdk` 0.7.2 (`AsyncTypeSafeClient.system_one`). The owner chose the SDK and its API endpoint on 2026-10-07. It is the only module that imports the SDK.
|
||||
3. **Wiring:** `open_decider` in `app/agent.py` returns the client when `TYPESAFE_API_KEY` is set and `NoDecider` otherwise; the `Assembly` carries it.
|
||||
4. **A scripted decider** for tests, `FakeDecider` in `app/adk/fake_runtime.py`.
|
||||
|
||||
Which package: three names exist on the index. `typesafe-sdk` is the vendor's (author TypeSafe AI, repository `typesafe-ai/typesafe-sdk-python`). `typesafe` is an unrelated type-checking library, and `typesafe-ai` is a third party's redirect shim; neither is used.
|
||||
|
||||
What the SDK settled, read from the installed package and a stand-in transport:
|
||||
|
||||
- The request is `POST /v1/systemone` with a bearer key; default endpoint `https://api.typesafe.ai`, default model `jev-latest`.
|
||||
- Questions are `Noul(instructions)`, `Choice(instructions, criteria={option: description})` and `Score(instructions, criteria=[levels])`.
|
||||
- Answers are `noul` (a probability, with no confidence of its own), `choice` with `confidence` and `probabilities`, and `score` with `confidence`. Usage reports `input_tokens` only: no cost and no remaining credit.
|
||||
|
||||
Limits kept, each with a test in `tests/tools/test_jev.py`:
|
||||
|
||||
| Practice | Built as |
|
||||
|---|---|
|
||||
| Size | Refused locally above 48,000 estimated tokens in all, or 24,000 for the state plus the longest question. Never truncated. |
|
||||
| Batching | Every question about one state goes in one request. |
|
||||
| Timeout | 5 seconds a request, 15 for a decision with its retries (the SDK's). |
|
||||
| Retries | 2, by the SDK: on 408, 429, 5xx, connection and timeout errors, with jittered backoff, honouring `Retry-After`. |
|
||||
| Rate | A token bucket, 5 a second with a burst of 10; the rate halves when a rate limit outlasts the retries. |
|
||||
| Concurrency | At most 8 requests in flight. |
|
||||
| Circuit breaker | Opens after 5 failures in 30 seconds for 60 seconds, then lets one probe through. |
|
||||
| Cache | Answers kept by a hash of the state and questions, up to 1,024. |
|
||||
| Budget | 500 calls or 1,000,000 input tokens per scope (the Execution), then fallbacks with one warning. |
|
||||
| Records | Each decision's outcome, seconds, rate limiting, input tokens and confidence; `summary()` totals them. |
|
||||
| Key | From `TYPESAFE_API_KEY` only; never logged. |
|
||||
|
||||
Settings: `TYPESAFE_API_KEY`, `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL` (the SDK's own names), and `JEV_TIMEOUT_SECONDS`, `JEV_REQUESTS_PER_SECOND`, `JEV_MAX_IN_FLIGHT`, `JEV_BUDGET_CALLS`, `JEV_BUDGET_TOKENS`.
|
||||
|
||||
Changed from the plan:
|
||||
|
||||
- The cache is per client, not per Execution. A decision depends only on its state and questions, so two Executions asking the same thing can share the answer.
|
||||
- A reply that leaves any question unanswered, or picks a choice that was not offered, counts as no decision for the whole request.
|
||||
- The budget is in calls and input tokens, not dollars, and there is no low-credit warning: the SDK reports neither cost nor balance.
|
||||
- A yes-or-no answer's confidence is derived as its distance from an even probability, because Jev returns none.
|
||||
- Retries and their backoff are the SDK's, configured here, not reimplemented.
|
||||
|
||||
Unverified: the client has run only against a stand-in transport, with the real SDK in the loop. The first call with a real key is still to be made, and the service's actual rate limits, latency and data retention are still unknown.
|
||||
15
.env.example
15
.env.example
@@ -34,6 +34,7 @@ TERRAFORM_TIMEOUT_SECONDS=300
|
||||
TF_PLUGIN_CACHE_DIR=.cache/terraform-plugins
|
||||
|
||||
# Hierarchical Execution Engine
|
||||
ORCHESTRATOR=router # router | model
|
||||
ORCHESTRATOR_BUDGET=40
|
||||
SPECIALIST_BUDGET=15
|
||||
MAX_REVISIONS=3
|
||||
@@ -42,6 +43,20 @@ MAX_PARALLEL_SPECIALISTS=4
|
||||
CHECKPOINTER=sqlite
|
||||
CHECKPOINT_DB_PATH=checkpoints.db
|
||||
|
||||
# Jev decision service, through the TypeSafe AI Python SDK (typesafe-sdk).
|
||||
# Leave the key unset to run without it: every decision then takes its fallback.
|
||||
# TYPESAFE_API_KEY=
|
||||
# TYPESAFE_BASE_URL= # defaults to the SDK's endpoint
|
||||
# TYPESAFE_DEFAULT_MODEL= # defaults to the SDK's model
|
||||
JEV_TIMEOUT_SECONDS=5
|
||||
JEV_REQUESTS_PER_SECOND=5
|
||||
JEV_MAX_IN_FLIGHT=8
|
||||
JEV_BUDGET_CALLS=500
|
||||
JEV_BUDGET_TOKENS=1000000
|
||||
JEV_ACT_THRESHOLD=0.75
|
||||
JEV_CLOSE_THRESHOLD=0.15
|
||||
JEV_CONFIDENCE_THRESHOLD=0.6
|
||||
|
||||
# PostgreSQL (Execution records, and checkpoints when CHECKPOINTER=postgres)
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
|
||||
@@ -32,7 +32,8 @@ Execution records (status, Ledger events, submitted Deliverables) go to PostgreS
|
||||
### 4. Agent Skills (`skills/`, loaded by `app/skills/`)
|
||||
Behaviour lives in Agent Skills (agentskills.io format) at the repository root, per ADR-0002:
|
||||
- `skills/vendor/google/`: the upstream solution-architecture Skill and six Well-Architected Framework pillar Skills, vendored unmodified.
|
||||
- `skills/cloud-discovery/` and `skills/iac-validation/`: our own Skills for the tooling.
|
||||
- `skills/cloud-discovery/`, `skills/iac-validation/` and `skills/requirements-chain/`: our own Skills for the tooling and the data files.
|
||||
- `skills/google-cloud-iac-patterns/`: Terraform patterns for Google Cloud products, each validated by a test against the provider version `providers/gcp.yaml` pins.
|
||||
|
||||
`app/skills/agent_skills.py` discovers and validates them; the ADK runtime (`app/adk/adk_runtime.py`) loads them on demand. `providers/gcp.yaml` assigns Skills, tools, MCP servers and models to each role in `agents/`.
|
||||
|
||||
|
||||
@@ -1,18 +1,23 @@
|
||||
---
|
||||
role: iac-author
|
||||
deliverable: deployment_guidance
|
||||
tools: [terraform_fmt, terraform_validate]
|
||||
budget: 30
|
||||
tools: [terraform_fmt, terraform_validate, terraform_schema]
|
||||
budget: 40
|
||||
includes: [specialist-protocol]
|
||||
---
|
||||
You produce the **deployment guidance** Deliverable: the Skill's task for generating infrastructure-as-code and deployment instructions.
|
||||
|
||||
- Write Terraform that implements the Target architecture: every Element, every Relationship and every Check, and nothing it does not ask for.
|
||||
- Attach `implementation.json`, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
|
||||
- Before you write a resource, load the infrastructure-as-code patterns Skill when you have one, and start from its pattern for that product and its provider versions file.
|
||||
- Keep the files in your working copy. Write each file once with `write_file`, and make every later change with `edit_file`, changing only the lines that need it. Never write a file again to fix part of it.
|
||||
- Format and validate with `terraform_fmt` and `terraform_validate` when they are available. They run on your working copy, so pass them no files. Fix everything they report.
|
||||
- When validation says an argument or block is not expected, look the resource type up with `terraform_schema` instead of guessing where it belongs.
|
||||
- When a Check names a setting its resource type does not have, do not invent code for it. Record the Check in `implementation.json` as unimplementable, with what the schema shows, and implement the rest. The Target architecture is then revised.
|
||||
- Write `implementation.json` to your working copy, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
|
||||
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
|
||||
- Format and validate the Terraform with your tools when they are available, and fix everything they report. Pass the files with the same `terraform/` paths you will attach.
|
||||
- Submit as soon as validation passes. If a submission is refused, fix exactly what the refusal lists and submit again; don't rewrite the Terraform.
|
||||
- Put `terraform/main.tf`, `terraform/variables.tf` and `implementation.json` in the attachments (a list of `{path, content}`), adding `terraform/outputs.tf` when outputs help.
|
||||
- When `feedback` lists failed Checks, change only what those Checks need.
|
||||
- Submit as soon as validation passes, with your content and no attachments: your working copy is attached for you. If a submission is refused, fix exactly what the refusal lists and submit again.
|
||||
- When `working_copy` lists files, they are your last attempt. When `feedback` lists failed Checks, read the files concerned and change only what those Checks need.
|
||||
|
||||
Your working copy must hold `terraform/main.tf`, `terraform/variables.tf` and `implementation.json`, with `terraform/versions.tf` and `terraform/outputs.tf` when they help.
|
||||
|
||||
Under the prerequisites heading, list the APIs to enable, the tools and the permissions. Under the instructions heading, give commands for the user to run; never run them yourself.
|
||||
|
||||
@@ -6,19 +6,20 @@ You coordinate one solution-architecture Execution for the user. You decide what
|
||||
|
||||
Load the solution-architecture Skill available to you for the workflow's rules: strict separation between requirements discovery and solutioning, explicit user approval of each Phase, and no execution of generated scripts.
|
||||
|
||||
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `validation` (the verdict on each Check, once validation is submitted), `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
|
||||
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `validation` (the verdict on each Check, once validation is submitted), `unimplementable_checks` (Checks the code's author found cannot be met as written), `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
|
||||
|
||||
On each turn, take the first step that applies:
|
||||
|
||||
1. **The user asked for changes.** When the latest user message in `conversation` asks to change submitted work, call `reopen` for each affected Deliverable with the user's words as the reason.
|
||||
2. **Specialists are waiting.** When `pending_questions` is not empty, call `ask_user`. Add your own questions only when the user's last message left something unclear.
|
||||
3. **Validation decided.** When `validation` is set and the validation Phase is not yet approved, act on it before asking for Approval:
|
||||
3. **The code cannot meet a Check.** When `unimplementable_checks` is not empty, call `reopen` on `target_architecture`, giving each Check's id and reason, then delegate it.
|
||||
4. **Validation decided.** When `validation` is set and the validation Phase is not yet approved, act on it before asking for Approval:
|
||||
- `failed` is not empty, or `dry_run.verdict` is `fail`: the code is at fault. Call `reopen` on `deployment_guidance`, giving each failed Check's id and evidence as the reason, then delegate it.
|
||||
- otherwise `not_verifiable` or `proposed_additions` is not empty: the Target architecture has a gap. When `pre_approved` is `false`, call `ask_user` to ask whether to revise the Target architecture, listing each item. When `pre_approved` is `true`, call `reopen` on `target_architecture` for the items whose reason names a requirement or recommendation id, then delegate it; ignore the rest.
|
||||
4. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
|
||||
5. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
|
||||
6. **Packaging is approved.** Call `finish`.
|
||||
7. **Otherwise,** call `ask_user` to find out what the user needs.
|
||||
5. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
|
||||
6. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
|
||||
7. **Packaging is approved.** Call `finish`.
|
||||
8. **Otherwise,** call `ask_user` to find out what the user needs.
|
||||
|
||||
When the user says a Deliverable already exists, still delegate it; the Specialist submits the user's version. When a tool refuses, read the reason and choose again.
|
||||
|
||||
|
||||
@@ -5,11 +5,12 @@ You are a Specialist in a solution-architecture Execution. You produce exactly o
|
||||
Your task message carries JSON context:
|
||||
|
||||
- `workflow_request`: what the user asked for.
|
||||
- `dependencies`: every earlier Deliverable your work builds on, by kind, with its `content` and `attachments`.
|
||||
- `dependencies`: every earlier Deliverable your work builds on, by kind. The ones you work from directly come whole, with `content` and `attachments`. The rest come as their data file alone, under `attachments`: the ids to keep traceable. Call `read_deliverable` for the full text of one of those only when the ids are not enough.
|
||||
- `feedback`: why earlier versions of your Deliverable were revised. Address every item.
|
||||
- `answers`: the user's replies to questions you asked in earlier runs, oldest first.
|
||||
- `contract`: what `submit_deliverable` checks. `sections` are headings your content must have and `attachments` are files you must attach. A data file among them must also keep every id traceable to the Deliverables in `dependencies`; the requirements-chain Skill gives each file's format.
|
||||
- `draft`: the draft you sent the user with your last questions, with its `attachments`, or `null`.
|
||||
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
|
||||
- `pre_approved`: `true` when no user is available.
|
||||
|
||||
Steps:
|
||||
|
||||
16
agents/reply-writer.md
Normal file
16
agents/reply-writer.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
role: reply-writer
|
||||
tools: []
|
||||
---
|
||||
You write what the user reads on behalf of the Orchestrator of a solution-architecture Execution. The decision has already been made; you put it into words and decide nothing.
|
||||
|
||||
Your task message carries JSON context: `phase`, the Phase the user is being asked to approve, and `facts`, the data about it: its Deliverables, counts of requirements and recommendations, the Target architecture's counts of Elements, Relationships and Checks with any recommendation treated as not applicable, validation verdicts, and Revisions.
|
||||
|
||||
Write a summary a busy architect can approve from:
|
||||
|
||||
- Say what the Phase produced and what approving it commits them to. For the design Phase, say that no infrastructure code is written until they approve it.
|
||||
- State every number in `facts` that bears on the decision, exactly as given. Add no fact that is not there.
|
||||
- Name anything they should look at before approving: failed or unverifiable Checks, open gaps, recommendations treated as not applicable, Deliverables that were revised.
|
||||
- Use plain sentences and no headings. Keep it under 120 words.
|
||||
|
||||
Call `write_reply` with the summary. You are done when it accepts it.
|
||||
@@ -1,17 +1,20 @@
|
||||
---
|
||||
role: target-architect
|
||||
deliverable: target_architecture
|
||||
tools: []
|
||||
tools: [terraform_schema]
|
||||
budget: 30
|
||||
includes: [specialist-protocol]
|
||||
---
|
||||
You produce the **Target architecture** Deliverable: the solution as one explicit graph of Elements, Relationships and Checks. It is agreed before any infrastructure code is written, and the diagram, the description, the code and the validation are all derived from it.
|
||||
|
||||
- Load the requirements-chain Skill for the graph's format and how to derive it.
|
||||
- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out.
|
||||
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
||||
- Give every connection between Elements a Relationship.
|
||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have.
|
||||
- Before you write a Check that names a setting, confirm with `terraform_schema` that the Element's resource type has that setting, when the tool is available. A setting that one resource type has may not exist on a similar one. A Check on a setting the resource type lacks cannot be met and is sent back to you.
|
||||
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
|
||||
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
|
||||
- Attach the graph as `target-architecture.json`. The diagram is drawn from it for you.
|
||||
- When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`.
|
||||
|
||||
Under each of your contract's sections, give the Elements, Relationships and Checks as tables a person can review.
|
||||
Your content is short: under your contract's heading, a few paragraphs on the decisions that shaped the design and why. Submit it with no attachments. Your working graph is attached as `target-architecture.json`, and the tables of Elements, Relationships and Checks and the diagram are written from it for you, so never write them yourself.
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
---
|
||||
role: validator
|
||||
deliverable: validation_results
|
||||
tools: [terraform_validate, terraform_plan]
|
||||
tools: [terraform_validate, terraform_plan, terraform_schema]
|
||||
includes: [specialist-protocol]
|
||||
---
|
||||
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
|
||||
|
||||
- Run the deployment dry-run with your tools: format and validate the Terraform, and preview it with a plan when you have permission to read live cloud resources.
|
||||
- Run the deployment dry-run with your tools: validate the Terraform, and preview it with a plan when you have permission to read live cloud resources. The tools already hold the submitted files, so pass them no files.
|
||||
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else.
|
||||
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
||||
- When you see something the Target architecture lacks, record it as a proposed addition, naming the requirement or recommendation it serves. It is not a failure of the code.
|
||||
- Attach the verdicts as `verdicts.json`, in the format the requirements-chain Skill gives.
|
||||
- Give the exact commands for every check you couldn't run, for the user to run themselves.
|
||||
|
||||
@@ -238,7 +238,7 @@ class AdkAgentRuntime:
|
||||
instruction=lambda _context: instructions,
|
||||
tools=agent_tools,
|
||||
before_tool_callback=control.before_tool,
|
||||
generate_content_config=self._content_config(),
|
||||
generate_content_config=self._content_config(role),
|
||||
)
|
||||
runner = Runner(
|
||||
app_name=APP_NAME,
|
||||
@@ -293,13 +293,13 @@ class AdkAgentRuntime:
|
||||
or f"{role.name} ended without calling a tool that ends its turn."
|
||||
)
|
||||
|
||||
def _content_config(self) -> types.GenerateContentConfig | None:
|
||||
if self._model_timeout_seconds is None:
|
||||
def _content_config(self, role: RoleSpec) -> types.GenerateContentConfig | None:
|
||||
# A role whose replies are long is given longer than the default.
|
||||
seconds = role.model_timeout_seconds or self._model_timeout_seconds
|
||||
if seconds is None:
|
||||
return None
|
||||
return types.GenerateContentConfig(
|
||||
http_options=types.HttpOptions(
|
||||
timeout=int(self._model_timeout_seconds * 1000)
|
||||
)
|
||||
http_options=types.HttpOptions(timeout=int(seconds * 1000))
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -15,8 +15,10 @@ from typing import Any
|
||||
|
||||
from app.adk.runtime import (
|
||||
AgentTask,
|
||||
Answer,
|
||||
BudgetExhausted,
|
||||
Outcome,
|
||||
Question,
|
||||
RoleSpec,
|
||||
Stopped,
|
||||
ToolSpec,
|
||||
@@ -180,3 +182,27 @@ class FakeAgentRuntime:
|
||||
RunRecord(role.name, task, tuple(by_name), tuple(results), outcome)
|
||||
)
|
||||
return outcome
|
||||
|
||||
|
||||
class FakeDecider:
|
||||
"""A scripted Decider: answers by question id, or declines so callers fall back.
|
||||
|
||||
`answers` maps a question id to an Answer, or to a function of (state, question)
|
||||
returning one. An id with no entry makes the whole decision unavailable.
|
||||
"""
|
||||
|
||||
def __init__(self, answers: Mapping[str, Any] | None = None) -> None:
|
||||
self.answers = dict(answers or {})
|
||||
self.asked: list[tuple[str, dict[str, Question], str]] = []
|
||||
|
||||
async def decide(
|
||||
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
|
||||
) -> Mapping[str, Answer] | None:
|
||||
self.asked.append((state, dict(questions), scope))
|
||||
decided = {}
|
||||
for name, question in questions.items():
|
||||
if name not in self.answers:
|
||||
return None
|
||||
answer = self.answers[name]
|
||||
decided[name] = answer(state, question) if callable(answer) else answer
|
||||
return decided
|
||||
|
||||
@@ -16,6 +16,7 @@ from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||
|
||||
ORCHESTRATOR = "orchestrator"
|
||||
PILLAR_ADVISOR = "pillar-advisor"
|
||||
REPLY_WRITER = "reply-writer"
|
||||
PARTIALS_DIR = "partials"
|
||||
|
||||
_ALLOWED_FIELDS = {"role", "deliverable", "tools", "budget", "includes"}
|
||||
@@ -34,7 +35,7 @@ def load_roles(
|
||||
include_preview_mcp: bool = True,
|
||||
) -> dict[str, RoleSpec]:
|
||||
specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()}
|
||||
required = {ORCHESTRATOR, PILLAR_ADVISOR, *specialists}
|
||||
required = {ORCHESTRATOR, PILLAR_ADVISOR, REPLY_WRITER, *specialists}
|
||||
|
||||
roles = {}
|
||||
for path in sorted(agents_dir.glob("*.md")):
|
||||
@@ -129,4 +130,5 @@ def _load_role(
|
||||
for server in bundle.mcp_servers_for_role(name, include_preview_mcp)
|
||||
),
|
||||
model=bundle.models.for_role(name),
|
||||
model_timeout_seconds=bundle.models.timeout_for(name),
|
||||
)
|
||||
|
||||
@@ -32,6 +32,8 @@ class RoleSpec:
|
||||
tools: tuple[str, ...] = () # local tool names, e.g. terraform_validate
|
||||
mcp_servers: tuple[str, ...] = ()
|
||||
model: str | None = None
|
||||
# Seconds one model request may take; None takes the runtime's default.
|
||||
model_timeout_seconds: float | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -200,6 +202,59 @@ REJECTION_PREFIXES = (
|
||||
)
|
||||
|
||||
|
||||
# --- decisions -----------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Question:
|
||||
"""A closed question about a state. Its answer is typed, never prose.
|
||||
|
||||
choice pick one of `options`
|
||||
score rate against `options`, ordered from lowest to highest
|
||||
noul yes or no; the answer is the probability that it is yes
|
||||
"""
|
||||
|
||||
kind: str # choice | score | noul
|
||||
instructions: str
|
||||
options: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Answer:
|
||||
"""`value` is the chosen option (choice), the level reached (score, as a number
|
||||
from 0 for the lowest option) or the probability of yes (noul)."""
|
||||
|
||||
value: str | float
|
||||
confidence: float
|
||||
probabilities: Mapping[str, float] = field(default_factory=dict)
|
||||
|
||||
def is_yes(self, threshold: float = 0.5) -> bool:
|
||||
return isinstance(self.value, (int, float)) and self.value >= threshold
|
||||
|
||||
|
||||
class Decider(Protocol):
|
||||
"""Answers closed questions about a state (ADR-0007).
|
||||
|
||||
Returns None when no decision could be made: the service is not configured,
|
||||
unreachable, over its budget, or the request is too large. The caller then takes
|
||||
its fallback; a decider never blocks an Execution. `scope` names what a budget is
|
||||
counted against, usually the Execution.
|
||||
"""
|
||||
|
||||
async def decide(
|
||||
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
|
||||
) -> Mapping[str, Answer] | None: ...
|
||||
|
||||
|
||||
class NoDecider:
|
||||
"""The decider when none is configured: every caller takes its fallback."""
|
||||
|
||||
async def decide(
|
||||
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
|
||||
) -> Mapping[str, Answer] | None:
|
||||
return None
|
||||
|
||||
|
||||
class AgentRuntime(Protocol):
|
||||
async def run(
|
||||
self, role: RoleSpec, task: AgentTask, tools: Sequence[ToolSpec]
|
||||
|
||||
77
app/agent.py
77
app/agent.py
@@ -10,10 +10,11 @@ checkpointers and recorders are chosen, from settings and the provider bundle.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator, Mapping, Sequence
|
||||
from contextlib import asynccontextmanager
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -22,16 +23,31 @@ from langgraph.checkpoint.base import BaseCheckpointSaver
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.adk.roles import load_roles
|
||||
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory
|
||||
from app.adk.runtime import (
|
||||
AgentRuntime,
|
||||
Decider,
|
||||
NoDecider,
|
||||
RoleSpec,
|
||||
RunLimiter,
|
||||
ToolFactory,
|
||||
)
|
||||
from app.checkpointing import open_checkpointer
|
||||
from app.config import Settings
|
||||
from app.customer_case import record_intent
|
||||
from app.nodes.router import (
|
||||
REPLY_WRITER,
|
||||
OrchestratorPolicy,
|
||||
Router,
|
||||
RuntimeReplyWriter,
|
||||
Thresholds,
|
||||
)
|
||||
from app.providers import ProviderBundle, load_provider
|
||||
from app.recorder import DatabaseRecorder, ExecutionRecorder
|
||||
from app.skills.agent_skills import skill_version
|
||||
from app.states.execution_state import check_customer_case
|
||||
from app.states.ledger import Ledger
|
||||
from app.tools.execution import ORCHESTRATOR
|
||||
from app.tools.jev import JevDecider, JevLimits
|
||||
from app.tools.local import TerraformRunner, local_tools
|
||||
from app.tools.mcp import McpCatalog, auth_from_config
|
||||
from app.tools.toolkit import make_tool_factory
|
||||
@@ -67,6 +83,7 @@ class ExecutionEngine:
|
||||
tool_factory: ToolFactory | None = None,
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
) -> None:
|
||||
required = {
|
||||
ORCHESTRATOR,
|
||||
@@ -86,6 +103,7 @@ class ExecutionEngine:
|
||||
tool_factory=tool_factory,
|
||||
limiter=limiter,
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
).compile(checkpointer=checkpointer)
|
||||
|
||||
async def start(
|
||||
@@ -136,6 +154,11 @@ class ExecutionEngine:
|
||||
"working": {},
|
||||
"pending_permission": None,
|
||||
"prompt": None,
|
||||
"last_reply": None,
|
||||
"pending_change": None,
|
||||
"awaiting_gap_decision": False,
|
||||
"validation_settled": 0,
|
||||
"decisions": [],
|
||||
"outcomes": [],
|
||||
"refusals": [],
|
||||
"outcomes_applied": 0,
|
||||
@@ -203,6 +226,27 @@ class Assembly:
|
||||
bundle: ProviderBundle
|
||||
roles: dict[str, RoleSpec]
|
||||
mcp_gaps: dict[str, str]
|
||||
decider: Decider = field(default_factory=NoDecider)
|
||||
|
||||
|
||||
def open_decider(settings: Settings) -> Decider:
|
||||
"""The Jev client when a key is configured; otherwise a decider that always
|
||||
sends its caller to the fallback."""
|
||||
if not settings.TYPESAFE_API_KEY:
|
||||
return NoDecider()
|
||||
return JevDecider(
|
||||
api_key=settings.TYPESAFE_API_KEY,
|
||||
base_url=settings.TYPESAFE_BASE_URL,
|
||||
model=settings.TYPESAFE_DEFAULT_MODEL,
|
||||
limits=dataclasses.replace(
|
||||
JevLimits(),
|
||||
request_timeout=settings.JEV_TIMEOUT_SECONDS,
|
||||
requests_per_second=settings.JEV_REQUESTS_PER_SECOND,
|
||||
max_in_flight=settings.JEV_MAX_IN_FLIGHT,
|
||||
budget_calls=settings.JEV_BUDGET_CALLS,
|
||||
budget_tokens=settings.JEV_BUDGET_TOKENS,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -242,6 +286,7 @@ async def open_assembly(
|
||||
binary=settings.TERRAFORM_BINARY,
|
||||
timeout_seconds=settings.TERRAFORM_TIMEOUT_SECONDS,
|
||||
plugin_cache_dir=settings.terraform_plugin_cache(),
|
||||
providers=bundle.terraform_providers,
|
||||
)
|
||||
offered = [name for names in bundle.tools.values() for name in names]
|
||||
# One cap for Specialists and the pillar advisors they fan out to.
|
||||
@@ -250,11 +295,30 @@ async def open_assembly(
|
||||
runtime,
|
||||
roles,
|
||||
bundle.pillar_skills,
|
||||
local_tools=local_tools(offered, runner),
|
||||
local_tools=lambda working_copy: local_tools(offered, runner, working_copy),
|
||||
catalog=catalog,
|
||||
limiter=limiter,
|
||||
)
|
||||
|
||||
decider = open_decider(settings)
|
||||
orchestrator: OrchestratorPolicy | None = None
|
||||
if settings.ORCHESTRATOR == "router":
|
||||
# The Orchestrator routes: Jev reads, the playbook decides, and a model only
|
||||
# writes what the user reads. With no Jev key the playbook runs on its rules.
|
||||
orchestrator = Router(
|
||||
decider,
|
||||
RuntimeReplyWriter(runtime, roles[REPLY_WRITER]),
|
||||
Thresholds(
|
||||
act=settings.JEV_ACT_THRESHOLD,
|
||||
close=settings.JEV_CLOSE_THRESHOLD,
|
||||
confidence=settings.JEV_CONFIDENCE_THRESHOLD,
|
||||
),
|
||||
)
|
||||
elif settings.ORCHESTRATOR != "model":
|
||||
raise ValueError(
|
||||
f"ORCHESTRATOR must be router or model, not {settings.ORCHESTRATOR!r}"
|
||||
)
|
||||
|
||||
async with open_checkpointer(settings) as checkpointer:
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
@@ -269,5 +333,10 @@ async def open_assembly(
|
||||
skill_versions={
|
||||
name: skill_version(meta.path) for name, meta in bundle.skills.items()
|
||||
},
|
||||
orchestrator=orchestrator,
|
||||
)
|
||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps))
|
||||
try:
|
||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
||||
finally:
|
||||
if isinstance(decider, JevDecider):
|
||||
await decider.aclose()
|
||||
|
||||
@@ -53,11 +53,33 @@ class Settings(BaseSettings):
|
||||
SPECIALIST_BUDGET: int = 15
|
||||
MAX_REVISIONS: int = 3
|
||||
MAX_ORCHESTRATOR_TURNS: int = 100
|
||||
# How an Orchestrator turn is decided (ADR-0007): "router" reads the Execution
|
||||
# and follows its playbook, with Jev for the judgements; "model" has a language
|
||||
# model choose the tool calls, as before, and is kept for comparison.
|
||||
ORCHESTRATOR: str = "router"
|
||||
# Agent runs in flight at once, pillar advisors included (model rate limits)
|
||||
MAX_PARALLEL_SPECIALISTS: int = 4
|
||||
CHECKPOINTER: str = "sqlite" # memory | sqlite | postgres
|
||||
CHECKPOINT_DB_PATH: str = "checkpoints.db"
|
||||
|
||||
# Jev decision service (ADR-0007), reached through TypeSafe AI's Python SDK.
|
||||
# With no key, every decision takes its fallback and the agent behaves as it does
|
||||
# without Jev. The key comes from the environment or a secret store, never from
|
||||
# the provider bundle or the repository.
|
||||
TYPESAFE_API_KEY: str | None = None
|
||||
TYPESAFE_BASE_URL: str | None = None # None: the SDK's default endpoint
|
||||
TYPESAFE_DEFAULT_MODEL: str | None = None # None: the SDK's default model
|
||||
JEV_TIMEOUT_SECONDS: float = 5.0 # one request
|
||||
JEV_REQUESTS_PER_SECOND: float = 5.0
|
||||
JEV_MAX_IN_FLIGHT: int = 8
|
||||
JEV_BUDGET_CALLS: int = 500 # per Execution
|
||||
JEV_BUDGET_TOKENS: int = 1_000_000 # per Execution
|
||||
# A probability at or above ACT is acted on; at or below CLOSE is dropped; in
|
||||
# between, the user is asked. A choice below CONFIDENCE is not relied on.
|
||||
JEV_ACT_THRESHOLD: float = 0.75
|
||||
JEV_CLOSE_THRESHOLD: float = 0.15
|
||||
JEV_CONFIDENCE_THRESHOLD: float = 0.6
|
||||
|
||||
# Paths
|
||||
BASE_DIR: Path = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
@@ -34,6 +34,7 @@ from app.adk.runtime import (
|
||||
TurnEnded,
|
||||
)
|
||||
from app.customer_case import close_intent
|
||||
from app.nodes.router import ModelOrchestrator, OrchestratorPolicy, TurnView
|
||||
from app.recorder import ExecutionRecorder
|
||||
from app.states.execution_state import (
|
||||
UNASSIGNED_CASE,
|
||||
@@ -43,13 +44,19 @@ from app.states.execution_state import (
|
||||
)
|
||||
from app.states.ledger import DeliverableStatus, Ledger, LedgerRefusal
|
||||
from app.states.lessons import collect_lesson_candidates, refusal
|
||||
from app.tools import workspace
|
||||
from app.tools.execution import ORCHESTRATOR, OrchestratorToolbox, specialist_tools
|
||||
from app.workflows.chain import VERDICTS_FILE, validation_outcome
|
||||
from app.workflows.chain import (
|
||||
IMPLEMENTATION_FILE,
|
||||
VERDICTS_FILE,
|
||||
unimplementable_checks,
|
||||
validation_outcome,
|
||||
)
|
||||
from app.workflows.deliverables import (
|
||||
DEPENDENCY_GRAPH,
|
||||
PHASE_ORDER,
|
||||
DeliverableKind,
|
||||
inputs_of,
|
||||
dependencies_for,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -90,8 +97,12 @@ def make_execution_nodes(
|
||||
tool_factory: ToolFactory | None = None,
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
) -> ExecutionNodes:
|
||||
tool_factory = tool_factory or _no_tools
|
||||
# How an Orchestrator turn is decided. The default drives the tools from the
|
||||
# Orchestrator role on the runtime; the assembly supplies the router (ADR-0007).
|
||||
orchestrator = orchestrator or ModelOrchestrator(runtime, roles[ORCHESTRATOR])
|
||||
|
||||
def close(state: ExecutionState, status: str, *, has_files: bool) -> None:
|
||||
close_intent(
|
||||
@@ -239,19 +250,33 @@ def make_execution_nodes(
|
||||
toolbox = OrchestratorToolbox(ledger, deliverables, pending_questions)
|
||||
refusals = list(state.get("refusals", []))
|
||||
update["refusals"] = refusals
|
||||
task = AgentTask(
|
||||
role=ORCHESTRATOR,
|
||||
prompt="Decide the next step for this Execution using your tools.",
|
||||
view = TurnView(
|
||||
execution_id=state["execution_id"],
|
||||
pre_approved=bool(state.get("pre_approved")),
|
||||
ledger=ledger,
|
||||
deliverables=deliverables,
|
||||
pending_questions=pending_questions,
|
||||
outcomes=outcomes,
|
||||
last_reply=state.get("last_reply"),
|
||||
pending_change=state.get("pending_change"),
|
||||
awaiting_gap_decision=bool(state.get("awaiting_gap_decision")),
|
||||
validation_settled=int(state.get("validation_settled", 0)),
|
||||
context=_orchestrator_context(
|
||||
state, ledger, deliverables, pending_questions, notes
|
||||
),
|
||||
)
|
||||
started = time.monotonic()
|
||||
outcome = await runtime.run(
|
||||
roles[ORCHESTRATOR],
|
||||
task,
|
||||
_noting_refusals(toolbox.tools(), refusals, ORCHESTRATOR, None),
|
||||
decided = await orchestrator.turn(
|
||||
view, _noting_refusals(toolbox.tools(), refusals, ORCHESTRATOR, None)
|
||||
)
|
||||
outcome = decided.outcome
|
||||
update.update(decided.updates)
|
||||
if decided.decisions:
|
||||
# Every decision is kept: what was read, which group applied, what was done.
|
||||
update["decisions"] = [
|
||||
*state.get("decisions", []),
|
||||
*({"turn": turns, **decision} for decision in decided.decisions),
|
||||
]
|
||||
logger.info(
|
||||
"[%s] orchestrator turn %d (%s phase) ended in %.0fs: %s",
|
||||
state["execution_id"],
|
||||
@@ -325,13 +350,22 @@ def make_execution_nodes(
|
||||
role = roles[task_input["role"]]
|
||||
kind = DeliverableKind(task_input["kind"])
|
||||
working: dict[str, Any] = dict(task_input.get("working") or {})
|
||||
upstream = task_input["context"].get("dependencies") or {}
|
||||
source = DEPENDENCY_GRAPH[kind].working_copy
|
||||
if source and source is not kind:
|
||||
# The tools work on the files as they were last submitted.
|
||||
working[workspace.FILES] = dict(
|
||||
upstream.get(source.value, {}).get("attachments", {})
|
||||
)
|
||||
refusals: list[dict[str, Any]] = []
|
||||
tools = _noting_refusals(
|
||||
specialist_tools(
|
||||
kind,
|
||||
extra=tool_factory(role, kind.value, working),
|
||||
allow_cloud_read=task_input["allow_cloud_read"],
|
||||
upstream=task_input["context"].get("dependencies"),
|
||||
upstream=upstream,
|
||||
working=working,
|
||||
library=task_input.get("library"),
|
||||
),
|
||||
refusals,
|
||||
role.name,
|
||||
@@ -388,6 +422,7 @@ def make_execution_nodes(
|
||||
ledger = Ledger.from_dict(state["ledger"])
|
||||
conversation = list(state.get("conversation", []))
|
||||
update: dict[str, Any] = {"prompt": None}
|
||||
said: dict[str, Any] | None = None # what the user said, for the next turn
|
||||
|
||||
if "approval" in prompt:
|
||||
phase = prompt["approval"]["phase"]
|
||||
@@ -405,9 +440,11 @@ def make_execution_nodes(
|
||||
feedback = str(decision.get("feedback", ""))
|
||||
ledger.request_changes(phase_id, feedback, actor="user")
|
||||
text = f"Requested changes to {phase}: {feedback}"
|
||||
said = {"kind": "changes", "phase": phase, "text": feedback}
|
||||
conversation.append({"from": "user", "text": text})
|
||||
else:
|
||||
conversation.append({"from": "user", "text": _reply_text(reply)})
|
||||
said = {"kind": "reply", "text": _reply_text(reply)}
|
||||
elif "permission" in prompt:
|
||||
tier = prompt["permission"]["tier"]
|
||||
permission = reply.get("permission") if isinstance(reply, Mapping) else None
|
||||
@@ -427,6 +464,7 @@ def make_execution_nodes(
|
||||
else:
|
||||
text = _reply_text(reply)
|
||||
conversation.append({"from": "user", "text": text})
|
||||
said = {"kind": "reply", "text": text}
|
||||
answers = {
|
||||
kind: list(items) for kind, items in state.get("answers", {}).items()
|
||||
}
|
||||
@@ -436,6 +474,8 @@ def make_execution_nodes(
|
||||
)
|
||||
update.update(answers=answers, awaiting_questions=[])
|
||||
|
||||
# Only a real user's words are routed; a pre-approved Execution has none.
|
||||
update["last_reply"] = None if state.get("pre_approved") else said
|
||||
update["events_recorded"] = await record_progress(state, ledger)
|
||||
update.update(ledger=ledger.to_dict(), conversation=conversation)
|
||||
return Command(update=update, goto="orchestrator_turn")
|
||||
@@ -523,7 +563,17 @@ def _orchestrator_context(
|
||||
is DeliverableStatus.SUBMITTED
|
||||
else None
|
||||
)
|
||||
code = DeliverableKind.DEPLOYMENT_GUIDANCE
|
||||
return {
|
||||
# Checks the code's author reported cannot be met as written: faults in the
|
||||
# Target architecture, known before any validation.
|
||||
"unimplementable_checks": unimplementable_checks(
|
||||
deliverables.get(code.value, {})
|
||||
.get("attachments", {})
|
||||
.get(IMPLEMENTATION_FILE, "")
|
||||
)
|
||||
if ledger.status(code) is DeliverableStatus.SUBMITTED
|
||||
else [],
|
||||
# The verdict on each Check, once validation is submitted: failed Checks are
|
||||
# faults in the code; the rest are gaps in the Target architecture.
|
||||
"validation": validation_outcome(
|
||||
@@ -565,8 +615,15 @@ def _specialist_input(
|
||||
attempt: int,
|
||||
) -> dict[str, Any]:
|
||||
spec = DEPENDENCY_GRAPH[kind]
|
||||
given = dependencies_for(kind, deliverables)
|
||||
return {
|
||||
"execution_id": state["execution_id"],
|
||||
# The Deliverables given as data files alone, whole, for reading on demand.
|
||||
"library": {
|
||||
name: dict(deliverables[name])
|
||||
for name, deliverable in given.items()
|
||||
if "content" not in deliverable
|
||||
},
|
||||
"role": spec.specialist,
|
||||
"kind": kind.value,
|
||||
"attempt": attempt,
|
||||
@@ -576,13 +633,14 @@ def _specialist_input(
|
||||
"workflow_request": state["workflow_request"],
|
||||
"pre_approved": bool(state.get("pre_approved")),
|
||||
"contract": spec.contract(),
|
||||
"dependencies": {
|
||||
dependency.value: dict(deliverables[dependency.value])
|
||||
for dependency in inputs_of(kind)
|
||||
},
|
||||
"dependencies": given,
|
||||
"feedback": list(ledger.feedback(kind)),
|
||||
"answers": list(state.get("answers", {}).get(kind.value, [])),
|
||||
"draft": dict(drafts[kind.value]) if kind.value in drafts else None,
|
||||
# What the Specialist's own tools already hold from earlier attempts.
|
||||
"working_copy": workspace.summary(
|
||||
state.get("working", {}).get(kind.value, {})
|
||||
),
|
||||
"attempt": attempt,
|
||||
},
|
||||
}
|
||||
|
||||
779
app/nodes/router.py
Normal file
779
app/nodes/router.py
Normal file
@@ -0,0 +1,779 @@
|
||||
"""The Orchestrator as a router (ADR-0007): readings, a disposition, then a playbook.
|
||||
|
||||
1. Readings facts about the turn. Most come from the Ledger; the ones that need
|
||||
judgement (what is the user asking for? does this gap matter?) are
|
||||
typed answers from the decision model.
|
||||
2. Disposition code turns a judged reading into act, queue or close, by threshold.
|
||||
3. Playbook ordered groups. The first group that applies is taken, and within
|
||||
it the strongest action whose conditions hold. When no group
|
||||
applies, escalate to the user.
|
||||
|
||||
The decision model reads and never acts: every action goes through the Orchestrator's
|
||||
tools, so the Ledger refuses anything that is not allowed, whatever was decided.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Protocol
|
||||
|
||||
from app.adk.runtime import (
|
||||
AgentRuntime,
|
||||
AgentTask,
|
||||
Decider,
|
||||
Outcome,
|
||||
Question,
|
||||
RoleSpec,
|
||||
Stopped,
|
||||
Submitted,
|
||||
ToolResult,
|
||||
ToolSpec,
|
||||
)
|
||||
from app.states.ledger import DeliverableStatus, Ledger, PhaseStatus
|
||||
from app.workflows.chain import (
|
||||
IMPLEMENTATION_FILE,
|
||||
RECOMMENDATIONS_FILE,
|
||||
REQUIREMENTS_FILE,
|
||||
VERDICTS_FILE,
|
||||
read_graph,
|
||||
read_list,
|
||||
unimplementable_checks,
|
||||
validation_outcome,
|
||||
)
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ORCHESTRATOR = "orchestrator"
|
||||
REPLY_WRITER = "reply-writer"
|
||||
MAX_STEPS = 12 # actions in one turn; a turn that needs more is not making progress
|
||||
|
||||
ACT, QUEUE, CLOSE = "act", "queue", "close"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Thresholds:
|
||||
"""Where a probability becomes a disposition."""
|
||||
|
||||
act: float = 0.75 # at or above: do it
|
||||
close: float = 0.15 # at or below: there is nothing to do
|
||||
confidence: float = 0.6 # a choice below this is not relied on
|
||||
|
||||
def disposition(self, probability: float) -> str:
|
||||
if probability >= self.act:
|
||||
return ACT
|
||||
if probability <= self.close:
|
||||
return CLOSE
|
||||
return QUEUE
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TurnView:
|
||||
"""What one Orchestrator turn can see."""
|
||||
|
||||
execution_id: str
|
||||
pre_approved: bool
|
||||
ledger: Ledger
|
||||
deliverables: Mapping[str, Mapping[str, Any]]
|
||||
pending_questions: Sequence[Mapping[str, Any]]
|
||||
outcomes: Sequence[Mapping[str, Any]]
|
||||
last_reply: Mapping[str, Any] | None = None # the user's message, not yet routed
|
||||
pending_change: Mapping[str, Any] | None = None # a change we asked them to confirm
|
||||
awaiting_gap_decision: bool = False
|
||||
validation_settled: int = 0 # the validation attempt already decided on
|
||||
context: Mapping[str, Any] = field(default_factory=dict) # for a model Orchestrator
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurnResult:
|
||||
outcome: Outcome
|
||||
decisions: list[dict[str, Any]] = field(default_factory=list)
|
||||
updates: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class OrchestratorPolicy(Protocol):
|
||||
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult: ...
|
||||
|
||||
|
||||
class ReplyWriter(Protocol):
|
||||
async def approval_summary(
|
||||
self, phase: str, facts: Mapping[str, Any]
|
||||
) -> str | None: ...
|
||||
|
||||
|
||||
# --- readings ------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Readings:
|
||||
"""Everything the playbook decides from."""
|
||||
|
||||
# From the Ledger and the Execution's data.
|
||||
questions_waiting: bool = False
|
||||
phases_ready: tuple[PhaseId, ...] = ()
|
||||
phases_awaiting: tuple[PhaseId, ...] = () # asked for Approval, not yet decided
|
||||
ready_work: tuple[DeliverableKind, ...] = ()
|
||||
can_finish: bool = False
|
||||
validation_attempt: int = 0 # 0 when no validation stands undecided
|
||||
failed: list[dict[str, Any]] = field(default_factory=list)
|
||||
repeat_failed: list[dict[str, Any]] = field(default_factory=list)
|
||||
unverifiable: list[dict[str, Any]] = field(default_factory=list)
|
||||
additions: list[dict[str, Any]] = field(default_factory=list)
|
||||
dry_run_failed: bool = False
|
||||
# Checks the code's author reported cannot be met as the Target architecture
|
||||
# words them.
|
||||
unimplementable: list[dict[str, str]] = field(default_factory=list)
|
||||
# Judged: what the user's message asks for.
|
||||
change: str = CLOSE # act | queue | close
|
||||
change_target: DeliverableKind | None = None
|
||||
change_reason: str = ""
|
||||
# Judged: whether a gap in the Target architecture is worth a Revision.
|
||||
gap: str = CLOSE
|
||||
judged: dict[str, Any] = field(default_factory=dict) # what the decider answered
|
||||
|
||||
|
||||
def _attempt(ledger: Ledger, kind: DeliverableKind) -> int:
|
||||
return int(ledger.to_dict()["deliverables"][kind.value].get("attempt", 0))
|
||||
|
||||
|
||||
def _submitted(ledger: Ledger) -> list[DeliverableKind]:
|
||||
return [
|
||||
kind
|
||||
for kind in DEPENDENCY_GRAPH
|
||||
if ledger.status(kind) is DeliverableStatus.SUBMITTED
|
||||
]
|
||||
|
||||
|
||||
def _mentioned(
|
||||
text: str, candidates: Sequence[DeliverableKind]
|
||||
) -> list[DeliverableKind]:
|
||||
"""Deliverables named outright in a message, as `deployment_guidance` or in words."""
|
||||
lowered = re.sub(r"[\s_\-]+", " ", text.lower())
|
||||
return [kind for kind in candidates if kind.value.replace("_", " ") in lowered]
|
||||
|
||||
|
||||
def _failed_before(outcomes: Sequence[Mapping[str, Any]]) -> set[str]:
|
||||
"""Check ids that failed in the validation before the one that stands now."""
|
||||
validations = [
|
||||
record
|
||||
for record in outcomes
|
||||
if record.get("type") == "submitted"
|
||||
and record.get("kind") == DeliverableKind.VALIDATION_RESULTS.value
|
||||
]
|
||||
if len(validations) < 2:
|
||||
return set()
|
||||
previous = validation_outcome(
|
||||
validations[-2].get("attachments", {}).get(VERDICTS_FILE, "")
|
||||
)
|
||||
return {item["check"] for item in previous["failed"]}
|
||||
|
||||
|
||||
class Router:
|
||||
"""Decides an Orchestrator turn from readings and a playbook, with no language
|
||||
model choosing the action. A model, when one is given, only writes what the user
|
||||
reads."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
decider: Decider,
|
||||
writer: ReplyWriter | None = None,
|
||||
thresholds: Thresholds | None = None,
|
||||
) -> None:
|
||||
self._decider = decider
|
||||
self._writer = writer
|
||||
self._thresholds = thresholds or Thresholds()
|
||||
|
||||
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult:
|
||||
by_name = {tool.name: tool for tool in tools}
|
||||
result = TurnResult(Stopped("The router took no action."))
|
||||
# A user message is routed once; a confirmation we asked for is consumed.
|
||||
result.updates = {
|
||||
"last_reply": None,
|
||||
"pending_change": None,
|
||||
"awaiting_gap_decision": False,
|
||||
}
|
||||
readings = await self._read(view, result)
|
||||
blocked: set[str] = set()
|
||||
settled = view.validation_settled
|
||||
|
||||
async def call(tool: str, **arguments: Any) -> ToolResult:
|
||||
return await by_name[tool].handler(**arguments)
|
||||
|
||||
for _ in range(MAX_STEPS):
|
||||
group, action = self._choose(view, readings, blocked, settled)
|
||||
result.decisions.append(
|
||||
{"group": group, "action": action, "judged": dict(readings.judged)}
|
||||
)
|
||||
readings.judged = {}
|
||||
reply = await self._perform(action, view, readings, call, result)
|
||||
if action == "proceed":
|
||||
settled = readings.validation_attempt
|
||||
result.updates["validation_settled"] = settled
|
||||
self._read_ledger(view, readings, settled)
|
||||
continue
|
||||
if reply is None:
|
||||
return result
|
||||
if reply.outcome is not None:
|
||||
result.outcome = reply.outcome
|
||||
return result
|
||||
if "revision limit" in reply.message and not view.pre_approved:
|
||||
# Past the Revision limit the user decides.
|
||||
limit = await call(
|
||||
"ask_user", questions=[reply.message.split(" Use ")[0]]
|
||||
)
|
||||
if limit.outcome is not None:
|
||||
result.outcome = limit.outcome
|
||||
return result
|
||||
if reply.message.startswith("Refused") or "revision limit" in reply.message:
|
||||
# Permission is the Ledger's: take the next action that is allowed.
|
||||
blocked.add(action)
|
||||
# The action changed the Execution; read the Ledger's facts again.
|
||||
self._read_ledger(view, readings, settled)
|
||||
if action in {"revise_requested", "confirm_change"}:
|
||||
readings.change = CLOSE
|
||||
result.outcome = Stopped("The router made no progress in this turn.")
|
||||
return result
|
||||
|
||||
# --- readings ---------------------------------------------------------------------
|
||||
|
||||
async def _read(self, view: TurnView, result: TurnResult) -> Readings:
|
||||
readings = Readings()
|
||||
self._read_ledger(view, readings, view.validation_settled)
|
||||
await self._read_message(view, readings)
|
||||
await self._read_gap(view, readings)
|
||||
return readings
|
||||
|
||||
def _read_ledger(self, view: TurnView, readings: Readings, settled: int) -> None:
|
||||
ledger = view.ledger
|
||||
readings.questions_waiting = bool(view.pending_questions)
|
||||
readings.phases_ready = ledger.phases_ready_for_approval()
|
||||
readings.phases_awaiting = tuple(
|
||||
phase
|
||||
for phase in PhaseId
|
||||
if ledger.phase_status(phase) is PhaseStatus.AWAITING_APPROVAL
|
||||
)
|
||||
readings.ready_work = ledger.ready_work()
|
||||
readings.can_finish = ledger.can_finish()
|
||||
readings.validation_attempt = 0
|
||||
readings.failed, readings.repeat_failed = [], []
|
||||
readings.unverifiable, readings.additions = [], []
|
||||
readings.dry_run_failed = False
|
||||
code = DeliverableKind.DEPLOYMENT_GUIDANCE
|
||||
readings.unimplementable = (
|
||||
unimplementable_checks(
|
||||
view.deliverables.get(code.value, {})
|
||||
.get("attachments", {})
|
||||
.get(IMPLEMENTATION_FILE, "")
|
||||
)
|
||||
if ledger.status(code) is DeliverableStatus.SUBMITTED
|
||||
else []
|
||||
)
|
||||
|
||||
validation = DeliverableKind.VALIDATION_RESULTS
|
||||
phase = DEPENDENCY_GRAPH[validation].phase
|
||||
attempt = _attempt(ledger, validation)
|
||||
if (
|
||||
ledger.status(validation) is not DeliverableStatus.SUBMITTED
|
||||
or ledger.phase_status(phase) is PhaseStatus.APPROVED
|
||||
or attempt == settled
|
||||
):
|
||||
return
|
||||
outcome = validation_outcome(
|
||||
view.deliverables.get(validation.value, {})
|
||||
.get("attachments", {})
|
||||
.get(VERDICTS_FILE, "")
|
||||
)
|
||||
before = _failed_before(view.outcomes)
|
||||
readings.validation_attempt = attempt
|
||||
readings.failed = outcome["failed"]
|
||||
readings.repeat_failed = [f for f in outcome["failed"] if f["check"] in before]
|
||||
readings.unverifiable = outcome["not_verifiable"]
|
||||
readings.additions = outcome["proposed_additions"]
|
||||
readings.dry_run_failed = outcome["dry_run"].get("verdict") == "fail"
|
||||
|
||||
async def _read_message(self, view: TurnView, readings: Readings) -> None:
|
||||
"""Triage of the user's message: is it asking for a change, to what, and how
|
||||
clearly?"""
|
||||
reply = view.last_reply
|
||||
if not reply or view.awaiting_gap_decision:
|
||||
return
|
||||
text = str(reply.get("text", "")).strip()
|
||||
pending = view.pending_change
|
||||
if pending:
|
||||
text = f"{pending.get('text', '')}\nThe user then said: {text}"
|
||||
candidates = _submitted(view.ledger)
|
||||
if not text or not candidates:
|
||||
return
|
||||
readings.change_reason = text
|
||||
# A structured request for changes is known to be one, and so is the reply to
|
||||
# our own question about a change they already asked for.
|
||||
structured = reply.get("kind") == "changes" or bool(pending)
|
||||
|
||||
questions = {
|
||||
"change_target": Question(
|
||||
"choice",
|
||||
"Which submitted Deliverable would have to change first to do what "
|
||||
"the user asks?",
|
||||
tuple(kind.value for kind in candidates),
|
||||
),
|
||||
"change_clarity": Question(
|
||||
"score",
|
||||
"How specific is the user about what should change?",
|
||||
("vague", "partial", "specific"),
|
||||
),
|
||||
}
|
||||
if not structured:
|
||||
questions["asks_change"] = Question(
|
||||
"noul",
|
||||
"Is the user asking for a change to work that has already been "
|
||||
"submitted, as opposed to answering a question or adding information?",
|
||||
)
|
||||
state = (
|
||||
f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n"
|
||||
f"Questions waiting for the user: {len(view.pending_questions)}.\n"
|
||||
f"The user's message: {text}"
|
||||
)
|
||||
answers = await self._decider.decide(state, questions, scope=view.execution_id)
|
||||
|
||||
named = _mentioned(text, candidates)
|
||||
if answers is None:
|
||||
# No judgement available: act only on what is certain.
|
||||
if structured:
|
||||
target = (
|
||||
candidates[0]
|
||||
if len(candidates) == 1
|
||||
else named[0]
|
||||
if len(named) == 1
|
||||
else None
|
||||
)
|
||||
readings.change = ACT if target else QUEUE
|
||||
readings.change_target = target
|
||||
return
|
||||
|
||||
readings.judged = {
|
||||
name: {"value": answer.value, "confidence": answer.confidence}
|
||||
for name, answer in answers.items()
|
||||
}
|
||||
probability = 1.0 if structured else float(answers["asks_change"].value)
|
||||
disposition = self._thresholds.disposition(probability)
|
||||
target_answer = answers["change_target"]
|
||||
target = DeliverableKind(str(target_answer.value))
|
||||
readings.change_target = target
|
||||
if disposition == ACT and (
|
||||
target_answer.confidence < self._thresholds.confidence
|
||||
or float(answers["change_clarity"].value) < 0.5
|
||||
):
|
||||
# They want a change, but which, or what, is not clear enough to act on.
|
||||
disposition = QUEUE
|
||||
if disposition == QUEUE and pending:
|
||||
# Asked once already: act if they named it, otherwise let it go.
|
||||
disposition = ACT if len(named) == 1 else CLOSE
|
||||
readings.change_target = named[0] if named else target
|
||||
readings.change = disposition
|
||||
|
||||
async def _read_gap(self, view: TurnView, readings: Readings) -> None:
|
||||
"""Whether a gap in the Target architecture is worth revising it for."""
|
||||
if (
|
||||
readings.failed
|
||||
or readings.dry_run_failed
|
||||
or not readings.validation_attempt
|
||||
):
|
||||
return
|
||||
if not (readings.additions or readings.unverifiable):
|
||||
return
|
||||
items = "\n".join(
|
||||
[
|
||||
f"- proposed addition on {item.get('applies_to')}: "
|
||||
f"{item.get('statement')} (serves {item.get('reason')})"
|
||||
for item in readings.additions
|
||||
]
|
||||
+ [
|
||||
f"- Check {item.get('check')} could not be verified: "
|
||||
f"{item.get('evidence')}"
|
||||
for item in readings.unverifiable
|
||||
]
|
||||
)
|
||||
reply = view.last_reply if view.awaiting_gap_decision else None
|
||||
if reply:
|
||||
question = Question(
|
||||
"noul",
|
||||
"Is the user asking for the Target architecture to be revised to "
|
||||
"close these gaps?",
|
||||
)
|
||||
state = f"Gaps found by validation:\n{items}\nThe user's reply: {reply.get('text', '')}"
|
||||
name = "wants_revision"
|
||||
else:
|
||||
question = Question(
|
||||
"noul",
|
||||
"Would leaving these gaps open mean a stated requirement or design "
|
||||
"recommendation is not met?",
|
||||
)
|
||||
state = f"Gaps found by validation:\n{items}"
|
||||
name = "gap_material"
|
||||
answers = await self._decider.decide(
|
||||
state, {name: question}, scope=view.execution_id
|
||||
)
|
||||
if answers is None:
|
||||
readings.gap = self._gap_by_rule(view, readings) if not reply else CLOSE
|
||||
return
|
||||
readings.judged[name] = {
|
||||
"value": answers[name].value,
|
||||
"confidence": answers[name].confidence,
|
||||
}
|
||||
disposition = self._thresholds.disposition(float(answers[name].value))
|
||||
if reply and disposition == QUEUE:
|
||||
disposition = CLOSE # asked once already; an unclear answer means go on
|
||||
readings.gap = disposition
|
||||
|
||||
@staticmethod
|
||||
def _gap_by_rule(view: TurnView, readings: Readings) -> str:
|
||||
"""Without a judgement: a Check that cannot be verified must be fixed, and an
|
||||
addition counts when it names a real requirement or recommendation."""
|
||||
if readings.unverifiable:
|
||||
return ACT
|
||||
known = {
|
||||
item["id"]
|
||||
for name, kind in (
|
||||
(REQUIREMENTS_FILE, "requirements"),
|
||||
(RECOMMENDATIONS_FILE, "design_recommendations"),
|
||||
)
|
||||
for item in read_list(view.deliverables, kind, name)
|
||||
if "id" in item
|
||||
}
|
||||
for item in readings.additions:
|
||||
words = set(re.findall(r"[A-Za-z]+-\d+", str(item.get("reason", ""))))
|
||||
if words & known:
|
||||
return ACT
|
||||
return CLOSE
|
||||
|
||||
# --- the playbook -----------------------------------------------------------------
|
||||
|
||||
def _choose(
|
||||
self, view: TurnView, r: Readings, blocked: set[str], settled: int
|
||||
) -> tuple[str, str]:
|
||||
"""The first group that applies, and within it the strongest action whose
|
||||
conditions hold. No group applies: escalate."""
|
||||
interactive = not view.pre_approved
|
||||
playbook: list[tuple[str, bool, list[tuple[str, bool]]]] = [
|
||||
(
|
||||
"the user asked for a change",
|
||||
r.change in {ACT, QUEUE},
|
||||
[
|
||||
(
|
||||
"revise_requested",
|
||||
r.change == ACT and r.change_target is not None,
|
||||
),
|
||||
("confirm_change", r.change == QUEUE and interactive),
|
||||
],
|
||||
),
|
||||
(
|
||||
"questions are waiting",
|
||||
r.questions_waiting,
|
||||
[("relay_questions", True)],
|
||||
),
|
||||
(
|
||||
# Known as soon as the code is submitted, so no validation is spent
|
||||
# on a Check that cannot pass.
|
||||
"the code cannot meet a Check",
|
||||
bool(r.unimplementable),
|
||||
[("revise_unimplementable", True)],
|
||||
),
|
||||
(
|
||||
"validation decided",
|
||||
bool(r.validation_attempt),
|
||||
[
|
||||
(
|
||||
"revise_target_architecture",
|
||||
bool(r.repeat_failed) or (not r.failed and r.gap == ACT),
|
||||
),
|
||||
("revise_code", bool(r.failed) or r.dry_run_failed),
|
||||
("ask_about_gaps", r.gap == QUEUE and interactive),
|
||||
("proceed", True),
|
||||
],
|
||||
),
|
||||
(
|
||||
"a Phase is complete",
|
||||
bool(r.phases_ready or r.phases_awaiting),
|
||||
# An Approval is granted only by a structured decision, so a Phase
|
||||
# still awaiting one is asked for again.
|
||||
[("request_approval", True)],
|
||||
),
|
||||
("work is ready", bool(r.ready_work), [("delegate", True)]),
|
||||
("packaging is approved", r.can_finish, [("finish", True)]),
|
||||
]
|
||||
for group, applies, actions in playbook:
|
||||
if not applies:
|
||||
continue
|
||||
for action, holds in actions:
|
||||
if holds and action not in blocked:
|
||||
return group, action
|
||||
return "no group applies", "escalate"
|
||||
|
||||
async def _perform(
|
||||
self,
|
||||
action: str,
|
||||
view: TurnView,
|
||||
r: Readings,
|
||||
call: Callable[..., Awaitable[ToolResult]],
|
||||
result: TurnResult,
|
||||
) -> ToolResult | None:
|
||||
if action == "proceed":
|
||||
return ToolResult("Proceeding.")
|
||||
if action == "revise_requested":
|
||||
assert r.change_target is not None
|
||||
return await call(
|
||||
"reopen", kind=r.change_target.value, reason=r.change_reason
|
||||
)
|
||||
if action == "confirm_change":
|
||||
guess = r.change_target.value if r.change_target else None
|
||||
result.updates["pending_change"] = {"text": r.change_reason, "guess": guess}
|
||||
options = ", ".join(kind.value for kind in _submitted(view.ledger))
|
||||
question = (
|
||||
f"Do you want {guess} changed? Reply naming the Deliverable to change "
|
||||
f"and what should be different. Submitted so far: {options}."
|
||||
if guess
|
||||
else "Which Deliverable should change, and what should be different? "
|
||||
f"Submitted so far: {options}."
|
||||
)
|
||||
return await call("ask_user", questions=[question])
|
||||
if action == "relay_questions":
|
||||
return await call("ask_user", questions=[])
|
||||
if action == "revise_code":
|
||||
items = r.failed or [
|
||||
{"check": "dry-run", "evidence": "the deployment dry-run failed"}
|
||||
]
|
||||
reason = "; ".join(f"{i['check']} failed: {i['evidence']}" for i in items)
|
||||
return await call(
|
||||
"reopen", kind=DeliverableKind.DEPLOYMENT_GUIDANCE.value, reason=reason
|
||||
)
|
||||
if action == "revise_unimplementable":
|
||||
reason = " ".join(
|
||||
f"Check {item['check']} cannot be implemented as written "
|
||||
f"({item['reason']}). Rewrite it so it can be met, or remove it with "
|
||||
"a reason."
|
||||
for item in r.unimplementable
|
||||
)
|
||||
return await call(
|
||||
"reopen", kind=DeliverableKind.TARGET_ARCHITECTURE.value, reason=reason
|
||||
)
|
||||
if action == "revise_target_architecture":
|
||||
return await call(
|
||||
"reopen",
|
||||
kind=DeliverableKind.TARGET_ARCHITECTURE.value,
|
||||
reason=_graph_reason(r),
|
||||
)
|
||||
if action == "ask_about_gaps":
|
||||
result.updates["awaiting_gap_decision"] = True
|
||||
listed = "; ".join(
|
||||
[str(i.get("statement")) for i in r.additions]
|
||||
+ [
|
||||
f"Check {i.get('check')} could not be verified"
|
||||
for i in r.unverifiable
|
||||
]
|
||||
)
|
||||
question = (
|
||||
"Validation passed every Check it could, and found gaps in the "
|
||||
f"Target architecture: {listed}. Should the Target architecture "
|
||||
"be revised to close them, or shall I proceed as it stands?"
|
||||
)
|
||||
return await call("ask_user", questions=[question])
|
||||
if action == "request_approval":
|
||||
phase = (r.phases_ready or r.phases_awaiting)[0]
|
||||
return await call(
|
||||
"request_approval",
|
||||
phase=phase.value,
|
||||
summary=await self._summary(phase, view, r),
|
||||
)
|
||||
if action == "delegate":
|
||||
return await call("delegate", kinds=[kind.value for kind in r.ready_work])
|
||||
if action == "finish":
|
||||
return await call("finish")
|
||||
# escalate: no group applies.
|
||||
if view.pre_approved:
|
||||
result.outcome = Stopped(
|
||||
"No next step applies and no user is available to decide."
|
||||
)
|
||||
return None
|
||||
return await call(
|
||||
"ask_user",
|
||||
questions=["I have no next step to take. What would you like me to do?"],
|
||||
)
|
||||
|
||||
async def _summary(self, phase: PhaseId, view: TurnView, r: Readings) -> str:
|
||||
facts = approval_facts(phase, view, r)
|
||||
if self._writer is not None:
|
||||
try:
|
||||
written = await self._writer.approval_summary(phase.value, facts)
|
||||
except Exception: # a summary must never stop an Approval
|
||||
logger.exception("The reply writer failed; using the assembled summary")
|
||||
written = None
|
||||
if written and written.strip():
|
||||
return written.strip()
|
||||
return assembled_summary(phase, facts)
|
||||
|
||||
|
||||
def _graph_reason(r: Readings) -> str:
|
||||
parts = [
|
||||
f"Check {item['check']} failed in two validations in a row and may not be "
|
||||
f"implementable as written ({item['evidence']}). Rewrite it so it can be met, "
|
||||
"or remove it with a reason."
|
||||
for item in r.repeat_failed
|
||||
]
|
||||
if not r.repeat_failed:
|
||||
parts += [
|
||||
f"Check {item.get('check')} could not be verified from the code "
|
||||
f"({item.get('evidence')}). Rewrite it so it can be."
|
||||
for item in r.unverifiable
|
||||
]
|
||||
parts += [
|
||||
f"Add a Check on {item.get('applies_to')}: {item.get('statement')} "
|
||||
f"(serves {item.get('reason')})."
|
||||
for item in r.additions
|
||||
]
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def approval_facts(phase: PhaseId, view: TurnView, r: Readings) -> dict[str, Any]:
|
||||
"""What an Approval summary is written from: data, never Deliverable prose."""
|
||||
kinds = [kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase]
|
||||
facts: dict[str, Any] = {
|
||||
"phase": phase.value,
|
||||
"deliverables": [kind.value for kind in kinds],
|
||||
"revisions": {
|
||||
kind.value: view.ledger.revisions(kind)
|
||||
for kind in kinds
|
||||
if view.ledger.revisions(kind)
|
||||
},
|
||||
}
|
||||
counts = {
|
||||
"requirements": len(
|
||||
read_list(view.deliverables, "requirements", REQUIREMENTS_FILE)
|
||||
),
|
||||
"recommendations": len(
|
||||
read_list(view.deliverables, "design_recommendations", RECOMMENDATIONS_FILE)
|
||||
),
|
||||
}
|
||||
facts["counts"] = {name: count for name, count in counts.items() if count}
|
||||
graph = read_graph(view.deliverables)
|
||||
if graph is not None:
|
||||
facts["target_architecture"] = graph.summary()
|
||||
validation = view.deliverables.get(DeliverableKind.VALIDATION_RESULTS.value)
|
||||
if (
|
||||
validation
|
||||
and phase is DEPENDENCY_GRAPH[DeliverableKind.VALIDATION_RESULTS].phase
|
||||
):
|
||||
outcome = validation_outcome(
|
||||
validation.get("attachments", {}).get(VERDICTS_FILE, "")
|
||||
)
|
||||
facts["validation"] = {
|
||||
"passed": len(outcome["passed"]),
|
||||
"failed": [item["check"] for item in outcome["failed"]],
|
||||
"not_verifiable": [item["check"] for item in outcome["not_verifiable"]],
|
||||
"open_gaps": [
|
||||
item.get("statement") for item in outcome["proposed_additions"]
|
||||
],
|
||||
}
|
||||
return facts
|
||||
|
||||
|
||||
def assembled_summary(phase: PhaseId, facts: Mapping[str, Any]) -> str:
|
||||
"""An Approval summary from the facts alone, used when no writer is available."""
|
||||
lines = [
|
||||
(
|
||||
f"The {phase.value} Phase is ready: "
|
||||
f"{', '.join(facts['deliverables'])} submitted."
|
||||
)
|
||||
]
|
||||
counts = facts.get("counts") or {}
|
||||
if counts:
|
||||
lines.append(
|
||||
", ".join(f"{count} {name}" for name, count in counts.items()) + "."
|
||||
)
|
||||
graph = facts.get("target_architecture")
|
||||
if graph:
|
||||
lines.append(
|
||||
f"Target architecture: {graph['elements']} Elements, "
|
||||
f"{graph['relationships']} Relationships, {graph['checks']} Checks."
|
||||
)
|
||||
validation = facts.get("validation")
|
||||
if validation:
|
||||
lines.append(
|
||||
f"Validation: {validation['passed']} Checks passed, "
|
||||
f"{len(validation['failed'])} failed, "
|
||||
f"{len(validation['not_verifiable'])} not verifiable."
|
||||
)
|
||||
if validation["open_gaps"]:
|
||||
lines.append(
|
||||
"Open gaps: " + "; ".join(map(str, validation["open_gaps"])) + "."
|
||||
)
|
||||
if facts.get("revisions"):
|
||||
lines.append(
|
||||
"Revised: "
|
||||
+ ", ".join(
|
||||
f"{kind} × {count}" for kind, count in facts["revisions"].items()
|
||||
)
|
||||
+ "."
|
||||
)
|
||||
return " ".join(lines)
|
||||
|
||||
|
||||
# --- the two ways an Orchestrator turn can be driven ---------------------------------
|
||||
|
||||
|
||||
class ModelOrchestrator:
|
||||
"""The Orchestrator as a reasoning agent: a language model chooses the tool calls.
|
||||
|
||||
Kept so the Ledger's rules can be driven in any order by a scripted runtime in
|
||||
tests, and so an evaluation can compare it with the router.
|
||||
"""
|
||||
|
||||
def __init__(self, runtime: AgentRuntime, role: RoleSpec) -> None:
|
||||
self._runtime = runtime
|
||||
self._role = role
|
||||
|
||||
async def turn(self, view: TurnView, tools: Sequence[ToolSpec]) -> TurnResult:
|
||||
task = AgentTask(
|
||||
role=ORCHESTRATOR,
|
||||
prompt="Decide the next step for this Execution using your tools.",
|
||||
context=view.context,
|
||||
)
|
||||
return TurnResult(await self._runtime.run(self._role, task, tools))
|
||||
|
||||
|
||||
class RuntimeReplyWriter:
|
||||
"""Writes what the user reads, on a language model. It is given a decision that
|
||||
has already been made and the facts, and it decides nothing."""
|
||||
|
||||
def __init__(self, runtime: AgentRuntime, role: RoleSpec) -> None:
|
||||
self._runtime = runtime
|
||||
self._role = role
|
||||
|
||||
async def approval_summary(
|
||||
self, phase: str, facts: Mapping[str, Any]
|
||||
) -> str | None:
|
||||
async def write_reply(text: str) -> ToolResult:
|
||||
if not text.strip():
|
||||
return ToolResult("Not submitted: the reply is empty.")
|
||||
return ToolResult("Written.", Submitted(text))
|
||||
|
||||
tool = ToolSpec(
|
||||
"write_reply",
|
||||
"Give the text the user will read. Ends your turn.",
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"text": {"type": "string"}},
|
||||
"required": ["text"],
|
||||
},
|
||||
write_reply,
|
||||
tier="execution",
|
||||
)
|
||||
task = AgentTask(
|
||||
role=REPLY_WRITER,
|
||||
prompt=f"Write the summary the user will approve the {phase} Phase from.",
|
||||
context={"phase": phase, "facts": dict(facts)},
|
||||
)
|
||||
outcome = await self._runtime.run(self._role, task, [tool])
|
||||
return outcome.content if isinstance(outcome, Submitted) else None
|
||||
@@ -42,10 +42,16 @@ class McpServerSpec:
|
||||
class ModelConfig:
|
||||
default: str
|
||||
roles: Mapping[str, str] = field(default_factory=dict)
|
||||
# Seconds one model request may take, for the roles whose replies are long. A
|
||||
# role not named here gets the runtime's default.
|
||||
timeouts: Mapping[str, float] = field(default_factory=dict)
|
||||
|
||||
def for_role(self, role: str) -> str:
|
||||
return self.roles.get(role, self.default)
|
||||
|
||||
def timeout_for(self, role: str) -> float | None:
|
||||
return self.timeouts.get(role)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProviderBundle:
|
||||
@@ -60,6 +66,9 @@ class ProviderBundle:
|
||||
role_mcp_servers: Mapping[str, tuple[str, ...]]
|
||||
role_skills: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
|
||||
pillar_skills: Mapping[str, str] = field(default_factory=dict)
|
||||
# The Terraform providers generated code is written for, pinned so the schema
|
||||
# lookup, the patterns and validation agree: {name: {source, version}}.
|
||||
terraform_providers: Mapping[str, Mapping[str, str]] = field(default_factory=dict)
|
||||
|
||||
def mcp_servers_for_role(
|
||||
self, role: str, include_preview: bool = True
|
||||
@@ -108,10 +117,34 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
||||
models = config.get("models") or {}
|
||||
if not isinstance(models.get("default"), str) or not models["default"]:
|
||||
raise fail("'models.default' is required")
|
||||
timeouts = {}
|
||||
for role, seconds in (models.get("timeout_seconds") or {}).items():
|
||||
if isinstance(seconds, bool) or not isinstance(seconds, (int, float)):
|
||||
raise fail(f"'models.timeout_seconds' for {role!r} must be a number")
|
||||
if seconds <= 0:
|
||||
raise fail(f"'models.timeout_seconds' for {role!r} must be positive")
|
||||
timeouts[str(role)] = float(seconds)
|
||||
model_config = ModelConfig(
|
||||
default=models["default"], roles=dict(models.get("roles") or {})
|
||||
default=models["default"],
|
||||
roles=dict(models.get("roles") or {}),
|
||||
timeouts=timeouts,
|
||||
)
|
||||
|
||||
terraform_providers = {}
|
||||
pins = (config.get("terraform") or {}).get("required_providers") or {}
|
||||
for provider, pin in pins.items():
|
||||
if not isinstance(pin, dict) or not all(
|
||||
isinstance(pin.get(key), str) and pin[key] for key in ("source", "version")
|
||||
):
|
||||
raise fail(
|
||||
f"terraform provider {provider!r} needs a 'source' and an exact "
|
||||
"'version', both strings"
|
||||
)
|
||||
terraform_providers[str(provider)] = {
|
||||
"source": pin["source"],
|
||||
"version": pin["version"],
|
||||
}
|
||||
|
||||
tools = {}
|
||||
for tier, tool_names in (config.get("tools") or {}).items():
|
||||
if tier not in TIERS:
|
||||
@@ -173,4 +206,5 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
||||
role_mcp_servers=role_mcp_servers,
|
||||
role_skills=role_skills,
|
||||
pillar_skills=pillar_skills,
|
||||
terraform_providers=terraform_providers,
|
||||
)
|
||||
|
||||
@@ -82,6 +82,13 @@ class ExecutionState(TypedDict, total=False):
|
||||
refusals: list[dict[str, Any]] # the Orchestrator's refused tool calls
|
||||
events_recorded: int
|
||||
|
||||
# Routing (ADR-0007).
|
||||
last_reply: dict[str, Any] | None # the user's message, until a turn routes it
|
||||
pending_change: dict[str, Any] | None # a change the user was asked to confirm
|
||||
awaiting_gap_decision: bool # the user was asked whether to close gaps
|
||||
validation_settled: int # the validation attempt already decided on
|
||||
decisions: list[dict[str, Any]] # every routing decision, in order
|
||||
|
||||
turns: int
|
||||
status: str # running | completed | completed_with_warnings
|
||||
warnings: list[str]
|
||||
|
||||
@@ -7,8 +7,7 @@ outcome.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from pathlib import PurePosixPath
|
||||
from collections.abc import Mapping, MutableMapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import (
|
||||
@@ -21,6 +20,14 @@ from app.adk.runtime import (
|
||||
files_from_argument,
|
||||
)
|
||||
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
|
||||
from app.tools.workspace import (
|
||||
file_tools,
|
||||
graph_text,
|
||||
graph_tools,
|
||||
keep_graph,
|
||||
safe_relative,
|
||||
working_files,
|
||||
)
|
||||
from app.workflows.deliverables import (
|
||||
DEPENDENCY_GRAPH,
|
||||
DeliverableKind,
|
||||
@@ -35,6 +42,7 @@ from app.workflows.target_architecture import (
|
||||
TargetArchitecture,
|
||||
parse_graph,
|
||||
render_mermaid,
|
||||
render_tables,
|
||||
)
|
||||
|
||||
ORCHESTRATOR = "orchestrator"
|
||||
@@ -244,29 +252,37 @@ class OrchestratorToolbox:
|
||||
return ToolResult("Finishing the Execution.", TurnEnded("finish"))
|
||||
|
||||
|
||||
def _safe_relative(name: str) -> bool:
|
||||
path = PurePosixPath(name)
|
||||
return bool(name) and not path.is_absolute() and ".." not in path.parts
|
||||
|
||||
|
||||
def specialist_tools(
|
||||
kind: DeliverableKind,
|
||||
extra: Sequence[ToolSpec],
|
||||
allow_cloud_read: bool | None,
|
||||
upstream: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
working: MutableMapping[str, Any] | None = None,
|
||||
library: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
) -> list[ToolSpec]:
|
||||
"""A Specialist's tools: submit, ask, and its role's tools with the cloud-read gate.
|
||||
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
||||
the cloud-read gate.
|
||||
|
||||
`upstream` is the Deliverables this one builds on; with it, a submission's links
|
||||
along the Requirements chain are checked.
|
||||
along the Requirements chain are checked. `working` is the Deliverable's working
|
||||
memory, where its working copy is kept. `library` is the earlier Deliverables
|
||||
given to the Specialist as data files alone, to read whole when it needs one.
|
||||
"""
|
||||
working = {} if working is None else working
|
||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||
own_files = DEPENDENCY_GRAPH[kind].working_copy is kind
|
||||
|
||||
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
|
||||
try:
|
||||
attachments = files_from_argument(attachments)
|
||||
except TypeError as exc:
|
||||
return _refused(f"attachments: {exc}")
|
||||
unsafe = [name for name in attachments if not _safe_relative(name)]
|
||||
# The working copy is the submission; a file passed here replaces its copy.
|
||||
if own_files:
|
||||
attachments = {**working_files(working), **attachments}
|
||||
if is_graph and GRAPH_ATTACHMENT not in attachments and graph_text(working):
|
||||
attachments[GRAPH_ATTACHMENT] = graph_text(working)
|
||||
unsafe = [name for name in attachments if not safe_relative(name)]
|
||||
if unsafe:
|
||||
return _refused(
|
||||
"attachment names must be relative paths without '..': "
|
||||
@@ -277,13 +293,27 @@ def specialist_tools(
|
||||
return ToolResult(
|
||||
"Not submitted. The Deliverable is missing: " + "; ".join(missing)
|
||||
)
|
||||
if kind is DeliverableKind.TARGET_ARCHITECTURE:
|
||||
# The diagram is a view of the graph, so the engine draws it.
|
||||
attachments[DIAGRAM_ATTACHMENT] = render_mermaid(
|
||||
parse_graph(attachments[GRAPH_ATTACHMENT])
|
||||
)
|
||||
if own_files:
|
||||
working_files(working).update(attachments)
|
||||
if is_graph:
|
||||
# The tables and the diagram are views of the graph, so the engine
|
||||
# writes them: they cannot disagree with it.
|
||||
graph = parse_graph(attachments[GRAPH_ATTACHMENT])
|
||||
keep_graph(working, attachments[GRAPH_ATTACHMENT])
|
||||
content = content.rstrip() + "\n\n" + render_tables(graph)
|
||||
attachments[DIAGRAM_ATTACHMENT] = render_mermaid(graph)
|
||||
return ToolResult(f"Submitted {kind}.", Submitted(content, attachments))
|
||||
|
||||
async def read_deliverable(kind: str) -> ToolResult:
|
||||
deliverable = (library or {}).get(kind)
|
||||
if deliverable is None:
|
||||
available = ", ".join(library or {}) or "none"
|
||||
return _refused(f"{kind} is not one you can read here: {available}")
|
||||
attached = ", ".join(deliverable.get("attachments", {})) or "none"
|
||||
return ToolResult(
|
||||
f"{deliverable.get('content', '')}\n\nAttachments: {attached}"
|
||||
)
|
||||
|
||||
async def needs_input(
|
||||
questions: Sequence[str], draft: str | None = None, attachments: Any = None
|
||||
) -> ToolResult:
|
||||
@@ -331,6 +361,23 @@ def specialist_tools(
|
||||
needs_input,
|
||||
tier="execution",
|
||||
),
|
||||
*(
|
||||
[
|
||||
ToolSpec(
|
||||
"read_deliverable",
|
||||
"Read the full text of an earlier Deliverable that `dependencies` "
|
||||
"gives you as its data file alone: "
|
||||
f"{', '.join(library)}. Use it only when the ids are not enough.",
|
||||
_schema({"kind": {"type": "string"}}, ["kind"]),
|
||||
read_deliverable,
|
||||
tier="execution",
|
||||
)
|
||||
]
|
||||
if library
|
||||
else ()
|
||||
),
|
||||
*(graph_tools(working) if is_graph else ()),
|
||||
*(file_tools(working) if own_files else ()),
|
||||
*(gate_cloud_read(tool, allow_cloud_read) for tool in extra),
|
||||
]
|
||||
|
||||
|
||||
327
app/tools/jev.py
Normal file
327
app/tools/jev.py
Normal file
@@ -0,0 +1,327 @@
|
||||
"""The Jev decision client: typed answers to closed questions, within strict limits.
|
||||
|
||||
Jev is TypeSafe AI's decision model. This module is the only one that speaks to it,
|
||||
through the vendor's Python SDK (`typesafe-sdk`), which owns the request shape, the
|
||||
timeout and the retries. Everything else here is ours: the service's limits are not
|
||||
published, so the client assumes they are strict and never lets Jev hold up an
|
||||
Execution. Any refusal, failure or timeout returns None and the caller takes its
|
||||
fallback.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from collections import OrderedDict, deque
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from typesafe_sdk import (
|
||||
AsyncTypeSafeClient,
|
||||
Choice,
|
||||
ChoiceAnswer,
|
||||
Noul,
|
||||
NoulAnswer,
|
||||
RetryPolicy,
|
||||
Score,
|
||||
ScoreAnswer,
|
||||
TypeSafeError,
|
||||
TypeSafeRateLimitError,
|
||||
)
|
||||
|
||||
from app.adk.runtime import Answer, Question
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class JevLimits:
|
||||
"""Every limit the client keeps. Starting values, to tune from measurement."""
|
||||
|
||||
max_tokens: int = 48_000 # state plus questions; the service allows 64,000
|
||||
max_question_tokens: int = 24_000 # state plus the longest question; allows 32,000
|
||||
request_timeout: float = 5.0 # one HTTP attempt
|
||||
total_timeout: float = 15.0 # a decision, retries included
|
||||
retries: int = 2
|
||||
backoff_base: float = 0.5
|
||||
backoff_cap: float = 8.0
|
||||
requests_per_second: float = 5.0
|
||||
burst: int = 10
|
||||
max_in_flight: int = 8
|
||||
breaker_failures: int = 5
|
||||
breaker_window: float = 30.0
|
||||
breaker_cooldown: float = 60.0
|
||||
cache_size: int = 1024
|
||||
budget_calls: int = 500 # per scope, usually one Execution
|
||||
budget_tokens: int = 1_000_000 # per scope; about $0.42 at the published price
|
||||
|
||||
|
||||
@dataclass
|
||||
class DecisionRecord:
|
||||
"""One attempt to decide, kept for the evaluation report and for tuning."""
|
||||
|
||||
scope: str
|
||||
questions: tuple[str, ...]
|
||||
# answered | cached | too_large | over_budget | breaker_open | failed
|
||||
outcome: str = "failed"
|
||||
seconds: float = 0.0
|
||||
rate_limited: bool = False
|
||||
input_tokens: int = 0
|
||||
confidence: Mapping[str, float] = field(default_factory=dict)
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""A deliberately high estimate: refusing locally is cheaper than a rejected call."""
|
||||
return len(text) // 3 + 1
|
||||
|
||||
|
||||
def encode_question(question: Question) -> Noul | Choice | Score:
|
||||
if question.kind == "choice":
|
||||
return Choice(
|
||||
instructions=question.instructions,
|
||||
criteria=dict.fromkeys(question.options),
|
||||
)
|
||||
if question.kind == "score":
|
||||
return Score(
|
||||
instructions=question.instructions, criteria=list(question.options)
|
||||
)
|
||||
return Noul(instructions=question.instructions)
|
||||
|
||||
|
||||
def decode_answer(question: Question, raw: object) -> Answer | None:
|
||||
"""The SDK's answer as ours, or None when it does not answer the question asked."""
|
||||
if question.kind == "choice" and isinstance(raw, ChoiceAnswer):
|
||||
# A choice outside the options asked for is not an answer.
|
||||
if raw.choice not in question.options:
|
||||
return None
|
||||
return Answer(raw.choice, raw.confidence, dict(raw.probabilities))
|
||||
if question.kind == "score" and isinstance(raw, ScoreAnswer):
|
||||
probabilities = {str(level): p for level, p in raw.probabilities.items()}
|
||||
return Answer(raw.score, raw.confidence, probabilities)
|
||||
if question.kind == "noul" and isinstance(raw, NoulAnswer):
|
||||
# Jev gives a probability for a yes-or-no question and no separate
|
||||
# confidence, so confidence is how far the probability is from even.
|
||||
return Answer(raw.noul, abs(2 * raw.noul - 1))
|
||||
return None
|
||||
|
||||
|
||||
class JevDecider:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
api_key: str,
|
||||
base_url: str | None = None,
|
||||
model: str | None = None,
|
||||
limits: JevLimits | None = None,
|
||||
transport: Any = None,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
) -> None:
|
||||
self.limits = limits or JevLimits()
|
||||
self._clock = clock
|
||||
self._sleep = sleep
|
||||
self._client = AsyncTypeSafeClient(
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
model=model,
|
||||
timeout=self.limits.request_timeout,
|
||||
# The SDK retries 408, 429 and 5xx, and connection and timeout errors,
|
||||
# with jittered backoff, and honours Retry-After.
|
||||
retry=RetryPolicy(
|
||||
max_retries=self.limits.retries,
|
||||
backoff_initial=self.limits.backoff_base,
|
||||
backoff_max=self.limits.backoff_cap,
|
||||
timeout=self.limits.total_timeout,
|
||||
),
|
||||
transport=transport,
|
||||
)
|
||||
self._in_flight = asyncio.Semaphore(self.limits.max_in_flight)
|
||||
# Token bucket: starts full, refills at `rate`, which drops when the service
|
||||
# says we are going too fast.
|
||||
self._rate = self.limits.requests_per_second
|
||||
self._tokens = float(self.limits.burst)
|
||||
self._refilled = clock()
|
||||
self._bucket = asyncio.Lock()
|
||||
self._failures: deque[float] = deque()
|
||||
self._open_until = 0.0
|
||||
self._probing = False
|
||||
self._cache: OrderedDict[str, dict[str, Answer]] = OrderedDict()
|
||||
self._calls: dict[str, int] = {}
|
||||
self._tokens_used: dict[str, int] = {}
|
||||
self._over_budget: set[str] = set()
|
||||
self.records: list[DecisionRecord] = []
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._client.aclose()
|
||||
|
||||
async def decide(
|
||||
self, state: str, questions: Mapping[str, Question], *, scope: str = ""
|
||||
) -> Mapping[str, Answer] | None:
|
||||
record = DecisionRecord(scope=scope, questions=tuple(questions))
|
||||
self.records.append(record)
|
||||
started = self._clock()
|
||||
try:
|
||||
return await self._decide(state, questions, scope, record)
|
||||
finally:
|
||||
record.seconds = self._clock() - started
|
||||
|
||||
async def _decide(
|
||||
self,
|
||||
state: str,
|
||||
questions: Mapping[str, Question],
|
||||
scope: str,
|
||||
record: DecisionRecord,
|
||||
) -> Mapping[str, Answer] | None:
|
||||
if not questions:
|
||||
record.outcome = "answered"
|
||||
return {}
|
||||
|
||||
asked = {
|
||||
name: {
|
||||
"kind": question.kind,
|
||||
"instructions": question.instructions,
|
||||
"options": list(question.options),
|
||||
}
|
||||
for name, question in questions.items()
|
||||
}
|
||||
sizes = [estimate_tokens(json.dumps(item)) for item in asked.values()]
|
||||
state_size = estimate_tokens(state)
|
||||
if (
|
||||
state_size + sum(sizes) > self.limits.max_tokens
|
||||
or state_size + max(sizes) > self.limits.max_question_tokens
|
||||
):
|
||||
# Never truncate: the caller must split by Deliverable or section.
|
||||
record.outcome = "too_large"
|
||||
return None
|
||||
|
||||
key = hashlib.sha256(
|
||||
json.dumps({"state": state, "questions": asked}, sort_keys=True).encode()
|
||||
).hexdigest()
|
||||
if key in self._cache:
|
||||
self._cache.move_to_end(key)
|
||||
record.outcome = "cached"
|
||||
return dict(self._cache[key])
|
||||
|
||||
if (
|
||||
self._calls.get(scope, 0) >= self.limits.budget_calls
|
||||
or self._tokens_used.get(scope, 0) >= self.limits.budget_tokens
|
||||
):
|
||||
if scope not in self._over_budget:
|
||||
self._over_budget.add(scope)
|
||||
logger.warning("Jev budget reached for %s; using fallbacks", scope)
|
||||
record.outcome = "over_budget"
|
||||
return None
|
||||
|
||||
if not self._breaker_allows():
|
||||
record.outcome = "breaker_open"
|
||||
return None
|
||||
|
||||
self._calls[scope] = self._calls.get(scope, 0) + 1
|
||||
await self._take_token()
|
||||
try:
|
||||
async with self._in_flight:
|
||||
response = await self._client.system_one(
|
||||
state=state,
|
||||
questions={
|
||||
name: encode_question(question)
|
||||
for name, question in questions.items()
|
||||
},
|
||||
)
|
||||
except TypeSafeRateLimitError:
|
||||
# Still rate limited after the SDK's retries: halve our own rate.
|
||||
record.rate_limited = True
|
||||
self._rate = max(self._rate / 2, 0.25)
|
||||
self._note_failure()
|
||||
return None
|
||||
except TypeSafeError as exc:
|
||||
logger.info("Jev request failed: %s", type(exc).__name__)
|
||||
self._note_failure()
|
||||
return None
|
||||
self._note_success()
|
||||
|
||||
used = response.usage.input_tokens or 0
|
||||
record.input_tokens = used
|
||||
self._tokens_used[scope] = self._tokens_used.get(scope, 0) + used
|
||||
|
||||
answers: dict[str, Answer] = {}
|
||||
for name, question in questions.items():
|
||||
answer = decode_answer(question, response.answers.get(name))
|
||||
if answer is None:
|
||||
# A reply that does not answer every question answers none.
|
||||
return None
|
||||
answers[name] = answer
|
||||
|
||||
record.outcome = "answered"
|
||||
record.confidence = {name: a.confidence for name, a in answers.items()}
|
||||
self._cache[key] = answers
|
||||
while len(self._cache) > self.limits.cache_size:
|
||||
self._cache.popitem(last=False)
|
||||
return dict(answers)
|
||||
|
||||
async def _take_token(self) -> None:
|
||||
async with self._bucket:
|
||||
while True:
|
||||
now = self._clock()
|
||||
self._tokens = min(
|
||||
float(self.limits.burst),
|
||||
self._tokens + (now - self._refilled) * self._rate,
|
||||
)
|
||||
self._refilled = now
|
||||
if self._tokens >= 1:
|
||||
self._tokens -= 1
|
||||
return
|
||||
await self._sleep((1 - self._tokens) / self._rate)
|
||||
|
||||
def _breaker_allows(self) -> bool:
|
||||
now = self._clock()
|
||||
if now < self._open_until:
|
||||
return False
|
||||
if self._open_until and not self._probing:
|
||||
# Cool-down over: let exactly one call through to test the service.
|
||||
self._probing = True
|
||||
return True
|
||||
|
||||
def _note_failure(self) -> None:
|
||||
now = self._clock()
|
||||
if self._probing:
|
||||
self._probing = False
|
||||
self._open_until = now + self.limits.breaker_cooldown
|
||||
return
|
||||
self._failures.append(now)
|
||||
while self._failures and now - self._failures[0] > self.limits.breaker_window:
|
||||
self._failures.popleft()
|
||||
if len(self._failures) >= self.limits.breaker_failures:
|
||||
self._failures.clear()
|
||||
self._open_until = now + self.limits.breaker_cooldown
|
||||
logger.warning(
|
||||
"Jev circuit open for %.0fs after repeated failures",
|
||||
self.limits.breaker_cooldown,
|
||||
)
|
||||
|
||||
def _note_success(self) -> None:
|
||||
self._failures.clear()
|
||||
self._open_until = 0.0
|
||||
self._probing = False
|
||||
|
||||
def summary(self) -> dict[str, Any]:
|
||||
"""Totals for the evaluation report."""
|
||||
outcomes: dict[str, int] = {}
|
||||
for record in self.records:
|
||||
outcomes[record.outcome] = outcomes.get(record.outcome, 0) + 1
|
||||
answered = [r for r in self.records if r.outcome == "answered"]
|
||||
return {
|
||||
"decisions": len(self.records),
|
||||
"outcomes": outcomes,
|
||||
"fallbacks": sum(
|
||||
count
|
||||
for name, count in outcomes.items()
|
||||
if name not in {"answered", "cached"}
|
||||
),
|
||||
"rate_limited": sum(1 for r in self.records if r.rate_limited),
|
||||
"seconds": round(sum(r.seconds for r in answered), 3),
|
||||
"input_tokens": sum(r.input_tokens for r in self.records),
|
||||
}
|
||||
@@ -1,16 +1,23 @@
|
||||
"""Local tools: a Mermaid syntax check and Terraform checks in a scratch directory.
|
||||
|
||||
Terraform runs only fmt, init -backend=false, validate and plan. No command that
|
||||
changes cloud resources is ever constructed.
|
||||
Terraform runs only fmt, init -backend=false, validate, plan and providers schema. No
|
||||
command that changes cloud resources is ever constructed.
|
||||
|
||||
A Specialist with a working copy (app/tools/workspace.py) runs the checks on it and
|
||||
passes no files; `terraform_fmt` formats the working copy in place. That is what
|
||||
keeps a fix cheap: the model changes a line and validates, without sending every file
|
||||
again on each call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import difflib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from collections.abc import Awaitable, Callable, Iterable, Mapping
|
||||
from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
@@ -79,10 +86,19 @@ class TerraformRunner:
|
||||
# Providers are downloaded here once and reused by every later check. Without it,
|
||||
# each validate or plan downloads its providers again into a scratch directory.
|
||||
plugin_cache_dir: str | None = None
|
||||
# The providers the schema lookup answers for, as the provider bundle pins them:
|
||||
# {local name: {"source": ..., "version": ...}}.
|
||||
providers: Mapping[str, Mapping[str, str]] = field(default_factory=dict)
|
||||
# Terraform's plugin cache is not safe for concurrent installs.
|
||||
_init_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False)
|
||||
_schema_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False)
|
||||
# resource type -> (provider label, the resource's schema block)
|
||||
_schemas: dict[str, tuple[str, dict[str, Any]]] | None = field(
|
||||
default=None, repr=False
|
||||
)
|
||||
|
||||
async def fmt(self, files: Mapping[str, str]) -> ToolResult:
|
||||
async def formatted(self, files: Mapping[str, str]) -> dict[str, str] | ToolResult:
|
||||
"""The files formatting changes, with their new content; or why it failed."""
|
||||
refusal = _check_files(files)
|
||||
if refusal:
|
||||
return refusal
|
||||
@@ -93,19 +109,24 @@ class TerraformRunner:
|
||||
)
|
||||
if code != 0:
|
||||
return ToolResult(f"terraform fmt failed:\n{output}")
|
||||
changed = [
|
||||
name for name in files if (root / name).read_text() != files[name]
|
||||
]
|
||||
if not changed:
|
||||
return ToolResult("terraform fmt: all files are already formatted.")
|
||||
formatted = "\n\n".join(
|
||||
f"--- {name}\n{(root / name).read_text()}" for name in changed
|
||||
)
|
||||
return ToolResult(
|
||||
_clip(
|
||||
f"terraform fmt reformatted {len(changed)} file(s):\n\n{formatted}"
|
||||
)
|
||||
)
|
||||
return {
|
||||
name: (root / name).read_text()
|
||||
for name in files
|
||||
if (root / name).read_text() != files[name]
|
||||
}
|
||||
|
||||
async def fmt(self, files: Mapping[str, str]) -> ToolResult:
|
||||
changed = await self.formatted(files)
|
||||
if isinstance(changed, ToolResult):
|
||||
return changed
|
||||
if not changed:
|
||||
return ToolResult("terraform fmt: all files are already formatted.")
|
||||
formatted = "\n\n".join(
|
||||
f"--- {name}\n{content}" for name, content in changed.items()
|
||||
)
|
||||
return ToolResult(
|
||||
_clip(f"terraform fmt reformatted {len(changed)} file(s):\n\n{formatted}")
|
||||
)
|
||||
|
||||
async def validate(self, files: Mapping[str, str]) -> ToolResult:
|
||||
refusal = _check_files(files)
|
||||
@@ -150,6 +171,87 @@ class TerraformRunner:
|
||||
_clip(f"terraform plan {verdict} (nothing was applied):\n{output}")
|
||||
)
|
||||
|
||||
async def schema(
|
||||
self, resource_type: str, block: str = "", find: str = ""
|
||||
) -> ToolResult:
|
||||
"""What a resource type accepts, from the pinned provider's own schema."""
|
||||
loaded = await self._load_schemas()
|
||||
if isinstance(loaded, ToolResult):
|
||||
return loaded
|
||||
if resource_type not in loaded:
|
||||
close = difflib.get_close_matches(resource_type, loaded, n=5, cutoff=0.6)
|
||||
hint = f" Close matches: {', '.join(close)}." if close else ""
|
||||
return ToolResult(
|
||||
f"No resource type {resource_type} in {self._provider_labels()}.{hint}"
|
||||
)
|
||||
label, root = loaded[resource_type]
|
||||
title = f"{resource_type} ({label})"
|
||||
if find:
|
||||
return ToolResult(_clip(_find_in_schema(title, root, find)))
|
||||
node = root
|
||||
for step in [part for part in block.split(".") if part]:
|
||||
nested = node.get("block_types", {}).get(step)
|
||||
if nested is None:
|
||||
inside = ", ".join(sorted(node.get("block_types", {}))) or "none"
|
||||
return ToolResult(
|
||||
f"{title} has no block {block!r}: {step!r} is not a block there. "
|
||||
f"Blocks at that level: {inside}."
|
||||
)
|
||||
node = nested["block"]
|
||||
heading = f"{title}, block {block}" if block else title
|
||||
return ToolResult(_clip(_render_schema(heading, node)))
|
||||
|
||||
def _provider_labels(self) -> str:
|
||||
return (
|
||||
", ".join(
|
||||
f"{pin['source']} {pin['version']}" for pin in self.providers.values()
|
||||
)
|
||||
or "no pinned provider"
|
||||
)
|
||||
|
||||
async def _load_schemas(self) -> dict[str, tuple[str, dict[str, Any]]] | ToolResult:
|
||||
async with self._schema_lock:
|
||||
if self._schemas is not None:
|
||||
return self._schemas
|
||||
if not self.providers:
|
||||
return ToolResult(
|
||||
"Schema lookup is unavailable: the Cloud provider bundle pins no "
|
||||
"Terraform provider."
|
||||
)
|
||||
required = "\n".join(
|
||||
f' {name} = {{ source = "{pin["source"]}", '
|
||||
f'version = "{pin["version"]}" }}'
|
||||
for name, pin in self.providers.items()
|
||||
)
|
||||
with tempfile.TemporaryDirectory(prefix="tf-schema-") as tmp:
|
||||
root = Path(tmp)
|
||||
(root / "versions.tf").write_text(
|
||||
f"terraform {{\n required_providers {{\n{required}\n }}\n}}\n"
|
||||
)
|
||||
failure = await self._init(root)
|
||||
if failure:
|
||||
return failure
|
||||
code, output = await self._run(
|
||||
[self.binary, "providers", "schema", "-json"], root
|
||||
)
|
||||
if code != 0:
|
||||
return ToolResult(
|
||||
_clip(f"terraform providers schema failed:\n{output}")
|
||||
)
|
||||
try:
|
||||
data = json.loads(output)
|
||||
except ValueError:
|
||||
return ToolResult("terraform providers schema returned no JSON.")
|
||||
labels = {pin["source"]: pin["version"] for pin in self.providers.values()}
|
||||
schemas: dict[str, tuple[str, dict[str, Any]]] = {}
|
||||
for address, provider in (data.get("provider_schemas") or {}).items():
|
||||
source = address.split("/", 1)[-1] # drop the registry host
|
||||
label = f"{source} {labels.get(source, '')}".strip()
|
||||
for name, resource in (provider.get("resource_schemas") or {}).items():
|
||||
schemas[name] = (label, resource.get("block") or {})
|
||||
self._schemas = schemas
|
||||
return schemas
|
||||
|
||||
async def _init(self, root: Path) -> ToolResult | None:
|
||||
command = [self.binary, "init", "-backend=false", "-input=false", "-no-color"]
|
||||
if self.plugin_cache_dir:
|
||||
@@ -236,6 +338,100 @@ def _clip(text: str) -> str:
|
||||
return text if len(text) <= _MAX_OUTPUT else text[:_MAX_OUTPUT] + "\n[truncated]"
|
||||
|
||||
|
||||
def _type(value: Any) -> str:
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
if isinstance(value, list) and len(value) == 2 and value[0] != "object":
|
||||
return f"{value[0]} of {_type(value[1])}"
|
||||
return "object"
|
||||
|
||||
|
||||
def _use(attribute: Mapping[str, Any]) -> str:
|
||||
if attribute.get("required"):
|
||||
return "required"
|
||||
return "optional" if attribute.get("optional") else "read-only"
|
||||
|
||||
|
||||
def _arguments(node: Mapping[str, Any]) -> dict[str, Mapping[str, Any]]:
|
||||
"""The arguments code may set: everything but the read-only ones."""
|
||||
return {
|
||||
name: attribute
|
||||
for name, attribute in sorted((node.get("attributes") or {}).items())
|
||||
if _use(attribute) != "read-only"
|
||||
}
|
||||
|
||||
|
||||
def _block_note(nested: Mapping[str, Any]) -> str:
|
||||
notes = [str(nested.get("nesting_mode", "block"))]
|
||||
if nested.get("min_items"):
|
||||
notes.append(f"at least {nested['min_items']}")
|
||||
if nested.get("max_items"):
|
||||
notes.append(f"at most {nested['max_items']}")
|
||||
return ", ".join(notes)
|
||||
|
||||
|
||||
def _render_schema(heading: str, node: Mapping[str, Any]) -> str:
|
||||
"""One level in full, and the blocks beneath it as an outline of names."""
|
||||
lines = [heading, "", "Arguments:"]
|
||||
arguments = _arguments(node)
|
||||
for name, attribute in arguments.items():
|
||||
described = " ".join(str(attribute.get("description") or "").split())[:160]
|
||||
deprecated = " DEPRECATED" if attribute.get("deprecated") else ""
|
||||
lines.append(
|
||||
f" {name} ({_type(attribute.get('type'))}, {_use(attribute)})"
|
||||
f"{deprecated}" + (f": {described}" if described else "")
|
||||
)
|
||||
if not arguments:
|
||||
lines.append(" none")
|
||||
|
||||
def outline(parent: Mapping[str, Any], depth: int) -> None:
|
||||
for name, nested in sorted((parent.get("block_types") or {}).items()):
|
||||
inner = nested.get("block") or {}
|
||||
names = ", ".join(
|
||||
f"{argument}*" if attribute.get("required") else argument
|
||||
for argument, attribute in _arguments(inner).items()
|
||||
)
|
||||
lines.append(
|
||||
f"{' ' * depth}{name} {{ }} ({_block_note(nested)}): {names or '-'}"
|
||||
)
|
||||
outline(inner, depth + 1)
|
||||
|
||||
lines += ["", "Blocks (* marks a required argument):"]
|
||||
before = len(lines)
|
||||
outline(node, 1)
|
||||
if len(lines) == before:
|
||||
lines.append(" none")
|
||||
lines += ["", "Pass `block`, such as a.b, for one block's arguments in full."]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _find_in_schema(title: str, root: Mapping[str, Any], wanted: str) -> str:
|
||||
"""Every argument or block whose name contains `wanted`, with where it sits."""
|
||||
needle = wanted.lower()
|
||||
found: list[str] = []
|
||||
|
||||
def search(node: Mapping[str, Any], path: str) -> None:
|
||||
for name, attribute in sorted((node.get("attributes") or {}).items()):
|
||||
if needle in name.lower():
|
||||
where = f"{path}.{name}" if path else name
|
||||
found.append(
|
||||
f" {where} ({_type(attribute.get('type'))}, {_use(attribute)})"
|
||||
)
|
||||
for name, nested in sorted((node.get("block_types") or {}).items()):
|
||||
where = f"{path}.{name}" if path else name
|
||||
if needle in name.lower():
|
||||
found.append(f" {where} {{ }} ({_block_note(nested)})")
|
||||
search(nested.get("block") or {}, where)
|
||||
|
||||
search(root, "")
|
||||
if not found:
|
||||
return (
|
||||
f"{title} has no argument or block named like {wanted!r}. It cannot be "
|
||||
"set on this resource type."
|
||||
)
|
||||
return f"{title}: {wanted!r} is found at\n" + "\n".join(found)
|
||||
|
||||
|
||||
_FILES_SCHEMA: dict[str, Any] = {
|
||||
**FILES_SCHEMA,
|
||||
"description": "Terraform files in one directory, e.g. terraform/main.tf",
|
||||
@@ -244,19 +440,53 @@ _FILES_SCHEMA: dict[str, Any] = {
|
||||
|
||||
def _with_files(
|
||||
method: Callable[..., Awaitable[ToolResult]],
|
||||
working_copy: MutableMapping[str, str] | None = None,
|
||||
) -> Callable[..., Awaitable[ToolResult]]:
|
||||
"""Adapt a runner method to take the FILES_SCHEMA argument."""
|
||||
"""Adapt a runner method to take the FILES_SCHEMA argument.
|
||||
|
||||
async def handler(files: Any, **kwargs: Any) -> ToolResult:
|
||||
With a working copy, the files are the Terraform files in it and the argument is
|
||||
optional: files passed anyway are written to the working copy first.
|
||||
"""
|
||||
|
||||
async def handler(files: Any = None, **kwargs: Any) -> ToolResult:
|
||||
try:
|
||||
converted = files_from_argument(files)
|
||||
except TypeError as exc:
|
||||
return ToolResult(f"Refused: files: {exc}")
|
||||
return await method(converted, **kwargs)
|
||||
if working_copy is None:
|
||||
return await method(converted, **kwargs)
|
||||
working_copy.update(converted)
|
||||
return await method(_terraform_files(working_copy), **kwargs)
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
def _terraform_files(files: Mapping[str, str]) -> dict[str, str]:
|
||||
return {
|
||||
name: content
|
||||
for name, content in files.items()
|
||||
if name.endswith(_TERRAFORM_SUFFIXES)
|
||||
}
|
||||
|
||||
|
||||
def _fmt_in_place(
|
||||
runner: TerraformRunner, working_copy: MutableMapping[str, str]
|
||||
) -> Callable[..., Awaitable[ToolResult]]:
|
||||
async def format_working_copy(files: Mapping[str, str]) -> ToolResult:
|
||||
changed = await runner.formatted(files)
|
||||
if isinstance(changed, ToolResult):
|
||||
return changed
|
||||
working_copy.update(changed)
|
||||
if not changed:
|
||||
return ToolResult("terraform fmt: all files are already formatted.")
|
||||
return ToolResult(
|
||||
f"terraform fmt reformatted {len(changed)} file(s) in your working "
|
||||
f"copy: {', '.join(sorted(changed))}."
|
||||
)
|
||||
|
||||
return _with_files(format_working_copy, working_copy)
|
||||
|
||||
|
||||
LOCAL_TOOLS: dict[str, tuple[str, str]] = {
|
||||
"mermaid_check": ("general", "Check a Mermaid diagram's basic structure."),
|
||||
"terraform_fmt": ("general", "Format Terraform files and return any that changed."),
|
||||
@@ -268,10 +498,48 @@ LOCAL_TOOLS: dict[str, tuple[str, str]] = {
|
||||
"cloud_read",
|
||||
"Preview Terraform against the live cloud environment with a plan. Never applies.",
|
||||
),
|
||||
"terraform_schema": (
|
||||
"general",
|
||||
(
|
||||
"Look up what a Terraform resource type accepts, from the provider's own "
|
||||
"schema: its arguments and blocks, one block in full with `block`, or "
|
||||
"whether an argument exists anywhere in it with `find`."
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
_ON_WORKING_COPY = " Runs on your working copy; pass no files."
|
||||
|
||||
|
||||
def local_tools(
|
||||
names: Iterable[str],
|
||||
runner: TerraformRunner,
|
||||
working_copy: MutableMapping[str, str] | None = None,
|
||||
) -> list[ToolSpec]:
|
||||
"""The named tools. With a `working_copy`, the Terraform checks run on it."""
|
||||
files_required = [] if working_copy is not None else ["files"]
|
||||
suffix = _ON_WORKING_COPY if working_copy is not None else ""
|
||||
|
||||
def checks(name: str, method: Callable[..., Awaitable[ToolResult]]) -> ToolSpec:
|
||||
properties: dict[str, Any] = {"files": _FILES_SCHEMA}
|
||||
if name == "terraform_plan":
|
||||
properties["variables"] = {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"},
|
||||
}
|
||||
tier, description = LOCAL_TOOLS[name]
|
||||
return ToolSpec(
|
||||
name,
|
||||
description + suffix,
|
||||
{
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
"required": files_required,
|
||||
},
|
||||
method,
|
||||
tier=tier,
|
||||
)
|
||||
|
||||
def local_tools(names: Iterable[str], runner: TerraformRunner) -> list[ToolSpec]:
|
||||
builders = {
|
||||
"mermaid_check": lambda: ToolSpec(
|
||||
"mermaid_check",
|
||||
@@ -284,44 +552,41 @@ def local_tools(names: Iterable[str], runner: TerraformRunner) -> list[ToolSpec]
|
||||
mermaid_check,
|
||||
tier="general",
|
||||
),
|
||||
"terraform_fmt": lambda: ToolSpec(
|
||||
"terraform_fmt": lambda: checks(
|
||||
"terraform_fmt",
|
||||
LOCAL_TOOLS["terraform_fmt"][1],
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"files": _FILES_SCHEMA},
|
||||
"required": ["files"],
|
||||
},
|
||||
_with_files(runner.fmt),
|
||||
tier="general",
|
||||
_fmt_in_place(runner, working_copy)
|
||||
if working_copy is not None
|
||||
else _with_files(runner.fmt),
|
||||
),
|
||||
"terraform_validate": lambda: ToolSpec(
|
||||
"terraform_validate",
|
||||
LOCAL_TOOLS["terraform_validate"][1],
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"files": _FILES_SCHEMA},
|
||||
"required": ["files"],
|
||||
},
|
||||
_with_files(runner.validate),
|
||||
tier="general",
|
||||
"terraform_validate": lambda: checks(
|
||||
"terraform_validate", _with_files(runner.validate, working_copy)
|
||||
),
|
||||
"terraform_plan": lambda: ToolSpec(
|
||||
"terraform_plan",
|
||||
LOCAL_TOOLS["terraform_plan"][1],
|
||||
"terraform_plan": lambda: checks(
|
||||
"terraform_plan", _with_files(runner.plan, working_copy)
|
||||
),
|
||||
"terraform_schema": lambda: ToolSpec(
|
||||
"terraform_schema",
|
||||
LOCAL_TOOLS["terraform_schema"][1],
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"files": _FILES_SCHEMA,
|
||||
"variables": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"},
|
||||
"resource_type": {
|
||||
"type": "string",
|
||||
"description": "For example google_cloud_run_v2_service",
|
||||
},
|
||||
"block": {
|
||||
"type": "string",
|
||||
"description": "A nested block, such as template.containers",
|
||||
},
|
||||
"find": {
|
||||
"type": "string",
|
||||
"description": "An argument or block name to look for",
|
||||
},
|
||||
},
|
||||
"required": ["files"],
|
||||
"required": ["resource_type"],
|
||||
},
|
||||
_with_files(runner.plan),
|
||||
tier="cloud_read",
|
||||
runner.schema,
|
||||
tier="general",
|
||||
),
|
||||
}
|
||||
return [builders[name]() for name in names]
|
||||
|
||||
@@ -2,19 +2,24 @@
|
||||
|
||||
A role gets the local tools its role file names, the tools of the remote MCP servers
|
||||
the provider bundle assigns it, and, for the design advisor, the pillar consultation.
|
||||
When the Deliverable has a working copy, the local tools are built on it.
|
||||
Cloud-read gating is applied later, per Execution, by `specialist_tools`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, MutableMapping, Sequence
|
||||
from collections.abc import Callable, Mapping, MutableMapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.roles import PILLAR_ADVISOR
|
||||
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory, ToolSpec
|
||||
from app.tools.mcp import McpCatalog
|
||||
from app.tools.pillars import pillar_consultation_tool
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
from app.tools.workspace import working_files
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||
|
||||
# The local tools, built on a working copy of files when the Deliverable has one.
|
||||
LocalToolBuilder = Callable[[MutableMapping[str, str] | None], Sequence[ToolSpec]]
|
||||
|
||||
|
||||
def make_tool_factory(
|
||||
@@ -22,11 +27,14 @@ def make_tool_factory(
|
||||
roles: Mapping[str, RoleSpec],
|
||||
pillar_skills: Mapping[str, str],
|
||||
*,
|
||||
local_tools: Sequence[ToolSpec] = (),
|
||||
local_tools: Sequence[ToolSpec] | LocalToolBuilder = (),
|
||||
catalog: McpCatalog | None = None,
|
||||
limiter: RunLimiter | None = None,
|
||||
) -> ToolFactory:
|
||||
local_by_name = {tool.name: tool for tool in local_tools}
|
||||
def build_local(
|
||||
working_copy: MutableMapping[str, str] | None,
|
||||
) -> Sequence[ToolSpec]:
|
||||
return local_tools(working_copy) if callable(local_tools) else local_tools
|
||||
|
||||
def mcp_tools(role: RoleSpec) -> list[ToolSpec]:
|
||||
return catalog.tools_for(role.mcp_servers) if catalog else []
|
||||
@@ -42,6 +50,16 @@ def make_tool_factory(
|
||||
def tools_for(
|
||||
role: RoleSpec, deliverable: str | None, working: MutableMapping[str, Any]
|
||||
) -> Sequence[ToolSpec]:
|
||||
has_working_copy = any(
|
||||
kind.value == deliverable and spec.working_copy is not None
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
)
|
||||
local_by_name = {
|
||||
tool.name: tool
|
||||
for tool in build_local(
|
||||
working_files(working) if has_working_copy else None
|
||||
)
|
||||
}
|
||||
tools = [local_by_name[name] for name in role.tools if name in local_by_name]
|
||||
tools += mcp_tools(role)
|
||||
if (
|
||||
|
||||
289
app/tools/workspace.py
Normal file
289
app/tools/workspace.py
Normal file
@@ -0,0 +1,289 @@
|
||||
"""A Specialist's working copy: files and a graph built up over several tool calls.
|
||||
|
||||
A model that must return a whole Deliverable in one reply spends most of its output
|
||||
repeating itself, and a long reply is what times out. With a working copy it writes a
|
||||
file once, changes only the lines that need changing, and adds to a graph a few items
|
||||
at a time. The working copy is kept with the Execution, so a Revision starts from the
|
||||
last attempt instead of from nothing.
|
||||
|
||||
It lives in the Deliverable's working memory:
|
||||
|
||||
working["files"] {path: content}
|
||||
working["graph"] the Target architecture, as target-architecture.json holds it
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping, MutableMapping, Sequence
|
||||
from pathlib import PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import ToolResult, ToolSpec
|
||||
|
||||
FILES = "files"
|
||||
GRAPH = "graph"
|
||||
_GRAPH_LISTS = ("elements", "relationships", "checks", "not_applicable")
|
||||
_MAX_READ = 40_000
|
||||
|
||||
|
||||
def safe_relative(name: str) -> bool:
|
||||
path = PurePosixPath(name)
|
||||
return bool(name) and not path.is_absolute() and ".." not in path.parts
|
||||
|
||||
|
||||
def working_files(working: MutableMapping[str, Any]) -> MutableMapping[str, str]:
|
||||
files = working.get(FILES)
|
||||
if not isinstance(files, MutableMapping):
|
||||
files = working[FILES] = {}
|
||||
return files
|
||||
|
||||
|
||||
def _schema(properties: Mapping[str, Any], required: Sequence[str]) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": dict(properties),
|
||||
"required": list(required),
|
||||
}
|
||||
|
||||
|
||||
def _lines(content: str) -> int:
|
||||
return content.count("\n") + (1 if content and not content.endswith("\n") else 0)
|
||||
|
||||
|
||||
def file_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
|
||||
"""Tools that write, change and read the files of a working copy."""
|
||||
files = working_files(working)
|
||||
|
||||
async def write_file(path: str, content: str) -> ToolResult:
|
||||
if not safe_relative(path):
|
||||
return ToolResult(
|
||||
f"Refused: {path!r} must be a relative path without '..'."
|
||||
)
|
||||
files[path] = content
|
||||
return ToolResult(f"Wrote {path} ({_lines(content)} lines).")
|
||||
|
||||
async def edit_file(path: str, old: str, new: str) -> ToolResult:
|
||||
if path not in files:
|
||||
return ToolResult(f"Refused: {path} is not in your working copy.")
|
||||
found = files[path].count(old) if old else 0
|
||||
if found != 1:
|
||||
return ToolResult(
|
||||
f"Refused: `old` must match exactly one place in {path}; it matches "
|
||||
f"{found}. Include more of the surrounding lines."
|
||||
)
|
||||
files[path] = files[path].replace(old, new)
|
||||
return ToolResult(f"Edited {path}.")
|
||||
|
||||
async def read_file(path: str = "") -> ToolResult:
|
||||
if not path:
|
||||
if not files:
|
||||
return ToolResult("Your working copy is empty.")
|
||||
return ToolResult(
|
||||
"Your working copy:\n"
|
||||
+ "\n".join(
|
||||
f"- {name} ({_lines(files[name])} lines)" for name in sorted(files)
|
||||
)
|
||||
)
|
||||
if path not in files:
|
||||
return ToolResult(f"Refused: {path} is not in your working copy.")
|
||||
content = files[path]
|
||||
if len(content) > _MAX_READ:
|
||||
content = content[:_MAX_READ] + "\n[truncated]"
|
||||
return ToolResult(content)
|
||||
|
||||
async def delete_file(path: str) -> ToolResult:
|
||||
if files.pop(path, None) is None:
|
||||
return ToolResult(f"Refused: {path} is not in your working copy.")
|
||||
return ToolResult(f"Deleted {path}.")
|
||||
|
||||
text = {"type": "string"}
|
||||
return [
|
||||
ToolSpec(
|
||||
"write_file",
|
||||
"Write one file of your working copy, replacing it if it exists. Your "
|
||||
"working copy is attached for you when you submit.",
|
||||
_schema({"path": text, "content": text}, ["path", "content"]),
|
||||
write_file,
|
||||
),
|
||||
ToolSpec(
|
||||
"edit_file",
|
||||
"Change part of one file: replace the single place `old` matches with "
|
||||
"`new`. Use this for a fix instead of writing the file again.",
|
||||
_schema({"path": text, "old": text, "new": text}, ["path", "old", "new"]),
|
||||
edit_file,
|
||||
),
|
||||
ToolSpec(
|
||||
"read_file",
|
||||
"Read one file of your working copy, or list its files when `path` is "
|
||||
"left out.",
|
||||
_schema({"path": text}, []),
|
||||
read_file,
|
||||
),
|
||||
ToolSpec(
|
||||
"delete_file",
|
||||
"Remove one file from your working copy.",
|
||||
_schema({"path": text}, ["path"]),
|
||||
delete_file,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
# --- the graph -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def working_graph(working: MutableMapping[str, Any]) -> dict[str, list[dict[str, Any]]]:
|
||||
graph = working.get(GRAPH)
|
||||
if not isinstance(graph, dict):
|
||||
graph = working[GRAPH] = {}
|
||||
for name in _GRAPH_LISTS:
|
||||
if not isinstance(graph.get(name), list):
|
||||
graph[name] = []
|
||||
return graph
|
||||
|
||||
|
||||
def graph_text(working: MutableMapping[str, Any]) -> str:
|
||||
"""The working graph as target-architecture.json, or "" when nothing is in it."""
|
||||
graph = working.get(GRAPH)
|
||||
if not isinstance(graph, dict) or not any(graph.get(n) for n in _GRAPH_LISTS):
|
||||
return ""
|
||||
return json.dumps(
|
||||
{name: graph.get(name, []) for name in _GRAPH_LISTS},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
|
||||
def keep_graph(working: MutableMapping[str, Any], text: str) -> None:
|
||||
"""Make a submitted graph the working graph, so a Revision amends it."""
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except ValueError:
|
||||
return
|
||||
if isinstance(data, dict):
|
||||
working[GRAPH] = {
|
||||
name: [item for item in data.get(name, []) if isinstance(item, dict)]
|
||||
for name in _GRAPH_LISTS
|
||||
}
|
||||
|
||||
|
||||
def _key(name: str) -> str:
|
||||
return "recommendation" if name == "not_applicable" else "id"
|
||||
|
||||
|
||||
def graph_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
|
||||
"""Tools that build the Target architecture a few items at a time."""
|
||||
|
||||
async def graph_put(
|
||||
elements: Sequence[Any] = (),
|
||||
relationships: Sequence[Any] = (),
|
||||
checks: Sequence[Any] = (),
|
||||
not_applicable: Sequence[Any] = (),
|
||||
) -> ToolResult:
|
||||
graph = working_graph(working)
|
||||
given = dict(
|
||||
zip(
|
||||
_GRAPH_LISTS,
|
||||
(elements, relationships, checks, not_applicable),
|
||||
strict=True,
|
||||
)
|
||||
)
|
||||
bad = [
|
||||
f"{name}[{index}]"
|
||||
for name, items in given.items()
|
||||
for index, item in enumerate(items or ())
|
||||
if not isinstance(item, Mapping)
|
||||
or not isinstance(item.get(_key(name)), str)
|
||||
or not item[_key(name)].strip()
|
||||
]
|
||||
if bad:
|
||||
return ToolResult(
|
||||
"Refused: each item must be an object with an `id` (a "
|
||||
f"`recommendation` under not_applicable): {', '.join(bad)}"
|
||||
)
|
||||
added = replaced = 0
|
||||
for name, items in given.items():
|
||||
key = _key(name)
|
||||
for item in items or ():
|
||||
for index, existing in enumerate(graph[name]):
|
||||
if existing.get(key) == item[key]:
|
||||
graph[name][index] = dict(item)
|
||||
replaced += 1
|
||||
break
|
||||
else:
|
||||
graph[name].append(dict(item))
|
||||
added += 1
|
||||
return ToolResult(f"Added {added}, replaced {replaced}. {_counts(graph)}")
|
||||
|
||||
async def graph_remove(ids: Sequence[str]) -> ToolResult:
|
||||
graph = working_graph(working)
|
||||
wanted = set(ids)
|
||||
removed = 0
|
||||
for name in _GRAPH_LISTS:
|
||||
kept = [item for item in graph[name] if item.get(_key(name)) not in wanted]
|
||||
removed += len(graph[name]) - len(kept)
|
||||
graph[name] = kept
|
||||
return ToolResult(f"Removed {removed}. {_counts(graph)}")
|
||||
|
||||
async def graph_read() -> ToolResult:
|
||||
return ToolResult(graph_text(working) or "Your working graph is empty.")
|
||||
|
||||
def items(*fields: str, optional: tuple[str, ...] = ()) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "array",
|
||||
"items": _schema(
|
||||
{name: {"type": "string"} for name in (*fields, *optional)}, fields
|
||||
),
|
||||
}
|
||||
|
||||
lists = {
|
||||
"elements": items(
|
||||
"id", "name", "product", "resource_type", optional=("component",)
|
||||
),
|
||||
"relationships": items("id", "from", "to", "kind"),
|
||||
"checks": items("id", "applies_to", "statement", "source"),
|
||||
"not_applicable": items("recommendation", "reason"),
|
||||
}
|
||||
return [
|
||||
ToolSpec(
|
||||
"graph_put",
|
||||
"Add items to your working graph, or replace the ones with the same id. "
|
||||
"Call it several times with a few items each. Your working graph is "
|
||||
"attached as target-architecture.json when you submit.",
|
||||
_schema(lists, []),
|
||||
graph_put,
|
||||
),
|
||||
ToolSpec(
|
||||
"graph_remove",
|
||||
"Remove items from your working graph by id (a not-applicable entry by "
|
||||
"its recommendation id).",
|
||||
_schema({"ids": {"type": "array", "items": {"type": "string"}}}, ["ids"]),
|
||||
graph_remove,
|
||||
),
|
||||
ToolSpec(
|
||||
"graph_read",
|
||||
"Read your working graph.",
|
||||
{"type": "object", "properties": {}},
|
||||
graph_read,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _counts(graph: Mapping[str, Sequence[Any]]) -> str:
|
||||
return (
|
||||
f"The graph has {len(graph['elements'])} Elements, "
|
||||
f"{len(graph['relationships'])} Relationships and "
|
||||
f"{len(graph['checks'])} Checks."
|
||||
)
|
||||
|
||||
|
||||
def summary(working: Mapping[str, Any]) -> dict[str, Any] | None:
|
||||
"""What a Specialist is told it already has, at the start of a run."""
|
||||
told: dict[str, Any] = {}
|
||||
files = working.get(FILES)
|
||||
if isinstance(files, Mapping) and files:
|
||||
told["files"] = sorted(files)
|
||||
graph = working.get(GRAPH)
|
||||
if isinstance(graph, Mapping) and any(graph.get(name) for name in _GRAPH_LISTS):
|
||||
told["graph"] = {name: len(graph.get(name) or ()) for name in _GRAPH_LISTS}
|
||||
return told or None
|
||||
@@ -242,20 +242,51 @@ def _covers_every_check(
|
||||
return problems
|
||||
|
||||
|
||||
def _split_implementation(text: str, name: str) -> tuple[str, str]:
|
||||
"""The entries that say where a Check is met, and those that say it cannot be."""
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except ValueError:
|
||||
return text, "[]"
|
||||
if not isinstance(data, list):
|
||||
return text, "[]"
|
||||
cannot = [
|
||||
item for item in data if isinstance(item, dict) and "unimplementable" in item
|
||||
]
|
||||
met = [item for item in data if item not in cannot]
|
||||
return json.dumps(met), json.dumps(cannot)
|
||||
|
||||
|
||||
def check_implementation(
|
||||
attachments: Mapping[str, str], upstream: Upstream | None
|
||||
) -> list[str]:
|
||||
name = IMPLEMENTATION_FILE
|
||||
items, problems = _objects(attachments[name], name, ("check", "file", "address"))
|
||||
problems += _covers_every_check(items, name, read_graph(upstream))
|
||||
met_text, cannot_text = _split_implementation(attachments[name], name)
|
||||
met, problems = _objects(met_text, name, ("check", "file", "address"))
|
||||
cannot, faults = _objects(cannot_text, name, ("check", "unimplementable"))
|
||||
problems += faults
|
||||
problems += _covers_every_check([*met, *cannot], name, read_graph(upstream))
|
||||
problems += [
|
||||
f"{name}: {item['check']} names {item['file']}, which is not attached"
|
||||
for item in items
|
||||
for item in met
|
||||
if item["file"] not in attachments
|
||||
]
|
||||
return problems
|
||||
|
||||
|
||||
def unimplementable_checks(implementation_text: str) -> list[dict[str, str]]:
|
||||
"""The Checks the code's author found cannot be met as written, with the reason.
|
||||
|
||||
Each is a fault in the Target architecture, not in the code: the Check names
|
||||
something the resource does not have.
|
||||
"""
|
||||
_, cannot_text = _split_implementation(implementation_text, IMPLEMENTATION_FILE)
|
||||
cannot, _ = _objects(cannot_text, IMPLEMENTATION_FILE, ("check", "unimplementable"))
|
||||
return [
|
||||
{"check": item["check"], "reason": item["unimplementable"]} for item in cannot
|
||||
]
|
||||
|
||||
|
||||
def check_verdicts(
|
||||
attachments: Mapping[str, str], upstream: Upstream | None
|
||||
) -> list[str]:
|
||||
|
||||
@@ -64,6 +64,14 @@ class DeliverableSpec:
|
||||
depends_on: tuple[DeliverableKind, ...] = ()
|
||||
sections: tuple[str, ...] = ()
|
||||
attachments: tuple[str, ...] = ()
|
||||
# Which earlier Deliverables the Specialist reads whole. None means the ones it
|
||||
# depends on. Every other input arrives as its data file alone: the ids the
|
||||
# Requirements chain runs through, without the prose written around them.
|
||||
reads_in_full: tuple[DeliverableKind, ...] | None = None
|
||||
# The Deliverable whose files this Specialist's tools work on. Its own kind: the
|
||||
# files are its working copy, kept across attempts and attached on submission.
|
||||
# Another kind: a fresh copy of that Deliverable's files on every attempt.
|
||||
working_copy: DeliverableKind | None = None
|
||||
|
||||
def contract(self) -> dict[str, list[str]]:
|
||||
"""What a submission must contain, as plain data for the Specialist's context."""
|
||||
@@ -126,8 +134,10 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
||||
PhaseId.SOLUTION_ARCHITECTURE,
|
||||
"target-architect",
|
||||
depends_on=(_K.PRODUCT_SELECTION, _K.DESIGN_RECOMMENDATIONS),
|
||||
sections=("Elements", "Relationships", "Checks"),
|
||||
# The engine adds architecture.mmd, rendered from the graph, on submission.
|
||||
# The Specialist writes the reasoning and the graph. On submission the
|
||||
# engine adds the Elements, Relationships and Checks tables to the content
|
||||
# and architecture.mmd to the attachments, both rendered from the graph.
|
||||
sections=("Design rationale",),
|
||||
attachments=(GRAPH_ATTACHMENT,),
|
||||
),
|
||||
DeliverableSpec(
|
||||
@@ -151,6 +161,9 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
||||
"terraform/variables.tf",
|
||||
IMPLEMENTATION_FILE,
|
||||
),
|
||||
# The graph is all the code is written from.
|
||||
reads_in_full=(),
|
||||
working_copy=_K.DEPLOYMENT_GUIDANCE,
|
||||
),
|
||||
DeliverableSpec(
|
||||
_K.VALIDATION_RESULTS,
|
||||
@@ -159,6 +172,7 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
||||
depends_on=(_K.DEPLOYMENT_GUIDANCE,),
|
||||
sections=("Deployment dry-run", "Architecture and policy analysis"),
|
||||
attachments=(VERDICTS_FILE,),
|
||||
working_copy=_K.DEPLOYMENT_GUIDANCE,
|
||||
),
|
||||
DeliverableSpec(
|
||||
_K.SOLUTION_GUIDE,
|
||||
@@ -175,6 +189,8 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
||||
"Validation plan",
|
||||
"References",
|
||||
),
|
||||
# The guide is assembled from every Deliverable's prose.
|
||||
reads_in_full=tuple(_K)[:-1],
|
||||
),
|
||||
)
|
||||
}
|
||||
@@ -218,6 +234,31 @@ def inputs_of(kind: DeliverableKind) -> tuple[DeliverableKind, ...]:
|
||||
)
|
||||
|
||||
|
||||
def dependencies_for(
|
||||
kind: DeliverableKind, deliverables: Mapping[str, Mapping[str, object]]
|
||||
) -> dict[str, dict[str, object]]:
|
||||
"""What a Specialist is given of each Deliverable it works from.
|
||||
|
||||
A Deliverable read in full arrives as submitted. Any other arrives as its data
|
||||
file alone, which is all the Requirements chain needs of it.
|
||||
"""
|
||||
spec = DEPENDENCY_GRAPH[kind]
|
||||
in_full = spec.depends_on if spec.reads_in_full is None else spec.reads_in_full
|
||||
given: dict[str, dict[str, object]] = {}
|
||||
for dependency in inputs_of(kind):
|
||||
deliverable = deliverables[dependency.value]
|
||||
if dependency in in_full or dependency not in DATA_CHECKS:
|
||||
given[dependency.value] = dict(deliverable)
|
||||
continue
|
||||
name = DATA_CHECKS[dependency][0]
|
||||
attachments = deliverable.get("attachments")
|
||||
files = attachments if isinstance(attachments, Mapping) else {}
|
||||
given[dependency.value] = {
|
||||
"attachments": {name: files[name]} if name in files else {}
|
||||
}
|
||||
return given
|
||||
|
||||
|
||||
def deliverables_in(phase: PhaseId) -> tuple[DeliverableKind, ...]:
|
||||
return tuple(kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase)
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ from langgraph.graph import END, START, StateGraph
|
||||
|
||||
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory
|
||||
from app.nodes.execution_nodes import make_execution_nodes
|
||||
from app.nodes.router import OrchestratorPolicy
|
||||
from app.recorder import ExecutionRecorder
|
||||
from app.states.execution_state import ExecutionState
|
||||
|
||||
@@ -34,6 +35,7 @@ def build_execution_graph(
|
||||
tool_factory: ToolFactory | None = None,
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
) -> StateGraph:
|
||||
nodes = make_execution_nodes(
|
||||
runtime=runtime,
|
||||
@@ -44,6 +46,7 @@ def build_execution_graph(
|
||||
tool_factory=tool_factory,
|
||||
limiter=limiter,
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
)
|
||||
|
||||
graph = StateGraph(ExecutionState)
|
||||
|
||||
@@ -6,7 +6,8 @@
|
||||
to the requirement or design recommendation it comes from
|
||||
|
||||
This module is pure: it parses the graph, states what is wrong with it, renders it as
|
||||
a diagram, and says whether the Requirements chain through it is complete.
|
||||
a diagram and as tables a person can review, and says whether the Requirements chain
|
||||
through it is complete.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -281,3 +282,64 @@ def render_mermaid(graph: TargetArchitecture) -> str:
|
||||
f"{nodes[relationship.target]}"
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --- so are the tables a person reviews ----------------------------------------------
|
||||
|
||||
|
||||
def _cell(text: str | None) -> str:
|
||||
return " ".join((text or "").replace("|", "\\|").split())
|
||||
|
||||
|
||||
def _table(
|
||||
heading: str, columns: Sequence[str], rows: Iterable[Sequence[str | None]]
|
||||
) -> str:
|
||||
lines = [
|
||||
f"## {heading}",
|
||||
"",
|
||||
"| " + " | ".join(columns) + " |",
|
||||
"|" + "|".join(" --- " for _ in columns) + "|",
|
||||
]
|
||||
lines += ["| " + " | ".join(_cell(value) for value in row) + " |" for row in rows]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_tables(graph: TargetArchitecture) -> str:
|
||||
"""The graph as markdown tables: what a person approving the design reads.
|
||||
|
||||
Written by code, so the tables cannot disagree with the graph and no model spends
|
||||
output repeating it.
|
||||
"""
|
||||
names = {element.id: element.name for element in graph.elements}
|
||||
parts = [
|
||||
_table(
|
||||
"Elements",
|
||||
("Id", "Name", "Product", "Resource type", "Component"),
|
||||
(
|
||||
(e.id, e.name, e.product, e.resource_type, e.component)
|
||||
for e in graph.elements
|
||||
),
|
||||
),
|
||||
_table(
|
||||
"Relationships",
|
||||
("Id", "From", "Kind", "To"),
|
||||
(
|
||||
(r.id, names[r.source], r.kind, names[r.target])
|
||||
for r in graph.relationships
|
||||
),
|
||||
),
|
||||
_table(
|
||||
"Checks",
|
||||
("Id", "Applies to", "Statement", "Source"),
|
||||
((c.id, c.applies_to, c.statement, c.source) for c in graph.checks),
|
||||
),
|
||||
]
|
||||
if graph.not_applicable:
|
||||
parts.append(
|
||||
_table(
|
||||
"Not applicable",
|
||||
("Recommendation", "Reason"),
|
||||
((n.recommendation, n.reason) for n in graph.not_applicable),
|
||||
)
|
||||
)
|
||||
return "\n\n".join(parts) + "\n"
|
||||
|
||||
@@ -13,7 +13,7 @@ An Execution is a hierarchy of ReAct agents shaped by the dependencies between D
|
||||
|
||||
The dependency graph is declared as data in code. The Orchestrator picks what to do next through tools that refuse work whose dependencies aren't met. Dependencies are guaranteed by code, while ordering and parallelism are the model's choice.
|
||||
|
||||
> **Under review (2026-10-07).** [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), proposed, would have the Orchestrator decide with Jev and run no language model. If accepted, it replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected.
|
||||
> **Amended (2026-10-07) by [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), accepted.** The Orchestrator is now a router: it decides by readings and a playbook in code, with Jev for the judgements, and runs no language model to choose an action. That replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected and still reason on language models.
|
||||
|
||||
## Considered Options
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: proposed
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Jev is the Orchestrator's decision point
|
||||
# Jev is the Orchestrator's decision point, and the Orchestrator is a router
|
||||
|
||||
[ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev ([Jev in AI agent loops](https://vercel.com/i/jev-agent-control)): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.
|
||||
|
||||
@@ -17,28 +17,53 @@ status: proposed
|
||||
| Update state | Application code | The Execution graph and its checkpoint |
|
||||
| Explain the result | A generative model | One small model call that writes an Approval summary or a question, from data it is given |
|
||||
|
||||
## How a turn is decided
|
||||
## The Orchestrator is a router
|
||||
|
||||
1. **Code lists the allowed actions** from the Ledger and the Execution's state: relay waiting questions, act on a validation outcome, request an Approval, delegate ready work, finish. This is the permission check, and it holds whatever Jev answers, because a typed decision can still be wrong.
|
||||
2. **Jev proposes the next action** from that list, and answers the other closed questions the turn needs, in one request where they rest on the same evidence:
|
||||
- Did the user's last message ask for a change to submitted work? (yes or no)
|
||||
- Which Deliverable does that change affect? (one of the Deliverables)
|
||||
- What should happen after this validation: revise the code, revise the Target architecture, or proceed? (one of three)
|
||||
A question whose options depend on another answer, or on evidence not yet fetched, is a second request.
|
||||
3. **Code carries the decision out** through the existing tools and records it: the questions, the answers, their probabilities and what was done. A resumed Execution reuses a recorded decision when its input is unchanged.
|
||||
4. **A generative model writes the reply**, only where the user reads prose: the summary they approve a Phase from, and any question the Orchestrator itself asks. It is given the decision and the data and decides nothing.
|
||||
With Jev as its decision point, the Orchestrator's job is routing: it reads what has arrived and the state of the Execution, asks Jev which route applies, and dispatches. It does no work of its own. There are four routes to choose.
|
||||
|
||||
Two guards from the pattern are kept in code. Repeated actions are limited, so the Orchestrator cannot loop on the same step: the turn limit and the Revision limit already do this. A failed call to Jev is recorded as a failure, distinct from an answer, so "Jev was unreachable" is never read as "nothing to do".
|
||||
| What is routed | The routes Jev chooses between | Who is dispatched |
|
||||
|---|---|---|
|
||||
| A message that starts something | a new Workflow request; a Change request against a Baseline; a question about work already delivered; not something this agent does | a full Execution; an Execution from the Baseline; the reply writer, from the delivered files; a polite refusal |
|
||||
| A message inside an Execution | an answer to a waiting question; a request to change submitted work, and which Deliverable; anything else | the Specialist that asked; a Revision of that Deliverable; a question back to the user |
|
||||
| A validation outcome | revise the code; revise the Target architecture; proceed | the IaC author; the target architect; the Approval |
|
||||
| Ready work, in a Change request | which Deliverables the change reaches, and which carry over from the Baseline unchanged | only the Specialists whose Deliverables are reached |
|
||||
|
||||
## Decisions to confirm
|
||||
Which Specialist produces a given Deliverable is still fixed by the dependency graph: Jev chooses the route, never the worker for it. And the Ledger still refuses a route whose Dependencies or Approvals are not met.
|
||||
|
||||
This ADR is proposed. Accepting it means agreeing to each of these:
|
||||
## How a turn is decided: readings, disposition, playbook
|
||||
|
||||
1. **No language model decides what the Orchestrator does.** Jev proposes and code disposes. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
|
||||
The owner's model for this is a four-stage pipeline: readings of what arrived, a disposition worked out in code, further readings of the state, and a playbook that takes the first group that applies and, within it, the strongest action whose conditions hold, escalating when no group applies.
|
||||
|
||||
1. **Readings.** Facts about the turn. Most are read from the Ledger and the Execution's data by code: are questions waiting, is a Phase complete, is work ready, which Checks failed, did a Check fail twice running. The ones that need judgement are typed answers from Jev: is the user asking for a change (yes or no), which Deliverable would have to change first (a choice), how specific are they (a score), does a gap in the Target architecture leave a requirement unmet (yes or no). Jev reads; it never picks the action.
|
||||
2. **Disposition.** Code turns a judged reading into act, queue or close by threshold: a probability of 0.75 or more is acted on, 0.15 or less is dropped, and anything between is put to the user. A choice with confidence under 0.6 is not relied on. The thresholds are settings.
|
||||
3. **Playbook.** Ordered groups; the first that applies is taken, and within it the strongest action whose conditions hold:
|
||||
|
||||
| Group | Actions, strongest first |
|
||||
|---|---|
|
||||
| The user asked for a change | revise the Deliverable it affects; else ask them to confirm which and what |
|
||||
| Questions are waiting | relay them |
|
||||
| Validation decided | revise the Target architecture (a Check failed twice running, or a gap that matters); else revise the code (a Check failed); else ask about the gaps; else proceed |
|
||||
| A Phase is complete | request its Approval |
|
||||
| Work is ready | delegate all of it |
|
||||
| Packaging is approved | finish |
|
||||
| No group applies | escalate: ask the user, or end with warnings when there is none |
|
||||
|
||||
4. **Permission.** Every action goes through the Orchestrator's tools, so the Ledger refuses what is not allowed whatever was decided, and the playbook then takes the next action that is. A typed answer can still be wrong.
|
||||
5. **Record.** Each decision is kept in the Execution's state: the turn, the group, the action and what Jev answered with its confidence.
|
||||
6. **Reply.** A generative model writes only what the user reads, the summary they approve a Phase from, out of data it is given. It runs after the decision and cannot change it. When it fails, the summary is assembled from the same data.
|
||||
|
||||
A failed call to Jev is not an answer. The client returns nothing, and the reading falls back to a rule that acts only on what is certain: a change is acted on when the Deliverable is named outright, and a gap is revised for when it names a real requirement or cannot be verified.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **No language model decides what the Orchestrator does.** Jev reads, and the playbook in code decides. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
|
||||
2. **Low confidence goes to the user, not to another model.** Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
|
||||
3. **Jev being unreachable does not stop an Execution.** After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
|
||||
4. **What is sent to Jev is the Execution's state, not its content:** Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
|
||||
5. **Jev does not choose Specialists.** Which Specialist produces a Deliverable is fixed by the dependency graph, and all ready work is delegated together. The pattern's specialist routing has nothing to route here.
|
||||
5. **The Orchestrator routes; Jev chooses the route.** That covers what kind of work a message starts, where a message inside an Execution goes, where a validation outcome goes, and, in a Change request, which Deliverables are reached. Jev never chooses which Specialist produces a Deliverable: the dependency graph fixes that.
|
||||
6. **In a new Execution every Deliverable is still produced.** Routing does not skip a Specialist for a new Workflow request, because each later Deliverable is checked against the ones before it. Skipping happens only in a Change request, where the Baseline supplies what is carried over.
|
||||
|
||||
## Considered Options
|
||||
|
||||
@@ -49,6 +74,9 @@ This ADR is proposed. Accepting it means agreeing to each of these:
|
||||
|
||||
## Consequences
|
||||
|
||||
- Jev is reached through TypeSafe AI's Python SDK, `typesafe-sdk`, pinned exactly (owner, 2026-10-07). `app/tools/jev.py` is the only module that imports it.
|
||||
- The Orchestrator as a language-model agent is kept behind the setting `ORCHESTRATOR=model`, for the tests that drive the Ledger's rules in arbitrary order and for comparing the two in an evaluation. The default is `router`. Remove the setting once the router is proven.
|
||||
|
||||
- The Orchestrator's instructions in `agents/orchestrator.md` become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.
|
||||
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
|
||||
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
"""Judges: graded answers about a Deliverable that string matching cannot give.
|
||||
|
||||
A judge takes a document and a set of questions and returns a score from 0 to 1 per
|
||||
question. The shape (one state, many typed questions) is deliberately the one a
|
||||
decision model takes, so the Gemini judge here can be swapped for one later.
|
||||
question. The shape is the one a decision model takes: one state, many typed
|
||||
questions, answered in a single call. `DeciderJudge` puts Jev in that seat, which is
|
||||
the judge to prefer for narrow, typed criteria at volume: it is cheap enough to ask
|
||||
every question of every run and returns the same answer for the same input.
|
||||
`GeminiJudge` is a language-model judge for comparison, and for when no Jev key is set.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -11,6 +14,8 @@ import json
|
||||
from collections.abc import Mapping
|
||||
from typing import Any, Protocol
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
|
||||
INSTRUCTIONS = (
|
||||
"You grade a document from a cloud solution architecture engagement. Answer every "
|
||||
"question about the document only, strictly and without benefit of the doubt. "
|
||||
@@ -31,6 +36,48 @@ class Judge(Protocol):
|
||||
...
|
||||
|
||||
|
||||
GRADES = ("not at all", "weakly", "mostly", "fully")
|
||||
|
||||
|
||||
class JudgeUnavailable(RuntimeError):
|
||||
"""The judge gave no answer; the check it serves is unanswered, not failed."""
|
||||
|
||||
|
||||
class DeciderJudge:
|
||||
"""Jev as the judge. The document is the state and carries no grading
|
||||
instructions; each question is asked on its own terms, all in one call."""
|
||||
|
||||
def __init__(self, decider: Decider, scope: str = "evaluation") -> None:
|
||||
self._decider = decider
|
||||
self._scope = scope
|
||||
|
||||
async def answer(
|
||||
self, document: str, questions: Mapping[str, Mapping[str, str]]
|
||||
) -> Mapping[str, float]:
|
||||
asked = {
|
||||
key: Question("score", item["text"], GRADES)
|
||||
if item["kind"] == "graded"
|
||||
else Question("noul", item["text"])
|
||||
for key, item in questions.items()
|
||||
}
|
||||
answers = await self._decider.decide(document, asked, scope=self._scope)
|
||||
if answers is None:
|
||||
raise JudgeUnavailable("Jev gave no answer")
|
||||
scores = {}
|
||||
for key, question in asked.items():
|
||||
value = float(answers[key].value)
|
||||
# A graded answer is a level from 0 to 3; a yes-or-no answer is the
|
||||
# probability of yes, counted as a yes at an even chance or better.
|
||||
scores[key] = (
|
||||
min(1.0, max(0.0, value / (len(GRADES) - 1)))
|
||||
if question.kind == "score"
|
||||
else 1.0
|
||||
if value >= 0.5
|
||||
else 0.0
|
||||
)
|
||||
return scores
|
||||
|
||||
|
||||
class GeminiJudge:
|
||||
"""A judge on the Gemini API or Vertex AI, chosen by the environment as ADK does."""
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ from app.workflows.target_architecture import (
|
||||
parse_graph,
|
||||
render_mermaid,
|
||||
)
|
||||
from eval.judge import Judge
|
||||
from eval.judge import Judge, JudgeUnavailable
|
||||
|
||||
PASS_THRESHOLD = 0.8
|
||||
CHECKLIST_COVERAGE = 0.7 # share of a pillar Skill's checklist the design must address
|
||||
@@ -403,7 +403,10 @@ async def _checklist_coverage(
|
||||
}
|
||||
if not recommendations or not questions:
|
||||
return Check(name, False, "no design recommendations or no checklists")
|
||||
scores = await judge.answer(recommendations, questions)
|
||||
try:
|
||||
scores = await judge.answer(recommendations, questions)
|
||||
except JudgeUnavailable as exc:
|
||||
return Check(name, False, f"judge unavailable: {exc}")
|
||||
per_pillar = {
|
||||
pillar: sum(scores.get(f"{pillar}.{i}", 0) for i in range(len(items)))
|
||||
/ len(items)
|
||||
@@ -419,10 +422,16 @@ async def _quality(judge: Judge, recommendations: str) -> Check:
|
||||
name = "recommendations_quality"
|
||||
if not recommendations:
|
||||
return Check(name, False, "no design recommendations")
|
||||
scores = await judge.answer(
|
||||
recommendations,
|
||||
{key: {"kind": "graded", "text": text} for key, text in QUALITY_RUBRIC.items()},
|
||||
)
|
||||
try:
|
||||
scores = await judge.answer(
|
||||
recommendations,
|
||||
{
|
||||
key: {"kind": "graded", "text": text}
|
||||
for key, text in QUALITY_RUBRIC.items()
|
||||
},
|
||||
)
|
||||
except JudgeUnavailable as exc:
|
||||
return Check(name, False, f"judge unavailable: {exc}")
|
||||
mean = sum(scores.values()) / len(QUALITY_RUBRIC)
|
||||
detail = ", ".join(f"{key} {scores.get(key, 0):.2f}" for key in QUALITY_RUBRIC)
|
||||
return Check(name, mean >= QUALITY_BAR, f"mean {mean:.2f}: {detail}")
|
||||
|
||||
@@ -8,8 +8,9 @@ Usage:
|
||||
GOOGLE_GENAI_USE_VERTEXAI=true GOOGLE_CLOUD_PROJECT=<project> GOOGLE_CLOUD_LOCATION=global \
|
||||
.venv/bin/python -m eval.solution_eval [--judge] [case-id ...]
|
||||
|
||||
--judge adds the judged checks (pillar checklist coverage, recommendation quality),
|
||||
using the provider bundle's default model unless EVAL_JUDGE_MODEL names another.
|
||||
--judge adds the judged checks (pillar checklist coverage, recommendation quality).
|
||||
The judge is Jev when TYPESAFE_API_KEY is set; otherwise, or with EVAL_JUDGE=gemini,
|
||||
a language model: the provider bundle's default unless EVAL_JUDGE_MODEL names another.
|
||||
A case marked "interactive" runs with a scripted user in the loop. The exit code is
|
||||
non-zero when a case fails or exceeds one of its thresholds.
|
||||
"""
|
||||
@@ -36,7 +37,7 @@ from app.nodes.execution_nodes import automatic_reply
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.states.execution_state import execution_dir
|
||||
from app.tools.local import TerraformRunner
|
||||
from eval.judge import GeminiJudge, Judge
|
||||
from eval.judge import DeciderJudge, GeminiJudge, Judge
|
||||
from eval.scoring import (
|
||||
CaseResult,
|
||||
CaseSpec,
|
||||
@@ -209,6 +210,8 @@ async def run_case(
|
||||
"agent_runs": dict(runtime.runs),
|
||||
"outcomes": dict(runtime.outcomes),
|
||||
"revisions": revisions,
|
||||
# Every routing decision: the group that applied, the action, what was judged.
|
||||
"decisions": interaction.state.get("decisions", []),
|
||||
"warnings": interaction.state.get("warnings", []),
|
||||
"conversation": interaction.state.get("conversation", []),
|
||||
"output_dir": str(execution_dir(run_dir, interaction.execution_id, case.id)),
|
||||
@@ -217,8 +220,13 @@ async def run_case(
|
||||
|
||||
|
||||
def make_judge(settings: Settings) -> Judge:
|
||||
"""Jev when a TypeSafe key is set, unless EVAL_JUDGE=gemini asks for the
|
||||
language-model judge; otherwise the language-model judge."""
|
||||
from app.agent import open_decider
|
||||
from app.providers import load_provider
|
||||
|
||||
if settings.TYPESAFE_API_KEY and os.environ.get("EVAL_JUDGE") != "gemini":
|
||||
return DeciderJudge(open_decider(settings))
|
||||
model = os.environ.get("EVAL_JUDGE_MODEL") or load_provider(
|
||||
settings.CLOUD_PROVIDER
|
||||
).models.default.removeprefix(GEMINI_PREFIX)
|
||||
|
||||
@@ -21,6 +21,13 @@ models:
|
||||
target-architect: gemini/gemini-3.1-pro-preview
|
||||
iac-author: gemini/gemini-3.1-pro-preview
|
||||
validator: gemini/gemini-3.1-pro-preview
|
||||
# Seconds one model request may take, for roles whose replies are long. Other
|
||||
# roles get MODEL_TIMEOUT_SECONDS. The target architect timed out at 180 s in both
|
||||
# attempts of the 2026-10-07 run.
|
||||
timeout_seconds:
|
||||
target-architect: 420
|
||||
iac-author: 300
|
||||
packager: 300
|
||||
|
||||
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
|
||||
role_skills:
|
||||
@@ -29,9 +36,9 @@ role_skills:
|
||||
decomposer: [google-cloud-solution-architecture, requirements-chain]
|
||||
product-selector: [google-cloud-solution-architecture, requirements-chain]
|
||||
design-advisor: [google-cloud-solution-architecture, requirements-chain]
|
||||
target-architect: [google-cloud-solution-architecture, requirements-chain]
|
||||
describer: [google-cloud-solution-architecture]
|
||||
iac-author: [google-cloud-solution-architecture, requirements-chain, iac-validation]
|
||||
target-architect: [google-cloud-solution-architecture, requirements-chain]
|
||||
iac-author: [google-cloud-solution-architecture, requirements-chain, iac-validation, google-cloud-iac-patterns]
|
||||
validator: [google-cloud-solution-architecture, requirements-chain, iac-validation]
|
||||
packager: [google-cloud-solution-architecture]
|
||||
|
||||
@@ -45,9 +52,18 @@ pillar_skills:
|
||||
sustainability: google-cloud-waf-sustainability
|
||||
|
||||
tools:
|
||||
general: [mermaid_check, terraform_fmt, terraform_validate]
|
||||
general: [mermaid_check, terraform_fmt, terraform_validate, terraform_schema]
|
||||
cloud_read: [terraform_plan]
|
||||
|
||||
# The Terraform providers generated code is written for. One exact version, so the
|
||||
# schema lookup, the patterns Skill and validation all describe the same provider.
|
||||
# The 2026-10-07 run left five different versions in the plugin cache.
|
||||
terraform:
|
||||
required_providers:
|
||||
google:
|
||||
source: hashicorp/google
|
||||
version: 8.6.0
|
||||
|
||||
mcp_auth:
|
||||
kind: google_adc # Application Default Credentials bearer token; sends X-Goog-User-Project
|
||||
|
||||
|
||||
@@ -14,3 +14,4 @@ psycopg2-binary>=2.9.9
|
||||
google-adk[extensions]==2.9.0 # Skills are Experimental: pin exactly; extensions brings LiteLLM
|
||||
mcp>=1.24,<2
|
||||
a2a-sdk[http-server]>=1.1,<2
|
||||
typesafe-sdk==0.7.2 # Jev decisions (ADR-0007); a young SDK, so pinned exactly
|
||||
|
||||
49
skills/google-cloud-iac-patterns/SKILL.md
Normal file
49
skills/google-cloud-iac-patterns/SKILL.md
Normal file
@@ -0,0 +1,49 @@
|
||||
---
|
||||
name: google-cloud-iac-patterns
|
||||
description: >-
|
||||
Terraform patterns for Google Cloud products that validate against one pinned
|
||||
version of the hashicorp/google provider: Cloud Run services and jobs,
|
||||
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
|
||||
networking, Cloud Scheduler, Artifact Registry, Monitoring and API
|
||||
enablement. Use when writing Terraform for a Google Cloud product, before
|
||||
writing the resource, to start from a pattern whose arguments are known to be
|
||||
in the right place.
|
||||
metadata:
|
||||
scope: deployment guidance
|
||||
provider: hashicorp/google 8.6.0
|
||||
---
|
||||
|
||||
# Google Cloud IaC patterns
|
||||
|
||||
Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0. A test runs that on each of them, so an argument you copy from a pattern is in the right block.
|
||||
|
||||
## How to use them
|
||||
|
||||
1. Write `assets/versions.tf` to your working copy as `terraform/versions.tf`, unchanged. It pins the provider version these patterns and the `terraform_schema` tool describe. Declare the `project_id` and `region` variables it uses in `terraform/variables.tf`.
|
||||
2. For each Element in the Target architecture, read the reference for its product with `load_skill_resource` and start from its pattern. Read only the references for products the Target architecture uses.
|
||||
3. Change the pattern to meet the Element's Checks. For a setting the pattern does not show, call `terraform_schema` with `find` set to its name before you write it.
|
||||
|
||||
| Product | Reference |
|
||||
| --- | --- |
|
||||
| Enabling APIs | `references/project-services.md` |
|
||||
| Service accounts and IAM grants | `references/iam.md` |
|
||||
| Cloud Run service | `references/cloud-run-service.md` |
|
||||
| Cloud Run job | `references/cloud-run-job.md` |
|
||||
| Pub/Sub | `references/pubsub.md` |
|
||||
| Cloud Storage | `references/cloud-storage.md` |
|
||||
| Secret Manager | `references/secret-manager.md` |
|
||||
| Cloud KMS | `references/kms.md` |
|
||||
| Cloud SQL | `references/cloud-sql.md` |
|
||||
| VPC networking | `references/networking.md` |
|
||||
| Cloud Scheduler | `references/cloud-scheduler.md` |
|
||||
| Artifact Registry | `references/artifact-registry.md` |
|
||||
| Cloud Monitoring and Logging | `references/monitoring.md` |
|
||||
|
||||
A product with no reference has no pattern yet: write it from `terraform_schema` and the product documentation.
|
||||
|
||||
## Rules that hold for every product
|
||||
|
||||
- A Cloud Run service and a Cloud Run job are different resource types with different arguments. A setting of one is not a setting of the other. `references/cloud-run-job.md` lists what a job lacks.
|
||||
- Grant a role with an `_iam_member` resource. An `_iam_binding` or `_iam_policy` resource removes grants made elsewhere.
|
||||
- Give a Google-managed service agent its role before the resource that needs it, with `depends_on`. A key, a dead-letter topic and a push subscription all need one.
|
||||
- Set `deletion_protection` from a variable on the resources that have it, so a test deployment can be removed.
|
||||
15
skills/google-cloud-iac-patterns/assets/versions.tf
Normal file
15
skills/google-cloud-iac-patterns/assets/versions.tf
Normal file
@@ -0,0 +1,15 @@
|
||||
terraform {
|
||||
required_version = ">= 1.9"
|
||||
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "8.6.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
# Artifact Registry
|
||||
|
||||
A Docker repository with a cleanup policy, and the grant that lets a runtime identity pull from it.
|
||||
|
||||
- `format = "DOCKER"` for container images. The image path is `REGION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE`.
|
||||
- Cleanup policies delete nothing while `cleanup_policy_dry_run` is `true`.
|
||||
- `docker_config.immutable_tags = true` stops a tag being moved to a different image.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
resource "google_service_account" "runtime" {
|
||||
project = var.project_id
|
||||
account_id = "image-puller"
|
||||
display_name = "Pulls workload images"
|
||||
}
|
||||
|
||||
resource "google_artifact_registry_repository" "images" {
|
||||
project = var.project_id
|
||||
location = var.region
|
||||
repository_id = "workload-images"
|
||||
format = "DOCKER"
|
||||
description = "Container images for the workload."
|
||||
cleanup_policy_dry_run = false
|
||||
|
||||
docker_config {
|
||||
immutable_tags = true
|
||||
}
|
||||
|
||||
cleanup_policies {
|
||||
id = "keep-recent"
|
||||
action = "KEEP"
|
||||
|
||||
most_recent_versions {
|
||||
keep_count = 10
|
||||
}
|
||||
}
|
||||
|
||||
cleanup_policies {
|
||||
id = "delete-old-untagged"
|
||||
action = "DELETE"
|
||||
|
||||
condition {
|
||||
tag_state = "UNTAGGED"
|
||||
older_than = "2592000s"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_artifact_registry_repository_iam_member" "runtime_pulls" {
|
||||
project = var.project_id
|
||||
location = google_artifact_registry_repository.images.location
|
||||
repository = google_artifact_registry_repository.images.name
|
||||
role = "roles/artifactregistry.reader"
|
||||
member = google_service_account.runtime.member
|
||||
}
|
||||
```
|
||||
95
skills/google-cloud-iac-patterns/references/cloud-run-job.md
Normal file
95
skills/google-cloud-iac-patterns/references/cloud-run-job.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# Cloud Run job
|
||||
|
||||
`google_cloud_run_v2_job`: a task that runs to completion. It is a different resource type from a service, and its template is nested one level deeper.
|
||||
|
||||
What a job does not have. A Check that asks for one of these on a job cannot be met:
|
||||
|
||||
- `startup_cpu_boost` and `cpu_idle`. A job's `resources` block has `limits` only.
|
||||
- `ingress`, `traffic`, `scaling`, `min_instance_count`, `max_instance_request_concurrency`. A job answers no requests.
|
||||
- `liveness_probe`. A job's container has a `startup_probe` only.
|
||||
|
||||
Where a job's settings sit:
|
||||
|
||||
- `task_count` and `parallelism` are in `template`.
|
||||
- `service_account`, `timeout`, `max_retries`, `execution_environment`, `vpc_access` and `encryption_key` are in `template.template`.
|
||||
- `image`, `env`, `args`, `command` and `resources` are in `template.template.containers`.
|
||||
- Whoever starts the job needs `roles/run.invoker` on it; Cloud Scheduler starts it with an HTTP call to the `:run` method.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "image" {
|
||||
type = string
|
||||
description = "Container image reference, including the tag or digest."
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
type = bool
|
||||
description = "Whether Terraform is prevented from deleting the job."
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "google_service_account" "replay" {
|
||||
project = var.project_id
|
||||
account_id = "replay-job"
|
||||
display_name = "Replay job"
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_job" "replay" {
|
||||
project = var.project_id
|
||||
name = "replay"
|
||||
location = var.region
|
||||
deletion_protection = var.deletion_protection
|
||||
|
||||
template {
|
||||
task_count = 1
|
||||
parallelism = 1
|
||||
|
||||
template {
|
||||
service_account = google_service_account.replay.email
|
||||
timeout = "3600s"
|
||||
max_retries = 3
|
||||
execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
|
||||
|
||||
containers {
|
||||
image = var.image
|
||||
args = ["--mode=replay"]
|
||||
|
||||
resources {
|
||||
limits = {
|
||||
cpu = "1"
|
||||
memory = "1Gi"
|
||||
}
|
||||
}
|
||||
|
||||
env {
|
||||
name = "BATCH_SIZE"
|
||||
value = "100"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_service_account" "operator" {
|
||||
project = var.project_id
|
||||
account_id = "replay-operator"
|
||||
display_name = "Starts the replay job"
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_job_iam_member" "operator_runs_replay" {
|
||||
project = var.project_id
|
||||
location = google_cloud_run_v2_job.replay.location
|
||||
name = google_cloud_run_v2_job.replay.name
|
||||
role = "roles/run.invoker"
|
||||
member = google_service_account.operator.member
|
||||
}
|
||||
```
|
||||
144
skills/google-cloud-iac-patterns/references/cloud-run-service.md
Normal file
144
skills/google-cloud-iac-patterns/references/cloud-run-service.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Cloud Run service
|
||||
|
||||
`google_cloud_run_v2_service`: a long-running service that answers requests.
|
||||
|
||||
Where the settings sit, which is what goes wrong most often:
|
||||
|
||||
- `ingress`, `deletion_protection` and `traffic` are on the resource.
|
||||
- `service_account`, `scaling`, `timeout`, `max_instance_request_concurrency`, `execution_environment`, `vpc_access` and `encryption_key` are in `template`.
|
||||
- `image`, `env`, `ports`, `resources` and the probes are in `template.containers`.
|
||||
- `cpu_idle` and `startup_cpu_boost` are in `template.containers.resources`, beside `limits`. They exist on a service only.
|
||||
- `min_instance_count` and `max_instance_count` are in `template.scaling`.
|
||||
- A secret is read through `env.value_source.secret_key_ref`; the service account needs `roles/secretmanager.secretAccessor` on that secret.
|
||||
- Direct VPC egress is `template.vpc_access.network_interfaces`. A connector is `template.vpc_access.connector`. Use one of them.
|
||||
- Public access is a grant of `roles/run.invoker` to `allUsers` together with `ingress = "INGRESS_TRAFFIC_ALL"`. A private service has neither.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "image" {
|
||||
type = string
|
||||
description = "Container image reference, including the tag or digest."
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
type = bool
|
||||
description = "Whether Terraform is prevented from deleting the service."
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "google_service_account" "ingest" {
|
||||
project = var.project_id
|
||||
account_id = "ingest-service"
|
||||
display_name = "Ingest service"
|
||||
}
|
||||
|
||||
resource "google_secret_manager_secret" "signing_key" {
|
||||
project = var.project_id
|
||||
secret_id = "webhook-signing-key"
|
||||
|
||||
replication {
|
||||
auto {}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_secret_manager_secret_iam_member" "ingest_reads_signing_key" {
|
||||
project = var.project_id
|
||||
secret_id = google_secret_manager_secret.signing_key.secret_id
|
||||
role = "roles/secretmanager.secretAccessor"
|
||||
member = google_service_account.ingest.member
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_service" "ingest" {
|
||||
project = var.project_id
|
||||
name = "ingest"
|
||||
location = var.region
|
||||
ingress = "INGRESS_TRAFFIC_ALL"
|
||||
deletion_protection = var.deletion_protection
|
||||
|
||||
template {
|
||||
service_account = google_service_account.ingest.email
|
||||
timeout = "30s"
|
||||
max_instance_request_concurrency = 80
|
||||
execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
|
||||
|
||||
scaling {
|
||||
min_instance_count = 1
|
||||
max_instance_count = 10
|
||||
}
|
||||
|
||||
containers {
|
||||
image = var.image
|
||||
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
|
||||
resources {
|
||||
limits = {
|
||||
cpu = "1"
|
||||
memory = "512Mi"
|
||||
}
|
||||
cpu_idle = true
|
||||
startup_cpu_boost = true
|
||||
}
|
||||
|
||||
env {
|
||||
name = "LOG_LEVEL"
|
||||
value = "info"
|
||||
}
|
||||
|
||||
env {
|
||||
name = "SIGNING_KEY"
|
||||
value_source {
|
||||
secret_key_ref {
|
||||
secret = google_secret_manager_secret.signing_key.secret_id
|
||||
version = "latest"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
startup_probe {
|
||||
initial_delay_seconds = 0
|
||||
period_seconds = 5
|
||||
failure_threshold = 6
|
||||
http_get {
|
||||
path = "/healthz"
|
||||
}
|
||||
}
|
||||
|
||||
liveness_probe {
|
||||
period_seconds = 30
|
||||
http_get {
|
||||
path = "/healthz"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
traffic {
|
||||
type = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
|
||||
percent = 100
|
||||
}
|
||||
|
||||
depends_on = [google_secret_manager_secret_iam_member.ingest_reads_signing_key]
|
||||
}
|
||||
|
||||
# Public access. Leave this out, and set ingress to INGRESS_TRAFFIC_INTERNAL_ONLY,
|
||||
# for a service only other workloads call.
|
||||
resource "google_cloud_run_v2_service_iam_member" "public" {
|
||||
project = var.project_id
|
||||
location = google_cloud_run_v2_service.ingest.location
|
||||
name = google_cloud_run_v2_service.ingest.name
|
||||
role = "roles/run.invoker"
|
||||
member = "allUsers"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
# Cloud Scheduler
|
||||
|
||||
A schedule that starts a Cloud Run job, and one that calls a private Cloud Run service.
|
||||
|
||||
- A job is started by an HTTP POST to its `:run` method with an OAuth token (`oauth_token`). A service is called with an OIDC token (`oidc_token`). Using the wrong one is refused at run time, not by `terraform validate`.
|
||||
- The schedule's service account needs `roles/run.invoker` on the job or service it starts.
|
||||
- `schedule` is unix-cron; `time_zone` is an IANA name.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "job_name" {
|
||||
type = string
|
||||
description = "Name of the Cloud Run job to start."
|
||||
}
|
||||
|
||||
variable "service_url" {
|
||||
type = string
|
||||
description = "HTTPS URL of the private Cloud Run service to call."
|
||||
}
|
||||
|
||||
resource "google_service_account" "scheduler" {
|
||||
project = var.project_id
|
||||
account_id = "nightly-scheduler"
|
||||
display_name = "Identity of the nightly schedules"
|
||||
}
|
||||
|
||||
resource "google_cloud_scheduler_job" "start_job" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
name = "start-nightly-job"
|
||||
schedule = "0 2 * * *"
|
||||
time_zone = "Etc/UTC"
|
||||
attempt_deadline = "180s"
|
||||
|
||||
retry_config {
|
||||
retry_count = 3
|
||||
}
|
||||
|
||||
http_target {
|
||||
http_method = "POST"
|
||||
uri = "https://run.googleapis.com/v2/projects/${var.project_id}/locations/${var.region}/jobs/${var.job_name}:run"
|
||||
|
||||
oauth_token {
|
||||
service_account_email = google_service_account.scheduler.email
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_scheduler_job" "call_service" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
name = "call-nightly-service"
|
||||
schedule = "30 2 * * *"
|
||||
time_zone = "Etc/UTC"
|
||||
|
||||
http_target {
|
||||
http_method = "POST"
|
||||
uri = "${var.service_url}/tasks/nightly"
|
||||
|
||||
oidc_token {
|
||||
service_account_email = google_service_account.scheduler.email
|
||||
audience = var.service_url
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
124
skills/google-cloud-iac-patterns/references/cloud-sql.md
Normal file
124
skills/google-cloud-iac-patterns/references/cloud-sql.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# Cloud SQL
|
||||
|
||||
A PostgreSQL instance with a private address, backups and IAM database authentication.
|
||||
|
||||
- A private address needs a reserved range and a service networking connection on the VPC first; the instance must `depends_on` the connection.
|
||||
- `availability_type = "REGIONAL"` is high availability; `"ZONAL"` is a single zone.
|
||||
- Backups, point-in-time recovery, flags and the address are all in `settings`.
|
||||
- `deletion_protection` on the resource guards Terraform; `settings.deletion_protection_enabled` guards the API.
|
||||
- Take the database password from Secret Manager at run time, or use IAM database authentication as here. Never put a password in the configuration.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "deletion_protection" {
|
||||
type = bool
|
||||
description = "Whether Terraform is prevented from deleting the instance."
|
||||
default = true
|
||||
}
|
||||
|
||||
resource "google_compute_network" "main" {
|
||||
project = var.project_id
|
||||
name = "main"
|
||||
auto_create_subnetworks = false
|
||||
}
|
||||
|
||||
resource "google_compute_global_address" "private_services" {
|
||||
project = var.project_id
|
||||
name = "private-services"
|
||||
purpose = "VPC_PEERING"
|
||||
address_type = "INTERNAL"
|
||||
prefix_length = 16
|
||||
network = google_compute_network.main.id
|
||||
}
|
||||
|
||||
resource "google_service_networking_connection" "private_services" {
|
||||
network = google_compute_network.main.id
|
||||
service = "servicenetworking.googleapis.com"
|
||||
reserved_peering_ranges = [google_compute_global_address.private_services.name]
|
||||
}
|
||||
|
||||
resource "google_service_account" "app" {
|
||||
project = var.project_id
|
||||
account_id = "orders-app"
|
||||
display_name = "Orders application"
|
||||
}
|
||||
|
||||
resource "google_sql_database_instance" "orders" {
|
||||
project = var.project_id
|
||||
name = "orders"
|
||||
region = var.region
|
||||
database_version = "POSTGRES_16"
|
||||
deletion_protection = var.deletion_protection
|
||||
|
||||
settings {
|
||||
tier = "db-custom-2-7680"
|
||||
edition = "ENTERPRISE"
|
||||
availability_type = "REGIONAL"
|
||||
disk_type = "PD_SSD"
|
||||
disk_autoresize = true
|
||||
deletion_protection_enabled = var.deletion_protection
|
||||
|
||||
ip_configuration {
|
||||
ipv4_enabled = false
|
||||
private_network = google_compute_network.main.id
|
||||
ssl_mode = "ENCRYPTED_ONLY"
|
||||
}
|
||||
|
||||
backup_configuration {
|
||||
enabled = true
|
||||
point_in_time_recovery_enabled = true
|
||||
start_time = "02:00"
|
||||
|
||||
backup_retention_settings {
|
||||
retained_backups = 14
|
||||
}
|
||||
}
|
||||
|
||||
database_flags {
|
||||
name = "cloudsql.iam_authentication"
|
||||
value = "on"
|
||||
}
|
||||
|
||||
maintenance_window {
|
||||
day = 7
|
||||
hour = 3
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [google_service_networking_connection.private_services]
|
||||
}
|
||||
|
||||
resource "google_sql_database" "orders" {
|
||||
project = var.project_id
|
||||
name = "orders"
|
||||
instance = google_sql_database_instance.orders.name
|
||||
}
|
||||
|
||||
resource "google_sql_user" "app" {
|
||||
project = var.project_id
|
||||
name = trimsuffix(google_service_account.app.email, ".gserviceaccount.com")
|
||||
instance = google_sql_database_instance.orders.name
|
||||
type = "CLOUD_IAM_SERVICE_ACCOUNT"
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "app_connects" {
|
||||
project = var.project_id
|
||||
role = "roles/cloudsql.client"
|
||||
member = google_service_account.app.member
|
||||
}
|
||||
|
||||
resource "google_project_iam_member" "app_logs_in" {
|
||||
project = var.project_id
|
||||
role = "roles/cloudsql.instanceUser"
|
||||
member = google_service_account.app.member
|
||||
}
|
||||
```
|
||||
67
skills/google-cloud-iac-patterns/references/cloud-storage.md
Normal file
67
skills/google-cloud-iac-patterns/references/cloud-storage.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Cloud Storage
|
||||
|
||||
A private bucket with a retention period and a lifecycle.
|
||||
|
||||
- Bucket names are global. Build the name from the project id.
|
||||
- `uniform_bucket_level_access = true` and `public_access_prevention = "enforced"` keep the bucket private.
|
||||
- `retention_policy.retention_period` is in seconds and stops objects being deleted before then. A lifecycle `Delete` rule with `age` in days removes them after. Retaining for 30 days and then deleting needs both.
|
||||
- Versioning and a retention policy cannot be used on the same bucket.
|
||||
- Customer-managed encryption is `encryption.default_kms_key_name`; the Cloud Storage service agent needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key first.
|
||||
- `force_destroy = false` stops Terraform deleting a bucket that holds objects.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
resource "google_service_account" "writer" {
|
||||
project = var.project_id
|
||||
account_id = "raw-events-writer"
|
||||
display_name = "Writes raw events"
|
||||
}
|
||||
|
||||
resource "google_storage_bucket" "raw_events" {
|
||||
project = var.project_id
|
||||
name = "${var.project_id}-raw-events"
|
||||
location = var.region
|
||||
storage_class = "STANDARD"
|
||||
uniform_bucket_level_access = true
|
||||
public_access_prevention = "enforced"
|
||||
force_destroy = false
|
||||
|
||||
retention_policy {
|
||||
retention_period = 2592000
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 31
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 7
|
||||
}
|
||||
action {
|
||||
type = "SetStorageClass"
|
||||
storage_class = "NEARLINE"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_storage_bucket_iam_member" "writer_creates_objects" {
|
||||
bucket = google_storage_bucket.raw_events.name
|
||||
role = "roles/storage.objectCreator"
|
||||
member = google_service_account.writer.member
|
||||
}
|
||||
```
|
||||
40
skills/google-cloud-iac-patterns/references/iam.md
Normal file
40
skills/google-cloud-iac-patterns/references/iam.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# Service accounts and IAM grants
|
||||
|
||||
One service account per workload identity, and one `_iam_member` resource per grant.
|
||||
|
||||
- `account_id` is 6 to 30 characters: lower-case letters, digits and hyphens.
|
||||
- Use the narrowest resource the role can be granted on: the bucket, topic, secret or service, in preference to the project.
|
||||
- A grant to a service account is `member = "serviceAccount:${...email}"`. The same value is the `member` attribute of the service account resource.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
resource "google_service_account" "worker" {
|
||||
project = var.project_id
|
||||
account_id = "event-worker"
|
||||
display_name = "Event worker"
|
||||
description = "Identity of the worker service."
|
||||
}
|
||||
|
||||
# A project-level role, only when no narrower resource can take the grant.
|
||||
resource "google_project_iam_member" "worker_metrics" {
|
||||
project = var.project_id
|
||||
role = "roles/monitoring.metricWriter"
|
||||
member = google_service_account.worker.member
|
||||
}
|
||||
|
||||
# Who may act as the service account, for example a deployment pipeline.
|
||||
resource "google_service_account_iam_member" "deployer_acts_as_worker" {
|
||||
service_account_id = google_service_account.worker.name
|
||||
role = "roles/iam.serviceAccountUser"
|
||||
member = "serviceAccount:deployer@${var.project_id}.iam.gserviceaccount.com"
|
||||
}
|
||||
```
|
||||
79
skills/google-cloud-iac-patterns/references/kms.md
Normal file
79
skills/google-cloud-iac-patterns/references/kms.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# Cloud KMS
|
||||
|
||||
A key ring and a key for customer-managed encryption, and the grant that lets a product use it.
|
||||
|
||||
- The key ring must be in the same location as the resource it protects.
|
||||
- Key rings and keys cannot be deleted. `prevent_destroy` stops Terraform trying.
|
||||
- `rotation_period` is in seconds, such as `"7776000s"` for 90 days.
|
||||
- The product's service agent, not the workload's service account, needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key. The resource that uses the key must `depends_on` that grant.
|
||||
- Service agents: Pub/Sub is `service-NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com`, Cloud Run is `service-NUMBER@serverless-robot-prod.iam.gserviceaccount.com`, Cloud Storage comes from the `google_storage_project_service_account` data source.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
data "google_project" "this" {
|
||||
project_id = var.project_id
|
||||
}
|
||||
|
||||
data "google_storage_project_service_account" "this" {
|
||||
project = var.project_id
|
||||
}
|
||||
|
||||
resource "google_kms_key_ring" "workload" {
|
||||
project = var.project_id
|
||||
name = "workload"
|
||||
location = var.region
|
||||
}
|
||||
|
||||
resource "google_kms_crypto_key" "data" {
|
||||
name = "data"
|
||||
key_ring = google_kms_key_ring.workload.id
|
||||
purpose = "ENCRYPT_DECRYPT"
|
||||
rotation_period = "7776000s"
|
||||
|
||||
lifecycle {
|
||||
prevent_destroy = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_kms_crypto_key_iam_member" "pubsub_uses_key" {
|
||||
crypto_key_id = google_kms_crypto_key.data.id
|
||||
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
|
||||
member = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
|
||||
}
|
||||
|
||||
resource "google_kms_crypto_key_iam_member" "storage_uses_key" {
|
||||
crypto_key_id = google_kms_crypto_key.data.id
|
||||
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
|
||||
member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic" "encrypted" {
|
||||
project = var.project_id
|
||||
name = "encrypted-events"
|
||||
kms_key_name = google_kms_crypto_key.data.id
|
||||
|
||||
depends_on = [google_kms_crypto_key_iam_member.pubsub_uses_key]
|
||||
}
|
||||
|
||||
resource "google_storage_bucket" "encrypted" {
|
||||
project = var.project_id
|
||||
name = "${var.project_id}-encrypted"
|
||||
location = var.region
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
encryption {
|
||||
default_kms_key_name = google_kms_crypto_key.data.id
|
||||
}
|
||||
|
||||
depends_on = [google_kms_crypto_key_iam_member.storage_uses_key]
|
||||
}
|
||||
```
|
||||
107
skills/google-cloud-iac-patterns/references/monitoring.md
Normal file
107
skills/google-cloud-iac-patterns/references/monitoring.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# Cloud Monitoring and Logging
|
||||
|
||||
A notification channel, an alert on a metric, a log-based metric and a log sink.
|
||||
|
||||
- An alert policy needs `combiner` and at least one `conditions` block. A threshold condition's `filter` must name both a `metric.type` and a `resource.type`.
|
||||
- Every `aggregations` block needs an `alignment_period` of at least `"60s"` and a `per_series_aligner` that fits the metric kind: `ALIGN_RATE` or `ALIGN_DELTA` for a delta or cumulative metric, `ALIGN_MEAN` or `ALIGN_MAX` for a gauge.
|
||||
- A log-based metric's name is used in an alert filter as `logging.googleapis.com/user/NAME`.
|
||||
- A sink's writer identity needs a grant on the destination, or nothing arrives.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "alert_email" {
|
||||
type = string
|
||||
description = "Address alerts are sent to."
|
||||
}
|
||||
|
||||
resource "google_monitoring_notification_channel" "email" {
|
||||
project = var.project_id
|
||||
display_name = "Operations email"
|
||||
type = "email"
|
||||
|
||||
labels = {
|
||||
email_address = var.alert_email
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_monitoring_alert_policy" "server_errors" {
|
||||
project = var.project_id
|
||||
display_name = "Cloud Run 5xx responses"
|
||||
combiner = "OR"
|
||||
|
||||
conditions {
|
||||
display_name = "5xx rate above 1 per second for 5 minutes"
|
||||
|
||||
condition_threshold {
|
||||
filter = "resource.type = \"cloud_run_revision\" AND metric.type = \"run.googleapis.com/request_count\" AND metric.labels.response_code_class = \"5xx\""
|
||||
comparison = "COMPARISON_GT"
|
||||
threshold_value = 1
|
||||
duration = "300s"
|
||||
|
||||
aggregations {
|
||||
alignment_period = "60s"
|
||||
per_series_aligner = "ALIGN_RATE"
|
||||
cross_series_reducer = "REDUCE_SUM"
|
||||
group_by_fields = ["resource.labels.service_name"]
|
||||
}
|
||||
|
||||
trigger {
|
||||
count = 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
notification_channels = [google_monitoring_notification_channel.email.name]
|
||||
|
||||
alert_strategy {
|
||||
auto_close = "1800s"
|
||||
}
|
||||
|
||||
documentation {
|
||||
content = "The service is returning server errors. Check the latest revision's logs."
|
||||
mime_type = "text/markdown"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_logging_metric" "rejected_signatures" {
|
||||
project = var.project_id
|
||||
name = "rejected_signatures"
|
||||
filter = "resource.type = \"cloud_run_revision\" AND jsonPayload.event = \"signature_rejected\""
|
||||
|
||||
metric_descriptor {
|
||||
metric_kind = "DELTA"
|
||||
value_type = "INT64"
|
||||
unit = "1"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_storage_bucket" "audit_logs" {
|
||||
project = var.project_id
|
||||
name = "${var.project_id}-audit-logs"
|
||||
location = var.region
|
||||
uniform_bucket_level_access = true
|
||||
}
|
||||
|
||||
resource "google_logging_project_sink" "audit" {
|
||||
project = var.project_id
|
||||
name = "audit-to-storage"
|
||||
destination = "storage.googleapis.com/${google_storage_bucket.audit_logs.name}"
|
||||
filter = "logName:\"cloudaudit.googleapis.com\""
|
||||
unique_writer_identity = true
|
||||
}
|
||||
|
||||
resource "google_storage_bucket_iam_member" "sink_writes" {
|
||||
bucket = google_storage_bucket.audit_logs.name
|
||||
role = "roles/storage.objectCreator"
|
||||
member = google_logging_project_sink.audit.writer_identity
|
||||
}
|
||||
```
|
||||
102
skills/google-cloud-iac-patterns/references/networking.md
Normal file
102
skills/google-cloud-iac-patterns/references/networking.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# VPC networking
|
||||
|
||||
A custom-mode VPC, a subnet, outbound internet through Cloud NAT, and a firewall rule.
|
||||
|
||||
- `auto_create_subnetworks = false` gives a custom-mode network: only the subnets you declare.
|
||||
- `private_ip_google_access = true` lets workloads without external addresses reach Google APIs.
|
||||
- Cloud Run reaches the VPC with Direct VPC egress: `template.vpc_access.network_interfaces` naming this network and subnet. It needs no connector resource.
|
||||
- Firewall rules match by `target_service_accounts` or `target_tags`. A rule with neither applies to the whole network.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
resource "google_compute_network" "main" {
|
||||
project = var.project_id
|
||||
name = "main"
|
||||
auto_create_subnetworks = false
|
||||
routing_mode = "REGIONAL"
|
||||
}
|
||||
|
||||
resource "google_compute_subnetwork" "workloads" {
|
||||
project = var.project_id
|
||||
name = "workloads"
|
||||
region = var.region
|
||||
network = google_compute_network.main.id
|
||||
ip_cidr_range = "10.10.0.0/24"
|
||||
private_ip_google_access = true
|
||||
|
||||
log_config {
|
||||
aggregation_interval = "INTERVAL_5_SEC"
|
||||
flow_sampling = 0.5
|
||||
metadata = "INCLUDE_ALL_METADATA"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_router" "egress" {
|
||||
project = var.project_id
|
||||
name = "egress"
|
||||
region = var.region
|
||||
network = google_compute_network.main.id
|
||||
}
|
||||
|
||||
resource "google_compute_router_nat" "egress" {
|
||||
project = var.project_id
|
||||
name = "egress"
|
||||
region = var.region
|
||||
router = google_compute_router.egress.name
|
||||
nat_ip_allocate_option = "AUTO_ONLY"
|
||||
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
|
||||
|
||||
log_config {
|
||||
enable = true
|
||||
filter = "ERRORS_ONLY"
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_firewall" "allow_health_checks" {
|
||||
project = var.project_id
|
||||
name = "allow-health-checks"
|
||||
network = google_compute_network.main.id
|
||||
direction = "INGRESS"
|
||||
priority = 1000
|
||||
|
||||
# Google Cloud load balancer health check ranges.
|
||||
source_ranges = ["130.211.0.0/22", "35.191.0.0/16"]
|
||||
target_tags = ["web"]
|
||||
|
||||
allow {
|
||||
protocol = "tcp"
|
||||
ports = ["8080"]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_service" "internal" {
|
||||
project = var.project_id
|
||||
name = "internal"
|
||||
location = var.region
|
||||
ingress = "INGRESS_TRAFFIC_INTERNAL_ONLY"
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = "us-docker.pkg.dev/cloudrun/container/hello"
|
||||
}
|
||||
|
||||
vpc_access {
|
||||
egress = "PRIVATE_RANGES_ONLY"
|
||||
|
||||
network_interfaces {
|
||||
network = google_compute_network.main.id
|
||||
subnetwork = google_compute_subnetwork.workloads.id
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Enabling APIs
|
||||
|
||||
Enable each API the Elements need, once, and make the first resource of each product depend on it.
|
||||
|
||||
- `disable_on_destroy = false` keeps the API on when the configuration is destroyed; other workloads in the project may use it.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
locals {
|
||||
services = [
|
||||
"run.googleapis.com",
|
||||
"pubsub.googleapis.com",
|
||||
"secretmanager.googleapis.com",
|
||||
]
|
||||
}
|
||||
|
||||
resource "google_project_service" "enabled" {
|
||||
for_each = toset(local.services)
|
||||
|
||||
project = var.project_id
|
||||
service = each.value
|
||||
disable_on_destroy = false
|
||||
}
|
||||
```
|
||||
111
skills/google-cloud-iac-patterns/references/pubsub.md
Normal file
111
skills/google-cloud-iac-patterns/references/pubsub.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Pub/Sub
|
||||
|
||||
A topic, a push subscription that calls a private Cloud Run service, and a dead-letter topic.
|
||||
|
||||
- A push subscription authenticates with `push_config.oidc_token`; its service account needs `roles/run.invoker` on the service it calls.
|
||||
- Dead-lettering needs two grants to the Pub/Sub service agent: `roles/pubsub.publisher` on the dead-letter topic and `roles/pubsub.subscriber` on the source subscription. Without them messages are never moved.
|
||||
- Give the dead-letter topic a subscription of its own, or the messages moved there are lost.
|
||||
- `max_delivery_attempts` is 5 to 100. `ack_deadline_seconds` is 10 to 600.
|
||||
- `message_retention_duration` is a duration in seconds, such as `"604800s"`.
|
||||
- Customer-managed encryption is `kms_key_name` on the topic; the Pub/Sub service agent needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key first.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "worker_url" {
|
||||
type = string
|
||||
description = "HTTPS URL of the service the subscription pushes to."
|
||||
}
|
||||
|
||||
data "google_project" "this" {
|
||||
project_id = var.project_id
|
||||
}
|
||||
|
||||
locals {
|
||||
pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
|
||||
}
|
||||
|
||||
resource "google_service_account" "push" {
|
||||
project = var.project_id
|
||||
account_id = "events-push"
|
||||
display_name = "Identity Pub/Sub pushes with"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic" "events" {
|
||||
project = var.project_id
|
||||
name = "events"
|
||||
message_retention_duration = "604800s"
|
||||
|
||||
message_storage_policy {
|
||||
allowed_persistence_regions = [var.region]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic" "dead_letter" {
|
||||
project = var.project_id
|
||||
name = "events-dead-letter"
|
||||
}
|
||||
|
||||
resource "google_pubsub_subscription" "dead_letter" {
|
||||
project = var.project_id
|
||||
name = "events-dead-letter"
|
||||
topic = google_pubsub_topic.dead_letter.id
|
||||
message_retention_duration = "1209600s"
|
||||
|
||||
expiration_policy {
|
||||
ttl = ""
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_pubsub_subscription" "worker" {
|
||||
project = var.project_id
|
||||
name = "events-worker"
|
||||
topic = google_pubsub_topic.events.id
|
||||
ack_deadline_seconds = 60
|
||||
|
||||
push_config {
|
||||
push_endpoint = var.worker_url
|
||||
|
||||
oidc_token {
|
||||
service_account_email = google_service_account.push.email
|
||||
audience = var.worker_url
|
||||
}
|
||||
}
|
||||
|
||||
retry_policy {
|
||||
minimum_backoff = "10s"
|
||||
maximum_backoff = "600s"
|
||||
}
|
||||
|
||||
dead_letter_policy {
|
||||
dead_letter_topic = google_pubsub_topic.dead_letter.id
|
||||
max_delivery_attempts = 5
|
||||
}
|
||||
|
||||
expiration_policy {
|
||||
ttl = ""
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "agent_publishes_dead_letters" {
|
||||
project = var.project_id
|
||||
topic = google_pubsub_topic.dead_letter.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = local.pubsub_agent
|
||||
}
|
||||
|
||||
resource "google_pubsub_subscription_iam_member" "agent_forwards_from_worker" {
|
||||
project = var.project_id
|
||||
subscription = google_pubsub_subscription.worker.name
|
||||
role = "roles/pubsub.subscriber"
|
||||
member = local.pubsub_agent
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,45 @@
|
||||
# Secret Manager
|
||||
|
||||
Terraform creates the secret and who may read it. The value is added outside Terraform, so it never enters the state or the repository.
|
||||
|
||||
- `replication` is required: `auto {}` for Google-managed locations, or `user_managed` with one `replicas` block per location.
|
||||
- Grant `roles/secretmanager.secretAccessor` on the secret, to the one identity that reads it.
|
||||
- Never write a `google_secret_manager_secret_version` with the value in the configuration.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
resource "google_service_account" "reader" {
|
||||
project = var.project_id
|
||||
account_id = "api-key-reader"
|
||||
display_name = "Reads the API key"
|
||||
}
|
||||
|
||||
resource "google_secret_manager_secret" "api_key" {
|
||||
project = var.project_id
|
||||
secret_id = "partner-api-key"
|
||||
|
||||
replication {
|
||||
user_managed {
|
||||
replicas {
|
||||
location = var.region
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_secret_manager_secret_iam_member" "reader_reads_api_key" {
|
||||
project = var.project_id
|
||||
secret_id = google_secret_manager_secret.api_key.secret_id
|
||||
role = "roles/secretmanager.secretAccessor"
|
||||
member = google_service_account.reader.member
|
||||
}
|
||||
```
|
||||
@@ -2,31 +2,35 @@
|
||||
name: iac-validation
|
||||
description: >-
|
||||
Checks Terraform infrastructure-as-code without provisioning anything, using
|
||||
the local terraform_fmt, terraform_validate and terraform_plan tools. Use when
|
||||
writing deployment guidance or validating it before deployment.
|
||||
the local terraform_fmt, terraform_validate, terraform_plan and
|
||||
terraform_schema tools. Use when writing deployment guidance or validating it
|
||||
before deployment.
|
||||
metadata:
|
||||
scope: deployment-guidance validation
|
||||
---
|
||||
|
||||
# IaC validation
|
||||
|
||||
Every check runs on a scratch copy of the files you pass. Nothing is applied, and no state is written.
|
||||
Every check runs on a scratch copy of your Terraform files. Nothing is applied, and no state is written.
|
||||
|
||||
The files are your working copy: the files you wrote with `write_file`, or, when you are validating, the files that were submitted. Pass the tools no files.
|
||||
|
||||
## Tools
|
||||
|
||||
- `terraform_fmt(files)`: returns the files formatting changed. Replace your copies with the returned content.
|
||||
- `terraform_validate(files)`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments.
|
||||
- `terraform_plan(files, variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires.
|
||||
- `terraform_fmt()`: formats your working copy in place and names the files it changed.
|
||||
- `terraform_validate()`: runs `init -backend=false` and `validate` offline. It reports syntax errors, unknown references, missing variables and invalid arguments.
|
||||
- `terraform_plan(variables)`: previews the changes against the live cloud environment. It reads live state, so the first call may pause for the user's permission. Pass every variable the configuration requires.
|
||||
- `terraform_schema(resource_type, block, find)`: what a resource type accepts, from the provider's own schema. With `resource_type` alone it lists the top-level arguments and outlines every block. With `block`, such as `template.containers`, it gives that block's arguments in full. With `find`, it says where an argument or block of that name sits, or that the resource type has none.
|
||||
|
||||
## When writing deployment guidance
|
||||
|
||||
1. Run `terraform_fmt`, then `terraform_validate`, on the complete set of files you will attach.
|
||||
2. Fix every error and run both again, until `terraform_validate` passes.
|
||||
3. Attach the formatted files.
|
||||
1. Run `terraform_fmt`, then `terraform_validate`.
|
||||
2. Fix each error with `edit_file`, changing only the lines concerned, and run `terraform_validate` again until it passes.
|
||||
3. On "An argument named X is not expected here" or "Unsupported block type", call `terraform_schema` with `find` set to the name. It is either in a different block, which the reply names, or it does not exist on that resource type. Never move it around by trial.
|
||||
|
||||
## When validating deployment guidance
|
||||
|
||||
1. Run `terraform_validate` on the attached files, and report its verdict and output.
|
||||
1. Run `terraform_validate`, and report its verdict and output.
|
||||
2. When permission to read the live environment is granted, run `terraform_plan` with the variable values from the requirements, or clearly marked placeholders. Summarise the resources to add, change and destroy, and flag any destroy or replace.
|
||||
3. When a check can't run (a missing CLI, a refused permission or a provider download failure), give the user the exact commands to run themselves:
|
||||
|
||||
@@ -40,4 +44,6 @@ Every check runs on a scratch copy of the files you pass. Nothing is applied, an
|
||||
|
||||
Give a verdict on every Check in the Target architecture and attach them as `verdicts.json`, in the format the requirements-chain Skill gives. The dry-run's result goes in its `dry_run` entry: `pass` when `terraform_validate` passes, `fail` with the error when it does not, and `not_run` when the tool could not run.
|
||||
|
||||
A Check fails when the code could meet it and does not. When `terraform_schema` shows the resource type has no such setting, no code could meet it: the verdict is `not_verifiable`, with the schema's answer as evidence.
|
||||
|
||||
Never suggest `terraform apply` as part of validation.
|
||||
|
||||
@@ -92,14 +92,23 @@ A Check is a statement that is true or false of the infrastructure code. Name th
|
||||
|
||||
Split a broad recommendation into several Checks, one per setting. A Check that the validator cannot verify from the code is sent back to you.
|
||||
|
||||
A Check must name a setting its Element's resource type really has. Two similar resource types often differ: a setting of a long-running service may not exist on a batch job. A Check on a setting the resource type lacks is sent back to you.
|
||||
|
||||
The tables of Elements, Relationships and Checks, and the diagram, are written from this file when you submit. Write the graph once, here.
|
||||
|
||||
### `implementation.json` (deployment guidance)
|
||||
|
||||
One entry per Check in the Target architecture, saying where the code satisfies it. `file` is an attached file; `address` is the resource address and argument.
|
||||
|
||||
```json
|
||||
[{"check": "K-1", "file": "terraform/main.tf", "address": "google_cloud_run_v2_service.ingest.ingress"}]
|
||||
[
|
||||
{"check": "K-1", "file": "terraform/main.tf", "address": "google_cloud_run_v2_service.ingest.ingress"},
|
||||
{"check": "K-7", "unimplementable": "google_cloud_run_v2_job has no startup_cpu_boost argument; it exists only on a service."}
|
||||
]
|
||||
```
|
||||
|
||||
When a Check names a setting its resource type does not have, give it an `unimplementable` entry in place of `file` and `address`, saying what the resource type lacks. Write no code for it. The Target architecture is revised before anything is validated.
|
||||
|
||||
Implement every Element, Relationship and Check, and nothing the Target architecture does not ask for. When you believe the Target architecture is wrong or incomplete, say so with `needs_input`; do not quietly build something else.
|
||||
|
||||
### `verdicts.json` (validation results)
|
||||
|
||||
@@ -271,6 +271,21 @@ async def test_model_timeout_is_set_on_every_request():
|
||||
assert model.requests[0].config.http_options.timeout == 90_000
|
||||
|
||||
|
||||
async def test_a_role_with_its_own_timeout_is_given_it_over_the_default():
|
||||
model = ScriptedLlm(script=[[call("delegate")]])
|
||||
runtime = AdkAgentRuntime(
|
||||
skill_dirs=SKILLS, model_factory=lambda role: model, model_timeout_seconds=90
|
||||
)
|
||||
|
||||
await runtime.run(
|
||||
RoleSpec("orchestrator", "Coordinate.", budget=5, model_timeout_seconds=420),
|
||||
TASK,
|
||||
[Recorder().tool("delegate", TurnEnded("delegate"))],
|
||||
)
|
||||
|
||||
assert model.requests[0].config.http_options.timeout == 420_000
|
||||
|
||||
|
||||
class MeteredLlm(ScriptedLlm):
|
||||
async def generate_content_async(
|
||||
self, llm_request: LlmRequest, stream: bool = False
|
||||
|
||||
@@ -19,7 +19,7 @@ def bundle():
|
||||
def test_every_engine_role_and_the_pillar_advisor_is_defined(bundle):
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
|
||||
expected = {"orchestrator", PILLAR_ADVISOR}
|
||||
expected = {"orchestrator", "reply-writer", PILLAR_ADVISOR}
|
||||
expected |= {spec.specialist for spec in DEPENDENCY_GRAPH.values()}
|
||||
assert set(roles) == expected
|
||||
for role in roles.values():
|
||||
@@ -46,12 +46,35 @@ def test_roles_take_skills_mcp_servers_and_models_from_the_bundle(bundle):
|
||||
def test_roles_name_only_the_local_tools_they_need(bundle):
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
|
||||
assert roles["iac-author"].tools == ("terraform_fmt", "terraform_validate")
|
||||
assert roles["target-architect"].tools == ()
|
||||
assert roles["validator"].tools == ("terraform_validate", "terraform_plan")
|
||||
assert roles["iac-author"].tools == (
|
||||
"terraform_fmt",
|
||||
"terraform_validate",
|
||||
"terraform_schema",
|
||||
)
|
||||
# The architect only looks settings up; it writes and runs no code.
|
||||
assert roles["target-architect"].tools == ("terraform_schema",)
|
||||
assert roles["validator"].tools == (
|
||||
"terraform_validate",
|
||||
"terraform_plan",
|
||||
"terraform_schema",
|
||||
)
|
||||
assert roles["requirements-analyst"].tools == ()
|
||||
|
||||
|
||||
def test_roles_with_long_replies_are_given_longer_for_a_model_request(bundle):
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
|
||||
assert roles["target-architect"].model_timeout_seconds == 420
|
||||
assert roles["decomposer"].model_timeout_seconds is None
|
||||
|
||||
|
||||
def test_the_code_author_is_given_the_patterns_for_its_cloud_provider(bundle):
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
|
||||
assert "google-cloud-iac-patterns" in roles["iac-author"].skills
|
||||
assert "google-cloud-iac-patterns" not in roles["validator"].skills
|
||||
|
||||
|
||||
def test_preview_mcp_servers_can_be_excluded(bundle):
|
||||
roles = load_roles(AGENTS_DIR, bundle, include_preview_mcp=False)
|
||||
|
||||
|
||||
@@ -475,3 +475,56 @@ async def test_a_scripted_user_drives_an_interactive_execution(tmp_path):
|
||||
assert any("Approved." in reply for reply in replies)
|
||||
assert any("europe-west2." in reply for reply in replies)
|
||||
assert final.state["deliverables"]["requirements"]["content"]
|
||||
|
||||
|
||||
# --- Jev as the judge -------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_jev_judges_every_question_in_one_call_with_no_grading_in_the_state():
|
||||
from app.adk.fake_runtime import FakeDecider
|
||||
from app.adk.runtime import Answer
|
||||
from eval.judge import DeciderJudge
|
||||
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"covered": Answer(0.91, 0.82),
|
||||
"missed": Answer(0.2, 0.6),
|
||||
"specific": Answer(2.0, 0.7), # "mostly", on a scale of four
|
||||
}
|
||||
)
|
||||
judge = DeciderJudge(decider)
|
||||
|
||||
scores = await judge.answer(
|
||||
"## Reliability\nUse two zones.",
|
||||
{
|
||||
"covered": {"kind": "yes_no", "text": "Is redundancy addressed?"},
|
||||
"missed": {"kind": "yes_no", "text": "Are backups tested?"},
|
||||
"specific": {"kind": "graded", "text": "Is it specific?"},
|
||||
},
|
||||
)
|
||||
|
||||
assert scores == {"covered": 1.0, "missed": 0.0, "specific": 2 / 3}
|
||||
[(state, questions, scope)] = decider.asked
|
||||
assert state == "## Reliability\nUse two zones." # the document, nothing else
|
||||
assert questions["covered"].kind == "noul"
|
||||
assert questions["specific"].options == ("not at all", "weakly", "mostly", "fully")
|
||||
assert scope == "evaluation"
|
||||
|
||||
|
||||
async def test_a_judge_that_gives_no_answer_leaves_the_check_unanswered_not_passed():
|
||||
from app.adk.runtime import NoDecider
|
||||
from eval.judge import DeciderJudge
|
||||
|
||||
result = await score_execution(
|
||||
CASE,
|
||||
execution(),
|
||||
HEADINGS,
|
||||
passing_terraform,
|
||||
judge=DeciderJudge(NoDecider()),
|
||||
pillar_checklists=CHECKLISTS,
|
||||
)
|
||||
|
||||
assert check(result, "pillar_checklists_covered").detail == (
|
||||
"judge unavailable: Jev gave no answer"
|
||||
)
|
||||
assert not check(result, "recommendations_quality").passed
|
||||
|
||||
@@ -15,7 +15,7 @@ from app.agent import ExecutionEngine
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.tools.execution import specialist_tools
|
||||
from app.tools.local import mermaid_check
|
||||
from app.workflows.chain import validation_outcome
|
||||
from app.workflows.chain import unimplementable_checks, validation_outcome
|
||||
from app.workflows.deliverables import (
|
||||
DEPENDENCY_GRAPH,
|
||||
DeliverableKind,
|
||||
@@ -327,6 +327,54 @@ def test_deployment_guidance_must_account_for_every_check():
|
||||
)
|
||||
|
||||
|
||||
def test_a_check_that_cannot_be_implemented_is_accounted_for_with_its_reason():
|
||||
args = sample_submission(K.DEPLOYMENT_GUIDANCE).args
|
||||
implementation = json.dumps(
|
||||
[
|
||||
{"check": "K-1", "file": "terraform/main.tf", "address": "a"},
|
||||
{
|
||||
"check": "K-2",
|
||||
"unimplementable": "example_service_account has no such argument",
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
missing = check_submission(
|
||||
K.DEPLOYMENT_GUIDANCE,
|
||||
args["content"],
|
||||
{**args["attachments"], "implementation.json": implementation},
|
||||
upstream(*EVERYTHING_BEFORE_CODE),
|
||||
)
|
||||
|
||||
assert missing == ()
|
||||
assert unimplementable_checks(implementation) == [
|
||||
{"check": "K-2", "reason": "example_service_account has no such argument"}
|
||||
]
|
||||
assert unimplementable_checks(SAMPLE_DATA["implementation.json"]) == []
|
||||
|
||||
|
||||
def test_an_unimplementable_entry_needs_its_reason():
|
||||
args = sample_submission(K.DEPLOYMENT_GUIDANCE).args
|
||||
implementation = json.dumps(
|
||||
[
|
||||
{"check": "K-1", "file": "terraform/main.tf", "address": "a"},
|
||||
{"check": "K-2", "unimplementable": ""},
|
||||
]
|
||||
)
|
||||
|
||||
missing = check_submission(
|
||||
K.DEPLOYMENT_GUIDANCE,
|
||||
args["content"],
|
||||
{**args["attachments"], "implementation.json": implementation},
|
||||
upstream(*EVERYTHING_BEFORE_CODE),
|
||||
)
|
||||
|
||||
assert missing == (
|
||||
"implementation.json[0]: needs unimplementable",
|
||||
"implementation.json: Check K-2 is not accounted for",
|
||||
)
|
||||
|
||||
|
||||
def verdicts(**changes) -> str:
|
||||
data = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||
data.update(changes)
|
||||
|
||||
598
tests/execution/test_router.py
Normal file
598
tests/execution/test_router.py
Normal file
@@ -0,0 +1,598 @@
|
||||
"""The Orchestrator as a router (ADR-0007): readings, a disposition, then a playbook.
|
||||
|
||||
No language model chooses an action here. Specialists are scripted; the Orchestrator
|
||||
is the Router, with a scripted decider standing in for Jev, or none at all.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import (
|
||||
SAMPLE_DATA,
|
||||
Call,
|
||||
FakeAgentRuntime,
|
||||
FakeDecider,
|
||||
sample_submission,
|
||||
)
|
||||
from app.adk.runtime import Answer, NoDecider, RoleSpec
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.router import (
|
||||
REPLY_WRITER,
|
||||
Router,
|
||||
RuntimeReplyWriter,
|
||||
Thresholds,
|
||||
)
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||
|
||||
K = DeliverableKind
|
||||
|
||||
ROLES = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused by the router.", budget=40),
|
||||
REPLY_WRITER: RoleSpec(REPLY_WRITER, "Write the reply.", budget=3),
|
||||
**{
|
||||
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def specialist(kind: DeliverableKind) -> str:
|
||||
return DEPENDENCY_GRAPH[kind].specialist
|
||||
|
||||
|
||||
def once() -> dict:
|
||||
return {
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
|
||||
|
||||
def validation(**changes) -> Call:
|
||||
"""A validation submission whose verdicts differ from the all-pass sample."""
|
||||
call = sample_submission(K.VALIDATION_RESULTS)
|
||||
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||
verdicts.update(changes)
|
||||
return Call(
|
||||
call.tool,
|
||||
{**call.args, "attachments": {"verdicts.json": json.dumps(verdicts)}},
|
||||
)
|
||||
|
||||
|
||||
def failing(check: str = "K-1") -> Call:
|
||||
checks = json.loads(SAMPLE_DATA["verdicts.json"])["checks"]
|
||||
for item in checks:
|
||||
if item["check"] == check:
|
||||
item.update(verdict="fail", evidence="min instances is 1")
|
||||
return validation(checks=checks)
|
||||
|
||||
|
||||
def addition(reason: str) -> Call:
|
||||
return validation(
|
||||
proposed_additions=[
|
||||
{"statement": "A timeout is set.", "applies_to": "E-1", "reason": reason}
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def engine_for(tmp_path, turns: dict, decider=None, writer: bool = False, **options):
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
router = Router(
|
||||
decider or NoDecider(),
|
||||
RuntimeReplyWriter(runtime, ROLES[REPLY_WRITER]) if writer else None,
|
||||
Thresholds(),
|
||||
)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=router,
|
||||
**options,
|
||||
)
|
||||
return engine, runtime
|
||||
|
||||
|
||||
def actions(result) -> list[str]:
|
||||
return [decision["action"] for decision in result.state["decisions"]]
|
||||
|
||||
|
||||
def reopened(result) -> list[tuple[str, str]]:
|
||||
return [
|
||||
(event["subject"], event["detail"])
|
||||
for event in result.state["ledger"]["events"]
|
||||
if event["type"] == "reopened"
|
||||
]
|
||||
|
||||
|
||||
def approve(phase: str) -> dict:
|
||||
return {"approval": {"phase": phase, "decision": "approve"}}
|
||||
|
||||
|
||||
# --- the playbook on rules alone -------------------------------------------------------
|
||||
|
||||
|
||||
async def test_an_execution_completes_with_no_model_choosing_an_action(tmp_path):
|
||||
engine, runtime = engine_for(tmp_path, once())
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert "orchestrator" not in {run.role for run in runtime.runs}
|
||||
assert actions(result) == [
|
||||
"delegate", # requirements
|
||||
"delegate", # technical decomposition
|
||||
"request_approval",
|
||||
"delegate", # product selection
|
||||
"delegate", # design recommendations
|
||||
"delegate", # Target architecture
|
||||
"request_approval",
|
||||
"delegate", # description and deployment guidance, together
|
||||
"request_approval",
|
||||
"delegate", # validation
|
||||
"proceed",
|
||||
"request_approval",
|
||||
"delegate", # solution guide
|
||||
"request_approval",
|
||||
"finish",
|
||||
]
|
||||
groups = {decision["group"] for decision in result.state["decisions"]}
|
||||
assert groups == {
|
||||
"work is ready",
|
||||
"a Phase is complete",
|
||||
"validation decided",
|
||||
"packaging is approved",
|
||||
}
|
||||
|
||||
|
||||
async def test_every_decision_is_recorded_with_its_turn_group_and_action(tmp_path):
|
||||
engine, _ = engine_for(tmp_path, once())
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
first = result.state["decisions"][0]
|
||||
assert first == {
|
||||
"turn": 1,
|
||||
"group": "work is ready",
|
||||
"action": "delegate",
|
||||
"judged": {},
|
||||
}
|
||||
assert [d["turn"] for d in result.state["decisions"]] == sorted(
|
||||
d["turn"] for d in result.state["decisions"]
|
||||
)
|
||||
|
||||
|
||||
async def test_when_no_group_applies_a_pre_approved_execution_ends_with_warnings(
|
||||
tmp_path,
|
||||
):
|
||||
turns = {specialist(K.REQUIREMENTS): [[]]} # stops without submitting
|
||||
|
||||
engine, _ = engine_for(tmp_path, turns)
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
# The Specialist's question is relayed, answered automatically, and it stops
|
||||
# again; the router never invents work.
|
||||
assert result.status == "completed_with_warnings"
|
||||
|
||||
|
||||
# --- Approvals: the summary is written from data, and only data approves ----------------
|
||||
|
||||
|
||||
async def test_the_approval_summary_is_assembled_from_the_data(tmp_path):
|
||||
engine, _ = engine_for(tmp_path, once())
|
||||
|
||||
first = await engine.start("Ingest webhooks")
|
||||
design = await engine.resume(first.execution_id, approve("requirements_discovery"))
|
||||
|
||||
assert first.prompt["approval"]["summary"] == (
|
||||
"The requirements_discovery Phase is ready: requirements, "
|
||||
"technical_decomposition submitted. 2 requirements."
|
||||
)
|
||||
summary = design.prompt["approval"]["summary"]
|
||||
assert "Target architecture: 2 Elements, 1 Relationships, 2 Checks." in summary
|
||||
assert design.prompt["approval"]["target_architecture"]["checks"] == 2
|
||||
|
||||
|
||||
async def test_a_model_writes_the_summary_but_decides_nothing(tmp_path):
|
||||
turns = {
|
||||
**once(),
|
||||
REPLY_WRITER: [[Call("write_reply", {"text": "Two requirements, ready."})]],
|
||||
}
|
||||
engine, runtime = engine_for(tmp_path, turns, writer=True)
|
||||
|
||||
first = await engine.start("Ingest webhooks")
|
||||
|
||||
assert first.prompt["approval"]["summary"] == "Two requirements, ready."
|
||||
writer_run = next(run for run in runtime.runs if run.role == REPLY_WRITER)
|
||||
assert writer_run.tools == ("write_reply",)
|
||||
assert writer_run.task.context["facts"]["deliverables"] == [
|
||||
"requirements",
|
||||
"technical_decomposition",
|
||||
]
|
||||
|
||||
|
||||
async def test_a_writer_that_fails_does_not_stop_the_approval(tmp_path):
|
||||
engine, _ = engine_for(tmp_path, once(), writer=True) # no writer turn scripted
|
||||
|
||||
first = await engine.start("Ingest webhooks")
|
||||
|
||||
assert first.prompt["approval"]["summary"].startswith(
|
||||
"The requirements_discovery Phase is ready"
|
||||
)
|
||||
|
||||
|
||||
async def test_a_text_reply_never_approves_and_the_approval_is_asked_again(tmp_path):
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"asks_change": Answer(0.03, 0.94),
|
||||
"change_target": Answer("requirements", 0.5),
|
||||
"change_clarity": Answer(0.0, 0.9),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, once(), decider)
|
||||
|
||||
first = await engine.start("Ingest webhooks")
|
||||
again = await engine.resume(first.execution_id, {"reply": "Looks good to me."})
|
||||
|
||||
assert again.prompt["approval"]["phase"] == "requirements_discovery"
|
||||
assert again.state["ledger"]["phases"]["requirements_discovery"] != "approved"
|
||||
|
||||
|
||||
# --- validation: the code, the graph, or proceed ----------------------------------------
|
||||
|
||||
|
||||
async def test_a_failed_check_sends_the_code_back_and_leaves_the_design_approved(
|
||||
tmp_path,
|
||||
):
|
||||
turns = once()
|
||||
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [
|
||||
[failing("K-1")],
|
||||
[sample_submission(K.VALIDATION_RESULTS)],
|
||||
]
|
||||
engine, runtime = engine_for(tmp_path, turns)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert reopened(result) == [
|
||||
("deployment_guidance", "K-1 failed: min instances is 1")
|
||||
]
|
||||
assert "revise_code" in actions(result)
|
||||
architect_runs = [
|
||||
run for run in runtime.runs if run.role == specialist(K.TARGET_ARCHITECTURE)
|
||||
]
|
||||
assert len(architect_runs) == 1
|
||||
|
||||
|
||||
async def test_a_check_the_code_cannot_meet_goes_back_to_the_graph_unvalidated(
|
||||
tmp_path,
|
||||
):
|
||||
"""The code's author found the Check names a setting the resource lacks. No
|
||||
validation is spent on it: the Target architecture is revised first."""
|
||||
cannot = sample_submission(K.DEPLOYMENT_GUIDANCE)
|
||||
implementation = json.loads(SAMPLE_DATA["implementation.json"])
|
||||
implementation[1] = {"check": "K-2", "unimplementable": "no such argument"}
|
||||
turns = once()
|
||||
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
|
||||
turns[specialist(K.ARCHITECTURE_DESCRIPTION)] *= 2
|
||||
turns[specialist(K.DEPLOYMENT_GUIDANCE)] = [
|
||||
[
|
||||
Call(
|
||||
cannot.tool,
|
||||
{
|
||||
**cannot.args,
|
||||
"attachments": {
|
||||
**cannot.args["attachments"],
|
||||
"implementation.json": json.dumps(implementation),
|
||||
},
|
||||
},
|
||||
)
|
||||
],
|
||||
[sample_submission(K.DEPLOYMENT_GUIDANCE)],
|
||||
]
|
||||
engine, runtime = engine_for(tmp_path, turns)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert reopened(result) == [
|
||||
(
|
||||
"target_architecture",
|
||||
(
|
||||
"Check K-2 cannot be implemented as written (no such argument). "
|
||||
"Rewrite it so it can be met, or remove it with a reason."
|
||||
),
|
||||
)
|
||||
]
|
||||
assert "the code cannot meet a Check" in {
|
||||
decision["group"] for decision in result.state["decisions"]
|
||||
}
|
||||
validations = [
|
||||
run for run in runtime.runs if run.role == specialist(K.VALIDATION_RESULTS)
|
||||
]
|
||||
assert len(validations) == 1
|
||||
|
||||
|
||||
async def test_a_check_that_fails_twice_running_is_sent_back_to_the_graph(tmp_path):
|
||||
"""The code was revised and the same Check failed again: the Check is suspect."""
|
||||
turns = once()
|
||||
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 3
|
||||
turns[specialist(K.ARCHITECTURE_DESCRIPTION)] *= 2
|
||||
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [
|
||||
[failing("K-1")],
|
||||
[failing("K-1")],
|
||||
[sample_submission(K.VALIDATION_RESULTS)],
|
||||
]
|
||||
engine, _ = engine_for(tmp_path, turns)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
[first, second] = reopened(result)
|
||||
assert first[0] == "deployment_guidance"
|
||||
assert second[0] == "target_architecture"
|
||||
assert "K-1 failed in two validations in a row" in second[1]
|
||||
assert actions(result).count("revise_code") == 1
|
||||
|
||||
|
||||
async def test_without_jev_a_gap_naming_a_real_requirement_revises_the_graph(tmp_path):
|
||||
turns = once()
|
||||
for kind in (
|
||||
K.TARGET_ARCHITECTURE,
|
||||
K.ARCHITECTURE_DESCRIPTION,
|
||||
K.DEPLOYMENT_GUIDANCE,
|
||||
):
|
||||
turns[specialist(kind)] *= 2
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [
|
||||
[addition("NFR-1 has no Check on timeouts")],
|
||||
[sample_submission(K.VALIDATION_RESULTS)],
|
||||
]
|
||||
engine, _ = engine_for(tmp_path, turns)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
[(kind, reason)] = reopened(result)
|
||||
assert kind == "target_architecture"
|
||||
assert reason == (
|
||||
"Add a Check on E-1: A timeout is set. (serves NFR-1 has no Check on timeouts)."
|
||||
)
|
||||
|
||||
|
||||
async def test_without_jev_a_gap_that_names_nothing_real_is_left(tmp_path):
|
||||
turns = once()
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [[addition("it would be nice")]]
|
||||
engine, _ = engine_for(tmp_path, turns)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert reopened(result) == []
|
||||
|
||||
|
||||
async def test_jev_judges_whether_a_gap_is_worth_a_revision(tmp_path):
|
||||
turns = once()
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [[addition("NFR-1")]]
|
||||
immaterial = FakeDecider({"gap_material": Answer(0.04, 0.92)})
|
||||
engine, _ = engine_for(tmp_path, turns, immaterial)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
# The rule alone would have revised the graph; the reading says it is not needed.
|
||||
assert reopened(result) == []
|
||||
judged = next(d for d in result.state["decisions"] if d["action"] == "proceed")
|
||||
assert judged["judged"] == {"gap_material": {"value": 0.04, "confidence": 0.92}}
|
||||
[(state, questions, scope)] = immaterial.asked
|
||||
assert "A timeout is set." in state and scope == result.execution_id
|
||||
assert questions["gap_material"].kind == "noul"
|
||||
|
||||
|
||||
async def test_an_uncertain_gap_is_put_to_the_user_who_decides(tmp_path):
|
||||
turns = once()
|
||||
for kind in (
|
||||
K.TARGET_ARCHITECTURE,
|
||||
K.ARCHITECTURE_DESCRIPTION,
|
||||
K.DEPLOYMENT_GUIDANCE,
|
||||
):
|
||||
turns[specialist(kind)] *= 2
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [
|
||||
[addition("NFR-1")],
|
||||
[sample_submission(K.VALIDATION_RESULTS)],
|
||||
]
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"gap_material": Answer(0.5, 0.0), # the grey zone
|
||||
"wants_revision": Answer(0.96, 0.92),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
|
||||
result = await engine.start("Ingest webhooks")
|
||||
for phase in (
|
||||
"requirements_discovery",
|
||||
"solution_architecture",
|
||||
"solution_implementation",
|
||||
):
|
||||
result = await engine.resume(result.execution_id, approve(phase))
|
||||
|
||||
[question] = result.prompt["questions"]
|
||||
assert (
|
||||
"found gaps in the Target architecture: A timeout is set."
|
||||
in (question["question"])
|
||||
)
|
||||
|
||||
result = await engine.resume(result.execution_id, {"reply": "Yes, revise it."})
|
||||
|
||||
assert reopened(result)[0][0] == "target_architecture"
|
||||
# Reopening the graph withdrew the design Approval, so it is asked for again.
|
||||
assert result.prompt["approval"]["phase"] == "solution_architecture"
|
||||
|
||||
|
||||
# --- the user's message: act, queue or close --------------------------------------------
|
||||
|
||||
|
||||
def to_design_approval(engine):
|
||||
async def run():
|
||||
first = await engine.start("Ingest webhooks")
|
||||
return await engine.resume(
|
||||
first.execution_id, approve("requirements_discovery")
|
||||
)
|
||||
|
||||
return run()
|
||||
|
||||
|
||||
def changes(phase: str, feedback: str) -> dict:
|
||||
return {"approval": {"phase": phase, "decision": "changes", "feedback": feedback}}
|
||||
|
||||
|
||||
async def test_a_requested_change_is_routed_to_the_deliverable_it_affects(tmp_path):
|
||||
turns = once()
|
||||
turns[specialist(K.DESIGN_RECOMMENDATIONS)] *= 2
|
||||
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"change_target": Answer("design_recommendations", 0.88),
|
||||
"change_clarity": Answer(1.8, 0.8),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
|
||||
design = await to_design_approval(engine)
|
||||
result = await engine.resume(
|
||||
design.execution_id,
|
||||
changes("solution_architecture", "Use customer-managed keys on the topic."),
|
||||
)
|
||||
|
||||
assert reopened(result) == [
|
||||
("design_recommendations", "Use customer-managed keys on the topic.")
|
||||
]
|
||||
# Product selection was not touched; the design is asked for again once redone.
|
||||
assert result.state["ledger"]["deliverables"]["product_selection"]["revisions"] == 0
|
||||
assert result.prompt["approval"]["phase"] == "solution_architecture"
|
||||
[(state, questions, _)] = decider.asked
|
||||
assert "The user's message: Use customer-managed keys on the topic." in state
|
||||
assert questions["change_target"].options == (
|
||||
"requirements",
|
||||
"technical_decomposition",
|
||||
"product_selection",
|
||||
"design_recommendations",
|
||||
"target_architecture",
|
||||
)
|
||||
assert "asks_change" not in questions # a structured request is known to be one
|
||||
|
||||
|
||||
async def test_an_unclear_change_is_confirmed_with_the_user_before_anything_moves(
|
||||
tmp_path,
|
||||
):
|
||||
turns = once()
|
||||
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"change_target": Answer("product_selection", 0.31), # not relied on
|
||||
"change_clarity": Answer(1.5, 0.7),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
|
||||
design = await to_design_approval(engine)
|
||||
asked = await engine.resume(
|
||||
design.execution_id, changes("solution_architecture", "Tighten it up.")
|
||||
)
|
||||
|
||||
assert reopened(asked) == []
|
||||
[question] = asked.prompt["questions"]
|
||||
assert question["question"].startswith("Do you want product_selection changed?")
|
||||
|
||||
result = await engine.resume(
|
||||
asked.execution_id,
|
||||
{"reply": "No, change the target architecture: add a Check on timeouts."},
|
||||
)
|
||||
|
||||
[(kind, reason)] = reopened(result)
|
||||
assert kind == "target_architecture"
|
||||
assert "Tighten it up." in reason and "add a Check on timeouts" in reason
|
||||
|
||||
|
||||
async def test_a_free_text_reply_that_asks_for_a_change_is_acted_on(tmp_path):
|
||||
turns = once()
|
||||
turns[specialist(K.REQUIREMENTS)] = [
|
||||
[sample_submission(K.REQUIREMENTS)],
|
||||
[sample_submission(K.REQUIREMENTS)],
|
||||
]
|
||||
turns[specialist(K.TECHNICAL_DECOMPOSITION)] *= 2
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"asks_change": Answer(0.93, 0.86),
|
||||
"change_target": Answer("requirements", 0.9),
|
||||
"change_clarity": Answer(1.9, 0.8),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
|
||||
first = await engine.start("Ingest webhooks")
|
||||
result = await engine.resume(
|
||||
first.execution_id, {"reply": "Retention must be 7 years, not 30 days."}
|
||||
)
|
||||
|
||||
assert reopened(result) == [
|
||||
("requirements", "Retention must be 7 years, not 30 days.")
|
||||
]
|
||||
|
||||
|
||||
async def test_without_jev_a_change_is_acted_on_only_when_the_deliverable_is_named(
|
||||
tmp_path,
|
||||
):
|
||||
turns = once()
|
||||
turns[specialist(K.DESIGN_RECOMMENDATIONS)] *= 2
|
||||
turns[specialist(K.TARGET_ARCHITECTURE)] *= 2
|
||||
engine, _ = engine_for(tmp_path, turns)
|
||||
|
||||
design = await to_design_approval(engine)
|
||||
unnamed = await engine.resume(
|
||||
design.execution_id, changes("solution_architecture", "Make it cheaper.")
|
||||
)
|
||||
|
||||
assert reopened(unnamed) == []
|
||||
assert unnamed.prompt["questions"][0]["question"].startswith(
|
||||
"Which Deliverable should change"
|
||||
)
|
||||
|
||||
named = await engine.resume(
|
||||
unnamed.execution_id, {"reply": "The design recommendations."}
|
||||
)
|
||||
|
||||
[(kind, reason)] = reopened(named)
|
||||
assert kind == "design_recommendations"
|
||||
assert reason == (
|
||||
"Make it cheaper.\nThe user then said: The design recommendations."
|
||||
)
|
||||
|
||||
|
||||
# --- permission is the Ledger's, whatever was decided ------------------------------------
|
||||
|
||||
|
||||
async def test_past_the_revision_limit_the_user_decides(tmp_path):
|
||||
turns = once()
|
||||
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2
|
||||
turns[specialist(K.VALIDATION_RESULTS)] = [[failing("K-2")], [failing("K-1")]]
|
||||
engine, _ = engine_for(tmp_path, turns, max_revisions=1)
|
||||
|
||||
result = await engine.start("Ingest webhooks")
|
||||
for phase in (
|
||||
"requirements_discovery",
|
||||
"solution_architecture",
|
||||
"solution_implementation",
|
||||
"solution_implementation",
|
||||
):
|
||||
result = await engine.resume(result.execution_id, approve(phase))
|
||||
|
||||
assert len(reopened(result)) == 1
|
||||
[question] = result.prompt["questions"]
|
||||
assert (
|
||||
"deployment_guidance has reached its revision limit (1)"
|
||||
in (question["question"])
|
||||
)
|
||||
480
tests/execution/test_working_copy.py
Normal file
480
tests/execution/test_working_copy.py
Normal file
@@ -0,0 +1,480 @@
|
||||
"""A Specialist's working copy, and what each Specialist is given to read.
|
||||
|
||||
Three things made an Execution slow: a Specialist rewrote a whole Deliverable in one
|
||||
reply, sent every file again on each tool call, and read every earlier Deliverable in
|
||||
full. A working copy and trimmed reading remove each of them.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import (
|
||||
SAMPLE_DATA,
|
||||
Call,
|
||||
FakeAgentRuntime,
|
||||
sample_submission,
|
||||
)
|
||||
from app.adk.runtime import NoDecider, RoleSpec, Submitted, ToolResult
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.tools.execution import specialist_tools
|
||||
from app.tools.local import TerraformRunner, local_tools
|
||||
from app.tools.toolkit import make_tool_factory
|
||||
from app.tools.workspace import file_tools, graph_tools
|
||||
from app.workflows.deliverables import (
|
||||
DEPENDENCY_GRAPH,
|
||||
DeliverableKind,
|
||||
dependencies_for,
|
||||
)
|
||||
from app.workflows.target_architecture import parse_graph, render_tables
|
||||
|
||||
K = DeliverableKind
|
||||
GRAPH = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||
UPSTREAM = {
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
|
||||
},
|
||||
"technical_decomposition": {
|
||||
"attachments": {"components.json": SAMPLE_DATA["components.json"]}
|
||||
},
|
||||
"product_selection": {
|
||||
"attachments": {"products.json": SAMPLE_DATA["products.json"]}
|
||||
},
|
||||
"design_recommendations": {
|
||||
"attachments": {"recommendations.json": SAMPLE_DATA["recommendations.json"]}
|
||||
},
|
||||
"target_architecture": {
|
||||
"attachments": {
|
||||
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def tools_of(kind, working, upstream=UPSTREAM, extra=()):
|
||||
return {
|
||||
tool.name: tool
|
||||
for tool in specialist_tools(
|
||||
kind, extra, allow_cloud_read=False, upstream=upstream, working=working
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
# --- the Target architecture is written once -------------------------------------------
|
||||
|
||||
|
||||
def test_tables_are_rendered_from_the_graph():
|
||||
tables = render_tables(parse_graph(SAMPLE_DATA["target-architecture.json"]))
|
||||
|
||||
assert "## Elements" in tables and "## Relationships" in tables
|
||||
assert (
|
||||
"| E-1 | Ingress service | Example Run | example_run_service | C-1 |" in tables
|
||||
)
|
||||
assert "| R-1 | Ingress service | runs as | Ingress identity |" in tables
|
||||
assert "| K-1 | E-1 | The service runs at least two instances. | NFR-1 |" in tables
|
||||
|
||||
|
||||
def test_a_table_cell_cannot_break_its_row():
|
||||
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||
graph["checks"][0]["statement"] = "a | b\nc"
|
||||
|
||||
assert "| a \\| b c |" in render_tables(parse_graph(json.dumps(graph)))
|
||||
|
||||
|
||||
async def test_the_engine_writes_the_tables_and_the_diagram_on_submission():
|
||||
tools = tools_of(K.TARGET_ARCHITECTURE, {})
|
||||
|
||||
result = await tools["submit_deliverable"].handler(
|
||||
content="## Design rationale\nOne service, one identity.\n",
|
||||
attachments={
|
||||
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
|
||||
},
|
||||
)
|
||||
|
||||
assert isinstance(result.outcome, Submitted)
|
||||
assert result.outcome.content.startswith("## Design rationale")
|
||||
assert "## Checks" in result.outcome.content
|
||||
assert result.outcome.attachments["architecture.mmd"].startswith("flowchart")
|
||||
|
||||
|
||||
async def test_the_graph_is_built_a_few_items_at_a_time_and_attached_on_submission():
|
||||
working: dict = {}
|
||||
tools = tools_of(K.TARGET_ARCHITECTURE, working)
|
||||
|
||||
await tools["graph_put"].handler(elements=GRAPH["elements"])
|
||||
await tools["graph_put"].handler(relationships=GRAPH["relationships"])
|
||||
put = await tools["graph_put"].handler(checks=GRAPH["checks"])
|
||||
assert put.message == (
|
||||
"Added 2, replaced 0. The graph has 2 Elements, 1 Relationships and 2 Checks."
|
||||
)
|
||||
|
||||
result = await tools["submit_deliverable"].handler(
|
||||
content="## Design rationale\nReasons.\n"
|
||||
)
|
||||
|
||||
assert isinstance(result.outcome, Submitted)
|
||||
submitted = json.loads(result.outcome.attachments["target-architecture.json"])
|
||||
assert [item["id"] for item in submitted["checks"]] == ["K-1", "K-2"]
|
||||
|
||||
|
||||
async def test_a_revision_amends_the_graph_that_was_submitted():
|
||||
working: dict = {}
|
||||
tools = tools_of(K.TARGET_ARCHITECTURE, working)
|
||||
await tools["submit_deliverable"].handler(
|
||||
content="## Design rationale\nReasons.\n",
|
||||
attachments={
|
||||
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
|
||||
},
|
||||
)
|
||||
|
||||
# A later attempt, with the working memory the Execution kept.
|
||||
tools = tools_of(K.TARGET_ARCHITECTURE, working)
|
||||
changed = {**GRAPH["checks"][0], "statement": "min_instance_count is at least 2"}
|
||||
put = await tools["graph_put"].handler(checks=[changed])
|
||||
read = await tools["graph_read"].handler()
|
||||
|
||||
assert put.message.startswith("Added 0, replaced 1.")
|
||||
assert "min_instance_count is at least 2" in read.message
|
||||
assert len(json.loads(read.message)["checks"]) == 2
|
||||
|
||||
|
||||
async def test_graph_items_can_be_removed_and_an_item_without_an_id_is_refused():
|
||||
working: dict = {}
|
||||
tools = {tool.name: tool for tool in graph_tools(working)}
|
||||
await tools["graph_put"].handler(
|
||||
elements=GRAPH["elements"],
|
||||
not_applicable=[{"recommendation": "DR-9", "reason": "Single region."}],
|
||||
)
|
||||
|
||||
removed = await tools["graph_remove"].handler(ids=["E-2", "DR-9"])
|
||||
refused = await tools["graph_put"].handler(checks=[{"statement": "no id"}])
|
||||
|
||||
assert removed.message.startswith("Removed 2.")
|
||||
assert refused.message.startswith("Refused") and "checks[0]" in refused.message
|
||||
assert [e["id"] for e in working["graph"]["elements"]] == ["E-1"]
|
||||
|
||||
|
||||
# --- files are written once and changed in place ----------------------------------------
|
||||
|
||||
|
||||
async def test_a_file_is_written_once_and_changed_in_place():
|
||||
working: dict = {}
|
||||
tools = {tool.name: tool for tool in file_tools(working)}
|
||||
|
||||
await tools["write_file"].handler(
|
||||
path="terraform/main.tf", content="a = 1\nb = 1\n"
|
||||
)
|
||||
edited = await tools["edit_file"].handler(
|
||||
path="terraform/main.tf", old="b = 1", new="b = 2"
|
||||
)
|
||||
listing = await tools["read_file"].handler()
|
||||
|
||||
assert edited.message == "Edited terraform/main.tf."
|
||||
assert working["files"] == {"terraform/main.tf": "a = 1\nb = 2\n"}
|
||||
assert "- terraform/main.tf (2 lines)" in listing.message
|
||||
|
||||
|
||||
async def test_an_edit_that_does_not_match_exactly_once_changes_nothing():
|
||||
working: dict = {"files": {"main.tf": "x = 1\nx = 1\n"}}
|
||||
tools = {tool.name: tool for tool in file_tools(working)}
|
||||
|
||||
twice = await tools["edit_file"].handler(path="main.tf", old="x = 1", new="x = 2")
|
||||
never = await tools["edit_file"].handler(path="main.tf", old="y", new="z")
|
||||
absent = await tools["edit_file"].handler(path="other.tf", old="x", new="y")
|
||||
unsafe = await tools["write_file"].handler(path="../main.tf", content="x")
|
||||
|
||||
assert "it matches 2" in twice.message and "it matches 0" in never.message
|
||||
assert absent.message.startswith("Refused") and unsafe.message.startswith("Refused")
|
||||
assert working["files"] == {"main.tf": "x = 1\nx = 1\n"}
|
||||
|
||||
|
||||
async def test_the_working_copy_is_what_is_submitted():
|
||||
working: dict = {}
|
||||
tools = tools_of(K.DEPLOYMENT_GUIDANCE, working)
|
||||
for path in ("terraform/main.tf", "terraform/variables.tf"):
|
||||
await tools["write_file"].handler(path=path, content=f"# {path}\n")
|
||||
await tools["write_file"].handler(
|
||||
path="implementation.json", content=SAMPLE_DATA["implementation.json"]
|
||||
)
|
||||
content = sample_submission(K.DEPLOYMENT_GUIDANCE).args["content"]
|
||||
|
||||
result = await tools["submit_deliverable"].handler(content=content)
|
||||
|
||||
assert isinstance(result.outcome, Submitted)
|
||||
assert set(result.outcome.attachments) == {
|
||||
"terraform/main.tf",
|
||||
"terraform/variables.tf",
|
||||
"implementation.json",
|
||||
}
|
||||
|
||||
|
||||
async def test_a_file_passed_on_submission_replaces_its_working_copy():
|
||||
working: dict = {"files": {"terraform/main.tf": "old\n"}}
|
||||
tools = tools_of(K.DEPLOYMENT_GUIDANCE, working)
|
||||
call = sample_submission(K.DEPLOYMENT_GUIDANCE)
|
||||
|
||||
result = await tools["submit_deliverable"].handler(**call.args)
|
||||
|
||||
assert isinstance(result.outcome, Submitted)
|
||||
assert result.outcome.attachments["terraform/main.tf"] != "old\n"
|
||||
assert working["files"]["terraform/main.tf"] != "old\n"
|
||||
|
||||
|
||||
def test_only_a_specialist_that_owns_files_can_write_them():
|
||||
assert "write_file" in tools_of(K.DEPLOYMENT_GUIDANCE, {})
|
||||
assert "write_file" not in tools_of(K.VALIDATION_RESULTS, {})
|
||||
assert "graph_put" in tools_of(K.TARGET_ARCHITECTURE, {})
|
||||
assert "graph_put" not in tools_of(K.DEPLOYMENT_GUIDANCE, {})
|
||||
|
||||
|
||||
# --- Terraform runs on the working copy -------------------------------------------------
|
||||
|
||||
|
||||
class Runner(TerraformRunner):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.validated: list[dict[str, str]] = []
|
||||
|
||||
async def validate(self, files):
|
||||
self.validated.append(dict(files))
|
||||
return ToolResult("terraform validate passed:\n")
|
||||
|
||||
async def formatted(self, files):
|
||||
return {name: content.strip() + "\n" for name, content in files.items()}
|
||||
|
||||
|
||||
async def test_terraform_checks_take_no_files_when_there_is_a_working_copy():
|
||||
runner = Runner()
|
||||
files = {"terraform/main.tf": "x = 1\n", "implementation.json": "[]"}
|
||||
(validate,) = local_tools(["terraform_validate"], runner, files)
|
||||
|
||||
await validate.handler()
|
||||
|
||||
assert validate.parameters["required"] == []
|
||||
assert "pass no files" in validate.description
|
||||
# Only the Terraform files are checked.
|
||||
assert runner.validated == [{"terraform/main.tf": "x = 1\n"}]
|
||||
|
||||
|
||||
async def test_formatting_changes_the_working_copy_and_returns_only_names():
|
||||
files = {"terraform/main.tf": " x = 1 \n"}
|
||||
(fmt,) = local_tools(["terraform_fmt"], Runner(), files)
|
||||
|
||||
result = await fmt.handler()
|
||||
|
||||
assert files == {"terraform/main.tf": "x = 1\n"}
|
||||
assert result.message == (
|
||||
"terraform fmt reformatted 1 file(s) in your working copy: terraform/main.tf."
|
||||
)
|
||||
|
||||
|
||||
async def test_files_passed_anyway_are_written_to_the_working_copy_first():
|
||||
runner = Runner()
|
||||
files: dict[str, str] = {}
|
||||
(validate,) = local_tools(["terraform_validate"], runner, files)
|
||||
|
||||
await validate.handler(files=[{"path": "terraform/main.tf", "content": "y"}])
|
||||
|
||||
assert files == {"terraform/main.tf": "y"}
|
||||
assert runner.validated == [files]
|
||||
|
||||
|
||||
def roles_with(tools_by_role: dict[str, tuple[str, ...]]) -> dict[str, RoleSpec]:
|
||||
return {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
**{
|
||||
spec.specialist: RoleSpec(
|
||||
spec.specialist,
|
||||
"Produce.",
|
||||
budget=15,
|
||||
tools=tools_by_role.get(spec.specialist, ()),
|
||||
)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
async def test_the_validator_checks_the_submitted_code_without_being_sent_it(tmp_path):
|
||||
runner = Runner()
|
||||
roles = roles_with({"validator": ("terraform_validate",)})
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
turns["validator"] = [
|
||||
[Call("terraform_validate", {}), sample_submission(K.VALIDATION_RESULTS)]
|
||||
]
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=roles,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
tool_factory=make_tool_factory(
|
||||
runtime,
|
||||
roles,
|
||||
{},
|
||||
local_tools=lambda working_copy: local_tools(
|
||||
["terraform_validate"], runner, working_copy
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
assert runner.validated == [
|
||||
{
|
||||
"terraform/main.tf": "# terraform/main.tf\n",
|
||||
"terraform/variables.tf": "# terraform/variables.tf\n",
|
||||
}
|
||||
]
|
||||
# The code is the code author's: validation attaches its verdicts and nothing else.
|
||||
validation = result.state["deliverables"]["validation_results"]
|
||||
assert set(validation["attachments"]) == {"verdicts.json"}
|
||||
|
||||
|
||||
# --- each Specialist reads what it works from -------------------------------------------
|
||||
|
||||
|
||||
def submitted() -> dict[str, dict]:
|
||||
return {
|
||||
kind.value: {
|
||||
"content": f"# {kind}\nProse.\n",
|
||||
"attachments": {
|
||||
name: SAMPLE_DATA.get(name, f"# {name}\n") for name in spec.attachments
|
||||
},
|
||||
}
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
|
||||
|
||||
def test_the_code_is_written_from_the_graph_and_the_ids_alone():
|
||||
given = dependencies_for(K.DEPLOYMENT_GUIDANCE, submitted())
|
||||
|
||||
assert given["target_architecture"] == {
|
||||
"attachments": {
|
||||
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
|
||||
}
|
||||
}
|
||||
assert all("content" not in deliverable for deliverable in given.values())
|
||||
assert set(given["requirements"]["attachments"]) == {"requirements.json"}
|
||||
|
||||
|
||||
def test_a_direct_dependency_is_read_whole_and_the_rest_as_data():
|
||||
given = dependencies_for(K.TARGET_ARCHITECTURE, submitted())
|
||||
|
||||
assert "content" in given["product_selection"]
|
||||
assert "content" in given["design_recommendations"]
|
||||
assert "content" not in given["requirements"]
|
||||
assert "content" not in given["technical_decomposition"]
|
||||
|
||||
|
||||
def test_the_validator_reads_the_code_whole_and_the_graph_as_data():
|
||||
given = dependencies_for(K.VALIDATION_RESULTS, submitted())
|
||||
|
||||
assert "terraform/main.tf" in given["deployment_guidance"]["attachments"]
|
||||
assert "content" not in given["target_architecture"]
|
||||
# A Deliverable with no data file has nothing to trim to, so it comes whole.
|
||||
assert "content" in given["architecture_description"]
|
||||
|
||||
|
||||
def test_the_solution_guide_is_assembled_from_every_deliverable_whole():
|
||||
given = dependencies_for(K.SOLUTION_GUIDE, submitted())
|
||||
|
||||
assert len(given) == len(DEPENDENCY_GRAPH) - 1
|
||||
assert all("content" in deliverable for deliverable in given.values())
|
||||
|
||||
|
||||
async def test_a_specialist_is_told_what_its_working_copy_already_holds(tmp_path):
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
failing = sample_submission(K.VALIDATION_RESULTS)
|
||||
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||
verdicts["checks"][0].update(verdict="fail", evidence="min instances is 1")
|
||||
turns["validator"] = [
|
||||
[
|
||||
Call(
|
||||
failing.tool,
|
||||
{
|
||||
**failing.args,
|
||||
"attachments": {"verdicts.json": json.dumps(verdicts)},
|
||||
},
|
||||
)
|
||||
],
|
||||
[sample_submission(K.VALIDATION_RESULTS)],
|
||||
]
|
||||
turns["iac-author"] = [[sample_submission(K.DEPLOYMENT_GUIDANCE)]] * 2
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=roles_with({}),
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
)
|
||||
|
||||
await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
first, second = (run for run in runtime.runs if run.role == "iac-author")
|
||||
assert first.task.context["working_copy"] is None
|
||||
assert second.task.context["working_copy"] == {
|
||||
"files": ["implementation.json", "terraform/main.tf", "terraform/variables.tf"]
|
||||
}
|
||||
|
||||
|
||||
async def test_a_deliverable_given_as_data_can_be_read_whole_on_demand():
|
||||
library = {
|
||||
"requirements": {
|
||||
"content": "# requirements\nProse.\n",
|
||||
"attachments": {"requirements.json": "[]"},
|
||||
}
|
||||
}
|
||||
tools = {
|
||||
tool.name: tool
|
||||
for tool in specialist_tools(
|
||||
K.DEPLOYMENT_GUIDANCE, (), allow_cloud_read=False, library=library
|
||||
)
|
||||
}
|
||||
|
||||
read = await tools["read_deliverable"].handler(kind="requirements")
|
||||
refused = await tools["read_deliverable"].handler(kind="solution_guide")
|
||||
|
||||
assert (
|
||||
read.message.startswith("# requirements")
|
||||
and "requirements.json" in read.message
|
||||
)
|
||||
assert refused.message.startswith("Refused")
|
||||
# With nothing trimmed there is nothing to read back.
|
||||
assert "read_deliverable" not in tools_of(K.REQUIREMENTS, {})
|
||||
|
||||
|
||||
async def test_the_engine_offers_the_trimmed_deliverables_for_reading(tmp_path):
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=roles_with({}),
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
)
|
||||
|
||||
await engine.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
author = next(run for run in runtime.runs if run.role == "iac-author")
|
||||
packager = next(run for run in runtime.runs if run.role == "packager")
|
||||
assert "read_deliverable" in author.tools
|
||||
assert "content" not in author.task.context["dependencies"]["requirements"]
|
||||
assert "read_deliverable" not in packager.tools
|
||||
@@ -12,6 +12,7 @@ def settings(tmp_path, **overrides) -> Settings:
|
||||
values: dict[str, Any] = {
|
||||
"CHECKPOINTER": "memory",
|
||||
"MCP_ENABLED": False,
|
||||
"ORCHESTRATOR": "model",
|
||||
"ORCHESTRATOR_BUDGET": 9,
|
||||
"SPECIALIST_BUDGET": 4,
|
||||
}
|
||||
@@ -50,3 +51,42 @@ async def test_preview_mcp_servers_follow_the_setting(tmp_path):
|
||||
|
||||
assert "asset_inventory" not in servers
|
||||
assert "compute" in servers
|
||||
|
||||
|
||||
async def test_the_assembly_routes_by_default_and_runs_without_a_jev_key(tmp_path):
|
||||
"""With no Jev key the router follows its playbook on rules alone."""
|
||||
from app.adk.fake_runtime import sample_submission
|
||||
from app.adk.runtime import NoDecider
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
runtime = FakeAgentRuntime(
|
||||
{"requirements-analyst": [[sample_submission(DeliverableKind.REQUIREMENTS)]]}
|
||||
)
|
||||
|
||||
async with open_assembly(
|
||||
settings(tmp_path, ORCHESTRATOR="router", TYPESAFE_API_KEY=None),
|
||||
runtime=runtime,
|
||||
recorder=InMemoryRecorder(),
|
||||
output_root=tmp_path,
|
||||
) as assembly:
|
||||
interaction = await assembly.engine.start("Design ingestion")
|
||||
|
||||
assert isinstance(assembly.decider, NoDecider)
|
||||
# The router delegated requirements with no Orchestrator model, then the
|
||||
# decomposer (unscripted here) stopped and its question reached the user.
|
||||
assert [run.role for run in runtime.runs] == ["requirements-analyst"]
|
||||
assert interaction.status == "awaiting_user"
|
||||
assert "reply-writer" in assembly.roles
|
||||
|
||||
|
||||
async def test_an_unknown_orchestrator_setting_is_refused(tmp_path):
|
||||
import pytest
|
||||
|
||||
with pytest.raises(ValueError, match="ORCHESTRATOR must be router or model"):
|
||||
async with open_assembly(
|
||||
settings(tmp_path, ORCHESTRATOR="llm"),
|
||||
runtime=FakeAgentRuntime({}),
|
||||
recorder=InMemoryRecorder(),
|
||||
output_root=tmp_path,
|
||||
):
|
||||
pass
|
||||
|
||||
62
tests/test_iac_patterns.py
Normal file
62
tests/test_iac_patterns.py
Normal file
@@ -0,0 +1,62 @@
|
||||
"""The IaC patterns Skill: every pattern validates against the pinned provider.
|
||||
|
||||
A pattern that does not validate teaches the mistake it was written to prevent, so
|
||||
each reference's Terraform is run through `terraform validate` with the Skill's own
|
||||
versions file. The provider comes from the plugin cache; with no cache and no
|
||||
network the test is skipped, not failed.
|
||||
"""
|
||||
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.providers import load_provider
|
||||
from app.tools.local import TerraformRunner
|
||||
|
||||
SKILL = "google-cloud-iac-patterns"
|
||||
ROOT = Path(__file__).resolve().parent.parent / "skills" / SKILL
|
||||
REFERENCES = sorted((ROOT / "references").glob("*.md"))
|
||||
_HCL = re.compile(r"^```hcl\n(.*?)^```", re.MULTILINE | re.DOTALL)
|
||||
|
||||
needs_terraform = pytest.mark.skipif(
|
||||
shutil.which("terraform") is None, reason="terraform CLI not installed"
|
||||
)
|
||||
|
||||
|
||||
def _runner() -> TerraformRunner:
|
||||
return TerraformRunner(plugin_cache_dir=Settings().terraform_plugin_cache())
|
||||
|
||||
|
||||
def test_every_reference_is_listed_in_the_skill_and_holds_a_pattern():
|
||||
skill = (ROOT / "SKILL.md").read_text(encoding="utf-8")
|
||||
assert REFERENCES
|
||||
for reference in REFERENCES:
|
||||
assert f"references/{reference.name}" in skill
|
||||
assert _HCL.findall(reference.read_text(encoding="utf-8")), reference.name
|
||||
listed = set(re.findall(r"references/([\w-]+\.md)", skill))
|
||||
assert listed == {reference.name for reference in REFERENCES}
|
||||
|
||||
|
||||
def test_the_skill_pins_the_version_the_provider_bundle_pins():
|
||||
pin = load_provider("gcp").terraform_providers["google"]
|
||||
versions = (ROOT / "assets" / "versions.tf").read_text(encoding="utf-8")
|
||||
skill = (ROOT / "SKILL.md").read_text(encoding="utf-8")
|
||||
assert f'source = "{pin["source"]}"' in versions
|
||||
assert f'version = "{pin["version"]}"' in versions
|
||||
assert f"{pin['source']} {pin['version']}" in skill
|
||||
|
||||
|
||||
@needs_terraform
|
||||
@pytest.mark.parametrize("reference", REFERENCES, ids=lambda path: path.stem)
|
||||
async def test_pattern_validates(reference: Path):
|
||||
files = {
|
||||
"terraform/versions.tf": (ROOT / "assets" / "versions.tf").read_text(),
|
||||
"terraform/main.tf": "\n".join(_HCL.findall(reference.read_text())),
|
||||
}
|
||||
result = await _runner().validate(files)
|
||||
if result.message.startswith("terraform init failed"):
|
||||
pytest.skip("the pinned provider could not be installed")
|
||||
assert result.message.startswith("terraform validate passed"), result.message
|
||||
@@ -218,3 +218,55 @@ def test_bundle_places_each_local_tool_in_its_own_tier():
|
||||
for tier, names in bundle.tools.items():
|
||||
for name in names:
|
||||
assert LOCAL_TOOLS[name][0] == tier, name
|
||||
|
||||
|
||||
def test_bundle_pins_terraform_providers_and_role_timeouts(tmp_path):
|
||||
config = minimal_bundle(
|
||||
tmp_path,
|
||||
models={"default": "gemini/m", "timeout_seconds": {"target-architect": 420}},
|
||||
terraform={
|
||||
"required_providers": {
|
||||
"cloud": {"source": "example/cloud", "version": "1.2.3"}
|
||||
}
|
||||
},
|
||||
)
|
||||
write_bundle(tmp_path, config)
|
||||
|
||||
bundle = load_provider("test", root=tmp_path)
|
||||
|
||||
assert bundle.terraform_providers == {
|
||||
"cloud": {"source": "example/cloud", "version": "1.2.3"}
|
||||
}
|
||||
assert bundle.models.timeout_for("target-architect") == 420
|
||||
assert bundle.models.timeout_for("decomposer") is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"overrides, message",
|
||||
[
|
||||
(
|
||||
{"terraform": {"required_providers": {"cloud": {"source": "x/cloud"}}}},
|
||||
"needs a 'source' and an exact 'version'",
|
||||
),
|
||||
(
|
||||
{"models": {"default": "gemini/m", "timeout_seconds": {"x": 0}}},
|
||||
"must be positive",
|
||||
),
|
||||
(
|
||||
{"models": {"default": "gemini/m", "timeout_seconds": {"x": "slow"}}},
|
||||
"must be a number",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_bad_terraform_pins_and_timeouts_are_rejected(tmp_path, overrides, message):
|
||||
write_bundle(tmp_path, minimal_bundle(tmp_path, **overrides))
|
||||
|
||||
with pytest.raises(ProviderConfigError, match=message):
|
||||
load_provider("test", root=tmp_path)
|
||||
|
||||
|
||||
def test_gcp_bundle_pins_one_exact_google_provider_version():
|
||||
pin = load_provider("gcp").terraform_providers["google"]
|
||||
|
||||
assert pin["source"] == "hashicorp/google"
|
||||
assert pin["version"].replace(".", "").isdigit() # exact, not a range
|
||||
|
||||
417
tests/tools/test_jev.py
Normal file
417
tests/tools/test_jev.py
Normal file
@@ -0,0 +1,417 @@
|
||||
"""The Jev decision client: typed answers within strict limits, and a fallback always.
|
||||
|
||||
Every test runs the real TypeSafe SDK against a stand-in transport; nothing here calls
|
||||
the live service.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
||||
import httpx2
|
||||
|
||||
from app.adk.runtime import Answer, NoDecider, Question
|
||||
from app.agent import open_decider
|
||||
from app.config import Settings
|
||||
from app.tools.jev import JevDecider, JevLimits, estimate_tokens
|
||||
|
||||
URGENT = Question("noul", "Does this need attention right now?")
|
||||
ROUTE = Question(
|
||||
"choice", "What comes next?", ("revise_code", "revise_graph", "proceed")
|
||||
)
|
||||
SEVERITY = Question("score", "How much does it matter?", ("low", "medium", "high"))
|
||||
ALL = {"urgent": URGENT, "route": ROUTE, "severity": SEVERITY}
|
||||
|
||||
ANSWERS = {
|
||||
"urgent": {"type": "noul", "noul": 0.97},
|
||||
"route": {
|
||||
"type": "choice",
|
||||
"choice": "revise_code",
|
||||
"confidence": 0.81,
|
||||
"probabilities": {"revise_code": 0.81, "revise_graph": 0.12, "proceed": 0.07},
|
||||
},
|
||||
"severity": {
|
||||
"type": "score",
|
||||
"score": 1.6,
|
||||
"confidence": 0.7,
|
||||
"legend": {"0": "low", "1": "medium", "2": "high"},
|
||||
"probabilities": {"0": 0.1, "1": 0.2, "2": 0.7},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class Clock:
|
||||
"""Time that moves only when something sleeps."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.now = 1000.0
|
||||
self.slept: list[float] = []
|
||||
|
||||
def __call__(self) -> float:
|
||||
return self.now
|
||||
|
||||
async def sleep(self, seconds: float) -> None:
|
||||
self.slept.append(seconds)
|
||||
self.now += seconds
|
||||
|
||||
|
||||
def ok(*names: str, tokens: int = 40) -> httpx2.Response:
|
||||
answers = {name: ANSWERS[name] for name in names} if names else ANSWERS
|
||||
return httpx2.Response(
|
||||
200,
|
||||
json={
|
||||
"model": "jev-latest",
|
||||
"answers": answers,
|
||||
"usage": {"input_tokens": tokens, "output_tokens": 0},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class Service:
|
||||
"""A stand-in Jev: replies from a queue, then with `default`; records requests.
|
||||
|
||||
A reply answers only the questions the request asked, as the service does.
|
||||
"""
|
||||
|
||||
def __init__(self, *replies, default=None) -> None:
|
||||
self.replies = list(replies)
|
||||
self.default = default
|
||||
self.requests: list[httpx2.Request] = []
|
||||
|
||||
def __call__(self, request: httpx2.Request) -> httpx2.Response:
|
||||
self.requests.append(request)
|
||||
reply = self.replies.pop(0) if self.replies else self.default
|
||||
if reply is None:
|
||||
return ok(*json.loads(request.content)["questions"])
|
||||
if isinstance(reply, Exception):
|
||||
raise reply
|
||||
return reply
|
||||
|
||||
@property
|
||||
def bodies(self) -> list[dict]:
|
||||
return [json.loads(request.content) for request in self.requests]
|
||||
|
||||
|
||||
def decider(service: Service, clock: Clock | None = None, **limits) -> JevDecider:
|
||||
clock = clock or Clock()
|
||||
limits.setdefault("backoff_base", 0.0) # no real waiting between SDK retries
|
||||
return JevDecider(
|
||||
api_key="tsk_secret",
|
||||
base_url="https://jev.example",
|
||||
limits=JevLimits(**limits),
|
||||
transport=httpx2.MockTransport(service),
|
||||
clock=clock,
|
||||
sleep=clock.sleep,
|
||||
)
|
||||
|
||||
|
||||
# --- the request and the typed answers -------------------------------------------------
|
||||
|
||||
|
||||
async def test_one_request_carries_the_state_and_every_question():
|
||||
service = Service()
|
||||
jev = decider(service)
|
||||
|
||||
answers = await jev.decide("K-30 failed twice.", ALL, scope="exec-1")
|
||||
|
||||
[request] = service.requests
|
||||
assert request.method == "POST"
|
||||
assert str(request.url) == "https://jev.example/v1/systemone"
|
||||
assert request.headers["Authorization"] == "Bearer tsk_secret"
|
||||
body = service.bodies[0]
|
||||
assert body["state"] == "K-30 failed twice."
|
||||
assert body["questions"] == {
|
||||
"urgent": {"type": "noul", "instructions": URGENT.instructions},
|
||||
"route": {
|
||||
"type": "choice",
|
||||
"instructions": ROUTE.instructions,
|
||||
"criteria": {"revise_code": None, "revise_graph": None, "proceed": None},
|
||||
},
|
||||
"severity": {
|
||||
"type": "score",
|
||||
"instructions": SEVERITY.instructions,
|
||||
"criteria": ["low", "medium", "high"],
|
||||
},
|
||||
}
|
||||
assert answers["route"] == Answer(
|
||||
"revise_code",
|
||||
0.81,
|
||||
{"revise_code": 0.81, "revise_graph": 0.12, "proceed": 0.07},
|
||||
)
|
||||
assert answers["severity"] == Answer(1.6, 0.7, {"0": 0.1, "1": 0.2, "2": 0.7})
|
||||
# A yes-or-no answer is a probability; its confidence is its distance from even.
|
||||
assert answers["urgent"].value == 0.97
|
||||
assert round(answers["urgent"].confidence, 2) == 0.94
|
||||
assert answers["urgent"].is_yes() and not Answer(0.2, 0.6).is_yes()
|
||||
|
||||
|
||||
async def test_a_choice_outside_the_options_or_a_missing_answer_is_no_decision():
|
||||
invented = httpx2.Response(
|
||||
200,
|
||||
json={
|
||||
"model": "jev-latest",
|
||||
"usage": {"input_tokens": 1, "output_tokens": 0},
|
||||
"answers": {
|
||||
"route": {
|
||||
"type": "choice",
|
||||
"choice": "delete_everything",
|
||||
"confidence": 0.99,
|
||||
"probabilities": {},
|
||||
}
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
assert await decider(Service(invented)).decide("s", {"route": ROUTE}) is None
|
||||
assert await decider(Service(ok("urgent"))).decide("s", ALL) is None
|
||||
assert (
|
||||
await decider(Service(httpx2.Response(200, text="not json"))).decide(
|
||||
"s", {"urgent": URGENT}
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
async def test_no_questions_needs_no_request():
|
||||
service = Service()
|
||||
|
||||
assert await decider(service).decide("state", {}) == {}
|
||||
assert service.requests == []
|
||||
|
||||
|
||||
# --- size --------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_request_over_the_size_limit_is_refused_locally_never_truncated():
|
||||
service = Service()
|
||||
jev = decider(service, max_tokens=100, max_question_tokens=100)
|
||||
state = "x" * 600
|
||||
|
||||
assert estimate_tokens(state) > 100
|
||||
assert await jev.decide(state, {"urgent": URGENT}) is None
|
||||
assert service.requests == []
|
||||
assert jev.records[-1].outcome == "too_large"
|
||||
|
||||
|
||||
# --- timeouts and retries (the SDK's, configured here) ------------------------------------
|
||||
|
||||
|
||||
async def test_temporary_failures_are_retried_then_answered():
|
||||
service = Service(httpx2.Response(503), httpx2.ConnectTimeout("slow"))
|
||||
jev = decider(service)
|
||||
|
||||
assert await jev.decide("s", {"urgent": URGENT}) is not None
|
||||
assert len(service.requests) == 3
|
||||
|
||||
|
||||
async def test_a_rate_limit_that_outlasts_the_retries_lowers_the_clients_own_rate():
|
||||
service = Service(default=httpx2.Response(429))
|
||||
jev = decider(service, retries=1)
|
||||
|
||||
assert await jev.decide("s", {"urgent": URGENT}) is None
|
||||
assert len(service.requests) == 2
|
||||
assert jev.records[-1].rate_limited
|
||||
assert jev._rate == JevLimits().requests_per_second / 2
|
||||
|
||||
|
||||
async def test_a_rejected_request_is_not_retried():
|
||||
service = Service(default=httpx2.Response(400, json={"error": "bad question"}))
|
||||
jev = decider(service)
|
||||
|
||||
assert await jev.decide("s", {"urgent": URGENT}) is None
|
||||
assert len(service.requests) == 1
|
||||
|
||||
|
||||
async def test_retries_stop_at_the_budget_and_the_caller_falls_back():
|
||||
service = Service(default=httpx2.Response(503))
|
||||
jev = decider(service, retries=2)
|
||||
|
||||
assert await jev.decide("s", {"urgent": URGENT}) is None
|
||||
assert len(service.requests) == 3
|
||||
assert jev.records[-1].outcome == "failed"
|
||||
|
||||
|
||||
# --- rate and concurrency ----------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_calls_beyond_the_burst_wait_for_the_bucket_to_refill():
|
||||
clock = Clock()
|
||||
service = Service()
|
||||
jev = decider(service, clock, burst=2, requests_per_second=4)
|
||||
|
||||
for index in range(3):
|
||||
await jev.decide(f"state {index}", {"urgent": URGENT})
|
||||
|
||||
assert len(service.requests) == 3
|
||||
assert clock.slept == [0.25] # one token at four a second
|
||||
|
||||
|
||||
async def test_calls_in_flight_never_exceed_the_cap():
|
||||
in_flight = peak = 0
|
||||
|
||||
async def slow(request: httpx2.Request) -> httpx2.Response:
|
||||
nonlocal in_flight, peak
|
||||
in_flight += 1
|
||||
peak = max(peak, in_flight)
|
||||
await asyncio.sleep(0.01)
|
||||
in_flight -= 1
|
||||
return ok("urgent")
|
||||
|
||||
jev = JevDecider(
|
||||
api_key="k",
|
||||
base_url="https://jev.example",
|
||||
limits=JevLimits(max_in_flight=2, burst=50),
|
||||
transport=httpx2.MockTransport(slow),
|
||||
)
|
||||
|
||||
await asyncio.gather(
|
||||
*(jev.decide(f"state {index}", {"urgent": URGENT}) for index in range(8))
|
||||
)
|
||||
|
||||
assert peak == 2
|
||||
|
||||
|
||||
# --- circuit breaker ---------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_repeated_failures_open_the_circuit_and_one_probe_closes_it():
|
||||
clock = Clock()
|
||||
service = Service(default=httpx2.Response(503))
|
||||
jev = decider(
|
||||
service, clock, retries=0, breaker_failures=3, breaker_cooldown=60, burst=50
|
||||
)
|
||||
|
||||
for index in range(3):
|
||||
assert await jev.decide(f"s{index}", {"urgent": URGENT}) is None
|
||||
sent = len(service.requests)
|
||||
|
||||
# Open: the service is not called at all.
|
||||
assert await jev.decide("s-open", {"urgent": URGENT}) is None
|
||||
assert len(service.requests) == sent
|
||||
assert jev.records[-1].outcome == "breaker_open"
|
||||
|
||||
# After the cool-down a single call probes; its success closes the circuit.
|
||||
clock.now += 61
|
||||
service.default = None
|
||||
assert await jev.decide("s-probe", {"urgent": URGENT}) is not None
|
||||
assert await jev.decide("s-after", {"urgent": URGENT}) is not None
|
||||
|
||||
|
||||
async def test_a_failed_probe_reopens_the_circuit():
|
||||
clock = Clock()
|
||||
service = Service(default=httpx2.Response(503))
|
||||
jev = decider(
|
||||
service, clock, retries=0, breaker_failures=2, breaker_cooldown=60, burst=50
|
||||
)
|
||||
for index in range(2):
|
||||
await jev.decide(f"s{index}", {"urgent": URGENT})
|
||||
|
||||
clock.now += 61
|
||||
await jev.decide("probe", {"urgent": URGENT})
|
||||
sent = len(service.requests)
|
||||
|
||||
assert await jev.decide("again", {"urgent": URGENT}) is None
|
||||
assert len(service.requests) == sent
|
||||
|
||||
|
||||
# --- cache and budget --------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_same_question_about_the_same_state_is_answered_from_memory():
|
||||
service = Service()
|
||||
jev = decider(service)
|
||||
|
||||
first = await jev.decide("unchanged", {"route": ROUTE})
|
||||
second = await jev.decide("unchanged", {"route": ROUTE})
|
||||
await jev.decide("changed", {"route": ROUTE})
|
||||
|
||||
assert first == second
|
||||
assert len(service.requests) == 2
|
||||
assert [r.outcome for r in jev.records] == ["answered", "cached", "answered"]
|
||||
|
||||
|
||||
async def test_each_execution_has_a_budget_of_calls_and_then_falls_back(caplog):
|
||||
service = Service()
|
||||
jev = decider(service, budget_calls=2, burst=50)
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
results = [
|
||||
await jev.decide(f"state {index}", {"urgent": URGENT}, scope="exec-1")
|
||||
for index in range(4)
|
||||
]
|
||||
other = await jev.decide("state", {"urgent": URGENT}, scope="exec-2")
|
||||
|
||||
assert [result is not None for result in results] == [True, True, False, False]
|
||||
assert other is not None # another Execution has its own budget
|
||||
assert len(service.requests) == 3
|
||||
assert caplog.text.count("budget reached for exec-1") == 1
|
||||
|
||||
|
||||
async def test_each_execution_has_a_budget_of_tokens():
|
||||
service = Service(default=ok("urgent", tokens=600))
|
||||
jev = decider(service, budget_tokens=1000, burst=50)
|
||||
|
||||
results = [
|
||||
await jev.decide(f"state {index}", {"urgent": URGENT}, scope="exec-1")
|
||||
for index in range(3)
|
||||
]
|
||||
|
||||
assert [result is not None for result in results] == [True, True, False]
|
||||
|
||||
|
||||
async def test_the_key_is_never_logged(caplog):
|
||||
service = Service(httpx2.Response(500), httpx2.Response(401))
|
||||
jev = decider(service, retries=0)
|
||||
|
||||
with caplog.at_level(logging.DEBUG):
|
||||
await jev.decide("a", {"urgent": URGENT})
|
||||
await jev.decide("b", {"urgent": URGENT})
|
||||
|
||||
assert "tsk_secret" not in caplog.text
|
||||
|
||||
|
||||
# --- what is recorded, and running without Jev -------------------------------------------
|
||||
|
||||
|
||||
async def test_every_decision_is_recorded_for_the_evaluation_report():
|
||||
service = Service(ok("urgent"), default=httpx2.Response(400))
|
||||
jev = decider(service)
|
||||
|
||||
await jev.decide("first", {"urgent": URGENT}, scope="exec-1")
|
||||
await jev.decide("first", {"urgent": URGENT}, scope="exec-1")
|
||||
await jev.decide("second", {"urgent": URGENT}, scope="exec-1")
|
||||
|
||||
assert round(jev.records[0].confidence["urgent"], 2) == 0.94
|
||||
assert jev.summary() == {
|
||||
"decisions": 3,
|
||||
"outcomes": {"answered": 1, "cached": 1, "failed": 1},
|
||||
"fallbacks": 1,
|
||||
"rate_limited": 0,
|
||||
"seconds": round(jev.records[0].seconds, 3),
|
||||
"input_tokens": 40,
|
||||
}
|
||||
|
||||
|
||||
async def test_without_a_key_every_caller_takes_its_fallback():
|
||||
unset = open_decider(Settings(_env_file=None, TYPESAFE_API_KEY=None))
|
||||
|
||||
assert isinstance(unset, NoDecider)
|
||||
assert await unset.decide("state", {"urgent": URGENT}) is None
|
||||
|
||||
|
||||
async def test_with_a_key_the_client_takes_its_limits_from_settings():
|
||||
configured = open_decider(
|
||||
Settings(
|
||||
_env_file=None,
|
||||
TYPESAFE_API_KEY="tsk_x",
|
||||
TYPESAFE_BASE_URL="https://jev.internal",
|
||||
JEV_REQUESTS_PER_SECOND=2,
|
||||
JEV_BUDGET_CALLS=40,
|
||||
)
|
||||
)
|
||||
|
||||
assert isinstance(configured, JevDecider)
|
||||
assert configured.limits.requests_per_second == 2
|
||||
assert configured.limits.budget_calls == 40
|
||||
assert configured.limits.max_tokens == 48_000
|
||||
await configured.aclose()
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Local tools: Mermaid check and Terraform fmt/validate/plan in a scratch directory."""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
@@ -140,6 +141,110 @@ async def test_plan_never_runs_apply_and_refuses_unsafe_variables(monkeypatch):
|
||||
assert "-var=region=europe-west2" in plan
|
||||
|
||||
|
||||
# --- schema lookup -----------------------------------------------------------------------
|
||||
|
||||
SCHEMA = {
|
||||
"provider_schemas": {
|
||||
"registry.terraform.io/example/cloud": {
|
||||
"resource_schemas": {
|
||||
"cloud_service": {
|
||||
"block": {
|
||||
"attributes": {
|
||||
"name": {"type": "string", "required": True},
|
||||
"labels": {"type": ["map", "string"], "optional": True},
|
||||
"uri": {"type": "string", "computed": True},
|
||||
},
|
||||
"block_types": {
|
||||
"template": {
|
||||
"nesting_mode": "list",
|
||||
"max_items": 1,
|
||||
"block": {
|
||||
"attributes": {
|
||||
"cpu_boost": {
|
||||
"type": "bool",
|
||||
"optional": True,
|
||||
"description": "Boost CPU at start.",
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
},
|
||||
"cloud_job": {"block": {"attributes": {"name": {"type": "string"}}}},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
PIN = {"cloud": {"source": "example/cloud", "version": "1.2.3"}}
|
||||
|
||||
|
||||
def schema_runner(monkeypatch) -> tuple[TerraformRunner, list[list[str]]]:
|
||||
commands: list[list[str]] = []
|
||||
|
||||
async def fake_run(self, args, cwd, env=None):
|
||||
commands.append(list(args))
|
||||
if args[1] == "init":
|
||||
assert (
|
||||
'cloud = { source = "example/cloud", version = "1.2.3" }'
|
||||
in (Path(cwd) / "versions.tf").read_text()
|
||||
)
|
||||
return 0, "ok"
|
||||
return 0, json.dumps(SCHEMA)
|
||||
|
||||
monkeypatch.setattr(TerraformRunner, "_run", fake_run)
|
||||
return TerraformRunner(providers=PIN), commands
|
||||
|
||||
|
||||
async def test_schema_lists_what_code_may_set_and_outlines_the_blocks(monkeypatch):
|
||||
runner, _ = schema_runner(monkeypatch)
|
||||
|
||||
result = await runner.schema("cloud_service")
|
||||
|
||||
assert result.message.startswith("cloud_service (example/cloud 1.2.3)")
|
||||
assert " name (string, required)" in result.message
|
||||
assert " labels (map of string, optional)" in result.message
|
||||
assert "uri" not in result.message # read-only: code cannot set it
|
||||
assert " template { } (list, at most 1): cpu_boost" in result.message
|
||||
|
||||
|
||||
async def test_schema_gives_one_block_in_full(monkeypatch):
|
||||
runner, _ = schema_runner(monkeypatch)
|
||||
|
||||
result = await runner.schema("cloud_service", block="template")
|
||||
missing = await runner.schema("cloud_service", block="template.containers")
|
||||
|
||||
assert " cpu_boost (bool, optional): Boost CPU at start." in result.message
|
||||
assert "'containers' is not a block there" in missing.message
|
||||
|
||||
|
||||
async def test_schema_says_where_a_setting_sits_or_that_it_cannot_be_set(monkeypatch):
|
||||
runner, _ = schema_runner(monkeypatch)
|
||||
|
||||
found = await runner.schema("cloud_service", find="cpu_boost")
|
||||
absent = await runner.schema("cloud_job", find="cpu_boost")
|
||||
|
||||
assert "template.cpu_boost (bool, optional)" in found.message
|
||||
assert "It cannot be set on this resource type." in absent.message
|
||||
|
||||
|
||||
async def test_schema_suggests_close_resource_types_and_is_loaded_once(monkeypatch):
|
||||
runner, commands = schema_runner(monkeypatch)
|
||||
|
||||
unknown = await runner.schema("cloud_servce")
|
||||
await runner.schema("cloud_job")
|
||||
|
||||
assert "Close matches: cloud_service" in unknown.message
|
||||
assert [command[1] for command in commands] == ["init", "providers"]
|
||||
assert "apply" not in [arg for command in commands for arg in command]
|
||||
|
||||
|
||||
async def test_schema_is_unavailable_without_a_pinned_provider():
|
||||
result = await TerraformRunner().schema("cloud_service")
|
||||
|
||||
assert "pins no Terraform provider" in result.message
|
||||
|
||||
|
||||
# --- registry ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -155,6 +260,7 @@ def test_local_tools_are_built_by_name_with_their_tier():
|
||||
"terraform_fmt",
|
||||
"terraform_validate",
|
||||
"terraform_plan",
|
||||
"terraform_schema",
|
||||
}
|
||||
with pytest.raises(KeyError, match="terraform_apply"):
|
||||
local_tools(["terraform_apply"], TerraformRunner())
|
||||
|
||||
Reference in New Issue
Block a user