This commit is contained in:
@@ -9,8 +9,8 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8 and 15 built; no evaluation run has completed on them; 9 awaits the owner; 10–14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. |
|
||||
| 5 | `iter-05-jev-decisions/` | planned, gated on open questions | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–20 built; no evaluation run has completed on them, so their effect on time and tokens is unmeasured; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; step 11's evaluation run is outstanding, so unproven with real models | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
|
||||
@@ -137,3 +137,35 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Rationale:** Each check runs `terraform init` in a new scratch directory, so every call downloaded its providers again. The setting `TF_PLUGIN_CACHE_DIR` existed and was unset by default.
|
||||
- **Where:** `TerraformRunner` in `app/tools/local.py`, `app/config.py`, `.env.example`.
|
||||
- **Acceptance criteria:** The cache is on by default and can be disabled. A second check of the same configuration does not download the provider. Concurrent checks cannot corrupt the cache.
|
||||
|
||||
## Step 17: A working copy for files and for the graph
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 to shorten a run
|
||||
- **Capability:** A Specialist writes a file or a graph once and afterwards changes only what needs changing. A Revision starts from the last attempt.
|
||||
- **Rationale:** In the 2026-10-07 run the IaC author made 57 model requests and used 2.35 million tokens: every `terraform_fmt` and `terraform_validate` call carried every file, `terraform_fmt` returned them all again, and each fix rewrote whole files. The target architect returned the whole graph and the same content again as tables in one reply, and timed out at 180 seconds in both attempts.
|
||||
- **Where:** `app/tools/workspace.py` (new), `specialist_tools` in `app/tools/execution.py`, `app/tools/local.py`, `app/tools/toolkit.py`, `DeliverableSpec.working_copy` in `app/workflows/deliverables.py`, `render_tables` in `app/workflows/target_architecture.py`, `agents/target-architect.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
|
||||
- **Acceptance criteria:** The Terraform tools take no files when the Deliverable has a working copy, and formatting returns file names only. The working copy is attached on submission. The validator's tools hold the submitted code without it being sent. The target architect can build the graph over several calls, and the tables and the diagram are written by code from the graph. On the next evaluation the IaC author's and the target architect's requests and tokens fall against the 2026-10-07 run.
|
||||
|
||||
## Step 18: Provider schema lookup and one pinned provider version
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** An agent can ask what a resource type accepts, and get the provider's own answer.
|
||||
- **Rationale:** All 13 Terraform refusals in the 2026-10-07 run were one mistake: `startup_cpu_boost` and `cpu_idle` in the wrong block, and on a Cloud Run job, which has neither. The agent moved the argument around by trial. The run also left five versions of the Google provider in the plugin cache, because nothing pinned one.
|
||||
- **Where:** `TerraformRunner.schema` in `app/tools/local.py`, `terraform.required_providers` in `providers/gcp.yaml`, `app/providers.py`, the role files of the target architect, the IaC author and the validator.
|
||||
- **Acceptance criteria:** `terraform_schema` lists a resource type's arguments and blocks, gives one block in full, and says where a named setting sits or that the resource type has none. The schema is read once per process. The bundle pins one exact provider version, and the lookup refuses to guess without one.
|
||||
|
||||
## Step 19: A Check the code cannot meet goes back to the Target architecture
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** A Check that names a setting its resource lacks is fixed where it was written, before any validation is spent on it.
|
||||
- **Rationale:** Check K-30 of the 2026-10-07 run asked for `startup_cpu_boost` on a Cloud Run job. The IaC author could not implement it and had no way to say so; the validator failed it; the code was reopened until the Revision limit ended the run with warnings.
|
||||
- **Where:** `check_implementation` and `unimplementable_checks` in `app/workflows/chain.py`, the playbook in `app/nodes/router.py`, `_orchestrator_context` in `app/nodes/execution_nodes.py`, `agents/orchestrator.md`, `agents/iac-author.md`, `agents/validator.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** `implementation.json` may record a Check as unimplementable with a reason, and that accounts for the Check. The router then reopens the Target architecture with the reason, before validation is delegated. The validator gives a Check on a setting the schema lacks the verdict not verifiable, not fail.
|
||||
|
||||
## Step 20: A model timeout per role
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07
|
||||
- **Capability:** A role whose replies are long is given longer for one model request than the rest.
|
||||
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
|
||||
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
|
||||
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Iteration:** 4, learning and Skills
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8 and 15 built; no evaluation run has completed on them (see `steps/step-8.md`); step 9 drafted and awaiting the owner; steps 10 to 14 planned. Steps 1 to 6 are in commit `7ee49ab`; 7, 8 and 15 are uncommitted.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; no evaluation run has completed on any of them (see `steps/step-8.md`), so every "tokens fall" or "Revisions fall" criterion is unmeasured; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 16
|
||||
**This directory covers:** steps 1 to 20
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -27,12 +27,16 @@
|
||||
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
|
||||
| 9 | ADR: the lessons loop and overlay Skills | B | S | drafted as `docs/adr/0004-lessons-and-overlays-arrive-as-skills.md`, status proposed; awaits the owner |
|
||||
| 10 | Lessons Skill and lesson curator | B | L | planned |
|
||||
| 11 | IaC patterns Skill | B | M | planned |
|
||||
| 11 | IaC patterns Skill | B | M | built; every pattern validates; effect on Revisions and tokens unmeasured |
|
||||
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
|
||||
| 13 | Trim Specialist context | C | M | planned |
|
||||
| 13 | Trim Specialist context | C | M | built; effect on tokens unmeasured |
|
||||
| 14 | Provider-owned contracts | C | M | planned |
|
||||
| 15 | Group Executions by Customer case | added | S | complete |
|
||||
| 16 | Cache Terraform providers between checks | added | S | complete |
|
||||
| 17 | A working copy for files and for the graph | added | M | built; effect on time and tokens unmeasured |
|
||||
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
@@ -49,5 +53,7 @@ Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that
|
||||
- Step 14: depends on step 4
|
||||
- Step 15: no dependencies (added on 2026-10-06 at the owner's request)
|
||||
- Step 16: no dependencies (added on 2026-10-07 at the owner's request)
|
||||
- Steps 17 to 20: no dependencies (added on 2026-10-07 at the owner's request, from the causes of the 60 minute run recorded in `../iter-06-target-architecture-graph/steps/`)
|
||||
- Step 11 uses step 18's pinned provider version; step 19 uses step 18's lookup to find the Checks it reports
|
||||
|
||||
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.
|
||||
|
||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-11.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-11.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 11 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07 as `skills/google-cloud-iac-patterns/`. The name carries the Cloud provider because the content does; the provider bundle assigns it to the IaC author (`role_skills.iac-author`), so another provider brings its own patterns Skill with no code change.
|
||||
|
||||
1. `SKILL.md` says how to use the patterns and lists one reference per product. `assets/versions.tf` pins `hashicorp/google` 8.6.0, the version `providers/gcp.yaml` pins (step 18).
|
||||
2. Thirteen references, each a short list of what goes wrong and one self-contained pattern: enabling APIs, IAM, Cloud Run service, Cloud Run job, Pub/Sub, Cloud Storage, Secret Manager, Cloud KMS, Cloud SQL, VPC networking, Cloud Scheduler, Artifact Registry, Monitoring and Logging.
|
||||
3. Seeded from the 2026-10-07 run's lesson candidates. All 13 Terraform refusals were `startup_cpu_boost` or `cpu_idle` in the wrong block or on a Cloud Run job. The service reference says which block each setting sits in; the job reference lists what a job does not have.
|
||||
|
||||
Differs from the plan: the plan's acceptance is a fall in IaC author Revisions or tokens on the three cases. **No evaluation has run, so that is unmeasured.** What is verified is that the patterns are correct.
|
||||
|
||||
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
|
||||
|
||||
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-13.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-13.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 13 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `DeliverableSpec.reads_in_full` names the earlier Deliverables a Specialist reads whole. Left unset it is the Deliverables it depends on.
|
||||
2. `dependencies_for(kind, deliverables)` gives every other input as its data file alone: `requirements.json`, `components.json` and so on. Those ids are all the Requirements chain checks need, so submission checks are unchanged.
|
||||
3. Two overrides. The IaC author reads nothing whole: the code is written from `target-architecture.json`. The packager reads everything whole, since the guide is assembled from every Deliverable's prose.
|
||||
4. A Deliverable given as data can be read whole with `read_deliverable(kind)`, offered only when something was trimmed.
|
||||
|
||||
Differs from the plan: the plan said "summaries". The data file is used instead. It already exists, is exact, and costs no model call to produce.
|
||||
|
||||
**Unmeasured:** the plan's acceptance is that describer tokens fall against the step 8 baseline. No evaluation has run.
|
||||
|
||||
Tests: `tests/execution/test_working_copy.py`, "each Specialist reads what it works from".
|
||||
21
.agents/plans/iter-04-learning-and-skills/steps/step-17.md
Normal file
21
.agents/plans/iter-04-learning-and-skills/steps/step-17.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Step 17 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07. `app/tools/workspace.py` keeps a working copy in the Deliverable's working memory, which the Execution already checkpoints and returns on every attempt.
|
||||
|
||||
Files (`working["files"]`), for the Deliverable whose `working_copy` is its own kind (deployment guidance):
|
||||
|
||||
1. `write_file`, `edit_file` (replaces the one place `old` matches, and refuses when it matches none or several), `read_file` (one file, or the list) and `delete_file`.
|
||||
2. `terraform_fmt`, `terraform_validate` and `terraform_plan` run on the Terraform files in the working copy and take no `files` argument. `terraform_fmt` formats in place and returns the names it changed. Files passed anyway are written to the working copy first, so a model that ignores the instruction still works.
|
||||
3. `submit_deliverable` attaches the working copy; a file passed as an attachment replaces its copy. The submitted files become the working copy, so a Revision starts from them, and the task context lists them under `working_copy`.
|
||||
|
||||
The validator's `working_copy` is deployment guidance: its tools are given a fresh copy of the submitted files on every attempt, it has no write tools, and nothing from the copy is attached to its Deliverable.
|
||||
|
||||
Graph (`working["graph"]`), for the Target architecture:
|
||||
|
||||
4. `graph_put` adds items or replaces those with the same id, `graph_remove` removes by id, `graph_read` returns the graph. The submitted graph becomes the working graph.
|
||||
5. `submit_deliverable` attaches the working graph when no `target-architecture.json` is passed.
|
||||
6. `render_tables` writes the Elements, Relationships, Checks and not-applicable tables from the graph, and the engine appends them to the content on submission. The contract's sections are now `Design rationale` alone. This closes the duplicate output iteration 6 left: the architect no longer writes the graph twice.
|
||||
|
||||
**Unmeasured:** no evaluation has run, so the fall in requests, tokens and time is expected, not shown. The mechanism is what is tested.
|
||||
|
||||
Tests: `tests/execution/test_working_copy.py` (21), including an Execution in which the validator validates the submitted code with no files passed.
|
||||
17
.agents/plans/iter-04-learning-and-skills/steps/step-18.md
Normal file
17
.agents/plans/iter-04-learning-and-skills/steps/step-18.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Step 18 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `providers/gcp.yaml` pins `terraform.required_providers.google` to `hashicorp/google` 8.6.0. `load_provider` requires a `source` and an exact `version`.
|
||||
2. `TerraformRunner.schema(resource_type, block, find)` writes a versions file for the pinned providers in a scratch directory, runs `terraform init -backend=false` and `terraform providers schema -json` once per process, and keeps the resource schemas in memory. Measured here: 0.6 seconds and a 9.7 MB reply for the Google provider, from the plugin cache.
|
||||
3. With `resource_type` alone it lists the arguments code may set (read-only ones are left out) and outlines every block. With `block` it gives one block's arguments with descriptions. With `find` it says where a setting sits, or that the resource type has none. An unknown resource type gets close matches.
|
||||
4. The tool is `terraform_schema`, tier general, given to the target architect, the IaC author and the validator.
|
||||
|
||||
Checked against the real provider on 2026-10-07:
|
||||
|
||||
- `find="startup_cpu_boost"` on `google_cloud_run_v2_job`: "has no argument or block named like 'startup_cpu_boost'. It cannot be set on this resource type."
|
||||
- the same on `google_cloud_run_v2_service`: `template.containers.resources.startup_cpu_boost (bool, optional)`.
|
||||
|
||||
Not done: generated code is not forced onto the pinned version. An agent that writes its own `required_providers` with another version is validated against that version. The patterns Skill gives it the pinned `versions.tf` to use.
|
||||
|
||||
Tests: `tests/tools/test_local.py`, "schema lookup", on a stand-in schema; `tests/test_providers.py` for the pin.
|
||||
12
.agents/plans/iter-04-learning-and-skills/steps/step-19.md
Normal file
12
.agents/plans/iter-04-learning-and-skills/steps/step-19.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Step 19 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. An `implementation.json` entry may be `{"check": "K-7", "unimplementable": "<what the resource type lacks>"}` in place of `file` and `address`. It accounts for the Check; an empty reason does not.
|
||||
2. `unimplementable_checks` reads those entries. The router reads them as soon as deployment guidance is submitted, and a new playbook group, "the code cannot meet a Check", sits before "validation decided" and "a Phase is complete". Its action reopens the Target architecture with each Check's id and reason. That returns the code to draft, so the group stops applying.
|
||||
3. A model Orchestrator is given the same list as `unimplementable_checks` in its context, and `agents/orchestrator.md` has the step.
|
||||
4. The validator is told to confirm with `terraform_schema` before failing a Check for a missing setting, and to give not verifiable when the resource type has no such setting. Not verifiable already routes to the Target architecture.
|
||||
|
||||
Past the Revision limit the reopen is refused as any other is: the user decides, or a pre-approved Execution ends with warnings.
|
||||
|
||||
Tests: `tests/execution/test_requirements_chain.py` (two) and `tests/execution/test_router.py`, "a Check the code cannot meet goes back to the graph unvalidated".
|
||||
@@ -0,0 +1,7 @@
|
||||
# Step 20 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07. `models.timeout_seconds` in the provider bundle maps a role to the seconds one model request may take; `RoleSpec.model_timeout_seconds` carries it and `AdkAgentRuntime` uses it in place of `MODEL_TIMEOUT_SECONDS`.
|
||||
|
||||
`providers/gcp.yaml` sets the target architect to 420 and the IaC author and the packager to 300. These are starting values, not measurements. With step 17 the architect's replies are short, so its 420 should rarely be reached.
|
||||
|
||||
Tests: `tests/agents/test_adk_runtime.py`, `tests/agents/test_roles.py`, `tests/test_providers.py`.
|
||||
@@ -135,6 +135,12 @@ Rules for every step from 3 onward:
|
||||
6. A generative model writes only what the user reads: an Approval summary, or a question of the Orchestrator's own, from the decision and the data. It runs after the decision and cannot change it. A Revision's reason is the user's words or the failed Checks.
|
||||
7. A failed Jev call is recorded as a failure, distinct from an answer, and retried within a fixed budget before the fallback.
|
||||
- **Pattern followed:** https://vercel.com/i/jev-agent-control. Its ownership split maps as: propose the next action, Jev; check permission, the Ledger; execute, the existing tools; update state, the Execution graph; explain the result, a generative model. Its specialist routing is not used, because the dependency graph fixes which Specialist produces each Deliverable.
|
||||
- **Routing (owner's direction, 2026-10-07):** the Orchestrator acts as a router with Jev choosing the route. Four things are routed:
|
||||
- a message that starts something: new Workflow request, Change request against a Baseline, question about delivered work, or out of scope (Choice);
|
||||
- a message inside an Execution: an answer to a waiting question, a request to change a named Deliverable, or anything else (Choice, then a Choice over the Deliverables);
|
||||
- a validation outcome: revise the code, revise the Target architecture, or proceed (Choice);
|
||||
- in a Change request, which Deliverables the change reaches (a Boolean per Deliverable, with the Requirements chain as the deterministic check).
|
||||
The route decides what is dispatched. Which Specialist produces a Deliverable stays fixed by the dependency graph, and every Deliverable is still produced in a new Execution. The first route needs an entry point that can start a Change request or answer from a case's files (`app/main.py`, `app/a2a_server.py`); the last depends on iteration 7.
|
||||
- **Fallbacks:** Below the confidence threshold, an interactive Execution asks the user and a pre-approved one takes the safest allowed action with a warning. When Jev is unreachable, the fixed order of the seven steps is used and recorded.
|
||||
- **Where:** `orchestrator_turn` in `app/nodes/execution_nodes.py`; `app/tools/execution.py`; `agents/orchestrator.md` becomes question wording and fixed text in a Skill or the bundle; `app/adk/roles.py` and `providers/gcp.yaml` no longer need an Orchestrator model.
|
||||
- **What is sent to Jev:** state, not content: Deliverable names and statuses, Check ids and verdicts, the user's latest message.
|
||||
@@ -169,3 +175,33 @@ Rules for every step from 3 onward:
|
||||
- **Fallback:** Treat it as changed.
|
||||
- **Where:** the graph comparison from iteration 7 step 4 (`app/workflows/target_architecture.py`), the change summary from iteration 7 step 6.
|
||||
- **Acceptance criteria:** A reworded but equivalent Check is not reported as changed; a Check that lost or gained a condition is. Every decision is recorded with its probability in `changes.json`.
|
||||
|
||||
## Step 16: Jev as the evaluation judge, calibrated
|
||||
|
||||
- **Capability:** The evaluation's judged checks are answered by Jev, and the judge is shown to agree with a person.
|
||||
- **Rationale:** LangChain's evaluation posts report Jev agreeing with a human reviewer on every pass-or-fail decision in a small experiment, with far lower variance than language-model judges, at $0.00035 a call. They also warn that low cost can amplify mistakes and that human review and judge alignment are still needed.
|
||||
- **Built:** `DeciderJudge` in `eval/judge.py` asks every question in one call, with the document as the state and no grading instructions in it. `eval/solution_eval.py --judge` uses it when `TYPESAFE_API_KEY` is set, and the language-model judge otherwise or with `EVAL_JUDGE=gemini`. A judge that gives no answer leaves its check unanswered, not passed.
|
||||
- **Open:** a small set of design recommendations labelled by a person against the pillar checklists and the quality rubric; agreement of each judge with those labels; variance over repeated runs on the same output. This replaces the stability criterion in step 10.
|
||||
- **Acceptance criteria:** On the labelled set, the Jev judge's pass-or-fail agreement and its variance are recorded beside the language-model judge's, and the evaluation uses the one that agrees better.
|
||||
|
||||
## Step 17: Trace every decision
|
||||
|
||||
- **Capability:** For any Execution, a person can see each decision, what it was given and what came back.
|
||||
- **Rationale:** LangChain's production post treats tracing as required for model-driven steps and describes LangSmith views for decision models. Here decisions are kept in the Execution's state and the evaluation report, and nothing is traced.
|
||||
- **Owner's decision needed:** whether to send traces to LangSmith. Traces carry the Workflow request and Deliverable text, so the same retention question as Jev applies. `langsmith` is already installed as a dependency of LangChain; LangGraph traces when its environment variables are set.
|
||||
- **Where:** environment settings for tracing; a traced span around `JevDecider.decide` in `app/tools/jev.py` carrying the questions, answers and probabilities; `docs/operations/`.
|
||||
- **Acceptance criteria:** One Execution's trace shows every Orchestrator turn, each Jev call with its inputs and outputs, and each Specialist run. With tracing off, nothing leaves the process.
|
||||
|
||||
## Step 18: Readings as their own graph step
|
||||
|
||||
- **Capability:** A resumed Execution reuses the decisions it already made.
|
||||
- **Rationale:** LangGraph saves state after each node. The router takes its readings and acts inside one node, so a failure in that node asks Jev again on retry. Jev is designed to return the same answer for the same input, and the client caches answers in memory, but neither survives a restart.
|
||||
- **Where:** `app/workflows/execution_graph.py` and `app/nodes/`: a node that takes the readings and saves them, before the node that applies the playbook.
|
||||
- **Acceptance criteria:** A test that fails the playbook node after the readings node shows the retry using the saved readings, with no second call to the decider.
|
||||
|
||||
## Step 19: Online evaluation of live Executions
|
||||
|
||||
- **Capability:** Every Execution is scored as it runs, not only benchmark cases.
|
||||
- **Rationale:** At Jev's cost, LangChain recommends scoring every run and alerting on risk flags such as leaked personal data or injected instructions.
|
||||
- **Where:** the judged checks from `eval/scoring.py`, run when an Execution ends and stored with its record; with step 17, as feedback on its trace.
|
||||
- **Acceptance criteria:** A completed Execution carries its judged scores. A flagged risk is recorded and surfaced without stopping the Execution.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** planned; gated on step 1. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 15
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 19
|
||||
|
||||
## What Jev is
|
||||
|
||||
@@ -49,6 +49,24 @@ On 2026-10-07 the owner named the nine use cases at https://vercel.com/i/jev-use
|
||||
|
||||
The page's own cautions apply to every step: Jev does not write replies, so a generative model drafts messages; exact conditions (a Dependency is met, a file is attached, an id exists) stay deterministic checks in code; and a typed answer can still be wrong, so start where outcomes can be inspected and corrected, and record every decision.
|
||||
|
||||
## Alignment with LangChain's published guidance
|
||||
|
||||
Checked on 2026-10-07 against three LangChain posts: "Building prod with Jev and LangGraph", "Jev agent evals with LangSmith" and "Jev is now available in LangSmith evals". Steps 16 to 19 close the gaps found.
|
||||
|
||||
| Their guidance | Here | Step |
|
||||
|---|---|---|
|
||||
| Code owns the workflow; Jev makes narrow, typed decisions | The Ledger and the playbook are code; Jev supplies readings | 12 |
|
||||
| Encode domain knowledge in the graph's topology, not in prompts | The dependency graph, Phases and playbook are data and code | 12 |
|
||||
| Several atomic questions against one state, in one call | Readings are batched per turn | 2, 12 |
|
||||
| Language models for open-ended work | Specialists write Deliverables; a model writes replies | 12 |
|
||||
| Humans in the loop through interrupts | Approvals and every user question are interrupts | done |
|
||||
| Checkpointing, so a failed run resumes with the decisions it made | The Execution is checkpointed; a decision is saved only when its turn ends | 18 |
|
||||
| Low confidence falls back to a language model (their example: under 0.7) | Low confidence goes to the user, by the owner's choice in ADR-0007 | differs on purpose |
|
||||
| Trace every decision, with its inputs and calibrated outputs | Decisions are kept in Execution state and the evaluation report; nothing is traced | 17 |
|
||||
| Jev as the judge: typed questions, no grading instructions in the state | `DeciderJudge` in `eval/judge.py`, used when a key is set | 16 |
|
||||
| Check a judge against human labels, and for repeatability | Not done; there is no labelled set | 16 |
|
||||
| Score every live run and alert on risk flags | Evaluation is offline only | 19 |
|
||||
|
||||
## Boundaries
|
||||
|
||||
Jev answers closed questions. Writing Deliverables, grounding product choices and producing infrastructure code stay with the large models. Dependencies and Approvals stay rules in the Ledger.
|
||||
@@ -58,7 +76,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | planned |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
@@ -67,11 +85,15 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 8 | Skill routing | J9 | planned |
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | drafted as `docs/adr/0007-jev-is-the-orchestrators-decision-point.md`, status proposed; awaits the owner |
|
||||
| 12 | Jev proposes the Orchestrator's next action; a model only writes replies | J1 | planned; the owner's priority for this iteration (2026-10-07) |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -90,5 +112,9 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
- Step 13: depends on step 2
|
||||
- Step 14: depends on step 2
|
||||
- Step 15: depends on step 2, iteration 7 step 4
|
||||
- Step 16: depends on step 2
|
||||
- Step 17: no dependencies
|
||||
- Step 18: depends on step 12
|
||||
- Step 19: depends on step 16, step 17
|
||||
|
||||
Steps 3 and 4 come first: they are small, need no ADR change, and target Revisions, where the tokens are. The Orchestrator is 11% of case 03's tokens; the IaC author and validator together used 1.43 million over 7 runs.
|
||||
|
||||
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
9
.agents/plans/iter-05-jev-decisions/steps/step-11.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 11 — Implementation Detail
|
||||
|
||||
ADR-0007 was accepted by the owner on 2026-10-07 ("this is exactly what I want, I approve ADR-0007"), after three refinements made the same day and recorded in it:
|
||||
|
||||
1. Jev is the decision point, in place of a different language model.
|
||||
2. The agent-control pattern at https://vercel.com/i/jev-agent-control is followed: a decision model reads, application code checks permission and executes, a generative model writes replies.
|
||||
3. The Orchestrator is a router, and decides by readings, a disposition and a playbook, on the model of a triage pipeline the owner supplied: the first group that applies, the strongest action whose conditions hold, escalate when none applies.
|
||||
|
||||
Jev is reached through TypeSafe AI's Python SDK and API endpoint (owner, 2026-10-07). The API key is still to be supplied.
|
||||
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
36
.agents/plans/iter-05-jev-decisions/steps/step-12.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Step 12 — Implementation Detail
|
||||
|
||||
Built against a scripted decider. No call has been made to the live Jev service, and no evaluation has run on it: the owner holds evaluation runs until the Jev work is in.
|
||||
|
||||
1. **`app/nodes/router.py`.** `Router` decides an Orchestrator turn in three stages: readings (`_read_ledger` from the Ledger and data; `_read_message` and `_read_gap` from the decider), a disposition by `Thresholds` (act at 0.75, close at 0.15, choice confidence 0.6), and the playbook in `_choose`: ordered groups, the strongest action whose conditions hold, escalate when none applies.
|
||||
2. **Permission stays with the tools.** The router calls the same `OrchestratorToolbox` tools the model did. A refused action is skipped for the rest of the turn and the next one that holds is taken. Past the Revision limit the user is asked.
|
||||
3. **What Jev is asked** (`Question` objects, batched when they rest on the same evidence):
|
||||
- of the user's message: `asks_change` (noul), `change_target` (choice over submitted Deliverables), `change_clarity` (score: vague, partial, specific);
|
||||
- of validation gaps: `gap_material` (noul);
|
||||
- of the user's reply about gaps: `wants_revision` (noul).
|
||||
What is sent is Deliverable names, the count of waiting questions, the user's message, and for gaps the proposed Check statements and unverifiable Check ids.
|
||||
4. **Rules when Jev gives no answer.** A requested change is acted on only when one Deliverable is named outright, otherwise the user is asked which. A free-text reply is treated as an answer. A gap is revised for when a Check cannot be verified or an addition names a real requirement or recommendation id.
|
||||
5. **A new reading from the run of 2026-10-07:** a Check that fails in two validations running is sent to the Target architecture, not the code. That run retried one unimplementable Check as code three times.
|
||||
6. **Decisions are recorded** in the Execution's state under `decisions`: turn, group, action and what was judged, with confidence.
|
||||
7. **The reply writer.** A new role, `agents/reply-writer.md`, writes the Approval summary from facts (`approval_facts`) through one tool. When it fails or is absent, `assembled_summary` builds the summary from the same facts.
|
||||
8. **Wiring.** `ORCHESTRATOR=router` (the default) or `model`; `open_assembly` builds the router with the decider and the thresholds (`JEV_ACT_THRESHOLD`, `JEV_CLOSE_THRESHOLD`, `JEV_CONFIDENCE_THRESHOLD`). State gained `last_reply`, `pending_change`, `awaiting_gap_decision`, `validation_settled` and `decisions`.
|
||||
|
||||
Changed from the plan and the ADR's first wording:
|
||||
|
||||
- **Jev does not propose the next action.** It supplies readings, and the playbook in code picks the action. This follows the owner's triage model and keeps every decision explainable by a rule and a threshold.
|
||||
- **The model Orchestrator is kept** behind `ORCHESTRATOR=model`. The 40 or so engine tests drive the Ledger's rules by scripting its tool calls in arbitrary order, which the router cannot do. `agents/orchestrator.md` is therefore unchanged and unused in router mode.
|
||||
- **An unclear change is confirmed once.** If the reply still does not settle it, a named Deliverable is acted on and anything else is dropped.
|
||||
- **A pre-approved Execution's automatic replies are not routed.** There is no user message to read.
|
||||
|
||||
Not built from the ADR's routes:
|
||||
|
||||
- **A message that starts something** (new request, Change request, question about delivered work, out of scope). It needs an entry point that can answer from a case's files, and Change requests (iteration 7).
|
||||
- **Which Deliverables a Change request reaches.** Iteration 7.
|
||||
|
||||
Tests: `tests/execution/test_router.py` (18), `tests/test_assembly.py`.
|
||||
|
||||
Unverified:
|
||||
|
||||
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
|
||||
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
|
||||
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
|
||||
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
44
.agents/plans/iter-05-jev-decisions/steps/step-2.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
Built ahead of step 1's answers, on the owner's instruction of 2026-10-07 that no evaluation be rerun until the Jev work is in. Nothing here has called the live service.
|
||||
|
||||
1. **The seam** (`app/adk/runtime.py`): `Question` (kind choice, score or noul, with instructions and options), `Answer` (value, confidence, probabilities) and the `Decider` protocol. `decide` returns the answers, or `None` when no decision could be made, and the caller then takes its fallback. `NoDecider` always returns `None`.
|
||||
2. **The client** (`app/tools/jev.py`): `JevDecider` calls Jev through TypeSafe AI's official Python SDK, `typesafe-sdk` 0.7.2 (`AsyncTypeSafeClient.system_one`). The owner chose the SDK and its API endpoint on 2026-10-07. It is the only module that imports the SDK.
|
||||
3. **Wiring:** `open_decider` in `app/agent.py` returns the client when `TYPESAFE_API_KEY` is set and `NoDecider` otherwise; the `Assembly` carries it.
|
||||
4. **A scripted decider** for tests, `FakeDecider` in `app/adk/fake_runtime.py`.
|
||||
|
||||
Which package: three names exist on the index. `typesafe-sdk` is the vendor's (author TypeSafe AI, repository `typesafe-ai/typesafe-sdk-python`). `typesafe` is an unrelated type-checking library, and `typesafe-ai` is a third party's redirect shim; neither is used.
|
||||
|
||||
What the SDK settled, read from the installed package and a stand-in transport:
|
||||
|
||||
- The request is `POST /v1/systemone` with a bearer key; default endpoint `https://api.typesafe.ai`, default model `jev-latest`.
|
||||
- Questions are `Noul(instructions)`, `Choice(instructions, criteria={option: description})` and `Score(instructions, criteria=[levels])`.
|
||||
- Answers are `noul` (a probability, with no confidence of its own), `choice` with `confidence` and `probabilities`, and `score` with `confidence`. Usage reports `input_tokens` only: no cost and no remaining credit.
|
||||
|
||||
Limits kept, each with a test in `tests/tools/test_jev.py`:
|
||||
|
||||
| Practice | Built as |
|
||||
|---|---|
|
||||
| Size | Refused locally above 48,000 estimated tokens in all, or 24,000 for the state plus the longest question. Never truncated. |
|
||||
| Batching | Every question about one state goes in one request. |
|
||||
| Timeout | 5 seconds a request, 15 for a decision with its retries (the SDK's). |
|
||||
| Retries | 2, by the SDK: on 408, 429, 5xx, connection and timeout errors, with jittered backoff, honouring `Retry-After`. |
|
||||
| Rate | A token bucket, 5 a second with a burst of 10; the rate halves when a rate limit outlasts the retries. |
|
||||
| Concurrency | At most 8 requests in flight. |
|
||||
| Circuit breaker | Opens after 5 failures in 30 seconds for 60 seconds, then lets one probe through. |
|
||||
| Cache | Answers kept by a hash of the state and questions, up to 1,024. |
|
||||
| Budget | 500 calls or 1,000,000 input tokens per scope (the Execution), then fallbacks with one warning. |
|
||||
| Records | Each decision's outcome, seconds, rate limiting, input tokens and confidence; `summary()` totals them. |
|
||||
| Key | From `TYPESAFE_API_KEY` only; never logged. |
|
||||
|
||||
Settings: `TYPESAFE_API_KEY`, `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL` (the SDK's own names), and `JEV_TIMEOUT_SECONDS`, `JEV_REQUESTS_PER_SECOND`, `JEV_MAX_IN_FLIGHT`, `JEV_BUDGET_CALLS`, `JEV_BUDGET_TOKENS`.
|
||||
|
||||
Changed from the plan:
|
||||
|
||||
- The cache is per client, not per Execution. A decision depends only on its state and questions, so two Executions asking the same thing can share the answer.
|
||||
- A reply that leaves any question unanswered, or picks a choice that was not offered, counts as no decision for the whole request.
|
||||
- The budget is in calls and input tokens, not dollars, and there is no low-credit warning: the SDK reports neither cost nor balance.
|
||||
- A yes-or-no answer's confidence is derived as its distance from an even probability, because Jev returns none.
|
||||
- Retries and their backoff are the SDK's, configured here, not reimplemented.
|
||||
|
||||
Unverified: the client has run only against a stand-in transport, with the real SDK in the loop. The first call with a real key is still to be made, and the service's actual rate limits, latency and data retention are still unknown.
|
||||
Reference in New Issue
Block a user