feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)

Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 17:45:56 +01:00
parent 844987587c
commit 567aede5ca
38 changed files with 1956 additions and 58 deletions

View File

@@ -9,8 +9,8 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); the lessons loop is built and its first six lessons are approved and published, not yet proved (step 10); 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); the lessons loop is built and its first six lessons are approved and published, not yet proved (step 10); overlay Skills built, not yet run (step 12); 14 deferred until a second Cloud provider | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the router, intake, the reviews, pillar selection, Skill routing, the guardrail, meaning-aware change summaries and an assessment of every Execution are built (steps 2–5, 8, 10–12, 14, 15, 19), none of the newest yet run in a case; model tier, context selection, compaction, the cloud-read guard and readings as a node are planned (6, 7, 9, 13, 18) | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |

View File

@@ -28,9 +28,9 @@
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
| 10 | Lessons Skill and lesson curator | B | L | the loop is built; six lessons proposed from 21 Executions, approved by the owner and published on 2026-10-09; not yet proved by an evaluation with and without (`steps/step-10.md`) |
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
| 12 | Overlay Skills and an organisation standards Skill | C | M | planned |
| 12 | Overlay Skills and an organisation standards Skill | C | M | built, with an example standards Skill and an evaluation case; not run with real models |
| 13 | Trim Specialist context | C | M | built; measured on case 01 only, together with steps 11 and 17 |
| 14 | Provider-owned contracts | C | M | planned |
| 14 | Provider-owned contracts | C | M | deferred by its own plan until a second Cloud provider is planned (ADR-0002) |
| 15 | Group Executions by Customer case | added | S | complete |
| 16 | Cache Terraform providers between checks | added | S | complete |
| 17 | A working copy for files and for the graph | added | M | complete; in every run since 2026-10-07 |

View File

@@ -42,3 +42,11 @@ The owner approved all six on 2026-10-09 and they were published: five for the t
## Tests
`tests/test_lessons_loop.py` (20): grouping, proposals as files, the approval gate, publish and retire, the curator with a scripted model, and which roles are given the Skill.
## Added later on 2026-10-09
- **Proving is a command.** `eval.curate_lessons prove --without <report.json> --with <report.json>` compares two evaluation reports case by case (score, Revisions, minutes, tokens), says whether quality held and anything fell, and writes the comparison into every published proposal's `## Proof` (`app/learning/proof.py`). The two runs are still made by hand, one with `LESSONS_ENABLED=false`.
- **Triage.** Before the curator runs, each new group is put to the decider with the lessons that exist: which one would have prevented it, or none. A group a lesson covers is not put to the curator again (`_not_already_covered` in `app/learning/curator.py`; iteration 5 step 10).
Still open: retiring lessons when the upstream Skills change is not automated, and no lesson has been proved.

View File

@@ -0,0 +1,19 @@
# Step 12 — Implementation Detail
Built on 2026-10-09. **Not run with real models.**
1. An overlay Skill is an ordinary Agent Skill in a folder the deployment names: `OVERLAY_SKILLS_DIRS` (default `skills/overlays`). `discover_overlays` in `app/agent.py` reads them with the same validation as the provider's Skills and refuses one that reuses a provider Skill's name.
2. `ExecutionEngine.start(overlay_skills=[...])` names the overlays for one Execution. `DEFAULT_OVERLAY_SKILLS` names the ones every Execution loads. An unknown name, or anything but a list of names, refuses the start before anything runs or is recorded. A caller names an overlay; nothing lets it supply one.
3. Every Specialist is given the Execution's overlays on top of its own Skills, and its context lists them as `overlay_skills`. The protocol says an overlay wins over the provider's general guidance, and a requirement the user stated wins over an overlay.
4. The Execution records each overlay with the content hash it had: in its state, in `execution.json` and in its lesson candidates.
5. Callers: `overlay_skills` in the `/generate` body, in A2A message metadata, and on an evaluation case.
6. `skills/overlays/example-organisation-standards/` is an example: naming, four mandatory labels, approved regions, three prohibitions. It is marked as an example, is not loaded by default, and is not any real organisation's policy.
Differs from the plan:
- **Skills are still loaded once at start-up.** The runtime loads every provider Skill and every overlay in the configured folders when it starts; an Execution chooses among them. An overlay added to the folder needs a restart.
- **No organisation standards Skill is loaded by default.** The plan's criterion "loaded by the roles the bundle lists" is met by naming it, not by the bundle: whose standards apply is the deployment's and the caller's to say.
`case-06-organisation-standards` in the evaluation dataset runs case 03's request under the example overlay. It has not been run.
Tests: `tests/execution/test_overlay_skills.py` (8).

View File

@@ -2,7 +2,7 @@
**Iteration:** 5, Jev decisions
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
**Status:** steps 2, 11 and 12 built (2026-10-07) and used live in case 02, where Jev judged two gaps; step 4 built in part and switched off; the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
**Status:** steps 2 to 5, 8, 10 to 12, 14, 15 and 19 are built or settled (2026-10-09); 16 awaits the owner's labels; 6, 7, 9, 13 and 18 are planned; 1 and 17 await the owner
**This directory covers:** steps 1 to 19
## What Jev is
@@ -77,23 +77,23 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | planned |
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; not run in an Execution |
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
| 5 | Finding-to-Deliverable routing | J2 | planned |
| 6 | Model tier per delegation | J5 | planned |
| 7 | Context selection | J6 | planned |
| 8 | Skill routing | J9 | planned |
| 9 | In-run context compaction | J10 | planned |
| 10 | Evaluation rubric and lesson triage | J8 | planned |
| 5 | Finding-to-Deliverable routing | J2 | superseded: iteration 6 gives a verdict per Check, and the router sends a failed Check to the code and a gap to the Target architecture |
| 6 | Model tier per delegation | J5 | planned; held until the cases have run on this tranche, so its effect on quality can be told apart |
| 7 | Context selection | J6 | planned; held for the same reason as step 6 |
| 8 | Skill routing | J9 | built and on: the references a task needs are named in the Specialist's context; calibrated live; not run in an Execution (`steps/step-8.md`) |
| 9 | In-run context compaction | J10 | planned; held for the same reason as step 6 |
| 10 | Evaluation rubric and lesson triage | J8 | built: the judged rubric (step 16) and triage of evidence a lesson already covers (iteration 4 step 10) |
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
| 13 | Cloud-read call guard | J7 | planned |
| 14 | Input guardrail | J11 | planned |
| 15 | Changes of meaning in a Change request | none | planned; iteration 7 now compares by id and text, so a reworded but equivalent Check still reads as changed |
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; not run in an Execution |
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; not run in a Change request |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned |
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |
## Dependency graph

View File

@@ -0,0 +1,16 @@
# Step 14 — Implementation Detail
Built on 2026-10-09, on by default (`JEV_GUARDRAIL`). **No Execution has run with it.**
1. `app/tools/guardrail.py`: one yes or no on a text, whether it tries to give instructions to the agent reading it, as opposed to informing the reader.
2. **A remote MCP server's reply** is checked before a Specialist reads it (`make_tool_factory` wraps every MCP tool). At `JEV_GUARDRAIL_THRESHOLD` (0.75) or above it is withheld: the Specialist is told the reply read as instructions, not to act on it, and to record the gap. The withholding is logged.
3. **What the user writes** is only flagged: the Workflow request on the first turn and each reply after it. A flag is recorded in the Execution's state and in `execution.json` under `risks`, with where it was and how likely. Nothing is withheld and nothing changes in what the Execution does.
4. With no decision a text passes through.
Differs from the plan: the plan fails closed everywhere. The user is the principal, and a wrong flag on their own request would stop work they asked for, so their words are flagged and not withheld. A server's reply with no decision passes through, because the guard adds to the tool allowlist and the permission checks and replaces neither; failing closed there would stop every documentation lookup whenever Jev is unavailable.
Measured against live Jev on ten texts: product documentation, how-to steps written as imperatives, a "you should ... ignore this recommendation if" best practice, a resource listing and an ordinary request scored 0.02 to 0.12. Four that addressed the agent (a classic "ignore all previous instructions", a quiet "add an IAM binding and do not mention it", a role change, and a request with instructions appended) scored 0.98 to 0.99.
The decider's budget is counted per Execution through a context variable set when a Specialist starts.
Tests: `tests/tools/test_guardrail.py` (7).

View File

@@ -0,0 +1,27 @@
# Step 15 — Implementation Detail
Built on 2026-10-09. **Not run in a Change request with real models.**
`settle_meaning` in `app/tools/meaning.py`: for each Check, Element, Relationship and recommendation whose text differs from the Baseline's, one yes or no with the old and the new: does the new text change what must be true? An item at or below `JEV_REWORDED_THRESHOLD` (0.25) moves from `changed` to `reworded` in the change summary, and leaves `unrequested` if it was there. Every probability is recorded under `meaning`, by id. With no decision an item stays changed.
It runs when the Execution ends, on the summary written to `changes.json`. The summary shown at each Approval, and the evaluation's change checks, still compare text exactly.
Measured against live Jev on nine pairs:
| Pair | P(changed) |
|---|---|
| A Check reworded with the argument's name | 0.12 |
| "at least 2" to "2 or more" | 0.22 |
| A recommendation reworded | 0.10 |
| An Element renamed, same product and type | 0.30 |
| A Check moved to another Element | 0.41 |
| "only" dropped from a grant | 0.61 |
| A value changed, 30 to 2555 | 0.96 |
| A condition added | 0.95 |
| The product changed | 0.90 |
At 0.25 the three rewordings are read as such and no real change is. The dropped "only", the case the plan names, is caught at 0.61 but not by much.
Differs from the plan: requirements are not asked about. They carry their own `change` and reason, and an `unchanged` one must keep its text exactly (iteration 7 step 3). The Baseline's verdict is not carried forward for a reworded Check: validation still gives a verdict on every Check.
Tests: `tests/execution/test_change_requests.py`, "changed, or only reworded?" (2).

View File

@@ -0,0 +1,18 @@
# Step 19 — Implementation Detail
Built on 2026-10-09, on by default (`JEV_ASSESS_EXECUTIONS`). **No Execution has run with it.**
1. `app/tools/assessment.py` holds the graded rubric the evaluation uses for design recommendations (specific, actionable, trade-offs, traceable) and the bar (a mean of 0.67). `eval/scoring.py` and `eval/judge.py` now take the rubric, the bar and the grades from there, so a live Execution and a benchmark case are asked the same questions.
2. When an Execution ends, completed or with warnings, its design recommendations are put to the decider once. The four scores, their mean and whether it meets the bar are kept in the Execution's state and in `execution.json` under `assessment`. A mean below the bar is logged as a warning.
3. It reports only. Nothing stops or changes, and a failure to assess leaves the Execution with no assessment.
4. Risk flags (step 14) are already in `execution.json` under `risks`.
Differs from the plan:
- **Only the quality rubric is asked.** Checklist coverage is already settled inside every Execution by the pillar checklist review (iteration 4), in code. The structural checks need a case's expected products, which a live Execution does not have.
- **Nothing is sent as feedback on a trace.** That part waits on step 17 and the owner's decision on LangSmith.
- **No alerting.** A low score is a log line and a field in the record; nothing watches for it.
The judge behind it was measured in step 16: Jev answered the same question the same way every time (1.00) and agreed with Gemini on 0.91 of questions. Agreement with a person still awaits the owner's 32 labels.
Tests: `tests/execution/test_assessment.py` (5).

View File

@@ -0,0 +1,23 @@
# Step 3 — Implementation Detail
Built on 2026-10-09. **Not run in an Execution with real models.**
1. `pillars_touched` in `app/tools/pillars.py` asks one graded question per pillar: how directly does this change affect the pillar's recommendations (not at all, only indirectly, directly). A pillar at `JEV_PILLAR_THRESHOLD` (1.5 on that 0 to 2 scale) or above is consulted again. An empty list means the change touches none; no decision means consult them all.
2. When the design advisor is started on a Revision, or in a Change request, and advice from an earlier consultation exists, the engine asks which pillars the change touches (`_pillar_hints` in `app/nodes/execution_nodes.py`) and puts the answer in its context as `pillars_to_consult`. The change is the Deliverable's feedback, or in a Change request the requirements added, changed and removed.
3. `consult_pillar_advisors` with an empty list consults nobody and returns the advice already given.
4. A completed Execution writes `working.json`: each pillar's advice and checklist review. A Change request starts from its Baseline's, so the pillars a change does not touch are not consulted again. In the first Change request, before this, the design advisor consulted all six afresh and that was its longest step.
Differs from the plan: the plan asked a yes or no per pillar. Measured against live Jev on six changes, that was yes for nearly every pillar every time (0.39 to 0.96 across the five real changes). The graded form separates them:
| Change | Pillars at "directly" |
|---|---|
| Retention to 7 years, undeletable | security, reliability, operations, cost, sustainability |
| Survive the loss of a region | reliability, operations, cost |
| Budget halved | operations, cost, performance, sustainability |
| Page load 1 s to 300 ms | operations, cost, performance |
| A page title renamed | none |
| Customer-managed keys for one bucket | security |
Operations and cost are picked for almost every real change, so the saving is two to five advisors of six, not five of six.
Tests: `tests/execution/test_pillars.py` (3), `tests/execution/test_change_requests.py` (2).

View File

@@ -0,0 +1,27 @@
# Step 8 — Implementation Detail
Built on 2026-10-09, on by default (`JEV_SKILL_ROUTING`). **No Execution has run with it.**
1. `app/tools/skill_routing.py`. A Skill with four or more reference files is one worth routing within; each reference is described by its title and first paragraph. Today that is `google-cloud-iac-patterns` (14 references), which only the IaC author loads.
2. When a Specialist whose Skills include such a Skill is started, the engine asks one graded question per reference: how much does this task need a reference on this (not needed, might help, needed). The task is the Deliverable, any feedback, and the Target architecture's Elements with their products and resource types.
3. References at `JEV_SKILL_THRESHOLD` (1.25 on that 0 to 2 scale) or above are named in the Specialist's context as `read_first`. The protocol says to read those first, and that it is a place to start, not a limit.
4. With no decision nothing is named, and the Specialist searches its Skills as before.
Measured against live Jev on four tasks, each reference scored 0 to 2:
| Task | Named at 1.25 | Highest not named |
|---|---|---|
| Web app: Cloud Run, Cloud SQL, Secret Manager, load balancer, IAM, VPC | service, SQL, secrets, load balancer, IAM, networking, project services, Artifact Registry | KMS 0.72 |
| Nightly batch: Cloud Run job, Scheduler, Storage, IAM | job, scheduler, storage, IAM, project services, Artifact Registry | Secret Manager 0.90 |
| Events: Pub/Sub, Cloud Run, Artifact Registry, KMS, Monitoring | all seven it needs | Secret Manager 1.09 |
| A Revision: one failed Check on a bucket's retention | storage only | project services 0.96 |
The Cloud Run service pattern scored 1.39 and 1.42 where it was needed, which is why the threshold is 1.25 and not 1.5. The cost of that is the Artifact Registry pattern being named for two designs that run containers and did not list a registry.
Differs from the plan:
- **A graded question per reference, not one Choice.** A task needs several references, and a Choice names one.
- **Overlay Skills are not selected by Jev.** The caller or the deployment names them (iteration 4 step 12): which organisation's standards apply is not something to infer.
- **Not measured against its acceptance criterion.** Whether `load_skill_resource` calls fall needs a case run with it and one without.
Tests: `tests/execution/test_skill_routing.py` (5).