feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -78,7 +78,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | a coverage review of the Target architecture built: one choice per listed condition; passes a live test on case 02's graph; off until a full case has run with it (`steps/step-4.md`); other checks planned |
|
||||
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
| 6 | Model tier per delegation | J5 | planned |
|
||||
| 7 | Context selection | J6 | planned |
|
||||
@@ -86,11 +86,11 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
@@ -49,3 +49,30 @@ Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04"
|
||||
|
||||
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
|
||||
|
||||
## Finished on 2026-10-08: the third case 04 fault, and intake
|
||||
|
||||
**A reply that restates what is agreed is an answer.** `_read_message` now shows the decider the agreed `requirements.json` and asks one more noul, `already_agreed`: does the message only confirm or repeat what those requirements say? At 0.75 or more the reply is closed as an answer, whatever `asks_change` says. Measured against live Jev with requirements holding the facts being restated:
|
||||
|
||||
| Reply | asks_change | already_agreed | Disposition |
|
||||
|---|---|---|---|
|
||||
| Restates the GDPR and secrets requirement | 0.12 | 0.78 | close |
|
||||
| Restates the availability requirement | 0.26 | 0.48 | queue (asked once, then let go) |
|
||||
| Restates the retention requirement | 0.52 | 0.81 | close |
|
||||
| "Approved. Please proceed." | 0.04 | 0.70 | close |
|
||||
| Raises availability to 99.99% | 0.85 | 0.01 | act |
|
||||
| Changes retention from 2 years to 7 | 0.89 | 0.01 | act |
|
||||
| Adds SMS notifications and a mobile app | 0.85 | 0.01 | act |
|
||||
| Changes secret rotation to 30 days | 0.87 | 0.02 | act |
|
||||
|
||||
**Intake: a message that starts something is read first.** `app/nodes/intake.py` asks one choice of a message that arrives with no Execution in progress: new request, change request, or question. The A2A server (`app/a2a_server.py`) acts on it:
|
||||
|
||||
- a question, at 0.75 confidence or more, is answered with a short statement of what the agent does and starts nothing;
|
||||
- a change request, when the Customer case has a completed Execution, is started as a new request with a note saying Change requests are not available yet (iteration 7 builds them);
|
||||
- anything else, any reading that is not confident, and every pre-approved task starts an Execution as before.
|
||||
|
||||
Measured against live Jev on ten messages: two new requests, two questions, two change requests with an earlier Execution, a question about earlier work, a different new request for the same case, a change with nothing delivered (taken as new), and "hello" (0.53, taken as new). All ten as intended.
|
||||
|
||||
Not built: acting on a change request (iteration 7 step 2), and answering a question about earlier work from that work.
|
||||
|
||||
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).
|
||||
|
||||
|
||||
20
.agents/plans/iter-05-jev-decisions/steps/step-16.md
Normal file
20
.agents/plans/iter-05-jev-decisions/steps/step-16.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Step 16 — Implementation Detail
|
||||
|
||||
The judge (`eval/judge.py`: `DeciderJudge` for Jev, `GeminiJudge` for a language model) was built earlier. Built on 2026-10-08: the calibration.
|
||||
|
||||
1. `eval/datasets/judge/` holds two saved design recommendations (cases 02 and 03, from runs that passed) and `labels.json`: 32 questions about them, 12 checklist items and the 4 quality-rubric questions for each document, with `label` empty.
|
||||
2. `eval/calibrate_judge.py` asks both judges every question several times and reports, per judge, stability (the same answer to the same question across runs) and agreement with the labels, and agreement between the two judges. `--build` regenerates the question set. A new Jev client is used for each run, because the client caches identical requests and a cached answer would make stability look perfect.
|
||||
|
||||
Measured on 2026-10-08, three runs, no labels yet (`eval/datasets/judge/calibration.json`):
|
||||
|
||||
| | Jev | Gemini (the bundle's default model) |
|
||||
|---|---|---|
|
||||
| Stability | 1.00 | 0.92 |
|
||||
| Mean answer | 0.37 | 0.32 |
|
||||
| Agreement between the two | 0.91 | |
|
||||
|
||||
So Jev gave the same answer every time, the language-model judge changed its answer on about one question in twelve, and the two agree on nine in ten.
|
||||
|
||||
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
|
||||
@@ -48,3 +48,14 @@ Turn it on for a full case before relying on it.
|
||||
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.
|
||||
|
||||
## Added on 2026-10-08
|
||||
|
||||
**What code cannot show is listed apart.** The first full run with the coverage review raised conditions such as an availability percentage, which no Check can verify. A requirement or recommendation now lists under `conditions` only what infrastructure configuration can show, and under `outside_code` what it cannot: an outcome measured in operation, behaviour of the application's own code, a process. `check_requirements` and `check_recommendations` require at least one entry between the two. `chain_gaps` asks for no Check from a constraint with `"conditions": []` and something under `outside_code`. The target architect writes no Check for `outside_code`; the validator proposes nothing for it and names it as something to verify after deployment.
|
||||
|
||||
Measured before the split, on the designs of two runs that passed validation with nothing proposed, using the conditions the agents themselves wrote: the coverage review raised 14 of 66 (case 02) and 8 of 51 (case 03). About half were of the kind the split removes (availability, recovery time, throughput, latency, what the application logs). The rest look like real gaps the validator did not report (three zones, cost labels, a load balancer). It has not been measured since the split, which needs new runs.
|
||||
|
||||
**Requirements name no products.** `RequirementsReview` in `app/tools/requirements_review.py` asks one noul per requirement: does it name a cloud provider's product or managed service? It is a third review on the same seam, behind `JEV_REQUIREMENTS_REVIEW` (off), with its own threshold, `JEV_REQUIREMENTS_THRESHOLD`, 0.3. Measured against live Jev: across the 61 requirements of three passed runs the highest answer was 0.10, so nothing was raised; requirements written to name a product scored 0.93 (Cloud Run), 0.61 (Secret Manager) and 0.34 (a Pub/Sub topic), and a database engine or a region scored 0.10 or less. At 0.3 that is three of three caught and none of 65 raised wrongly. Three written examples is a small sample.
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py` (2 more), `tests/execution/test_requirements_chain.py` (2 more).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user