feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -49,3 +49,30 @@ Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04"
|
||||
|
||||
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
|
||||
|
||||
## Finished on 2026-10-08: the third case 04 fault, and intake
|
||||
|
||||
**A reply that restates what is agreed is an answer.** `_read_message` now shows the decider the agreed `requirements.json` and asks one more noul, `already_agreed`: does the message only confirm or repeat what those requirements say? At 0.75 or more the reply is closed as an answer, whatever `asks_change` says. Measured against live Jev with requirements holding the facts being restated:
|
||||
|
||||
| Reply | asks_change | already_agreed | Disposition |
|
||||
|---|---|---|---|
|
||||
| Restates the GDPR and secrets requirement | 0.12 | 0.78 | close |
|
||||
| Restates the availability requirement | 0.26 | 0.48 | queue (asked once, then let go) |
|
||||
| Restates the retention requirement | 0.52 | 0.81 | close |
|
||||
| "Approved. Please proceed." | 0.04 | 0.70 | close |
|
||||
| Raises availability to 99.99% | 0.85 | 0.01 | act |
|
||||
| Changes retention from 2 years to 7 | 0.89 | 0.01 | act |
|
||||
| Adds SMS notifications and a mobile app | 0.85 | 0.01 | act |
|
||||
| Changes secret rotation to 30 days | 0.87 | 0.02 | act |
|
||||
|
||||
**Intake: a message that starts something is read first.** `app/nodes/intake.py` asks one choice of a message that arrives with no Execution in progress: new request, change request, or question. The A2A server (`app/a2a_server.py`) acts on it:
|
||||
|
||||
- a question, at 0.75 confidence or more, is answered with a short statement of what the agent does and starts nothing;
|
||||
- a change request, when the Customer case has a completed Execution, is started as a new request with a note saying Change requests are not available yet (iteration 7 builds them);
|
||||
- anything else, any reading that is not confident, and every pre-approved task starts an Execution as before.
|
||||
|
||||
Measured against live Jev on ten messages: two new requests, two questions, two change requests with an earlier Execution, a question about earlier work, a different new request for the same case, a change with nothing delivered (taken as new), and "hello" (0.53, taken as new). All ten as intended.
|
||||
|
||||
Not built: acting on a change request (iteration 7 step 2), and answering a question about earlier work from that work.
|
||||
|
||||
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user