fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator turns on two router faults: - A reply that quoted the agent's question back was judged on that question. A reply now reaches the router with quoted questions and Q:/A: labels removed, and the questions are given to the decider as background. - At a Revision limit the reply was read as another change request for the same Deliverable, and the refusal repeated. The user is now told once, the reply is taken as the answer, that Deliverable is not reopened again, and the Execution carries on with it as it stands. - Any run that asks the user four times running with no work done in between now ends with a warning. The validator is told to confirm with the user in an interactive Execution. Plans record the fixes and the rerun, which did not finish: both fixes held, and a third fault appeared that is not fixed here. A scripted answer restating a requirement, given to a draft-confirmation question, reopens requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
@@ -86,7 +86,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit (`../iter-06-target-architecture-graph/steps/step-11.md`) |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
|
||||
@@ -34,3 +34,18 @@ Unverified:
|
||||
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
|
||||
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
|
||||
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
|
||||
|
||||
## Faults found by evaluation case 04, fixed on 2026-10-08
|
||||
|
||||
Case 04 is the first interactive run. It stopped at 100 Orchestrator turns (`../../iter-06-target-architecture-graph/steps/step-11.md`). Three changes:
|
||||
|
||||
1. **Only the user's own words are their message.** A reply to questions now reaches the router with the questions it answered (`asked`) and with any quoted question and `Q:`/`A:` labels removed from its text (`_own_words` in `app/nodes/execution_nodes.py`). `_read_message` gives the questions to the decider as background and the cleaned text as the message. Before, a describer's "please confirm my draft" question, quoted back by the scripted user, was judged a request to change the Target architecture, twice.
|
||||
2. **A Revision limit is told to the user once.** When a reopen is refused at the limit, the router records the Deliverable in `revision_limit_asked` and sets `awaiting_limit_decision`. The reply is not read as a request for changes, every action that would reopen that Deliverable is barred from then on, and the Execution carries on with it as it stands. Before, the reply was read as another change request for the same Deliverable and the refusal repeated until the turn limit.
|
||||
3. **A run that only asks is stopped.** The engine counts questions put to the user with no change in any Deliverable's status or attempt, or any Phase's status, in between. At the fourth it ends with a warning (`MAX_IDLE_ASKS`). This holds for any Orchestrator, whatever the cause.
|
||||
|
||||
Also: `agents/validator.md` now tells the validator to confirm with the user before submitting in an interactive Execution, which the case's `specialists_ask_before_submitting` check found it did not.
|
||||
|
||||
Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04" (5).
|
||||
|
||||
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
|
||||
|
||||
|
||||
@@ -56,7 +56,7 @@ flowchart TD
|
||||
| 8 | Deployment guidance implements the graph | M | complete |
|
||||
| 9 | Validation gives a verdict per Check | M | complete |
|
||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) ends with warnings at 75% on two router faults (`steps/step-11.md`); thresholds outstanding |
|
||||
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding |
|
||||
|
||||
Size: S under a day, M a few days.
|
||||
|
||||
|
||||
@@ -137,3 +137,28 @@ Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not compl
|
||||
|
||||
Not yet fixed: the two router faults above.
|
||||
|
||||
## Run of 2026-10-08, 12:22 UTC, unfinished (`.cache/eval-logs/case-04-20261008T122200Z.log`; no report)
|
||||
|
||||
Case 04 again, with the three router fixes of iteration 5 step 12 (uncommitted at the time). The process stopped at about 14:48 local time, 86 minutes in, when the session that started it ended. It was not cancelled and wrote no score, report or token count.
|
||||
|
||||
What the fixes did:
|
||||
|
||||
- No reopen came from a question quoted back in a reply.
|
||||
- At 14:41 `requirements` reached its Revision limit. The user was told once and the Execution carried on; there was no loop.
|
||||
- The schema review raised items at submission twice (1, then 3), its first use in a real Execution.
|
||||
- The unimplementable-Check route fired once and the design was revised in 92 seconds; the description was kept.
|
||||
|
||||
What went wrong:
|
||||
|
||||
1. **A scripted answer was read as a new requirement, three times.** The scripted user answers by matching words in the question against patterns written for the requirements analyst. Asked by the validator, and later the design advisor and the IaC author, to confirm a draft, it replied with requirement statements ("GDPR applies to employee names and emails...", "99.5% during business hours...", "PostgreSQL. Under 20 GB..."). Jev read each as a request to change `requirements`, and the router reopened it: twice, then the limit. Each reopen returned every later Deliverable to draft, so the Execution was redone from the start twice.
|
||||
2. Validation failed one Check twice (an alert condition; a log bucket not locked). Each was sent back to the code, which is the intended route.
|
||||
|
||||
The cause of the first is mostly the harness: no person answers "please confirm my draft" with a requirement. It also shows a weakness in the router: a reply that restates what a submitted Deliverable already says is treated as a request to change it.
|
||||
|
||||
Not yet changed:
|
||||
|
||||
- Scripted answers should apply only to the Specialist they were written for, and a draft-confirmation question should get an approval.
|
||||
- A reply that restates what a submitted Deliverable already says should be read as an answer.
|
||||
|
||||
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user