fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator turns on two router faults: - A reply that quoted the agent's question back was judged on that question. A reply now reaches the router with quoted questions and Q:/A: labels removed, and the questions are given to the decider as background. - At a Revision limit the reply was read as another change request for the same Deliverable, and the refusal repeated. The user is now told once, the reply is taken as the answer, that Deliverable is not reopened again, and the Execution carries on with it as it stands. - Any run that asks the user four times running with no work done in between now ends with a warning. The validator is told to confirm with the user in an interactive Execution. Plans record the fixes and the rerun, which did not finish: both fixes held, and a third fault appeared that is not fixed here. A scripted answer restating a requirement, given to a draft-confirmation question, reopens requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -34,3 +34,18 @@ Unverified:
|
||||
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
|
||||
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
|
||||
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
|
||||
|
||||
## Faults found by evaluation case 04, fixed on 2026-10-08
|
||||
|
||||
Case 04 is the first interactive run. It stopped at 100 Orchestrator turns (`../../iter-06-target-architecture-graph/steps/step-11.md`). Three changes:
|
||||
|
||||
1. **Only the user's own words are their message.** A reply to questions now reaches the router with the questions it answered (`asked`) and with any quoted question and `Q:`/`A:` labels removed from its text (`_own_words` in `app/nodes/execution_nodes.py`). `_read_message` gives the questions to the decider as background and the cleaned text as the message. Before, a describer's "please confirm my draft" question, quoted back by the scripted user, was judged a request to change the Target architecture, twice.
|
||||
2. **A Revision limit is told to the user once.** When a reopen is refused at the limit, the router records the Deliverable in `revision_limit_asked` and sets `awaiting_limit_decision`. The reply is not read as a request for changes, every action that would reopen that Deliverable is barred from then on, and the Execution carries on with it as it stands. Before, the reply was read as another change request for the same Deliverable and the refusal repeated until the turn limit.
|
||||
3. **A run that only asks is stopped.** The engine counts questions put to the user with no change in any Deliverable's status or attempt, or any Phase's status, in between. At the fourth it ends with a warning (`MAX_IDLE_ASKS`). This holds for any Orchestrator, whatever the cause.
|
||||
|
||||
Also: `agents/validator.md` now tells the validator to confirm with the user before submitting in an interactive Execution, which the case's `specialists_ask_before_submitting` check found it did not.
|
||||
|
||||
Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04" (5).
|
||||
|
||||
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user