fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator turns on two router faults: - A reply that quoted the agent's question back was judged on that question. A reply now reaches the router with quoted questions and Q:/A: labels removed, and the questions are given to the decider as background. - At a Revision limit the reply was read as another change request for the same Deliverable, and the refusal repeated. The user is now told once, the reply is taken as the answer, that Deliverable is not reopened again, and the Execution carries on with it as it stands. - Any run that asks the user four times running with no work done in between now ends with a warning. The validator is told to confirm with the user in an interactive Execution. Plans record the fixes and the rerun, which did not finish: both fixes held, and a third fault appeared that is not fixed here. A scripted answer restating a requirement, given to a draft-confirmation question, reopens requirements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -137,3 +137,28 @@ Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not compl
|
||||
|
||||
Not yet fixed: the two router faults above.
|
||||
|
||||
## Run of 2026-10-08, 12:22 UTC, unfinished (`.cache/eval-logs/case-04-20261008T122200Z.log`; no report)
|
||||
|
||||
Case 04 again, with the three router fixes of iteration 5 step 12 (uncommitted at the time). The process stopped at about 14:48 local time, 86 minutes in, when the session that started it ended. It was not cancelled and wrote no score, report or token count.
|
||||
|
||||
What the fixes did:
|
||||
|
||||
- No reopen came from a question quoted back in a reply.
|
||||
- At 14:41 `requirements` reached its Revision limit. The user was told once and the Execution carried on; there was no loop.
|
||||
- The schema review raised items at submission twice (1, then 3), its first use in a real Execution.
|
||||
- The unimplementable-Check route fired once and the design was revised in 92 seconds; the description was kept.
|
||||
|
||||
What went wrong:
|
||||
|
||||
1. **A scripted answer was read as a new requirement, three times.** The scripted user answers by matching words in the question against patterns written for the requirements analyst. Asked by the validator, and later the design advisor and the IaC author, to confirm a draft, it replied with requirement statements ("GDPR applies to employee names and emails...", "99.5% during business hours...", "PostgreSQL. Under 20 GB..."). Jev read each as a request to change `requirements`, and the router reopened it: twice, then the limit. Each reopen returned every later Deliverable to draft, so the Execution was redone from the start twice.
|
||||
2. Validation failed one Check twice (an alert condition; a log bucket not locked). Each was sent back to the code, which is the intended route.
|
||||
|
||||
The cause of the first is mostly the harness: no person answers "please confirm my draft" with a requirement. It also shows a weakness in the router: a reply that restates what a submitted Deliverable already says is treated as a request to change it.
|
||||
|
||||
Not yet changed:
|
||||
|
||||
- Scripted answers should apply only to the Specialist they were written for, and a draft-confirmation question should get an approval.
|
||||
- A reply that restates what a submitted Deliverable already says should be read as an answer.
|
||||
|
||||
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user