fix: read only the user's own words, and stop the router looping at a limit

Case 04, the first interactive evaluation, stopped at 100 Orchestrator
turns on two router faults:

- A reply that quoted the agent's question back was judged on that question.
  A reply now reaches the router with quoted questions and Q:/A: labels
  removed, and the questions are given to the decider as background.
- At a Revision limit the reply was read as another change request for the
  same Deliverable, and the refusal repeated. The user is now told once,
  the reply is taken as the answer, that Deliverable is not reopened again,
  and the Execution carries on with it as it stands.
- Any run that asks the user four times running with no work done in
  between now ends with a warning.

The validator is told to confirm with the user in an interactive Execution.

Plans record the fixes and the rerun, which did not finish: both fixes
held, and a third fault appeared that is not fixed here. A scripted answer
restating a requirement, given to a draft-confirmation question, reopens
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 15:05:03 +01:00
parent c9f59be274
commit d41607f505
11 changed files with 321 additions and 9 deletions

View File

@@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. | | 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. | | 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. | | 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | | 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. | | 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. | | 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | | 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |

View File

@@ -86,7 +86,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 9 | In-run context compaction | J10 | planned | | 9 | In-run context compaction | J10 | planned |
| 10 | Evaluation rubric and lesson triage | J8 | planned | | 10 | Evaluation rubric and lesson triage | J8 | planned |
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 | | 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit (`../iter-06-target-architecture-graph/steps/step-11.md`) | | 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` |
| 13 | Cloud-read call guard | J7 | planned | | 13 | Cloud-read call guard | J7 | planned |
| 14 | Input guardrail | J11 | planned | | 14 | Input guardrail | J11 | planned |
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 | | 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |

View File

@@ -34,3 +34,18 @@ Unverified:
- Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned. - Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned.
- Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does. - Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does.
- Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it. - Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it.
## Faults found by evaluation case 04, fixed on 2026-10-08
Case 04 is the first interactive run. It stopped at 100 Orchestrator turns (`../../iter-06-target-architecture-graph/steps/step-11.md`). Three changes:
1. **Only the user's own words are their message.** A reply to questions now reaches the router with the questions it answered (`asked`) and with any quoted question and `Q:`/`A:` labels removed from its text (`_own_words` in `app/nodes/execution_nodes.py`). `_read_message` gives the questions to the decider as background and the cleaned text as the message. Before, a describer's "please confirm my draft" question, quoted back by the scripted user, was judged a request to change the Target architecture, twice.
2. **A Revision limit is told to the user once.** When a reopen is refused at the limit, the router records the Deliverable in `revision_limit_asked` and sets `awaiting_limit_decision`. The reply is not read as a request for changes, every action that would reopen that Deliverable is barred from then on, and the Execution carries on with it as it stands. Before, the reply was read as another change request for the same Deliverable and the refusal repeated until the turn limit.
3. **A run that only asks is stopped.** The engine counts questions put to the user with no change in any Deliverable's status or attempt, or any Phase's status, in between. At the fourth it ends with a warning (`MAX_IDLE_ASKS`). This holds for any Orchestrator, whatever the cause.
Also: `agents/validator.md` now tells the validator to confirm with the user before submitting in an interactive Execution, which the case's `specialists_ask_before_submitting` check found it did not.
Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04" (5).
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.

View File

@@ -56,7 +56,7 @@ flowchart TD
| 8 | Deployment guidance implements the graph | M | complete | | 8 | Deployment guidance implements the graph | M | complete |
| 9 | Validation gives a verdict per Check | M | complete | | 9 | Validation gives a verdict per Check | M | complete |
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model | | 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) ends with warnings at 75% on two router faults (`steps/step-11.md`); thresholds outstanding | | 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding |
Size: S under a day, M a few days. Size: S under a day, M a few days.

View File

@@ -137,3 +137,28 @@ Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not compl
Not yet fixed: the two router faults above. Not yet fixed: the two router faults above.
## Run of 2026-10-08, 12:22 UTC, unfinished (`.cache/eval-logs/case-04-20261008T122200Z.log`; no report)
Case 04 again, with the three router fixes of iteration 5 step 12 (uncommitted at the time). The process stopped at about 14:48 local time, 86 minutes in, when the session that started it ended. It was not cancelled and wrote no score, report or token count.
What the fixes did:
- No reopen came from a question quoted back in a reply.
- At 14:41 `requirements` reached its Revision limit. The user was told once and the Execution carried on; there was no loop.
- The schema review raised items at submission twice (1, then 3), its first use in a real Execution.
- The unimplementable-Check route fired once and the design was revised in 92 seconds; the description was kept.
What went wrong:
1. **A scripted answer was read as a new requirement, three times.** The scripted user answers by matching words in the question against patterns written for the requirements analyst. Asked by the validator, and later the design advisor and the IaC author, to confirm a draft, it replied with requirement statements ("GDPR applies to employee names and emails...", "99.5% during business hours...", "PostgreSQL. Under 20 GB..."). Jev read each as a request to change `requirements`, and the router reopened it: twice, then the limit. Each reopen returned every later Deliverable to draft, so the Execution was redone from the start twice.
2. Validation failed one Check twice (an alert condition; a log bucket not locked). Each was sent back to the code, which is the intended route.
The cause of the first is mostly the harness: no person answers "please confirm my draft" with a requirement. It also shows a weakness in the router: a reply that restates what a submitted Deliverable already says is treated as a request to change it.
Not yet changed:
- Scripted answers should apply only to the Specialist they were written for, and a draft-confirmation question should get an approval.
- A reply that restates what a submitted Deliverable already says should be read as an answer.
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.

View File

@@ -15,6 +15,7 @@ You produce the **validation results** Deliverable: the Skill's pre-deployment v
- Record your verdicts with `verdict_put`, a few Checks per call, as you reach them: the Checks of one Element, then the next. Record the dry-run's result and each proposed addition the same way. Short calls are what keep your replies from timing out, and what you record is kept if your run is cut short. `verdict_read` shows which Checks still have none. - Record your verdicts with `verdict_put`, a few Checks per call, as you reach them: the Checks of one Element, then the next. Record the dry-run's result and each proposed addition the same way. Short calls are what keep your replies from timing out, and what you record is kept if your run is cut short. `verdict_read` shows which Checks still have none.
- When `working_copy` shows verdicts recorded, your last run was cut short on this same code: call `verdict_read` and carry on from there. - When `working_copy` shows verdicts recorded, your last run was cut short on this same code: call `verdict_read` and carry on from there.
- Submit with your content and no attachments when every Check has a verdict: what you recorded is attached as `verdicts.json`, in the format the requirements-chain Skill gives. - Submit with your content and no attachments when every Check has a verdict: what you recorded is attached as `verdicts.json`, in the format the requirements-chain Skill gives.
- When `pre_approved` is `false`, confirm with the user before you submit, as every Specialist does: call `needs_input` with a summary of the verdicts and pass your content as `draft`. Your recorded verdicts are kept and attached when you submit afterwards.
- Give the exact commands for every check you couldn't run, for the user to run themselves. - Give the exact commands for every check you couldn't run, for the user to run themselves.
End each of your contract's sections with a pass or fail verdict. End each of your contract's sections with a pass or fail verdict.

View File

@@ -163,6 +163,10 @@ class ExecutionEngine:
"pending_change": None, "pending_change": None,
"awaiting_gap_decision": False, "awaiting_gap_decision": False,
"validation_settled": 0, "validation_settled": 0,
"awaiting_limit_decision": False,
"revision_limit_asked": [],
"idle_asks": 0,
"ask_fingerprint": "",
"decisions": [], "decisions": [],
"outcomes": [], "outcomes": [],
"refusals": [], "refusals": [],

View File

@@ -64,6 +64,8 @@ logger = logging.getLogger(__name__)
# Working-memory key: the last submission of a Deliverable with what it was written from. # Working-memory key: the last submission of a Deliverable with what it was written from.
STANDING = "standing" STANDING = "standing"
# Questions put to the user running, with no work done between, before stopping.
MAX_IDLE_ASKS = 3
NO_USER_REPLY = ( NO_USER_REPLY = (
"No user is available. State your assumption, record it in the Deliverable under " "No user is available. State your assumption, record it in the Deliverable under "
@@ -266,6 +268,8 @@ def make_execution_nodes(
pending_change=state.get("pending_change"), pending_change=state.get("pending_change"),
awaiting_gap_decision=bool(state.get("awaiting_gap_decision")), awaiting_gap_decision=bool(state.get("awaiting_gap_decision")),
validation_settled=int(state.get("validation_settled", 0)), validation_settled=int(state.get("validation_settled", 0)),
awaiting_limit_decision=bool(state.get("awaiting_limit_decision")),
revision_limit_asked=tuple(state.get("revision_limit_asked", ())),
context=_orchestrator_context( context=_orchestrator_context(
state, ledger, deliverables, pending_questions, notes state, ledger, deliverables, pending_questions, notes
), ),
@@ -332,6 +336,22 @@ def make_execution_nodes(
return ask({"approval": toolbox.approval}) return ask({"approval": toolbox.approval})
if isinstance(outcome, TurnEnded) and outcome.tool == "ask_user": if isinstance(outcome, TurnEnded) and outcome.tool == "ask_user":
# Asking again and again with nothing done in between is a loop, not a
# conversation: stop it, whatever caused it.
fingerprint = _progress(ledger)
idle = (
state.get("idle_asks", 0) + 1
if fingerprint == state.get("ask_fingerprint")
else 0
)
update.update(idle_asks=idle, ask_fingerprint=fingerprint)
if idle >= MAX_IDLE_ASKS:
warnings.append(
f"Stopped: the user was asked {idle + 1} times running with "
"no work done in between."
)
update["events_recorded"] = await record_progress(state, ledger)
return finish_with("end_with_warnings", warnings=warnings)
update["events_recorded"] = await record_progress( update["events_recorded"] = await record_progress(
state, ledger, None if pre_approved else "awaiting_user" state, ledger, None if pre_approved else "awaiting_user"
) )
@@ -519,7 +539,9 @@ def make_execution_nodes(
else: else:
text = _reply_text(reply) text = _reply_text(reply)
conversation.append({"from": "user", "text": text}) conversation.append({"from": "user", "text": text})
said = {"kind": "reply", "text": text} put = [item["question"] for item in prompt.get("questions", [])]
# What is routed is what the user said, not our questions quoted back.
said = {"kind": "reply", "text": _own_words(text, put), "asked": put}
answers = { answers = {
kind: list(items) for kind, items in state.get("answers", {}).items() kind: list(items) for kind, items in state.get("answers", {}).items()
} }
@@ -775,6 +797,35 @@ def _reply_text(reply: object) -> str:
return str(reply) return str(reply)
def _own_words(text: str, asked: Sequence[str]) -> str:
"""A reply without the questions it quotes, and without Q: and A: labels."""
for question in sorted(asked, key=len, reverse=True):
text = text.replace(question, "")
lines = []
for line in text.splitlines():
line = re.sub(r"^\s*(?:Q|A)\s*:\s*", "", line).strip()
if line:
lines.append(line)
return "\n".join(lines)
def _progress(ledger: Ledger) -> str:
"""The work done so far, as one comparable value: each Deliverable's status and
attempt, and each Phase's status."""
data = ledger.to_dict()
return json.dumps(
{
"deliverables": {
kind: [record.get("status"), record.get("attempt")]
for kind, record in data["deliverables"].items()
},
"phases": data.get("phases"),
},
sort_keys=True,
default=str,
)
def _render_prompt(prompt: Mapping[str, Any]) -> str: def _render_prompt(prompt: Mapping[str, Any]) -> str:
if "approval" in prompt: if "approval" in prompt:
approval = prompt["approval"] approval = prompt["approval"]

View File

@@ -84,6 +84,10 @@ class TurnView:
pending_change: Mapping[str, Any] | None = None # a change we asked them to confirm pending_change: Mapping[str, Any] | None = None # a change we asked them to confirm
awaiting_gap_decision: bool = False awaiting_gap_decision: bool = False
validation_settled: int = 0 # the validation attempt already decided on validation_settled: int = 0 # the validation attempt already decided on
# The user's reply answers "this Deliverable reached its Revision limit".
awaiting_limit_decision: bool = False
# Deliverables at their Revision limit that the user has been told about.
revision_limit_asked: Sequence[str] = ()
context: Mapping[str, Any] = field(default_factory=dict) # for a model Orchestrator context: Mapping[str, Any] = field(default_factory=dict) # for a model Orchestrator
@@ -135,6 +139,25 @@ class Readings:
judged: dict[str, Any] = field(default_factory=dict) # what the decider answered judged: dict[str, Any] = field(default_factory=dict) # what the decider answered
# The actions that reopen a Deliverable, and which one.
_REOPENS = (
"revise_requested",
"revise_code",
"revise_target_architecture",
"revise_unimplementable",
)
def _reopened_by(action: str, readings: Readings) -> DeliverableKind | None:
if action == "revise_requested":
return readings.change_target
if action == "revise_code":
return DeliverableKind.DEPLOYMENT_GUIDANCE
if action in {"revise_target_architecture", "revise_unimplementable"}:
return DeliverableKind.TARGET_ARCHITECTURE
return None
def _attempt(ledger: Ledger, kind: DeliverableKind) -> int: def _attempt(ledger: Ledger, kind: DeliverableKind) -> int:
return int(ledger.to_dict()["deliverables"][kind.value].get("attempt", 0)) return int(ledger.to_dict()["deliverables"][kind.value].get("attempt", 0))
@@ -194,16 +217,25 @@ class Router:
"last_reply": None, "last_reply": None,
"pending_change": None, "pending_change": None,
"awaiting_gap_decision": False, "awaiting_gap_decision": False,
"awaiting_limit_decision": False,
} }
readings = await self._read(view, result) readings = await self._read(view, result)
blocked: set[str] = set() blocked: set[str] = set()
settled = view.validation_settled settled = view.validation_settled
# Told once, the user is not asked again: a Deliverable at its limit is no
# longer reopened, and the Execution carries on with it as it stands.
at_limit = set(view.revision_limit_asked)
async def call(tool: str, **arguments: Any) -> ToolResult: async def call(tool: str, **arguments: Any) -> ToolResult:
return await by_name[tool].handler(**arguments) return await by_name[tool].handler(**arguments)
for _ in range(MAX_STEPS): for _ in range(MAX_STEPS):
group, action = self._choose(view, readings, blocked, settled) barred = blocked | {
name
for name in _REOPENS
if (kind := _reopened_by(name, readings)) and kind.value in at_limit
}
group, action = self._choose(view, readings, barred, settled)
result.decisions.append( result.decisions.append(
{"group": group, "action": action, "judged": dict(readings.judged)} {"group": group, "action": action, "judged": dict(readings.judged)}
) )
@@ -220,9 +252,20 @@ class Router:
result.outcome = reply.outcome result.outcome = reply.outcome
return result return result
if "revision limit" in reply.message and not view.pre_approved: if "revision limit" in reply.message and not view.pre_approved:
# Past the Revision limit the user decides. # Past the Revision limit the user is told, once. Their reply is
# read as the answer to this, never as a new request for changes.
kind = _reopened_by(action, readings)
if kind:
at_limit.add(kind.value)
result.updates["revision_limit_asked"] = sorted(at_limit)
result.updates["awaiting_limit_decision"] = True
limit = await call( limit = await call(
"ask_user", questions=[reply.message.split(" Use ")[0]] "ask_user",
questions=[
reply.message.split(" Use ")[0]
+ " It will not be revised again in this Execution; I will "
"carry on with it as it stands. Reply to continue."
],
) )
if limit.outcome is not None: if limit.outcome is not None:
result.outcome = limit.outcome result.outcome = limit.outcome
@@ -298,7 +341,7 @@ class Router:
"""Triage of the user's message: is it asking for a change, to what, and how """Triage of the user's message: is it asking for a change, to what, and how
clearly?""" clearly?"""
reply = view.last_reply reply = view.last_reply
if not reply or view.awaiting_gap_decision: if not reply or view.awaiting_gap_decision or view.awaiting_limit_decision:
return return
text = str(reply.get("text", "")).strip() text = str(reply.get("text", "")).strip()
pending = view.pending_change pending = view.pending_change
@@ -308,6 +351,7 @@ class Router:
if not text or not candidates: if not text or not candidates:
return return
readings.change_reason = text readings.change_reason = text
asked = [str(question) for question in reply.get("asked") or ()]
# A structured request for changes is known to be one, and so is the reply to # A structured request for changes is known to be one, and so is the reply to
# our own question about a change they already asked for. # our own question about a change they already asked for.
structured = reply.get("kind") == "changes" or bool(pending) structured = reply.get("kind") == "changes" or bool(pending)
@@ -334,7 +378,16 @@ class Router:
state = ( state = (
f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n" f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n"
f"Questions waiting for the user: {len(view.pending_questions)}.\n" f"Questions waiting for the user: {len(view.pending_questions)}.\n"
f"The user's message: {text}" # What we asked is background. Only the user's own words are their
# message: our question echoed back is not something they said.
+ (
"The user was replying to these questions from the agents:\n"
+ "\n".join(f"- {question}" for question in asked)
+ "\n"
if asked
else ""
)
+ f"The user's message: {text}"
) )
answers = await self._decider.decide(state, questions, scope=view.execution_id) answers = await self._decider.decide(state, questions, scope=view.execution_id)

View File

@@ -87,6 +87,10 @@ class ExecutionState(TypedDict, total=False):
pending_change: dict[str, Any] | None # a change the user was asked to confirm pending_change: dict[str, Any] | None # a change the user was asked to confirm
awaiting_gap_decision: bool # the user was asked whether to close gaps awaiting_gap_decision: bool # the user was asked whether to close gaps
validation_settled: int # the validation attempt already decided on validation_settled: int # the validation attempt already decided on
awaiting_limit_decision: bool # the user was told a Revision limit was reached
revision_limit_asked: list[str] # Deliverables the user has been told that about
idle_asks: int # questions put to the user running, with nothing changing between
ask_fingerprint: str # the Ledger's state when the user was last asked
decisions: list[dict[str, Any]] # every routing decision, in order decisions: list[dict[str, Any]] # every routing decision, in order
turns: int turns: int

View File

@@ -596,3 +596,162 @@ async def test_past_the_revision_limit_the_user_decides(tmp_path):
"deployment_guidance has reached its revision limit (1)" "deployment_guidance has reached its revision limit (1)"
in (question["question"]) in (question["question"])
) )
# --- the two faults of evaluation case 04 (2026-10-08) -----------------------------------
async def test_a_reply_at_the_revision_limit_is_the_answer_and_the_run_carries_on(
tmp_path,
):
"""The user is told once. Their reply is not read as a new request for changes,
the Deliverable is not reopened again, and nothing is asked a second time."""
turns = once()
turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2
turns[specialist(K.VALIDATION_RESULTS)] = [[failing("K-2")], [failing("K-1")]]
# A decider that would call any reply a clear request to change the code.
decider = FakeDecider(
{
"asks_change": Answer(0.95, 0.9),
"change_target": Answer("deployment_guidance", 0.95),
"change_clarity": Answer(1.9, 0.8),
}
)
engine, _ = engine_for(tmp_path, turns, decider, max_revisions=1)
result = await engine.start("Ingest webhooks")
for phase in (
"requirements_discovery",
"solution_architecture",
"solution_implementation",
"solution_implementation",
):
result = await engine.resume(result.execution_id, approve(phase))
[question] = result.prompt["questions"]
assert "I will carry on with it as it stands" in question["question"]
result = await engine.resume(
result.execution_id, {"reply": "Please change the code again."}
)
# It moves on to the Approval of validation; the code was reopened once only.
assert result.prompt["approval"]["phase"] == "solution_validation"
assert len(reopened(result)) == 1
assert result.state["revision_limit_asked"] == ["deployment_guidance"]
assert decider.asked == [] # the reply was never put to the decider
async def test_only_the_users_own_words_are_read_as_their_message(tmp_path):
"""A reply that quotes our question back must not be judged on our words."""
question = "Please confirm the requirements draft. Shall I change anything?"
turns = once()
turns[specialist(K.REQUIREMENTS)] = [
[Call("needs_input", {"questions": [question]})],
[sample_submission(K.REQUIREMENTS)],
]
decider = FakeDecider(
{
"asks_change": Answer(0.05, 0.9),
"change_target": Answer("requirements", 0.5),
"change_clarity": Answer(0.2, 0.5),
}
)
engine, _ = engine_for(tmp_path, turns, decider)
first = await engine.start("Ingest webhooks")
result = await engine.resume(
first.execution_id, {"reply": f"Q: {question}\nA: Yes, approved."}
)
# Nothing has been submitted yet, so nothing is judged on that turn. Let the
# next reply, to the decomposer's approval prompt, be a text one.
result = await engine.resume(result.execution_id, {"reply": "Looks fine."})
assert result.state["last_reply"] is None
state, _, _ = decider.asked[-1]
assert state.endswith("The user's message: Looks fine.")
def test_a_quoted_question_is_removed_from_what_the_user_said():
from app.nodes.execution_nodes import _own_words
asked = ["Please confirm the draft. Shall I change the design?", "Which region?"]
reply = (
"Q: Please confirm the draft. Shall I change the design?\nA: Yes, approved.\n\n"
"Q: Which region?\nA: europe-west2"
)
assert _own_words(reply, asked) == "Yes, approved.\neurope-west2"
assert _own_words("Make retention 7 years.", asked) == "Make retention 7 years."
async def test_the_questions_asked_are_given_to_the_decider_as_background(tmp_path):
from app.nodes.router import TurnView
from app.states.ledger import Ledger
ledger = Ledger()
ledger.start([K.REQUIREMENTS], actor="orchestrator")
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
decider = FakeDecider(
{
"asks_change": Answer(0.05, 0.9),
"change_target": Answer("requirements", 0.5),
"change_clarity": Answer(0.2, 0.5),
}
)
router = Router(decider, None, Thresholds())
view = TurnView(
execution_id="e1",
pre_approved=False,
ledger=ledger,
deliverables={},
pending_questions=[],
outcomes=[],
last_reply={
"kind": "reply",
"text": "Yes, approved.",
"asked": ["Shall I change the design?"],
},
)
from app.nodes.router import Readings
await router._read_message(view, Readings())
state, _, _ = decider.asked[0]
assert "replying to these questions from the agents:\n- Shall I change" in state
assert state.endswith("The user's message: Yes, approved.")
async def test_asking_the_user_again_and_again_with_nothing_done_stops_the_run(
tmp_path,
):
"""Whatever causes it, a run that only asks is stopped, not left to the turn limit."""
from app.nodes.execution_nodes import MAX_IDLE_ASKS
from app.nodes.router import TurnResult
class OnlyAsks:
"""An Orchestrator that asks the user something on every turn."""
async def turn(self, view, tools):
ask = next(tool for tool in tools if tool.name == "ask_user")
reply = await ask.handler(questions=["What now?"])
return TurnResult(reply.outcome)
engine = ExecutionEngine(
runtime=FakeAgentRuntime({}),
roles=ROLES,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=OnlyAsks(),
)
result = await engine.start("Ingest webhooks")
asked = 1
while result.status == "awaiting_user" and asked < 20:
result = await engine.resume(result.execution_id, {"reply": "Carry on."})
asked += 1
assert result.status == "completed_with_warnings"
assert asked == MAX_IDLE_ASKS + 1
assert "asked" in result.state["warnings"][-1]
assert "no work done in between" in result.state["warnings"][-1]