diff --git a/.agents/plans/README.md b/.agents/plans/README.md index d8a5df8..64cd146 100644 --- a/.agents/plans/README.md +++ b/.agents/plans/README.md @@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio | 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. | | 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. | | 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. | -| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | +| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | | 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. | | 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. | | 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | diff --git a/.agents/plans/iter-05-jev-decisions/MAP.md b/.agents/plans/iter-05-jev-decisions/MAP.md index 0416764..cf61956 100644 --- a/.agents/plans/iter-05-jev-decisions/MAP.md +++ b/.agents/plans/iter-05-jev-decisions/MAP.md @@ -86,7 +86,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an | 9 | In-run context compaction | J10 | planned | | 10 | Evaluation rubric and lesson triage | J8 | planned | | 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 | -| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit (`../iter-06-target-architecture-graph/steps/step-11.md`) | +| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` | | 13 | Cloud-read call guard | J7 | planned | | 14 | Input guardrail | J11 | planned | | 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 | diff --git a/.agents/plans/iter-05-jev-decisions/steps/step-12.md b/.agents/plans/iter-05-jev-decisions/steps/step-12.md index b609c0c..48d45c8 100644 --- a/.agents/plans/iter-05-jev-decisions/steps/step-12.md +++ b/.agents/plans/iter-05-jev-decisions/steps/step-12.md @@ -34,3 +34,18 @@ Unverified: - Whether Jev's answers to these five questions are good enough at these thresholds. The thresholds are the ones in the owner's example and are untuned. - Whether the reply writer's summaries are accurate. It is instructed to state only the facts it is given; nothing checks that it does. - Run time. The Orchestrator made 31 model requests in the last run; the router makes none, and one small writer call per Approval. No run has measured it. + +## Faults found by evaluation case 04, fixed on 2026-10-08 + +Case 04 is the first interactive run. It stopped at 100 Orchestrator turns (`../../iter-06-target-architecture-graph/steps/step-11.md`). Three changes: + +1. **Only the user's own words are their message.** A reply to questions now reaches the router with the questions it answered (`asked`) and with any quoted question and `Q:`/`A:` labels removed from its text (`_own_words` in `app/nodes/execution_nodes.py`). `_read_message` gives the questions to the decider as background and the cleaned text as the message. Before, a describer's "please confirm my draft" question, quoted back by the scripted user, was judged a request to change the Target architecture, twice. +2. **A Revision limit is told to the user once.** When a reopen is refused at the limit, the router records the Deliverable in `revision_limit_asked` and sets `awaiting_limit_decision`. The reply is not read as a request for changes, every action that would reopen that Deliverable is barred from then on, and the Execution carries on with it as it stands. Before, the reply was read as another change request for the same Deliverable and the refusal repeated until the turn limit. +3. **A run that only asks is stopped.** The engine counts questions put to the user with no change in any Deliverable's status or attempt, or any Phase's status, in between. At the fourth it ends with a warning (`MAX_IDLE_ASKS`). This holds for any Orchestrator, whatever the cause. + +Also: `agents/validator.md` now tells the validator to confirm with the user before submitting in an interactive Execution, which the case's `specialists_ask_before_submitting` check found it did not. + +Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04" (5). + +Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`. + diff --git a/.agents/plans/iter-06-target-architecture-graph/MAP.md b/.agents/plans/iter-06-target-architecture-graph/MAP.md index e234c12..4ce9868 100644 --- a/.agents/plans/iter-06-target-architecture-graph/MAP.md +++ b/.agents/plans/iter-06-target-architecture-graph/MAP.md @@ -56,7 +56,7 @@ flowchart TD | 8 | Deployment guidance implements the graph | M | complete | | 9 | Validation gives a verdict per Check | M | complete | | 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model | -| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) ends with warnings at 75% on two router faults (`steps/step-11.md`); thresholds outstanding | +| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding | Size: S under a day, M a few days. diff --git a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md index 3a42579..c26b549 100644 --- a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md +++ b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md @@ -137,3 +137,28 @@ Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not compl Not yet fixed: the two router faults above. +## Run of 2026-10-08, 12:22 UTC, unfinished (`.cache/eval-logs/case-04-20261008T122200Z.log`; no report) + +Case 04 again, with the three router fixes of iteration 5 step 12 (uncommitted at the time). The process stopped at about 14:48 local time, 86 minutes in, when the session that started it ended. It was not cancelled and wrote no score, report or token count. + +What the fixes did: + +- No reopen came from a question quoted back in a reply. +- At 14:41 `requirements` reached its Revision limit. The user was told once and the Execution carried on; there was no loop. +- The schema review raised items at submission twice (1, then 3), its first use in a real Execution. +- The unimplementable-Check route fired once and the design was revised in 92 seconds; the description was kept. + +What went wrong: + +1. **A scripted answer was read as a new requirement, three times.** The scripted user answers by matching words in the question against patterns written for the requirements analyst. Asked by the validator, and later the design advisor and the IaC author, to confirm a draft, it replied with requirement statements ("GDPR applies to employee names and emails...", "99.5% during business hours...", "PostgreSQL. Under 20 GB..."). Jev read each as a request to change `requirements`, and the router reopened it: twice, then the limit. Each reopen returned every later Deliverable to draft, so the Execution was redone from the start twice. +2. Validation failed one Check twice (an alert condition; a log bucket not locked). Each was sent back to the code, which is the intended route. + +The cause of the first is mostly the harness: no person answers "please confirm my draft" with a requirement. It also shows a weakness in the router: a reply that restates what a submitted Deliverable already says is treated as a request to change it. + +Not yet changed: + +- Scripted answers should apply only to the Specialist they were written for, and a draft-confirmation question should get an approval. +- A reply that restates what a submitted Deliverable already says should be read as an answer. + +Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one. + diff --git a/agents/validator.md b/agents/validator.md index 28f8f1f..68dded8 100644 --- a/agents/validator.md +++ b/agents/validator.md @@ -15,6 +15,7 @@ You produce the **validation results** Deliverable: the Skill's pre-deployment v - Record your verdicts with `verdict_put`, a few Checks per call, as you reach them: the Checks of one Element, then the next. Record the dry-run's result and each proposed addition the same way. Short calls are what keep your replies from timing out, and what you record is kept if your run is cut short. `verdict_read` shows which Checks still have none. - When `working_copy` shows verdicts recorded, your last run was cut short on this same code: call `verdict_read` and carry on from there. - Submit with your content and no attachments when every Check has a verdict: what you recorded is attached as `verdicts.json`, in the format the requirements-chain Skill gives. +- When `pre_approved` is `false`, confirm with the user before you submit, as every Specialist does: call `needs_input` with a summary of the verdicts and pass your content as `draft`. Your recorded verdicts are kept and attached when you submit afterwards. - Give the exact commands for every check you couldn't run, for the user to run themselves. End each of your contract's sections with a pass or fail verdict. diff --git a/app/agent.py b/app/agent.py index 971fa59..b9ba3cd 100644 --- a/app/agent.py +++ b/app/agent.py @@ -163,6 +163,10 @@ class ExecutionEngine: "pending_change": None, "awaiting_gap_decision": False, "validation_settled": 0, + "awaiting_limit_decision": False, + "revision_limit_asked": [], + "idle_asks": 0, + "ask_fingerprint": "", "decisions": [], "outcomes": [], "refusals": [], diff --git a/app/nodes/execution_nodes.py b/app/nodes/execution_nodes.py index 64c783f..89a56e3 100644 --- a/app/nodes/execution_nodes.py +++ b/app/nodes/execution_nodes.py @@ -64,6 +64,8 @@ logger = logging.getLogger(__name__) # Working-memory key: the last submission of a Deliverable with what it was written from. STANDING = "standing" +# Questions put to the user running, with no work done between, before stopping. +MAX_IDLE_ASKS = 3 NO_USER_REPLY = ( "No user is available. State your assumption, record it in the Deliverable under " @@ -266,6 +268,8 @@ def make_execution_nodes( pending_change=state.get("pending_change"), awaiting_gap_decision=bool(state.get("awaiting_gap_decision")), validation_settled=int(state.get("validation_settled", 0)), + awaiting_limit_decision=bool(state.get("awaiting_limit_decision")), + revision_limit_asked=tuple(state.get("revision_limit_asked", ())), context=_orchestrator_context( state, ledger, deliverables, pending_questions, notes ), @@ -332,6 +336,22 @@ def make_execution_nodes( return ask({"approval": toolbox.approval}) if isinstance(outcome, TurnEnded) and outcome.tool == "ask_user": + # Asking again and again with nothing done in between is a loop, not a + # conversation: stop it, whatever caused it. + fingerprint = _progress(ledger) + idle = ( + state.get("idle_asks", 0) + 1 + if fingerprint == state.get("ask_fingerprint") + else 0 + ) + update.update(idle_asks=idle, ask_fingerprint=fingerprint) + if idle >= MAX_IDLE_ASKS: + warnings.append( + f"Stopped: the user was asked {idle + 1} times running with " + "no work done in between." + ) + update["events_recorded"] = await record_progress(state, ledger) + return finish_with("end_with_warnings", warnings=warnings) update["events_recorded"] = await record_progress( state, ledger, None if pre_approved else "awaiting_user" ) @@ -519,7 +539,9 @@ def make_execution_nodes( else: text = _reply_text(reply) conversation.append({"from": "user", "text": text}) - said = {"kind": "reply", "text": text} + put = [item["question"] for item in prompt.get("questions", [])] + # What is routed is what the user said, not our questions quoted back. + said = {"kind": "reply", "text": _own_words(text, put), "asked": put} answers = { kind: list(items) for kind, items in state.get("answers", {}).items() } @@ -775,6 +797,35 @@ def _reply_text(reply: object) -> str: return str(reply) +def _own_words(text: str, asked: Sequence[str]) -> str: + """A reply without the questions it quotes, and without Q: and A: labels.""" + for question in sorted(asked, key=len, reverse=True): + text = text.replace(question, "") + lines = [] + for line in text.splitlines(): + line = re.sub(r"^\s*(?:Q|A)\s*:\s*", "", line).strip() + if line: + lines.append(line) + return "\n".join(lines) + + +def _progress(ledger: Ledger) -> str: + """The work done so far, as one comparable value: each Deliverable's status and + attempt, and each Phase's status.""" + data = ledger.to_dict() + return json.dumps( + { + "deliverables": { + kind: [record.get("status"), record.get("attempt")] + for kind, record in data["deliverables"].items() + }, + "phases": data.get("phases"), + }, + sort_keys=True, + default=str, + ) + + def _render_prompt(prompt: Mapping[str, Any]) -> str: if "approval" in prompt: approval = prompt["approval"] diff --git a/app/nodes/router.py b/app/nodes/router.py index eab2b07..926a06c 100644 --- a/app/nodes/router.py +++ b/app/nodes/router.py @@ -84,6 +84,10 @@ class TurnView: pending_change: Mapping[str, Any] | None = None # a change we asked them to confirm awaiting_gap_decision: bool = False validation_settled: int = 0 # the validation attempt already decided on + # The user's reply answers "this Deliverable reached its Revision limit". + awaiting_limit_decision: bool = False + # Deliverables at their Revision limit that the user has been told about. + revision_limit_asked: Sequence[str] = () context: Mapping[str, Any] = field(default_factory=dict) # for a model Orchestrator @@ -135,6 +139,25 @@ class Readings: judged: dict[str, Any] = field(default_factory=dict) # what the decider answered +# The actions that reopen a Deliverable, and which one. +_REOPENS = ( + "revise_requested", + "revise_code", + "revise_target_architecture", + "revise_unimplementable", +) + + +def _reopened_by(action: str, readings: Readings) -> DeliverableKind | None: + if action == "revise_requested": + return readings.change_target + if action == "revise_code": + return DeliverableKind.DEPLOYMENT_GUIDANCE + if action in {"revise_target_architecture", "revise_unimplementable"}: + return DeliverableKind.TARGET_ARCHITECTURE + return None + + def _attempt(ledger: Ledger, kind: DeliverableKind) -> int: return int(ledger.to_dict()["deliverables"][kind.value].get("attempt", 0)) @@ -194,16 +217,25 @@ class Router: "last_reply": None, "pending_change": None, "awaiting_gap_decision": False, + "awaiting_limit_decision": False, } readings = await self._read(view, result) blocked: set[str] = set() settled = view.validation_settled + # Told once, the user is not asked again: a Deliverable at its limit is no + # longer reopened, and the Execution carries on with it as it stands. + at_limit = set(view.revision_limit_asked) async def call(tool: str, **arguments: Any) -> ToolResult: return await by_name[tool].handler(**arguments) for _ in range(MAX_STEPS): - group, action = self._choose(view, readings, blocked, settled) + barred = blocked | { + name + for name in _REOPENS + if (kind := _reopened_by(name, readings)) and kind.value in at_limit + } + group, action = self._choose(view, readings, barred, settled) result.decisions.append( {"group": group, "action": action, "judged": dict(readings.judged)} ) @@ -220,9 +252,20 @@ class Router: result.outcome = reply.outcome return result if "revision limit" in reply.message and not view.pre_approved: - # Past the Revision limit the user decides. + # Past the Revision limit the user is told, once. Their reply is + # read as the answer to this, never as a new request for changes. + kind = _reopened_by(action, readings) + if kind: + at_limit.add(kind.value) + result.updates["revision_limit_asked"] = sorted(at_limit) + result.updates["awaiting_limit_decision"] = True limit = await call( - "ask_user", questions=[reply.message.split(" Use ")[0]] + "ask_user", + questions=[ + reply.message.split(" Use ")[0] + + " It will not be revised again in this Execution; I will " + "carry on with it as it stands. Reply to continue." + ], ) if limit.outcome is not None: result.outcome = limit.outcome @@ -298,7 +341,7 @@ class Router: """Triage of the user's message: is it asking for a change, to what, and how clearly?""" reply = view.last_reply - if not reply or view.awaiting_gap_decision: + if not reply or view.awaiting_gap_decision or view.awaiting_limit_decision: return text = str(reply.get("text", "")).strip() pending = view.pending_change @@ -308,6 +351,7 @@ class Router: if not text or not candidates: return readings.change_reason = text + asked = [str(question) for question in reply.get("asked") or ()] # A structured request for changes is known to be one, and so is the reply to # our own question about a change they already asked for. structured = reply.get("kind") == "changes" or bool(pending) @@ -334,7 +378,16 @@ class Router: state = ( f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n" f"Questions waiting for the user: {len(view.pending_questions)}.\n" - f"The user's message: {text}" + # What we asked is background. Only the user's own words are their + # message: our question echoed back is not something they said. + + ( + "The user was replying to these questions from the agents:\n" + + "\n".join(f"- {question}" for question in asked) + + "\n" + if asked + else "" + ) + + f"The user's message: {text}" ) answers = await self._decider.decide(state, questions, scope=view.execution_id) diff --git a/app/states/execution_state.py b/app/states/execution_state.py index 722f477..2f4b5e6 100644 --- a/app/states/execution_state.py +++ b/app/states/execution_state.py @@ -87,6 +87,10 @@ class ExecutionState(TypedDict, total=False): pending_change: dict[str, Any] | None # a change the user was asked to confirm awaiting_gap_decision: bool # the user was asked whether to close gaps validation_settled: int # the validation attempt already decided on + awaiting_limit_decision: bool # the user was told a Revision limit was reached + revision_limit_asked: list[str] # Deliverables the user has been told that about + idle_asks: int # questions put to the user running, with nothing changing between + ask_fingerprint: str # the Ledger's state when the user was last asked decisions: list[dict[str, Any]] # every routing decision, in order turns: int diff --git a/tests/execution/test_router.py b/tests/execution/test_router.py index f7a4596..305e1ce 100644 --- a/tests/execution/test_router.py +++ b/tests/execution/test_router.py @@ -596,3 +596,162 @@ async def test_past_the_revision_limit_the_user_decides(tmp_path): "deployment_guidance has reached its revision limit (1)" in (question["question"]) ) + + +# --- the two faults of evaluation case 04 (2026-10-08) ----------------------------------- + + +async def test_a_reply_at_the_revision_limit_is_the_answer_and_the_run_carries_on( + tmp_path, +): + """The user is told once. Their reply is not read as a new request for changes, + the Deliverable is not reopened again, and nothing is asked a second time.""" + turns = once() + turns[specialist(K.DEPLOYMENT_GUIDANCE)] *= 2 + turns[specialist(K.VALIDATION_RESULTS)] = [[failing("K-2")], [failing("K-1")]] + # A decider that would call any reply a clear request to change the code. + decider = FakeDecider( + { + "asks_change": Answer(0.95, 0.9), + "change_target": Answer("deployment_guidance", 0.95), + "change_clarity": Answer(1.9, 0.8), + } + ) + engine, _ = engine_for(tmp_path, turns, decider, max_revisions=1) + + result = await engine.start("Ingest webhooks") + for phase in ( + "requirements_discovery", + "solution_architecture", + "solution_implementation", + "solution_implementation", + ): + result = await engine.resume(result.execution_id, approve(phase)) + [question] = result.prompt["questions"] + assert "I will carry on with it as it stands" in question["question"] + + result = await engine.resume( + result.execution_id, {"reply": "Please change the code again."} + ) + + # It moves on to the Approval of validation; the code was reopened once only. + assert result.prompt["approval"]["phase"] == "solution_validation" + assert len(reopened(result)) == 1 + assert result.state["revision_limit_asked"] == ["deployment_guidance"] + assert decider.asked == [] # the reply was never put to the decider + + +async def test_only_the_users_own_words_are_read_as_their_message(tmp_path): + """A reply that quotes our question back must not be judged on our words.""" + question = "Please confirm the requirements draft. Shall I change anything?" + turns = once() + turns[specialist(K.REQUIREMENTS)] = [ + [Call("needs_input", {"questions": [question]})], + [sample_submission(K.REQUIREMENTS)], + ] + decider = FakeDecider( + { + "asks_change": Answer(0.05, 0.9), + "change_target": Answer("requirements", 0.5), + "change_clarity": Answer(0.2, 0.5), + } + ) + engine, _ = engine_for(tmp_path, turns, decider) + first = await engine.start("Ingest webhooks") + + result = await engine.resume( + first.execution_id, {"reply": f"Q: {question}\nA: Yes, approved."} + ) + # Nothing has been submitted yet, so nothing is judged on that turn. Let the + # next reply, to the decomposer's approval prompt, be a text one. + result = await engine.resume(result.execution_id, {"reply": "Looks fine."}) + + assert result.state["last_reply"] is None + state, _, _ = decider.asked[-1] + assert state.endswith("The user's message: Looks fine.") + + +def test_a_quoted_question_is_removed_from_what_the_user_said(): + from app.nodes.execution_nodes import _own_words + + asked = ["Please confirm the draft. Shall I change the design?", "Which region?"] + reply = ( + "Q: Please confirm the draft. Shall I change the design?\nA: Yes, approved.\n\n" + "Q: Which region?\nA: europe-west2" + ) + + assert _own_words(reply, asked) == "Yes, approved.\neurope-west2" + assert _own_words("Make retention 7 years.", asked) == "Make retention 7 years." + + +async def test_the_questions_asked_are_given_to_the_decider_as_background(tmp_path): + from app.nodes.router import TurnView + from app.states.ledger import Ledger + + ledger = Ledger() + ledger.start([K.REQUIREMENTS], actor="orchestrator") + ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst") + decider = FakeDecider( + { + "asks_change": Answer(0.05, 0.9), + "change_target": Answer("requirements", 0.5), + "change_clarity": Answer(0.2, 0.5), + } + ) + router = Router(decider, None, Thresholds()) + view = TurnView( + execution_id="e1", + pre_approved=False, + ledger=ledger, + deliverables={}, + pending_questions=[], + outcomes=[], + last_reply={ + "kind": "reply", + "text": "Yes, approved.", + "asked": ["Shall I change the design?"], + }, + ) + from app.nodes.router import Readings + + await router._read_message(view, Readings()) + + state, _, _ = decider.asked[0] + assert "replying to these questions from the agents:\n- Shall I change" in state + assert state.endswith("The user's message: Yes, approved.") + + +async def test_asking_the_user_again_and_again_with_nothing_done_stops_the_run( + tmp_path, +): + """Whatever causes it, a run that only asks is stopped, not left to the turn limit.""" + from app.nodes.execution_nodes import MAX_IDLE_ASKS + from app.nodes.router import TurnResult + + class OnlyAsks: + """An Orchestrator that asks the user something on every turn.""" + + async def turn(self, view, tools): + ask = next(tool for tool in tools if tool.name == "ask_user") + reply = await ask.handler(questions=["What now?"]) + return TurnResult(reply.outcome) + + engine = ExecutionEngine( + runtime=FakeAgentRuntime({}), + roles=ROLES, + recorder=InMemoryRecorder(), + checkpointer=InMemorySaver(), + output_root=tmp_path, + orchestrator=OnlyAsks(), + ) + + result = await engine.start("Ingest webhooks") + asked = 1 + while result.status == "awaiting_user" and asked < 20: + result = await engine.resume(result.execution_id, {"reply": "Carry on."}) + asked += 1 + + assert result.status == "completed_with_warnings" + assert asked == MAX_IDLE_ASKS + 1 + assert "asked" in result.state["warnings"][-1] + assert "no work done in between" in result.state["warnings"][-1]