diff --git a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md index 40ae7d0..0074b22 100644 --- a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md +++ b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md @@ -18,3 +18,41 @@ Outstanding, and why: - **Thresholds** from iteration 4 step 8 are still unset. Do not mark this step complete until a run meets the criteria. (`legacy-baseline` was removed on 2026-10-07 on the owner's instruction, ahead of this.) + +## Evaluation run of 2026-10-07, case 01 (commit `e924682`) + +Report: `eval/results/20261007T125917Z/report.json`. Result: `completed_with_warnings`, score 73% (8 of 11 checks), 3,627 seconds, 4.25 million tokens, 144 model requests. No solution guide was written. + +| | Stopped run, 2026-10-06 | This run | +|---|---|---| +| First pass to a validation | about 17 minutes | 25 minutes | +| Validation passes | 3 | 2 | +| Failing items per pass | 4, then 7, then 5 findings | 0 failed Checks and 7 proposed additions, then 1 failed Check, 1 not verifiable and 1 proposed addition | +| Revisions | deployment guidance × 3 | Target architecture × 1, deployment guidance × 3 | +| Outcome | stopped at 35 minutes | ended with warnings at 60 minutes | + +What held: + +- Every chain link was accepted on a first attempt except the Target architecture, which was refused once per attempt (7 requirements with no Check; then 4 Elements with no Check) and fixed inside its own run. +- `requirements_chain_complete` and `diagram_matches_graph` passed. +- The Terraform met every Check on its first attempt (0 failed of the first graph's Checks). +- The Orchestrator followed the new routing: additions reopened the Target architecture; a failed Check reopened the code. +- After one revision of the graph the validator proposed 1 addition, down from 7. + +What failed, in order of cause: + +1. **A bug in this iteration (fixed after the run, with a test).** Once the deployment guidance was reopened, the Orchestrator's context still carried the old validation's verdicts. It reopened the code for K-30 three times in a row without a validation in between, reaching the Revision limit. `_orchestrator_context` now shows a validation only while the validation results stand as submitted. +2. **A Check that cannot be implemented had nowhere to go.** K-30 required `startup_cpu_boost` on a Cloud Run job. Terraform rejects that argument there, so the IaC author could not satisfy it, and the validator marked it failed, which sends it back to the code. The fault was in the graph. The IaC author used 57 requests and 2.35 million tokens, 55% of the run. +3. **The target architect put a recommendation on an Element it does not apply to.** K-30 came from DR-19, proposed by the validator in pass 1 for two services and applied to the job as well. +4. **Model timeouts.** The target architect's model request timed out at 180 seconds in both attempts; each cost about 3 minutes. + +Not yet exercised: the second pass also had a Check marked not verifiable (K-55) and a proposed addition (NFR-15). The failed-Check route took precedence, as instructed, so the gap route was not taken a second time. + +Acceptance criteria: not met. The case did not complete or reach 0.8. + +Changes this points to, not yet made: + +- Let the IaC author report a Check it cannot implement, with the tool's error as evidence, so that it is routed to the Target architecture and not retried as code. +- Have the validator distinguish "the code does not do this" from "this cannot be done to this resource". +- Check a proposed addition against the Element it names before it becomes a Check. +- Raise or tune the model timeout for the target architect. diff --git a/app/nodes/execution_nodes.py b/app/nodes/execution_nodes.py index f2b79cd..59435e6 100644 --- a/app/nodes/execution_nodes.py +++ b/app/nodes/execution_nodes.py @@ -41,7 +41,7 @@ from app.states.execution_state import ( execution_dir, lessons_file, ) -from app.states.ledger import Ledger, LedgerRefusal +from app.states.ledger import DeliverableStatus, Ledger, LedgerRefusal from app.states.lessons import collect_lesson_candidates, refusal from app.tools.execution import ORCHESTRATOR, OrchestratorToolbox, specialist_tools from app.workflows.chain import VERDICTS_FILE, validation_outcome @@ -515,7 +515,14 @@ def _orchestrator_context( pending_questions: Sequence[Mapping[str, Any]], notes: Sequence[str], ) -> dict[str, Any]: - validation = deliverables.get(DeliverableKind.VALIDATION_RESULTS.value) + # Only a validation that still stands: once anything it judged is reopened, its + # verdicts are about work that no longer exists. + validation = ( + deliverables.get(DeliverableKind.VALIDATION_RESULTS.value) + if ledger.status(DeliverableKind.VALIDATION_RESULTS) + is DeliverableStatus.SUBMITTED + else None + ) return { # The verdict on each Check, once validation is submitted: failed Checks are # faults in the code; the rest are gaps in the Target architecture. diff --git a/tests/execution/test_requirements_chain.py b/tests/execution/test_requirements_chain.py index 124b4ad..0211f79 100644 --- a/tests/execution/test_requirements_chain.py +++ b/tests/execution/test_requirements_chain.py @@ -607,6 +607,9 @@ async def test_the_orchestrator_is_told_what_validation_decided(tmp_path): {"check": "K-1", "evidence": "min instances is 1"} ] assert after["validation"]["proposed_additions"] == [] + # Once the code is reopened those verdicts are about work that no longer exists, + # so the next turn must not see them and reopen the same thing again. + assert turns[8].task.context["validation"] is None # Revising the code leaves the approved design standing. phases = result.state["ledger"]["phases"] assert phases["solution_architecture"] == "approved"