feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
78
app/tools/requirements_review.py
Normal file
78
app/tools/requirements_review.py
Normal file
@@ -0,0 +1,78 @@
|
||||
"""A review of the requirements: they describe the problem, not the solution.
|
||||
|
||||
Requirements discovery is kept apart from solutioning, so a requirement names no
|
||||
product. That rule was an instruction the engine could not enforce; the evaluation
|
||||
checked it only afterwards, against a list of product names. This asks the decider
|
||||
(ADR-0007), once per requirement, whether it names a cloud product or managed
|
||||
service. It never blocks: what it raises is returned to the Specialist once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
from app.workflows.chain import REQUIREMENTS_FILE, read_list
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TEXT = 200
|
||||
|
||||
|
||||
class RequirementsReview:
|
||||
def __init__(self, decider: Decider, *, act: float = 0.3) -> None:
|
||||
self._decider = decider
|
||||
# Lower than the router's threshold, from measurement on 2026-10-08: across
|
||||
# 61 requirements from three passed runs the highest answer was 0.10, and
|
||||
# requirements written to name a product scored 0.34, 0.61 and 0.93.
|
||||
self._act = act
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
kind: DeliverableKind,
|
||||
attachments: Mapping[str, str],
|
||||
upstream: Mapping[str, Mapping[str, Any]] | None,
|
||||
scope: str = "",
|
||||
) -> Sequence[str]:
|
||||
"""The requirements that name a product, each as a line for the Specialist."""
|
||||
if kind is not DeliverableKind.REQUIREMENTS:
|
||||
return ()
|
||||
items = [
|
||||
item
|
||||
for item in read_list(
|
||||
{"requirements": {"attachments": attachments}},
|
||||
"requirements",
|
||||
REQUIREMENTS_FILE,
|
||||
)
|
||||
if isinstance(item.get("id"), str) and isinstance(item.get("text"), str)
|
||||
]
|
||||
if not items:
|
||||
return ()
|
||||
answers = await self._decider.decide(
|
||||
"Requirements for a workload, written before any product is chosen.",
|
||||
{
|
||||
item["id"]: Question(
|
||||
"noul",
|
||||
f'Requirement: "{item["text"]}". Does it name a specific cloud '
|
||||
"provider's product or managed service, as opposed to "
|
||||
"describing what is needed?",
|
||||
)
|
||||
for item in items
|
||||
},
|
||||
scope=scope,
|
||||
)
|
||||
if not answers:
|
||||
return ()
|
||||
raised = [
|
||||
f'{item["id"]} names a product or service: "{item["text"][:_TEXT]}". '
|
||||
"Say what is needed, not what to use"
|
||||
for item in items
|
||||
if (answer := answers.get(item["id"])) is not None
|
||||
and answer.is_yes(self._act)
|
||||
]
|
||||
if raised:
|
||||
logger.info("[%s] requirements review raised %d", scope, len(raised))
|
||||
return tuple(raised)
|
||||
Reference in New Issue
Block a user