Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
79 lines
2.9 KiB
Python
79 lines
2.9 KiB
Python
"""A review of the requirements: they describe the problem, not the solution.
|
|
|
|
Requirements discovery is kept apart from solutioning, so a requirement names no
|
|
product. That rule was an instruction the engine could not enforce; the evaluation
|
|
checked it only afterwards, against a list of product names. This asks the decider
|
|
(ADR-0007), once per requirement, whether it names a cloud product or managed
|
|
service. It never blocks: what it raises is returned to the Specialist once.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from collections.abc import Mapping, Sequence
|
|
from typing import Any
|
|
|
|
from app.adk.runtime import Decider, Question
|
|
from app.workflows.chain import REQUIREMENTS_FILE, read_list
|
|
from app.workflows.deliverables import DeliverableKind
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
_TEXT = 200
|
|
|
|
|
|
class RequirementsReview:
|
|
def __init__(self, decider: Decider, *, act: float = 0.3) -> None:
|
|
self._decider = decider
|
|
# Lower than the router's threshold, from measurement on 2026-10-08: across
|
|
# 61 requirements from three passed runs the highest answer was 0.10, and
|
|
# requirements written to name a product scored 0.34, 0.61 and 0.93.
|
|
self._act = act
|
|
|
|
async def __call__(
|
|
self,
|
|
kind: DeliverableKind,
|
|
attachments: Mapping[str, str],
|
|
upstream: Mapping[str, Mapping[str, Any]] | None,
|
|
scope: str = "",
|
|
) -> Sequence[str]:
|
|
"""The requirements that name a product, each as a line for the Specialist."""
|
|
if kind is not DeliverableKind.REQUIREMENTS:
|
|
return ()
|
|
items = [
|
|
item
|
|
for item in read_list(
|
|
{"requirements": {"attachments": attachments}},
|
|
"requirements",
|
|
REQUIREMENTS_FILE,
|
|
)
|
|
if isinstance(item.get("id"), str) and isinstance(item.get("text"), str)
|
|
]
|
|
if not items:
|
|
return ()
|
|
answers = await self._decider.decide(
|
|
"Requirements for a workload, written before any product is chosen.",
|
|
{
|
|
item["id"]: Question(
|
|
"noul",
|
|
f'Requirement: "{item["text"]}". Does it name a specific cloud '
|
|
"provider's product or managed service, as opposed to "
|
|
"describing what is needed?",
|
|
)
|
|
for item in items
|
|
},
|
|
scope=scope,
|
|
)
|
|
if not answers:
|
|
return ()
|
|
raised = [
|
|
f'{item["id"]} names a product or service: "{item["text"][:_TEXT]}". '
|
|
"Say what is needed, not what to use"
|
|
for item in items
|
|
if (answer := answers.get(item["id"])) is not None
|
|
and answer.is_yes(self._act)
|
|
]
|
|
if raised:
|
|
logger.info("[%s] requirements review raised %d", scope, len(raised))
|
|
return tuple(raised)
|