feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s

A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.

Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 19:30:12 +01:00
parent f9cc34817b
commit a48468d477
15 changed files with 375 additions and 17 deletions

View File

@@ -45,6 +45,7 @@
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
| 29 | What the request does not say is a question for the customer | none | M | built 2026-10-10: a fourth checklist outcome, a questions report, and the judged check counts it; not run (`steps/step-29.md`) |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -0,0 +1,24 @@
# Step 29 — What the request does not say is a question for the customer
Built on 2026-10-10, on the owner's direction. **Not run with real models.**
## What prompted it
The owner labelled the judge's 32 questions (iteration 5 step 16): yes to all 24 checklist items. Both judges had answered no to most; they agreed with the labels on 28% and 16%. The owner's reading was deliberate: where a checklist item is unanswered because the customer's request gives no information about it, that is not the design failing a standard. It is missing information, and should come back as a question to put to the customer.
The checklist review had three outcomes: addressed, does not apply, not addressed. The third was the only place such an item could go, and it reads as a fault.
## What was built
- **A fourth outcome, `needs_customer_input`** ("To confirm with the customer"), in `app/tools/pillars.py`. A pillar advisor gives it to an item that could apply where its brief does not say what would settle it, and must give the question to ask; without the question its submission is refused. `not_addressed` is now only for an item the brief gives enough to settle, that applies, and that nothing is recommended for.
- **The questions are reported.** Code writes a "Questions for the customer" table into the design recommendations, under the checklist review, saying they are not faults in the design. An Execution also writes `customer-questions.md` beside its other files, and `execution.json` names it. The packager carries the section into the guide.
- **The evaluation counts it as accounted for.** The judged check asks whether the recommendations account for an item by recommending it, by explaining why it does not apply, or by recording it as something to confirm with the customer (`checklist_question` in `app/tools/assessment.py`, used by the evaluation and by calibration).
- **Glossary:** Customer question.
Nothing is asked of the user during an Execution for these: they are reported. Whether an interactive Execution should put them to the user before the design is approved is not decided.
## The judge's question set
The owner's labels were not kept. The two saved documents predate the checklist review, so they hold neither statuses nor questions, and the question's wording has changed. The set has to be rebuilt from design recommendations produced after this step (`eval.calibrate_judge --build`, after replacing the two documents), and labelled again. That needs a run.
Tests: `tests/execution/test_pillars.py` (4 more).

View File

@@ -75,7 +75,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| Step | What | Report ref | Status |
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 1 | Resolve the open questions | none | the vendor's answers on retention, training, context and rate limits recorded on 2026-10-10, with the owner's decisions; retention for client work is held (`steps/step-1.md`) |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
@@ -90,7 +90,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; ran in case 05, where nothing was reworded |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; the owner labelled the set on 2026-10-10, which showed the question was wrong, not the judges; the set is to be rebuilt and relabelled after iteration 4 step 29 (`steps/step-16.md`) |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |

View File

@@ -0,0 +1,37 @@
# Step 1 — The open questions, and the owner's decisions of 2026-10-10
## What the vendor says
Supplied by the owner on 2026-10-10. Not verified here against the vendor's documents.
| Question | Answer |
|---|---|
| Data retention | No fixed period for a standard account: data is kept "for as long as reasonably necessary". Zero data retention is available to enterprise customers on request, through Cloudflare Workers AI, or through Vercel AI Gateway. |
| Model training | Customer prompts, inputs and outputs are not used to train or fine-tune the vendor's models. |
| Context | 64,000 tokens a request in total, with 32,000 for the state plus the longest question. |
| Rate limits | Up to 80 requests a second and 100,000 tokens a second on the standard hosted service, adjusted with platform traffic. |
Against the client's own limits (`JevLimits` in `app/tools/jev.py`): it caps a request at 48,000 and 24,000 tokens, inside the service's; and throttles itself to 5 requests a second with 8 in flight, a sixteenth of the service's rate. Nothing needs changing for either.
Still open: latency, and whether the vendor's SDK can be pointed at either gateway.
## What is sent to Jev
ADR-0007's decision 4 limits an Orchestrator decision to the Execution's state. The uses built since send content: the user's messages and the request (routing, intake, guardrail), replies from documentation servers (guardrail), the design recommendations (assessment), the text of Checks, Elements and recommendations (meaning), a change and the list of Elements (pillar selection, reference routing), and each requirement's text when the requirements review is on.
## The owner's decisions
| Question | Decision |
|---|---|
| Re-verify only what changed after a Revision | **No.** Every validation stays complete. The saving was 5 to 10% on some runs, and the risk fell on the validator. |
| The requirements review (`JEV_REQUIREMENTS_REVIEW`) | **On for the next runs that happen anyway; the default if they are clean.** No run is to be made for it. Cases 01 and 02 are held until the owner says. |
| Tracing (step 17) | **Held.** No decision. |
| The judge against a person (step 16) | **The owner will label the 32 questions.** |
| Jev's data retention, for client work | **Held.** No decision. |
## Advice given, with no decision asked
- With no training use, the remaining exposure is content kept by the vendor for an unstated time. Zero retention is the only route that costs the agent nothing. A switch that keeps Deliverable content away from Jev is a stopgap; even routing a reply sends that reply.
- The owner may look for an open-source version of Jev. That would end the retention question. Its cost is that every threshold, tuned against the hosted model's probabilities, would need measuring again, and the two properties relied on (the same answer every time, clean separation) would need proving again. The agent is placed for it: one interface, every threshold a setting, every use a fallback.
- **A gap either way:** the calibrations in this plan were made with one-off scripts; the example sets are not in the repository. Kept as a runnable suite they could be put to any model. Not built.
- The owner's 32 labels are the one measure that depends on no model.

View File

@@ -18,3 +18,9 @@ So Jev gave the same answer every time, the language-model judge changed its ans
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
## The owner's labels, 2026-10-10
The owner labelled all 32: yes to every checklist item, "mostly" on every quality question. Against them Jev agreed on 28% and Gemini on 16%; the judges answer no to most checklist items (means 0.37 and 0.32), and agree with each other on 88%.
The disagreement was about the question, not the judges. The owner counts an item the request gives no information about as something to ask the customer, not as unaddressed. That is now an outcome of the checklist review and part of the judged question (iteration 4 step 29). The labels were not kept, and the set is to be rebuilt from documents produced after that change.

View File

@@ -108,6 +108,10 @@ _Avoid_: finding, rule, control, test
The unbroken trace from each requirement, through the components that serve it and the Elements and Checks that realise it, to the code that implements them and the validation verdict on each Check.
_Avoid_: traceability matrix, lineage
**Customer question**:
Something the architecture depends on that the customer's Intent does not say, recorded with the question to put to them. It marks missing information, not a fault in the design, and its answer may change a recommendation.
_Avoid_: gap, failure, open issue, assumption (which is a default the agents chose)
**Solution guide**:
The packaged Deliverable that assembles every other Deliverable into the Cloud provider's solution architecture template.
_Avoid_: final report, package

View File

@@ -11,6 +11,7 @@ You produce the **design recommendations** Deliverable: the Skill's task for rec
- Otherwise, when `feedback` is not empty or `baseline` is set you are revising: pass `pillars` naming only the pillars the change touches. The guidance you were given before is returned for the others.
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
- The advisors' checklist reviews, and any questions they raise for the customer, are written into your Deliverable for you when you submit. Do not copy them, and do not write a recommendation to cover a question an advisor left for the customer.
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture. Give each its `conditions`: one entry for each separate thing it asks to be configured, and for each component it must hold on when it names several. Put under `outside_code` anything it asks of the application's own code or of people, and any property of a region or product that only its provider publishes, such as a carbon rating: no infrastructure configuration can show those. The condition is then the choice configuration can show, such as which region is set.
Your contract's sections are one per pillar. The advisors' review of each pillar's checklist is added to your Deliverable for you when you submit, so do not write it yourself.

View File

@@ -10,6 +10,7 @@ You produce the **solution guide** Deliverable: the Skill's packaging of every a
- Consolidate the content of every Deliverable in `dependencies`; rewrite for flow, but keep every requirement, product decision, citation, recommendation, command and validation verdict.
- Include the architecture diagram, which is the `architecture.mmd` attachment of the Target architecture, unchanged as a `mermaid` code block, and the Terraform as code blocks.
- List the Target architecture's Checks with their validation verdicts in the validation section.
- When the design recommendations hold a "Questions for the customer" section, carry every question into the guide's design and configuration recommendations section, under the same heading and with the same wording. They are what the request did not say enough to settle; say so, and do not present them as faults or as recommendations.
In a Change request (`baseline` is set), open the executive summary with what changed and why, from `baseline.changes`: the requirements added, changed and removed with their reasons, then what that changed in the architecture. Name separately anything listed under `unrequested`: it changed without a changed requirement reaching it.

View File

@@ -9,8 +9,14 @@ Your task message carries the design advisor's brief in `brief` and the pillar's
1. Load the Skill available to you.
2. Evaluate the workload in the brief against the Skill's core principles and validation checklist.
3. Write prioritised, actionable recommendations for each relevant component, naming the products the Skill lists for each gap and citing its guidance.
4. List any assessment questions from the Skill that the brief leaves unanswered, as open questions; don't ask the user.
4. Do not ask the user anything, and do not guess what the brief does not say.
5. Review the Skill's validation checklist item by item, when it has one. For every item give its `status`: `addressed` when your recommendations cover it, `not_applicable` when it does not apply to this workload or is not something an architecture can settle (an organisation's process, for example), or `not_addressed` when it applies and you recommend nothing for it. In `note`, say in a few words how it is addressed or why it does not apply. Copy each item's text as the Skill has it, shortened to its first sentence.
5. Review the Skill's validation checklist item by item, when it has one. For every item give its `status`:
- `addressed` when your recommendations cover it.
- `not_applicable` when it does not apply to this workload.
- `needs_customer_input` when it could apply and the brief does not say what would settle it: how the customer's organisation works, what they already have, or what they want. Give the `question` to put to the customer, written so someone who knows their organisation can answer it. This is the right status whenever the gap is in what you were told, not in the design. Do not invent a recommendation to cover it.
- `not_addressed` only when the brief gives you what you need, the item applies, and you recommend nothing for it.
In `note`, say in a few words how it is addressed, why it does not apply, or what the brief leaves unsaid. Copy each item's text as the Skill has it, shortened to its first sentence.
Call `submit_advice` with your recommendations as markdown in `content` and the review in `checklist`. You are done when it accepts them.

View File

@@ -54,7 +54,13 @@ from app.tools.execution import ORCHESTRATOR, OrchestratorToolbox, specialist_to
from app.tools.guardrail import EXECUTION_SCOPE, instructs_agent
from app.tools.meaning import settle_meaning
from app.tools.modules import MODULE_TOOL, MODULES_OPTED_IN
from app.tools.pillars import ADVICE_KEY, KEPT_FOR_CHANGES, pillars_touched
from app.tools.pillars import (
ADVICE_KEY,
KEPT_FOR_CHANGES,
QUESTIONS_HEADING,
customer_questions,
pillars_touched,
)
from app.tools.skill_routing import read_first
from app.workflows.chain import (
IMPLEMENTATION_FILE,
@@ -83,6 +89,7 @@ NO_USER_REPLY = (
"'Assumptions', and proceed."
)
GUIDE_FILENAME = "solution-architecture-guide.md"
QUESTIONS_FILENAME = "customer-questions.md" # what to ask the customer
MANIFEST_FILENAME = "execution.json" # request, status and files of one Execution
@@ -1237,7 +1244,11 @@ def _written_before(out_dir: Path) -> set[str]:
return set()
if not isinstance(manifest, dict):
return set()
names = {manifest.get("solution_guide"), manifest.get("changes")}
names = {
manifest.get("solution_guide"),
manifest.get("changes"),
manifest.get("customer_questions"),
}
for entry in (manifest.get("deliverables") or {}).values():
if isinstance(entry, dict):
names.add(entry.get("document"))
@@ -1293,6 +1304,28 @@ def _write_outputs(
if kept:
# Working material a Change request starts from, such as each pillar's advice.
files[WORKING_FILENAME] = json.dumps(kept, indent=2, ensure_ascii=False)
questions = customer_questions(
state.get("working", {}).get(DeliverableKind.DESIGN_RECOMMENDATIONS.value, {})
)
if questions:
# What the request did not say enough to settle, as questions to put to
# the customer. Missing information, not a fault in the design.
files[QUESTIONS_FILENAME] = "\n".join(
[
f"# {QUESTIONS_HEADING}",
"",
(
"The request did not say enough to settle these. Each answer "
"may change a recommendation."
),
"",
*(
f"- **{item['pillar']}**, {item['item']} {item['question']}"
for item in questions
),
"",
]
)
baseline = state.get("baseline") or None
if baseline:
# What this Change request changed against its Baseline, worked out from
@@ -1334,6 +1367,8 @@ def _write_outputs(
phase.value: ledger.phase_status(phase).value for phase in PHASE_ORDER
},
"solution_guide": GUIDE_FILENAME if guide else None,
# Questions the request left open, for the customer; None when there are none.
"customer_questions": QUESTIONS_FILENAME if questions else None,
"changes": CHANGES_FILE if baseline else None,
"deliverables": by_kind,
}

View File

@@ -32,6 +32,21 @@ QUALITY_RUBRIC: Mapping[str, str] = {
_JUDGED = "design_recommendations"
def checklist_question(pillar: str, item: str) -> str:
"""Whether the recommendations account for one item of a pillar's checklist.
An item is accounted for in three ways. The third matters: a customer's request
seldom says enough to settle every item, and what it leaves unsaid is a
question for them, not a fault in the design (the owner, 2026-10-10).
"""
return (
f"Do the recommendations account for this {pillar} checklist item in any "
"of these ways: by recommending it, by explaining why it does not apply, "
"or by recording it as something to confirm with the customer? "
f"{item}"
)
async def assess(
decider: Decider,
deliverables: Mapping[str, Mapping[str, Any]],

View File

@@ -39,12 +39,18 @@ REVIEW_HEADING = "Architecture framework checklist review"
# What a Change request carries over from its Baseline's working memory.
KEPT_FOR_CHANGES = (ADVICE_KEY, CHECKLIST_KEY)
_CHANGE_LIMIT = 6_000
STATUSES = ("addressed", "not_applicable", "not_addressed")
NEEDS_CUSTOMER = "needs_customer_input"
STATUSES = ("addressed", "not_applicable", NEEDS_CUSTOMER, "not_addressed")
_STATUS_WORDS = {
"addressed": "Addressed",
"not_applicable": "Does not apply",
# The item applies, and the customer's request does not say what would settle
# it. That is missing information, not a fault in the design: it is reported
# with the question to put to the customer.
NEEDS_CUSTOMER: "To confirm with the customer",
"not_addressed": "Not addressed",
}
QUESTIONS_HEADING = "Questions for the customer"
AFFECTED = ("not at all", "only indirectly", "directly")
@@ -215,17 +221,29 @@ def _submit_advice_tool(review: list[dict[str, str]]) -> ToolSpec:
async def submit_advice(content: str, checklist: Sequence[Any] = ()) -> ToolResult:
if not content.strip():
return ToolResult("Not submitted: the advice is empty.")
review[:] = [
entries = [
{
"item": " ".join(str(entry.get("item", "")).split()),
"status": str(entry.get("status", "")),
"note": " ".join(str(entry.get("note", "")).split()),
"question": " ".join(str(entry.get("question") or "").split()),
}
for entry in checklist or ()
if isinstance(entry, Mapping)
and str(entry.get("item", "")).strip()
and entry.get("status") in STATUSES
]
unasked = [
entry["item"]
for entry in entries
if entry["status"] == NEEDS_CUSTOMER and not entry["question"]
]
if unasked:
return ToolResult(
"Not submitted: an item to confirm with the customer needs the "
"`question` to put to them. Missing for: " + "; ".join(unasked)
)
review[:] = entries
return ToolResult("Advice submitted.", Submitted(content))
text = {"type": "string"}
@@ -248,8 +266,15 @@ def _submit_advice_tool(review: list[dict[str, str]]) -> ToolSpec:
"status": {"type": "string", "enum": list(STATUSES)},
"note": {
"type": "string",
"description": "How your advice addresses it, or "
"why it does not apply to this workload.",
"description": "How your advice addresses it, why "
"it does not apply to this workload, or what the "
"request leaves unsaid.",
},
"question": {
"type": "string",
"description": "For needs_customer_input only: "
"the question to put to the customer, answerable "
"by someone who knows their organisation.",
},
},
"required": ["item", "status", "note"],
@@ -274,7 +299,8 @@ def render_checklist_review(working: Mapping[str, Any]) -> str:
"",
(
"Each pillar advisor's review of its pillar's checklist for this "
"workload: what its advice addresses, and what does not apply and why."
"workload: what its advice addresses, what does not apply and why, and "
"what the request does not say enough to settle."
),
]
for pillar, entries in reviews.items():
@@ -301,4 +327,48 @@ def render_checklist_review(working: Mapping[str, Any]) -> str:
for entry in entries
if isinstance(entry, Mapping)
]
questions = customer_questions(working)
if questions:
parts += [
"",
f"### {QUESTIONS_HEADING}",
"",
(
"The request does not say enough to settle these. They are not "
"faults in the design: each is something to ask the customer, and "
"the answer may change a recommendation."
),
"",
"| Pillar | About | Question |",
"| --- | --- | --- |",
]
parts += [
"| "
+ " | ".join(
value.replace("|", "\\|")
for value in (item["pillar"], item["item"], item["question"])
)
+ " |"
for item in questions
]
return "\n".join(parts) + "\n"
def customer_questions(working: Mapping[str, Any]) -> list[dict[str, str]]:
"""What the advisors found the request does not say: each checklist item that
needs the customer's answer, with the question to put to them."""
reviews = working.get(CHECKLIST_KEY)
if not isinstance(reviews, Mapping):
return []
return [
{
"pillar": str(pillar),
"item": str(entry.get("item", "")),
"question": str(entry.get("question", "")),
}
for pillar, entries in reviews.items()
for entry in entries or ()
if isinstance(entry, Mapping)
and entry.get("status") == NEEDS_CUSTOMER
and entry.get("question")
]

View File

@@ -31,6 +31,7 @@ from typing import Any
from app.agent import open_decider
from app.config import get_settings
from app.providers import load_provider
from app.tools.assessment import checklist_question
from eval.judge import DeciderJudge, GeminiJudge, Judge, JudgeUnavailable
from eval.scoring import QUALITY_RUBRIC, checklist_items
@@ -55,9 +56,7 @@ def build(seed: int = 7) -> list[dict[str, Any]]:
"id": f"{document.stem}.{pillar}.{len(items)}",
"document": document.name,
"kind": "yes_no",
"text": f"Do the recommendations address this {pillar} "
"checklist item, either by recommending it or by "
f"explaining why it does not apply? {item}",
"text": checklist_question(pillar, item),
"label": None,
}
)

View File

@@ -20,7 +20,7 @@ from typing import Any
from app.adk.runtime import ToolResult
from app.agent import Interaction
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC, checklist_question
from app.tools.local import mermaid_check
from app.workflows.chain import (
COMPONENTS_FILE,
@@ -417,8 +417,7 @@ async def _checklist_coverage(
questions = {
f"{pillar}.{index}": {
"kind": "yes_no",
"text": f"Do the recommendations address this {pillar} checklist item, "
f"either by recommending it or by explaining why it does not apply? {item}",
"text": checklist_question(pillar, item),
}
for pillar, items in checklists.items()
for index, item in enumerate(items)

View File

@@ -464,3 +464,163 @@ async def test_an_empty_list_of_pillars_consults_none_and_returns_what_was_said(
assert len(runtime.runs) == 3 # nobody was consulted a second time
for pillar in PILLARS:
assert f"## {pillar} (from your earlier consultation)" in again.message
# --- what the request does not say is a question for the customer ------------------------
def unsure(question: str | None = "Do you have a central identity provider?"):
def turn(task):
entry = {
"item": "Identity is managed centrally",
"status": "needs_customer_input",
"note": "The request does not say how staff identities are managed.",
}
if question is not None:
entry["question"] = question
first = Call("submit_advice", {"content": "Advice.", "checklist": [entry]})
# A second attempt, used only when the first is refused.
fixed = Call(
"submit_advice",
{
"content": "Advice.",
"checklist": [{**entry, "question": "Who manages staff identities?"}],
},
)
return [first, fixed]
return turn
async def test_an_item_the_request_cannot_settle_is_kept_with_its_question():
from app.tools.pillars import CHECKLIST_KEY, customer_questions
memory: dict = {}
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [unsure()] * 3})
await consult(
pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory), brief="b"
)
(entry,) = memory[CHECKLIST_KEY]["security"]
assert entry["status"] == "needs_customer_input"
assert customer_questions(memory)[0] == {
"pillar": "security",
"item": "Identity is managed centrally",
"question": "Do you have a central identity provider?",
}
assert len(customer_questions(memory)) == len(PILLARS)
async def test_an_item_for_the_customer_must_carry_the_question_to_ask():
from app.tools.pillars import CHECKLIST_KEY
memory: dict = {}
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [unsure(question=None)] * 3})
await consult(
pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory), brief="b"
)
refused, accepted = (reply for _, reply in runtime.runs[0].results)
assert refused.startswith("Not submitted: an item to confirm with the customer")
assert "Identity is managed centrally" in refused
(entry,) = memory[CHECKLIST_KEY]["security"]
assert entry["question"] == "Who manages staff identities?"
assert accepted == "Advice submitted."
def test_the_questions_are_reported_as_missing_information_not_as_faults():
from app.tools.pillars import (
CHECKLIST_KEY,
QUESTIONS_HEADING,
render_checklist_review,
)
working = {
CHECKLIST_KEY: {
"security": [
{
"item": "Scans run in the pipeline",
"status": "addressed",
"note": "DR-3",
},
{
"item": "Identity is managed centrally",
"status": "needs_customer_input",
"note": "Not said.",
"question": "Do you have a central identity provider?",
},
]
}
}
text = render_checklist_review(working)
assert "| Identity is managed centrally | To confirm with the customer |" in text
assert f"### {QUESTIONS_HEADING}" in text
assert "They are not faults in the design" in text
assert (
"| security | Identity is managed centrally | Do you have a central "
"identity provider? |"
) in text
# With nothing to ask there is no questions section.
only_addressed = {
CHECKLIST_KEY: {"security": working[CHECKLIST_KEY]["security"][:1]}
}
assert QUESTIONS_HEADING not in render_checklist_review(only_addressed)
async def test_an_execution_writes_its_questions_for_the_customer(tmp_path):
import json
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import sample_submission
from app.adk.runtime import NoDecider
from app.agent import ExecutionEngine
from app.nodes.router import Router, Thresholds
from app.recorder import InMemoryRecorder
from app.tools.pillars import CONSULT_TOOL
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
kind = DeliverableKind.DESIGN_RECOMMENDATIONS
advisor = DEPENDENCY_GRAPH[kind].specialist
roles = {
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
PILLAR_ADVISOR: ADVISOR,
**{
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
for spec in DEPENDENCY_GRAPH.values()
},
}
turns = {
spec.specialist: [[sample_submission(k)]]
for k, spec in DEPENDENCY_GRAPH.items()
}
turns[advisor] = [[Call(CONSULT_TOOL, {"brief": "b"}), sample_submission(kind)]]
turns[PILLAR_ADVISOR] = [unsure()] * 3
runtime = FakeAgentRuntime(turns)
engine = ExecutionEngine(
runtime=runtime,
roles=roles,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
orchestrator=Router(NoDecider(), None, Thresholds()),
tool_factory=make_tool_factory(runtime, roles, pillar_skills=PILLARS),
)
result = await engine.start("Ingest", customer_case="acme", pre_approved=True)
assert result.status == "completed"
out = next(tmp_path.rglob("execution.json")).parent
manifest = json.loads((out / "execution.json").read_text())
assert manifest["customer_questions"] == "customer-questions.md"
questions = (out / "customer-questions.md").read_text()
assert questions.startswith("# Questions for the customer")
assert "Do you have a central identity provider?" in questions
assert (
"### Questions for the customer"
in (out / "deliverables" / "design_recommendations.md").read_text()
)