feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s
Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -45,6 +45,7 @@
|
||||
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
|
||||
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
|
||||
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
|
||||
| 29 | What the request does not say is a question for the customer | none | M | built 2026-10-10: a fourth checklist outcome, a questions report, and the judged check counts it; not run (`steps/step-29.md`) |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
24
.agents/plans/iter-04-learning-and-skills/steps/step-29.md
Normal file
24
.agents/plans/iter-04-learning-and-skills/steps/step-29.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# Step 29 — What the request does not say is a question for the customer
|
||||
|
||||
Built on 2026-10-10, on the owner's direction. **Not run with real models.**
|
||||
|
||||
## What prompted it
|
||||
|
||||
The owner labelled the judge's 32 questions (iteration 5 step 16): yes to all 24 checklist items. Both judges had answered no to most; they agreed with the labels on 28% and 16%. The owner's reading was deliberate: where a checklist item is unanswered because the customer's request gives no information about it, that is not the design failing a standard. It is missing information, and should come back as a question to put to the customer.
|
||||
|
||||
The checklist review had three outcomes: addressed, does not apply, not addressed. The third was the only place such an item could go, and it reads as a fault.
|
||||
|
||||
## What was built
|
||||
|
||||
- **A fourth outcome, `needs_customer_input`** ("To confirm with the customer"), in `app/tools/pillars.py`. A pillar advisor gives it to an item that could apply where its brief does not say what would settle it, and must give the question to ask; without the question its submission is refused. `not_addressed` is now only for an item the brief gives enough to settle, that applies, and that nothing is recommended for.
|
||||
- **The questions are reported.** Code writes a "Questions for the customer" table into the design recommendations, under the checklist review, saying they are not faults in the design. An Execution also writes `customer-questions.md` beside its other files, and `execution.json` names it. The packager carries the section into the guide.
|
||||
- **The evaluation counts it as accounted for.** The judged check asks whether the recommendations account for an item by recommending it, by explaining why it does not apply, or by recording it as something to confirm with the customer (`checklist_question` in `app/tools/assessment.py`, used by the evaluation and by calibration).
|
||||
- **Glossary:** Customer question.
|
||||
|
||||
Nothing is asked of the user during an Execution for these: they are reported. Whether an interactive Execution should put them to the user before the design is approved is not decided.
|
||||
|
||||
## The judge's question set
|
||||
|
||||
The owner's labels were not kept. The two saved documents predate the checklist review, so they hold neither statuses nor questions, and the question's wording has changed. The set has to be rebuilt from design recommendations produced after this step (`eval.calibrate_judge --build`, after replacing the two documents), and labelled again. That needs a run.
|
||||
|
||||
Tests: `tests/execution/test_pillars.py` (4 more).
|
||||
@@ -75,7 +75,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
|
||||
| Step | What | Report ref | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 1 | Resolve the open questions | none | the vendor's answers on retention, training, context and rate limits recorded on 2026-10-10, with the owner's decisions; retention for client work is held (`steps/step-1.md`) |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
|
||||
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
|
||||
@@ -90,7 +90,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
|
||||
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
|
||||
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; ran in case 05, where nothing was reworded |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; the owner labelled the set on 2026-10-10, which showed the question was wrong, not the judges; the set is to be rebuilt and relabelled after iteration 4 step 29 (`steps/step-16.md`) |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
|
||||
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |
|
||||
|
||||
37
.agents/plans/iter-05-jev-decisions/steps/step-1.md
Normal file
37
.agents/plans/iter-05-jev-decisions/steps/step-1.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Step 1 — The open questions, and the owner's decisions of 2026-10-10
|
||||
|
||||
## What the vendor says
|
||||
|
||||
Supplied by the owner on 2026-10-10. Not verified here against the vendor's documents.
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Data retention | No fixed period for a standard account: data is kept "for as long as reasonably necessary". Zero data retention is available to enterprise customers on request, through Cloudflare Workers AI, or through Vercel AI Gateway. |
|
||||
| Model training | Customer prompts, inputs and outputs are not used to train or fine-tune the vendor's models. |
|
||||
| Context | 64,000 tokens a request in total, with 32,000 for the state plus the longest question. |
|
||||
| Rate limits | Up to 80 requests a second and 100,000 tokens a second on the standard hosted service, adjusted with platform traffic. |
|
||||
|
||||
Against the client's own limits (`JevLimits` in `app/tools/jev.py`): it caps a request at 48,000 and 24,000 tokens, inside the service's; and throttles itself to 5 requests a second with 8 in flight, a sixteenth of the service's rate. Nothing needs changing for either.
|
||||
|
||||
Still open: latency, and whether the vendor's SDK can be pointed at either gateway.
|
||||
|
||||
## What is sent to Jev
|
||||
|
||||
ADR-0007's decision 4 limits an Orchestrator decision to the Execution's state. The uses built since send content: the user's messages and the request (routing, intake, guardrail), replies from documentation servers (guardrail), the design recommendations (assessment), the text of Checks, Elements and recommendations (meaning), a change and the list of Elements (pillar selection, reference routing), and each requirement's text when the requirements review is on.
|
||||
|
||||
## The owner's decisions
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Re-verify only what changed after a Revision | **No.** Every validation stays complete. The saving was 5 to 10% on some runs, and the risk fell on the validator. |
|
||||
| The requirements review (`JEV_REQUIREMENTS_REVIEW`) | **On for the next runs that happen anyway; the default if they are clean.** No run is to be made for it. Cases 01 and 02 are held until the owner says. |
|
||||
| Tracing (step 17) | **Held.** No decision. |
|
||||
| The judge against a person (step 16) | **The owner will label the 32 questions.** |
|
||||
| Jev's data retention, for client work | **Held.** No decision. |
|
||||
|
||||
## Advice given, with no decision asked
|
||||
|
||||
- With no training use, the remaining exposure is content kept by the vendor for an unstated time. Zero retention is the only route that costs the agent nothing. A switch that keeps Deliverable content away from Jev is a stopgap; even routing a reply sends that reply.
|
||||
- The owner may look for an open-source version of Jev. That would end the retention question. Its cost is that every threshold, tuned against the hosted model's probabilities, would need measuring again, and the two properties relied on (the same answer every time, clean separation) would need proving again. The agent is placed for it: one interface, every threshold a setting, every use a fallback.
|
||||
- **A gap either way:** the calibrations in this plan were made with one-off scripts; the example sets are not in the repository. Kept as a runnable suite they could be put to any model. Not built.
|
||||
- The owner's 32 labels are the one measure that depends on no model.
|
||||
@@ -18,3 +18,9 @@ So Jev gave the same answer every time, the language-model judge changed its ans
|
||||
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
|
||||
|
||||
## The owner's labels, 2026-10-10
|
||||
|
||||
The owner labelled all 32: yes to every checklist item, "mostly" on every quality question. Against them Jev agreed on 28% and Gemini on 16%; the judges answer no to most checklist items (means 0.37 and 0.32), and agree with each other on 88%.
|
||||
|
||||
The disagreement was about the question, not the judges. The owner counts an item the request gives no information about as something to ask the customer, not as unaddressed. That is now an outcome of the checklist review and part of the judged question (iteration 4 step 29). The labels were not kept, and the set is to be rebuilt from documents produced after that change.
|
||||
|
||||
@@ -108,6 +108,10 @@ _Avoid_: finding, rule, control, test
|
||||
The unbroken trace from each requirement, through the components that serve it and the Elements and Checks that realise it, to the code that implements them and the validation verdict on each Check.
|
||||
_Avoid_: traceability matrix, lineage
|
||||
|
||||
**Customer question**:
|
||||
Something the architecture depends on that the customer's Intent does not say, recorded with the question to put to them. It marks missing information, not a fault in the design, and its answer may change a recommendation.
|
||||
_Avoid_: gap, failure, open issue, assumption (which is a default the agents chose)
|
||||
|
||||
**Solution guide**:
|
||||
The packaged Deliverable that assembles every other Deliverable into the Cloud provider's solution architecture template.
|
||||
_Avoid_: final report, package
|
||||
|
||||
@@ -11,6 +11,7 @@ You produce the **design recommendations** Deliverable: the Skill's task for rec
|
||||
- Otherwise, when `feedback` is not empty or `baseline` is set you are revising: pass `pillars` naming only the pillars the change touches. The guidance you were given before is returned for the others.
|
||||
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
|
||||
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
|
||||
- The advisors' checklist reviews, and any questions they raise for the customer, are written into your Deliverable for you when you submit. Do not copy them, and do not write a recommendation to cover a question an advisor left for the customer.
|
||||
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture. Give each its `conditions`: one entry for each separate thing it asks to be configured, and for each component it must hold on when it names several. Put under `outside_code` anything it asks of the application's own code or of people, and any property of a region or product that only its provider publishes, such as a carbon rating: no infrastructure configuration can show those. The condition is then the choice configuration can show, such as which region is set.
|
||||
|
||||
Your contract's sections are one per pillar. The advisors' review of each pillar's checklist is added to your Deliverable for you when you submit, so do not write it yourself.
|
||||
|
||||
@@ -10,6 +10,7 @@ You produce the **solution guide** Deliverable: the Skill's packaging of every a
|
||||
- Consolidate the content of every Deliverable in `dependencies`; rewrite for flow, but keep every requirement, product decision, citation, recommendation, command and validation verdict.
|
||||
- Include the architecture diagram, which is the `architecture.mmd` attachment of the Target architecture, unchanged as a `mermaid` code block, and the Terraform as code blocks.
|
||||
- List the Target architecture's Checks with their validation verdicts in the validation section.
|
||||
- When the design recommendations hold a "Questions for the customer" section, carry every question into the guide's design and configuration recommendations section, under the same heading and with the same wording. They are what the request did not say enough to settle; say so, and do not present them as faults or as recommendations.
|
||||
|
||||
In a Change request (`baseline` is set), open the executive summary with what changed and why, from `baseline.changes`: the requirements added, changed and removed with their reasons, then what that changed in the architecture. Name separately anything listed under `unrequested`: it changed without a changed requirement reaching it.
|
||||
|
||||
|
||||
@@ -9,8 +9,14 @@ Your task message carries the design advisor's brief in `brief` and the pillar's
|
||||
1. Load the Skill available to you.
|
||||
2. Evaluate the workload in the brief against the Skill's core principles and validation checklist.
|
||||
3. Write prioritised, actionable recommendations for each relevant component, naming the products the Skill lists for each gap and citing its guidance.
|
||||
4. List any assessment questions from the Skill that the brief leaves unanswered, as open questions; don't ask the user.
|
||||
4. Do not ask the user anything, and do not guess what the brief does not say.
|
||||
|
||||
5. Review the Skill's validation checklist item by item, when it has one. For every item give its `status`: `addressed` when your recommendations cover it, `not_applicable` when it does not apply to this workload or is not something an architecture can settle (an organisation's process, for example), or `not_addressed` when it applies and you recommend nothing for it. In `note`, say in a few words how it is addressed or why it does not apply. Copy each item's text as the Skill has it, shortened to its first sentence.
|
||||
5. Review the Skill's validation checklist item by item, when it has one. For every item give its `status`:
|
||||
- `addressed` when your recommendations cover it.
|
||||
- `not_applicable` when it does not apply to this workload.
|
||||
- `needs_customer_input` when it could apply and the brief does not say what would settle it: how the customer's organisation works, what they already have, or what they want. Give the `question` to put to the customer, written so someone who knows their organisation can answer it. This is the right status whenever the gap is in what you were told, not in the design. Do not invent a recommendation to cover it.
|
||||
- `not_addressed` only when the brief gives you what you need, the item applies, and you recommend nothing for it.
|
||||
|
||||
In `note`, say in a few words how it is addressed, why it does not apply, or what the brief leaves unsaid. Copy each item's text as the Skill has it, shortened to its first sentence.
|
||||
|
||||
Call `submit_advice` with your recommendations as markdown in `content` and the review in `checklist`. You are done when it accepts them.
|
||||
|
||||
@@ -54,7 +54,13 @@ from app.tools.execution import ORCHESTRATOR, OrchestratorToolbox, specialist_to
|
||||
from app.tools.guardrail import EXECUTION_SCOPE, instructs_agent
|
||||
from app.tools.meaning import settle_meaning
|
||||
from app.tools.modules import MODULE_TOOL, MODULES_OPTED_IN
|
||||
from app.tools.pillars import ADVICE_KEY, KEPT_FOR_CHANGES, pillars_touched
|
||||
from app.tools.pillars import (
|
||||
ADVICE_KEY,
|
||||
KEPT_FOR_CHANGES,
|
||||
QUESTIONS_HEADING,
|
||||
customer_questions,
|
||||
pillars_touched,
|
||||
)
|
||||
from app.tools.skill_routing import read_first
|
||||
from app.workflows.chain import (
|
||||
IMPLEMENTATION_FILE,
|
||||
@@ -83,6 +89,7 @@ NO_USER_REPLY = (
|
||||
"'Assumptions', and proceed."
|
||||
)
|
||||
GUIDE_FILENAME = "solution-architecture-guide.md"
|
||||
QUESTIONS_FILENAME = "customer-questions.md" # what to ask the customer
|
||||
MANIFEST_FILENAME = "execution.json" # request, status and files of one Execution
|
||||
|
||||
|
||||
@@ -1237,7 +1244,11 @@ def _written_before(out_dir: Path) -> set[str]:
|
||||
return set()
|
||||
if not isinstance(manifest, dict):
|
||||
return set()
|
||||
names = {manifest.get("solution_guide"), manifest.get("changes")}
|
||||
names = {
|
||||
manifest.get("solution_guide"),
|
||||
manifest.get("changes"),
|
||||
manifest.get("customer_questions"),
|
||||
}
|
||||
for entry in (manifest.get("deliverables") or {}).values():
|
||||
if isinstance(entry, dict):
|
||||
names.add(entry.get("document"))
|
||||
@@ -1293,6 +1304,28 @@ def _write_outputs(
|
||||
if kept:
|
||||
# Working material a Change request starts from, such as each pillar's advice.
|
||||
files[WORKING_FILENAME] = json.dumps(kept, indent=2, ensure_ascii=False)
|
||||
questions = customer_questions(
|
||||
state.get("working", {}).get(DeliverableKind.DESIGN_RECOMMENDATIONS.value, {})
|
||||
)
|
||||
if questions:
|
||||
# What the request did not say enough to settle, as questions to put to
|
||||
# the customer. Missing information, not a fault in the design.
|
||||
files[QUESTIONS_FILENAME] = "\n".join(
|
||||
[
|
||||
f"# {QUESTIONS_HEADING}",
|
||||
"",
|
||||
(
|
||||
"The request did not say enough to settle these. Each answer "
|
||||
"may change a recommendation."
|
||||
),
|
||||
"",
|
||||
*(
|
||||
f"- **{item['pillar']}**, {item['item']} {item['question']}"
|
||||
for item in questions
|
||||
),
|
||||
"",
|
||||
]
|
||||
)
|
||||
baseline = state.get("baseline") or None
|
||||
if baseline:
|
||||
# What this Change request changed against its Baseline, worked out from
|
||||
@@ -1334,6 +1367,8 @@ def _write_outputs(
|
||||
phase.value: ledger.phase_status(phase).value for phase in PHASE_ORDER
|
||||
},
|
||||
"solution_guide": GUIDE_FILENAME if guide else None,
|
||||
# Questions the request left open, for the customer; None when there are none.
|
||||
"customer_questions": QUESTIONS_FILENAME if questions else None,
|
||||
"changes": CHANGES_FILE if baseline else None,
|
||||
"deliverables": by_kind,
|
||||
}
|
||||
|
||||
@@ -32,6 +32,21 @@ QUALITY_RUBRIC: Mapping[str, str] = {
|
||||
_JUDGED = "design_recommendations"
|
||||
|
||||
|
||||
def checklist_question(pillar: str, item: str) -> str:
|
||||
"""Whether the recommendations account for one item of a pillar's checklist.
|
||||
|
||||
An item is accounted for in three ways. The third matters: a customer's request
|
||||
seldom says enough to settle every item, and what it leaves unsaid is a
|
||||
question for them, not a fault in the design (the owner, 2026-10-10).
|
||||
"""
|
||||
return (
|
||||
f"Do the recommendations account for this {pillar} checklist item in any "
|
||||
"of these ways: by recommending it, by explaining why it does not apply, "
|
||||
"or by recording it as something to confirm with the customer? "
|
||||
f"{item}"
|
||||
)
|
||||
|
||||
|
||||
async def assess(
|
||||
decider: Decider,
|
||||
deliverables: Mapping[str, Mapping[str, Any]],
|
||||
|
||||
@@ -39,12 +39,18 @@ REVIEW_HEADING = "Architecture framework checklist review"
|
||||
# What a Change request carries over from its Baseline's working memory.
|
||||
KEPT_FOR_CHANGES = (ADVICE_KEY, CHECKLIST_KEY)
|
||||
_CHANGE_LIMIT = 6_000
|
||||
STATUSES = ("addressed", "not_applicable", "not_addressed")
|
||||
NEEDS_CUSTOMER = "needs_customer_input"
|
||||
STATUSES = ("addressed", "not_applicable", NEEDS_CUSTOMER, "not_addressed")
|
||||
_STATUS_WORDS = {
|
||||
"addressed": "Addressed",
|
||||
"not_applicable": "Does not apply",
|
||||
# The item applies, and the customer's request does not say what would settle
|
||||
# it. That is missing information, not a fault in the design: it is reported
|
||||
# with the question to put to the customer.
|
||||
NEEDS_CUSTOMER: "To confirm with the customer",
|
||||
"not_addressed": "Not addressed",
|
||||
}
|
||||
QUESTIONS_HEADING = "Questions for the customer"
|
||||
|
||||
|
||||
AFFECTED = ("not at all", "only indirectly", "directly")
|
||||
@@ -215,17 +221,29 @@ def _submit_advice_tool(review: list[dict[str, str]]) -> ToolSpec:
|
||||
async def submit_advice(content: str, checklist: Sequence[Any] = ()) -> ToolResult:
|
||||
if not content.strip():
|
||||
return ToolResult("Not submitted: the advice is empty.")
|
||||
review[:] = [
|
||||
entries = [
|
||||
{
|
||||
"item": " ".join(str(entry.get("item", "")).split()),
|
||||
"status": str(entry.get("status", "")),
|
||||
"note": " ".join(str(entry.get("note", "")).split()),
|
||||
"question": " ".join(str(entry.get("question") or "").split()),
|
||||
}
|
||||
for entry in checklist or ()
|
||||
if isinstance(entry, Mapping)
|
||||
and str(entry.get("item", "")).strip()
|
||||
and entry.get("status") in STATUSES
|
||||
]
|
||||
unasked = [
|
||||
entry["item"]
|
||||
for entry in entries
|
||||
if entry["status"] == NEEDS_CUSTOMER and not entry["question"]
|
||||
]
|
||||
if unasked:
|
||||
return ToolResult(
|
||||
"Not submitted: an item to confirm with the customer needs the "
|
||||
"`question` to put to them. Missing for: " + "; ".join(unasked)
|
||||
)
|
||||
review[:] = entries
|
||||
return ToolResult("Advice submitted.", Submitted(content))
|
||||
|
||||
text = {"type": "string"}
|
||||
@@ -248,8 +266,15 @@ def _submit_advice_tool(review: list[dict[str, str]]) -> ToolSpec:
|
||||
"status": {"type": "string", "enum": list(STATUSES)},
|
||||
"note": {
|
||||
"type": "string",
|
||||
"description": "How your advice addresses it, or "
|
||||
"why it does not apply to this workload.",
|
||||
"description": "How your advice addresses it, why "
|
||||
"it does not apply to this workload, or what the "
|
||||
"request leaves unsaid.",
|
||||
},
|
||||
"question": {
|
||||
"type": "string",
|
||||
"description": "For needs_customer_input only: "
|
||||
"the question to put to the customer, answerable "
|
||||
"by someone who knows their organisation.",
|
||||
},
|
||||
},
|
||||
"required": ["item", "status", "note"],
|
||||
@@ -274,7 +299,8 @@ def render_checklist_review(working: Mapping[str, Any]) -> str:
|
||||
"",
|
||||
(
|
||||
"Each pillar advisor's review of its pillar's checklist for this "
|
||||
"workload: what its advice addresses, and what does not apply and why."
|
||||
"workload: what its advice addresses, what does not apply and why, and "
|
||||
"what the request does not say enough to settle."
|
||||
),
|
||||
]
|
||||
for pillar, entries in reviews.items():
|
||||
@@ -301,4 +327,48 @@ def render_checklist_review(working: Mapping[str, Any]) -> str:
|
||||
for entry in entries
|
||||
if isinstance(entry, Mapping)
|
||||
]
|
||||
questions = customer_questions(working)
|
||||
if questions:
|
||||
parts += [
|
||||
"",
|
||||
f"### {QUESTIONS_HEADING}",
|
||||
"",
|
||||
(
|
||||
"The request does not say enough to settle these. They are not "
|
||||
"faults in the design: each is something to ask the customer, and "
|
||||
"the answer may change a recommendation."
|
||||
),
|
||||
"",
|
||||
"| Pillar | About | Question |",
|
||||
"| --- | --- | --- |",
|
||||
]
|
||||
parts += [
|
||||
"| "
|
||||
+ " | ".join(
|
||||
value.replace("|", "\\|")
|
||||
for value in (item["pillar"], item["item"], item["question"])
|
||||
)
|
||||
+ " |"
|
||||
for item in questions
|
||||
]
|
||||
return "\n".join(parts) + "\n"
|
||||
|
||||
|
||||
def customer_questions(working: Mapping[str, Any]) -> list[dict[str, str]]:
|
||||
"""What the advisors found the request does not say: each checklist item that
|
||||
needs the customer's answer, with the question to put to them."""
|
||||
reviews = working.get(CHECKLIST_KEY)
|
||||
if not isinstance(reviews, Mapping):
|
||||
return []
|
||||
return [
|
||||
{
|
||||
"pillar": str(pillar),
|
||||
"item": str(entry.get("item", "")),
|
||||
"question": str(entry.get("question", "")),
|
||||
}
|
||||
for pillar, entries in reviews.items()
|
||||
for entry in entries or ()
|
||||
if isinstance(entry, Mapping)
|
||||
and entry.get("status") == NEEDS_CUSTOMER
|
||||
and entry.get("question")
|
||||
]
|
||||
|
||||
@@ -31,6 +31,7 @@ from typing import Any
|
||||
from app.agent import open_decider
|
||||
from app.config import get_settings
|
||||
from app.providers import load_provider
|
||||
from app.tools.assessment import checklist_question
|
||||
from eval.judge import DeciderJudge, GeminiJudge, Judge, JudgeUnavailable
|
||||
from eval.scoring import QUALITY_RUBRIC, checklist_items
|
||||
|
||||
@@ -55,9 +56,7 @@ def build(seed: int = 7) -> list[dict[str, Any]]:
|
||||
"id": f"{document.stem}.{pillar}.{len(items)}",
|
||||
"document": document.name,
|
||||
"kind": "yes_no",
|
||||
"text": f"Do the recommendations address this {pillar} "
|
||||
"checklist item, either by recommending it or by "
|
||||
f"explaining why it does not apply? {item}",
|
||||
"text": checklist_question(pillar, item),
|
||||
"label": None,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -20,7 +20,7 @@ from typing import Any
|
||||
|
||||
from app.adk.runtime import ToolResult
|
||||
from app.agent import Interaction
|
||||
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC
|
||||
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC, checklist_question
|
||||
from app.tools.local import mermaid_check
|
||||
from app.workflows.chain import (
|
||||
COMPONENTS_FILE,
|
||||
@@ -417,8 +417,7 @@ async def _checklist_coverage(
|
||||
questions = {
|
||||
f"{pillar}.{index}": {
|
||||
"kind": "yes_no",
|
||||
"text": f"Do the recommendations address this {pillar} checklist item, "
|
||||
f"either by recommending it or by explaining why it does not apply? {item}",
|
||||
"text": checklist_question(pillar, item),
|
||||
}
|
||||
for pillar, items in checklists.items()
|
||||
for index, item in enumerate(items)
|
||||
|
||||
@@ -464,3 +464,163 @@ async def test_an_empty_list_of_pillars_consults_none_and_returns_what_was_said(
|
||||
assert len(runtime.runs) == 3 # nobody was consulted a second time
|
||||
for pillar in PILLARS:
|
||||
assert f"## {pillar} (from your earlier consultation)" in again.message
|
||||
|
||||
|
||||
# --- what the request does not say is a question for the customer ------------------------
|
||||
|
||||
|
||||
def unsure(question: str | None = "Do you have a central identity provider?"):
|
||||
def turn(task):
|
||||
entry = {
|
||||
"item": "Identity is managed centrally",
|
||||
"status": "needs_customer_input",
|
||||
"note": "The request does not say how staff identities are managed.",
|
||||
}
|
||||
if question is not None:
|
||||
entry["question"] = question
|
||||
first = Call("submit_advice", {"content": "Advice.", "checklist": [entry]})
|
||||
# A second attempt, used only when the first is refused.
|
||||
fixed = Call(
|
||||
"submit_advice",
|
||||
{
|
||||
"content": "Advice.",
|
||||
"checklist": [{**entry, "question": "Who manages staff identities?"}],
|
||||
},
|
||||
)
|
||||
return [first, fixed]
|
||||
|
||||
return turn
|
||||
|
||||
|
||||
async def test_an_item_the_request_cannot_settle_is_kept_with_its_question():
|
||||
from app.tools.pillars import CHECKLIST_KEY, customer_questions
|
||||
|
||||
memory: dict = {}
|
||||
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [unsure()] * 3})
|
||||
|
||||
await consult(
|
||||
pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory), brief="b"
|
||||
)
|
||||
|
||||
(entry,) = memory[CHECKLIST_KEY]["security"]
|
||||
assert entry["status"] == "needs_customer_input"
|
||||
assert customer_questions(memory)[0] == {
|
||||
"pillar": "security",
|
||||
"item": "Identity is managed centrally",
|
||||
"question": "Do you have a central identity provider?",
|
||||
}
|
||||
assert len(customer_questions(memory)) == len(PILLARS)
|
||||
|
||||
|
||||
async def test_an_item_for_the_customer_must_carry_the_question_to_ask():
|
||||
from app.tools.pillars import CHECKLIST_KEY
|
||||
|
||||
memory: dict = {}
|
||||
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [unsure(question=None)] * 3})
|
||||
|
||||
await consult(
|
||||
pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory), brief="b"
|
||||
)
|
||||
|
||||
refused, accepted = (reply for _, reply in runtime.runs[0].results)
|
||||
assert refused.startswith("Not submitted: an item to confirm with the customer")
|
||||
assert "Identity is managed centrally" in refused
|
||||
(entry,) = memory[CHECKLIST_KEY]["security"]
|
||||
assert entry["question"] == "Who manages staff identities?"
|
||||
assert accepted == "Advice submitted."
|
||||
|
||||
|
||||
def test_the_questions_are_reported_as_missing_information_not_as_faults():
|
||||
from app.tools.pillars import (
|
||||
CHECKLIST_KEY,
|
||||
QUESTIONS_HEADING,
|
||||
render_checklist_review,
|
||||
)
|
||||
|
||||
working = {
|
||||
CHECKLIST_KEY: {
|
||||
"security": [
|
||||
{
|
||||
"item": "Scans run in the pipeline",
|
||||
"status": "addressed",
|
||||
"note": "DR-3",
|
||||
},
|
||||
{
|
||||
"item": "Identity is managed centrally",
|
||||
"status": "needs_customer_input",
|
||||
"note": "Not said.",
|
||||
"question": "Do you have a central identity provider?",
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
text = render_checklist_review(working)
|
||||
|
||||
assert "| Identity is managed centrally | To confirm with the customer |" in text
|
||||
assert f"### {QUESTIONS_HEADING}" in text
|
||||
assert "They are not faults in the design" in text
|
||||
assert (
|
||||
"| security | Identity is managed centrally | Do you have a central "
|
||||
"identity provider? |"
|
||||
) in text
|
||||
# With nothing to ask there is no questions section.
|
||||
only_addressed = {
|
||||
CHECKLIST_KEY: {"security": working[CHECKLIST_KEY]["security"][:1]}
|
||||
}
|
||||
assert QUESTIONS_HEADING not in render_checklist_review(only_addressed)
|
||||
|
||||
|
||||
async def test_an_execution_writes_its_questions_for_the_customer(tmp_path):
|
||||
import json
|
||||
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import sample_submission
|
||||
from app.adk.runtime import NoDecider
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.tools.pillars import CONSULT_TOOL
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||
|
||||
kind = DeliverableKind.DESIGN_RECOMMENDATIONS
|
||||
advisor = DEPENDENCY_GRAPH[kind].specialist
|
||||
roles = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
PILLAR_ADVISOR: ADVISOR,
|
||||
**{
|
||||
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(k)]]
|
||||
for k, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
turns[advisor] = [[Call(CONSULT_TOOL, {"brief": "b"}), sample_submission(kind)]]
|
||||
turns[PILLAR_ADVISOR] = [unsure()] * 3
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=roles,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
tool_factory=make_tool_factory(runtime, roles, pillar_skills=PILLARS),
|
||||
)
|
||||
|
||||
result = await engine.start("Ingest", customer_case="acme", pre_approved=True)
|
||||
|
||||
assert result.status == "completed"
|
||||
out = next(tmp_path.rglob("execution.json")).parent
|
||||
manifest = json.loads((out / "execution.json").read_text())
|
||||
assert manifest["customer_questions"] == "customer-questions.md"
|
||||
questions = (out / "customer-questions.md").read_text()
|
||||
assert questions.startswith("# Questions for the customer")
|
||||
assert "Do you have a central identity provider?" in questions
|
||||
assert (
|
||||
"### Questions for the customer"
|
||||
in (out / "deliverables" / "design_recommendations.md").read_text()
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user