Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-29.md
Jonathan Boniface a48468d477
Some checks failed
validation / verify (push) Failing after 11s
feat: what a request does not say is a question for the customer
A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.

Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:30:12 +01:00

2.5 KiB

Step 29 — What the request does not say is a question for the customer

Built on 2026-10-10, on the owner's direction. Not run with real models.

What prompted it

The owner labelled the judge's 32 questions (iteration 5 step 16): yes to all 24 checklist items. Both judges had answered no to most; they agreed with the labels on 28% and 16%. The owner's reading was deliberate: where a checklist item is unanswered because the customer's request gives no information about it, that is not the design failing a standard. It is missing information, and should come back as a question to put to the customer.

The checklist review had three outcomes: addressed, does not apply, not addressed. The third was the only place such an item could go, and it reads as a fault.

What was built

  • A fourth outcome, needs_customer_input ("To confirm with the customer"), in app/tools/pillars.py. A pillar advisor gives it to an item that could apply where its brief does not say what would settle it, and must give the question to ask; without the question its submission is refused. not_addressed is now only for an item the brief gives enough to settle, that applies, and that nothing is recommended for.
  • The questions are reported. Code writes a "Questions for the customer" table into the design recommendations, under the checklist review, saying they are not faults in the design. An Execution also writes customer-questions.md beside its other files, and execution.json names it. The packager carries the section into the guide.
  • The evaluation counts it as accounted for. The judged check asks whether the recommendations account for an item by recommending it, by explaining why it does not apply, or by recording it as something to confirm with the customer (checklist_question in app/tools/assessment.py, used by the evaluation and by calibration).
  • Glossary: Customer question.

Nothing is asked of the user during an Execution for these: they are reported. Whether an interactive Execution should put them to the user before the design is approved is not decided.

The judge's question set

The owner's labels were not kept. The two saved documents predate the checklist review, so they hold neither statuses nor questions, and the question's wording has changed. The set has to be rebuilt from design recommendations produced after this step (eval.calibrate_judge --build, after replacing the two documents), and labelled again. That needs a run.

Tests: tests/execution/test_pillars.py (4 more).