feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s

A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.

Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 19:30:12 +01:00
parent f9cc34817b
commit a48468d477
15 changed files with 375 additions and 17 deletions

View File

@@ -45,6 +45,7 @@
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
| 29 | What the request does not say is a question for the customer | none | M | built 2026-10-10: a fourth checklist outcome, a questions report, and the judged check counts it; not run (`steps/step-29.md`) |
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.

View File

@@ -0,0 +1,24 @@
# Step 29 — What the request does not say is a question for the customer
Built on 2026-10-10, on the owner's direction. **Not run with real models.**
## What prompted it
The owner labelled the judge's 32 questions (iteration 5 step 16): yes to all 24 checklist items. Both judges had answered no to most; they agreed with the labels on 28% and 16%. The owner's reading was deliberate: where a checklist item is unanswered because the customer's request gives no information about it, that is not the design failing a standard. It is missing information, and should come back as a question to put to the customer.
The checklist review had three outcomes: addressed, does not apply, not addressed. The third was the only place such an item could go, and it reads as a fault.
## What was built
- **A fourth outcome, `needs_customer_input`** ("To confirm with the customer"), in `app/tools/pillars.py`. A pillar advisor gives it to an item that could apply where its brief does not say what would settle it, and must give the question to ask; without the question its submission is refused. `not_addressed` is now only for an item the brief gives enough to settle, that applies, and that nothing is recommended for.
- **The questions are reported.** Code writes a "Questions for the customer" table into the design recommendations, under the checklist review, saying they are not faults in the design. An Execution also writes `customer-questions.md` beside its other files, and `execution.json` names it. The packager carries the section into the guide.
- **The evaluation counts it as accounted for.** The judged check asks whether the recommendations account for an item by recommending it, by explaining why it does not apply, or by recording it as something to confirm with the customer (`checklist_question` in `app/tools/assessment.py`, used by the evaluation and by calibration).
- **Glossary:** Customer question.
Nothing is asked of the user during an Execution for these: they are reported. Whether an interactive Execution should put them to the user before the design is approved is not decided.
## The judge's question set
The owner's labels were not kept. The two saved documents predate the checklist review, so they hold neither statuses nor questions, and the question's wording has changed. The set has to be rebuilt from design recommendations produced after this step (`eval.calibrate_judge --build`, after replacing the two documents), and labelled again. That needs a run.
Tests: `tests/execution/test_pillars.py` (4 more).

View File

@@ -75,7 +75,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| Step | What | Report ref | Status |
|---|---|---|---|
| 1 | Resolve the open questions | none | open, owner's input |
| 1 | Resolve the open questions | none | the vendor's answers on retention, training, context and rate limits recorded on 2026-10-10, with the owner's decisions; retention for client work is held (`steps/step-1.md`) |
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
| 3 | Pillars a Revision touches | J3 | built, as a graded question; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
@@ -90,7 +90,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
| 13 | Cloud-read call guard | J7 | planned; cloud reads are off in every case, so it could not be exercised |
| 14 | Input guardrail | J11 | built and on: an MCP reply that instructs the agent is withheld, the user's text is flagged; calibrated live; ran in cases 05 and 06 on 2026-10-09 |
| 15 | Changes of meaning in a Change request | none | built: reworded items are listed apart in changes.json; calibrated live; ran in case 05, where nothing was reworded |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; the owner labelled the set on 2026-10-10, which showed the question was wrong, not the judges; the set is to be rebuilt and relabelled after iteration 4 step 29 (`steps/step-16.md`) |
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
| 18 | Readings as their own graph step | none | planned; it splits the Orchestrator turn that case 04 only just passes, to save a repeated Jev call after a crash |
| 19 | Online evaluation of live Executions | none | built and on: each Execution's design recommendations are judged when it ends and the scores kept in its record; feedback on a trace waits on step 17 (`steps/step-19.md`) |

View File

@@ -0,0 +1,37 @@
# Step 1 — The open questions, and the owner's decisions of 2026-10-10
## What the vendor says
Supplied by the owner on 2026-10-10. Not verified here against the vendor's documents.
| Question | Answer |
|---|---|
| Data retention | No fixed period for a standard account: data is kept "for as long as reasonably necessary". Zero data retention is available to enterprise customers on request, through Cloudflare Workers AI, or through Vercel AI Gateway. |
| Model training | Customer prompts, inputs and outputs are not used to train or fine-tune the vendor's models. |
| Context | 64,000 tokens a request in total, with 32,000 for the state plus the longest question. |
| Rate limits | Up to 80 requests a second and 100,000 tokens a second on the standard hosted service, adjusted with platform traffic. |
Against the client's own limits (`JevLimits` in `app/tools/jev.py`): it caps a request at 48,000 and 24,000 tokens, inside the service's; and throttles itself to 5 requests a second with 8 in flight, a sixteenth of the service's rate. Nothing needs changing for either.
Still open: latency, and whether the vendor's SDK can be pointed at either gateway.
## What is sent to Jev
ADR-0007's decision 4 limits an Orchestrator decision to the Execution's state. The uses built since send content: the user's messages and the request (routing, intake, guardrail), replies from documentation servers (guardrail), the design recommendations (assessment), the text of Checks, Elements and recommendations (meaning), a change and the list of Elements (pillar selection, reference routing), and each requirement's text when the requirements review is on.
## The owner's decisions
| Question | Decision |
|---|---|
| Re-verify only what changed after a Revision | **No.** Every validation stays complete. The saving was 5 to 10% on some runs, and the risk fell on the validator. |
| The requirements review (`JEV_REQUIREMENTS_REVIEW`) | **On for the next runs that happen anyway; the default if they are clean.** No run is to be made for it. Cases 01 and 02 are held until the owner says. |
| Tracing (step 17) | **Held.** No decision. |
| The judge against a person (step 16) | **The owner will label the 32 questions.** |
| Jev's data retention, for client work | **Held.** No decision. |
## Advice given, with no decision asked
- With no training use, the remaining exposure is content kept by the vendor for an unstated time. Zero retention is the only route that costs the agent nothing. A switch that keeps Deliverable content away from Jev is a stopgap; even routing a reply sends that reply.
- The owner may look for an open-source version of Jev. That would end the retention question. Its cost is that every threshold, tuned against the hosted model's probabilities, would need measuring again, and the two properties relied on (the same answer every time, clean separation) would need proving again. The agent is placed for it: one interface, every threshold a setting, every use a fallback.
- **A gap either way:** the calibrations in this plan were made with one-off scripts; the example sets are not in the repository. Kept as a runnable suite they could be put to any model. Not built.
- The owner's 32 labels are the one measure that depends on no model.

View File

@@ -18,3 +18,9 @@ So Jev gave the same answer every time, the language-model judge changed its ans
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
## The owner's labels, 2026-10-10
The owner labelled all 32: yes to every checklist item, "mostly" on every quality question. Against them Jev agreed on 28% and Gemini on 16%; the judges answer no to most checklist items (means 0.37 and 0.32), and agree with each other on 88%.
The disagreement was about the question, not the judges. The owner counts an item the request gives no information about as something to ask the customer, not as unaddressed. That is now an outcome of the checklist review and part of the judged question (iteration 4 step 29). The labels were not kept, and the set is to be rebuilt from documents produced after that change.