feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s
Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
37
.agents/plans/iter-05-jev-decisions/steps/step-1.md
Normal file
37
.agents/plans/iter-05-jev-decisions/steps/step-1.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Step 1 — The open questions, and the owner's decisions of 2026-10-10
|
||||
|
||||
## What the vendor says
|
||||
|
||||
Supplied by the owner on 2026-10-10. Not verified here against the vendor's documents.
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Data retention | No fixed period for a standard account: data is kept "for as long as reasonably necessary". Zero data retention is available to enterprise customers on request, through Cloudflare Workers AI, or through Vercel AI Gateway. |
|
||||
| Model training | Customer prompts, inputs and outputs are not used to train or fine-tune the vendor's models. |
|
||||
| Context | 64,000 tokens a request in total, with 32,000 for the state plus the longest question. |
|
||||
| Rate limits | Up to 80 requests a second and 100,000 tokens a second on the standard hosted service, adjusted with platform traffic. |
|
||||
|
||||
Against the client's own limits (`JevLimits` in `app/tools/jev.py`): it caps a request at 48,000 and 24,000 tokens, inside the service's; and throttles itself to 5 requests a second with 8 in flight, a sixteenth of the service's rate. Nothing needs changing for either.
|
||||
|
||||
Still open: latency, and whether the vendor's SDK can be pointed at either gateway.
|
||||
|
||||
## What is sent to Jev
|
||||
|
||||
ADR-0007's decision 4 limits an Orchestrator decision to the Execution's state. The uses built since send content: the user's messages and the request (routing, intake, guardrail), replies from documentation servers (guardrail), the design recommendations (assessment), the text of Checks, Elements and recommendations (meaning), a change and the list of Elements (pillar selection, reference routing), and each requirement's text when the requirements review is on.
|
||||
|
||||
## The owner's decisions
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Re-verify only what changed after a Revision | **No.** Every validation stays complete. The saving was 5 to 10% on some runs, and the risk fell on the validator. |
|
||||
| The requirements review (`JEV_REQUIREMENTS_REVIEW`) | **On for the next runs that happen anyway; the default if they are clean.** No run is to be made for it. Cases 01 and 02 are held until the owner says. |
|
||||
| Tracing (step 17) | **Held.** No decision. |
|
||||
| The judge against a person (step 16) | **The owner will label the 32 questions.** |
|
||||
| Jev's data retention, for client work | **Held.** No decision. |
|
||||
|
||||
## Advice given, with no decision asked
|
||||
|
||||
- With no training use, the remaining exposure is content kept by the vendor for an unstated time. Zero retention is the only route that costs the agent nothing. A switch that keeps Deliverable content away from Jev is a stopgap; even routing a reply sends that reply.
|
||||
- The owner may look for an open-source version of Jev. That would end the retention question. Its cost is that every threshold, tuned against the hosted model's probabilities, would need measuring again, and the two properties relied on (the same answer every time, clean separation) would need proving again. The agent is placed for it: one interface, every threshold a setting, every use a fallback.
|
||||
- **A gap either way:** the calibrations in this plan were made with one-off scripts; the example sets are not in the repository. Kept as a runnable suite they could be put to any model. Not built.
|
||||
- The owner's 32 labels are the one measure that depends on no model.
|
||||
Reference in New Issue
Block a user