feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005). A new target-architect Specialist produces one graph of Elements, Relationships and Checks; the diagram is rendered from it, deployment guidance must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations carry ids in data files, and a submission with a gap in the Requirements chain is refused. Also in this commit: - Customer cases group Executions, with an intent history per case - lesson candidates are captured when an Execution ends - evaluation gains chain, verdict, judged and interactive checks, and per-case thresholds - Terraform providers are cached between checks - the requirements analyst proposes defaults instead of asking - the legacy-baseline case and the checks that only read it are removed - ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed - plans for iterations 4 to 7 under .agents/plans/ Unproven with a real model: no evaluation run has completed on this design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -13,6 +13,8 @@ An Execution is a hierarchy of ReAct agents shaped by the dependencies between D
|
||||
|
||||
The dependency graph is declared as data in code. The Orchestrator picks what to do next through tools that refuse work whose dependencies aren't met. Dependencies are guaranteed by code, while ordering and parallelism are the model's choice.
|
||||
|
||||
> **Under review (2026-10-07).** [ADR-0007](0007-jev-is-the-orchestrators-decision-point.md), proposed, would have the Orchestrator decide with Jev and run no language model. If accepted, it replaces "the Orchestrator ... is an ADK `LlmAgent`" above and reverses the third option rejected below. Specialists are unaffected.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **ADK end to end** (root `LlmAgent` with `sub_agents`/`AgentTool`, and ADK sessions for pauses): rejected. It departs from the KAB workflow standard, and LangGraph's checkpointed `interrupt()` is the proven platform path for resumable user Approvals.
|
||||
|
||||
37
docs/adr/0004-lessons-and-overlays-arrive-as-skills.md
Normal file
37
docs/adr/0004-lessons-and-overlays-arrive-as-skills.md
Normal file
@@ -0,0 +1,37 @@
|
||||
---
|
||||
status: proposed
|
||||
---
|
||||
|
||||
# Lessons and overlays arrive as Skills
|
||||
|
||||
An Execution starts clean ([CONTEXT.md](../../CONTEXT.md)) unless it is a Change request with a named Baseline ([ADR-0006](0006-a-change-request-starts-from-a-baseline-execution.md)), so nothing one Execution learns may flow into the next as hidden state. What an Execution gets wrong is still worth keeping. We decide that everything that changes how agents behave between Executions is a Skill change, of one of two kinds.
|
||||
|
||||
- **Lessons.** Each Execution records **lesson candidates**: Revisions and their reasons, validator findings, tool refusals, the user's change requests, and work that stopped. An offline curator proposes lessons from candidates across Executions. A person approves each one. An approved lesson is added to a lessons Skill that the provider bundle assigns to roles.
|
||||
- **Overlays.** An organisation's standards or one engagement's constraints are written as their own Skills. The provider bundle lists the ones every Execution loads. A caller may name further overlay Skills when it starts an Execution.
|
||||
|
||||
Vendored Skills stay unmodified ([ADR-0002](0002-skills-are-the-portable-unit-of-behaviour.md)); lessons and overlays sit beside them.
|
||||
|
||||
## Decisions to confirm
|
||||
|
||||
This ADR is proposed. Accepting it means agreeing to each of these:
|
||||
|
||||
1. **No lesson takes effect without a person's approval.** The curator proposes; it never edits a Skill that agents load. We prefer slower learning to an agent that rewrites its own instructions.
|
||||
2. **A lesson must be shown to help.** It is kept only if the evaluation, run with and without it, holds quality and lowers Revisions, time or tokens.
|
||||
3. **Lessons are per Cloud provider.** The lessons Skill is listed in that provider's bundle, since most lessons name its products.
|
||||
4. **Overlay Skills come only from directories the deployment configures.** A caller names an overlay by its Skill name; it cannot upload one. Each passes the same Agent Skills validation as bundled Skills, and an unknown name refuses the Execution at the start.
|
||||
5. **Every Execution records the Skills it loaded and a content hash of each,** so a result can be traced to the instructions that produced it.
|
||||
6. **Lesson candidates are client data.** They quote user feedback and name resources. They are stored with the Execution's records, never in the repository, and a lesson is written so that it carries no client detail.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Memory carried between Executions** (a vector store of past Deliverables or feedback, retrieved at run time): rejected. It breaks "every Execution starts clean", can leak one client's content into another's Execution, and cannot be reviewed or rolled back as a diff.
|
||||
- **Automatic Skill edits** (the curator applies lessons directly): rejected for now. Without review, one bad Execution can degrade every later one, and nothing ties a regression to its cause.
|
||||
- **Editing the vendored Skills in place**: rejected by ADR-0002. Upstream changes must arrive as a clean diff.
|
||||
- **Overlays as prompt text passed by the caller**: rejected. Free text bypasses Skill validation and progressive loading, and it is an injection path.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Learning is slow by design: a lesson reaches agents only after capture, proposal, review and an evaluation run.
|
||||
- The evaluation must be able to see a difference a Skill makes. Its structural checks cannot, so the judged checks and the thresholds on Revisions, time and tokens become part of the loop.
|
||||
- The glossary gains **Lesson candidate**, **Lesson** and **Overlay Skill** once this is accepted.
|
||||
- Skills can no longer be loaded once at start-up for all Executions: the runtime must resolve a role's Skills per Execution.
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# The target architecture is an explicit graph, and code comes after it
|
||||
|
||||
Today the design passes from Specialist to Specialist as prose, and the infrastructure code is written before anyone has stated what it must satisfy. In an evaluation run on 2026-10-06 the validator found 4 problems, then 7, then 5 different ones, across three Revisions of the same Terraform: the bar was discovered after the code existed, and it moved. We decide that the **Target architecture** is one explicit graph, held as data, that is established before any infrastructure code is written, and that everything after it is derived from it and checked against it.
|
||||
|
||||
The graph has three kinds of thing:
|
||||
|
||||
- **Elements**: every component of the solution and the resource that realises it, with the product selected for it.
|
||||
- **Relationships**: how Elements connect: calls, publishes to, reads from, is granted access to.
|
||||
- **Checks**: on each Element and Relationship, the numbered, verifiable conditions it must meet. Each Check names the requirement or design recommendation it comes from.
|
||||
|
||||
This makes the **Requirements chain** explicit: every requirement leads to the components that serve it, to the Elements and Checks that realise it, to the code that implements them, and to a validation verdict. A link that is missing is refused, not discovered later.
|
||||
|
||||
## Who provides what
|
||||
|
||||
The user states what they want to achieve, at a high level, in the Workflow request, and answers the questions that only they can answer. The agents produce everything in the graph: every Element, Relationship and Check is derived by Specialists from the requirements, the Cloud provider's architecture framework and its documented best practices. The user never writes or edits the graph. They approve it, from a summary, before code is written.
|
||||
|
||||
## What changes
|
||||
|
||||
- **A new Deliverable, the Target architecture,** follows product selection and design recommendations. Its graph is an attachment; its content is the readable summary.
|
||||
- **Completeness is a submission rule.** The Target architecture is refused when a requirement, a selected product or a design recommendation has no Element or Check, or when a Relationship or Check refers to an Element that is not in the graph.
|
||||
- **Later Deliverables are views of the graph.** The architecture diagram is rendered from its Elements and Relationships. The architecture description describes them. The deployment guidance implements every Element, Relationship and Check.
|
||||
- **Validation is a verdict per Check,** pass or fail with evidence, not a list the validator composes on each pass. Something the validator notices that the graph lacks is a proposed addition to the Target architecture. It cannot fail the code. If it matters, the Target architecture is revised, and the code follows.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **The user approves the Target architecture before code is written.** The solution architecture Phase splits in two: design, which ends with the Target architecture, and implementation, which holds the deployment guidance. The upstream Skill has four Phases; this makes five, and is a deliberate departure from it.
|
||||
2. **A new Specialist produces the Target architecture.** The design advisor stays responsible for recommendations only.
|
||||
3. **Code renders the diagram from the graph.** No model draws it, so the picture cannot disagree with the architecture, and the diagrammer role goes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Keep the prose chain and tighten the validator's instructions**: rejected. The validator would still decide what to check after the code exists, and nothing would make two passes agree.
|
||||
- **A build plan for the infrastructure code only**, beside the architecture: rejected. It would be a second description of the same system, free to drift from the diagram and the description. The graph has to be the architecture itself.
|
||||
- **Let the infrastructure code be the model** and derive the diagram and checks from Terraform: rejected. It puts code first, which is the order that failed, and it ties the architecture to one infrastructure-as-code tool.
|
||||
- **Raise the Revision limit**: rejected. It spends more on a loop that has no fixed target.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Requirements reach code through data, not prose, so a requirement that nothing implements is caught when the Target architecture is submitted.
|
||||
- A Revision either closes a known Check or it does not. Validation findings stop growing between passes.
|
||||
- More is decided before code: the Specialist that writes the Target architecture carries work the infrastructure author and validator did implicitly, and a gap in the graph now costs a Revision of the Target architecture and everything after it.
|
||||
- The Phase list is no longer the upstream Skill's. The solution guide still follows the upstream template; only where Approvals fall changes.
|
||||
- The vocabulary of the graph is part of the glossary: Target architecture, Element, Relationship, Check, Requirements chain. "Dependency" keeps its existing meaning, between Deliverables.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# A change request starts from a baseline Execution
|
||||
|
||||
A user's requirements change after an architecture has been delivered. They need to say what has changed, at a high level, and get the architecture updated, not rebuilt from nothing with no link to what was agreed before. Today that is impossible by design: every Execution starts clean, and a Session or Customer case only groups Executions. We decide that a caller may start an Execution as a **Change request** against a **Baseline**: a completed Execution of the same Customer case. The new Execution begins from the Baseline's approved Deliverables and the user's description of what has changed, and it delivers a full, updated set plus a statement of what changed and why.
|
||||
|
||||
This is the one, explicit exception to "every Execution starts clean". It is named by the caller, limited to the same Customer case, and recorded in the Execution's manifest. Nothing flows between Executions unless a Baseline is named.
|
||||
|
||||
## How a Change request runs
|
||||
|
||||
- **The user says what changed,** in the Workflow request, at the same high level as an original request: "retention goes from 30 days to 7 years", "it must now also serve a second region". It becomes the next statement in the Customer case's Intent history.
|
||||
- **The agents work out the consequences.** The requirements analyst revises the Baseline's requirements, keeping every id that still holds and marking each requirement added, changed or removed. Each later Specialist starts from the Baseline's version of its Deliverable and changes only what the changed requirements reach.
|
||||
- **The Requirements chain shows the reach.** Because requirements, components, Elements and Checks carry ids ([ADR-0005](0005-the-target-architecture-is-an-explicit-graph.md)), a changed requirement leads to the Elements and Checks it touches. Everything it does not reach is carried over unchanged.
|
||||
- **The user approves the change, not the whole.** Each Approval shows what was added, changed and removed against the Baseline.
|
||||
- **Validation covers every Check,** carried over or new, so an unchanged part that a change has broken is still caught.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-07, with each of these:
|
||||
|
||||
1. **A Change request is a new Execution,** with its own id, Approvals and files. The Baseline is never modified, so every delivered version of a Customer case's architecture remains on record.
|
||||
2. **The Baseline must be a completed Execution of the same Customer case.** One case can never start from another's work.
|
||||
3. **The output is complete, not a patch.** A Change request delivers the full updated Deliverables and a change summary, so the latest Execution of a case always stands on its own.
|
||||
4. **A Change request may itself be a Baseline,** so a case's Executions form a line of versions.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Reopen the completed Execution** and revise its Deliverables in place: rejected. It destroys the record of what was delivered and approved, and an Execution that never ends cannot be audited.
|
||||
- **Start clean and ask the user to restate everything** with the change included: rejected. It loses every decision and Approval already made, changes parts nobody asked to change, and gives no account of what differs.
|
||||
- **Carry context automatically within a Customer case:** rejected. Hidden carry-over is what "starts clean" exists to prevent; the caller must choose the Baseline.
|
||||
- **Deliver only a diff** of the Deliverables: rejected. The reader would have to assemble the current architecture from a chain of patches.
|
||||
|
||||
## Consequences
|
||||
|
||||
- "Every Execution starts clean" becomes "every Execution starts clean unless the caller names a Baseline". The glossary entries for Execution, Session and Customer case change to say so, and gain **Change request** and **Baseline**.
|
||||
- This depends on ADR-0005. Without ids along the Requirements chain there is nothing to trace a change through, and a Change request would be a rewrite with extra context.
|
||||
- A Change request against a Baseline produced by older Skills is revised under the Skills loaded now; the change summary can therefore include changes the user did not ask for. They must be shown as such.
|
||||
- Lesson candidates gain a strong signal: what users change after delivery.
|
||||
55
docs/adr/0007-jev-is-the-orchestrators-decision-point.md
Normal file
55
docs/adr/0007-jev-is-the-orchestrators-decision-point.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
status: proposed
|
||||
---
|
||||
|
||||
# Jev is the Orchestrator's decision point
|
||||
|
||||
[ADR-0003](0003-langgraph-holds-state-adk-agents-reason.md) made the Orchestrator a reasoning agent on a large language model. In practice its turns are slow and mostly mechanical: in the evaluation run of 2026-10-06 it took about 7 of 35 minutes, and stalled four times before doing the only thing available. We decide that the Orchestrator follows the agent-control pattern Vercel describes for Jev ([Jev in AI agent loops](https://vercel.com/i/jev-agent-control)): Jev, a decision model that answers typed questions with probabilities, proposes the next action; application code checks that it is allowed, carries it out and records it; a generative model only writes what the user reads. Specialists are unchanged: they still reason and write on language models.
|
||||
|
||||
## Who owns what
|
||||
|
||||
| Responsibility | Owner in the pattern | Here |
|
||||
|---|---|---|
|
||||
| Interpret the request | A model with supplied context | The requirements analyst and the Specialists after it |
|
||||
| Propose the next action | The decision model | Jev, asked typed questions about the Execution's state |
|
||||
| Check permission | Application code | The Ledger: Dependencies, Approvals, the Revision limit, the cloud-read gate |
|
||||
| Execute the action | The tool runner | The Orchestrator's existing tools, called by code |
|
||||
| Update state | Application code | The Execution graph and its checkpoint |
|
||||
| Explain the result | A generative model | One small model call that writes an Approval summary or a question, from data it is given |
|
||||
|
||||
## How a turn is decided
|
||||
|
||||
1. **Code lists the allowed actions** from the Ledger and the Execution's state: relay waiting questions, act on a validation outcome, request an Approval, delegate ready work, finish. This is the permission check, and it holds whatever Jev answers, because a typed decision can still be wrong.
|
||||
2. **Jev proposes the next action** from that list, and answers the other closed questions the turn needs, in one request where they rest on the same evidence:
|
||||
- Did the user's last message ask for a change to submitted work? (yes or no)
|
||||
- Which Deliverable does that change affect? (one of the Deliverables)
|
||||
- What should happen after this validation: revise the code, revise the Target architecture, or proceed? (one of three)
|
||||
A question whose options depend on another answer, or on evidence not yet fetched, is a second request.
|
||||
3. **Code carries the decision out** through the existing tools and records it: the questions, the answers, their probabilities and what was done. A resumed Execution reuses a recorded decision when its input is unchanged.
|
||||
4. **A generative model writes the reply**, only where the user reads prose: the summary they approve a Phase from, and any question the Orchestrator itself asks. It is given the decision and the data and decides nothing.
|
||||
|
||||
Two guards from the pattern are kept in code. Repeated actions are limited, so the Orchestrator cannot loop on the same step: the turn limit and the Revision limit already do this. A failed call to Jev is recorded as a failure, distinct from an answer, so "Jev was unreachable" is never read as "nothing to do".
|
||||
|
||||
## Decisions to confirm
|
||||
|
||||
This ADR is proposed. Accepting it means agreeing to each of these:
|
||||
|
||||
1. **No language model decides what the Orchestrator does.** Jev proposes and code disposes. A generative model writes user-facing text only. This replaces the part of ADR-0003 that makes the Orchestrator a reasoning agent.
|
||||
2. **Low confidence goes to the user, not to another model.** Below a decision's threshold, an interactive Execution asks the user in plain terms; a pre-approved one takes the safest allowed action and records a warning.
|
||||
3. **Jev being unreachable does not stop an Execution.** After a fixed retry budget the Orchestrator falls back to the fixed order of its seven steps, in code, and records that it did.
|
||||
4. **What is sent to Jev is the Execution's state, not its content:** Deliverable names and statuses, Check ids and verdicts, and the user's latest message. Deliverable text and discovered cloud data are not sent for an Orchestrator decision.
|
||||
5. **Jev does not choose Specialists.** Which Specialist produces a Deliverable is fixed by the dependency graph, and all ready work is delegated together. The pattern's specialist routing has nothing to route here.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **A smaller language model as Orchestrator:** rejected by the owner. It keeps the slow, open-ended turn and the stalls, at lower cost.
|
||||
- **No generative model at all, with fixed wording for summaries and questions:** this ADR's first draft. Rejected to follow the pattern, which pairs Jev with a generative model for replies: an Approval summary is what a person decides from and reads better written than assembled.
|
||||
- **Code only, with fixed rules:** rejected. It cannot tell what a user's free-text reply is asking for.
|
||||
- **Code for the mechanical steps and the existing model for judgement:** rejected in favour of one decision point. It keeps two ways of deciding and the large model in the loop.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The Orchestrator's instructions in `agents/orchestrator.md` become the wording of Jev's questions and the brief for the reply writer, held in a Skill or the provider bundle so they stay reviewable and portable.
|
||||
- Jev becomes a dependency of every interactive Execution. Its availability, latency, rate limits and data retention must be known before this is built; none is documented publicly.
|
||||
- Decisions become measurable: each has a confidence, so thresholds can be tuned from recorded Executions.
|
||||
- An unusual request that fits none of the questions is handled by asking the user, where a language model might have improvised.
|
||||
Reference in New Issue
Block a user