feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)

The store's rules now sit on a Blobs layer: a folder, as before, or a
prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND.
The layout is the same in both; the record is written last.

- S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials
- an s3 extra (boto3), installed in the image; moto for tests
- every store test runs on both kinds of storage

Retention, the platform's sign-in and each kept file as a platform
artifact are left: the last needs the platform's S3 backend, which it does
not yet have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:08:39 +01:00
parent 1ec939f4fd
commit 83df77005a
9 changed files with 314 additions and 57 deletions

View File

@@ -7,7 +7,7 @@ One directory per iteration. Each holds the plan for that iteration and, for fin
| # | Directory | Status | What it is |
| --- | --- | --- | --- |
| 1 | `iter-01-execution-engine/` | steps 1 to 6 built on 2026-10-13; 7 planned | The Execution engine: brought from the Reference, given a Workflow, and proved on a toy Workflow and on the Reference's own. |
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1 and K2 (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1, K2 and K3's object storage (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
The plan across both repositories is `../tom_analysis/IMPLEMENTATION_PLAN.md`.

View File

@@ -2,7 +2,7 @@
**Iteration:** 2, the Orchestration engine
**Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`).
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1 and K2 built 2026-10-21
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1, K2 and object storage (K3) built 2026-10-21
## Decided by the owner, 2026-10-20
@@ -27,4 +27,4 @@
| F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned |
| K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow |
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow |
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | planned |
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | object storage built (`steps/step-K3.md`); retention, the platform's sign-in and platform artifacts not chosen for now, the last needing the platform team |

View File

@@ -0,0 +1,30 @@
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
## Scope, as the owner chose it
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
## What was built
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
## Verified
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
- 606 tests pass; `ruff` and `mypy` clean.
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
## Not done
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
- Nothing kept on the volume before is moved: nothing had been kept.

View File

@@ -5,8 +5,8 @@
#
# Which agents it routes to is configuration, mounted at /app/agents.yaml
# (AGENTS_FILE); deploy/agents.local.yaml is the local KAB platform's. What agents
# deliver is kept in /data/store (ADR-0004); give it a volume, a PUBLIC_URL and a
# LINK_SECRET.
# deliver is kept in /data/store (ADR-0004), or in an S3-compatible bucket with
# STORE_BACKEND=s3; give it a volume or a bucket, a PUBLIC_URL and a LINK_SECRET.
FROM python:3.12-slim AS builder
@@ -14,7 +14,7 @@ WORKDIR /build
COPY pyproject.toml ./
COPY execution_engine/ ./execution_engine/
COPY orchestration/ ./orchestration/
RUN pip install --no-cache-dir --prefix=/install .
RUN pip install --no-cache-dir --prefix=/install ".[s3]"
FROM python:3.12-slim

View File

@@ -27,9 +27,16 @@ class OrchestrationSettings(BaseSettings):
# agent's work takes.
CONNECT_TIMEOUT_SECONDS: float = 10
# What agents deliver is kept here and served at signed links (ADR-0004).
# Unset: nothing is kept, and a caller asking for links gets none.
# What agents deliver is kept and served at signed links (ADR-0004): in a folder
# (`volume`, STORE_DIR; unset, nothing is kept) or in an S3-compatible bucket
# (`s3`). The bucket's credentials are the standard AWS_ACCESS_KEY_ID and
# AWS_SECRET_ACCESS_KEY.
STORE_BACKEND: str = "volume"
STORE_DIR: str | None = None
S3_BUCKET: str | None = None
S3_PREFIX: str = ""
S3_ENDPOINT_URL: str | None = None # MinIO, or another S3-compatible service
S3_REGION: str = "us-east-1"
# The base of every link: the address a browser reaches this service at, which
# is not its address inside the platform's network. Unset: A2A_PUBLIC_URL.
PUBLIC_URL: str | None = None

View File

@@ -29,7 +29,7 @@ from orchestration.delivery import Delivery
from orchestration.links import Signer
from orchestration.relay import AgentClients, RelayExecutor
from orchestration.serving import kept_routes
from orchestration.store import Store
from orchestration.store import Store, store_from_settings
logger = logging.getLogger(__name__)
@@ -121,8 +121,8 @@ def _delivery(
) -> Delivery | None:
"""Where what agents deliver is kept, and how it is linked to; None when
nothing is to be kept."""
if store is None and settings.STORE_DIR:
store = Store(settings.STORE_DIR)
if store is None:
store = store_from_settings(settings)
if store is None:
return None
if signer is None:

View File

@@ -1,10 +1,11 @@
"""What agents delivered, kept by tenant, Customer case and Execution (ADR-0004).
<root>/tenants/<tenant>/cases/<case>/executions/<id>/record.json
/files/<name as delivered>
tenants/<tenant>/cases/<case>/executions/<id>/record.json
/files/<name as delivered>
An Execution is written once. Its files are what the agent delivered, by the names
it gave them; its record says what it was.
it gave them; its record says what it was. The same layout is kept in a folder or
under a prefix in an S3-compatible bucket.
"""
from __future__ import annotations
@@ -15,7 +16,10 @@ import re
import time
from collections.abc import Mapping
from pathlib import Path, PurePosixPath
from typing import Any
from typing import TYPE_CHECKING, Any, Protocol
if TYPE_CHECKING:
from orchestration.config import OrchestrationSettings
_SLUG = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
_MEDIA_TYPES = {
@@ -53,21 +57,128 @@ def media_type(name: str) -> str:
return _MEDIA_TYPES.get(PurePosixPath(name).suffix.lower(), _TEXT)
class Store:
class Blobs(Protocol):
"""Where a store's keys are kept: a folder, or a bucket."""
def put(self, key: str, data: bytes, content_type: str) -> None: ...
def get(self, key: str) -> bytes | None: ...
def exists(self, key: str) -> bool: ...
def keys(self, prefix: str) -> list[str]: ...
class VolumeBlobs:
"""Keys as files under a folder."""
def __init__(self, root: str | Path) -> None:
self.root = Path(root)
def _case(self, tenant: str, case: str) -> Path:
return (
self.root
/ "tenants"
/ slug(tenant, "Tenant")
/ "cases"
/ slug(case, "Case")
def _path(self, key: str) -> Path:
path = (self.root / key).resolve()
if self.root.resolve() not in path.parents:
raise ValueError(f"{key!r} is outside the store")
return path
def put(self, key: str, data: bytes, content_type: str) -> None:
path = self._path(key)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
def get(self, key: str) -> bytes | None:
path = self._path(key)
return path.read_bytes() if path.is_file() else None
def exists(self, key: str) -> bool:
return self._path(key).is_file()
def keys(self, prefix: str) -> list[str]:
folder = self.root / prefix
if not folder.is_dir():
return []
return [
path.relative_to(self.root).as_posix()
for path in folder.rglob("*")
if path.is_file()
]
class S3Blobs:
"""Keys as objects under a prefix in an S3-compatible bucket (MinIO locally)."""
def __init__(
self, client: Any, bucket: str, prefix: str = "", create: bool = False
) -> None:
self._client = client
self.bucket = bucket
self.prefix = prefix.lstrip("/")
if create:
self._ensure_bucket()
def _ensure_bucket(self) -> None:
from botocore.exceptions import ClientError
try:
self._client.head_bucket(Bucket=self.bucket)
except ClientError:
self._client.create_bucket(Bucket=self.bucket)
def put(self, key: str, data: bytes, content_type: str) -> None:
self._client.put_object(
Bucket=self.bucket,
Key=self.prefix + key,
Body=data,
ContentType=content_type,
)
def _execution(self, tenant: str, case: str, execution_id: str) -> Path:
return self._case(tenant, case) / "executions" / slug(execution_id, "Execution")
def get(self, key: str) -> bytes | None:
from botocore.exceptions import ClientError
try:
response = self._client.get_object(
Bucket=self.bucket, Key=self.prefix + key
)
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
return None
raise
data: bytes = response["Body"].read()
return data
def exists(self, key: str) -> bool:
from botocore.exceptions import ClientError
try:
self._client.head_object(Bucket=self.bucket, Key=self.prefix + key)
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
return False
raise
return True
def keys(self, prefix: str) -> list[str]:
found: list[str] = []
pages = self._client.get_paginator("list_objects_v2").paginate(
Bucket=self.bucket, Prefix=self.prefix + prefix
)
for page in pages:
for item in page.get("Contents", []):
found.append(str(item["Key"])[len(self.prefix) :])
return found
class Store:
def __init__(self, blobs: Blobs | str | Path) -> None:
self.blobs: Blobs = (
VolumeBlobs(blobs) if isinstance(blobs, (str, Path)) else blobs
)
@staticmethod
def _case(tenant: str, case: str) -> str:
return f"tenants/{slug(tenant, 'Tenant')}/cases/{slug(case, 'Case')}"
def _execution(self, tenant: str, case: str, execution_id: str) -> str:
return (
f"{self._case(tenant, case)}/executions/{slug(execution_id, 'Execution')}"
)
def keep(
self,
@@ -86,12 +197,12 @@ class Store:
already kept is refused."""
folder = self._execution(tenant, case, execution_id)
paths = {name: file_name(name) for name in files}
if folder.exists():
if self.blobs.exists(f"{folder}/record.json"):
raise ValueError(f"Execution {execution_id} is already kept")
for name, path in paths.items():
target = folder / "files" / path
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(files[name], encoding="utf-8")
self.blobs.put(
f"{folder}/files/{path}", files[name].encode("utf-8"), media_type(name)
)
record = {
"tenant": tenant,
"customer_case": case,
@@ -105,20 +216,26 @@ class Store:
"kept_at": dt.datetime.now(dt.UTC).isoformat(),
"kept_ns": time.time_ns(),
}
folder.mkdir(parents=True, exist_ok=True)
(folder / "record.json").write_text(json.dumps(record, indent=2), "utf-8")
# The record last: an Execution is kept once its record is.
self.blobs.put(
f"{folder}/record.json",
json.dumps(record, indent=2).encode("utf-8"),
"application/json",
)
return record
def record(
self, tenant: str, case: str, execution_id: str
) -> dict[str, Any] | None:
try:
path = self._execution(tenant, case, execution_id) / "record.json"
data = self.blobs.get(
f"{self._execution(tenant, case, execution_id)}/record.json"
)
except ValueError:
return None
if not path.is_file():
if data is None:
return None
record: dict[str, Any] = json.loads(path.read_text("utf-8"))
record: dict[str, Any] = json.loads(data)
return record
def file(
@@ -126,13 +243,13 @@ class Store:
) -> tuple[bytes, str] | None:
"""A kept file's content and media type, or None."""
try:
folder = self._execution(tenant, case, execution_id) / "files"
target = folder / file_name(name)
key = (
f"{self._execution(tenant, case, execution_id)}/files/{file_name(name)}"
)
data = self.blobs.get(key)
except ValueError:
return None
if not target.is_file() or folder.resolve() not in target.resolve().parents:
return None
return target.read_bytes(), media_type(name)
return None if data is None else (data, media_type(name))
def files(self, tenant: str, case: str, execution_id: str) -> dict[str, str]:
"""Every file kept for an Execution, by name."""
@@ -147,12 +264,17 @@ class Store:
def case(self, tenant: str, case: str) -> list[dict[str, Any]]:
"""The Customer case's Executions, newest first."""
try:
folder = self._case(tenant, case) / "executions"
prefix = f"{self._case(tenant, case)}/executions/"
except ValueError:
return []
records = [
json.loads(path.read_text("utf-8")) for path in folder.glob("*/record.json")
]
records = []
for key in self.blobs.keys(prefix):
# <id>/record.json; a delivered file of that name is under files/.
parts = key[len(prefix) :].split("/")
if len(parts) == 2 and parts[1] == "record.json":
data = self.blobs.get(key)
if data is not None:
records.append(json.loads(data))
return sorted(records, key=lambda r: r.get("kept_ns", 0), reverse=True)
def latest(self, tenant: str, case: str, work: str) -> dict[str, Any] | None:
@@ -165,3 +287,25 @@ class Store:
),
None,
)
def store_from_settings(settings: OrchestrationSettings) -> Store | None:
"""The store the settings name; None when nothing is to be kept."""
if settings.STORE_BACKEND == "s3":
import boto3
if not settings.S3_BUCKET:
raise ValueError("STORE_BACKEND=s3 needs S3_BUCKET")
client = boto3.client(
"s3",
endpoint_url=settings.S3_ENDPOINT_URL or None,
region_name=settings.S3_REGION,
)
return Store(
S3Blobs(client, settings.S3_BUCKET, settings.S3_PREFIX, create=True)
)
if settings.STORE_BACKEND != "volume":
raise ValueError(
f"STORE_BACKEND must be volume or s3, not {settings.STORE_BACKEND!r}"
)
return Store(VolumeBlobs(settings.STORE_DIR)) if settings.STORE_DIR else None

View File

@@ -30,7 +30,11 @@ dependencies = [
]
[project.optional-dependencies]
# The Orchestration engine's store on S3-compatible object storage (ADR-0004).
s3 = ["boto3>=1.34"]
dev = [
"boto3>=1.34",
"moto[s3]>=5.0",
"pytest>=8.0.0",
"pytest-asyncio>=0.23.0",
"ruff>=0.6.0",
@@ -59,3 +63,8 @@ cache-dir = ".cache/ruff"
[tool.mypy]
cache_dir = ".cache/mypy"
[[tool.mypy.overrides]]
# The AWS SDK and its test double ship without type information.
module = ["boto3", "boto3.*", "botocore", "botocore.*", "moto", "moto.*"]
ignore_missing_imports = true

View File

@@ -2,10 +2,30 @@
from __future__ import annotations
import pytest
from collections.abc import Iterator
from typing import Any
import boto3
import pytest
from moto import mock_aws
from orchestration.config import OrchestrationSettings
from orchestration.links import Signer
from orchestration.store import Store
from orchestration.store import S3Blobs, Store, VolumeBlobs, store_from_settings
@pytest.fixture(params=["volume", "s3"])
def make_store(request: Any, tmp_path: Any) -> Iterator[Any]:
"""A store on each kind of storage: a folder, and S3 (moto in this process)."""
if request.param == "volume":
yield lambda: Store(VolumeBlobs(tmp_path))
return
with mock_aws():
client = boto3.client("s3", region_name="us-east-1")
yield lambda: Store(
S3Blobs(client, "kept", prefix="orchestration/", create=True)
)
FILES = {
"guide.md": "# Guide",
@@ -30,8 +50,8 @@ def kept(store: Store, execution_id: str = "a1", **overrides: object) -> dict:
return store.keep(**values)
def test_an_execution_is_kept_with_its_record(tmp_path):
store = Store(tmp_path)
def test_an_execution_is_kept_with_its_record(make_store):
store = make_store()
record = kept(store)
@@ -43,8 +63,8 @@ def test_an_execution_is_kept_with_its_record(tmp_path):
assert store.record("t1", "acme", "a1") == record
def test_a_file_comes_back_with_its_media_type(tmp_path):
store = Store(tmp_path)
def test_a_file_comes_back_with_its_media_type(make_store):
store = make_store()
kept(store)
assert store.file("t1", "acme", "a1", "guide.md") == (
@@ -59,8 +79,8 @@ def test_a_file_comes_back_with_its_media_type(tmp_path):
assert store.file("t1", "acme", "a1", "missing.md") is None
def test_what_is_kept_is_never_changed(tmp_path):
store = Store(tmp_path)
def test_what_is_kept_is_never_changed(make_store):
store = make_store()
kept(store)
with pytest.raises(ValueError, match="a1"):
@@ -70,8 +90,8 @@ def test_what_is_kept_is_never_changed(tmp_path):
@pytest.mark.parametrize(
"name", ["../escape.md", "/etc/passwd", "a/../../b", "a\\b", ""]
)
def test_a_file_name_outside_the_execution_is_refused(tmp_path, name):
store = Store(tmp_path)
def test_a_file_name_outside_the_execution_is_refused(make_store, name):
store = make_store()
with pytest.raises(ValueError):
kept(store, files={name: "x"})
@@ -79,13 +99,13 @@ def test_a_file_name_outside_the_execution_is_refused(tmp_path, name):
@pytest.mark.parametrize("slug", ["..", "a/b", "", "a b"])
def test_a_tenant_or_case_that_is_not_a_slug_is_refused(tmp_path, slug):
def test_a_tenant_or_case_that_is_not_a_slug_is_refused(make_store, slug):
with pytest.raises(ValueError):
kept(Store(tmp_path), case=slug)
kept(make_store(), case=slug)
def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path):
store = Store(tmp_path)
def test_latest_is_the_newest_completed_execution_of_the_work(make_store):
store = make_store()
kept(store, "a1")
kept(store, "a2")
kept(store, "a3", status="completed_with_warnings")
@@ -101,6 +121,53 @@ def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path):
]
def test_a_delivered_file_named_like_a_record_is_not_taken_for_one(make_store):
store = make_store()
kept(store, files={"record.json": '{"not": "a record"}'})
assert [r["execution_id"] for r in store.case("t1", "acme")] == ["a1"]
def test_object_storage_keeps_the_same_layout_under_its_prefix():
with mock_aws():
client = boto3.client("s3", region_name="us-east-1")
kept(Store(S3Blobs(client, "kept", prefix="orchestration/", create=True)))
keys = sorted(
item["Key"] for item in client.list_objects_v2(Bucket="kept")["Contents"]
)
base = "orchestration/tenants/t1/cases/acme/executions/a1"
assert keys == [
f"{base}/files/graph.json",
f"{base}/files/guide.md",
f"{base}/files/terraform/main.tf",
f"{base}/record.json",
]
def test_the_settings_choose_the_storage(tmp_path, monkeypatch):
assert store_from_settings(OrchestrationSettings(_env_file=None)) is None # type: ignore[call-arg]
volume = store_from_settings(
OrchestrationSettings(_env_file=None, STORE_DIR=str(tmp_path)) # type: ignore[call-arg]
)
assert isinstance(volume, Store) and isinstance(volume.blobs, VolumeBlobs)
monkeypatch.setenv("AWS_ACCESS_KEY_ID", "test")
monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "test")
with mock_aws():
s3 = store_from_settings(
OrchestrationSettings( # type: ignore[call-arg]
_env_file=None,
STORE_BACKEND="s3",
S3_BUCKET="kept",
S3_REGION="us-east-1",
)
)
assert isinstance(s3, Store) and isinstance(s3.blobs, S3Blobs)
kept(s3)
assert s3.record("t1", "acme", "a1") is not None
def test_a_signed_path_is_accepted_and_a_changed_one_is_not():
signer = Signer("secret", ttl_seconds=60, now=lambda: 1000)