feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)

The store's rules now sit on a Blobs layer: a folder, as before, or a
prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND.
The layout is the same in both; the record is written last.

- S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials
- an s3 extra (boto3), installed in the image; moto for tests
- every store test runs on both kinds of storage

Retention, the platform's sign-in and each kept file as a platform
artifact are left: the last needs the platform's S3 backend, which it does
not yet have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:08:39 +01:00
parent 1ec939f4fd
commit 83df77005a
9 changed files with 314 additions and 57 deletions

View File

@@ -7,7 +7,7 @@ One directory per iteration. Each holds the plan for that iteration and, for fin
| # | Directory | Status | What it is | | # | Directory | Status | What it is |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| 1 | `iter-01-execution-engine/` | steps 1 to 6 built on 2026-10-13; 7 planned | The Execution engine: brought from the Reference, given a Workflow, and proved on a toy Workflow and on the Reference's own. | | 1 | `iter-01-execution-engine/` | steps 1 to 6 built on 2026-10-13; 7 planned | The Execution engine: brought from the Reference, given a Workflow, and proved on a toy Workflow and on the Reference's own. |
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1 and K2 (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. | | 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1, K2 and K3's object storage (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
The plan across both repositories is `../tom_analysis/IMPLEMENTATION_PLAN.md`. The plan across both repositories is `../tom_analysis/IMPLEMENTATION_PLAN.md`.

View File

@@ -2,7 +2,7 @@
**Iteration:** 2, the Orchestration engine **Iteration:** 2, the Orchestration engine
**Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`). **Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`).
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1 and K2 built 2026-10-21 **Status:** F1 proposed; F2 and F4 built 2026-10-20; K1, K2 and object storage (K3) built 2026-10-21
## Decided by the owner, 2026-10-20 ## Decided by the owner, 2026-10-20
@@ -27,4 +27,4 @@
| F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned | | F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned |
| K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow | | K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow |
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow | | K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow |
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | planned | | K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | object storage built (`steps/step-K3.md`); retention, the platform's sign-in and platform artifacts not chosen for now, the last needing the platform team |

View File

@@ -0,0 +1,30 @@
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
## Scope, as the owner chose it
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
## What was built
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
## Verified
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
- 606 tests pass; `ruff` and `mypy` clean.
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
## Not done
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
- Nothing kept on the volume before is moved: nothing had been kept.

View File

@@ -5,8 +5,8 @@
# #
# Which agents it routes to is configuration, mounted at /app/agents.yaml # Which agents it routes to is configuration, mounted at /app/agents.yaml
# (AGENTS_FILE); deploy/agents.local.yaml is the local KAB platform's. What agents # (AGENTS_FILE); deploy/agents.local.yaml is the local KAB platform's. What agents
# deliver is kept in /data/store (ADR-0004); give it a volume, a PUBLIC_URL and a # deliver is kept in /data/store (ADR-0004), or in an S3-compatible bucket with
# LINK_SECRET. # STORE_BACKEND=s3; give it a volume or a bucket, a PUBLIC_URL and a LINK_SECRET.
FROM python:3.12-slim AS builder FROM python:3.12-slim AS builder
@@ -14,7 +14,7 @@ WORKDIR /build
COPY pyproject.toml ./ COPY pyproject.toml ./
COPY execution_engine/ ./execution_engine/ COPY execution_engine/ ./execution_engine/
COPY orchestration/ ./orchestration/ COPY orchestration/ ./orchestration/
RUN pip install --no-cache-dir --prefix=/install . RUN pip install --no-cache-dir --prefix=/install ".[s3]"
FROM python:3.12-slim FROM python:3.12-slim

View File

@@ -27,9 +27,16 @@ class OrchestrationSettings(BaseSettings):
# agent's work takes. # agent's work takes.
CONNECT_TIMEOUT_SECONDS: float = 10 CONNECT_TIMEOUT_SECONDS: float = 10
# What agents deliver is kept here and served at signed links (ADR-0004). # What agents deliver is kept and served at signed links (ADR-0004): in a folder
# Unset: nothing is kept, and a caller asking for links gets none. # (`volume`, STORE_DIR; unset, nothing is kept) or in an S3-compatible bucket
# (`s3`). The bucket's credentials are the standard AWS_ACCESS_KEY_ID and
# AWS_SECRET_ACCESS_KEY.
STORE_BACKEND: str = "volume"
STORE_DIR: str | None = None STORE_DIR: str | None = None
S3_BUCKET: str | None = None
S3_PREFIX: str = ""
S3_ENDPOINT_URL: str | None = None # MinIO, or another S3-compatible service
S3_REGION: str = "us-east-1"
# The base of every link: the address a browser reaches this service at, which # The base of every link: the address a browser reaches this service at, which
# is not its address inside the platform's network. Unset: A2A_PUBLIC_URL. # is not its address inside the platform's network. Unset: A2A_PUBLIC_URL.
PUBLIC_URL: str | None = None PUBLIC_URL: str | None = None

View File

@@ -29,7 +29,7 @@ from orchestration.delivery import Delivery
from orchestration.links import Signer from orchestration.links import Signer
from orchestration.relay import AgentClients, RelayExecutor from orchestration.relay import AgentClients, RelayExecutor
from orchestration.serving import kept_routes from orchestration.serving import kept_routes
from orchestration.store import Store from orchestration.store import Store, store_from_settings
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -121,8 +121,8 @@ def _delivery(
) -> Delivery | None: ) -> Delivery | None:
"""Where what agents deliver is kept, and how it is linked to; None when """Where what agents deliver is kept, and how it is linked to; None when
nothing is to be kept.""" nothing is to be kept."""
if store is None and settings.STORE_DIR: if store is None:
store = Store(settings.STORE_DIR) store = store_from_settings(settings)
if store is None: if store is None:
return None return None
if signer is None: if signer is None:

View File

@@ -1,10 +1,11 @@
"""What agents delivered, kept by tenant, Customer case and Execution (ADR-0004). """What agents delivered, kept by tenant, Customer case and Execution (ADR-0004).
<root>/tenants/<tenant>/cases/<case>/executions/<id>/record.json tenants/<tenant>/cases/<case>/executions/<id>/record.json
/files/<name as delivered> /files/<name as delivered>
An Execution is written once. Its files are what the agent delivered, by the names An Execution is written once. Its files are what the agent delivered, by the names
it gave them; its record says what it was. it gave them; its record says what it was. The same layout is kept in a folder or
under a prefix in an S3-compatible bucket.
""" """
from __future__ import annotations from __future__ import annotations
@@ -15,7 +16,10 @@ import re
import time import time
from collections.abc import Mapping from collections.abc import Mapping
from pathlib import Path, PurePosixPath from pathlib import Path, PurePosixPath
from typing import Any from typing import TYPE_CHECKING, Any, Protocol
if TYPE_CHECKING:
from orchestration.config import OrchestrationSettings
_SLUG = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") _SLUG = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
_MEDIA_TYPES = { _MEDIA_TYPES = {
@@ -53,21 +57,128 @@ def media_type(name: str) -> str:
return _MEDIA_TYPES.get(PurePosixPath(name).suffix.lower(), _TEXT) return _MEDIA_TYPES.get(PurePosixPath(name).suffix.lower(), _TEXT)
class Store: class Blobs(Protocol):
"""Where a store's keys are kept: a folder, or a bucket."""
def put(self, key: str, data: bytes, content_type: str) -> None: ...
def get(self, key: str) -> bytes | None: ...
def exists(self, key: str) -> bool: ...
def keys(self, prefix: str) -> list[str]: ...
class VolumeBlobs:
"""Keys as files under a folder."""
def __init__(self, root: str | Path) -> None: def __init__(self, root: str | Path) -> None:
self.root = Path(root) self.root = Path(root)
def _case(self, tenant: str, case: str) -> Path: def _path(self, key: str) -> Path:
return ( path = (self.root / key).resolve()
self.root if self.root.resolve() not in path.parents:
/ "tenants" raise ValueError(f"{key!r} is outside the store")
/ slug(tenant, "Tenant") return path
/ "cases"
/ slug(case, "Case") def put(self, key: str, data: bytes, content_type: str) -> None:
path = self._path(key)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
def get(self, key: str) -> bytes | None:
path = self._path(key)
return path.read_bytes() if path.is_file() else None
def exists(self, key: str) -> bool:
return self._path(key).is_file()
def keys(self, prefix: str) -> list[str]:
folder = self.root / prefix
if not folder.is_dir():
return []
return [
path.relative_to(self.root).as_posix()
for path in folder.rglob("*")
if path.is_file()
]
class S3Blobs:
"""Keys as objects under a prefix in an S3-compatible bucket (MinIO locally)."""
def __init__(
self, client: Any, bucket: str, prefix: str = "", create: bool = False
) -> None:
self._client = client
self.bucket = bucket
self.prefix = prefix.lstrip("/")
if create:
self._ensure_bucket()
def _ensure_bucket(self) -> None:
from botocore.exceptions import ClientError
try:
self._client.head_bucket(Bucket=self.bucket)
except ClientError:
self._client.create_bucket(Bucket=self.bucket)
def put(self, key: str, data: bytes, content_type: str) -> None:
self._client.put_object(
Bucket=self.bucket,
Key=self.prefix + key,
Body=data,
ContentType=content_type,
) )
def _execution(self, tenant: str, case: str, execution_id: str) -> Path: def get(self, key: str) -> bytes | None:
return self._case(tenant, case) / "executions" / slug(execution_id, "Execution") from botocore.exceptions import ClientError
try:
response = self._client.get_object(
Bucket=self.bucket, Key=self.prefix + key
)
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
return None
raise
data: bytes = response["Body"].read()
return data
def exists(self, key: str) -> bool:
from botocore.exceptions import ClientError
try:
self._client.head_object(Bucket=self.bucket, Key=self.prefix + key)
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
return False
raise
return True
def keys(self, prefix: str) -> list[str]:
found: list[str] = []
pages = self._client.get_paginator("list_objects_v2").paginate(
Bucket=self.bucket, Prefix=self.prefix + prefix
)
for page in pages:
for item in page.get("Contents", []):
found.append(str(item["Key"])[len(self.prefix) :])
return found
class Store:
def __init__(self, blobs: Blobs | str | Path) -> None:
self.blobs: Blobs = (
VolumeBlobs(blobs) if isinstance(blobs, (str, Path)) else blobs
)
@staticmethod
def _case(tenant: str, case: str) -> str:
return f"tenants/{slug(tenant, 'Tenant')}/cases/{slug(case, 'Case')}"
def _execution(self, tenant: str, case: str, execution_id: str) -> str:
return (
f"{self._case(tenant, case)}/executions/{slug(execution_id, 'Execution')}"
)
def keep( def keep(
self, self,
@@ -86,12 +197,12 @@ class Store:
already kept is refused.""" already kept is refused."""
folder = self._execution(tenant, case, execution_id) folder = self._execution(tenant, case, execution_id)
paths = {name: file_name(name) for name in files} paths = {name: file_name(name) for name in files}
if folder.exists(): if self.blobs.exists(f"{folder}/record.json"):
raise ValueError(f"Execution {execution_id} is already kept") raise ValueError(f"Execution {execution_id} is already kept")
for name, path in paths.items(): for name, path in paths.items():
target = folder / "files" / path self.blobs.put(
target.parent.mkdir(parents=True, exist_ok=True) f"{folder}/files/{path}", files[name].encode("utf-8"), media_type(name)
target.write_text(files[name], encoding="utf-8") )
record = { record = {
"tenant": tenant, "tenant": tenant,
"customer_case": case, "customer_case": case,
@@ -105,20 +216,26 @@ class Store:
"kept_at": dt.datetime.now(dt.UTC).isoformat(), "kept_at": dt.datetime.now(dt.UTC).isoformat(),
"kept_ns": time.time_ns(), "kept_ns": time.time_ns(),
} }
folder.mkdir(parents=True, exist_ok=True) # The record last: an Execution is kept once its record is.
(folder / "record.json").write_text(json.dumps(record, indent=2), "utf-8") self.blobs.put(
f"{folder}/record.json",
json.dumps(record, indent=2).encode("utf-8"),
"application/json",
)
return record return record
def record( def record(
self, tenant: str, case: str, execution_id: str self, tenant: str, case: str, execution_id: str
) -> dict[str, Any] | None: ) -> dict[str, Any] | None:
try: try:
path = self._execution(tenant, case, execution_id) / "record.json" data = self.blobs.get(
f"{self._execution(tenant, case, execution_id)}/record.json"
)
except ValueError: except ValueError:
return None return None
if not path.is_file(): if data is None:
return None return None
record: dict[str, Any] = json.loads(path.read_text("utf-8")) record: dict[str, Any] = json.loads(data)
return record return record
def file( def file(
@@ -126,13 +243,13 @@ class Store:
) -> tuple[bytes, str] | None: ) -> tuple[bytes, str] | None:
"""A kept file's content and media type, or None.""" """A kept file's content and media type, or None."""
try: try:
folder = self._execution(tenant, case, execution_id) / "files" key = (
target = folder / file_name(name) f"{self._execution(tenant, case, execution_id)}/files/{file_name(name)}"
)
data = self.blobs.get(key)
except ValueError: except ValueError:
return None return None
if not target.is_file() or folder.resolve() not in target.resolve().parents: return None if data is None else (data, media_type(name))
return None
return target.read_bytes(), media_type(name)
def files(self, tenant: str, case: str, execution_id: str) -> dict[str, str]: def files(self, tenant: str, case: str, execution_id: str) -> dict[str, str]:
"""Every file kept for an Execution, by name.""" """Every file kept for an Execution, by name."""
@@ -147,12 +264,17 @@ class Store:
def case(self, tenant: str, case: str) -> list[dict[str, Any]]: def case(self, tenant: str, case: str) -> list[dict[str, Any]]:
"""The Customer case's Executions, newest first.""" """The Customer case's Executions, newest first."""
try: try:
folder = self._case(tenant, case) / "executions" prefix = f"{self._case(tenant, case)}/executions/"
except ValueError: except ValueError:
return [] return []
records = [ records = []
json.loads(path.read_text("utf-8")) for path in folder.glob("*/record.json") for key in self.blobs.keys(prefix):
] # <id>/record.json; a delivered file of that name is under files/.
parts = key[len(prefix) :].split("/")
if len(parts) == 2 and parts[1] == "record.json":
data = self.blobs.get(key)
if data is not None:
records.append(json.loads(data))
return sorted(records, key=lambda r: r.get("kept_ns", 0), reverse=True) return sorted(records, key=lambda r: r.get("kept_ns", 0), reverse=True)
def latest(self, tenant: str, case: str, work: str) -> dict[str, Any] | None: def latest(self, tenant: str, case: str, work: str) -> dict[str, Any] | None:
@@ -165,3 +287,25 @@ class Store:
), ),
None, None,
) )
def store_from_settings(settings: OrchestrationSettings) -> Store | None:
"""The store the settings name; None when nothing is to be kept."""
if settings.STORE_BACKEND == "s3":
import boto3
if not settings.S3_BUCKET:
raise ValueError("STORE_BACKEND=s3 needs S3_BUCKET")
client = boto3.client(
"s3",
endpoint_url=settings.S3_ENDPOINT_URL or None,
region_name=settings.S3_REGION,
)
return Store(
S3Blobs(client, settings.S3_BUCKET, settings.S3_PREFIX, create=True)
)
if settings.STORE_BACKEND != "volume":
raise ValueError(
f"STORE_BACKEND must be volume or s3, not {settings.STORE_BACKEND!r}"
)
return Store(VolumeBlobs(settings.STORE_DIR)) if settings.STORE_DIR else None

View File

@@ -30,7 +30,11 @@ dependencies = [
] ]
[project.optional-dependencies] [project.optional-dependencies]
# The Orchestration engine's store on S3-compatible object storage (ADR-0004).
s3 = ["boto3>=1.34"]
dev = [ dev = [
"boto3>=1.34",
"moto[s3]>=5.0",
"pytest>=8.0.0", "pytest>=8.0.0",
"pytest-asyncio>=0.23.0", "pytest-asyncio>=0.23.0",
"ruff>=0.6.0", "ruff>=0.6.0",
@@ -59,3 +63,8 @@ cache-dir = ".cache/ruff"
[tool.mypy] [tool.mypy]
cache_dir = ".cache/mypy" cache_dir = ".cache/mypy"
[[tool.mypy.overrides]]
# The AWS SDK and its test double ship without type information.
module = ["boto3", "boto3.*", "botocore", "botocore.*", "moto", "moto.*"]
ignore_missing_imports = true

View File

@@ -2,10 +2,30 @@
from __future__ import annotations from __future__ import annotations
import pytest from collections.abc import Iterator
from typing import Any
import boto3
import pytest
from moto import mock_aws
from orchestration.config import OrchestrationSettings
from orchestration.links import Signer from orchestration.links import Signer
from orchestration.store import Store from orchestration.store import S3Blobs, Store, VolumeBlobs, store_from_settings
@pytest.fixture(params=["volume", "s3"])
def make_store(request: Any, tmp_path: Any) -> Iterator[Any]:
"""A store on each kind of storage: a folder, and S3 (moto in this process)."""
if request.param == "volume":
yield lambda: Store(VolumeBlobs(tmp_path))
return
with mock_aws():
client = boto3.client("s3", region_name="us-east-1")
yield lambda: Store(
S3Blobs(client, "kept", prefix="orchestration/", create=True)
)
FILES = { FILES = {
"guide.md": "# Guide", "guide.md": "# Guide",
@@ -30,8 +50,8 @@ def kept(store: Store, execution_id: str = "a1", **overrides: object) -> dict:
return store.keep(**values) return store.keep(**values)
def test_an_execution_is_kept_with_its_record(tmp_path): def test_an_execution_is_kept_with_its_record(make_store):
store = Store(tmp_path) store = make_store()
record = kept(store) record = kept(store)
@@ -43,8 +63,8 @@ def test_an_execution_is_kept_with_its_record(tmp_path):
assert store.record("t1", "acme", "a1") == record assert store.record("t1", "acme", "a1") == record
def test_a_file_comes_back_with_its_media_type(tmp_path): def test_a_file_comes_back_with_its_media_type(make_store):
store = Store(tmp_path) store = make_store()
kept(store) kept(store)
assert store.file("t1", "acme", "a1", "guide.md") == ( assert store.file("t1", "acme", "a1", "guide.md") == (
@@ -59,8 +79,8 @@ def test_a_file_comes_back_with_its_media_type(tmp_path):
assert store.file("t1", "acme", "a1", "missing.md") is None assert store.file("t1", "acme", "a1", "missing.md") is None
def test_what_is_kept_is_never_changed(tmp_path): def test_what_is_kept_is_never_changed(make_store):
store = Store(tmp_path) store = make_store()
kept(store) kept(store)
with pytest.raises(ValueError, match="a1"): with pytest.raises(ValueError, match="a1"):
@@ -70,8 +90,8 @@ def test_what_is_kept_is_never_changed(tmp_path):
@pytest.mark.parametrize( @pytest.mark.parametrize(
"name", ["../escape.md", "/etc/passwd", "a/../../b", "a\\b", ""] "name", ["../escape.md", "/etc/passwd", "a/../../b", "a\\b", ""]
) )
def test_a_file_name_outside_the_execution_is_refused(tmp_path, name): def test_a_file_name_outside_the_execution_is_refused(make_store, name):
store = Store(tmp_path) store = make_store()
with pytest.raises(ValueError): with pytest.raises(ValueError):
kept(store, files={name: "x"}) kept(store, files={name: "x"})
@@ -79,13 +99,13 @@ def test_a_file_name_outside_the_execution_is_refused(tmp_path, name):
@pytest.mark.parametrize("slug", ["..", "a/b", "", "a b"]) @pytest.mark.parametrize("slug", ["..", "a/b", "", "a b"])
def test_a_tenant_or_case_that_is_not_a_slug_is_refused(tmp_path, slug): def test_a_tenant_or_case_that_is_not_a_slug_is_refused(make_store, slug):
with pytest.raises(ValueError): with pytest.raises(ValueError):
kept(Store(tmp_path), case=slug) kept(make_store(), case=slug)
def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path): def test_latest_is_the_newest_completed_execution_of_the_work(make_store):
store = Store(tmp_path) store = make_store()
kept(store, "a1") kept(store, "a1")
kept(store, "a2") kept(store, "a2")
kept(store, "a3", status="completed_with_warnings") kept(store, "a3", status="completed_with_warnings")
@@ -101,6 +121,53 @@ def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path):
] ]
def test_a_delivered_file_named_like_a_record_is_not_taken_for_one(make_store):
store = make_store()
kept(store, files={"record.json": '{"not": "a record"}'})
assert [r["execution_id"] for r in store.case("t1", "acme")] == ["a1"]
def test_object_storage_keeps_the_same_layout_under_its_prefix():
with mock_aws():
client = boto3.client("s3", region_name="us-east-1")
kept(Store(S3Blobs(client, "kept", prefix="orchestration/", create=True)))
keys = sorted(
item["Key"] for item in client.list_objects_v2(Bucket="kept")["Contents"]
)
base = "orchestration/tenants/t1/cases/acme/executions/a1"
assert keys == [
f"{base}/files/graph.json",
f"{base}/files/guide.md",
f"{base}/files/terraform/main.tf",
f"{base}/record.json",
]
def test_the_settings_choose_the_storage(tmp_path, monkeypatch):
assert store_from_settings(OrchestrationSettings(_env_file=None)) is None # type: ignore[call-arg]
volume = store_from_settings(
OrchestrationSettings(_env_file=None, STORE_DIR=str(tmp_path)) # type: ignore[call-arg]
)
assert isinstance(volume, Store) and isinstance(volume.blobs, VolumeBlobs)
monkeypatch.setenv("AWS_ACCESS_KEY_ID", "test")
monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "test")
with mock_aws():
s3 = store_from_settings(
OrchestrationSettings( # type: ignore[call-arg]
_env_file=None,
STORE_BACKEND="s3",
S3_BUCKET="kept",
S3_REGION="us-east-1",
)
)
assert isinstance(s3, Store) and isinstance(s3.blobs, S3Blobs)
kept(s3)
assert s3.record("t1", "acme", "a1") is not None
def test_a_signed_path_is_accepted_and_a_changed_one_is_not(): def test_a_signed_path_is_accepted_and_a_changed_one_is_not():
signer = Signer("secret", ttl_seconds=60, now=lambda: 1000) signer = Signer("secret", ttl_seconds=60, now=lambda: 1000)