feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)

The store's rules now sit on a Blobs layer: a folder, as before, or a
prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND.
The layout is the same in both; the record is written last.

- S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials
- an s3 extra (boto3), installed in the image; moto for tests
- every store test runs on both kinds of storage

Retention, the platform's sign-in and each kept file as a platform
artifact are left: the last needs the platform's S3 backend, which it does
not yet have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:08:39 +01:00
parent 1ec939f4fd
commit 83df77005a
9 changed files with 314 additions and 57 deletions

View File

@@ -0,0 +1,30 @@
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
## Scope, as the owner chose it
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
## What was built
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
## Verified
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
- 606 tests pass; `ruff` and `mypy` clean.
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
## Not done
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
- Nothing kept on the volume before is moved: nothing had been kept.