feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)

The store's rules now sit on a Blobs layer: a folder, as before, or a
prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND.
The layout is the same in both; the record is written last.

- S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials
- an s3 extra (boto3), installed in the image; moto for tests
- every store test runs on both kinds of storage

Retention, the platform's sign-in and each kept file as a platform
artifact are left: the last needs the platform's S3 backend, which it does
not yet have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:08:39 +01:00
parent 1ec939f4fd
commit 83df77005a
9 changed files with 314 additions and 57 deletions

View File

@@ -2,7 +2,7 @@
**Iteration:** 2, the Orchestration engine
**Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`).
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1 and K2 built 2026-10-21
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1, K2 and object storage (K3) built 2026-10-21
## Decided by the owner, 2026-10-20
@@ -27,4 +27,4 @@
| F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned |
| K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow |
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow |
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | planned |
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | object storage built (`steps/step-K3.md`); retention, the platform's sign-in and platform artifacts not chosen for now, the last needing the platform team |

View File

@@ -0,0 +1,30 @@
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
## Scope, as the owner chose it
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
## What was built
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
## Verified
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
- 606 tests pass; `ruff` and `mypy` clean.
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
## Not done
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
- Nothing kept on the volume before is moved: nothing had been kept.