feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)
The store's rules now sit on a Blobs layer: a folder, as before, or a prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND. The layout is the same in both; the record is written last. - S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials - an s3 extra (boto3), installed in the image; moto for tests - every store test runs on both kinds of storage Retention, the platform's sign-in and each kept file as a platform artifact are left: the last needs the platform's S3 backend, which it does not yet have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
|
||||
|
||||
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
|
||||
|
||||
## Scope, as the owner chose it
|
||||
|
||||
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
|
||||
|
||||
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
|
||||
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
|
||||
|
||||
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
|
||||
|
||||
## What was built
|
||||
|
||||
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
|
||||
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
|
||||
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
|
||||
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
|
||||
|
||||
## Verified
|
||||
|
||||
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
|
||||
- 606 tests pass; `ruff` and `mypy` clean.
|
||||
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
|
||||
|
||||
## Not done
|
||||
|
||||
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
|
||||
- Nothing kept on the volume before is moved: nothing had been kept.
|
||||
Reference in New Issue
Block a user