feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)
The store's rules now sit on a Blobs layer: a folder, as before, or a prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND. The layout is the same in both; the record is written last. - S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials - an s3 extra (boto3), installed in the image; moto for tests - every store test runs on both kinds of storage Retention, the platform's sign-in and each kept file as a platform artifact are left: the last needs the platform's S3 backend, which it does not yet have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,7 +7,7 @@ One directory per iteration. Each holds the plan for that iteration and, for fin
|
||||
| # | Directory | Status | What it is |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | `iter-01-execution-engine/` | steps 1 to 6 built on 2026-10-13; 7 planned | The Execution engine: brought from the Reference, given a Workflow, and proved on a toy Workflow and on the Reference's own. |
|
||||
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1 and K2 (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
|
||||
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1, K2 and K3's object storage (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
|
||||
|
||||
The plan across both repositories is `../tom_analysis/IMPLEMENTATION_PLAN.md`.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 2, the Orchestration engine
|
||||
**Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`).
|
||||
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1 and K2 built 2026-10-21
|
||||
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1, K2 and object storage (K3) built 2026-10-21
|
||||
|
||||
## Decided by the owner, 2026-10-20
|
||||
|
||||
@@ -27,4 +27,4 @@
|
||||
| F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned |
|
||||
| K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow |
|
||||
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow |
|
||||
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | planned |
|
||||
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | object storage built (`steps/step-K3.md`); retention, the platform's sign-in and platform artifacts not chosen for now, the last needing the platform team |
|
||||
|
||||
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
|
||||
|
||||
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
|
||||
|
||||
## Scope, as the owner chose it
|
||||
|
||||
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
|
||||
|
||||
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
|
||||
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
|
||||
|
||||
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
|
||||
|
||||
## What was built
|
||||
|
||||
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
|
||||
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
|
||||
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
|
||||
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
|
||||
|
||||
## Verified
|
||||
|
||||
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
|
||||
- 606 tests pass; `ruff` and `mypy` clean.
|
||||
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
|
||||
|
||||
## Not done
|
||||
|
||||
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
|
||||
- Nothing kept on the volume before is moved: nothing had been kept.
|
||||
Reference in New Issue
Block a user