Files
orchestration_engine/.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Jonathan Boniface 83df77005a feat: keep delivered work in S3-compatible object storage (ADR-0004 phase 3)
The store's rules now sit on a Blobs layer: a folder, as before, or a
prefix in an S3-compatible bucket (MinIO locally), chosen by STORE_BACKEND.
The layout is the same in both; the record is written last.

- S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; AWS credentials
- an s3 extra (boto3), installed in the image; moto for tests
- every store test runs on both kinds of storage

Retention, the platform's sign-in and each kept file as a platform
artifact are left: the last needs the platform's S3 backend, which it does
not yet have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:08:39 +01:00

2.5 KiB

Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)

Built 2026-10-21 on feat/object-store, from feat/sources-from.

Scope, as the owner chose it

Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:

  • The platform's S3 and GCS artifact backends are not implemented (agent-builder-api/src/storage/router.py: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
  • The only artifact route the internal service token may call, POST /api/v1/executions/{id}/artifacts, registers a step's whole output (step_N/output.json); POST /projects/{id}/artifacts needs a person's sign-in.

So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.

What was built

  • store.py: the store's rules (names, write-once, records, latest) over a Blobs layer with two kinds: VolumeBlobs, a folder, and S3Blobs, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}. The record is written last, so an Execution is kept once its record is. A delivered file named record.json is not taken for one.
  • Settings: STORE_BACKEND (volume or s3), S3_BUCKET, S3_PREFIX, S3_ENDPOINT_URL, S3_REGION; credentials are the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. store_from_settings makes the store.
  • Packaging: the s3 extra (boto3); the image installs it. moto in the dev extra; mypy told the AWS SDK has no types.
  • Local deployment: the dev kit's MinIO (agent-builder-minio, started alone from the observability profile), bucket orchestration, its root credentials from the kit's .env.

Verified

  • Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered record.json, and the settings choosing the storage.
  • 606 tests pass; ruff and mypy clean.
  • Deployed: healthy, with the orchestration bucket made and reachable on MinIO. No agent was run.

Not done

  • Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
  • Nothing kept on the volume before is moved: nothing had been kept.