Compare commits
4 Commits
feat/deliv
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 5ea4fa4065 | |||
| fcac4e425c | |||
| 83df77005a | |||
| 1ec939f4fd |
@@ -7,7 +7,7 @@ One directory per iteration. Each holds the plan for that iteration and, for fin
|
||||
| # | Directory | Status | What it is |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | `iter-01-execution-engine/` | steps 1 to 6 built on 2026-10-13; 7 planned | The Execution engine: brought from the Reference, given a Workflow, and proved on a toy Workflow and on the Reference's own. |
|
||||
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1 (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
|
||||
| 2 | `iter-02-orchestration-engine/` | F1 proposed; F2 and F4 built on 2026-10-20; K1, K2 and K3's object storage (ADR-0004) built on 2026-10-21 | The Orchestration engine: routes each request to the agent for its work and relays everything back. |
|
||||
|
||||
The plan across both repositories is `../tom_analysis/IMPLEMENTATION_PLAN.md`.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 2, the Orchestration engine
|
||||
**Intent:** The service every request comes through. It decides which agent the work belongs to, dispatches it, and relays every question and Approval. First behind it: the operating model agent (`../tom_analysis`).
|
||||
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1 built 2026-10-21
|
||||
**Status:** F1 proposed; F2 and F4 built 2026-10-20; K1, K2 and object storage (K3) built 2026-10-21
|
||||
|
||||
## Decided by the owner, 2026-10-20
|
||||
|
||||
@@ -23,8 +23,8 @@
|
||||
| F1 | ADR-0003: one Orchestration engine above the agents | Accepted by the owner | proposed |
|
||||
| F2 | The service: an A2A server, a configured list of agents, a request passed through to one agent with every prompt relayed; its image; on the local platform | An Execution of the operating model agent runs through it, Approvals included, with the agent untouched | built (`steps/step-2.md`); the platform workflow through it is step 4 of `../tom_analysis` iteration 2 |
|
||||
| F3 | The decision: readings in code, the Jev question, thresholds, the fallbacks, the record | A labelled set of messages is routed correctly; low confidence asks; no Jev key still routes what is named | planned |
|
||||
| F4 | "Both": architecture, then its files as `sources` to the operating model agent | One request yields both sets of Deliverables under one Customer case | built against scripted agents (`steps/step-4.md`), ahead of F3 at the owner's request; not yet run with real models |
|
||||
| F4 | "Both": architecture, then its files as `sources` to the operating model agent | One request yields both sets of Deliverables under one Customer case | built (`steps/step-4.md`), ahead of F3 at the owner's request; run with real models 2026-10-21: the solution architecture agent's work handed to the operating model agent, both completed in 74.9 minutes, kept in MinIO (`../solution_architecture_agent/.agents/plans/iter-01-on-the-engine/steps/step-2.md`) |
|
||||
| F5 | Later messages to the agent that owns the work | A change to the architecture is followed by a change to the operating model | planned |
|
||||
| K1 | What agents deliver, kept and served at signed links; a platform step given one document of links (ADR-0004, phase 1) | A platform step's output links to each delivered file, and each link serves it | built (`steps/step-K1.md`); the owner's run through the platform to follow |
|
||||
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | planned |
|
||||
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | planned |
|
||||
| K2 | `sources_from`: a request names kept work instead of carrying its files; the sequence moved onto it | An operating model started from the latest architecture of a Customer case | built (`steps/step-K2.md`); the owner's run through the platform to follow |
|
||||
| K3 | Object storage shared with the platform, the platform's sign-in, retention, each file as a platform artifact | A kept guide is listed among the project's artifacts | object storage built (`steps/step-K3.md`); retention, the platform's sign-in and platform artifacts not chosen for now, the last needing the platform team |
|
||||
|
||||
25
.agents/plans/iter-02-orchestration-engine/steps/step-K2.md
Normal file
25
.agents/plans/iter-02-orchestration-engine/steps/step-K2.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# Step K2: building on kept work by reference (ADR-0004, phase 2)
|
||||
|
||||
Built 2026-10-21 on `feat/sources-from`, from `feat/delivery-store`.
|
||||
|
||||
## What was built
|
||||
|
||||
- **`sources_from`:** a request names work kept earlier instead of carrying its files. It may be:
|
||||
- a link from an earlier document of links, its `record` or its `latest`: what a platform step maps from an earlier step's output. The signature is checked; the link must be of the request's tenant when the request names one, and of its Customer case when it names one;
|
||||
- `latest/<work>`: the newest completed Execution of that work in the request's Customer case;
|
||||
- an Execution's id in the request's Customer case.
|
||||
- **What the agent is sent:** the kept files as `sources`, with the Orchestration engine's record that the work completed (`execution.json`: status, Customer case, Execution, the request, the work). A request naming no Customer case takes the kept work's.
|
||||
- **Nothing starts, and the caller is told why,** when the reference resolves to nothing, to work that did not end `completed`, to another tenant or Customer case, when the link was changed, when nothing is kept here, or when `sources` is given as well. The reply is a completed task with `{"status": "not_started", "reason": ...}`.
|
||||
- **The sequence on the same footing:** an agent after another is given what was kept of the one before (`Delivery.sources`), as a request naming it in `sources_from` would be. Without a store, it is given the files relayed, as before.
|
||||
- **`Store.files`, `Delivery.resolve`, `Delivery.sources`;** the record `execution.json` moved to `delivery.py`.
|
||||
- **The card** offers `sources_from`.
|
||||
|
||||
## Verified
|
||||
|
||||
- 12 tests in `test_sources_from.py`: each of the three forms; a `latest` link followed to the newest work; a changed link, another tenant, another Customer case, nothing kept, work that ended with warnings, and `sources` with `sources_from` each starting nothing; a sequence's next agent given what was kept; the card.
|
||||
- 589 tests pass; `ruff` and `mypy` clean.
|
||||
- Deployed on the local platform; the card offers `sources_from`. No agent was run: the owner runs it through the platform.
|
||||
|
||||
## Using it from the platform
|
||||
|
||||
A second workflow step on the Orchestration engine with `work: operating_model` and `sources_from` mapped from the first step's `record`; or, in a later workflow, `sources_from: latest/solution_architecture` with the Customer case. The operating model agent's own fields (who operates it, business outcomes) go in that step's `scope`.
|
||||
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
30
.agents/plans/iter-02-orchestration-engine/steps/step-K3.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Step K3: what is kept, in object storage (ADR-0004, phase 3, first part)
|
||||
|
||||
Built 2026-10-21 on `feat/object-store`, from `feat/sources-from`.
|
||||
|
||||
## Scope, as the owner chose it
|
||||
|
||||
Phase 3 was planned as object storage shared with the platform, the platform's sign-in, retention, and each kept file as a platform artifact. Looking at the platform first changed it:
|
||||
|
||||
- **The platform's S3 and GCS artifact backends are not implemented** (`agent-builder-api/src/storage/router.py`: "S3 backend not yet implemented"). It serves artifacts from MongoDB, Azure Blob, Neo4j and its execution volume only.
|
||||
- **The only artifact route the internal service token may call,** `POST /api/v1/executions/{id}/artifacts`, registers a step's whole output (`step_N/output.json`); `POST /projects/{id}/artifacts` needs a person's sign-in.
|
||||
|
||||
So each kept file as a platform artifact is the platform team's change, and the platform's sign-in would help API callers only. The owner chose object storage alone, for now.
|
||||
|
||||
## What was built
|
||||
|
||||
- **`store.py`:** the store's rules (names, write-once, records, `latest`) over a `Blobs` layer with two kinds: `VolumeBlobs`, a folder, and `S3Blobs`, a prefix in an S3-compatible bucket, made if missing. The layout is the same in both: `tenants/<tenant>/cases/<case>/executions/<id>/{record.json, files/…}`. The record is written last, so an Execution is kept once its record is. A delivered file named `record.json` is not taken for one.
|
||||
- **Settings:** `STORE_BACKEND` (`volume` or `s3`), `S3_BUCKET`, `S3_PREFIX`, `S3_ENDPOINT_URL`, `S3_REGION`; credentials are the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `store_from_settings` makes the store.
|
||||
- **Packaging:** the `s3` extra (`boto3`); the image installs it. `moto` in the dev extra; mypy told the AWS SDK has no types.
|
||||
- **Local deployment:** the dev kit's MinIO (`agent-builder-minio`, started alone from the `observability` profile), bucket `orchestration`, its root credentials from the kit's `.env`.
|
||||
|
||||
## Verified
|
||||
|
||||
- Every store test runs on both kinds of storage, S3 through moto in the process; and the object layout under its prefix, a delivered `record.json`, and the settings choosing the storage.
|
||||
- 606 tests pass; `ruff` and `mypy` clean.
|
||||
- Deployed: healthy, with the `orchestration` bucket made and reachable on MinIO. No agent was run.
|
||||
|
||||
## Not done
|
||||
|
||||
- Retention, the platform's sign-in, and each kept file as a platform artifact (a proposal to the platform team) remain, not chosen for now.
|
||||
- Nothing kept on the volume before is moved: nothing had been kept.
|
||||
@@ -5,8 +5,8 @@
|
||||
#
|
||||
# Which agents it routes to is configuration, mounted at /app/agents.yaml
|
||||
# (AGENTS_FILE); deploy/agents.local.yaml is the local KAB platform's. What agents
|
||||
# deliver is kept in /data/store (ADR-0004); give it a volume, a PUBLIC_URL and a
|
||||
# LINK_SECRET.
|
||||
# deliver is kept in /data/store (ADR-0004), or in an S3-compatible bucket with
|
||||
# STORE_BACKEND=s3; give it a volume or a bucket, a PUBLIC_URL and a LINK_SECRET.
|
||||
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
@@ -14,7 +14,7 @@ WORKDIR /build
|
||||
COPY pyproject.toml ./
|
||||
COPY execution_engine/ ./execution_engine/
|
||||
COPY orchestration/ ./orchestration/
|
||||
RUN pip install --no-cache-dir --prefix=/install .
|
||||
RUN pip install --no-cache-dir --prefix=/install ".[s3]"
|
||||
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
@@ -158,6 +158,14 @@ def input_contract(
|
||||
"`files`: another agent's delivered work."
|
||||
),
|
||||
},
|
||||
"sources_from": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Work kept earlier to start from instead of its files: an "
|
||||
"earlier step's `record` or `latest` link, `latest/<work>`, "
|
||||
"or an Execution's id in this Customer case."
|
||||
),
|
||||
},
|
||||
"deliver": {
|
||||
"type": "string",
|
||||
"enum": ["links", "files"],
|
||||
|
||||
@@ -27,9 +27,16 @@ class OrchestrationSettings(BaseSettings):
|
||||
# agent's work takes.
|
||||
CONNECT_TIMEOUT_SECONDS: float = 10
|
||||
|
||||
# What agents deliver is kept here and served at signed links (ADR-0004).
|
||||
# Unset: nothing is kept, and a caller asking for links gets none.
|
||||
# What agents deliver is kept and served at signed links (ADR-0004): in a folder
|
||||
# (`volume`, STORE_DIR; unset, nothing is kept) or in an S3-compatible bucket
|
||||
# (`s3`). The bucket's credentials are the standard AWS_ACCESS_KEY_ID and
|
||||
# AWS_SECRET_ACCESS_KEY.
|
||||
STORE_BACKEND: str = "volume"
|
||||
STORE_DIR: str | None = None
|
||||
S3_BUCKET: str | None = None
|
||||
S3_PREFIX: str = ""
|
||||
S3_ENDPOINT_URL: str | None = None # MinIO, or another S3-compatible service
|
||||
S3_REGION: str = "us-east-1"
|
||||
# The base of every link: the address a browser reaches this service at, which
|
||||
# is not its address inside the platform's network. Unset: A2A_PUBLIC_URL.
|
||||
PUBLIC_URL: str | None = None
|
||||
|
||||
@@ -3,11 +3,12 @@ signer and the address a caller reaches them at, together."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
from urllib.parse import parse_qs, quote, unquote, urlsplit
|
||||
|
||||
from orchestration.links import Signer
|
||||
from orchestration.store import Store
|
||||
@@ -16,7 +17,11 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
LOCAL_TENANT = "local" # what is kept for a caller that names no tenant
|
||||
UNASSIGNED = "unassigned" # for a request that names no Customer case
|
||||
# The Orchestration engine's own record that an Execution completed, sent with its
|
||||
# files to the agent that builds on them.
|
||||
RECORD = "execution.json"
|
||||
_NOT_SLUG = re.compile(r"[^A-Za-z0-9._-]+")
|
||||
_KEPT_PATH = re.compile(r"^/tenants/([^/]+)/cases/([^/]+)/(executions|latest)/([^/]+)$")
|
||||
|
||||
|
||||
def _slug(value: Any, default: str) -> str:
|
||||
@@ -68,6 +73,76 @@ class Delivery:
|
||||
logger.exception("Could not keep %s Execution %s", work, execution_id)
|
||||
return None
|
||||
|
||||
# --- building on what was kept (phase 2) ---------------------------------------
|
||||
|
||||
def resolve(self, reference: str, fields: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""The kept Execution a request names in `sources_from`: a link from an
|
||||
earlier document, `latest/<work>`, or an Execution's id in the request's
|
||||
Customer case. Raises ValueError, saying why, for anything else."""
|
||||
tenant, case = self.place(fields)
|
||||
named_case = fields.get("customer_case")
|
||||
if reference.startswith(("http://", "https://", "/")):
|
||||
url = urlsplit(reference)
|
||||
query = parse_qs(url.query)
|
||||
exp, sig = (query.get(key, [None])[0] for key in ("exp", "sig"))
|
||||
if not self.signer.verify(url.path, exp, sig):
|
||||
raise ValueError("The link in sources_from is not valid.")
|
||||
match = _KEPT_PATH.match(url.path)
|
||||
if not match:
|
||||
raise ValueError("The link in sources_from is not to kept work.")
|
||||
tenant_of, case, kind, ident = match.groups()
|
||||
if fields.get("tenant_id") and tenant_of != tenant:
|
||||
raise ValueError("The link in sources_from is of another tenant.")
|
||||
tenant = tenant_of
|
||||
if named_case and _slug(named_case, UNASSIGNED) != case:
|
||||
raise ValueError(
|
||||
f"The work in sources_from is of the Customer case {case}, "
|
||||
f"not {named_case}."
|
||||
)
|
||||
ident = unquote(ident)
|
||||
record = (
|
||||
self.store.record(tenant, case, ident)
|
||||
if kind == "executions"
|
||||
else self.store.latest(tenant, case, ident)
|
||||
)
|
||||
elif reference.startswith("latest/"):
|
||||
work = reference.removeprefix("latest/")
|
||||
record = self.store.latest(tenant, case, work)
|
||||
if record is None:
|
||||
raise ValueError(
|
||||
f"No {work} work has completed in the Customer case {case}."
|
||||
)
|
||||
else:
|
||||
record = self.store.record(tenant, case, reference)
|
||||
if record is None:
|
||||
raise ValueError(
|
||||
f"Nothing is kept for {reference!r} in the Customer case {case}."
|
||||
)
|
||||
if record["status"] != "completed":
|
||||
raise ValueError(
|
||||
f"The {record['work']} work {record['execution_id']} ended "
|
||||
f"{record['status']}; only work that completed is built on."
|
||||
)
|
||||
return record
|
||||
|
||||
def sources(self, record: Mapping[str, Any]) -> dict[str, str]:
|
||||
"""A kept Execution as an agent's sources: its files, with the record that
|
||||
it completed."""
|
||||
files = self.store.files(
|
||||
record["tenant"], record["customer_case"], record["execution_id"]
|
||||
)
|
||||
completion = {
|
||||
"status": record["status"],
|
||||
"customer_case": record["customer_case"]
|
||||
if record["customer_case"] != UNASSIGNED
|
||||
else None,
|
||||
"execution_id": record["execution_id"],
|
||||
"workflow_request": record["request"],
|
||||
"work": record["work"],
|
||||
}
|
||||
files[RECORD] = json.dumps({k: v for k, v in completion.items() if v})
|
||||
return files
|
||||
|
||||
# --- addresses -------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -29,7 +29,7 @@ from orchestration.delivery import Delivery
|
||||
from orchestration.links import Signer
|
||||
from orchestration.relay import AgentClients, RelayExecutor
|
||||
from orchestration.serving import kept_routes
|
||||
from orchestration.store import Store
|
||||
from orchestration.store import Store, store_from_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -121,8 +121,8 @@ def _delivery(
|
||||
) -> Delivery | None:
|
||||
"""Where what agents deliver is kept, and how it is linked to; None when
|
||||
nothing is to be kept."""
|
||||
if store is None and settings.STORE_DIR:
|
||||
store = Store(settings.STORE_DIR)
|
||||
if store is None:
|
||||
store = store_from_settings(settings)
|
||||
if store is None:
|
||||
return None
|
||||
if signer is None:
|
||||
|
||||
@@ -44,7 +44,7 @@ from google.protobuf import json_format
|
||||
|
||||
from orchestration.agents import Agent
|
||||
from orchestration.agents import Sequence as AgentSequence
|
||||
from orchestration.delivery import Delivery
|
||||
from orchestration.delivery import RECORD, UNASSIGNED, Delivery
|
||||
from orchestration.routing import WORK, route
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -62,9 +62,8 @@ _PAUSES = {TaskState.TASK_STATE_INPUT_REQUIRED, TaskState.TASK_STATE_AUTH_REQUIR
|
||||
# What every agent of a sequence is sent of the caller's fields. The rest go to the
|
||||
# first agent only, or to the agent whose work names them: `operating_model: {...}`.
|
||||
_CARRIED = ("customer_case", "pre_approved", "request")
|
||||
# The Orchestration engine's own record that an agent's Execution completed, sent
|
||||
# with its files to the next agent.
|
||||
RECORD = "execution.json"
|
||||
# Where a request names work kept earlier to build on (ADR-0004, phase 2).
|
||||
SOURCES_FROM = "sources_from"
|
||||
# How a caller is given what agents delivered (ADR-0004): one document of links to
|
||||
# what was kept, or each file as the agent sent it.
|
||||
DELIVER, LINKS, FILES = "deliver", "links", "files"
|
||||
@@ -87,6 +86,8 @@ class Plan:
|
||||
deliver: str = FILES
|
||||
# Each agent that completed, by its work: its Execution, how it ended, links.
|
||||
results: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
# What was kept of each, by its work, for the agent after it to build on.
|
||||
kept: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
|
||||
def metadata(self, step: int, sources: Mapping[str, str] | None) -> dict[str, Any]:
|
||||
"""The metadata for the agent at `step`."""
|
||||
@@ -203,8 +204,36 @@ class RelayExecutor(AgentExecutor):
|
||||
work=str(options.get(WORK) or decision.agents[0].work),
|
||||
deliver=_deliver(options),
|
||||
)
|
||||
if SOURCES_FROM in fields:
|
||||
try:
|
||||
self._build_on(fields)
|
||||
except ValueError as exc:
|
||||
await updater.complete(
|
||||
updater.new_agent_message(
|
||||
[
|
||||
new_text_part(f"Nothing was started: {exc}"),
|
||||
new_data_part(
|
||||
{"status": "not_started", "reason": str(exc)}
|
||||
),
|
||||
]
|
||||
)
|
||||
)
|
||||
return
|
||||
await self._run(task_id, plan, 0, self._opening(plan, 0, None), updater)
|
||||
|
||||
def _build_on(self, fields: dict[str, Any]) -> None:
|
||||
"""Make the work a request names in `sources_from` its sources, and its
|
||||
Customer case the request's when it names none."""
|
||||
reference = str(fields.pop(SOURCES_FROM) or "")
|
||||
if self._delivery is None:
|
||||
raise ValueError("Nothing is kept here to build on.")
|
||||
if fields.get("sources"):
|
||||
raise ValueError("Give sources or sources_from, not both.")
|
||||
record = self._delivery.resolve(reference, fields)
|
||||
if not fields.get("customer_case") and record["customer_case"] != UNASSIGNED:
|
||||
fields["customer_case"] = record["customer_case"]
|
||||
fields["sources"] = self._delivery.sources(record)
|
||||
|
||||
async def _resume(
|
||||
self,
|
||||
task_id: str,
|
||||
@@ -381,6 +410,7 @@ class RelayExecutor(AgentExecutor):
|
||||
else None
|
||||
)
|
||||
if record is not None and self._delivery is not None:
|
||||
plan.kept[agent.work] = record
|
||||
result.update(self._delivery.links(record))
|
||||
else:
|
||||
result["files"] = {name: None for name in sorted(files)}
|
||||
@@ -429,17 +459,22 @@ class RelayExecutor(AgentExecutor):
|
||||
not_started,
|
||||
)
|
||||
return
|
||||
record = {
|
||||
"status": "completed",
|
||||
"customer_case": plan.fields.get("customer_case"),
|
||||
"execution_id": plan.results[agent.work]["execution_id"] or None,
|
||||
"workflow_request": plan.text,
|
||||
"work": agent.work,
|
||||
}
|
||||
sources = {
|
||||
**files,
|
||||
RECORD: json.dumps({k: v for k, v in record.items() if v is not None}),
|
||||
}
|
||||
kept = plan.kept.get(agent.work)
|
||||
if kept is not None and self._delivery is not None:
|
||||
# What was kept, as a request naming it in sources_from would have it.
|
||||
sources = self._delivery.sources(kept)
|
||||
else:
|
||||
record = {
|
||||
"status": "completed",
|
||||
"customer_case": plan.fields.get("customer_case"),
|
||||
"execution_id": plan.results[agent.work]["execution_id"] or None,
|
||||
"workflow_request": plan.text,
|
||||
"work": agent.work,
|
||||
}
|
||||
sources = {
|
||||
**files,
|
||||
RECORD: json.dumps({k: v for k, v in record.items() if v is not None}),
|
||||
}
|
||||
following = plan.steps[step + 1]
|
||||
await updater.update_status(
|
||||
TaskState.TASK_STATE_WORKING,
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
"""What agents delivered, kept by tenant, Customer case and Execution (ADR-0004).
|
||||
|
||||
<root>/tenants/<tenant>/cases/<case>/executions/<id>/record.json
|
||||
/files/<name as delivered>
|
||||
tenants/<tenant>/cases/<case>/executions/<id>/record.json
|
||||
/files/<name as delivered>
|
||||
|
||||
An Execution is written once. Its files are what the agent delivered, by the names
|
||||
it gave them; its record says what it was.
|
||||
it gave them; its record says what it was. The same layout is kept in a folder or
|
||||
under a prefix in an S3-compatible bucket.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -15,7 +16,10 @@ import re
|
||||
import time
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any, Protocol
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from orchestration.config import OrchestrationSettings
|
||||
|
||||
_SLUG = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
|
||||
_MEDIA_TYPES = {
|
||||
@@ -53,21 +57,128 @@ def media_type(name: str) -> str:
|
||||
return _MEDIA_TYPES.get(PurePosixPath(name).suffix.lower(), _TEXT)
|
||||
|
||||
|
||||
class Store:
|
||||
class Blobs(Protocol):
|
||||
"""Where a store's keys are kept: a folder, or a bucket."""
|
||||
|
||||
def put(self, key: str, data: bytes, content_type: str) -> None: ...
|
||||
def get(self, key: str) -> bytes | None: ...
|
||||
def exists(self, key: str) -> bool: ...
|
||||
def keys(self, prefix: str) -> list[str]: ...
|
||||
|
||||
|
||||
class VolumeBlobs:
|
||||
"""Keys as files under a folder."""
|
||||
|
||||
def __init__(self, root: str | Path) -> None:
|
||||
self.root = Path(root)
|
||||
|
||||
def _case(self, tenant: str, case: str) -> Path:
|
||||
return (
|
||||
self.root
|
||||
/ "tenants"
|
||||
/ slug(tenant, "Tenant")
|
||||
/ "cases"
|
||||
/ slug(case, "Case")
|
||||
def _path(self, key: str) -> Path:
|
||||
path = (self.root / key).resolve()
|
||||
if self.root.resolve() not in path.parents:
|
||||
raise ValueError(f"{key!r} is outside the store")
|
||||
return path
|
||||
|
||||
def put(self, key: str, data: bytes, content_type: str) -> None:
|
||||
path = self._path(key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
|
||||
def get(self, key: str) -> bytes | None:
|
||||
path = self._path(key)
|
||||
return path.read_bytes() if path.is_file() else None
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
return self._path(key).is_file()
|
||||
|
||||
def keys(self, prefix: str) -> list[str]:
|
||||
folder = self.root / prefix
|
||||
if not folder.is_dir():
|
||||
return []
|
||||
return [
|
||||
path.relative_to(self.root).as_posix()
|
||||
for path in folder.rglob("*")
|
||||
if path.is_file()
|
||||
]
|
||||
|
||||
|
||||
class S3Blobs:
|
||||
"""Keys as objects under a prefix in an S3-compatible bucket (MinIO locally)."""
|
||||
|
||||
def __init__(
|
||||
self, client: Any, bucket: str, prefix: str = "", create: bool = False
|
||||
) -> None:
|
||||
self._client = client
|
||||
self.bucket = bucket
|
||||
self.prefix = prefix.lstrip("/")
|
||||
if create:
|
||||
self._ensure_bucket()
|
||||
|
||||
def _ensure_bucket(self) -> None:
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
try:
|
||||
self._client.head_bucket(Bucket=self.bucket)
|
||||
except ClientError:
|
||||
self._client.create_bucket(Bucket=self.bucket)
|
||||
|
||||
def put(self, key: str, data: bytes, content_type: str) -> None:
|
||||
self._client.put_object(
|
||||
Bucket=self.bucket,
|
||||
Key=self.prefix + key,
|
||||
Body=data,
|
||||
ContentType=content_type,
|
||||
)
|
||||
|
||||
def _execution(self, tenant: str, case: str, execution_id: str) -> Path:
|
||||
return self._case(tenant, case) / "executions" / slug(execution_id, "Execution")
|
||||
def get(self, key: str) -> bytes | None:
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
try:
|
||||
response = self._client.get_object(
|
||||
Bucket=self.bucket, Key=self.prefix + key
|
||||
)
|
||||
except ClientError as exc:
|
||||
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
|
||||
return None
|
||||
raise
|
||||
data: bytes = response["Body"].read()
|
||||
return data
|
||||
|
||||
def exists(self, key: str) -> bool:
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
try:
|
||||
self._client.head_object(Bucket=self.bucket, Key=self.prefix + key)
|
||||
except ClientError as exc:
|
||||
if exc.response.get("Error", {}).get("Code") in {"NoSuchKey", "404"}:
|
||||
return False
|
||||
raise
|
||||
return True
|
||||
|
||||
def keys(self, prefix: str) -> list[str]:
|
||||
found: list[str] = []
|
||||
pages = self._client.get_paginator("list_objects_v2").paginate(
|
||||
Bucket=self.bucket, Prefix=self.prefix + prefix
|
||||
)
|
||||
for page in pages:
|
||||
for item in page.get("Contents", []):
|
||||
found.append(str(item["Key"])[len(self.prefix) :])
|
||||
return found
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, blobs: Blobs | str | Path) -> None:
|
||||
self.blobs: Blobs = (
|
||||
VolumeBlobs(blobs) if isinstance(blobs, (str, Path)) else blobs
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _case(tenant: str, case: str) -> str:
|
||||
return f"tenants/{slug(tenant, 'Tenant')}/cases/{slug(case, 'Case')}"
|
||||
|
||||
def _execution(self, tenant: str, case: str, execution_id: str) -> str:
|
||||
return (
|
||||
f"{self._case(tenant, case)}/executions/{slug(execution_id, 'Execution')}"
|
||||
)
|
||||
|
||||
def keep(
|
||||
self,
|
||||
@@ -86,12 +197,12 @@ class Store:
|
||||
already kept is refused."""
|
||||
folder = self._execution(tenant, case, execution_id)
|
||||
paths = {name: file_name(name) for name in files}
|
||||
if folder.exists():
|
||||
if self.blobs.exists(f"{folder}/record.json"):
|
||||
raise ValueError(f"Execution {execution_id} is already kept")
|
||||
for name, path in paths.items():
|
||||
target = folder / "files" / path
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(files[name], encoding="utf-8")
|
||||
self.blobs.put(
|
||||
f"{folder}/files/{path}", files[name].encode("utf-8"), media_type(name)
|
||||
)
|
||||
record = {
|
||||
"tenant": tenant,
|
||||
"customer_case": case,
|
||||
@@ -105,20 +216,26 @@ class Store:
|
||||
"kept_at": dt.datetime.now(dt.UTC).isoformat(),
|
||||
"kept_ns": time.time_ns(),
|
||||
}
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
(folder / "record.json").write_text(json.dumps(record, indent=2), "utf-8")
|
||||
# The record last: an Execution is kept once its record is.
|
||||
self.blobs.put(
|
||||
f"{folder}/record.json",
|
||||
json.dumps(record, indent=2).encode("utf-8"),
|
||||
"application/json",
|
||||
)
|
||||
return record
|
||||
|
||||
def record(
|
||||
self, tenant: str, case: str, execution_id: str
|
||||
) -> dict[str, Any] | None:
|
||||
try:
|
||||
path = self._execution(tenant, case, execution_id) / "record.json"
|
||||
data = self.blobs.get(
|
||||
f"{self._execution(tenant, case, execution_id)}/record.json"
|
||||
)
|
||||
except ValueError:
|
||||
return None
|
||||
if not path.is_file():
|
||||
if data is None:
|
||||
return None
|
||||
record: dict[str, Any] = json.loads(path.read_text("utf-8"))
|
||||
record: dict[str, Any] = json.loads(data)
|
||||
return record
|
||||
|
||||
def file(
|
||||
@@ -126,23 +243,38 @@ class Store:
|
||||
) -> tuple[bytes, str] | None:
|
||||
"""A kept file's content and media type, or None."""
|
||||
try:
|
||||
folder = self._execution(tenant, case, execution_id) / "files"
|
||||
target = folder / file_name(name)
|
||||
key = (
|
||||
f"{self._execution(tenant, case, execution_id)}/files/{file_name(name)}"
|
||||
)
|
||||
data = self.blobs.get(key)
|
||||
except ValueError:
|
||||
return None
|
||||
if not target.is_file() or folder.resolve() not in target.resolve().parents:
|
||||
return None
|
||||
return target.read_bytes(), media_type(name)
|
||||
return None if data is None else (data, media_type(name))
|
||||
|
||||
def files(self, tenant: str, case: str, execution_id: str) -> dict[str, str]:
|
||||
"""Every file kept for an Execution, by name."""
|
||||
record = self.record(tenant, case, execution_id) or {}
|
||||
found: dict[str, str] = {}
|
||||
for name in record.get("files", []):
|
||||
kept = self.file(tenant, case, execution_id, name)
|
||||
if kept is not None:
|
||||
found[name] = kept[0].decode("utf-8")
|
||||
return found
|
||||
|
||||
def case(self, tenant: str, case: str) -> list[dict[str, Any]]:
|
||||
"""The Customer case's Executions, newest first."""
|
||||
try:
|
||||
folder = self._case(tenant, case) / "executions"
|
||||
prefix = f"{self._case(tenant, case)}/executions/"
|
||||
except ValueError:
|
||||
return []
|
||||
records = [
|
||||
json.loads(path.read_text("utf-8")) for path in folder.glob("*/record.json")
|
||||
]
|
||||
records = []
|
||||
for key in self.blobs.keys(prefix):
|
||||
# <id>/record.json; a delivered file of that name is under files/.
|
||||
parts = key[len(prefix) :].split("/")
|
||||
if len(parts) == 2 and parts[1] == "record.json":
|
||||
data = self.blobs.get(key)
|
||||
if data is not None:
|
||||
records.append(json.loads(data))
|
||||
return sorted(records, key=lambda r: r.get("kept_ns", 0), reverse=True)
|
||||
|
||||
def latest(self, tenant: str, case: str, work: str) -> dict[str, Any] | None:
|
||||
@@ -155,3 +287,25 @@ class Store:
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def store_from_settings(settings: OrchestrationSettings) -> Store | None:
|
||||
"""The store the settings name; None when nothing is to be kept."""
|
||||
if settings.STORE_BACKEND == "s3":
|
||||
import boto3
|
||||
|
||||
if not settings.S3_BUCKET:
|
||||
raise ValueError("STORE_BACKEND=s3 needs S3_BUCKET")
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=settings.S3_ENDPOINT_URL or None,
|
||||
region_name=settings.S3_REGION,
|
||||
)
|
||||
return Store(
|
||||
S3Blobs(client, settings.S3_BUCKET, settings.S3_PREFIX, create=True)
|
||||
)
|
||||
if settings.STORE_BACKEND != "volume":
|
||||
raise ValueError(
|
||||
f"STORE_BACKEND must be volume or s3, not {settings.STORE_BACKEND!r}"
|
||||
)
|
||||
return Store(VolumeBlobs(settings.STORE_DIR)) if settings.STORE_DIR else None
|
||||
|
||||
@@ -30,7 +30,11 @@ dependencies = [
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
# The Orchestration engine's store on S3-compatible object storage (ADR-0004).
|
||||
s3 = ["boto3>=1.34"]
|
||||
dev = [
|
||||
"boto3>=1.34",
|
||||
"moto[s3]>=5.0",
|
||||
"pytest>=8.0.0",
|
||||
"pytest-asyncio>=0.23.0",
|
||||
"ruff>=0.6.0",
|
||||
@@ -59,3 +63,8 @@ cache-dir = ".cache/ruff"
|
||||
|
||||
[tool.mypy]
|
||||
cache_dir = ".cache/mypy"
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
# The AWS SDK and its test double ship without type information.
|
||||
module = ["boto3", "boto3.*", "botocore", "botocore.*", "moto", "moto.*"]
|
||||
ignore_missing_imports = true
|
||||
|
||||
258
tests/orchestration/test_sources_from.py
Normal file
258
tests/orchestration/test_sources_from.py
Normal file
@@ -0,0 +1,258 @@
|
||||
"""A request names work kept earlier instead of carrying its files (ADR-0004,
|
||||
phase 2): `sources_from` is a link from an earlier document, `latest/<work>`, or
|
||||
an Execution's id in the request's Customer case."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from tests.orchestration.test_links import PLATFORM, Harness, document
|
||||
from tests.orchestration.test_sequence import ScriptedAgents, finished, struct
|
||||
|
||||
GRAPH = '{"elements": []}'
|
||||
|
||||
|
||||
def designed(**more: Any) -> ScriptedAgents:
|
||||
"""An architecture agent that has designed once, and an operations agent ready."""
|
||||
turns: dict[str, list[Any]] = {
|
||||
"architecture": [finished("a1", {"graph.json": GRAPH, "guide.md": "# Guide"})],
|
||||
"operations": [finished("o1", {"model.json": "{}"})],
|
||||
}
|
||||
turns.update(more)
|
||||
return ScriptedAgents(turns)
|
||||
|
||||
|
||||
async def design(harness: Harness, case: str = "acme") -> dict[str, Any]:
|
||||
return document(
|
||||
await harness.send(
|
||||
"A landing zone",
|
||||
{**PLATFORM, "work": "architecture", "customer_case": case},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def not_started(task: dict[str, Any]) -> str:
|
||||
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||
parts = task["status"]["message"]["parts"]
|
||||
data = next(p["data"] for p in parts if "data" in p)
|
||||
assert data["status"] == "not_started"
|
||||
return " ".join(p.get("text", "") for p in parts)
|
||||
|
||||
|
||||
def sent_sources(harness: Harness) -> dict[str, Any]:
|
||||
(message,) = harness.agents.received["operations"]
|
||||
metadata = struct(message)
|
||||
assert "sources_from" not in metadata
|
||||
return metadata
|
||||
|
||||
|
||||
async def test_the_latest_work_of_the_case_is_the_agents_sources(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
await design(harness)
|
||||
|
||||
await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "acme",
|
||||
"sources_from": "latest/architecture",
|
||||
},
|
||||
)
|
||||
|
||||
sent = sent_sources(harness)
|
||||
assert sent["sources"]["graph.json"] == GRAPH
|
||||
assert sent["sources"]["guide.md"] == "# Guide"
|
||||
assert json.loads(sent["sources"]["execution.json"]) == {
|
||||
"status": "completed",
|
||||
"customer_case": "acme",
|
||||
"execution_id": "a1",
|
||||
"workflow_request": "A landing zone",
|
||||
"work": "architecture",
|
||||
}
|
||||
|
||||
|
||||
async def test_a_link_from_an_earlier_document_names_the_work_and_its_case(tmp_path):
|
||||
# A platform step maps an earlier step's `record` into `sources_from`.
|
||||
harness = Harness(tmp_path, designed())
|
||||
earlier = await design(harness)
|
||||
|
||||
await harness.send(
|
||||
"Operate it",
|
||||
{**PLATFORM, "work": "operations", "sources_from": earlier["record"]},
|
||||
)
|
||||
|
||||
sent = sent_sources(harness)
|
||||
assert sent["customer_case"] == "acme"
|
||||
assert sent["sources"]["graph.json"] == GRAPH
|
||||
|
||||
|
||||
async def test_a_latest_link_is_followed_to_the_newest_completed_work(tmp_path):
|
||||
harness = Harness(
|
||||
tmp_path,
|
||||
designed(
|
||||
architecture=[
|
||||
finished("a1", {"graph.json": "1"}),
|
||||
finished("a2", {"graph.json": "2"}),
|
||||
]
|
||||
),
|
||||
)
|
||||
first = await design(harness)
|
||||
await design(harness)
|
||||
|
||||
await harness.send(
|
||||
"Operate it",
|
||||
{**PLATFORM, "work": "operations", "sources_from": first["latest"]},
|
||||
)
|
||||
|
||||
assert sent_sources(harness)["sources"]["graph.json"] == "2"
|
||||
|
||||
|
||||
async def test_an_execution_of_the_case_is_named_by_its_id(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
await design(harness)
|
||||
|
||||
await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "acme",
|
||||
"sources_from": "a1",
|
||||
},
|
||||
)
|
||||
|
||||
assert sent_sources(harness)["sources"]["graph.json"] == GRAPH
|
||||
|
||||
|
||||
async def test_a_link_that_was_changed_starts_nothing(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
earlier = await design(harness)
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"sources_from": earlier["record"].replace("/a1?", "/a2?"),
|
||||
},
|
||||
)
|
||||
|
||||
assert "not valid" in not_started(task)
|
||||
assert harness.agents.received["operations"] == []
|
||||
|
||||
|
||||
async def test_a_link_of_another_tenant_starts_nothing(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
earlier = await design(harness)
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"tenant_id": "t2",
|
||||
"work": "operations",
|
||||
"sources_from": earlier["record"],
|
||||
},
|
||||
)
|
||||
|
||||
assert "tenant" in not_started(task)
|
||||
assert harness.agents.received["operations"] == []
|
||||
|
||||
|
||||
async def test_work_of_another_customer_case_starts_nothing(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
earlier = await design(harness)
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "other",
|
||||
"sources_from": earlier["record"],
|
||||
},
|
||||
)
|
||||
|
||||
assert "acme" in not_started(task)
|
||||
|
||||
|
||||
async def test_nothing_kept_to_build_on_starts_nothing(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "acme",
|
||||
"sources_from": "latest/architecture",
|
||||
},
|
||||
)
|
||||
|
||||
assert "architecture" in not_started(task)
|
||||
assert harness.agents.received["operations"] == []
|
||||
|
||||
|
||||
async def test_work_that_ended_with_warnings_is_not_built_on(tmp_path):
|
||||
harness = Harness(
|
||||
tmp_path,
|
||||
designed(
|
||||
architecture=[
|
||||
finished("a1", {"graph.json": GRAPH}, status="completed_with_warnings")
|
||||
]
|
||||
),
|
||||
)
|
||||
await design(harness)
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "acme",
|
||||
"sources_from": "a1",
|
||||
},
|
||||
)
|
||||
|
||||
assert "completed_with_warnings" in not_started(task)
|
||||
|
||||
|
||||
async def test_sources_and_sources_from_together_start_nothing(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
await design(harness)
|
||||
|
||||
task = await harness.send(
|
||||
"Operate it",
|
||||
{
|
||||
**PLATFORM,
|
||||
"work": "operations",
|
||||
"customer_case": "acme",
|
||||
"sources_from": "a1",
|
||||
"sources": {"x.json": "{}"},
|
||||
},
|
||||
)
|
||||
|
||||
assert "sources" in not_started(task)
|
||||
|
||||
|
||||
async def test_a_sequence_gives_the_next_agent_what_was_kept(tmp_path):
|
||||
harness = Harness(tmp_path, designed())
|
||||
|
||||
await harness.send(
|
||||
"A landing zone",
|
||||
{**PLATFORM, "work": "architecture_and_operations", "customer_case": "acme"},
|
||||
)
|
||||
|
||||
sources = sent_sources(harness)["sources"]
|
||||
assert sources["graph.json"] == GRAPH
|
||||
assert json.loads(sources["execution.json"])["execution_id"] == "a1"
|
||||
|
||||
|
||||
async def test_the_card_offers_sources_from(tmp_path):
|
||||
harness = Harness(tmp_path, ScriptedAgents({}))
|
||||
async with harness.client() as caller:
|
||||
skill = (await caller.get("/.well-known/agent-card.json")).json()["skills"][0]
|
||||
|
||||
assert skill["inputSchema"]["properties"]["sources_from"]["type"] == "string"
|
||||
@@ -2,10 +2,30 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import boto3
|
||||
import pytest
|
||||
from moto import mock_aws
|
||||
|
||||
from orchestration.config import OrchestrationSettings
|
||||
from orchestration.links import Signer
|
||||
from orchestration.store import Store
|
||||
from orchestration.store import S3Blobs, Store, VolumeBlobs, store_from_settings
|
||||
|
||||
|
||||
@pytest.fixture(params=["volume", "s3"])
|
||||
def make_store(request: Any, tmp_path: Any) -> Iterator[Any]:
|
||||
"""A store on each kind of storage: a folder, and S3 (moto in this process)."""
|
||||
if request.param == "volume":
|
||||
yield lambda: Store(VolumeBlobs(tmp_path))
|
||||
return
|
||||
with mock_aws():
|
||||
client = boto3.client("s3", region_name="us-east-1")
|
||||
yield lambda: Store(
|
||||
S3Blobs(client, "kept", prefix="orchestration/", create=True)
|
||||
)
|
||||
|
||||
|
||||
FILES = {
|
||||
"guide.md": "# Guide",
|
||||
@@ -30,8 +50,8 @@ def kept(store: Store, execution_id: str = "a1", **overrides: object) -> dict:
|
||||
return store.keep(**values)
|
||||
|
||||
|
||||
def test_an_execution_is_kept_with_its_record(tmp_path):
|
||||
store = Store(tmp_path)
|
||||
def test_an_execution_is_kept_with_its_record(make_store):
|
||||
store = make_store()
|
||||
|
||||
record = kept(store)
|
||||
|
||||
@@ -43,8 +63,8 @@ def test_an_execution_is_kept_with_its_record(tmp_path):
|
||||
assert store.record("t1", "acme", "a1") == record
|
||||
|
||||
|
||||
def test_a_file_comes_back_with_its_media_type(tmp_path):
|
||||
store = Store(tmp_path)
|
||||
def test_a_file_comes_back_with_its_media_type(make_store):
|
||||
store = make_store()
|
||||
kept(store)
|
||||
|
||||
assert store.file("t1", "acme", "a1", "guide.md") == (
|
||||
@@ -59,8 +79,8 @@ def test_a_file_comes_back_with_its_media_type(tmp_path):
|
||||
assert store.file("t1", "acme", "a1", "missing.md") is None
|
||||
|
||||
|
||||
def test_what_is_kept_is_never_changed(tmp_path):
|
||||
store = Store(tmp_path)
|
||||
def test_what_is_kept_is_never_changed(make_store):
|
||||
store = make_store()
|
||||
kept(store)
|
||||
|
||||
with pytest.raises(ValueError, match="a1"):
|
||||
@@ -70,8 +90,8 @@ def test_what_is_kept_is_never_changed(tmp_path):
|
||||
@pytest.mark.parametrize(
|
||||
"name", ["../escape.md", "/etc/passwd", "a/../../b", "a\\b", ""]
|
||||
)
|
||||
def test_a_file_name_outside_the_execution_is_refused(tmp_path, name):
|
||||
store = Store(tmp_path)
|
||||
def test_a_file_name_outside_the_execution_is_refused(make_store, name):
|
||||
store = make_store()
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
kept(store, files={name: "x"})
|
||||
@@ -79,13 +99,13 @@ def test_a_file_name_outside_the_execution_is_refused(tmp_path, name):
|
||||
|
||||
|
||||
@pytest.mark.parametrize("slug", ["..", "a/b", "", "a b"])
|
||||
def test_a_tenant_or_case_that_is_not_a_slug_is_refused(tmp_path, slug):
|
||||
def test_a_tenant_or_case_that_is_not_a_slug_is_refused(make_store, slug):
|
||||
with pytest.raises(ValueError):
|
||||
kept(Store(tmp_path), case=slug)
|
||||
kept(make_store(), case=slug)
|
||||
|
||||
|
||||
def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path):
|
||||
store = Store(tmp_path)
|
||||
def test_latest_is_the_newest_completed_execution_of_the_work(make_store):
|
||||
store = make_store()
|
||||
kept(store, "a1")
|
||||
kept(store, "a2")
|
||||
kept(store, "a3", status="completed_with_warnings")
|
||||
@@ -101,6 +121,53 @@ def test_latest_is_the_newest_completed_execution_of_the_work(tmp_path):
|
||||
]
|
||||
|
||||
|
||||
def test_a_delivered_file_named_like_a_record_is_not_taken_for_one(make_store):
|
||||
store = make_store()
|
||||
kept(store, files={"record.json": '{"not": "a record"}'})
|
||||
|
||||
assert [r["execution_id"] for r in store.case("t1", "acme")] == ["a1"]
|
||||
|
||||
|
||||
def test_object_storage_keeps_the_same_layout_under_its_prefix():
|
||||
with mock_aws():
|
||||
client = boto3.client("s3", region_name="us-east-1")
|
||||
kept(Store(S3Blobs(client, "kept", prefix="orchestration/", create=True)))
|
||||
|
||||
keys = sorted(
|
||||
item["Key"] for item in client.list_objects_v2(Bucket="kept")["Contents"]
|
||||
)
|
||||
|
||||
base = "orchestration/tenants/t1/cases/acme/executions/a1"
|
||||
assert keys == [
|
||||
f"{base}/files/graph.json",
|
||||
f"{base}/files/guide.md",
|
||||
f"{base}/files/terraform/main.tf",
|
||||
f"{base}/record.json",
|
||||
]
|
||||
|
||||
|
||||
def test_the_settings_choose_the_storage(tmp_path, monkeypatch):
|
||||
assert store_from_settings(OrchestrationSettings(_env_file=None)) is None # type: ignore[call-arg]
|
||||
volume = store_from_settings(
|
||||
OrchestrationSettings(_env_file=None, STORE_DIR=str(tmp_path)) # type: ignore[call-arg]
|
||||
)
|
||||
assert isinstance(volume, Store) and isinstance(volume.blobs, VolumeBlobs)
|
||||
monkeypatch.setenv("AWS_ACCESS_KEY_ID", "test")
|
||||
monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "test")
|
||||
with mock_aws():
|
||||
s3 = store_from_settings(
|
||||
OrchestrationSettings( # type: ignore[call-arg]
|
||||
_env_file=None,
|
||||
STORE_BACKEND="s3",
|
||||
S3_BUCKET="kept",
|
||||
S3_REGION="us-east-1",
|
||||
)
|
||||
)
|
||||
assert isinstance(s3, Store) and isinstance(s3.blobs, S3Blobs)
|
||||
kept(s3)
|
||||
assert s3.record("t1", "acme", "a1") is not None
|
||||
|
||||
|
||||
def test_a_signed_path_is_accepted_and_a_changed_one_is_not():
|
||||
signer = Signer("secret", ttl_seconds=60, now=lambda: 1000)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user