This commit is contained in:
84
README.md
84
README.md
@@ -1,14 +1,82 @@
|
||||
# KAB Content Ingestion Agent
|
||||
|
||||
Contract-first ingestion for GitHub and SharePoint into a governed KAB-compatible knowledge store. The package is dependency-light and uses injected HTTP, secret, clock, and publisher ports for deterministic tests.
|
||||
Contract-first, multi-tenant ingestion for GitHub REST and Microsoft Graph SharePoint into a governed KAB-compatible knowledge store.
|
||||
|
||||
## Quick start
|
||||
```bash
|
||||
python -m pytest -q
|
||||
python -m compileall kab_ingestion
|
||||
The package is dependency-light, utilizing hexagonal architecture ports (HTTP, SecretStore, Publisher) for deterministic testing and zero-token leakage.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Codebase Alignment
|
||||
|
||||
The codebase is organized into two primary layers with a unified domain alignment:
|
||||
|
||||
```
|
||||
+-----------------------------------+
|
||||
| app/ |
|
||||
| (Starlette/FastAPI Web Layer & |
|
||||
| LangGraph Execution Workflows) |
|
||||
+-----------------+-----------------+
|
||||
|
|
||||
Imports & Delegates
|
||||
v
|
||||
+-----------------------------------+
|
||||
| kab_ingestion/ |
|
||||
| (Core Domain Models, Connectors, |
|
||||
| Orchestrator & Governance) |
|
||||
+-----------------------------------+
|
||||
```
|
||||
|
||||
See `docs/contract.md`, `docs/operations.md`, and `examples/config.yaml`.
|
||||
### 1. Core Ingestion Domain Layer (`kab_ingestion/`)
|
||||
- **[kab_ingestion/models.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/models.py)**: Canonical domain models (`Document`, `ACL`, `Provenance`, `SyncCursor`, `IngestionRequest`, `IngestionResult`).
|
||||
- **[kab_ingestion/github.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/github.py)** & **[kab_ingestion/sharepoint.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/sharepoint.py)**: Source connectors supporting full tree syncs, incremental delta syncs, and webhook event parsing.
|
||||
- **[kab_ingestion/orchestrator.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/orchestrator.py)**: Fault-tolerant batch sync engine with bounded retries (`max_retries=3`) and dead-lettering.
|
||||
- **[kab_ingestion/governance.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/governance.py)**: Strict multi-tenant isolation and ACL validation.
|
||||
|
||||
## Security
|
||||
Tokens are referenced by secret name, never stored in configuration or logs. Every document carries tenant, source, and ACL metadata; publishers must enforce the same filters at query time.
|
||||
### 2. Application & Workflow Layer (`app/`)
|
||||
- **[app/agent.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/agent.py)**: Service factory `create_agent_app()` and `IngestionService`.
|
||||
- **[app/card.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/card.py)**: Agent discovery card (`AGENT_CARD`).
|
||||
- **[app/workflows/ingestion_workflow.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/workflows/ingestion_workflow.py)**: 4-Phase LangGraph workflow runner.
|
||||
- **[app/routes.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/routes.py)**: HTTP REST routes (`/health`, `/card`, `/ingest`).
|
||||
|
||||
### 3. Package Re-export Alignment (`app/kab_ingestion`)
|
||||
- **`app/kab_ingestion`**: Operates as a lightweight re-export alignment layer. All domain models, connectors, and ports re-export directly from the canonical `kab_ingestion` root package, guaranteeing full backward compatibility and preventing duplicate definitions.
|
||||
|
||||
---
|
||||
|
||||
## LangGraph 4-Phase Execution Contract
|
||||
|
||||
Every ingestion workflow implements the platform's 4-phase execution contract:
|
||||
|
||||
1. **Discovery** (`app/nodes/discovery_node.py`): Non-blocking check for existing resources and connector availability.
|
||||
2. **Validation** (`app/nodes/validation_node.py`): Validates `IngestionRequest` format and allowed sources; fails fast on error (`has_errors: True` → `END`).
|
||||
3. **Generation** (`app/nodes/generation_node.py`): Connects to GitHub/SharePoint source connectors and fetches normalized documents into `fetched_documents`.
|
||||
4. **Deployment** (`app/nodes/deployment_node.py`): Enforces governance approval and tenant ACL isolation, publishing approved documents to target storage.
|
||||
|
||||
---
|
||||
|
||||
## Security & Tenant Isolation
|
||||
|
||||
- **Secret References (`secret_ref`)**: Tokens are referenced by secret name (e.g. `secrets/github-read`), never stored in configuration files or logs.
|
||||
- **Tenant Isolation & ACLs**: Every document carries `tenant_id` and `ACL {principals, groups, visibility}`. Tenant mismatch results in automatic publish rejection.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start & Testing
|
||||
|
||||
### Running Tests
|
||||
```bash
|
||||
python -m pytest -q
|
||||
```
|
||||
|
||||
### Compiling Packages
|
||||
```bash
|
||||
python -m compileall kab_ingestion app
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation References
|
||||
|
||||
- **Contract Specification**: [spec/ingestion-contract.md](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/spec/ingestion-contract.md)
|
||||
- **Configuration Example**: [examples/config.yaml](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/examples/config.yaml)
|
||||
- **Kubernetes Manifests**: [k8s/](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/k8s)
|
||||
|
||||
Reference in New Issue
Block a user