83 lines
5.2 KiB
Markdown
83 lines
5.2 KiB
Markdown
# KAB Content Ingestion Agent
|
|
|
|
Contract-first, multi-tenant ingestion for GitHub REST and Microsoft Graph SharePoint into a governed KAB-compatible knowledge store.
|
|
|
|
The package is dependency-light, utilizing hexagonal architecture ports (HTTP, SecretStore, Publisher) for deterministic testing and zero-token leakage.
|
|
|
|
---
|
|
|
|
## Architecture & Codebase Alignment
|
|
|
|
The codebase is organized into two primary layers with a unified domain alignment:
|
|
|
|
```
|
|
+-----------------------------------+
|
|
| app/ |
|
|
| (Starlette/FastAPI Web Layer & |
|
|
| LangGraph Execution Workflows) |
|
|
+-----------------+-----------------+
|
|
|
|
|
Imports & Delegates
|
|
v
|
|
+-----------------------------------+
|
|
| kab_ingestion/ |
|
|
| (Core Domain Models, Connectors, |
|
|
| Orchestrator & Governance) |
|
|
+-----------------------------------+
|
|
```
|
|
|
|
### 1. Core Ingestion Domain Layer (`kab_ingestion/`)
|
|
- **[kab_ingestion/models.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/models.py)**: Canonical domain models (`Document`, `ACL`, `Provenance`, `SyncCursor`, `IngestionRequest`, `IngestionResult`).
|
|
- **[kab_ingestion/github.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/github.py)** & **[kab_ingestion/sharepoint.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/sharepoint.py)**: Source connectors supporting full tree syncs, incremental delta syncs, and webhook event parsing.
|
|
- **[kab_ingestion/orchestrator.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/orchestrator.py)**: Fault-tolerant batch sync engine with bounded retries (`max_retries=3`) and dead-lettering.
|
|
- **[kab_ingestion/governance.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/kab_ingestion/governance.py)**: Strict multi-tenant isolation and ACL validation.
|
|
|
|
### 2. Application & Workflow Layer (`app/`)
|
|
- **[app/agent.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/agent.py)**: Service factory `create_agent_app()` and `IngestionService`.
|
|
- **[app/card.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/card.py)**: Agent discovery card (`AGENT_CARD`).
|
|
- **[app/workflows/ingestion_workflow.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/workflows/ingestion_workflow.py)**: 4-Phase LangGraph workflow runner.
|
|
- **[app/routes.py](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/app/routes.py)**: HTTP REST routes (`/health`, `/card`, `/ingest`).
|
|
|
|
### 3. Package Re-export Alignment (`app/kab_ingestion`)
|
|
- **`app/kab_ingestion`**: Operates as a lightweight re-export alignment layer. All domain models, connectors, and ports re-export directly from the canonical `kab_ingestion` root package, guaranteeing full backward compatibility and preventing duplicate definitions.
|
|
|
|
---
|
|
|
|
## LangGraph 4-Phase Execution Contract
|
|
|
|
Every ingestion workflow implements the platform's 4-phase execution contract:
|
|
|
|
1. **Discovery** (`app/nodes/discovery_node.py`): Non-blocking check for existing resources and connector availability.
|
|
2. **Validation** (`app/nodes/validation_node.py`): Validates `IngestionRequest` format and allowed sources; fails fast on error (`has_errors: True` → `END`).
|
|
3. **Generation** (`app/nodes/generation_node.py`): Connects to GitHub/SharePoint source connectors and fetches normalized documents into `fetched_documents`.
|
|
4. **Deployment** (`app/nodes/deployment_node.py`): Enforces governance approval and tenant ACL isolation, publishing approved documents to target storage.
|
|
|
|
---
|
|
|
|
## Security & Tenant Isolation
|
|
|
|
- **Secret References (`secret_ref`)**: Tokens are referenced by secret name (e.g. `secrets/github-read`), never stored in configuration files or logs.
|
|
- **Tenant Isolation & ACLs**: Every document carries `tenant_id` and `ACL {principals, groups, visibility}`. Tenant mismatch results in automatic publish rejection.
|
|
|
|
---
|
|
|
|
## Quick Start & Testing
|
|
|
|
### Running Tests
|
|
```bash
|
|
python -m pytest -q
|
|
```
|
|
|
|
### Compiling Packages
|
|
```bash
|
|
python -m compileall kab_ingestion app
|
|
```
|
|
|
|
---
|
|
|
|
## Documentation References
|
|
|
|
- **Contract Specification**: [spec/ingestion-contract.md](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/spec/ingestion-contract.md)
|
|
- **Configuration Example**: [examples/config.yaml](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/examples/config.yaml)
|
|
- **Kubernetes Manifests**: [k8s/](file:///home/jonathanboniface/platform-engineering/kyndryl-agent-builder/content-ingestion-agent/k8s)
|