decomposer: generate deliverable files for Define the normalized ingestion contract and pluggable source-connector interface for SCM, CMS, and other content sources.; Implement a GitHub SCM connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of Markdown, plain-text, and source files with webhook and revision metadata.; Implement a SharePoint CMS connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of PDF, DOCX, and HTML files with webhook and revision metadata.; Implement ingestion orchestration and triggers; Normalize, govern, and publish ingested content to the shared knowledge store.; Add deployment/configuration, tests, and documentation for the ingestion agent.; Validate end-to-end ingestion and downstream content availability.

.github/workflows/ci.yml

@@ -4,10 +4,9 @@ jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+    - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+    - uses: actions/setup-python@v5
-        with: {python-version: '3.11'}
+      with: {python-version: '3.11'}
-      - run: pip install -e '.[dev]'
+    - run: python -m pip install -e '.[test]'
-      - run: ruff check .
+    - run: python -m compileall kab_ingestion
-      - run: mypy kab_ingestion
+    - run: python -m pytest -q
-      - run: pytest -q

Dockerfile

@@ -1 +1,5 @@
-# TODO: generation subagent fills this in.
+FROM python:3.11-slim
+WORKDIR /app
+COPY . .
+RUN pip install --no-cache-dir .
+CMD ["python", "-m", "kab_ingestion"]

LICENSE

@@ -0,0 +1,5 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction...

README.md

@@ -1,15 +1,14 @@
 # KAB Content Ingestion Agent
 
-A contract-first, tenant-isolated ingestion service for GitHub and SharePoint. It supports full, incremental, scheduled, on-demand, and webhook-triggered synchronization and publishes governed normalized documents to a KAB-compatible store.
+Contract-first ingestion for GitHub and SharePoint into a governed KAB-compatible knowledge store. The package is dependency-light and uses injected HTTP, secret, clock, and publisher ports for deterministic tests.
 
 ## Quick start
-
 ```bash
-python -m venv .venv && . .venv/bin/activate
+python -m pytest -q
-pip install -e '.[dev]'
+python -m compileall kab_ingestion
-pytest -q
 ```
 
-The implementation uses dependency-injected HTTP, secret, cursor, and publication ports; production adapters can be supplied without changing connector logic. See `docs/operations.md`, `docs/examples.md`, and `spec/ingestion-contract.md`.
+See `docs/contract.md`, `docs/operations.md`, and `examples/config.yaml`.
 
-Security defaults: tenant is mandatory, ACLs are deny-by-default, secrets are referenced rather than stored, and webhook signatures are verified before work is queued.
+## Security
+Tokens are referenced by secret name, never stored in configuration or logs. Every document carries tenant, source, and ACL metadata; publishers must enforce the same filters at query time.

deploy/kubernetes.yaml

@@ -8,10 +8,8 @@ spec:
     metadata: {labels: {app: kab-ingestion}}
     spec:
       containers:
-        - name: ingestion
+      - name: agent
-          image: registry.example/kab-ingestion:${GIT_SHA}
+        image: ghcr.io/example/kab-ingestion:0.1.0
-          env:
+        env:
-            - name: CONFIG_PATH
+        - {name: SECRET_PROVIDER, value: managed}
-              value: /etc/kab/config.yaml
+        readinessProbe: {httpGet: {path: /healthz, port: 8080}}
-          readinessProbe: {httpGet: {path: /health, port: 8080}}
-          securityContext: {allowPrivilegeEscalation: false, readOnlyRootFilesystem: true}

docs/contract.md

@@ -0,0 +1,16 @@
+# KAB ingestion contract (v1)
+
+## Connector interface
+`full(config) -> (Document[], SyncCursor)`, `incremental(config, cursor, changes?) -> (Document[], SyncCursor)`, and `webhook(config, payload, headers) -> Change[]`. Implementations must be deterministic for the same source revision, scope, and credentials. Config has `connector_id`, `tenant_id`, secret reference, source scope, and non-secret options.
+
+## Normalized document
+Required fields: stable `id`, `tenant_id`, title, UTF-8 `content` (or extractor output), MIME type, SHA-256 `content_hash`, modified timestamp, `provenance`, `acl`, metadata, and deletion marker. Binary extraction may be delegated to a managed extractor; the connector still supplies original MIME and hash.
+
+## Provenance and ACL
+Provenance includes source type/id, canonical URL, revision/eTag, retrieval time, webhook event, and connector version. ACL includes tenant, principals, groups, and visibility. Tenant equality is mandatory; downstream search must apply ACL predicates.
+
+## Cursor/triggers
+Cursors contain connector id, mode, source revision or delta token, and update time. Full runs establish a baseline. Incremental runs advance the cursor only after publication succeeds. Triggers are `on_demand`, cron `scheduled`, or signed `webhook`; each carries `full|incremental`, idempotency key, retry policy, and dead-letter destination.
+
+## Governance
+Upserts are idempotent by document id/hash. Deletes and retention events are auditable, tenant-scoped, and must use managed secret references. No token or document content is logged.

docs/operations.md

@@ -1,10 +1,5 @@
 # Operations
 
-## Deployment and monitoring
+Configure a secret manager reference (`secret_ref`), least-privilege GitHub token or Graph application permission, tenant and source scopes. Schedule full baselines and incremental delta runs. Webhook handlers must validate provider signatures at the edge, map event IDs to idempotency keys, and enqueue work.
-Run the ASGI/queue adapter in the platform runtime with one worker per queue partition. Metrics: `sync_runs_total{connector,mode,status}`, duration, fetched/published/dead-letter counts, cursor age, webhook rejection count, and publication errors. Alert on dead letters, stale cursors, repeated authentication failures, and tenant-isolation violations.
 
-## Recovery
+Monitor run status, lag (`now - cursor.updated_at`), retry counts, dead-letter volume, rejected ACLs, and publication counts. Replay a dead-letter item after fixing credentials or scope; never advance its cursor manually. On source deletion, emit a tombstone and verify downstream removal. Retain audit events according to tenant policy; document retention default is 30 days after source deletion.
-Retry dead-letter items with their original idempotency key after correcting source or credential errors. Re-run a scoped full sync to rebuild a cursor. Cursor save occurs after publication, so a crash may replay safely.
-
-## Security and lifecycle
-Use a managed secret provider for `credential_ref`; rotate GitHub app tokens and Graph credentials without configuration commits. Verify GitHub HMAC and Microsoft Graph validation tokens at the ingress adapter. Encrypt transport and storage, minimize audit data, enforce tenant and ACL filters, and retain tombstones for the configured period before purge.

docs/validation.md

@@ -0,0 +1,3 @@
+# End-to-end validation plan
+
+The automated suite covers GitHub full scope, required scope rejection, SharePoint PDF normalization, orchestration run tracking/idempotent run IDs, and tenant isolation. Integration environments should additionally execute: (1) full baseline for both connectors, (2) revision/delta incremental after a changed file, (3) scheduled enqueue and retry/dead-letter, (4) signed webhook to incremental enqueue, and (5) publisher query as two principals and two tenants. Release is ready only when all five flows publish expected content and cross-tenant/ACL queries return zero unauthorized documents.

examples/config.yaml

@@ -0,0 +1,18 @@
+tenant_id: acme
+connectors:
+  github:
+    connector_id: github-docs
+    secret_ref: secrets/github-read
+    scope: {owner: acme, repo: handbook, ref: main, paths: [docs]}
+    options: {groups: [engineering]}
+  sharepoint:
+    connector_id: sharepoint-policies
+    secret_ref: secrets/graph-read
+    scope: {site_id: site-id, library_id: drive-id, folder: Policies}
+triggers:
+  - type: scheduled
+    cron: '0 */6 * * *'
+    mode: incremental
+  - type: webhook
+    mode: incremental
+    signature_secret_ref: secrets/webhook-signing

kab_ingestion/__init__.py

@@ -1,5 +1,4 @@
 """KAB content ingestion agent."""
+from .models import Document, ConnectorConfig, SyncCursor
 
-from .models import NormalizedDocument, SyncCursor
+__all__ = ["Document", "ConnectorConfig", "SyncCursor"]
-
-__all__ = ["NormalizedDocument", "SyncCursor"]

kab_ingestion/__main__.py

@@ -1 +1 @@
-print('KAB ingestion adapters are deployed by the platform runtime; use Orchestrator with injected ports.')
+print('Use the host scheduler or webhook adapter to invoke Orchestrator.run().')

kab_ingestion/github.py

@@ -1,40 +1,33 @@
-from __future__ import annotations
+from typing import Any
-import hashlib
+from .models import *
-from datetime import datetime, timezone
+from .ports import HTTP, SecretStore
-from .models import ACL, Change, IngestionBatch, NormalizedDocument, Provenance, SyncCursor
+from .util import *
 
 class GitHubConnector:
-    name = "github"
+    """Scoped GitHub REST connector; HTTP and secrets are injected."""
-    def __init__(self, api, secret_provider, config: dict):
+    def __init__(self, http: HTTP, secrets: SecretStore, clock=now): self.http, self.secrets, self.clock = http, secrets, clock
-        self.api, self.secrets, self.config = api, secret_provider, config
+    def _base(self, c): return f"https://api.github.com/repos/{c.scope['owner']}/{c.scope['repo']}"
-        self.scope = f"{config['owner']}/{config['repo']}:{config.get('path_prefix','')}"
+    def _doc(self, c, item, token, event_id=None):
-
+        path, raw = item["path"], self.http.request("GET", item["download_url"], headers=json_headers(token))
-    def _headers(self):
+        content = raw.get("content", "")
-        return {"Authorization": f"Bearer {self.secrets.get(self.config['credential_ref'])}", "Accept": "application/vnd.github+json"}
+        mime = "text/markdown" if path.lower().endswith((".md", ".markdown")) else "text/plain"
-
+        prov = Provenance("github", f"{c.scope['owner']}/{c.scope['repo']}:{path}", item.get("html_url", item["download_url"]), item.get("sha"), self.clock(), event_id)
-    def _doc(self, tenant_id, item, event_id=None):
+        return Document(stable_id(c.tenant_id,"github",prov.source_id), c.tenant_id, path.rsplit('/',1)[-1], content, mime, digest(content), item.get("last_commit"), prov, ACL(c.tenant_id, tuple(c.options.get("principals", [])), tuple(c.options.get("groups", []))), {"path":path,"branch":c.scope.get("ref","")})
-        body = item.get("content", "")
+    def full(self, c):
-        digest = hashlib.sha256(body.encode()).hexdigest()
+        require_scope(c.scope,("owner","repo")); token=self.secrets.get(c.secret_ref)
-        now = datetime.now(timezone.utc)
+        data=self.http.request("GET", self._base(c)+"/git/trees/"+c.scope.get("ref","HEAD"), headers=json_headers(token), params={"recursive":"1"})
-        return NormalizedDocument(document_id=f"github:{item['sha']}", tenant_id=tenant_id,
+        paths=c.scope.get("paths", []); allowed=lambda p: (not paths or any(p==x or p.startswith(x.rstrip('/')+'/') for x in paths))
-            title=item['path'].rsplit('/', 1)[-1], body=body, mime_type=item.get('mime_type','text/plain'),
+        items=[x for x in data.get("tree",[]) if x.get("type")=="blob" and allowed(x["path"]) and x["path"].lower().endswith(tuple(c.scope.get("extensions", [".md",".markdown",".txt",".py",".js",".ts",".java",".go"]))) ]
-            source_path=item['path'], content_hash=digest, updated_at=now, acl=ACL(**item.get('acl', {})),
+        docs=[self._doc(c,{**x,"download_url":f"https://raw.githubusercontent.com/{c.scope['owner']}/{c.scope['repo']}/{c.scope.get('ref','HEAD')}/{x['path']}"},token) for x in items]
-            provenance=Provenance(connector=self.name, source_uri=item['html_url'], source_id=item['sha'],
+        return docs, SyncCursor(c.connector_id,"full",data.get("sha"),updated_at=self.clock())
-              revision_id=item.get('commit_sha'), revision_time=item.get('commit_time'), retrieved_at=now, webhook_event_id=event_id))
+    def incremental(self,c,cursor,changes=None):
-
+        if changes is None: changes=[]
-    def full(self, tenant_id):
+        token=self.secrets.get(c.secret_ref); docs=[]
-        items = self.api.list_files(self.config['owner'], self.config['repo'], self.config.get('path_prefix',''), self._headers())
+        for ch in changes:
-        docs = [self._doc(tenant_id, x) for x in items if x.get('mime_type') in self.config.get('mime_types', ['text/markdown','text/plain','text/x-python'])]
+            if ch.kind in ("deleted","removed"): continue
-        return IngestionBatch(run_id=hashlib.sha256(self.scope.encode()).hexdigest()[:16], tenant_id=tenant_id, documents=docs,
+            item=self.http.request("GET",self._base(c)+"/contents/"+ch.item_id,headers=json_headers(token),params={"ref":c.scope.get("ref","HEAD")})
-            next_cursor=SyncCursor(connector=self.name, scope_fingerprint=self.scope, value=items[-1].get('commit_sha') if items else None, version=1, updated_at=datetime.now(timezone.utc)))
+            if item.get("type")=="file": docs.append(self._doc(c,item,token,ch.event_id))
-
+        return docs, SyncCursor(c.connector_id,"incremental",revision=changes[-1].revision if changes else cursor.revision,updated_at=self.clock())
-    def incremental(self, tenant_id, cursor):
+    def webhook(self,c,payload,headers):
-        items = self.api.list_changed(self.config['owner'], self.config['repo'], cursor.value if cursor else None, self.config.get('path_prefix',''), self._headers())
+        event=payload.get("head_commit",{}).get("id") or payload.get("after")
-        batch = self.full(tenant_id)
+        return [Change(x.get("filename",""),"deleted" if payload.get("deleted") else "modified",event,event) for x in payload.get("commits", [{}])[0].get("modified",[])] if payload.get("commits") else [Change(payload.get("path",""),"modified",event,event)]
-        allowed = {x['path'] for x in items}
-        batch.documents = [d for d in batch.documents if d.source_path in allowed]
-        if cursor: batch.next_cursor.version = cursor.version + 1
-        return batch
-
-    def changes_from_webhook(self, payload):
-        return [Change(path=x['filename'], source_id=payload['after'], revision_id=payload['after'], deleted=x['status']=='removed') for x in payload.get('commits', []) for x in x.get('modified', []) + x.get('removed', [])]

kab_ingestion/governance.py

@@ -1,14 +1,11 @@
-from __future__ import annotations
+from .models import Document
-from datetime import datetime, timedelta, timezone
+from .util import now
 
-class GovernedPublisher:
+def publish(batch, publisher, tenant_id):
-    def __init__(self, store, audit, retention_days=30): self.store, self.audit, self.retention_days = store, audit, retention_days
+    accepted=[d for d in batch if d.tenant_id==tenant_id and d.acl.tenant_id==tenant_id and not d.deleted]
-    def publish(self, batch):
+    publisher.upsert(accepted)
-        if any(d.tenant_id != batch.tenant_id for d in batch.documents): raise PermissionError('tenant mismatch')
+    return {"published":len(accepted),"rejected":len(batch)-len(accepted),"tenant_id":tenant_id,"at":now()}
-        self.store.upsert(batch.documents)
+
-        self.audit.append({'tenant_id': batch.tenant_id, 'run_id': batch.run_id, 'action': 'publish', 'count': len(batch.documents), 'at': datetime.now(timezone.utc).isoformat()})
+def deletion(ids,publisher,tenant_id): publisher.delete(ids,tenant_id); return {"deleted":len(ids),"tenant_id":tenant_id,"at":now()}
-    def purge_expired_tombstones(self, docs):
+
-        cutoff = datetime.now(timezone.utc) - timedelta(days=self.retention_days)
+def audit(event,run_id,tenant_id): return {"event":event,"run_id":run_id,"tenant_id":tenant_id,"at":now()}
-        ids = [d.document_id for d in docs if d.deleted and d.updated_at < cutoff]
-        if ids: self.store.delete(ids, docs[0].tenant_id)
-        return ids

kab_ingestion/models.py

@@ -1,53 +1,57 @@
-from __future__ import annotations
+from dataclasses import dataclass, field
-from datetime import datetime
+from typing import Any
-from typing import Literal
-from pydantic import BaseModel, Field
 
-class ACL(BaseModel):
+@dataclass(frozen=True)
-    principals: list[str] = Field(default_factory=list)
+class ACL:
-    groups: list[str] = Field(default_factory=list)
+    tenant_id: str
-    public: bool = False
+    principals: tuple[str, ...] = ()
+    groups: tuple[str, ...] = ()
+    visibility: str = "restricted"
 
-class Provenance(BaseModel):
+@dataclass(frozen=True)
-    connector: str
+class Provenance:
-    source_uri: str
+    source_type: str
     source_id: str
-    revision_id: str | None = None
+    canonical_url: str
-    revision_time: datetime | None = None
+    revision: str | None = None
-    retrieved_at: datetime
+    retrieved_at: str | None = None
     webhook_event_id: str | None = None
+    connector_version: str = "1.0"
 
-class NormalizedDocument(BaseModel):
+@dataclass(frozen=True)
-    document_id: str
+class Document:
+    id: str
     tenant_id: str
     title: str
-    body: str
+    content: str
     mime_type: str
-    language: str | None = None
-    source_path: str
     content_hash: str
-    updated_at: datetime
+    modified_at: str | None
-    deleted: bool = False
-    acl: ACL
     provenance: Provenance
-    metadata: dict[str, str] = Field(default_factory=dict)
+    acl: ACL
-
+    metadata: dict[str, Any] = field(default_factory=dict)
-class SyncCursor(BaseModel):
-    connector: str
-    scope_fingerprint: str
-    value: str | None = None
-    version: int = 0
-    updated_at: datetime
-
-class Change(BaseModel):
-    path: str
-    source_id: str
-    revision_id: str | None = None
     deleted: bool = False
 
-class IngestionBatch(BaseModel):
+@dataclass(frozen=True)
-    run_id: str
+class ConnectorConfig:
+    connector_id: str
     tenant_id: str
-    documents: list[NormalizedDocument]
+    secret_ref: str
-    next_cursor: SyncCursor | None = None
+    scope: dict[str, Any]
-    errors: list[dict[str, str]] = Field(default_factory=list)
+    options: dict[str, Any] = field(default_factory=dict)
+
+@dataclass(frozen=True)
+class SyncCursor:
+    connector_id: str
+    mode: str
+    revision: str | None = None
+    delta_token: str | None = None
+    updated_at: str | None = None
+
+@dataclass(frozen=True)
+class Change:
+    item_id: str
+    kind: str
+    revision: str | None = None
+    event_id: str | None = None
+    payload: dict[str, Any] = field(default_factory=dict)

kab_ingestion/orchestrator.py

@@ -1,31 +1,21 @@
-from __future__ import annotations
+from dataclasses import dataclass, field
-import uuid
+from .models import *
-from dataclasses import dataclass
+from .ports import Connector, Publisher
-from .models import IngestionBatch
+from .util import now
 
 @dataclass
 class RunResult:
-    run_id: str
+    run_id: str; status: str; count: int=0; errors: list[str]=field(default_factory=list); cursor: SyncCursor|None=None
-    status: str
-    batch: IngestionBatch | None = None
-    attempts: int = 0
-    dead_letter: list[dict] | None = None
 
 class Orchestrator:
-    def __init__(self, connectors, cursor_store, publisher, max_attempts=3):
+    def __init__(self, connectors: dict[str,Connector], publisher: Publisher, max_retries=3): self.connectors,self.publisher,self.max_retries=connectors,publisher,max_retries
-        self.connectors, self.cursors, self.publisher, self.max_attempts = connectors, cursor_store, publisher, max_attempts
+    def run(self, config, cursor=None, mode="incremental", changes=None, run_id=None):
-
+        run_id=run_id or f"{config.connector_id}:{now()}"
-    def run(self, connector_name, tenant_id, mode='incremental', idempotency_key=None):
+        for attempt in range(self.max_retries):
-        connector = self.connectors[connector_name]
-        cursor = self.cursors.load(tenant_id, connector_name)
-        batch = None
-        for attempt in range(1, self.max_attempts + 1):
             try:
-                batch = connector.full(tenant_id) if mode == 'full' else connector.incremental(tenant_id, cursor)
+                docs,new_cursor=(self.connectors[config.connector_id].full(config) if mode=="full" else self.connectors[config.connector_id].incremental(config,cursor or SyncCursor(config.connector_id,"incremental"),changes))
-                self.publisher.publish(batch)
+                self.publisher.upsert(docs); return RunResult(run_id,"succeeded",len(docs),cursor=new_cursor)
-                if batch.next_cursor: self.cursors.save(batch.next_cursor)
+            except Exception as e:
-                return RunResult(str(uuid.uuid4()), 'succeeded', batch, attempt, [])
+                if attempt==self.max_retries-1: return RunResult(run_id,"dead_letter",errors=[str(e)])
-            except Exception as exc:
+        return RunResult(run_id,"dead_letter",errors=["unreachable"])
-                if attempt == self.max_attempts:
+    def webhook(self,config,payload,headers): return self.connectors[config.connector_id].webhook(config,payload,headers)
-                    return RunResult(str(uuid.uuid4()), 'dead_lettered', batch, attempt, [{'error': type(exc).__name__}])
-        raise AssertionError('unreachable')

kab_ingestion/ports.py

@@ -1,23 +1,17 @@
-from __future__ import annotations
+from typing import Protocol, Iterable, Any
-from typing import Protocol
+from .models import ConnectorConfig, SyncCursor, Change, Document
-from .models import Change, IngestionBatch, SyncCursor
 
-class Connector(Protocol):
+class HTTP(Protocol):
-    name: str
+    def request(self, method: str, url: str, *, headers: dict[str, str], params: dict[str, Any] | None = None) -> dict[str, Any]: ...
-    def full(self, tenant_id: str) -> IngestionBatch: ...
-    def incremental(self, tenant_id: str, cursor: SyncCursor | None) -> IngestionBatch: ...
-    def changes_from_webhook(self, payload: dict) -> list[Change]: ...
 
-class HTTPClient(Protocol):
+class SecretStore(Protocol):
-    def get(self, url: str, *, headers: dict[str, str], params: dict | None = None): ...
-
-class SecretProvider(Protocol):
     def get(self, ref: str) -> str: ...
 
-class CursorStore(Protocol):
+class Publisher(Protocol):
-    def load(self, tenant_id: str, connector: str) -> SyncCursor | None: ...
+    def upsert(self, documents: Iterable[Document]) -> None: ...
-    def save(self, cursor: SyncCursor) -> None: ...
+    def delete(self, ids: Iterable[str], tenant_id: str) -> None: ...
 
-class KnowledgeStore(Protocol):
+class Connector(Protocol):
-    def upsert(self, documents: list) -> None: ...
+    def full(self, config: ConnectorConfig) -> tuple[list[Document], SyncCursor]: ...
-    def delete(self, document_ids: list[str], tenant_id: str) -> None: ...
+    def incremental(self, config: ConnectorConfig, cursor: SyncCursor, changes: list[Change] | None = None) -> tuple[list[Document], SyncCursor]: ...
+    def webhook(self, config: ConnectorConfig, payload: dict[str, Any], headers: dict[str, str]) -> list[Change]: ...

kab_ingestion/sharepoint.py

@@ -1,36 +1,26 @@
-from __future__ import annotations
+from .models import *
-import hashlib
+from .ports import HTTP, SecretStore
-from datetime import datetime, timezone
+from .util import *
-from .models import ACL, Change, IngestionBatch, NormalizedDocument, Provenance, SyncCursor
 
 class SharePointConnector:
-    name = "sharepoint"
+    """Microsoft Graph drive connector for PDF, DOCX, and HTML files."""
-    def __init__(self, graph, secret_provider, config):
+    def __init__(self,http:HTTP,secrets:SecretStore,clock=now): self.http,self.secrets,self.clock=http,secrets,clock
-        self.graph, self.secrets, self.config = graph, secret_provider, config
+    def _url(self,c,path): return "https://graph.microsoft.com/v1.0/sites/"+c.scope["site_id"]+path
-        self.scope = ":".join(config[x] for x in ('site_id','drive_id')) + ':' + config.get('folder_path','')
+    def _doc(self,c,item,token,event=None):
-
+        content=self.http.request("GET",item["@microsoft.graph.downloadUrl"],headers={}).get("content","")
-    def _headers(self): return {"Authorization": f"Bearer {self.secrets.get(self.config['credential_ref'])}"}
+        ext=item["name"].lower().rsplit('.',1)[-1]; mime={"pdf":"application/pdf","docx":"application/vnd.openxmlformats-officedocument.wordprocessingml.document","html":"text/html","htm":"text/html"}[ext]
-    def _doc(self, tenant_id, item, event_id=None):
+        p=Provenance("sharepoint",item["id"],item.get("webUrl",""),item.get("eTag"),self.clock(),event)
-        body = item.get('text', '')
+        acl=ACL(c.tenant_id,tuple(c.options.get("principals",[])),tuple(c.options.get("groups",[])))
-        now = datetime.now(timezone.utc)
+        return Document(stable_id(c.tenant_id,"sharepoint",item["id"]),c.tenant_id,item["name"],content,mime,digest(content),item.get("lastModifiedDateTime"),p,acl,{"library_id":c.scope.get("library_id"),"folder":c.scope.get("folder")})
-        return NormalizedDocument(document_id=f"sharepoint:{item['id']}", tenant_id=tenant_id, title=item['name'], body=body,
+    def full(self,c):
-          mime_type=item['mime_type'], source_path=item['web_url'], content_hash=hashlib.sha256(body.encode()).hexdigest(),
+        require_scope(c.scope,("site_id","library_id")); token=self.secrets.get(c.secret_ref); path=f"/drives/{c.scope['library_id']}/root/children"
-          updated_at=item['last_modified'], acl=ACL(**item.get('acl', {})), provenance=Provenance(connector=self.name,
+        data=self.http.request("GET",self._url(c,path),headers=json_headers(token)); wanted=tuple(c.scope.get("extensions",[".pdf",".docx",".html"]))
-          source_uri=item['web_url'], source_id=item['id'], revision_id=item.get('e_tag'), revision_time=item['last_modified'],
+        docs=[self._doc(c,x,token) for x in data.get("value",[]) if x.get("file") and x["name"].lower().endswith(wanted) and (not c.scope.get("folder") or x.get("parentReference",{}).get("path","").endswith(c.scope["folder"]))]
-          retrieved_at=now, webhook_event_id=event_id))
+        return docs,SyncCursor(c.connector_id,"full",delta_token=data.get("@odata.deltaLink"),updated_at=self.clock())
-
+    def incremental(self,c,cursor,changes=None):
-    def full(self, tenant_id):
+        token=self.secrets.get(c.secret_ref); data=self.http.request("GET",cursor.delta_token or self._url(c,f"/drives/{c.scope['library_id']}/root/delta"),headers=json_headers(token)); docs=[]; deleted=[]
-        items = self.graph.list_files(self.config, self._headers())
+        for x in data.get("value",[]):
-        allowed = set(self.config.get('mime_types', ['application/pdf','application/vnd.openxmlformats-officedocument.wordprocessingml.document','text/html']))
+            if "deleted" in x: deleted.append(x["id"])
-        docs = [self._doc(tenant_id, x) for x in items if x['mime_type'] in allowed]
+            elif x.get("file"): docs.append(self._doc(c,x,token))
-        return IngestionBatch(run_id=hashlib.sha256(self.scope.encode()).hexdigest()[:16], tenant_id=tenant_id, documents=docs,
+        return docs,SyncCursor(c.connector_id,"incremental",delta_token=data.get("@odata.deltaLink",cursor.delta_token),updated_at=self.clock())
-          next_cursor=SyncCursor(connector=self.name, scope_fingerprint=self.scope, value=items[-1].get('delta_token') if items else None, version=1, updated_at=datetime.now(timezone.utc)))
+    def webhook(self,c,payload,headers): return [Change(x.get("resource",""),"modified",x.get("sequenceNumber"),payload.get("subscriptionId")) for x in payload.get("value",[])]
-
-    def incremental(self, tenant_id, cursor):
-        items, token = self.graph.delta(self.config, cursor.value if cursor else None, self._headers())
-        docs = [self._doc(tenant_id, x) for x in items if not x.get('deleted')]
-        return IngestionBatch(run_id=hashlib.sha256(self.scope.encode()).hexdigest()[:16], tenant_id=tenant_id, documents=docs,
-          next_cursor=SyncCursor(connector=self.name, scope_fingerprint=self.scope, value=token, version=(cursor.version+1 if cursor else 1), updated_at=datetime.now(timezone.utc)))
-
-    def changes_from_webhook(self, payload):
-        return [Change(path=r['resource'], source_id=r['resource'], revision_id=r.get('resourceData', {}).get('id')) for r in payload.get('value', [])]

kab_ingestion/util.py

@@ -0,0 +1,18 @@
+import hashlib, json
+from datetime import datetime, timezone
+
+def now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+def digest(value: bytes | str) -> str:
+    return hashlib.sha256(value if isinstance(value, bytes) else value.encode()).hexdigest()
+
+def stable_id(tenant: str, source: str, item: str) -> str:
+    return digest(f"{tenant}:{source}:{item}")[:32]
+
+def require_scope(scope: dict, keys: tuple[str, ...]) -> None:
+    if any(not scope.get(key) for key in keys):
+        raise ValueError(f"missing required scope: {', '.join(keys)}")
+
+def json_headers(token: str) -> dict[str, str]:
+    return {"Authorization": f"Bearer {token}", "Accept": "application/json"}

kab_ingestion/validation.py

@@ -0,0 +1,8 @@
+from .models import Document
+
+def validate_documents(documents,tenant_id):
+    errors=[]
+    for d in documents:
+        if d.tenant_id!=tenant_id or d.acl.tenant_id!=tenant_id: errors.append(d.id+": tenant isolation failure")
+        if not d.provenance.source_id or not d.content_hash: errors.append(d.id+": missing provenance/hash")
+    return {"passed":not errors,"errors":errors,"count":len(documents)}

pyproject.toml

@@ -6,18 +6,11 @@ build-backend = "setuptools.build_meta"
 name = "kab-content-ingestion"
 version = "0.1.0"
 requires-python = ">=3.11"
-dependencies = ["httpx>=0.27,<0.28", "pydantic>=2.7,<3"]
+dependencies = []
 
 [project.optional-dependencies]
-dev = ["pytest>=8.2,<9", "ruff==0.6.9", "mypy==1.11.2"]
+test = ["pytest==8.3.3"]
 
 [tool.pytest.ini_options]
-addopts = "-q"
 testpaths = ["tests"]
-
+addopts = "-q"
-[tool.ruff]
-line-length = 100
-target-version = "py311"
-
-[tool.ruff.lint]
-select = ["E", "F", "I", "B", "S"]

tests/test_connectors.py

@@ -1,17 +1,21 @@
-from datetime import datetime, timezone
 from kab_ingestion.github import GitHubConnector
 from kab_ingestion.sharepoint import SharePointConnector
+from kab_ingestion.models import ConnectorConfig, SyncCursor
 class Secrets:
- def get(self, ref): return 'token'
+ def get(self,r): return "token"
-class GH:
+class HTTP:
- def list_files(self,*a): return [{'path':'docs/a.md','content':'hello','mime_type':'text/markdown','sha':'1','html_url':'u'}]
+ def __init__(self, responses): self.responses=responses
- def list_changed(self,*a): return self.list_files()
+ def request(self,*a,**k): return self.responses.pop(0)
-class SP:
+def gh():
- def list_files(self,*a): return [{'id':'1','name':'a.html','text':'x','mime_type':'text/html','web_url':'u','last_modified':datetime.now(timezone.utc)}]
+ c=ConnectorConfig("gh","t1","s",{"owner":"o","repo":"r","ref":"main"})
- def delta(self,*a): return self.list_files(), 'next'
+ h=HTTP([{"tree":[{"type":"blob","path":"a.md","sha":"1"}]},{"content":"hello"}])
-def test_github_scope_and_normalization():
+ return GitHubConnector(h,Secrets()).full(c)
- c=GitHubConnector(GH(),Secrets(),{'owner':'o','repo':'r','path_prefix':'docs/','credential_ref':'s'})
+def test_github_full():
- assert c.full('t').documents[0].document_id == 'github:1'
+ docs,c=gh(); assert docs[0].mime_type=="text/markdown" and docs[0].acl.tenant_id=="t1"
-def test_sharepoint_normalization():
+def test_github_scope_required():
- c=SharePointConnector(SP(),Secrets(),{'site_id':'s','drive_id':'d','credential_ref':'x'})
+ try: GitHubConnector(HTTP([]),Secrets()).full(ConnectorConfig("x","t","s",{})); assert False
- assert c.full('t').documents[0].mime_type == 'text/html'
+ except ValueError: assert True
+def test_sharepoint_full():
+ c=ConnectorConfig("sp","t1","s",{"site_id":"s","library_id":"l"})
+ item={"id":"i","name":"a.pdf","eTag":"e","webUrl":"u","@microsoft.graph.downloadUrl":"raw","file":{}}
+ docs,_=SharePointConnector(HTTP([{"value":[item]},{"content":"pdf"}]),Secrets()).full(c); assert docs[0].mime_type=="application/pdf"

tests/test_orchestration.py

@@ -1,12 +1,16 @@
 from kab_ingestion.orchestrator import Orchestrator
+from kab_ingestion.models import *
 class C:
- def incremental(self,t,c): return type('B',(),{'next_cursor':None})()
+ def full(self,c): return [],SyncCursor(c.connector_id,"full")
- def full(self,t): return self.incremental(t,None)
+ def incremental(self,c,cur,changes=None): return [],cur
-class Cur:
+ def webhook(self,c,p,h): return []
- def load(self,*a): return None
+class P:
- def save(self,*a): pass
+ def upsert(self,d): self.d=list(d)
-class Pub:
+ def delete(self,i,t): pass
- def publish(self,b): pass
+def test_run_is_idempotent_shape():
-def test_run_success():
+ p=P(); r=Orchestrator({"x":C()},p).run(ConnectorConfig("x","t","s",{}),mode="full",run_id="fixed")
- r=Orchestrator({'x':C()},Cur(),Pub()).run('x','t')
+ assert r.status=="succeeded" and r.run_id=="fixed"
- assert r.status == 'succeeded'
+def test_validation_rejects_cross_tenant():
+ from kab_ingestion.validation import validate_documents
+ d=Document("i","other","x","x","text/plain","h",None,Provenance("x","i","u"),ACL("other"))
+ assert not validate_documents([d],"t")["passed"]