agents/content-ingestion-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

decomposer: generate deliverable files for Define the normalized ingestion contract and pluggable source-connector interface for SCM, CMS, and other content sources.; Implement a GitHub SCM connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of Markdown, plain-text, and source files with webhook and revision metadata.; Implement a SharePoint CMS connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of PDF, DOCX, and HTML files with webhook and revision metadata.; Implement ingestion orchestration and triggers; Normalize, govern, and publish ingested content to the shared knowledge store.; Add deployment/configuration, tests, and documentation for the ingestion agent.; Validate end-to-end ingestion and downstream content availability.
Some checks failed
ci / test (push) Failing after 8s
Details

Browse Source
This commit is contained in:
demo-bot
2026-09-01 14:03:17 +00:00
parent 780d0d9cd6
commit 34d9dd0f1f
26 changed files with 512 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
spec/ingestion-contract.md Normal file
View File

@@ -0,0 +1,25 @@
# KAB ingestion contract (v1)
## Connector interface
A connector is configured with `tenant_id`, a source-specific scope, `credential_ref`, accepted MIME types, and trigger policy. It exposes `full(tenant_id)`, `incremental(tenant_id, cursor)`, and `changes_from_webhook(payload)`. Both sync methods return an `IngestionBatch`; webhook parsing returns source changes and never retrieves content. Connectors must be deterministic for the same source revision, idempotent by `document_id + content_hash`, and must never log credentials or content.
## Normalized document
Required fields: `document_id`, `tenant_id`, `title`, `body`, `mime_type`, `source_path`, `content_hash`, `updated_at`, `acl`, and `provenance`. Optional `language`, `metadata`, and `deleted` are supported. `document_id` is a stable namespaced source ID. Body is UTF-8 extracted text; unsupported or extraction-failed content is rejected to the batch error list.
## Provenance
`connector`, immutable `source_uri` and `source_id`, optional `revision_id`, optional `revision_time`, `retrieved_at`, and optional `webhook_event_id` provide traceability. Source revision metadata must be retained with publication.
## ACL and isolation
Every document carries `tenant_id` and `ACL {principals, groups, public}`. Publication must enforce tenant equality and downstream retrieval must filter by tenant plus principal/group intersection; `public` never crosses tenants. Missing ACL means deny all.
## Cursors and triggers
A cursor contains connector, scope fingerprint, opaque source value, monotonic version, and update time. Full sync starts a new cursor; incremental uses the stored cursor and atomically saves the returned cursor after publication. Trigger modes are `on_demand`, `schedule` (cron/interval), and `webhook`; each chooses `full` or `incremental`, has retry/backoff and an idempotency key. Webhooks are authenticated, bounded, and enqueue work rather than doing network retrieval inline.
## Lifecycle
Deletes are tombstone documents first, then physically deleted after retention policy. Audit events include tenant, run, source, action, result, and timestamp but no secret or body.