agents/content-ingestion-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

decomposer: generate deliverable files for Define the normalized ingestion contract and pluggable source-connector interface for SCM, CMS, and other content sources.; Implement a GitHub SCM connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of Markdown, plain-text, and source files with webhook and revision metadata.; Implement a SharePoint CMS connector that conforms to the normalized ingestion contract and supports scoped full and incremental ingestion of PDF, DOCX, and HTML files with webhook and revision metadata.; Implement ingestion orchestration and triggers; Normalize, govern, and publish ingested content to the shared knowledge store.; Add deployment/configuration, tests, and documentation for the ingestion agent.; Validate end-to-end ingestion and downstream content availability.
Some checks failed
ci / test (push) Failing after 8s
Details

Browse Source
This commit is contained in:
demo-bot
2026-09-01 14:03:17 +00:00
parent 780d0d9cd6
commit 34d9dd0f1f
26 changed files with 512 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/examples.md Normal file
View File

@@ -0,0 +1,3 @@
# Examples
`config/example.yaml` shows repository, path, site, library, and folder scopes. A full run is `Trigger('github','acme',mode='full',source='on_demand',idempotency_key='run-1')`; a scheduled run uses `source='schedule'`; a webhook handler validates the signature, converts the payload with `changes_from_webhook`, and enqueues an incremental trigger.

10
docs/operations.md Normal file
View File

@@ -0,0 +1,10 @@
# Operations
## Deployment and monitoring
Run the ASGI/queue adapter in the platform runtime with one worker per queue partition. Metrics: `sync_runs_total{connector,mode,status}`, duration, fetched/published/dead-letter counts, cursor age, webhook rejection count, and publication errors. Alert on dead letters, stale cursors, repeated authentication failures, and tenant-isolation violations.
## Recovery
Retry dead-letter items with their original idempotency key after correcting source or credential errors. Re-run a scoped full sync to rebuild a cursor. Cursor save occurs after publication, so a crash may replay safely.
## Security and lifecycle
Use a managed secret provider for `credential_ref`; rotate GitHub app tokens and Graph credentials without configuration commits. Verify GitHub HMAC and Microsoft Graph validation tokens at the ingress adapter. Encrypt transport and storage, minimize audit data, enforce tenant and ACL filters, and retain tombstones for the configured period before purge.

3
docs/validation-report.md Normal file
View File

@@ -0,0 +1,3 @@
# End-to-end validation plan
The offline harness covers full, incremental, scheduled, and webhook trigger validation. Connector integration tests must run against source API fixtures with signature fixtures and a fake KAB store. Release acceptance requires: all pytest, Ruff 0.6.9, and mypy 1.11.2 checks pass; replayed webhook is idempotent; cursor advances only after publication; deleted files produce tombstones; downstream queries cannot cross tenant or ACL boundaries; and dead-letter replay succeeds. The repository includes the executable trigger harness in `scripts/e2e_validation.py`.