Add CP1 Discovery agents in both patterns
All checks were successful
check agents / check (push) Successful in 10s
All checks were successful
check agents / check (push) Successful in 10s
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
# KAB dev: MCP server registration fails with HTTP 500 for the default tenant
|
||||
|
||||
> **Moved here on 2026-10-29.** Written while the agents were in two separate repositories, both now in this one:
|
||||
> `cp1-agents` → `sub-agents/ALZ-F01: Discovery Workflow Agent/`, and `cp1-agents-individual` → `individual-agents/ALZ-F01: Discovery Workflow Agent/`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Environment | kab-dev (AKS `crucible-kab-dev-aks`), platform from `modernization-platform` @ `7adfc87f` (2026-10-19) |
|
||||
| Component | MCP Gateway (`kyn-agent-builder-mcp-gateway`, namespace `core-components`, Helm release `kyn`, chart `agent-builder-mcp-gateway-0.1.0`) |
|
||||
| Reported | 2026-10-29 |
|
||||
| Severity | High: no MCP server can be registered in the default tenant, so no agent that uses MCP tools can be created or run there |
|
||||
| Status | Root cause confirmed; one-call remediation identified; permanent fix needed in the platform bootstrap |
|
||||
|
||||
## Summary
|
||||
|
||||
Registering an MCP server in the KAB UI fails with `500 Internal Server Error` on
|
||||
`POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/servers`. The MCP server itself is healthy, and
|
||||
its key is accepted.
|
||||
|
||||
The failure is a gap in platform provisioning. The MCP Gateway keeps its own tenant registry, and the gateway tenant
|
||||
for KAB's **default tenant** (`6ada663eb5b88053a9e43723`) has never been linked to it. KAB creates its default tenant
|
||||
at startup without running the provisioning workflow. That workflow is the only automated path that creates the
|
||||
matching gateway tenant. Each request from the default tenant therefore resolves to no gateway tenant, and Postgres
|
||||
row-level security refuses the insert.
|
||||
|
||||
Locally, the dev kit closes this gap with `scripts/link-mcp-tenants.sh`. The AKS installation has no equivalent step.
|
||||
|
||||
**Immediate fix:** one idempotent call, made as a KAB platform admin, that sets the gateway's `default` tenant's
|
||||
`external_id` to the KAB default tenant id (see [Remediation](#remediation)). **Permanent fix:** register the
|
||||
bootstrap tenant with the gateway as part of platform bootstrap.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- **In the UI:** registering a server fails with the browser console error `Failed to load resource: the server responded with a status of 500 ()`.
|
||||
- **The server list is empty** for the default tenant, even though the gateway logs show health-check rounds.
|
||||
- **Every proxied request** from the default tenant logs a warning on the gateway:
|
||||
`tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved (the caller will see no data)`.
|
||||
|
||||
## Evidence
|
||||
|
||||
**Gateway log for one failed registration (2026-10-29 16:07:29 UTC, request `9f0021b407a1080ccd50dd860744d647`; the same sequence appeared at 13:15:06 and about 13:19):**
|
||||
|
||||
```text
|
||||
[warning] tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved
|
||||
[info] Server 2d10e0e1-... auth check: required=True
|
||||
[info] Validating API key for server 2d10e0e1-...
|
||||
[info] API key validated successfully for server 2d10e0e1-...
|
||||
[info] Encrypted API key for server: 2d10e0e1-...
|
||||
[error] Failed to create server: ... InsufficientPrivilegeError: new row violates row-level security policy for table "servers"
|
||||
[INSERT INTO servers (... tenant_id ...) VALUES (... '6ada663eb5b88053a9e43723' ...)]
|
||||
[error] Database error: 500: Failed to create server
|
||||
[info] http_request_completed method=POST path=/api/v1/admin/servers status_code=500
|
||||
```
|
||||
|
||||
The connection test (`POST /api/v1/admin/servers/test-connection`) passes, and the gateway's authenticated
|
||||
`tools/list` call to the MCP server succeeds. The failure happens only at the database insert.
|
||||
|
||||
**Gateway configuration (Deployment `env`, set by Helm values):**
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `MCP_GATEWAY_TENANCY_MODE` | `single` |
|
||||
| `DEFAULT_TENANT_ID` | `default` |
|
||||
|
||||
**Provisioning activity:**
|
||||
- **No tenant registrations:** the gateway logged no `POST /api/v1/admin/tenants` call in the 6 hours checked.
|
||||
- **No provisioning runs:** the platform worker (`kyn-agent-builder-platform-worker`, queue `platform-tenant-lifecycle`) has registered `provision_mcp_gateway_tenant`, but logged no provisioning run since it started at 13:55 UTC.
|
||||
|
||||
**The MCP server under test (`alz-discovery-sample`, namespace `mcp-servers`) is healthy:**
|
||||
- `/health` reports healthy.
|
||||
- Unauthenticated calls get 401.
|
||||
- With its key: `initialize`, `tools/list` (20 tools) and `tools/call` all succeed from inside the cluster and from the gateway pod.
|
||||
|
||||
## Root cause: traced end to end
|
||||
|
||||
All paths below are relative to `modernization-platform/packages/`.
|
||||
|
||||
### 1. KAB creates the default tenant without provisioning it
|
||||
|
||||
`agent-builder-api/src/database/init_database.py` (lines 78–110) runs at API startup (`main.py`, "Initialize
|
||||
database with required collections and default teams"). If the `tenants` collection is empty, it inserts
|
||||
"Default Organization" directly into MongoDB. The tenant id is the MongoDB ObjectId returned by `insert_one`, which
|
||||
is the 24-hex form of `6ada663eb5b88053a9e43723`. The code states the gap itself:
|
||||
|
||||
> This tenant never runs the provisioning workflow, which is what normally writes these fields ...
|
||||
|
||||
It makes up for one consequence (it stamps `database_name` and `temporal_namespace`), but not for the missing MCP
|
||||
Gateway tenant. `init_default_data.py` (the CLI seeder, `init_default_user_and_team`) inserts its "Default Tenant"
|
||||
the same way, with no reference to provisioning or the gateway.
|
||||
|
||||
### 2. Normal tenants are provisioned, including the gateway
|
||||
|
||||
`agent-builder-api/src/api/tenant_routes.py`, `POST /api/v1/tenants/`, calls:
|
||||
1. **`db.create_tenant`,** with `provisioning_status="pending"`;
|
||||
2. **`_provisioning_input`,** which passes `tenant_slug` in `config` for the gateway;
|
||||
3. **`_start_provisioning`,** which starts `TenantProvisioningWorkflow` on `platform-tenant-lifecycle`.
|
||||
|
||||
Among that workflow's activities is
|
||||
`agent-builder-temporal-workers/src/temporal/activities/tenant_lifecycle/mcp_gateway_mgmt.py`,
|
||||
`provision_mcp_gateway_tenant`. It posts to the gateway directly:
|
||||
|
||||
```json
|
||||
POST {MCP_GATEWAY_URL}/api/v1/admin/tenants (X-Internal-Service-Token)
|
||||
{"id": "<tenant slug>", "name": "<tenant name>", "external_id": "<KAB tenant id>", "status": "active"}
|
||||
```
|
||||
|
||||
Its docstring records the history: "Until this step existed the row had to be created by hand
|
||||
(local-dev-kit/scripts/link-mcp-tenants.sh), so a freshly provisioned tenant opened the MCP pages to nothing."
|
||||
|
||||
### 3. The gateway resolves tenants only through that registry
|
||||
|
||||
`agent-builder-mcp-gateway/src/mcp_gateway/core/tenancy.py`, `resolve_tenant_ref` (around line 305), maps the
|
||||
tenant reference KAB sends to a gateway tenant by `Tenant.id == ref OR Tenant.external_id == ref`. If nothing
|
||||
matches, it returns the reference unchanged, on purpose: "binding it matches no rows, so an unknown tenant sees
|
||||
nothing instead of another tenant's data." The gateway's only tenant is `default` (from `DEFAULT_TENANT_ID`), whose
|
||||
`external_id` doesn't hold `6ada663eb5b88053a9e43723`. So the new server row is stamped
|
||||
`tenant_id = '6ada663eb5b88053a9e43723'`.
|
||||
|
||||
### 4. Row-level security rejects that row
|
||||
|
||||
`agent-builder-mcp-gateway/alembic/versions/20260723_1000_tenancy_full_rollout.py` puts `servers` (and `tools`,
|
||||
`mcp_tools`, `resources`, `prompts` and others) under the catalog policy:
|
||||
|
||||
```sql
|
||||
EFFECTIVE_TENANT := COALESCE(NULLIF(current_setting('app.current_tenant', true), ''), 'default')
|
||||
CREATE POLICY tenant_catalog ON servers
|
||||
USING (tenant_id IS NULL OR tenant_id = EFFECTIVE_TENANT)
|
||||
WITH CHECK (tenant_id = EFFECTIVE_TENANT OR (tenant_id IS NULL AND EFFECTIVE_TENANT = 'default'))
|
||||
```
|
||||
|
||||
In `single` mode, `SingleTenantStrategy` sets no session tenant, so `EFFECTIVE_TENANT` is `'default'`. The row's
|
||||
`'6ada663eb5b88053a9e43723'` fails the check, Postgres raises `InsufficientPrivilegeError`, and the API returns 500.
|
||||
The same unresolved tenant is why the server list shows nothing.
|
||||
|
||||
### 5. Why it works locally and not in AKS
|
||||
|
||||
The dev kit runs `agent-builder-local-dev-kit/scripts/link-mcp-tenants.sh`. It reads the platform's tenants from
|
||||
MongoDB, maps the **first** one (the bootstrap default) to gateway tenant `default`, sets `external_id` to its `_id`,
|
||||
and maps every later tenant by slug. The AKS Helm values (`agent-builder-kubernetes-deployment/AKS/helm-umbrella-chart/v2.2.0/values_files/az_dev.yaml`
|
||||
around line 1303, and v2.3.0 `az_staging.yaml`) configure only the gateway connection for
|
||||
`provision_mcp_gateway_tenant`, which covers new tenants. **Nothing in the AKS installation links the bootstrap
|
||||
tenant.**
|
||||
|
||||
## Impact
|
||||
|
||||
- **No MCP servers in the default tenant:** none can be registered, and none would show to its users.
|
||||
- **No working agents there:** agents can't be given an MCP key (`MCP_AGENT_API_KEY`), so any agent that declares MCP tools fails every request at startup with `MCP_AGENT_API_KEY is not set, so the MCP gateway will refuse the agent`.
|
||||
- **Blocks:** the CP1 Agentic Landing Zone Discovery deployment, in both patterns: `cp1-agents` (orchestrator) and `cp1-agents-individual` (eight agents).
|
||||
- **Likely other environments too:** any environment installed from the AKS or EKS charts with a bootstrap default tenant has the same gap unless it was linked by hand.
|
||||
|
||||
## Remediation
|
||||
|
||||
### Immediate (no deploy, no Helm change)
|
||||
|
||||
Make the call the provisioning activity would have made, through KAB's proxy, as a **KAB platform admin**. The
|
||||
proxy only forwards `api/v1/admin/tenants` to callers with platform-admin rights; see
|
||||
`agent-builder-api/src/api/routes/mcp_gateway_proxy.py`, `_CROSS_TENANT_PATTERNS`.
|
||||
|
||||
```sh
|
||||
curl -sS -X POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/tenants \
|
||||
-H "Authorization: Bearer <platform admin session token>" -H "Content-Type: application/json" \
|
||||
-d '{"id": "default", "name": "Default Organization", "external_id": "6ada663eb5b88053a9e43723", "status": "active"}'
|
||||
```
|
||||
|
||||
The endpoint is an idempotent upsert (`agent-builder-mcp-gateway/src/mcp_gateway/api/endpoints/tenant_admin.py`,
|
||||
`upsert_tenant`). It keeps the gateway's `default` tenant and records KAB's id on it, exactly as
|
||||
`link-mcp-tenants.sh` does locally. `DEFAULT_TENANT_ID` doesn't need to change.
|
||||
|
||||
**Don't** instead set `DEFAULT_TENANT_ID=6ada663eb5b88053a9e43723`. That names a gateway tenant row that doesn't
|
||||
exist, and Helm would revert a manual change.
|
||||
|
||||
### Permanent (platform change)
|
||||
|
||||
Options, best first:
|
||||
|
||||
1. **Register the bootstrap tenant during bootstrap.** When `init_database.py` creates or finds the bootstrap tenant, make the same `POST /api/v1/admin/tenants` call as `provision_mcp_gateway_tenant`, with gateway id `default`. Alternatively, start the provisioning workflow's gateway step for it.
|
||||
2. **Add a post-install / post-upgrade Helm hook Job** that does what `link-mcp-tenants.sh` does, using the gateway API rather than direct SQL.
|
||||
3. **Reconcile on a schedule:** a platform-worker job that upserts a gateway tenant for every active KAB tenant with no `external_id` match, which also repairs any tenant whose provisioning failed.
|
||||
|
||||
## Verification
|
||||
|
||||
1. **After the upsert,** `GET .../api/v1/mcp-gateway/api/v1/admin/tenants` lists `default` with `external_id: 6ada663eb5b88053a9e43723`.
|
||||
2. **The warning stops:** new requests from the default tenant no longer log `matches no tenant id or external_id`.
|
||||
3. **Registration succeeds:** registering `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` returns 2xx, and the server appears in the list.
|
||||
4. **Agents get keys:** creating an agent issues an `MCP_AGENT_API_KEY`, and that agent's requests no longer fail at startup.
|
||||
|
||||
## Related finding
|
||||
|
||||
The gateway's OIDC just-in-time provisioning failed for `jonathan.boniface@kyndryl.com` with a unique-key violation
|
||||
on `users` (`[OIDCVerifier] JIT provisioning failed ... UniqueViolationError`). The user probably exists already under
|
||||
another id. `OIDC_DEFAULT_ROLES` is `["viewer"]`. Check this user's gateway role after the tenant fix, in case server
|
||||
registration then fails on permissions instead.
|
||||
|
||||
## Method
|
||||
|
||||
- **Read-only checks against `crucible-kab-dev-aks`:** gateway, API and platform-worker logs, and Deployment `env` (plain values only).
|
||||
- **Code reading in `modernization-platform` @ `7adfc87f`.**
|
||||
- **Nothing in `core-components` was changed.**
|
||||
- **No Secret values or database credentials were read:** an attempt to query the gateway database directly was refused, and was not retried.
|
||||
|
||||
## Addendum (2026-10-29, 17:45 UTC): why the 500 persists, and a second defect
|
||||
|
||||
**The error is unchanged.** At 17:38:32 `POST /api/v1/admin/servers` failed again with `new row violates row-level security policy for table "servers"`. At 17:38–17:40 the gateway also logged `tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved`. The KAB default tenant is still unmapped.
|
||||
|
||||
**The one tenant upsert seen didn't touch it.** `POST /api/v1/admin/tenants` → 200 at 17:06:28 came from `kyn-agent-builder-platform-worker` (10.20.10.17), so it was a `TenantProvisioningWorkflow` for a newly created tenant. That activity (`temporal-workers/.../tenant_lifecycle/mcp_gateway_mgmt.py`) upserts `id = <tenant slug>` and `external_id = <platform id>`. It doesn't create a mapping for `6ada663eb5b88053a9e43723`.
|
||||
|
||||
### The full code path
|
||||
|
||||
| Step | Code | What happens |
|
||||
|---|---|---|
|
||||
| 1 | `agent-builder-api/src/api/routes/mcp_gateway_proxy.py` `_identity_context` | Forwards `X-Tenant-ID` = the tenant selected in the UI (`6ada…`). It sends `default` only when a platform admin has **no** tenant selected. |
|
||||
| 2 | `mcp-gateway/src/mcp_gateway/gateway/auth.py` (service-token path) | `resolve_tenant_ref` → `bind_tenant`. An unmapped reference is bound **as-is**. |
|
||||
| 3 | `mcp-gateway/src/mcp_gateway/db/repositories.py` `ServerRepository.create` | If the bound tenant ≠ `DEFAULT_TENANT_ID` (`default`), it stamps `servers.tenant_id = <bound tenant>`; otherwise it stamps `NULL` (global). |
|
||||
| 4 | `mcp-gateway/src/mcp_gateway/core/tenancy.py` `SingleTenantStrategy` | Single mode **never** issues `set_config('app.current_tenant', …)`. |
|
||||
| 5 | `alembic/versions/20260723_1000_tenancy_full_rollout.py` `tenant_catalog` | `WITH CHECK (tenant_id = EFFECTIVE OR (tenant_id IS NULL AND EFFECTIVE = 'default'))`. With no setting, EFFECTIVE is always `'default'`. |
|
||||
|
||||
So in `MCP_GATEWAY_TENANCY_MODE=single`, **a server insert succeeds only when the bound tenant is exactly `default`**. That gives a second defect, beyond the missing mapping:
|
||||
|
||||
- **Unmapped tenant** (today): `tenant_id='6ada…'` ≠ `'default'`, so it's refused.
|
||||
- **Mapped tenant whose gateway id isn't `default`** (what `TenantProvisioningWorkflow` creates for every new tenant, id = slug): `tenant_id='<slug>'` ≠ `'default'`, so it's refused too. In single mode, server registration fails for **every tenant except one mapped to `id='default'`**.
|
||||
|
||||
The `link-default-tenant` Job (an upsert of `{id:"default", external_id:"6ada…"}`) still fixes the KAB default tenant. It does nothing for other tenants.
|
||||
|
||||
### Options
|
||||
|
||||
| # | Option | Who | Effect |
|
||||
|---|---|---|---|
|
||||
| A | **Register from "All tenants".** A platform admin clears the tenant selector in the UI, so the proxy sends `X-Tenant-ID: default` and the server is stored as a global entry (`tenant_id NULL`). | Platform admin (`is_platform_admin`) | Works today, with no cluster change. The server is global; in single mode every tenant sees all global servers. |
|
||||
| B | Map the default tenant: apply `secrets.yaml` and `mcp-gateway-link-default-tenant.job.yaml`, or a Global Admin calls `POST /api/v1/mcp-gateway/api/v1/admin/tenants` with `{"id":"default","external_id":"6ada663eb5b88053a9e43723",…}` | Global Admin, or approval for a `core-components` change | Registration works with the default tenant selected. |
|
||||
| C | Platform fix: in single mode, make `ServerRepository.create` stamp `NULL` regardless of the bound tenant, **or** run the gateway in `pooled` mode (needs the restricted DB role and Alembic, per `docs/TENANCY.md`) | Platform team | Fixes registration for every tenant. Single mode then matches its documented promise ("byte-identical to the pre-tenancy behavior"). |
|
||||
| D | Platform fix: the bootstrap in `init_database.py` should run `provision_mcp_gateway_tenant` for the default tenant, with `id = DEFAULT_TENANT_ID` | Platform team | Stops a fresh install from shipping an unmapped default tenant. |
|
||||
Reference in New Issue
Block a user