All checks were successful
check agents / check (push) Successful in 10s
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
232 lines
16 KiB
Markdown
232 lines
16 KiB
Markdown
# KAB dev: MCP server registration fails with HTTP 500 for the default tenant
|
||
|
||
> **Moved here on 2026-10-29.** Written while the agents were in two separate repositories, both now in this one:
|
||
> `cp1-agents` → `sub-agents/ALZ-F01: Discovery Workflow Agent/`, and `cp1-agents-individual` → `individual-agents/ALZ-F01: Discovery Workflow Agent/`.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Environment | kab-dev (AKS `crucible-kab-dev-aks`), platform from `modernization-platform` @ `7adfc87f` (2026-10-19) |
|
||
| Component | MCP Gateway (`kyn-agent-builder-mcp-gateway`, namespace `core-components`, Helm release `kyn`, chart `agent-builder-mcp-gateway-0.1.0`) |
|
||
| Reported | 2026-10-29 |
|
||
| Severity | High: no MCP server can be registered in the default tenant, so no agent that uses MCP tools can be created or run there |
|
||
| Status | Root cause confirmed; one-call remediation identified; permanent fix needed in the platform bootstrap |
|
||
|
||
## Summary
|
||
|
||
Registering an MCP server in the KAB UI fails with `500 Internal Server Error` on
|
||
`POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/servers`. The MCP server itself is healthy, and
|
||
its key is accepted.
|
||
|
||
The failure is a gap in platform provisioning. The MCP Gateway keeps its own tenant registry, and the gateway tenant
|
||
for KAB's **default tenant** (`6ada663eb5b88053a9e43723`) has never been linked to it. KAB creates its default tenant
|
||
at startup without running the provisioning workflow. That workflow is the only automated path that creates the
|
||
matching gateway tenant. Each request from the default tenant therefore resolves to no gateway tenant, and Postgres
|
||
row-level security refuses the insert.
|
||
|
||
Locally, the dev kit closes this gap with `scripts/link-mcp-tenants.sh`. The AKS installation has no equivalent step.
|
||
|
||
**Immediate fix:** one idempotent call, made as a KAB platform admin, that sets the gateway's `default` tenant's
|
||
`external_id` to the KAB default tenant id (see [Remediation](#remediation)). **Permanent fix:** register the
|
||
bootstrap tenant with the gateway as part of platform bootstrap.
|
||
|
||
## Symptoms
|
||
|
||
- **In the UI:** registering a server fails with the browser console error `Failed to load resource: the server responded with a status of 500 ()`.
|
||
- **The server list is empty** for the default tenant, even though the gateway logs show health-check rounds.
|
||
- **Every proxied request** from the default tenant logs a warning on the gateway:
|
||
`tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved (the caller will see no data)`.
|
||
|
||
## Evidence
|
||
|
||
**Gateway log for one failed registration (2026-10-29 16:07:29 UTC, request `9f0021b407a1080ccd50dd860744d647`; the same sequence appeared at 13:15:06 and about 13:19):**
|
||
|
||
```text
|
||
[warning] tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved
|
||
[info] Server 2d10e0e1-... auth check: required=True
|
||
[info] Validating API key for server 2d10e0e1-...
|
||
[info] API key validated successfully for server 2d10e0e1-...
|
||
[info] Encrypted API key for server: 2d10e0e1-...
|
||
[error] Failed to create server: ... InsufficientPrivilegeError: new row violates row-level security policy for table "servers"
|
||
[INSERT INTO servers (... tenant_id ...) VALUES (... '6ada663eb5b88053a9e43723' ...)]
|
||
[error] Database error: 500: Failed to create server
|
||
[info] http_request_completed method=POST path=/api/v1/admin/servers status_code=500
|
||
```
|
||
|
||
The connection test (`POST /api/v1/admin/servers/test-connection`) passes, and the gateway's authenticated
|
||
`tools/list` call to the MCP server succeeds. The failure happens only at the database insert.
|
||
|
||
**Gateway configuration (Deployment `env`, set by Helm values):**
|
||
|
||
| Variable | Value |
|
||
|---|---|
|
||
| `MCP_GATEWAY_TENANCY_MODE` | `single` |
|
||
| `DEFAULT_TENANT_ID` | `default` |
|
||
|
||
**Provisioning activity:**
|
||
- **No tenant registrations:** the gateway logged no `POST /api/v1/admin/tenants` call in the 6 hours checked.
|
||
- **No provisioning runs:** the platform worker (`kyn-agent-builder-platform-worker`, queue `platform-tenant-lifecycle`) has registered `provision_mcp_gateway_tenant`, but logged no provisioning run since it started at 13:55 UTC.
|
||
|
||
**The MCP server under test (`alz-discovery-sample`, namespace `mcp-servers`) is healthy:**
|
||
- `/health` reports healthy.
|
||
- Unauthenticated calls get 401.
|
||
- With its key: `initialize`, `tools/list` (20 tools) and `tools/call` all succeed from inside the cluster and from the gateway pod.
|
||
|
||
## Root cause: traced end to end
|
||
|
||
All paths below are relative to `modernization-platform/packages/`.
|
||
|
||
### 1. KAB creates the default tenant without provisioning it
|
||
|
||
`agent-builder-api/src/database/init_database.py` (lines 78–110) runs at API startup (`main.py`, "Initialize
|
||
database with required collections and default teams"). If the `tenants` collection is empty, it inserts
|
||
"Default Organization" directly into MongoDB. The tenant id is the MongoDB ObjectId returned by `insert_one`, which
|
||
is the 24-hex form of `6ada663eb5b88053a9e43723`. The code states the gap itself:
|
||
|
||
> This tenant never runs the provisioning workflow, which is what normally writes these fields ...
|
||
|
||
It makes up for one consequence (it stamps `database_name` and `temporal_namespace`), but not for the missing MCP
|
||
Gateway tenant. `init_default_data.py` (the CLI seeder, `init_default_user_and_team`) inserts its "Default Tenant"
|
||
the same way, with no reference to provisioning or the gateway.
|
||
|
||
### 2. Normal tenants are provisioned, including the gateway
|
||
|
||
`agent-builder-api/src/api/tenant_routes.py`, `POST /api/v1/tenants/`, calls:
|
||
1. **`db.create_tenant`,** with `provisioning_status="pending"`;
|
||
2. **`_provisioning_input`,** which passes `tenant_slug` in `config` for the gateway;
|
||
3. **`_start_provisioning`,** which starts `TenantProvisioningWorkflow` on `platform-tenant-lifecycle`.
|
||
|
||
Among that workflow's activities is
|
||
`agent-builder-temporal-workers/src/temporal/activities/tenant_lifecycle/mcp_gateway_mgmt.py`,
|
||
`provision_mcp_gateway_tenant`. It posts to the gateway directly:
|
||
|
||
```json
|
||
POST {MCP_GATEWAY_URL}/api/v1/admin/tenants (X-Internal-Service-Token)
|
||
{"id": "<tenant slug>", "name": "<tenant name>", "external_id": "<KAB tenant id>", "status": "active"}
|
||
```
|
||
|
||
Its docstring records the history: "Until this step existed the row had to be created by hand
|
||
(local-dev-kit/scripts/link-mcp-tenants.sh), so a freshly provisioned tenant opened the MCP pages to nothing."
|
||
|
||
### 3. The gateway resolves tenants only through that registry
|
||
|
||
`agent-builder-mcp-gateway/src/mcp_gateway/core/tenancy.py`, `resolve_tenant_ref` (around line 305), maps the
|
||
tenant reference KAB sends to a gateway tenant by `Tenant.id == ref OR Tenant.external_id == ref`. If nothing
|
||
matches, it returns the reference unchanged, on purpose: "binding it matches no rows, so an unknown tenant sees
|
||
nothing instead of another tenant's data." The gateway's only tenant is `default` (from `DEFAULT_TENANT_ID`), whose
|
||
`external_id` doesn't hold `6ada663eb5b88053a9e43723`. So the new server row is stamped
|
||
`tenant_id = '6ada663eb5b88053a9e43723'`.
|
||
|
||
### 4. Row-level security rejects that row
|
||
|
||
`agent-builder-mcp-gateway/alembic/versions/20260723_1000_tenancy_full_rollout.py` puts `servers` (and `tools`,
|
||
`mcp_tools`, `resources`, `prompts` and others) under the catalog policy:
|
||
|
||
```sql
|
||
EFFECTIVE_TENANT := COALESCE(NULLIF(current_setting('app.current_tenant', true), ''), 'default')
|
||
CREATE POLICY tenant_catalog ON servers
|
||
USING (tenant_id IS NULL OR tenant_id = EFFECTIVE_TENANT)
|
||
WITH CHECK (tenant_id = EFFECTIVE_TENANT OR (tenant_id IS NULL AND EFFECTIVE_TENANT = 'default'))
|
||
```
|
||
|
||
In `single` mode, `SingleTenantStrategy` sets no session tenant, so `EFFECTIVE_TENANT` is `'default'`. The row's
|
||
`'6ada663eb5b88053a9e43723'` fails the check, Postgres raises `InsufficientPrivilegeError`, and the API returns 500.
|
||
The same unresolved tenant is why the server list shows nothing.
|
||
|
||
### 5. Why it works locally and not in AKS
|
||
|
||
The dev kit runs `agent-builder-local-dev-kit/scripts/link-mcp-tenants.sh`. It reads the platform's tenants from
|
||
MongoDB, maps the **first** one (the bootstrap default) to gateway tenant `default`, sets `external_id` to its `_id`,
|
||
and maps every later tenant by slug. The AKS Helm values (`agent-builder-kubernetes-deployment/AKS/helm-umbrella-chart/v2.2.0/values_files/az_dev.yaml`
|
||
around line 1303, and v2.3.0 `az_staging.yaml`) configure only the gateway connection for
|
||
`provision_mcp_gateway_tenant`, which covers new tenants. **Nothing in the AKS installation links the bootstrap
|
||
tenant.**
|
||
|
||
## Impact
|
||
|
||
- **No MCP servers in the default tenant:** none can be registered, and none would show to its users.
|
||
- **No working agents there:** agents can't be given an MCP key (`MCP_AGENT_API_KEY`), so any agent that declares MCP tools fails every request at startup with `MCP_AGENT_API_KEY is not set, so the MCP gateway will refuse the agent`.
|
||
- **Blocks:** the CP1 Agentic Landing Zone Discovery deployment, in both patterns: `cp1-agents` (orchestrator) and `cp1-agents-individual` (eight agents).
|
||
- **Likely other environments too:** any environment installed from the AKS or EKS charts with a bootstrap default tenant has the same gap unless it was linked by hand.
|
||
|
||
## Remediation
|
||
|
||
### Immediate (no deploy, no Helm change)
|
||
|
||
Make the call the provisioning activity would have made, through KAB's proxy, as a **KAB platform admin**. The
|
||
proxy only forwards `api/v1/admin/tenants` to callers with platform-admin rights; see
|
||
`agent-builder-api/src/api/routes/mcp_gateway_proxy.py`, `_CROSS_TENANT_PATTERNS`.
|
||
|
||
```sh
|
||
curl -sS -X POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/tenants \
|
||
-H "Authorization: Bearer <platform admin session token>" -H "Content-Type: application/json" \
|
||
-d '{"id": "default", "name": "Default Organization", "external_id": "6ada663eb5b88053a9e43723", "status": "active"}'
|
||
```
|
||
|
||
The endpoint is an idempotent upsert (`agent-builder-mcp-gateway/src/mcp_gateway/api/endpoints/tenant_admin.py`,
|
||
`upsert_tenant`). It keeps the gateway's `default` tenant and records KAB's id on it, exactly as
|
||
`link-mcp-tenants.sh` does locally. `DEFAULT_TENANT_ID` doesn't need to change.
|
||
|
||
**Don't** instead set `DEFAULT_TENANT_ID=6ada663eb5b88053a9e43723`. That names a gateway tenant row that doesn't
|
||
exist, and Helm would revert a manual change.
|
||
|
||
### Permanent (platform change)
|
||
|
||
Options, best first:
|
||
|
||
1. **Register the bootstrap tenant during bootstrap.** When `init_database.py` creates or finds the bootstrap tenant, make the same `POST /api/v1/admin/tenants` call as `provision_mcp_gateway_tenant`, with gateway id `default`. Alternatively, start the provisioning workflow's gateway step for it.
|
||
2. **Add a post-install / post-upgrade Helm hook Job** that does what `link-mcp-tenants.sh` does, using the gateway API rather than direct SQL.
|
||
3. **Reconcile on a schedule:** a platform-worker job that upserts a gateway tenant for every active KAB tenant with no `external_id` match, which also repairs any tenant whose provisioning failed.
|
||
|
||
## Verification
|
||
|
||
1. **After the upsert,** `GET .../api/v1/mcp-gateway/api/v1/admin/tenants` lists `default` with `external_id: 6ada663eb5b88053a9e43723`.
|
||
2. **The warning stops:** new requests from the default tenant no longer log `matches no tenant id or external_id`.
|
||
3. **Registration succeeds:** registering `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` returns 2xx, and the server appears in the list.
|
||
4. **Agents get keys:** creating an agent issues an `MCP_AGENT_API_KEY`, and that agent's requests no longer fail at startup.
|
||
|
||
## Related finding
|
||
|
||
The gateway's OIDC just-in-time provisioning failed for `jonathan.boniface@kyndryl.com` with a unique-key violation
|
||
on `users` (`[OIDCVerifier] JIT provisioning failed ... UniqueViolationError`). The user probably exists already under
|
||
another id. `OIDC_DEFAULT_ROLES` is `["viewer"]`. Check this user's gateway role after the tenant fix, in case server
|
||
registration then fails on permissions instead.
|
||
|
||
## Method
|
||
|
||
- **Read-only checks against `crucible-kab-dev-aks`:** gateway, API and platform-worker logs, and Deployment `env` (plain values only).
|
||
- **Code reading in `modernization-platform` @ `7adfc87f`.**
|
||
- **Nothing in `core-components` was changed.**
|
||
- **No Secret values or database credentials were read:** an attempt to query the gateway database directly was refused, and was not retried.
|
||
|
||
## Addendum (2026-10-29, 17:45 UTC): why the 500 persists, and a second defect
|
||
|
||
**The error is unchanged.** At 17:38:32 `POST /api/v1/admin/servers` failed again with `new row violates row-level security policy for table "servers"`. At 17:38–17:40 the gateway also logged `tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved`. The KAB default tenant is still unmapped.
|
||
|
||
**The one tenant upsert seen didn't touch it.** `POST /api/v1/admin/tenants` → 200 at 17:06:28 came from `kyn-agent-builder-platform-worker` (10.20.10.17), so it was a `TenantProvisioningWorkflow` for a newly created tenant. That activity (`temporal-workers/.../tenant_lifecycle/mcp_gateway_mgmt.py`) upserts `id = <tenant slug>` and `external_id = <platform id>`. It doesn't create a mapping for `6ada663eb5b88053a9e43723`.
|
||
|
||
### The full code path
|
||
|
||
| Step | Code | What happens |
|
||
|---|---|---|
|
||
| 1 | `agent-builder-api/src/api/routes/mcp_gateway_proxy.py` `_identity_context` | Forwards `X-Tenant-ID` = the tenant selected in the UI (`6ada…`). It sends `default` only when a platform admin has **no** tenant selected. |
|
||
| 2 | `mcp-gateway/src/mcp_gateway/gateway/auth.py` (service-token path) | `resolve_tenant_ref` → `bind_tenant`. An unmapped reference is bound **as-is**. |
|
||
| 3 | `mcp-gateway/src/mcp_gateway/db/repositories.py` `ServerRepository.create` | If the bound tenant ≠ `DEFAULT_TENANT_ID` (`default`), it stamps `servers.tenant_id = <bound tenant>`; otherwise it stamps `NULL` (global). |
|
||
| 4 | `mcp-gateway/src/mcp_gateway/core/tenancy.py` `SingleTenantStrategy` | Single mode **never** issues `set_config('app.current_tenant', …)`. |
|
||
| 5 | `alembic/versions/20260723_1000_tenancy_full_rollout.py` `tenant_catalog` | `WITH CHECK (tenant_id = EFFECTIVE OR (tenant_id IS NULL AND EFFECTIVE = 'default'))`. With no setting, EFFECTIVE is always `'default'`. |
|
||
|
||
So in `MCP_GATEWAY_TENANCY_MODE=single`, **a server insert succeeds only when the bound tenant is exactly `default`**. That gives a second defect, beyond the missing mapping:
|
||
|
||
- **Unmapped tenant** (today): `tenant_id='6ada…'` ≠ `'default'`, so it's refused.
|
||
- **Mapped tenant whose gateway id isn't `default`** (what `TenantProvisioningWorkflow` creates for every new tenant, id = slug): `tenant_id='<slug>'` ≠ `'default'`, so it's refused too. In single mode, server registration fails for **every tenant except one mapped to `id='default'`**.
|
||
|
||
The `link-default-tenant` Job (an upsert of `{id:"default", external_id:"6ada…"}`) still fixes the KAB default tenant. It does nothing for other tenants.
|
||
|
||
### Options
|
||
|
||
| # | Option | Who | Effect |
|
||
|---|---|---|---|
|
||
| A | **Register from "All tenants".** A platform admin clears the tenant selector in the UI, so the proxy sends `X-Tenant-ID: default` and the server is stored as a global entry (`tenant_id NULL`). | Platform admin (`is_platform_admin`) | Works today, with no cluster change. The server is global; in single mode every tenant sees all global servers. |
|
||
| B | Map the default tenant: apply `secrets.yaml` and `mcp-gateway-link-default-tenant.job.yaml`, or a Global Admin calls `POST /api/v1/mcp-gateway/api/v1/admin/tenants` with `{"id":"default","external_id":"6ada663eb5b88053a9e43723",…}` | Global Admin, or approval for a `core-components` change | Registration works with the default tenant selected. |
|
||
| C | Platform fix: in single mode, make `ServerRepository.create` stamp `NULL` regardless of the bound tenant, **or** run the gateway in `pooled` mode (needs the restricted DB role and Alembic, per `docs/TENANCY.md`) | Platform team | Fixes registration for every tenant. Single mode then matches its documented promise ("byte-identical to the pre-tenancy behavior"). |
|
||
| D | Platform fix: the bootstrap in `init_database.py` should run `provision_mcp_gateway_tenant` for the default tenant, with `id = DEFAULT_TENANT_ID` | Platform team | Stops a fresh install from shipping an unmapped default tenant. |
|