ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 KiB
KAB dev: MCP server registration fails with HTTP 500 for the default tenant
Moved here on 2026-10-29. Written while the agents were in two separate repositories, both now in this one:
cp1-agents→sub-agents/ALZ-F01: Discovery Workflow Agent/, andcp1-agents-individual→individual-agents/ALZ-F01: Discovery Workflow Agent/.
| Environment | kab-dev (AKS crucible-kab-dev-aks), platform from modernization-platform @ 7adfc87f (2026-10-19) |
| Component | MCP Gateway (kyn-agent-builder-mcp-gateway, namespace core-components, Helm release kyn, chart agent-builder-mcp-gateway-0.1.0) |
| Reported | 2026-10-29 |
| Severity | High: no MCP server can be registered in the default tenant, so no agent that uses MCP tools can be created or run there |
| Status | Root cause confirmed; one-call remediation identified; permanent fix needed in the platform bootstrap |
Summary
Registering an MCP server in the KAB UI fails with 500 Internal Server Error on
POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/servers. The MCP server itself is healthy, and
its key is accepted.
The failure is a gap in platform provisioning. The MCP Gateway keeps its own tenant registry, and the gateway tenant
for KAB's default tenant (6ada663eb5b88053a9e43723) has never been linked to it. KAB creates its default tenant
at startup without running the provisioning workflow. That workflow is the only automated path that creates the
matching gateway tenant. Each request from the default tenant therefore resolves to no gateway tenant, and Postgres
row-level security refuses the insert.
Locally, the dev kit closes this gap with scripts/link-mcp-tenants.sh. The AKS installation has no equivalent step.
Immediate fix: one idempotent call, made as a KAB platform admin, that sets the gateway's default tenant's
external_id to the KAB default tenant id (see Remediation). Permanent fix: register the
bootstrap tenant with the gateway as part of platform bootstrap.
Symptoms
- In the UI: registering a server fails with the browser console error
Failed to load resource: the server responded with a status of 500 (). - The server list is empty for the default tenant, even though the gateway logs show health-check rounds.
- Every proxied request from the default tenant logs a warning on the gateway:
tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved (the caller will see no data).
Evidence
Gateway log for one failed registration (2026-10-29 16:07:29 UTC, request 9f0021b407a1080ccd50dd860744d647; the same sequence appeared at 13:15:06 and about 13:19):
[warning] tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved
[info] Server 2d10e0e1-... auth check: required=True
[info] Validating API key for server 2d10e0e1-...
[info] API key validated successfully for server 2d10e0e1-...
[info] Encrypted API key for server: 2d10e0e1-...
[error] Failed to create server: ... InsufficientPrivilegeError: new row violates row-level security policy for table "servers"
[INSERT INTO servers (... tenant_id ...) VALUES (... '6ada663eb5b88053a9e43723' ...)]
[error] Database error: 500: Failed to create server
[info] http_request_completed method=POST path=/api/v1/admin/servers status_code=500
The connection test (POST /api/v1/admin/servers/test-connection) passes, and the gateway's authenticated
tools/list call to the MCP server succeeds. The failure happens only at the database insert.
Gateway configuration (Deployment env, set by Helm values):
| Variable | Value |
|---|---|
MCP_GATEWAY_TENANCY_MODE |
single |
DEFAULT_TENANT_ID |
default |
Provisioning activity:
- No tenant registrations: the gateway logged no
POST /api/v1/admin/tenantscall in the 6 hours checked. - No provisioning runs: the platform worker (
kyn-agent-builder-platform-worker, queueplatform-tenant-lifecycle) has registeredprovision_mcp_gateway_tenant, but logged no provisioning run since it started at 13:55 UTC.
The MCP server under test (alz-discovery-sample, namespace mcp-servers) is healthy:
/healthreports healthy.- Unauthenticated calls get 401.
- With its key:
initialize,tools/list(20 tools) andtools/callall succeed from inside the cluster and from the gateway pod.
Root cause: traced end to end
All paths below are relative to modernization-platform/packages/.
1. KAB creates the default tenant without provisioning it
agent-builder-api/src/database/init_database.py (lines 78–110) runs at API startup (main.py, "Initialize
database with required collections and default teams"). If the tenants collection is empty, it inserts
"Default Organization" directly into MongoDB. The tenant id is the MongoDB ObjectId returned by insert_one, which
is the 24-hex form of 6ada663eb5b88053a9e43723. The code states the gap itself:
This tenant never runs the provisioning workflow, which is what normally writes these fields ...
It makes up for one consequence (it stamps database_name and temporal_namespace), but not for the missing MCP
Gateway tenant. init_default_data.py (the CLI seeder, init_default_user_and_team) inserts its "Default Tenant"
the same way, with no reference to provisioning or the gateway.
2. Normal tenants are provisioned, including the gateway
agent-builder-api/src/api/tenant_routes.py, POST /api/v1/tenants/, calls:
db.create_tenant, withprovisioning_status="pending";_provisioning_input, which passestenant_sluginconfigfor the gateway;_start_provisioning, which startsTenantProvisioningWorkflowonplatform-tenant-lifecycle.
Among that workflow's activities is
agent-builder-temporal-workers/src/temporal/activities/tenant_lifecycle/mcp_gateway_mgmt.py,
provision_mcp_gateway_tenant. It posts to the gateway directly:
POST {MCP_GATEWAY_URL}/api/v1/admin/tenants (X-Internal-Service-Token)
{"id": "<tenant slug>", "name": "<tenant name>", "external_id": "<KAB tenant id>", "status": "active"}
Its docstring records the history: "Until this step existed the row had to be created by hand (local-dev-kit/scripts/link-mcp-tenants.sh), so a freshly provisioned tenant opened the MCP pages to nothing."
3. The gateway resolves tenants only through that registry
agent-builder-mcp-gateway/src/mcp_gateway/core/tenancy.py, resolve_tenant_ref (around line 305), maps the
tenant reference KAB sends to a gateway tenant by Tenant.id == ref OR Tenant.external_id == ref. If nothing
matches, it returns the reference unchanged, on purpose: "binding it matches no rows, so an unknown tenant sees
nothing instead of another tenant's data." The gateway's only tenant is default (from DEFAULT_TENANT_ID), whose
external_id doesn't hold 6ada663eb5b88053a9e43723. So the new server row is stamped
tenant_id = '6ada663eb5b88053a9e43723'.
4. Row-level security rejects that row
agent-builder-mcp-gateway/alembic/versions/20260723_1000_tenancy_full_rollout.py puts servers (and tools,
mcp_tools, resources, prompts and others) under the catalog policy:
EFFECTIVE_TENANT := COALESCE(NULLIF(current_setting('app.current_tenant', true), ''), 'default')
CREATE POLICY tenant_catalog ON servers
USING (tenant_id IS NULL OR tenant_id = EFFECTIVE_TENANT)
WITH CHECK (tenant_id = EFFECTIVE_TENANT OR (tenant_id IS NULL AND EFFECTIVE_TENANT = 'default'))
In single mode, SingleTenantStrategy sets no session tenant, so EFFECTIVE_TENANT is 'default'. The row's
'6ada663eb5b88053a9e43723' fails the check, Postgres raises InsufficientPrivilegeError, and the API returns 500.
The same unresolved tenant is why the server list shows nothing.
5. Why it works locally and not in AKS
The dev kit runs agent-builder-local-dev-kit/scripts/link-mcp-tenants.sh. It reads the platform's tenants from
MongoDB, maps the first one (the bootstrap default) to gateway tenant default, sets external_id to its _id,
and maps every later tenant by slug. The AKS Helm values (agent-builder-kubernetes-deployment/AKS/helm-umbrella-chart/v2.2.0/values_files/az_dev.yaml
around line 1303, and v2.3.0 az_staging.yaml) configure only the gateway connection for
provision_mcp_gateway_tenant, which covers new tenants. Nothing in the AKS installation links the bootstrap
tenant.
Impact
- No MCP servers in the default tenant: none can be registered, and none would show to its users.
- No working agents there: agents can't be given an MCP key (
MCP_AGENT_API_KEY), so any agent that declares MCP tools fails every request at startup withMCP_AGENT_API_KEY is not set, so the MCP gateway will refuse the agent. - Blocks: the CP1 Agentic Landing Zone Discovery deployment, in both patterns:
cp1-agents(orchestrator) andcp1-agents-individual(eight agents). - Likely other environments too: any environment installed from the AKS or EKS charts with a bootstrap default tenant has the same gap unless it was linked by hand.
Remediation
Immediate (no deploy, no Helm change)
Make the call the provisioning activity would have made, through KAB's proxy, as a KAB platform admin. The
proxy only forwards api/v1/admin/tenants to callers with platform-admin rights; see
agent-builder-api/src/api/routes/mcp_gateway_proxy.py, _CROSS_TENANT_PATTERNS.
curl -sS -X POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/tenants \
-H "Authorization: Bearer <platform admin session token>" -H "Content-Type: application/json" \
-d '{"id": "default", "name": "Default Organization", "external_id": "6ada663eb5b88053a9e43723", "status": "active"}'
The endpoint is an idempotent upsert (agent-builder-mcp-gateway/src/mcp_gateway/api/endpoints/tenant_admin.py,
upsert_tenant). It keeps the gateway's default tenant and records KAB's id on it, exactly as
link-mcp-tenants.sh does locally. DEFAULT_TENANT_ID doesn't need to change.
Don't instead set DEFAULT_TENANT_ID=6ada663eb5b88053a9e43723. That names a gateway tenant row that doesn't
exist, and Helm would revert a manual change.
Permanent (platform change)
Options, best first:
- Register the bootstrap tenant during bootstrap. When
init_database.pycreates or finds the bootstrap tenant, make the samePOST /api/v1/admin/tenantscall asprovision_mcp_gateway_tenant, with gateway iddefault. Alternatively, start the provisioning workflow's gateway step for it. - Add a post-install / post-upgrade Helm hook Job that does what
link-mcp-tenants.shdoes, using the gateway API rather than direct SQL. - Reconcile on a schedule: a platform-worker job that upserts a gateway tenant for every active KAB tenant with no
external_idmatch, which also repairs any tenant whose provisioning failed.
Verification
- After the upsert,
GET .../api/v1/mcp-gateway/api/v1/admin/tenantslistsdefaultwithexternal_id: 6ada663eb5b88053a9e43723. - The warning stops: new requests from the default tenant no longer log
matches no tenant id or external_id. - Registration succeeds: registering
http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110returns 2xx, and the server appears in the list. - Agents get keys: creating an agent issues an
MCP_AGENT_API_KEY, and that agent's requests no longer fail at startup.
Related finding
The gateway's OIDC just-in-time provisioning failed for jonathan.boniface@kyndryl.com with a unique-key violation
on users ([OIDCVerifier] JIT provisioning failed ... UniqueViolationError). The user probably exists already under
another id. OIDC_DEFAULT_ROLES is ["viewer"]. Check this user's gateway role after the tenant fix, in case server
registration then fails on permissions instead.
Method
- Read-only checks against
crucible-kab-dev-aks: gateway, API and platform-worker logs, and Deploymentenv(plain values only). - Code reading in
modernization-platform@7adfc87f. - Nothing in
core-componentswas changed. - No Secret values or database credentials were read: an attempt to query the gateway database directly was refused, and was not retried.
Addendum (2026-10-29, 17:45 UTC): why the 500 persists, and a second defect
The error is unchanged. At 17:38:32 POST /api/v1/admin/servers failed again with new row violates row-level security policy for table "servers". At 17:38–17:40 the gateway also logged tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved. The KAB default tenant is still unmapped.
The one tenant upsert seen didn't touch it. POST /api/v1/admin/tenants → 200 at 17:06:28 came from kyn-agent-builder-platform-worker (10.20.10.17), so it was a TenantProvisioningWorkflow for a newly created tenant. That activity (temporal-workers/.../tenant_lifecycle/mcp_gateway_mgmt.py) upserts id = <tenant slug> and external_id = <platform id>. It doesn't create a mapping for 6ada663eb5b88053a9e43723.
The full code path
| Step | Code | What happens |
|---|---|---|
| 1 | agent-builder-api/src/api/routes/mcp_gateway_proxy.py _identity_context |
Forwards X-Tenant-ID = the tenant selected in the UI (6ada…). It sends default only when a platform admin has no tenant selected. |
| 2 | mcp-gateway/src/mcp_gateway/gateway/auth.py (service-token path) |
resolve_tenant_ref → bind_tenant. An unmapped reference is bound as-is. |
| 3 | mcp-gateway/src/mcp_gateway/db/repositories.py ServerRepository.create |
If the bound tenant ≠ DEFAULT_TENANT_ID (default), it stamps servers.tenant_id = <bound tenant>; otherwise it stamps NULL (global). |
| 4 | mcp-gateway/src/mcp_gateway/core/tenancy.py SingleTenantStrategy |
Single mode never issues set_config('app.current_tenant', …). |
| 5 | alembic/versions/20260723_1000_tenancy_full_rollout.py tenant_catalog |
WITH CHECK (tenant_id = EFFECTIVE OR (tenant_id IS NULL AND EFFECTIVE = 'default')). With no setting, EFFECTIVE is always 'default'. |
So in MCP_GATEWAY_TENANCY_MODE=single, a server insert succeeds only when the bound tenant is exactly default. That gives a second defect, beyond the missing mapping:
- Unmapped tenant (today):
tenant_id='6ada…'≠'default', so it's refused. - Mapped tenant whose gateway id isn't
default(whatTenantProvisioningWorkflowcreates for every new tenant, id = slug):tenant_id='<slug>'≠'default', so it's refused too. In single mode, server registration fails for every tenant except one mapped toid='default'.
The link-default-tenant Job (an upsert of {id:"default", external_id:"6ada…"}) still fixes the KAB default tenant. It does nothing for other tenants.
Options
| # | Option | Who | Effect |
|---|---|---|---|
| A | Register from "All tenants". A platform admin clears the tenant selector in the UI, so the proxy sends X-Tenant-ID: default and the server is stored as a global entry (tenant_id NULL). |
Platform admin (is_platform_admin) |
Works today, with no cluster change. The server is global; in single mode every tenant sees all global servers. |
| B | Map the default tenant: apply secrets.yaml and mcp-gateway-link-default-tenant.job.yaml, or a Global Admin calls POST /api/v1/mcp-gateway/api/v1/admin/tenants with {"id":"default","external_id":"6ada663eb5b88053a9e43723",…} |
Global Admin, or approval for a core-components change |
Registration works with the default tenant selected. |
| C | Platform fix: in single mode, make ServerRepository.create stamp NULL regardless of the bound tenant, or run the gateway in pooled mode (needs the restricted DB role and Alembic, per docs/TENANCY.md) |
Platform team | Fixes registration for every tenant. Single mode then matches its documented promise ("byte-identical to the pre-tenancy behavior"). |
| D | Platform fix: the bootstrap in init_database.py should run provision_mcp_gateway_tenant for the default tenant, with id = DEFAULT_TENANT_ID |
Platform team | Stops a fresh install from shipping an unmapped default tenant. |