Add CP1 Discovery agents in both patterns
All checks were successful
check agents / check (push) Successful in 10s
All checks were successful
check agents / check (push) Successful in 10s
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
12
docs/platform-issues/mcp-gateway-default-tenant/README.md
Normal file
12
docs/platform-issues/mcp-gateway-default-tenant/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# MCP gateway: registering a server fails with a 500
|
||||
|
||||
Open since 2026-10-29, on `crucible-kab-dev-aks`. The full analysis is in [report-2026-10-29.md](report-2026-10-29.md).
|
||||
|
||||
| File | What it is | Applied? |
|
||||
|---|---|---|
|
||||
| [report-2026-10-29.md](report-2026-10-29.md) | Root cause, code path, evidence and options | — |
|
||||
| `kyn-agent-builder-mcp-gateway.deployment.yaml` | The gateway Deployment, cleaned up (`DEFAULT_TENANT_ID=default`, no `TENANT_ID`); no different from the live one | No change needed |
|
||||
| `secrets.yaml` | Secret `kab-tenant-id`. It holds KAB's default tenant id, which is an identifier and not a credential. | **No** |
|
||||
| `mcp-gateway-link-default-tenant.job.yaml` | A one-off Job that maps KAB's default tenant to the gateway tenant `default` | **No.** It changes `core-components`, so it needs approval. |
|
||||
|
||||
The Job fixes registration only while the default tenant is selected. In the gateway's single tenancy mode, tenants that KAB provisions later still fail. See option C in the report.
|
||||
@@ -0,0 +1,153 @@
|
||||
# MCP Gateway Deployment for kab-dev (namespace core-components), from the live object of Helm release `kyn`.
|
||||
#
|
||||
# Server-generated fields (status, uid, resourceVersion, creationTimestamp, generation, the revision annotation)
|
||||
# are removed so this file can be applied. Helm owns this object: a manual apply is overwritten by the next
|
||||
# `helm upgrade`, so make lasting changes in the release's values file too.
|
||||
#
|
||||
# Tenancy (see documents/kab-dev-mcp-gateway-default-tenant-500-2026-10-29.md):
|
||||
# - DEFAULT_TENANT_ID is read by the gateway (settings.default_tenant_id) and must stay "default": the database
|
||||
# default and the row-level security policies hard-code 'default' as the effective tenant in single mode.
|
||||
# - KAB's default tenant id is NOT configured here. The gateway maps it through the tenant row's external_id,
|
||||
# which mcp-gateway-link-default-tenant.job.yaml sets. A TENANT_ID variable is not read by the gateway, and the
|
||||
# earlier draft's reference to a Secret named "default" would have stopped the pod starting
|
||||
# (CreateContainerConfigError).
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
annotations:
|
||||
meta.helm.sh/release-name: kyn
|
||||
meta.helm.sh/release-namespace: core-components
|
||||
labels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
app.kubernetes.io/version: latest
|
||||
helm.sh/chart: agent-builder-mcp-gateway-0.1.0
|
||||
name: kyn-agent-builder-mcp-gateway
|
||||
namespace: core-components
|
||||
spec:
|
||||
progressDeadlineSeconds: 600
|
||||
replicas: 1
|
||||
revisionHistoryLimit: 10
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 25%
|
||||
maxUnavailable: 25%
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
app.kubernetes.io/version: latest
|
||||
helm.sh/chart: agent-builder-mcp-gateway-0.1.0
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: ENVIRONMENT
|
||||
value: production
|
||||
- name: DEBUG
|
||||
value: "false"
|
||||
- name: HOST
|
||||
value: 0.0.0.0
|
||||
- name: PORT
|
||||
value: "8000"
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-database-url
|
||||
name: agent-builder-secrets
|
||||
- name: INTERNAL_SERVICE_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: token
|
||||
name: internal-service-secret
|
||||
- name: DATABASE_POOL_SIZE
|
||||
value: "20"
|
||||
- name: REDIS_URL
|
||||
value: redis://redis.kab-data.svc.cluster.local:6379/0
|
||||
- name: REDIS_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: redis-password
|
||||
name: agent-builder-secrets
|
||||
- name: JWT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-jwt-secret
|
||||
name: agent-builder-secrets
|
||||
- name: MCP_ENCRYPTION_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-encryption-key
|
||||
name: agent-builder-secrets
|
||||
- name: AUTH_ENABLED
|
||||
value: "true"
|
||||
- name: CORS_ORIGINS
|
||||
value: '["https://app.kab-dev.kyndemo.live"]'
|
||||
- name: OIDC_ENABLED
|
||||
value: "true"
|
||||
- name: OIDC_AUTHORITY
|
||||
value: https://id.kab-dev.kyndemo.live/realms/kab
|
||||
- name: OIDC_AUDIENCE
|
||||
value: account
|
||||
- name: OIDC_DEFAULT_ROLES
|
||||
value: '["viewer"]'
|
||||
- name: ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-admin-password
|
||||
name: agent-builder-secrets
|
||||
- name: MCP_GATEWAY_TENANCY_MODE
|
||||
value: single
|
||||
# The gateway's own default tenant id. Keep "default": the RLS policies and the column default use it.
|
||||
- name: DEFAULT_TENANT_ID
|
||||
value: default
|
||||
image: cruciblecoredevacr.azurecr.io/kab/agent-builder-mcp-gateway:v3.1.1
|
||||
imagePullPolicy: Always
|
||||
livenessProbe:
|
||||
failureThreshold: 5
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: 8000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 15
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 5
|
||||
name: agent-builder-mcp-gateway
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
name: http
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 3
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: 8000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
dnsPolicy: ClusterFirst
|
||||
restartPolicy: Always
|
||||
schedulerName: default-scheduler
|
||||
securityContext: {}
|
||||
serviceAccount: default
|
||||
serviceAccountName: default
|
||||
terminationGracePeriodSeconds: 30
|
||||
@@ -0,0 +1,66 @@
|
||||
# Link KAB's default tenant to the MCP Gateway's "default" tenant (kab-dev, namespace core-components).
|
||||
#
|
||||
# KAB creates its default tenant at startup without the provisioning workflow, so the workflow's
|
||||
# provision_mcp_gateway_tenant step never ran for it. Without that link, every MCP server registration from the
|
||||
# default tenant fails row-level security (HTTP 500). This Job makes the same idempotent call the step makes,
|
||||
# with the same credential:
|
||||
#
|
||||
# POST /api/v1/admin/tenants X-Internal-Service-Token: <internal-service-secret/token>
|
||||
# {"id": "default", "name": "Default Organization", "external_id": <kab-tenant-id/tenant-id>, "status": "active"}
|
||||
#
|
||||
# The gateway tenant id stays "default" (its RLS policies use it); KAB's id becomes its external_id, which is what
|
||||
# the gateway resolves proxied requests by. It is the AKS equivalent of the dev kit's link-mcp-tenants.sh.
|
||||
#
|
||||
# Apply secrets.yaml first. Safe to run again: the endpoint is an upsert. The Job deletes itself an hour after it
|
||||
# finishes. Neither key is printed: the token is sent as a header, and only the response body is logged.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: mcp-gateway-link-default-tenant
|
||||
namespace: core-components
|
||||
labels:
|
||||
app.kubernetes.io/part-of: kab
|
||||
app.kubernetes.io/component: mcp-gateway-tenant-link
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/component: mcp-gateway-tenant-link
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: link
|
||||
image: docker.io/curlimages/curl:8.11.1
|
||||
env:
|
||||
- name: INTERNAL_SERVICE_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: internal-service-secret
|
||||
key: token
|
||||
- name: KAB_TENANT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kab-tenant-id
|
||||
key: tenant-id
|
||||
- name: GATEWAY_URL
|
||||
value: http://kyn-agent-builder-mcp-gateway.core-components.svc.cluster.local:8000
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
body=$(printf '{"id":"default","name":"Default Organization","external_id":"%s","status":"active"}' "$KAB_TENANT_ID")
|
||||
code=$(curl -sS -o /tmp/response -w '%{http_code}' -X POST "$GATEWAY_URL/api/v1/admin/tenants" \
|
||||
-H "X-Internal-Service-Token: $INTERNAL_SERVICE_TOKEN" -H "Content-Type: application/json" -d "$body")
|
||||
echo "upsert: HTTP $code"; cat /tmp/response; echo
|
||||
[ "$code" -lt 300 ] || exit 1
|
||||
curl -sS "$GATEWAY_URL/api/v1/admin/tenants" -H "X-Internal-Service-Token: $INTERNAL_SERVICE_TOKEN"; echo
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
@@ -0,0 +1,231 @@
|
||||
# KAB dev: MCP server registration fails with HTTP 500 for the default tenant
|
||||
|
||||
> **Moved here on 2026-10-29.** Written while the agents were in two separate repositories, both now in this one:
|
||||
> `cp1-agents` → `sub-agents/ALZ-F01: Discovery Workflow Agent/`, and `cp1-agents-individual` → `individual-agents/ALZ-F01: Discovery Workflow Agent/`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Environment | kab-dev (AKS `crucible-kab-dev-aks`), platform from `modernization-platform` @ `7adfc87f` (2026-10-19) |
|
||||
| Component | MCP Gateway (`kyn-agent-builder-mcp-gateway`, namespace `core-components`, Helm release `kyn`, chart `agent-builder-mcp-gateway-0.1.0`) |
|
||||
| Reported | 2026-10-29 |
|
||||
| Severity | High: no MCP server can be registered in the default tenant, so no agent that uses MCP tools can be created or run there |
|
||||
| Status | Root cause confirmed; one-call remediation identified; permanent fix needed in the platform bootstrap |
|
||||
|
||||
## Summary
|
||||
|
||||
Registering an MCP server in the KAB UI fails with `500 Internal Server Error` on
|
||||
`POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/servers`. The MCP server itself is healthy, and
|
||||
its key is accepted.
|
||||
|
||||
The failure is a gap in platform provisioning. The MCP Gateway keeps its own tenant registry, and the gateway tenant
|
||||
for KAB's **default tenant** (`6ada663eb5b88053a9e43723`) has never been linked to it. KAB creates its default tenant
|
||||
at startup without running the provisioning workflow. That workflow is the only automated path that creates the
|
||||
matching gateway tenant. Each request from the default tenant therefore resolves to no gateway tenant, and Postgres
|
||||
row-level security refuses the insert.
|
||||
|
||||
Locally, the dev kit closes this gap with `scripts/link-mcp-tenants.sh`. The AKS installation has no equivalent step.
|
||||
|
||||
**Immediate fix:** one idempotent call, made as a KAB platform admin, that sets the gateway's `default` tenant's
|
||||
`external_id` to the KAB default tenant id (see [Remediation](#remediation)). **Permanent fix:** register the
|
||||
bootstrap tenant with the gateway as part of platform bootstrap.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- **In the UI:** registering a server fails with the browser console error `Failed to load resource: the server responded with a status of 500 ()`.
|
||||
- **The server list is empty** for the default tenant, even though the gateway logs show health-check rounds.
|
||||
- **Every proxied request** from the default tenant logs a warning on the gateway:
|
||||
`tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved (the caller will see no data)`.
|
||||
|
||||
## Evidence
|
||||
|
||||
**Gateway log for one failed registration (2026-10-29 16:07:29 UTC, request `9f0021b407a1080ccd50dd860744d647`; the same sequence appeared at 13:15:06 and about 13:19):**
|
||||
|
||||
```text
|
||||
[warning] tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved
|
||||
[info] Server 2d10e0e1-... auth check: required=True
|
||||
[info] Validating API key for server 2d10e0e1-...
|
||||
[info] API key validated successfully for server 2d10e0e1-...
|
||||
[info] Encrypted API key for server: 2d10e0e1-...
|
||||
[error] Failed to create server: ... InsufficientPrivilegeError: new row violates row-level security policy for table "servers"
|
||||
[INSERT INTO servers (... tenant_id ...) VALUES (... '6ada663eb5b88053a9e43723' ...)]
|
||||
[error] Database error: 500: Failed to create server
|
||||
[info] http_request_completed method=POST path=/api/v1/admin/servers status_code=500
|
||||
```
|
||||
|
||||
The connection test (`POST /api/v1/admin/servers/test-connection`) passes, and the gateway's authenticated
|
||||
`tools/list` call to the MCP server succeeds. The failure happens only at the database insert.
|
||||
|
||||
**Gateway configuration (Deployment `env`, set by Helm values):**
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `MCP_GATEWAY_TENANCY_MODE` | `single` |
|
||||
| `DEFAULT_TENANT_ID` | `default` |
|
||||
|
||||
**Provisioning activity:**
|
||||
- **No tenant registrations:** the gateway logged no `POST /api/v1/admin/tenants` call in the 6 hours checked.
|
||||
- **No provisioning runs:** the platform worker (`kyn-agent-builder-platform-worker`, queue `platform-tenant-lifecycle`) has registered `provision_mcp_gateway_tenant`, but logged no provisioning run since it started at 13:55 UTC.
|
||||
|
||||
**The MCP server under test (`alz-discovery-sample`, namespace `mcp-servers`) is healthy:**
|
||||
- `/health` reports healthy.
|
||||
- Unauthenticated calls get 401.
|
||||
- With its key: `initialize`, `tools/list` (20 tools) and `tools/call` all succeed from inside the cluster and from the gateway pod.
|
||||
|
||||
## Root cause: traced end to end
|
||||
|
||||
All paths below are relative to `modernization-platform/packages/`.
|
||||
|
||||
### 1. KAB creates the default tenant without provisioning it
|
||||
|
||||
`agent-builder-api/src/database/init_database.py` (lines 78–110) runs at API startup (`main.py`, "Initialize
|
||||
database with required collections and default teams"). If the `tenants` collection is empty, it inserts
|
||||
"Default Organization" directly into MongoDB. The tenant id is the MongoDB ObjectId returned by `insert_one`, which
|
||||
is the 24-hex form of `6ada663eb5b88053a9e43723`. The code states the gap itself:
|
||||
|
||||
> This tenant never runs the provisioning workflow, which is what normally writes these fields ...
|
||||
|
||||
It makes up for one consequence (it stamps `database_name` and `temporal_namespace`), but not for the missing MCP
|
||||
Gateway tenant. `init_default_data.py` (the CLI seeder, `init_default_user_and_team`) inserts its "Default Tenant"
|
||||
the same way, with no reference to provisioning or the gateway.
|
||||
|
||||
### 2. Normal tenants are provisioned, including the gateway
|
||||
|
||||
`agent-builder-api/src/api/tenant_routes.py`, `POST /api/v1/tenants/`, calls:
|
||||
1. **`db.create_tenant`,** with `provisioning_status="pending"`;
|
||||
2. **`_provisioning_input`,** which passes `tenant_slug` in `config` for the gateway;
|
||||
3. **`_start_provisioning`,** which starts `TenantProvisioningWorkflow` on `platform-tenant-lifecycle`.
|
||||
|
||||
Among that workflow's activities is
|
||||
`agent-builder-temporal-workers/src/temporal/activities/tenant_lifecycle/mcp_gateway_mgmt.py`,
|
||||
`provision_mcp_gateway_tenant`. It posts to the gateway directly:
|
||||
|
||||
```json
|
||||
POST {MCP_GATEWAY_URL}/api/v1/admin/tenants (X-Internal-Service-Token)
|
||||
{"id": "<tenant slug>", "name": "<tenant name>", "external_id": "<KAB tenant id>", "status": "active"}
|
||||
```
|
||||
|
||||
Its docstring records the history: "Until this step existed the row had to be created by hand
|
||||
(local-dev-kit/scripts/link-mcp-tenants.sh), so a freshly provisioned tenant opened the MCP pages to nothing."
|
||||
|
||||
### 3. The gateway resolves tenants only through that registry
|
||||
|
||||
`agent-builder-mcp-gateway/src/mcp_gateway/core/tenancy.py`, `resolve_tenant_ref` (around line 305), maps the
|
||||
tenant reference KAB sends to a gateway tenant by `Tenant.id == ref OR Tenant.external_id == ref`. If nothing
|
||||
matches, it returns the reference unchanged, on purpose: "binding it matches no rows, so an unknown tenant sees
|
||||
nothing instead of another tenant's data." The gateway's only tenant is `default` (from `DEFAULT_TENANT_ID`), whose
|
||||
`external_id` doesn't hold `6ada663eb5b88053a9e43723`. So the new server row is stamped
|
||||
`tenant_id = '6ada663eb5b88053a9e43723'`.
|
||||
|
||||
### 4. Row-level security rejects that row
|
||||
|
||||
`agent-builder-mcp-gateway/alembic/versions/20260723_1000_tenancy_full_rollout.py` puts `servers` (and `tools`,
|
||||
`mcp_tools`, `resources`, `prompts` and others) under the catalog policy:
|
||||
|
||||
```sql
|
||||
EFFECTIVE_TENANT := COALESCE(NULLIF(current_setting('app.current_tenant', true), ''), 'default')
|
||||
CREATE POLICY tenant_catalog ON servers
|
||||
USING (tenant_id IS NULL OR tenant_id = EFFECTIVE_TENANT)
|
||||
WITH CHECK (tenant_id = EFFECTIVE_TENANT OR (tenant_id IS NULL AND EFFECTIVE_TENANT = 'default'))
|
||||
```
|
||||
|
||||
In `single` mode, `SingleTenantStrategy` sets no session tenant, so `EFFECTIVE_TENANT` is `'default'`. The row's
|
||||
`'6ada663eb5b88053a9e43723'` fails the check, Postgres raises `InsufficientPrivilegeError`, and the API returns 500.
|
||||
The same unresolved tenant is why the server list shows nothing.
|
||||
|
||||
### 5. Why it works locally and not in AKS
|
||||
|
||||
The dev kit runs `agent-builder-local-dev-kit/scripts/link-mcp-tenants.sh`. It reads the platform's tenants from
|
||||
MongoDB, maps the **first** one (the bootstrap default) to gateway tenant `default`, sets `external_id` to its `_id`,
|
||||
and maps every later tenant by slug. The AKS Helm values (`agent-builder-kubernetes-deployment/AKS/helm-umbrella-chart/v2.2.0/values_files/az_dev.yaml`
|
||||
around line 1303, and v2.3.0 `az_staging.yaml`) configure only the gateway connection for
|
||||
`provision_mcp_gateway_tenant`, which covers new tenants. **Nothing in the AKS installation links the bootstrap
|
||||
tenant.**
|
||||
|
||||
## Impact
|
||||
|
||||
- **No MCP servers in the default tenant:** none can be registered, and none would show to its users.
|
||||
- **No working agents there:** agents can't be given an MCP key (`MCP_AGENT_API_KEY`), so any agent that declares MCP tools fails every request at startup with `MCP_AGENT_API_KEY is not set, so the MCP gateway will refuse the agent`.
|
||||
- **Blocks:** the CP1 Agentic Landing Zone Discovery deployment, in both patterns: `cp1-agents` (orchestrator) and `cp1-agents-individual` (eight agents).
|
||||
- **Likely other environments too:** any environment installed from the AKS or EKS charts with a bootstrap default tenant has the same gap unless it was linked by hand.
|
||||
|
||||
## Remediation
|
||||
|
||||
### Immediate (no deploy, no Helm change)
|
||||
|
||||
Make the call the provisioning activity would have made, through KAB's proxy, as a **KAB platform admin**. The
|
||||
proxy only forwards `api/v1/admin/tenants` to callers with platform-admin rights; see
|
||||
`agent-builder-api/src/api/routes/mcp_gateway_proxy.py`, `_CROSS_TENANT_PATTERNS`.
|
||||
|
||||
```sh
|
||||
curl -sS -X POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/tenants \
|
||||
-H "Authorization: Bearer <platform admin session token>" -H "Content-Type: application/json" \
|
||||
-d '{"id": "default", "name": "Default Organization", "external_id": "6ada663eb5b88053a9e43723", "status": "active"}'
|
||||
```
|
||||
|
||||
The endpoint is an idempotent upsert (`agent-builder-mcp-gateway/src/mcp_gateway/api/endpoints/tenant_admin.py`,
|
||||
`upsert_tenant`). It keeps the gateway's `default` tenant and records KAB's id on it, exactly as
|
||||
`link-mcp-tenants.sh` does locally. `DEFAULT_TENANT_ID` doesn't need to change.
|
||||
|
||||
**Don't** instead set `DEFAULT_TENANT_ID=6ada663eb5b88053a9e43723`. That names a gateway tenant row that doesn't
|
||||
exist, and Helm would revert a manual change.
|
||||
|
||||
### Permanent (platform change)
|
||||
|
||||
Options, best first:
|
||||
|
||||
1. **Register the bootstrap tenant during bootstrap.** When `init_database.py` creates or finds the bootstrap tenant, make the same `POST /api/v1/admin/tenants` call as `provision_mcp_gateway_tenant`, with gateway id `default`. Alternatively, start the provisioning workflow's gateway step for it.
|
||||
2. **Add a post-install / post-upgrade Helm hook Job** that does what `link-mcp-tenants.sh` does, using the gateway API rather than direct SQL.
|
||||
3. **Reconcile on a schedule:** a platform-worker job that upserts a gateway tenant for every active KAB tenant with no `external_id` match, which also repairs any tenant whose provisioning failed.
|
||||
|
||||
## Verification
|
||||
|
||||
1. **After the upsert,** `GET .../api/v1/mcp-gateway/api/v1/admin/tenants` lists `default` with `external_id: 6ada663eb5b88053a9e43723`.
|
||||
2. **The warning stops:** new requests from the default tenant no longer log `matches no tenant id or external_id`.
|
||||
3. **Registration succeeds:** registering `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` returns 2xx, and the server appears in the list.
|
||||
4. **Agents get keys:** creating an agent issues an `MCP_AGENT_API_KEY`, and that agent's requests no longer fail at startup.
|
||||
|
||||
## Related finding
|
||||
|
||||
The gateway's OIDC just-in-time provisioning failed for `jonathan.boniface@kyndryl.com` with a unique-key violation
|
||||
on `users` (`[OIDCVerifier] JIT provisioning failed ... UniqueViolationError`). The user probably exists already under
|
||||
another id. `OIDC_DEFAULT_ROLES` is `["viewer"]`. Check this user's gateway role after the tenant fix, in case server
|
||||
registration then fails on permissions instead.
|
||||
|
||||
## Method
|
||||
|
||||
- **Read-only checks against `crucible-kab-dev-aks`:** gateway, API and platform-worker logs, and Deployment `env` (plain values only).
|
||||
- **Code reading in `modernization-platform` @ `7adfc87f`.**
|
||||
- **Nothing in `core-components` was changed.**
|
||||
- **No Secret values or database credentials were read:** an attempt to query the gateway database directly was refused, and was not retried.
|
||||
|
||||
## Addendum (2026-10-29, 17:45 UTC): why the 500 persists, and a second defect
|
||||
|
||||
**The error is unchanged.** At 17:38:32 `POST /api/v1/admin/servers` failed again with `new row violates row-level security policy for table "servers"`. At 17:38–17:40 the gateway also logged `tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved`. The KAB default tenant is still unmapped.
|
||||
|
||||
**The one tenant upsert seen didn't touch it.** `POST /api/v1/admin/tenants` → 200 at 17:06:28 came from `kyn-agent-builder-platform-worker` (10.20.10.17), so it was a `TenantProvisioningWorkflow` for a newly created tenant. That activity (`temporal-workers/.../tenant_lifecycle/mcp_gateway_mgmt.py`) upserts `id = <tenant slug>` and `external_id = <platform id>`. It doesn't create a mapping for `6ada663eb5b88053a9e43723`.
|
||||
|
||||
### The full code path
|
||||
|
||||
| Step | Code | What happens |
|
||||
|---|---|---|
|
||||
| 1 | `agent-builder-api/src/api/routes/mcp_gateway_proxy.py` `_identity_context` | Forwards `X-Tenant-ID` = the tenant selected in the UI (`6ada…`). It sends `default` only when a platform admin has **no** tenant selected. |
|
||||
| 2 | `mcp-gateway/src/mcp_gateway/gateway/auth.py` (service-token path) | `resolve_tenant_ref` → `bind_tenant`. An unmapped reference is bound **as-is**. |
|
||||
| 3 | `mcp-gateway/src/mcp_gateway/db/repositories.py` `ServerRepository.create` | If the bound tenant ≠ `DEFAULT_TENANT_ID` (`default`), it stamps `servers.tenant_id = <bound tenant>`; otherwise it stamps `NULL` (global). |
|
||||
| 4 | `mcp-gateway/src/mcp_gateway/core/tenancy.py` `SingleTenantStrategy` | Single mode **never** issues `set_config('app.current_tenant', …)`. |
|
||||
| 5 | `alembic/versions/20260723_1000_tenancy_full_rollout.py` `tenant_catalog` | `WITH CHECK (tenant_id = EFFECTIVE OR (tenant_id IS NULL AND EFFECTIVE = 'default'))`. With no setting, EFFECTIVE is always `'default'`. |
|
||||
|
||||
So in `MCP_GATEWAY_TENANCY_MODE=single`, **a server insert succeeds only when the bound tenant is exactly `default`**. That gives a second defect, beyond the missing mapping:
|
||||
|
||||
- **Unmapped tenant** (today): `tenant_id='6ada…'` ≠ `'default'`, so it's refused.
|
||||
- **Mapped tenant whose gateway id isn't `default`** (what `TenantProvisioningWorkflow` creates for every new tenant, id = slug): `tenant_id='<slug>'` ≠ `'default'`, so it's refused too. In single mode, server registration fails for **every tenant except one mapped to `id='default'`**.
|
||||
|
||||
The `link-default-tenant` Job (an upsert of `{id:"default", external_id:"6ada…"}`) still fixes the KAB default tenant. It does nothing for other tenants.
|
||||
|
||||
### Options
|
||||
|
||||
| # | Option | Who | Effect |
|
||||
|---|---|---|---|
|
||||
| A | **Register from "All tenants".** A platform admin clears the tenant selector in the UI, so the proxy sends `X-Tenant-ID: default` and the server is stored as a global entry (`tenant_id NULL`). | Platform admin (`is_platform_admin`) | Works today, with no cluster change. The server is global; in single mode every tenant sees all global servers. |
|
||||
| B | Map the default tenant: apply `secrets.yaml` and `mcp-gateway-link-default-tenant.job.yaml`, or a Global Admin calls `POST /api/v1/mcp-gateway/api/v1/admin/tenants` with `{"id":"default","external_id":"6ada663eb5b88053a9e43723",…}` | Global Admin, or approval for a `core-components` change | Registration works with the default tenant selected. |
|
||||
| C | Platform fix: in single mode, make `ServerRepository.create` stamp `NULL` regardless of the bound tenant, **or** run the gateway in `pooled` mode (needs the restricted DB role and Alembic, per `docs/TENANCY.md`) | Platform team | Fixes registration for every tenant. Single mode then matches its documented promise ("byte-identical to the pre-tenancy behavior"). |
|
||||
| D | Platform fix: the bootstrap in `init_database.py` should run `provision_mcp_gateway_tenant` for the default tenant, with `id = DEFAULT_TENANT_ID` | Platform team | Stops a fresh install from shipping an unmapped default tenant. |
|
||||
15
docs/platform-issues/mcp-gateway-default-tenant/secrets.yaml
Normal file
15
docs/platform-issues/mcp-gateway-default-tenant/secrets.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# KAB's default tenant id, for mcp-gateway-link-default-tenant.job.yaml.
|
||||
#
|
||||
# It is an identifier, not a credential: it appears in every KAB request and in the gateway's logs. It is kept in
|
||||
# a Secret only so the Job reads it the same way it reads the internal service token. stringData takes the plain
|
||||
# value; Kubernetes stores it base64-encoded under data. The gateway Deployment does not read it.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: kab
|
||||
name: kab-tenant-id
|
||||
namespace: core-components
|
||||
type: Opaque
|
||||
stringData:
|
||||
tenant-id: "6ada663eb5b88053a9e43723"
|
||||
Reference in New Issue
Block a user