Add CP1 Discovery agents in both patterns
All checks were successful
check agents / check (push) Successful in 10s
All checks were successful
check agents / check (push) Successful in 10s
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
# Workflow Artifacts step: can't select an earlier step's output within the same run (and how hand-offs reach the next agent)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-10-29 |
|
||||
| Environment | KAB dev (`crucible-kab-dev-aks`), Temporal workers as in `modernization-platform` @ `c89aedcb9` |
|
||||
| Component | `agent-builder-temporal-workers`: the generic workflow executor, and the `select_artifacts` activity ("Artifacts" step) |
|
||||
| Severity | Medium. Nothing fails, but artifacts can't be used to hand work between the steps of one run, and the step can't be left unconfigured in a reusable workflow. |
|
||||
| Status | Open; reported to the platform team for confirmation |
|
||||
| Found by | Code review while building the CP1 Discovery workflows (`sub-agents/` and `individual-agents/`, ALZ-F01) |
|
||||
|
||||
## Summary
|
||||
|
||||
You can place an Artifacts step (`select_artifacts`) anywhere in a workflow, including between two agent steps. Within one run, though, it can't select anything an earlier step of that same run produced. The executor registers a run's step outputs as project artifacts only **after every step has finished**. While the run is in progress, its own outputs aren't artifacts yet.
|
||||
|
||||
Step-to-step data within a run therefore has to travel through `from_steps`. That works, and both CP1 Discovery workflows rely on it. But it means the Artifacts step is useful only for artifacts that **existed before the run started**, and those have to be named by id, fixed in the workflow definition.
|
||||
|
||||
## What we expected
|
||||
|
||||
For workflows to use Artifacts steps to hand work between agents. For example, a step selecting "the inventory produced by step s3 of this run" for later steps, with the artifact's type, provenance and size limits applied. This is how the artifact design reads (`PROJECT_WORKFLOW_ARTIFACT_DESIGN.md`).
|
||||
|
||||
## What happens
|
||||
|
||||
| # | Behaviour | Code (`packages/agent-builder-temporal-workers/src/temporal/`) |
|
||||
|---|---|---|
|
||||
| 1 | Artifacts for a run are registered once, after every step has run. They're also registered on failure, after the failing step. | `workflows/generic_workflow_executor.py:1025` `await self._execute_workflow()`, then `:1029` `await self._register_artifacts(workflow_input)`; failure path `:1040`; implementation `:2330` |
|
||||
| 2 | The Artifacts step is a pass-through. Its output isn't stored and is never registered as an artifact. | `workflows/generic_workflow_executor.py:1223` `is_pass_through_selection = step_activity == "select_artifacts"`; `:1251` "Skipped storage … pass-through" |
|
||||
| 3 | `select_artifacts` reads only the project's `artifacts` collection. It has no way to refer to "this run" or "step sN". | `activities/artifacts_activities.py:71` `SelectArtifactsActivity` |
|
||||
| 4 | It fails if `artifact_ids` is empty, so a reusable workflow can't ship with the step left unconfigured. | `activities/artifacts_activities.py:98` "No artifact_ids provided" |
|
||||
| 5 | Besides an artifact's `_id`, an entry in `artifact_ids` also matches the artifact's `workflow_id` or `name`. | `activities/artifacts_activities.py:122–124` |
|
||||
|
||||
Behaviours 1 and 3 together mean an Artifacts step placed after step s1 finds no s1 artifact. It finds only artifacts from earlier, finished runs.
|
||||
|
||||
## Impact on the CP1 workflows
|
||||
|
||||
- **Both ALZ-F01 Discovery workflows** pass data between steps with `from_steps` (`s1.tasks`, `s3.records` and so on). Neither needs this fix to run. The committed exports have no Artifacts step. `scripts/kab_import_workflow.py --artifact-id <id>` adds one at the front for artifacts that already exist.
|
||||
- **Large outputs between steps:** with `from_steps`, a step's whole output is passed inline. The individual-agents pattern has no other channel, and it can't use artifacts' size limits or file references within a run (individual-agents ADR 0010).
|
||||
- **Between workflows (F01 → F02 Assessment):** this is where artifacts are needed. It works only once the Discovery run has finished, and only if the F02 workflow names F01's artifacts, by id or by one of the matches in row 5. The id-based route means editing the F02 workflow for every engagement.
|
||||
|
||||
## Workarounds in use
|
||||
|
||||
1. **Within a run:** `from_steps` for every hand-off. The sub-agent pattern also keeps a run workspace on the executions volume (sub-agents ADR 0010).
|
||||
2. **Existing artifacts:** an Artifacts step built with fixed ids (`--artifact-id`), wired only into the step that needs them.
|
||||
3. **Between workflows (not yet tested):** select by an earlier workflow's id, using the `workflow_id` match in row 5, so that a single entry brings in that workflow's outputs. Still to confirm on the dev platform:
|
||||
- Is the stored `workflow_id` the workflow definition's id or the run's?
|
||||
- Does the match return the outputs of every run, or of one run only?
|
||||
|
||||
## How one agent's output reaches the next agent's context
|
||||
|
||||
Traced through the workers (`modernization-platform` @ `c89aedcb9`) and the agent runtime (the repository's `agent-builder-agent-runtime` is 0.37.0; the agents pin 0.35.7). The `temporal-workers` limits below are the live values on `crucible-kab-dev-aks`.
|
||||
|
||||
| Hop | What happens | Code |
|
||||
|---|---|---|
|
||||
| 1. The agent replies | The reply's text is parsed as JSON. If it doesn't parse, the reply is kept as a plain string. | `activities/base_activity.py` `_process_agent_result` (~1200) |
|
||||
| 2. The executor stores it | A step output over **100 KB** is written to the PV, and only a reference is kept in workflow state. This is only to get it through Temporal. | `workflows/generic_workflow_executor.py:1189–1215` |
|
||||
| 3. `from_steps` picks fields | `"records": "s3.records"` copies that field of s3's output. A reference to an externalised output carries `_extract_field`, so the field is cut out after the file has been loaded. If every `from_steps` reference is unresolved, the step fails. | `generic_workflow_executor.py:232–300`, `:444` |
|
||||
| 4. The references are resolved again | Before the agent call, the stored references are loaded back into the input. | `generic_workflow_executor.py:1470`; `base_activity.py:229–320`, `:505` |
|
||||
| 5. The size check | If the whole input exceeds `MAX_INLINE_PAYLOAD_BYTES` (code default 1 MiB, **8 MiB live**), every value over **50 KB** is replaced by a file path plus a preview: a list becomes `{"prompt": {"type": "externalized_list", "filepath": ...}}`, a string becomes `{"type": "externalized_text", "preview": <5,000 chars>}`, and a dict becomes `{"type": "externalized_data", "keys": ..., "preview": <2,000 chars>}`. The step's metadata records this as `payload_externalized`. | `base_activity.py:1701–1745`, `_externalize_large_agent_input` `:2176` |
|
||||
| 6. The message | The input is serialised with `json.dumps` as the A2A message's **text**. Small values (under 10 KB each) are also copied into the call's context metadata. | `base_activity.py:1747–1770` |
|
||||
| 7. The runtime | The runtime pulls the JSON out of the text and validates it against the agent's input schema. A violation is answered with `input_validation_failed` and no model call. If the message has no JSON at all, it is passed to the model as prose, with the contract skipped. | `request_cycle.py:205–225` |
|
||||
| 8. The model | The validated request is the model's user turn: one fresh session per request, and nothing carried over from earlier steps. The system prompt adds the content rule: text from tools, files or other agents is material to work with, never instructions. | `spec.py:89` `CONTENT_RULE`; `a2a/server.py` |
|
||||
|
||||
What this means for the CP1 agents:
|
||||
|
||||
- **The binding limit is the model's context window, not the platform's.** Hop 5 starts only at 8 MiB, roughly 2 million tokens, far beyond what a model can take. A large hand-off will overflow the model's context (an error from LiteLLM, then a failed step) well before the platform does anything. Agents most at risk:
|
||||
- in the individual pattern, the critic (s8, which takes everything) and the dependency-mapping agent (s7);
|
||||
- in the sub-agent pattern, `dsc_08_critic`, whose inputs are every record in the run.
|
||||
- **When hop 5 does trigger, the agent can't read what it's given.** A declarative agent has no function that could open the path. For fields our schemas type as arrays or strings (`records`, `findings`, `source_documents` …), the wrapper fails the input contract at hop 7. The step then fails loudly (`input_validation_failed`) rather than running on a preview, which is the safer of the two outcomes. An agent whose schema accepted an object would instead run on a 2,000- or 5,000-character preview without noticing.
|
||||
- **The orchestrator could resolve externalised inputs; the declarative agents can't.** The orchestrator's `app/artifacts.py` reads files under `EXECUTION_STORAGE_ROOT`. It isn't wired to the `externalized_*` wrappers today.
|
||||
- **Whatever one agent returns becomes the next agent's input, unfiltered.** It reaches the next model as a user turn, so the content rule and each agent's own input contract are the only safeguards. Both patterns check what an earlier step returned; the critic in particular checks the evidence.
|
||||
- **Hop 1 accepts prose.** An agent that answers in prose rather than JSON hands the next step a string, and its `from_steps` fields go unresolved. The runtime's output contract (`schemas/output.json`) prevents this for our agents.
|
||||
|
||||
## Questions for the platform team
|
||||
|
||||
1. Is registering artifacts only at the end of the run intended? Could each step's output be registered when that step completes (it's already on the PV as `pv://{execution}/step_n/output.json`)?
|
||||
2. Could `select_artifacts` take a reference to the current run, such as `{"from_step": "s3"}` or `{"execution": "current", "step": "s3"}`, resolved by the executor?
|
||||
3. Could `artifact_ids` come from the run form, or from `from_workflow`, so that a reusable workflow doesn't need ids fixed when it's built? And could an empty selection be allowed (skip, rather than fail)?
|
||||
4. What does the `workflow_id` match select: one workflow definition's artifacts across all of its runs, or one run's? Is matching by `name` meant for production use, or is it a fallback?
|
||||
5. Could `_externalize_large_agent_input` hand an agent its externalised value as an A2A file part, or as a declared `x-storage-ref` field, instead of a wrapper object the agent's schema can't accept? And could its 50 KB per-value threshold be configured?
|
||||
6. Is there a planned way to limit a hand-off to what fits the receiving agent's model: a per-step token budget, or summarising before the hand-off?
|
||||
|
||||
## Evidence
|
||||
|
||||
Code reading, of `modernization-platform` @ `c89aedcb9`, plus read-only `kubectl get` of the workers' plain environment values. Nothing was run against the platform, and nothing was changed.
|
||||
12
docs/platform-issues/mcp-gateway-default-tenant/README.md
Normal file
12
docs/platform-issues/mcp-gateway-default-tenant/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# MCP gateway: registering a server fails with a 500
|
||||
|
||||
Open since 2026-10-29, on `crucible-kab-dev-aks`. The full analysis is in [report-2026-10-29.md](report-2026-10-29.md).
|
||||
|
||||
| File | What it is | Applied? |
|
||||
|---|---|---|
|
||||
| [report-2026-10-29.md](report-2026-10-29.md) | Root cause, code path, evidence and options | — |
|
||||
| `kyn-agent-builder-mcp-gateway.deployment.yaml` | The gateway Deployment, cleaned up (`DEFAULT_TENANT_ID=default`, no `TENANT_ID`); no different from the live one | No change needed |
|
||||
| `secrets.yaml` | Secret `kab-tenant-id`. It holds KAB's default tenant id, which is an identifier and not a credential. | **No** |
|
||||
| `mcp-gateway-link-default-tenant.job.yaml` | A one-off Job that maps KAB's default tenant to the gateway tenant `default` | **No.** It changes `core-components`, so it needs approval. |
|
||||
|
||||
The Job fixes registration only while the default tenant is selected. In the gateway's single tenancy mode, tenants that KAB provisions later still fail. See option C in the report.
|
||||
@@ -0,0 +1,153 @@
|
||||
# MCP Gateway Deployment for kab-dev (namespace core-components), from the live object of Helm release `kyn`.
|
||||
#
|
||||
# Server-generated fields (status, uid, resourceVersion, creationTimestamp, generation, the revision annotation)
|
||||
# are removed so this file can be applied. Helm owns this object: a manual apply is overwritten by the next
|
||||
# `helm upgrade`, so make lasting changes in the release's values file too.
|
||||
#
|
||||
# Tenancy (see documents/kab-dev-mcp-gateway-default-tenant-500-2026-10-29.md):
|
||||
# - DEFAULT_TENANT_ID is read by the gateway (settings.default_tenant_id) and must stay "default": the database
|
||||
# default and the row-level security policies hard-code 'default' as the effective tenant in single mode.
|
||||
# - KAB's default tenant id is NOT configured here. The gateway maps it through the tenant row's external_id,
|
||||
# which mcp-gateway-link-default-tenant.job.yaml sets. A TENANT_ID variable is not read by the gateway, and the
|
||||
# earlier draft's reference to a Secret named "default" would have stopped the pod starting
|
||||
# (CreateContainerConfigError).
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
annotations:
|
||||
meta.helm.sh/release-name: kyn
|
||||
meta.helm.sh/release-namespace: core-components
|
||||
labels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
app.kubernetes.io/version: latest
|
||||
helm.sh/chart: agent-builder-mcp-gateway-0.1.0
|
||||
name: kyn-agent-builder-mcp-gateway
|
||||
namespace: core-components
|
||||
spec:
|
||||
progressDeadlineSeconds: 600
|
||||
replicas: 1
|
||||
revisionHistoryLimit: 10
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 25%
|
||||
maxUnavailable: 25%
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/instance: kyn
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: agent-builder-mcp-gateway
|
||||
app.kubernetes.io/version: latest
|
||||
helm.sh/chart: agent-builder-mcp-gateway-0.1.0
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: ENVIRONMENT
|
||||
value: production
|
||||
- name: DEBUG
|
||||
value: "false"
|
||||
- name: HOST
|
||||
value: 0.0.0.0
|
||||
- name: PORT
|
||||
value: "8000"
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-database-url
|
||||
name: agent-builder-secrets
|
||||
- name: INTERNAL_SERVICE_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: token
|
||||
name: internal-service-secret
|
||||
- name: DATABASE_POOL_SIZE
|
||||
value: "20"
|
||||
- name: REDIS_URL
|
||||
value: redis://redis.kab-data.svc.cluster.local:6379/0
|
||||
- name: REDIS_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: redis-password
|
||||
name: agent-builder-secrets
|
||||
- name: JWT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-jwt-secret
|
||||
name: agent-builder-secrets
|
||||
- name: MCP_ENCRYPTION_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-encryption-key
|
||||
name: agent-builder-secrets
|
||||
- name: AUTH_ENABLED
|
||||
value: "true"
|
||||
- name: CORS_ORIGINS
|
||||
value: '["https://app.kab-dev.kyndemo.live"]'
|
||||
- name: OIDC_ENABLED
|
||||
value: "true"
|
||||
- name: OIDC_AUTHORITY
|
||||
value: https://id.kab-dev.kyndemo.live/realms/kab
|
||||
- name: OIDC_AUDIENCE
|
||||
value: account
|
||||
- name: OIDC_DEFAULT_ROLES
|
||||
value: '["viewer"]'
|
||||
- name: ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
key: mcp-gateway-admin-password
|
||||
name: agent-builder-secrets
|
||||
- name: MCP_GATEWAY_TENANCY_MODE
|
||||
value: single
|
||||
# The gateway's own default tenant id. Keep "default": the RLS policies and the column default use it.
|
||||
- name: DEFAULT_TENANT_ID
|
||||
value: default
|
||||
image: cruciblecoredevacr.azurecr.io/kab/agent-builder-mcp-gateway:v3.1.1
|
||||
imagePullPolicy: Always
|
||||
livenessProbe:
|
||||
failureThreshold: 5
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: 8000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 15
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 5
|
||||
name: agent-builder-mcp-gateway
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
name: http
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 3
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: 8000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
dnsPolicy: ClusterFirst
|
||||
restartPolicy: Always
|
||||
schedulerName: default-scheduler
|
||||
securityContext: {}
|
||||
serviceAccount: default
|
||||
serviceAccountName: default
|
||||
terminationGracePeriodSeconds: 30
|
||||
@@ -0,0 +1,66 @@
|
||||
# Link KAB's default tenant to the MCP Gateway's "default" tenant (kab-dev, namespace core-components).
|
||||
#
|
||||
# KAB creates its default tenant at startup without the provisioning workflow, so the workflow's
|
||||
# provision_mcp_gateway_tenant step never ran for it. Without that link, every MCP server registration from the
|
||||
# default tenant fails row-level security (HTTP 500). This Job makes the same idempotent call the step makes,
|
||||
# with the same credential:
|
||||
#
|
||||
# POST /api/v1/admin/tenants X-Internal-Service-Token: <internal-service-secret/token>
|
||||
# {"id": "default", "name": "Default Organization", "external_id": <kab-tenant-id/tenant-id>, "status": "active"}
|
||||
#
|
||||
# The gateway tenant id stays "default" (its RLS policies use it); KAB's id becomes its external_id, which is what
|
||||
# the gateway resolves proxied requests by. It is the AKS equivalent of the dev kit's link-mcp-tenants.sh.
|
||||
#
|
||||
# Apply secrets.yaml first. Safe to run again: the endpoint is an upsert. The Job deletes itself an hour after it
|
||||
# finishes. Neither key is printed: the token is sent as a header, and only the response body is logged.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: mcp-gateway-link-default-tenant
|
||||
namespace: core-components
|
||||
labels:
|
||||
app.kubernetes.io/part-of: kab
|
||||
app.kubernetes.io/component: mcp-gateway-tenant-link
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/component: mcp-gateway-tenant-link
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: link
|
||||
image: docker.io/curlimages/curl:8.11.1
|
||||
env:
|
||||
- name: INTERNAL_SERVICE_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: internal-service-secret
|
||||
key: token
|
||||
- name: KAB_TENANT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: kab-tenant-id
|
||||
key: tenant-id
|
||||
- name: GATEWAY_URL
|
||||
value: http://kyn-agent-builder-mcp-gateway.core-components.svc.cluster.local:8000
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
body=$(printf '{"id":"default","name":"Default Organization","external_id":"%s","status":"active"}' "$KAB_TENANT_ID")
|
||||
code=$(curl -sS -o /tmp/response -w '%{http_code}' -X POST "$GATEWAY_URL/api/v1/admin/tenants" \
|
||||
-H "X-Internal-Service-Token: $INTERNAL_SERVICE_TOKEN" -H "Content-Type: application/json" -d "$body")
|
||||
echo "upsert: HTTP $code"; cat /tmp/response; echo
|
||||
[ "$code" -lt 300 ] || exit 1
|
||||
curl -sS "$GATEWAY_URL/api/v1/admin/tenants" -H "X-Internal-Service-Token: $INTERNAL_SERVICE_TOKEN"; echo
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
@@ -0,0 +1,231 @@
|
||||
# KAB dev: MCP server registration fails with HTTP 500 for the default tenant
|
||||
|
||||
> **Moved here on 2026-10-29.** Written while the agents were in two separate repositories, both now in this one:
|
||||
> `cp1-agents` → `sub-agents/ALZ-F01: Discovery Workflow Agent/`, and `cp1-agents-individual` → `individual-agents/ALZ-F01: Discovery Workflow Agent/`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Environment | kab-dev (AKS `crucible-kab-dev-aks`), platform from `modernization-platform` @ `7adfc87f` (2026-10-19) |
|
||||
| Component | MCP Gateway (`kyn-agent-builder-mcp-gateway`, namespace `core-components`, Helm release `kyn`, chart `agent-builder-mcp-gateway-0.1.0`) |
|
||||
| Reported | 2026-10-29 |
|
||||
| Severity | High: no MCP server can be registered in the default tenant, so no agent that uses MCP tools can be created or run there |
|
||||
| Status | Root cause confirmed; one-call remediation identified; permanent fix needed in the platform bootstrap |
|
||||
|
||||
## Summary
|
||||
|
||||
Registering an MCP server in the KAB UI fails with `500 Internal Server Error` on
|
||||
`POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/servers`. The MCP server itself is healthy, and
|
||||
its key is accepted.
|
||||
|
||||
The failure is a gap in platform provisioning. The MCP Gateway keeps its own tenant registry, and the gateway tenant
|
||||
for KAB's **default tenant** (`6ada663eb5b88053a9e43723`) has never been linked to it. KAB creates its default tenant
|
||||
at startup without running the provisioning workflow. That workflow is the only automated path that creates the
|
||||
matching gateway tenant. Each request from the default tenant therefore resolves to no gateway tenant, and Postgres
|
||||
row-level security refuses the insert.
|
||||
|
||||
Locally, the dev kit closes this gap with `scripts/link-mcp-tenants.sh`. The AKS installation has no equivalent step.
|
||||
|
||||
**Immediate fix:** one idempotent call, made as a KAB platform admin, that sets the gateway's `default` tenant's
|
||||
`external_id` to the KAB default tenant id (see [Remediation](#remediation)). **Permanent fix:** register the
|
||||
bootstrap tenant with the gateway as part of platform bootstrap.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- **In the UI:** registering a server fails with the browser console error `Failed to load resource: the server responded with a status of 500 ()`.
|
||||
- **The server list is empty** for the default tenant, even though the gateway logs show health-check rounds.
|
||||
- **Every proxied request** from the default tenant logs a warning on the gateway:
|
||||
`tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved (the caller will see no data)`.
|
||||
|
||||
## Evidence
|
||||
|
||||
**Gateway log for one failed registration (2026-10-29 16:07:29 UTC, request `9f0021b407a1080ccd50dd860744d647`; the same sequence appeared at 13:15:06 and about 13:19):**
|
||||
|
||||
```text
|
||||
[warning] tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved
|
||||
[info] Server 2d10e0e1-... auth check: required=True
|
||||
[info] Validating API key for server 2d10e0e1-...
|
||||
[info] API key validated successfully for server 2d10e0e1-...
|
||||
[info] Encrypted API key for server: 2d10e0e1-...
|
||||
[error] Failed to create server: ... InsufficientPrivilegeError: new row violates row-level security policy for table "servers"
|
||||
[INSERT INTO servers (... tenant_id ...) VALUES (... '6ada663eb5b88053a9e43723' ...)]
|
||||
[error] Database error: 500: Failed to create server
|
||||
[info] http_request_completed method=POST path=/api/v1/admin/servers status_code=500
|
||||
```
|
||||
|
||||
The connection test (`POST /api/v1/admin/servers/test-connection`) passes, and the gateway's authenticated
|
||||
`tools/list` call to the MCP server succeeds. The failure happens only at the database insert.
|
||||
|
||||
**Gateway configuration (Deployment `env`, set by Helm values):**
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `MCP_GATEWAY_TENANCY_MODE` | `single` |
|
||||
| `DEFAULT_TENANT_ID` | `default` |
|
||||
|
||||
**Provisioning activity:**
|
||||
- **No tenant registrations:** the gateway logged no `POST /api/v1/admin/tenants` call in the 6 hours checked.
|
||||
- **No provisioning runs:** the platform worker (`kyn-agent-builder-platform-worker`, queue `platform-tenant-lifecycle`) has registered `provision_mcp_gateway_tenant`, but logged no provisioning run since it started at 13:55 UTC.
|
||||
|
||||
**The MCP server under test (`alz-discovery-sample`, namespace `mcp-servers`) is healthy:**
|
||||
- `/health` reports healthy.
|
||||
- Unauthenticated calls get 401.
|
||||
- With its key: `initialize`, `tools/list` (20 tools) and `tools/call` all succeed from inside the cluster and from the gateway pod.
|
||||
|
||||
## Root cause: traced end to end
|
||||
|
||||
All paths below are relative to `modernization-platform/packages/`.
|
||||
|
||||
### 1. KAB creates the default tenant without provisioning it
|
||||
|
||||
`agent-builder-api/src/database/init_database.py` (lines 78–110) runs at API startup (`main.py`, "Initialize
|
||||
database with required collections and default teams"). If the `tenants` collection is empty, it inserts
|
||||
"Default Organization" directly into MongoDB. The tenant id is the MongoDB ObjectId returned by `insert_one`, which
|
||||
is the 24-hex form of `6ada663eb5b88053a9e43723`. The code states the gap itself:
|
||||
|
||||
> This tenant never runs the provisioning workflow, which is what normally writes these fields ...
|
||||
|
||||
It makes up for one consequence (it stamps `database_name` and `temporal_namespace`), but not for the missing MCP
|
||||
Gateway tenant. `init_default_data.py` (the CLI seeder, `init_default_user_and_team`) inserts its "Default Tenant"
|
||||
the same way, with no reference to provisioning or the gateway.
|
||||
|
||||
### 2. Normal tenants are provisioned, including the gateway
|
||||
|
||||
`agent-builder-api/src/api/tenant_routes.py`, `POST /api/v1/tenants/`, calls:
|
||||
1. **`db.create_tenant`,** with `provisioning_status="pending"`;
|
||||
2. **`_provisioning_input`,** which passes `tenant_slug` in `config` for the gateway;
|
||||
3. **`_start_provisioning`,** which starts `TenantProvisioningWorkflow` on `platform-tenant-lifecycle`.
|
||||
|
||||
Among that workflow's activities is
|
||||
`agent-builder-temporal-workers/src/temporal/activities/tenant_lifecycle/mcp_gateway_mgmt.py`,
|
||||
`provision_mcp_gateway_tenant`. It posts to the gateway directly:
|
||||
|
||||
```json
|
||||
POST {MCP_GATEWAY_URL}/api/v1/admin/tenants (X-Internal-Service-Token)
|
||||
{"id": "<tenant slug>", "name": "<tenant name>", "external_id": "<KAB tenant id>", "status": "active"}
|
||||
```
|
||||
|
||||
Its docstring records the history: "Until this step existed the row had to be created by hand
|
||||
(local-dev-kit/scripts/link-mcp-tenants.sh), so a freshly provisioned tenant opened the MCP pages to nothing."
|
||||
|
||||
### 3. The gateway resolves tenants only through that registry
|
||||
|
||||
`agent-builder-mcp-gateway/src/mcp_gateway/core/tenancy.py`, `resolve_tenant_ref` (around line 305), maps the
|
||||
tenant reference KAB sends to a gateway tenant by `Tenant.id == ref OR Tenant.external_id == ref`. If nothing
|
||||
matches, it returns the reference unchanged, on purpose: "binding it matches no rows, so an unknown tenant sees
|
||||
nothing instead of another tenant's data." The gateway's only tenant is `default` (from `DEFAULT_TENANT_ID`), whose
|
||||
`external_id` doesn't hold `6ada663eb5b88053a9e43723`. So the new server row is stamped
|
||||
`tenant_id = '6ada663eb5b88053a9e43723'`.
|
||||
|
||||
### 4. Row-level security rejects that row
|
||||
|
||||
`agent-builder-mcp-gateway/alembic/versions/20260723_1000_tenancy_full_rollout.py` puts `servers` (and `tools`,
|
||||
`mcp_tools`, `resources`, `prompts` and others) under the catalog policy:
|
||||
|
||||
```sql
|
||||
EFFECTIVE_TENANT := COALESCE(NULLIF(current_setting('app.current_tenant', true), ''), 'default')
|
||||
CREATE POLICY tenant_catalog ON servers
|
||||
USING (tenant_id IS NULL OR tenant_id = EFFECTIVE_TENANT)
|
||||
WITH CHECK (tenant_id = EFFECTIVE_TENANT OR (tenant_id IS NULL AND EFFECTIVE_TENANT = 'default'))
|
||||
```
|
||||
|
||||
In `single` mode, `SingleTenantStrategy` sets no session tenant, so `EFFECTIVE_TENANT` is `'default'`. The row's
|
||||
`'6ada663eb5b88053a9e43723'` fails the check, Postgres raises `InsufficientPrivilegeError`, and the API returns 500.
|
||||
The same unresolved tenant is why the server list shows nothing.
|
||||
|
||||
### 5. Why it works locally and not in AKS
|
||||
|
||||
The dev kit runs `agent-builder-local-dev-kit/scripts/link-mcp-tenants.sh`. It reads the platform's tenants from
|
||||
MongoDB, maps the **first** one (the bootstrap default) to gateway tenant `default`, sets `external_id` to its `_id`,
|
||||
and maps every later tenant by slug. The AKS Helm values (`agent-builder-kubernetes-deployment/AKS/helm-umbrella-chart/v2.2.0/values_files/az_dev.yaml`
|
||||
around line 1303, and v2.3.0 `az_staging.yaml`) configure only the gateway connection for
|
||||
`provision_mcp_gateway_tenant`, which covers new tenants. **Nothing in the AKS installation links the bootstrap
|
||||
tenant.**
|
||||
|
||||
## Impact
|
||||
|
||||
- **No MCP servers in the default tenant:** none can be registered, and none would show to its users.
|
||||
- **No working agents there:** agents can't be given an MCP key (`MCP_AGENT_API_KEY`), so any agent that declares MCP tools fails every request at startup with `MCP_AGENT_API_KEY is not set, so the MCP gateway will refuse the agent`.
|
||||
- **Blocks:** the CP1 Agentic Landing Zone Discovery deployment, in both patterns: `cp1-agents` (orchestrator) and `cp1-agents-individual` (eight agents).
|
||||
- **Likely other environments too:** any environment installed from the AKS or EKS charts with a bootstrap default tenant has the same gap unless it was linked by hand.
|
||||
|
||||
## Remediation
|
||||
|
||||
### Immediate (no deploy, no Helm change)
|
||||
|
||||
Make the call the provisioning activity would have made, through KAB's proxy, as a **KAB platform admin**. The
|
||||
proxy only forwards `api/v1/admin/tenants` to callers with platform-admin rights; see
|
||||
`agent-builder-api/src/api/routes/mcp_gateway_proxy.py`, `_CROSS_TENANT_PATTERNS`.
|
||||
|
||||
```sh
|
||||
curl -sS -X POST https://api.kab-dev.kyndemo.live/api/v1/mcp-gateway/api/v1/admin/tenants \
|
||||
-H "Authorization: Bearer <platform admin session token>" -H "Content-Type: application/json" \
|
||||
-d '{"id": "default", "name": "Default Organization", "external_id": "6ada663eb5b88053a9e43723", "status": "active"}'
|
||||
```
|
||||
|
||||
The endpoint is an idempotent upsert (`agent-builder-mcp-gateway/src/mcp_gateway/api/endpoints/tenant_admin.py`,
|
||||
`upsert_tenant`). It keeps the gateway's `default` tenant and records KAB's id on it, exactly as
|
||||
`link-mcp-tenants.sh` does locally. `DEFAULT_TENANT_ID` doesn't need to change.
|
||||
|
||||
**Don't** instead set `DEFAULT_TENANT_ID=6ada663eb5b88053a9e43723`. That names a gateway tenant row that doesn't
|
||||
exist, and Helm would revert a manual change.
|
||||
|
||||
### Permanent (platform change)
|
||||
|
||||
Options, best first:
|
||||
|
||||
1. **Register the bootstrap tenant during bootstrap.** When `init_database.py` creates or finds the bootstrap tenant, make the same `POST /api/v1/admin/tenants` call as `provision_mcp_gateway_tenant`, with gateway id `default`. Alternatively, start the provisioning workflow's gateway step for it.
|
||||
2. **Add a post-install / post-upgrade Helm hook Job** that does what `link-mcp-tenants.sh` does, using the gateway API rather than direct SQL.
|
||||
3. **Reconcile on a schedule:** a platform-worker job that upserts a gateway tenant for every active KAB tenant with no `external_id` match, which also repairs any tenant whose provisioning failed.
|
||||
|
||||
## Verification
|
||||
|
||||
1. **After the upsert,** `GET .../api/v1/mcp-gateway/api/v1/admin/tenants` lists `default` with `external_id: 6ada663eb5b88053a9e43723`.
|
||||
2. **The warning stops:** new requests from the default tenant no longer log `matches no tenant id or external_id`.
|
||||
3. **Registration succeeds:** registering `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` returns 2xx, and the server appears in the list.
|
||||
4. **Agents get keys:** creating an agent issues an `MCP_AGENT_API_KEY`, and that agent's requests no longer fail at startup.
|
||||
|
||||
## Related finding
|
||||
|
||||
The gateway's OIDC just-in-time provisioning failed for `jonathan.boniface@kyndryl.com` with a unique-key violation
|
||||
on `users` (`[OIDCVerifier] JIT provisioning failed ... UniqueViolationError`). The user probably exists already under
|
||||
another id. `OIDC_DEFAULT_ROLES` is `["viewer"]`. Check this user's gateway role after the tenant fix, in case server
|
||||
registration then fails on permissions instead.
|
||||
|
||||
## Method
|
||||
|
||||
- **Read-only checks against `crucible-kab-dev-aks`:** gateway, API and platform-worker logs, and Deployment `env` (plain values only).
|
||||
- **Code reading in `modernization-platform` @ `7adfc87f`.**
|
||||
- **Nothing in `core-components` was changed.**
|
||||
- **No Secret values or database credentials were read:** an attempt to query the gateway database directly was refused, and was not retried.
|
||||
|
||||
## Addendum (2026-10-29, 17:45 UTC): why the 500 persists, and a second defect
|
||||
|
||||
**The error is unchanged.** At 17:38:32 `POST /api/v1/admin/servers` failed again with `new row violates row-level security policy for table "servers"`. At 17:38–17:40 the gateway also logged `tenant reference '6ada663eb5b88053a9e43723' matches no tenant id or external_id; binding it unresolved`. The KAB default tenant is still unmapped.
|
||||
|
||||
**The one tenant upsert seen didn't touch it.** `POST /api/v1/admin/tenants` → 200 at 17:06:28 came from `kyn-agent-builder-platform-worker` (10.20.10.17), so it was a `TenantProvisioningWorkflow` for a newly created tenant. That activity (`temporal-workers/.../tenant_lifecycle/mcp_gateway_mgmt.py`) upserts `id = <tenant slug>` and `external_id = <platform id>`. It doesn't create a mapping for `6ada663eb5b88053a9e43723`.
|
||||
|
||||
### The full code path
|
||||
|
||||
| Step | Code | What happens |
|
||||
|---|---|---|
|
||||
| 1 | `agent-builder-api/src/api/routes/mcp_gateway_proxy.py` `_identity_context` | Forwards `X-Tenant-ID` = the tenant selected in the UI (`6ada…`). It sends `default` only when a platform admin has **no** tenant selected. |
|
||||
| 2 | `mcp-gateway/src/mcp_gateway/gateway/auth.py` (service-token path) | `resolve_tenant_ref` → `bind_tenant`. An unmapped reference is bound **as-is**. |
|
||||
| 3 | `mcp-gateway/src/mcp_gateway/db/repositories.py` `ServerRepository.create` | If the bound tenant ≠ `DEFAULT_TENANT_ID` (`default`), it stamps `servers.tenant_id = <bound tenant>`; otherwise it stamps `NULL` (global). |
|
||||
| 4 | `mcp-gateway/src/mcp_gateway/core/tenancy.py` `SingleTenantStrategy` | Single mode **never** issues `set_config('app.current_tenant', …)`. |
|
||||
| 5 | `alembic/versions/20260723_1000_tenancy_full_rollout.py` `tenant_catalog` | `WITH CHECK (tenant_id = EFFECTIVE OR (tenant_id IS NULL AND EFFECTIVE = 'default'))`. With no setting, EFFECTIVE is always `'default'`. |
|
||||
|
||||
So in `MCP_GATEWAY_TENANCY_MODE=single`, **a server insert succeeds only when the bound tenant is exactly `default`**. That gives a second defect, beyond the missing mapping:
|
||||
|
||||
- **Unmapped tenant** (today): `tenant_id='6ada…'` ≠ `'default'`, so it's refused.
|
||||
- **Mapped tenant whose gateway id isn't `default`** (what `TenantProvisioningWorkflow` creates for every new tenant, id = slug): `tenant_id='<slug>'` ≠ `'default'`, so it's refused too. In single mode, server registration fails for **every tenant except one mapped to `id='default'`**.
|
||||
|
||||
The `link-default-tenant` Job (an upsert of `{id:"default", external_id:"6ada…"}`) still fixes the KAB default tenant. It does nothing for other tenants.
|
||||
|
||||
### Options
|
||||
|
||||
| # | Option | Who | Effect |
|
||||
|---|---|---|---|
|
||||
| A | **Register from "All tenants".** A platform admin clears the tenant selector in the UI, so the proxy sends `X-Tenant-ID: default` and the server is stored as a global entry (`tenant_id NULL`). | Platform admin (`is_platform_admin`) | Works today, with no cluster change. The server is global; in single mode every tenant sees all global servers. |
|
||||
| B | Map the default tenant: apply `secrets.yaml` and `mcp-gateway-link-default-tenant.job.yaml`, or a Global Admin calls `POST /api/v1/mcp-gateway/api/v1/admin/tenants` with `{"id":"default","external_id":"6ada663eb5b88053a9e43723",…}` | Global Admin, or approval for a `core-components` change | Registration works with the default tenant selected. |
|
||||
| C | Platform fix: in single mode, make `ServerRepository.create` stamp `NULL` regardless of the bound tenant, **or** run the gateway in `pooled` mode (needs the restricted DB role and Alembic, per `docs/TENANCY.md`) | Platform team | Fixes registration for every tenant. Single mode then matches its documented promise ("byte-identical to the pre-tenancy behavior"). |
|
||||
| D | Platform fix: the bootstrap in `init_database.py` should run `provision_mcp_gateway_tenant` for the default tenant, with `id = DEFAULT_TENANT_ID` | Platform team | Stops a fresh install from shipping an unmapped default tenant. |
|
||||
15
docs/platform-issues/mcp-gateway-default-tenant/secrets.yaml
Normal file
15
docs/platform-issues/mcp-gateway-default-tenant/secrets.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# KAB's default tenant id, for mcp-gateway-link-default-tenant.job.yaml.
|
||||
#
|
||||
# It is an identifier, not a credential: it appears in every KAB request and in the gateway's logs. It is kept in
|
||||
# a Secret only so the Job reads it the same way it reads the internal service token. stringData takes the plain
|
||||
# value; Kubernetes stores it base64-encoded under data. The gateway Deployment does not read it.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: kab
|
||||
name: kab-tenant-id
|
||||
namespace: core-components
|
||||
type: Opaque
|
||||
stringData:
|
||||
tenant-id: "6ada663eb5b88053a9e43723"
|
||||
Reference in New Issue
Block a user