> For the complete documentation index, see [llms.txt](https://docs.rootcause.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rootcause.ai/installation-and-deployment/deployment.md). # Deployment Guide This guide takes you from a prepared Kubernetes cluster to a running RootCause Platform. It assumes your infrastructure team has completed the checklist in the Requirements doc. There are four CLI steps to bootstrap the operator, then everything else happens in the Admin UI. *** ## Before you start Confirm you have: * [ ] Kubernetes cluster running (1.26+) with `kubectl` cluster-admin access * [ ] Helm 3.12+ installed * [ ] Ingress controller deployed with wildcard DNS pointing to its IP * [ ] Object storage ready: a single bucket/container with credentials — or nothing, if you plan to use the in-cluster Garage option * [ ] Container registry credentials from RootCause (GitLab deploy token) * [ ] DNS base domain and subdomains planned * [ ] TLS certificate available * [ ] Identity provider details ready (if using external OIDC or SAML) * [ ] LLM provider API keys ready (at least two providers recommended — see Step 7) If anything is missing, go back to the [Requirements doc](/installation-and-deployment/requirements.md) and hand the checklist to your infrastructure team. *** ## Step 1: Create the namespace ```bash kubectl create namespace rootcause ``` All RootCause components will be deployed into this namespace. **Verify:** ```bash kubectl get namespace rootcause ``` *** ## Step 2: Create registry credentials The operator and platform images are hosted on GitLab Container Registry. Create an image pull secret: ```bash kubectl create secret docker-registry regcred \ -n rootcause \ --docker-server=registry.gitlab.com \ --docker-username= \ --docker-password= ``` Use the GitLab deploy token provided by RootCause. It needs `read_registry` scope. The operator reuses this secret for both image pulls and OCI chart pulls — no separate chart registry secret is needed. **Verify:** ```bash kubectl get secret regcred -n rootcause ``` *** ## Step 3: Install the unified MongoDB Kubernetes Operator (MCK) The platform uses the unified MongoDB Kubernetes Operator (MCK — MongoDB Controllers for Kubernetes) to manage its MongoDB replica set and MongoDB Search (mongot). MongoDB Search is part of every deployment, and only MCK can deploy it — the legacy MongoDB Community Operator cannot, and the RootCause Operator's preflight checks reject it. ```bash helm install mongodb-kubernetes mongodb-kubernetes \ --repo https://mongodb.github.io/helm-charts \ --version 1.9.1 \ -n rootcause \ --set operator.watchNamespace=rootcause ``` The RootCause Operator pins MCK chart version `1.9.1` for compatibility with the dependency charts it deploys. > **Skip this step** if a compatible MCK is already installed cluster-wide. It should be configured to watch all namespaces. If an MCK is present but too old, the Admin UI's Rollout panel offers an **Upgrade MongoDB operator** button that remediates it in place. **Verify:** ```bash kubectl get pods -n rootcause ``` You should see one `mongodb-kubernetes-operator-...` pod in `Running` state. *** ## Step 4: Install the RootCause Operator ```bash helm install rootcause-operator \ oci://registry.gitlab.com/perceptura/client-deployments/releases-platform/charts/rootcause-operator \ -n rootcause ``` The chart defaults `imagePullSecrets` to `regcred`, so no extra flags are needed. **Verify:** ```bash kubectl get pods -n rootcause ``` You should see: ``` rootcause-operator-controller-... 1/1 Running rootcause-operator-admin-... 1/1 Running mongodb-kubernetes-operator-... 1/1 Running ``` *** ## Step 5: Access the Admin UI > **Preflight checks:** on first startup the operator seeds a `RootCauseInstallation` in `mode: preflight`. Nothing is deployed yet, but the reconcile loop immediately runs live cluster checks — required CRDs, registry pull secret, MongoDB operator (MCK) version and schema compatibility — and populates the release catalog, so the Bootstrap page shows live verdicts and version dropdowns before you configure anything. Applying the wizard configuration in Step 6 switches the installation to `mode: active` and starts the actual deployment. Port-forward the Admin UI to your local machine: ```bash kubectl port-forward -n rootcause svc/rootcause-operator-admin 3000:3000 ``` Open in your browser. ### Log in The operator generates a master password during installation. Retrieve it: ```bash kubectl get secret rootcause-bootstrap-auth -n rootcause \ -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d ``` Enter this password on the login page. *** ## Step 6: Configure via the Bootstrap Wizard The Admin UI presents a wizard with several sections. Walk through each one. ### Release Versions * **Dependencies chart version** and **Platform chart version**: Use the latest versions unless RootCause support has told you otherwise. ### Installation * **Namespace**: Shown read-only — the namespace the operator was installed into. * **Client ID**: Your organization identifier (provided by RootCause, e.g., `acme-corp`). ### Storage Choose a storage backend and enter its connection details: | Backend | What to enter | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **S3 / S3-compatible** | Region, access key ID, secret access key; optional endpoint URL for S3-compatible stores | | **Azure Blob Storage** | Storage account, endpoint suffix (`core.windows.net`), and one auth method: connection string, account key, or service principal (tenant ID, client ID, client secret) | | **Google Cloud Storage** | Project ID and service account credentials | | **Garage (self-hosted, in-cluster)** | Nothing — fully automated | For cloud backends, enter a **single bucket/container name**. Datasets, digital twins, and ML models are stored as sub-directories of this one bucket (`ROOTCAUSE_BUCKET`). For **Garage**, the operator deploys an S3-compatible object store inside the cluster, creates the bucket, generates credentials, and wires everything to the platform. Pick a mode: * **Single node** — 1 replica, RF=1. For development and single-node clusters. * **Production** — 3 replicas, RF=3, pod anti-affinity. Requires 3+ schedulable nodes. Set the storage capacity (PVC size per replica, default `50Gi`). Also set the **Kubernetes storage class** for persistent volumes (e.g., `managed-csi`, `gp3`, `standard-rwo`). ### Networking | Field | What to enter | | ----------------- | -------------------------------------------------------------------------------- | | **Base domain** | Your base domain (e.g., `rootcause.example.com`) | | **Ingress class** | `nginx` for most deployments. `azure/application-gateway` for Azure App Gateway. | | **Subdomains** | Platform, Auth, and LiteLLM subdomains (e.g., `platform`, `auth`, `litellm`) | #### Ingress annotations Add annotations under **All ingresses (base)** based on your ingress controller: **nginx:** | Key | Value | | ------------------------------------------ | ------ | | `nginx.ingress.kubernetes.io/ssl-redirect` | `true` | For TLS with a pre-existing wildcard certificate, select **Manual (bring your own secret)** under TLS mode and provide the secret name. **Azure Application Gateway:** | Key | Value | | -------------------------------------------- | ------- | | `appgw.ingress.kubernetes.io/ssl-redirect` | `false` | | `appgw.ingress.kubernetes.io/use-private-ip` | `true` | For each ingress (Platform, Auth, LiteLLM), also add: | Key | Value | | --------------------------------------------------- | ------------------------- | | `appgw.ingress.kubernetes.io/appgw-ssl-certificate` | *(your certificate name)* | > **Azure Application Gateway note:** App Gateway does not resolve loopback external URLs from within the cluster. You will need a `hostAliases` resource patch in the Advanced section — see [Azure Application Gateway patches](#azure-application-gateway-patches) below. ### Identity & Access Choose based on the decision you made in the Requirements doc: **External OIDC (recommended)** | Field | What to enter | | ------------------- | ------------------------------------------------------------------------------------------- | | Authentication mode | OIDC | | Deploy FusionAuth | No | | Issuer | Your IdP's issuer URL | | Client ID | Your application's client ID | | Client secret | Your application's client secret | | Well-known URL | Your IdP's OpenID configuration URL | | Logout URL | Your IdP's logout endpoint (include a `post_logout_redirect_uri` back to your platform URL) | > **Azure EntraID specifics:** The issuer URL is `https://login.microsoftonline.com//v2.0`. In the Azure Portal, ensure your app registration has: > > 1. **Redirect URI**: `https://./api/auth/callback/login` (type: Web) > 2. **Token configuration**: Include `email`, `profile`, and `openid` scopes > 3. **API permissions**: `openid`, `profile`, `email` (Microsoft Graph, delegated) **External SAML** | Field | What to enter | | -------------------- | ------------------------------------------------------------- | | Authentication mode | SAML | | Deploy FusionAuth | No | | *(remaining fields)* | Metadata URL or XML, entity ID, and certificate from your IdP | **Managed FusionAuth (POC / no existing IdP)** | Field | What to enter | | -------------------- | ----------------- | | Authentication mode | Built-in (no SSO) | | Deploy FusionAuth | Yes | | FusionAuth API key | *(leave blank)* | | FusionAuth Tenant ID | *(leave blank)* | Leave credentials blank. The operator auto-provisions everything: API keys, tenant, application, and OIDC configuration. ### Email (optional) SMTP credentials for outbound platform email: user, passkey, host, port, and an optional CA certificate (only needed when the SMTP server uses a private CA). Host defaults to `smtp.rootcau.se` and port to `30025`. Skip this section if you don't need outbound email. ### Telemetry (optional) An opt-in, in-cluster observability stack — off by default. When enabled, logs and metrics flow through an OpenTelemetry Collector gateway into local Loki + Prometheus, with a passwordless Grafana over both (reach it with `kubectl port-forward`). Configure retention days (default 7) and volume sizes, and optionally forward everything to an external OTLP collector (gRPC or HTTP). ### Dependencies Each dependency is deployed by the operator by default, or connected to an external instance: | Dependency | Deploy (default) | External | | ---------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------- | | PostgreSQL | Deployed by the operator | Host, port, username, password, SSL mode | | MongoDB | Set replicas: 1 for testing, 3 for production. MongoDB Search (mongot) is deployed alongside by default. | Connection URI | | Redis | Set replicas: 1 for testing, 3 for production | Connection URL or JSON sentinel config | | Temporal | Deployed by the operator | Frontend URL (`host:7233`) | | LiteLLM | Deployed by the operator | URL of your existing LiteLLM instance | ### Node Selector (optional) Constrain **all** workloads — every pod across both the dependencies and platform charts — to nodes matching label key/value pairs. For example, `usage: rootcause` to match a dedicated AKS node pool label. ### Compute Node Pool (optional) Route heavy ML jobs and digital twins (causal discovery, digital twin training, simulation) onto a dedicated node pool that scales to zero when idle. This applies only to the ephemeral Job pods the Data Service spawns — not the long-running services. Leave empty to run all ML jobs on the default nodes. The contract is the same on every cloud: 1. **You provision** a node group that is tainted `rootcause.ai/workload=compute:NoSchedule`, labeled `rootcause.ai/nodepool=compute`, and autoscaling with **min = 0**. 2. **The wizard fields** stamp a matching node selector + toleration onto the spawned ML Job pods. Those pods stay `Pending` until a compute node exists. 3. **Your cluster autoscaler** sees the `Pending` pod and scales the pool `0 → 1`, then back to `0` when the job finishes. The operator does not create node groups. AKS example (built-in autoscaler, nothing extra to deploy): ```bash az aks nodepool add \ --resource-group --cluster-name \ --name compute --mode User --node-vm-size Standard_E4s_v3 --node-count 0 \ --enable-cluster-autoscaler --min-count 0 --max-count 2 \ --node-taints rootcause.ai/workload=compute:NoSchedule \ --labels rootcause.ai/nodepool=compute ``` On GKE, use the built-in autoscaler the same way. On EKS, self-deploy Cluster Autoscaler (or Karpenter) with node-template tags for the taint/label so the group can scale from zero. Optional fields: * **Pool name** — logical name used for pool-scoped capacity checks (e.g., `compute`) * **Max node memory GiB** — lets the platform reject physically impossible workloads while the pool is at zero * **Job kinds** — which ML job kinds are routed to the pool. Default: the heavy kinds (`causal_discovery`, `causal_discovery_aggregate`, `digital_twin`, `simulation`); small, frequent ontology jobs stay on the default pool to avoid scale-from-zero latency. ### Components Configure replicas, resources, and extra environment variables per long-running component. Defaults work for testing. For production, use these as a starting point: | Component | Replicas | CPU request | Memory request | | ------------ | -------- | ----------- | -------------- | | Platform | 2-3 | 500m | 1 Gi | | Data Service | 2-3 | 1 | 2 Gi | ML jobs are not configured here — they run as ephemeral Kubernetes Job pods with per-job resources set automatically by the Data Service. See the **Upgrades & Operations** doc for detailed scaling guidance. ### Service Accounts (optional) Create Kubernetes service accounts with annotations and assign them to the `platform` and/or `data-service` deployments. Useful for workload identity (e.g., AWS IAM Roles for Service Accounts). ### Advanced For most deployments, you can skip this section. It's here for edge cases. * **Resource patches**: Deep-merge patches into rendered Kubernetes manifests (annotations, tolerations, node selectors, `hostAliases`, etc.) * **Raw Helm overrides**: Free-form YAML merged over computed values for any chart setting not covered by the wizard #### Azure Application Gateway patches Azure Application Gateway does not resolve external URLs from within the cluster. The platform pod needs a `hostAliases` entry to route auth traffic to the Application Gateway's IP directly. Add a **Deployment** resource patch: | Field | Value | | ----------- | ------------------ | | Chart | platform | | Kind | deployment | | Object name | rootcause-platform | Patch content: ```yaml spec: template: spec: hostAliases: - ip: "" hostnames: - "." ``` App Gateway also requires explicit path definitions. Add these in **Raw Helm Overrides**: **Platform chart overrides:** ```yaml platform: ingress: hosts: - host: . paths: - path: / pathType: Exact port: 80 - path: /* pathType: Prefix port: 80 ``` **Dependencies chart overrides:** ```yaml fusionauth: ingress: paths: - path: /* pathType: Prefix - path: / pathType: Exact ``` > Without both path types, App Gateway may return 502 errors on some requests. ### Effective Values Preview Before applying, this section shows the final Helm values the operator computes from your configuration (wizard fields, patches, and raw overrides merged) — use it to sanity-check what will actually be deployed. ### Deploy Review the **Deployment Summary** at the bottom of the wizard, then click **Apply configuration**. The operator will: 1. Create required secrets (storage credentials, platform config) 2. Create the `RootCauseInstallation` custom resource 3. Deploy infrastructure dependencies (PostgreSQL, Redis, MongoDB, Temporal, LiteLLM, and optionally FusionAuth) 4. Deploy platform services (Platform, Data Service) Watch the **Deployment status** panel on the right. The phase progresses through `Reconciling` to `Ready`, typically in 2-5 minutes. *** ## Step 7: Configure LLM models After the platform is deployed, configure LLM access on the **LLM** page in the Admin UI. Configure at least two providers (e.g., one OpenAI, one Anthropic) so fallbacks keep the platform working through provider outages. > If you configured an external LiteLLM instance in the wizard, manage models in your own instance instead, then continue to Step 8. The LLM page has four sections. Work top to bottom: ### Provider Keys Add an API key per LLM provider. The keys are stored as a Kubernetes secret in the installation namespace — you never need to log in to the LiteLLM UI or create virtual keys; the operator reads the LiteLLM master key itself. ### Models Register the LiteLLM models backed by those provider keys: pick the provider, the upstream model, and a model name. Repeat for each model you want available. ### Tiers Map the platform's model tiers (small / medium / large) to default models. The platform picks a tier per task; these defaults decide which registered model serves each tier. ### Fallbacks Define per-model fallback chains so the platform always has a working LLM, even during provider outages: select a primary model and add one or more fallbacks in order of preference. > **Tip:** Match fallbacks by weight class — if your primary is GPT-4, fall back to Claude Sonnet, not to a smaller model. The exact model doesn't matter as much as matching capability. ### Where this configuration lives The model registry and fallback configuration are persisted declaratively in the `RootCauseInstallation` CR (`spec.config.dependenciesConfig.litellm.config`); the LiteLLM ConfigMap is rendered from that spec on every reconcile. Tier routing is stored in MongoDB. This configuration **survives upgrades and reinstalls** — there is nothing to export or screenshot. Exporting the CR (see Upgrades & Operations) backs it up along with everything else. **Verify:** the Models section lists your models without runtime sync errors, and each tier has a default model assigned. *** ## Step 8: Create platform users Navigate to the **Users** page in the Admin UI. **With managed FusionAuth:** 1. Click **+ Add user** 2. Enter email, password, first name, last name 3. Click **Create user** The operator creates a FusionAuth account, registers it to the platform application, and adds the email to the admin list automatically. **With external OIDC or SAML:** Create users in your external identity provider (EntraID, Okta, etc.), then add their email addresses in the **Platform admin emails** section on the Users page. These emails are stored in the CR and mounted as `PLATFORM_ADMIN_EMAILS` on the platform deployment. *** ## Step 9: Log in and verify Navigate to `https://.`. * With FusionAuth: click "Continue with OIDC SSO" and log in with the credentials from Step 8 * With external OIDC/SAML: you'll be redirected to your identity provider ### Post-install smoke test Run through these checks to confirm everything is working: | # | Check | How | | - | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | 1 | All pods running | `kubectl get pods -n rootcause` — all should be `Running` or `Completed` | | 2 | Admin UI accessible | loads (with port-forward active) | | 3 | Platform login works | Navigate to platform URL, complete login flow, reach the home page | | 4 | LLM responds | Admin UI **LLM** page shows your models without sync errors; ask the platform's AI a question and get a response | | 5 | Data works end-to-end | Create a workspace, open a Data View, confirm it loads | If any check fails, see the **Troubleshooting** section in the Upgrades & Operations doc. *** ## Next step Proceed to the **Upgrades & Operations** doc for day-2 operations: applying updates, rolling back, scaling, and troubleshooting.