> For the complete documentation index, see [llms.txt](https://docs.rootcause.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rootcause.ai/installation-and-deployment/upgrades-and-operations.md).

# Upgrades & Operations

Day-2 operations for the RootCause Platform. This doc covers upgrades, rollback, scaling, configuration changes, and troubleshooting.

**Who this is for:** Data scientists can handle upgrades and routine operations (Sections 1-3). Infrastructure teams should review the backup, rollback, scaling, and troubleshooting sections.

***

## 1. How updates work

The RootCause Operator automatically checks the OCI registry for new chart versions. When an update is available, it appears on the **Releases** page in the Admin UI.

Key points:

* **You can jump directly to any version** — no need to step through intermediate releases
* **Dependencies and platform are upgraded together** — the operator manages both Helm releases as a unit
* **Updates are applied through the Admin UI** — no Helm commands needed for routine upgrades

***

## 2. Upgrading

### Before you upgrade (recommended)

The operator does not require backups before upgrading. However, we recommend these steps as best practice — they take a few minutes and give you a recovery path if anything unexpected happens.

**Recommended backup checklist:**

```bash
# Export the CR spec (captures your full configuration)
kubectl get rootcauseinstallation rootcause -n rootcause -o yaml > rc-backup-$(date +%Y%m%d).yaml

# PostgreSQL dump (Temporal, FusionAuth, and LiteLLM databases)
kubectl exec -it postgres-0 -n rootcause -- \
  pg_dumpall -U postgres > postgres-backup-$(date +%Y%m%d).sql

# MongoDB dump
kubectl exec -it perceptura-mongo-0 -n rootcause -- \
  mongodump --out=/backup/$(date +%Y%m%d)
```

The CR export already captures your LLM configuration — the model registry and fallback chains live in `spec.config.dependenciesConfig.litellm.config`, not in the database — so there is nothing separate to export for LiteLLM.

Also recommended:

* Check release notes for the target version
* Test in a staging environment if you have one

### Applying an upgrade

**Via the Admin UI (recommended):**

1. Open the Admin UI
2. Go to the **Releases** page
3. Select the target version
4. Click **Request upgrade**
5. Watch the **Deployment status** panel — the phase progresses through `Reconciling` to `Ready`

**Via CLI** (for automation or GitOps workflows):

```bash
# Update the chart versions in the CR spec
kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"release":{"dependenciesChartVersion":"<new-version>","platformChartVersion":"<new-version>"}}}'
```

The operator detects the spec change and reconciles automatically.

**Deferring until ML jobs finish:** an upgrade restarts the platform, so any in-progress ML jobs (causal discovery, digital twin training, etc.) would lose their work. The Releases page shows how many ML jobs are currently running and lets you request the upgrade **deferred until ML jobs are idle**. The operator then holds the action, sets the phase to `WaitingForMLJobs`, records the pending action and the live job list in the CR status (`status.pendingAction`, `status.activeMlJobs`), rechecks every 30 seconds, and applies the upgrade automatically once the last job finishes. Force reconcile and undeploy support the same deferral (`spec.actions.deferUntilMLJobsIdle`).

**Monitor the upgrade:**

```bash
# Watch pods restart
kubectl get pods -n rootcause -w

# Check installation status
kubectl get rootcauseinstallation -n rootcause -o jsonpath='{.status.phase}'
```

**Verify:** Run the same smoke test from the Deployment guide (Step 9) — all pods running, Admin UI accessible, platform login works, LLM responds, data loads.

***

## 3. Rolling back

The operator treats rollback the same as any other version change. To go back to a previous version, you select it and apply it — the operator deploys whichever version you tell it to.

This works because the operator manages the full deployment state. It doesn't need to "undo" anything — it simply deploys the version you specify.

### Steps

1. Open the Admin UI > **Releases** page
2. Select the previous version you want to return to
3. Click **Request upgrade** (yes, "upgrade" — the operator doesn't distinguish between moving forward or backward)
4. Watch the **Deployment status** panel
5. Verify with the smoke test

### When implicit rollback isn't enough

In rare cases — for example, if an upgrade included a database schema migration that isn't backward-compatible — selecting the previous version may not fully restore the old behavior. If this happens and you took backups before upgrading:

**Restore PostgreSQL:**

```bash
kubectl exec -i postgres-0 -n rootcause -- \
  psql -U postgres < postgres-backup-<date>.sql
```

**Restore MongoDB:**

```bash
kubectl exec -it perceptura-mongo-0 -n rootcause -- \
  mongorestore --drop /backup/<date>
```

LiteLLM model and fallback configuration lives in the CR spec, so it is reapplied automatically on every reconcile — nothing to re-enter.

Then select the previous version on the Releases page and apply it.

***

## 4. Scaling

Two things scale independently:

* **Long-running services** (Platform, Data Service) — fixed replica counts and resources, set in the **Components** section of the bootstrap wizard. Change values and click **Apply configuration** — the operator reconciles. The platform charts do not ship Horizontal Pod Autoscalers; replicas are whatever you set.
* **ML job pods** — ephemeral Kubernetes Jobs the Data Service spawns per task. There is no ML Jobs deployment to scale; capacity is a function of the nodes available to schedule those pods.

### Long-running services

**Replica counts by workload size:**

| Component    | Dev/Test | Small prod (50 users) | Medium prod (200 users) | Large prod (200+ users) |
| ------------ | -------- | --------------------- | ----------------------- | ----------------------- |
| Platform     | 1        | 2                     | 3                       | 5                       |
| Data Service | 1        | 3                     | 5                       | 10                      |

**Resource limits:**

**Platform (Web UI):**

|        | Request | Limit |
| ------ | ------- | ----- |
| CPU    | 500m    | 2     |
| Memory | 1 Gi    | 4 Gi  |

**Data Service:**

|        | Request | Limit |
| ------ | ------- | ----- |
| CPU    | 1       | 4     |
| Memory | 2 Gi    | 8 Gi  |

### ML job pods

ML workloads (causal discovery, digital twin training, simulation, ontology extraction) run as Kubernetes Jobs created per task: the pod runs to completion and is cleaned up. The Data Service sets resources per job kind:

| Job kind                     | CPU request / limit | Memory request / limit |
| ---------------------------- | ------------------- | ---------------------- |
| Causal discovery             | 4 / 16              | 8 Gi / 32 Gi           |
| Causal discovery (aggregate) | 2 / 8               | 16 Gi / 48 Gi          |
| Digital twin                 | 2 / 8               | 4 Gi / 16 Gi           |
| Simulation                   | 2 / 4               | 2 Gi / 8 Gi            |
| Ontology jobs                | 1-2 / 4-8           | 2-8 Gi / 8-20 Gi       |

If a job is OOM-killed, the retry escalates memory 1.5x per OOM, capped at 85% of the largest compute node's memory (or 64 Gi when the node size is unknown). Workloads that would exceed the cap are rejected as too large instead of thrashing.

To tune defaults without a code change, set the `ML_JOB_RESOURCES_OVERRIDE` environment variable on the Data Service (extra environment variables in the wizard's Components section) — a JSON mapping of job kind to partial resource fields:

```json
{"causal_discovery": {"memory_request": "16Gi", "memory_limit": "64Gi"}}
```

### Sizing the compute node pool

If you configured the **Compute Node Pool** section of the wizard, the heavy job kinds are routed to a dedicated node group that your cluster autoscaler scales from zero on demand. Sizing guidance:

* **Node size**: jobs are memory-bound. The largest default limit is 48 Gi (aggregate causal discovery), so nodes with 64 GB RAM are a sensible floor; use larger nodes for very large models (100+ variables), which escalate toward the node cap on retries.
* **Max node count**: bounds how many heavy jobs run in parallel. 2-4 is enough for most teams.
* **Max node memory GiB**: set this wizard field so the platform can reject physically impossible workloads even while the pool is scaled to zero.
* Job pods sitting in `Pending` for a few minutes while the pool scales `0 → 1` is normal, not a failure.

### Scaling databases

Databases scale through the Dependencies section of the wizard or via Raw Helm Overrides:

| Database   | How to scale                                                                        | Production recommendation           |
| ---------- | ----------------------------------------------------------------------------------- | ----------------------------------- |
| MongoDB    | Increase replica set members (1 → 3 for HA, a wizard toggle) and storage size       | 3 replicas, 100 Gi+ storage         |
| PostgreSQL | Vertical scaling (increase CPU/memory). Add read replicas for read-heavy workloads. | 2-4 CPU, 4-8 Gi RAM, 10 Gi+ storage |
| Redis      | Increase replicas (1 → 3, a wizard toggle) and memory                               | 1-4 CPU, 2-8 Gi RAM                 |

### Node placement

* **All workloads**: the wizard's **Node Selector** section applies a node selector to every pod across both charts (e.g., `usage: rootcause` for a dedicated node pool).
* **ML job pods only**: the **Compute Node Pool** section stamps a node selector + toleration onto the spawned Job pods, stored in the CR as:

```yaml
spec:
  config:
    computePool:
      nodePool: compute
      nodeSelector:
        rootcause.ai/nodepool: compute
      tolerations:
        - key: rootcause.ai/workload
          operator: Equal
          value: compute
          effect: NoSchedule
```

***

## 5. Changing configuration

Change any setting in the bootstrap wizard and click **Apply configuration**. The operator detects the spec change (generation bump) and reconciles — upgrading both Helm releases with the updated values.

Changes take effect within minutes. The Deployment status panel shows the reconciliation progress.

**Force reconcile:** If the installation is stuck or you need to re-apply values without changing the spec, click **Force reconcile** in the Deployment status panel. Via CLI:

```bash
kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"actions":{"forceReconcileNonce":"'$(uuidgen)'"}}}'
```

***

## 6. Managing secrets

The **Secrets** page in the Admin UI lists all namespace secrets. You can view, create, edit, and delete secrets. Missing secrets required by the platform are highlighted.

**Rotating credentials:**

1. Update the secret value on the Secrets page (or via `kubectl`)
2. Restart the affected pods so they pick up the new value:

```bash
# Example: restart data-service after rotating storage credentials
kubectl rollout restart deployment rootcause-platform-data-service -n rootcause
```

> LLM provider API keys are managed on the **LLM** page (Provider Keys section), not here.

***

## 7. Managing users

### With managed FusionAuth

Use the **Users** page in the Admin UI to add, edit, or remove users. The operator manages FusionAuth accounts and the platform admin email list automatically.

### With external OIDC/SAML

Manage user accounts in your identity provider (EntraID, Okta, etc.). In the Admin UI, use the **Platform admin emails** section on the Users page to control which users have admin access. These emails are stored in the CR and mounted as `PLATFORM_ADMIN_EMAILS`.

***

## 8. Uninstalling

On the Bootstrap page, scroll to **Danger Zone**, check the confirmation box, and click **Request undeploy**. The operator will:

1. Uninstall the platform Helm release
2. Uninstall the dependencies Helm release
3. Clean up operator-managed secrets

The phase moves through `Uninstalling` while this runs (or `WaitingForMLJobs` first, if you deferred the undeploy until running ML jobs finish). The installation CR is preserved in `Uninstalled` phase. You can redeploy by updating the spec and clicking Apply configuration — the operator will do a fresh deployment.

> **Warning:** Undeploying destroys all data in the deployed databases (PostgreSQL, MongoDB, Redis). Take backups before undeploying if you need to preserve data.

***

## 9. Troubleshooting

### Check operator logs

```bash
kubectl logs deploy/rootcause-operator-controller -n rootcause
```

### Check installation status

```bash
kubectl get rootcauseinstallation -n rootcause -o yaml
```

Key fields:

| Field                  | What it tells you                                                                                               |
| ---------------------- | --------------------------------------------------------------------------------------------------------------- |
| `status.phase`         | Current state: `Ready`, `Reconciling`, `WaitingForMLJobs`, `Degraded`, `Blocked`, `Uninstalling`, `Uninstalled` |
| `status.lastError`     | Error message if the phase is `Degraded`                                                                        |
| `status.conditions`    | Detailed condition status for each subsystem                                                                    |
| `status.activeMlJobs`  | ML Job pods currently running (count and job list)                                                              |
| `status.pendingAction` | Action (upgrade, force reconcile, undeploy) deferred until ML jobs finish                                       |

### Check Helm releases

```bash
helm list -n rootcause
```

### Check pod logs

```bash
# Platform logs
kubectl logs -l app=platform -n rootcause --tail=50

# Data service logs
kubectl logs -l app=data-service -n rootcause --tail=50

# Check logs from a crashed container
kubectl logs <pod-name> -n rootcause --previous
```

### Common issues

| Issue                                     | Cause                                                        | Fix                                                                                          |
| ----------------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| `ImagePullBackOff`                        | Missing or invalid pull secret                               | Verify `regcred` exists with correct credentials                                             |
| `Degraded` with Helm error                | Chart values validation failure                              | Check `status.lastError` and fix the configuration in the wizard                             |
| FusionAuth stuck in Maintenance Mode      | `silentMode` not enabled                                     | Ensure operator version includes the silent mode fix (0.1.14+)                               |
| OIDC login returns `invalid_redirect_uri` | Wildcard validation not enabled                              | Ensure operator version includes `authorizedURLValidationPolicy: AllowWildcards` (0.1.16+)   |
| PVC stuck in Pending                      | No matching storage class                                    | Set `storageClass` in the bootstrap wizard to match your cluster                             |
| Pods stuck in Pending                     | Insufficient cluster resources                               | Scale up nodes or reduce replica counts                                                      |
| ML job pod Pending for a few minutes      | Compute node pool scaling from zero                          | Normal. If it persists, check your cluster autoscaler and the pool's taint/label.            |
| Phase stuck in `WaitingForMLJobs`         | A requested action was deferred until running ML jobs finish | Wait for the jobs (see `status.activeMlJobs`), or re-request the action without the deferral |
| `CrashLoopBackOff`                        | Application error on startup                                 | Check pod logs with `kubectl logs <pod-name> --previous`                                     |
| Upgrade stuck in `Reconciling`            | Operator unable to complete reconciliation                   | Check operator logs, then try Force reconcile                                                |

### Upgrade-specific issues

| Issue                                  | Cause                                                                                                  | Fix                                                                                                       |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| Pods not restarting after upgrade      | Old pods still running                                                                                 | `kubectl rollout restart deployment <name> -n rootcause`                                                  |
| LiteLLM models missing after reinstall | `spec.config.dependenciesConfig.litellm.config` empty in the CR (e.g., CR restored from an old backup) | Re-add models on the Admin UI **LLM** page — they are stored in the CR and re-rendered on every reconcile |
| Configuration errors after upgrade     | New chart version has different value schema                                                           | Check `status.lastError`, update wizard fields to match new schema                                        |

***

## Quick reference: useful commands

```bash
# Get Admin UI master password
kubectl get secret rootcause-bootstrap-auth -n rootcause \
  -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

# Get LiteLLM master password
kubectl get secret rootcause-dependencies-litellm-secrets \
  -n rootcause -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d

# Export CR spec (backup your configuration)
kubectl get rootcauseinstallation rootcause -n rootcause -o yaml > rc-backup.yaml

# Watch all pods
kubectl get pods -n rootcause -w

# Check installation phase
kubectl get rootcauseinstallation -n rootcause -o jsonpath='{.status.phase}'

# Force reconcile
kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"actions":{"forceReconcileNonce":"'$(uuidgen)'"}}}'

# Check events (sorted by time)
kubectl get events -n rootcause --sort-by='.lastTimestamp'
```