Upgrades & Operations

Day-2 operations for the RootCause Platform. This doc covers upgrades, rollback, scaling, configuration changes, and troubleshooting.

Who this is for: Data scientists can handle upgrades and routine operations (Sections 1-3). Infrastructure teams should review the backup, rollback, scaling, and troubleshooting sections.

1. How updates work

The RootCause Operator automatically checks the OCI registry for new chart versions. When an update is available, it appears on the Releases page in the Admin UI.

Key points:

You can jump directly to any version — no need to step through intermediate releases
Dependencies and platform are upgraded together — the operator manages both Helm releases as a unit
Updates are applied through the Admin UI — no Helm commands needed for routine upgrades

2. Upgrading

Before you upgrade (recommended)

The operator does not require backups before upgrading. However, we recommend these steps as best practice — they take a few minutes and give you a recovery path if anything unexpected happens.

Recommended backup checklist:

# Export the CR spec (captures your full configuration)
kubectl get rootcauseinstallation rootcause -n rootcause -o yaml > rc-backup-$(date +%Y%m%d).yaml

# PostgreSQL dump (Temporal, FusionAuth, and LiteLLM databases)
kubectl exec -it postgres-0 -n rootcause -- \
  pg_dumpall -U postgres > postgres-backup-$(date +%Y%m%d).sql

# MongoDB dump
kubectl exec -it perceptura-mongo-0 -n rootcause -- \
  mongodump --out=/backup/$(date +%Y%m%d)

The CR export already captures your LLM configuration — the model registry and fallback chains live in spec.config.dependenciesConfig.litellm.config, not in the database — so there is nothing separate to export for LiteLLM.

Also recommended:

Check release notes for the target version
Test in a staging environment if you have one

Applying an upgrade

Via the Admin UI (recommended):

Open the Admin UI
Go to the Releases page
Select the target version
Click Request upgrade
Watch the Deployment status panel — the phase progresses through Reconciling to Ready

Via CLI (for automation or GitOps workflows):

# Update the chart versions in the CR spec
kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"release":{"dependenciesChartVersion":"<new-version>","platformChartVersion":"<new-version>"}}}'

The operator detects the spec change and reconciles automatically.

Deferring until ML jobs finish: an upgrade restarts the platform, so any in-progress ML jobs (causal discovery, digital twin training, etc.) would lose their work. The Releases page shows how many ML jobs are currently running and lets you request the upgrade deferred until ML jobs are idle. The operator then holds the action, sets the phase to WaitingForMLJobs, records the pending action and the live job list in the CR status (status.pendingAction, status.activeMlJobs), rechecks every 30 seconds, and applies the upgrade automatically once the last job finishes. Force reconcile and undeploy support the same deferral (spec.actions.deferUntilMLJobsIdle).

Monitor the upgrade:

# Watch pods restart
kubectl get pods -n rootcause -w

# Check installation status
kubectl get rootcauseinstallation -n rootcause -o jsonpath='{.status.phase}'

Verify: Run the same smoke test from the Deployment guide (Step 9) — all pods running, Admin UI accessible, platform login works, LLM responds, data loads.

3. Rolling back

The operator treats rollback the same as any other version change. To go back to a previous version, you select it and apply it — the operator deploys whichever version you tell it to.

This works because the operator manages the full deployment state. It doesn't need to "undo" anything — it simply deploys the version you specify.

Steps

Open the Admin UI > Releases page
Select the previous version you want to return to
Click Request upgrade (yes, "upgrade" — the operator doesn't distinguish between moving forward or backward)
Watch the Deployment status panel
Verify with the smoke test

When implicit rollback isn't enough

In rare cases — for example, if an upgrade included a database schema migration that isn't backward-compatible — selecting the previous version may not fully restore the old behavior. If this happens and you took backups before upgrading:

Restore PostgreSQL:

kubectl exec -i postgres-0 -n rootcause -- \
  psql -U postgres < postgres-backup-<date>.sql

Restore MongoDB:

kubectl exec -it perceptura-mongo-0 -n rootcause -- \
  mongorestore --drop /backup/<date>

LiteLLM model and fallback configuration lives in the CR spec, so it is reapplied automatically on every reconcile — nothing to re-enter.

Then select the previous version on the Releases page and apply it.

4. Scaling

Two things scale independently:

Long-running services (Platform, Data Service) — fixed replica counts and resources, set in the Components section of the bootstrap wizard. Change values and click Apply configuration — the operator reconciles. The platform charts do not ship Horizontal Pod Autoscalers; replicas are whatever you set.
ML job pods — ephemeral Kubernetes Jobs the Data Service spawns per task. There is no ML Jobs deployment to scale; capacity is a function of the nodes available to schedule those pods.

Long-running services

Replica counts by workload size:

Component

Dev/Test

Small prod (50 users)

Medium prod (200 users)

Large prod (200+ users)

Platform

Data Service

Resource limits:

Platform (Web UI):

Request

Limit

CPU

500m

Memory

1 Gi

4 Gi

Data Service:

Request

Limit

CPU

Memory

2 Gi

8 Gi

ML job pods

ML workloads (causal discovery, digital twin training, simulation, ontology extraction) run as Kubernetes Jobs created per task: the pod runs to completion and is cleaned up. The Data Service sets resources per job kind:

Job kind

CPU request / limit

Memory request / limit

Causal discovery

4 / 16

8 Gi / 32 Gi

Causal discovery (aggregate)

2 / 8

16 Gi / 48 Gi

Digital twin

2 / 8

4 Gi / 16 Gi

Simulation

2 / 4

2 Gi / 8 Gi

Ontology jobs

1-2 / 4-8

2-8 Gi / 8-20 Gi

If a job is OOM-killed, the retry escalates memory 1.5x per OOM, capped at 85% of the largest compute node's memory (or 64 Gi when the node size is unknown). Workloads that would exceed the cap are rejected as too large instead of thrashing.

To tune defaults without a code change, set the ML_JOB_RESOURCES_OVERRIDE environment variable on the Data Service (extra environment variables in the wizard's Components section) — a JSON mapping of job kind to partial resource fields:

{"causal_discovery": {"memory_request": "16Gi", "memory_limit": "64Gi"}}

Sizing the compute node pool

If you configured the Compute Node Pool section of the wizard, the heavy job kinds are routed to a dedicated node group that your cluster autoscaler scales from zero on demand. Sizing guidance:

Node size: jobs are memory-bound. The largest default limit is 48 Gi (aggregate causal discovery), so nodes with 64 GB RAM are a sensible floor; use larger nodes for very large models (100+ variables), which escalate toward the node cap on retries.
Max node count: bounds how many heavy jobs run in parallel. 2-4 is enough for most teams.
Max node memory GiB: set this wizard field so the platform can reject physically impossible workloads even while the pool is scaled to zero.
Job pods sitting in Pending for a few minutes while the pool scales 0 → 1 is normal, not a failure.

Scaling databases

Databases scale through the Dependencies section of the wizard or via Raw Helm Overrides:

Database

How to scale

Production recommendation

MongoDB

Increase replica set members (1 → 3 for HA, a wizard toggle) and storage size

3 replicas, 100 Gi+ storage

PostgreSQL

Vertical scaling (increase CPU/memory). Add read replicas for read-heavy workloads.

2-4 CPU, 4-8 Gi RAM, 10 Gi+ storage

Redis

Increase replicas (1 → 3, a wizard toggle) and memory

1-4 CPU, 2-8 Gi RAM

Node placement

All workloads: the wizard's Node Selector section applies a node selector to every pod across both charts (e.g., usage: rootcause for a dedicated node pool).
ML job pods only: the Compute Node Pool section stamps a node selector + toleration onto the spawned Job pods, stored in the CR as:

spec:
  config:
    computePool:
      nodePool: compute
      nodeSelector:
        rootcause.ai/nodepool: compute
      tolerations:
        - key: rootcause.ai/workload
          operator: Equal
          value: compute
          effect: NoSchedule

5. Changing configuration

Change any setting in the bootstrap wizard and click Apply configuration. The operator detects the spec change (generation bump) and reconciles — upgrading both Helm releases with the updated values.

Changes take effect within minutes. The Deployment status panel shows the reconciliation progress.

Force reconcile: If the installation is stuck or you need to re-apply values without changing the spec, click Force reconcile in the Deployment status panel. Via CLI:

kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"actions":{"forceReconcileNonce":"'$(uuidgen)'"}}}'

6. Managing secrets

The Secrets page in the Admin UI lists all namespace secrets. You can view, create, edit, and delete secrets. Missing secrets required by the platform are highlighted.

Rotating credentials:

Update the secret value on the Secrets page (or via kubectl)
Restart the affected pods so they pick up the new value:

# Example: restart data-service after rotating storage credentials
kubectl rollout restart deployment rootcause-platform-data-service -n rootcause

LLM provider API keys are managed on the LLM page (Provider Keys section), not here.

7. Managing users

With managed FusionAuth

Use the Users page in the Admin UI to add, edit, or remove users. The operator manages FusionAuth accounts and the platform admin email list automatically.

With external OIDC/SAML

Manage user accounts in your identity provider (EntraID, Okta, etc.). In the Admin UI, use the Platform admin emails section on the Users page to control which users have admin access. These emails are stored in the CR and mounted as PLATFORM_ADMIN_EMAILS.

8. Uninstalling

On the Bootstrap page, scroll to Danger Zone, check the confirmation box, and click Request undeploy. The operator will:

Uninstall the platform Helm release
Uninstall the dependencies Helm release
Clean up operator-managed secrets

The phase moves through Uninstalling while this runs (or WaitingForMLJobs first, if you deferred the undeploy until running ML jobs finish). The installation CR is preserved in Uninstalled phase. You can redeploy by updating the spec and clicking Apply configuration — the operator will do a fresh deployment.

Warning: Undeploying destroys all data in the deployed databases (PostgreSQL, MongoDB, Redis). Take backups before undeploying if you need to preserve data.

9. Troubleshooting

Check operator logs

kubectl logs deploy/rootcause-operator-controller -n rootcause

Check installation status

kubectl get rootcauseinstallation -n rootcause -o yaml

Key fields:

Field

What it tells you

status.phase

Current state: Ready, Reconciling, WaitingForMLJobs, Degraded, Blocked, Uninstalling, Uninstalled

status.lastError

Error message if the phase is Degraded

status.conditions

Detailed condition status for each subsystem

status.activeMlJobs

ML Job pods currently running (count and job list)

status.pendingAction

Action (upgrade, force reconcile, undeploy) deferred until ML jobs finish

Check Helm releases

helm list -n rootcause

Check pod logs

# Platform logs
kubectl logs -l app=platform -n rootcause --tail=50

# Data service logs
kubectl logs -l app=data-service -n rootcause --tail=50

# Check logs from a crashed container
kubectl logs <pod-name> -n rootcause --previous

Common issues

Issue

Cause

Fix

ImagePullBackOff

Missing or invalid pull secret

Verify regcred exists with correct credentials

Degraded with Helm error

Chart values validation failure

Check status.lastError and fix the configuration in the wizard

FusionAuth stuck in Maintenance Mode

silentMode not enabled

Ensure operator version includes the silent mode fix (0.1.14+)

OIDC login returns invalid_redirect_uri

Wildcard validation not enabled

Ensure operator version includes authorizedURLValidationPolicy: AllowWildcards (0.1.16+)

PVC stuck in Pending

No matching storage class

Set storageClass in the bootstrap wizard to match your cluster

Pods stuck in Pending

Insufficient cluster resources

Scale up nodes or reduce replica counts

ML job pod Pending for a few minutes

Compute node pool scaling from zero

Normal. If it persists, check your cluster autoscaler and the pool's taint/label.

Phase stuck in WaitingForMLJobs

A requested action was deferred until running ML jobs finish

Wait for the jobs (see status.activeMlJobs), or re-request the action without the deferral

CrashLoopBackOff

Application error on startup

Check pod logs with kubectl logs <pod-name> --previous

Upgrade stuck in Reconciling

Operator unable to complete reconciliation

Check operator logs, then try Force reconcile

Upgrade-specific issues

Issue

Cause

Fix

Pods not restarting after upgrade

Old pods still running

kubectl rollout restart deployment <name> -n rootcause

LiteLLM models missing after reinstall

spec.config.dependenciesConfig.litellm.config empty in the CR (e.g., CR restored from an old backup)

Re-add models on the Admin UI LLM page — they are stored in the CR and re-rendered on every reconcile

Configuration errors after upgrade

New chart version has different value schema

Check status.lastError, update wizard fields to match new schema

Quick reference: useful commands

# Get Admin UI master password
kubectl get secret rootcause-bootstrap-auth -n rootcause \
  -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

# Get LiteLLM master password
kubectl get secret rootcause-dependencies-litellm-secrets \
  -n rootcause -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d

# Export CR spec (backup your configuration)
kubectl get rootcauseinstallation rootcause -n rootcause -o yaml > rc-backup.yaml

# Watch all pods
kubectl get pods -n rootcause -w

# Check installation phase
kubectl get rootcauseinstallation -n rootcause -o jsonpath='{.status.phase}'

# Force reconcile
kubectl patch rootcauseinstallation rootcause -n rootcause \
  --type=merge -p '{"spec":{"actions":{"forceReconcileNonce":"'$(uuidgen)'"}}}'

# Check events (sorted by time)
kubectl get events -n rootcause --sort-by='.lastTimestamp'

PreviousDeployment Guide

Last updated 1 month ago