Scaling

This guide covers scaling your RootCause.ai deployment for production workloads. Proper scaling ensures responsive performance under load while optimizing resource costs.

---

Component Scaling Guidelines

Different components have different scaling characteristics:

Component	Scaling Type	Bottleneck
Platform (UI)	Horizontal	Concurrent users
Data Service	Horizontal	API requests
ML Jobs	Horizontal	Discovery/simulation queue
MongoDB	Vertical + Sharding	Data volume
PostgreSQL	Vertical	Temporal workflows
Redis	Vertical	Cache size
RabbitMQ	Horizontal	Message throughput

---

Replica Counts

Development / Testing

platform:
  replicaCount: 1

dataService:
  replicaCount: 1

mlJobs:
  replicaCount: 2

Production (Small)

Up to 50 concurrent users, moderate simulation load:

platform:
  replicaCount: 2

dataService:
  replicaCount: 3

mlJobs:
  replicaCount: 5

Production (Medium)

Up to 200 concurrent users, heavy simulation load:

platform:
  replicaCount: 3

dataService:
  replicaCount: 5

mlJobs:
  replicaCount: 10

Production (Large)

Enterprise scale:

platform:
  replicaCount: 5

dataService:
  replicaCount: 10

mlJobs:
  replicaCount: 20

---

Resource Limits

Platform (Web UI)

platform:
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 2
      memory: 4Gi

Data Service

dataService:
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 4
      memory: 8Gi

ML Jobs

ML Jobs are CPU and memory intensive:

mlJobs:
  resources:
    requests:
      cpu: 2
      memory: 4Gi
    limits:
      cpu: 8
      memory: 16Gi

For very large models (100+ variables):

mlJobs:
  resources:
    requests:
      cpu: 4
      memory: 8Gi
    limits:
      cpu: 16
      memory: 32Gi

---

Horizontal Pod Autoscaling

Enable automatic scaling based on load:

Platform HPA

platform:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

Data Service HPA

dataService:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 15
    targetCPUUtilizationPercentage: 60

ML Jobs HPA

mlJobs:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 30
    targetCPUUtilizationPercentage: 80

Or scale based on RabbitMQ queue depth:

mlJobs:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 30
    metrics:
      - type: External
        external:
          metric:
            name: rabbitmq_queue_messages
            selector:
              matchLabels:
                queue: ml-jobs
          target:
            type: Value
            value: 10

---

Database Scaling

MongoDB

MongoDB uses replica sets. For higher throughput:

mongodb:
  replicaSet:
    members: 3  # Minimum for HA
  
  resources:
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 8
      memory: 32Gi
  
  persistence:
    size: 500Gi

For very large datasets, consider sharding.

PostgreSQL

PostgreSQL scales vertically:

postgres:
  primary:
    resources:
      requests:
        cpu: 2
        memory: 4Gi
      limits:
        cpu: 8
        memory: 16Gi
    
    persistence:
      size: 200Gi

For read-heavy workloads, add replicas:

postgres:
  readReplicas:
    replicaCount: 2

Redis

redis:
  master:
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 4
        memory: 8Gi
  
  replica:
    replicaCount: 2

---

Node Affinity

Ensure ML Jobs run on appropriate nodes:

mlJobs:
  nodeSelector:
    node-type: compute
  
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "ml"
      effect: "NoSchedule"

For GPU workloads (local LLM):

mlJobs:
  nodeSelector:
    accelerator: nvidia-gpu
  
  resources:
    limits:
      nvidia.com/gpu: 1

---

Pod Disruption Budgets

Ensure availability during updates:

platform:
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

dataService:
  podDisruptionBudget:
    enabled: true
    minAvailable: 2

---

Monitoring Scaling Decisions

Track these metrics to inform scaling:

Platform/Data Service:

• Request latency (P50, P95, P99)

• Request rate (RPS)

• Error rate

• CPU/Memory utilization

ML Jobs:

• Queue depth (pending jobs)

• Job duration

• Success/failure rate

• CPU/Memory utilization

Databases:

• Connection count

• Query latency

• Replication lag

• Disk I/O

# Example Prometheus queries
# Request rate
rate(http_requests_total{service="data-service"}[5m])

# Queue depth
rabbitmq_queue_messages{queue="ml-jobs"}

# P95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

---

Cost Optimization

Right-size resources:

1. Start with recommended values

2. Monitor actual usage for 1-2 weeks

3. Adjust requests to match P95 usage

4. Set limits at 2x requests

Use spot/preemptible instances for ML Jobs:

mlJobs:
  tolerations:
    - key: "kubernetes.io/e2e-eviction"
      operator: "Exists"
  
  nodeSelector:
    node.kubernetes.io/lifecycle: spot

Scale down during off-hours:

# Using KEDA or custom CronJob
mlJobs:
  autoscaling:
    minReplicas: 1  # Night/weekend
    # Scheduled scale up during business hours

---

Performance Benchmarks

Typical performance expectations:

Workload	Recommended Config	Expected Performance
Small (10 users, light ML)	2/2/3 replicas	<500ms API response
Medium (50 users, moderate ML)	3/5/10 replicas	<1s API, <10min discovery
Large (200 users, heavy ML)	5/10/20 replicas	<2s API, <30min discovery

Discovery time scales with:

• Number of variables (exponential impact)

• Number of rows (linear impact)

• Data complexity (non-linear)

---

Next Steps

With scaling configured:

• Review Upgrading for zero-downtime updates

• Set up monitoring and alerting

• Implement backup procedures