Self-Hosted: Running Hudu highly available on AWS

This article covers configuring the self-hosted Helm chart for a highly available (HA) deployment on AWS: no single point of failure in the pieces that keep Hudu itself reachable, and a safe path through upgrades. It assumes you're already familiar with a standard install — see "Deploying Hudu with the Helm chart" if you haven't installed the chart before.

What "highly available" means here

HA, for this chart, means the web tier and the database can survive a node loss, an AZ loss, or a rolling upgrade without an outage. That's the boundary the rest of this article is built around, and it's worth being explicit about what's inside it and what isn't:

Web is stateless and scales horizontally — this is the piece the chart is designed to make redundant.
The database is where your data actually lives, so it's the other piece that has to stay up (or fail over quickly) for Hudu to be up.
Redis backs Sidekiq's job queues and ActionCable. It matters for Hudu to function well, but it isn't a system of record — see "Redis" below.
The worker is not part of this boundary. As of this chart version it runs as a single replica, full stop — there's no HA story for it today. A worker outage delays background jobs (emails, imports, scheduled reports) until it recovers, but it does not take down the app: as long as web and the database are healthy, users can keep using Hudu. See "Worker/scheduler" below for why this is a hard chart constraint, not a tuning knob.

Reference architecture

                         ┌─────────────────────┐
                         │   ALB / Ingress      │
                         │ (multi-AZ, TLS term) │
                         └──────────┬───────────┘
                                    │
                 ┌──────────────────┼──────────────────┐
                 │                  │                   │
           ┌─────▼─────┐      ┌─────▼─────┐      ┌─────▼─────┐
           │  web pod  │      │  web pod  │      │  web pod  │   AZ-a / AZ-b / AZ-c
           │ (replica) │      │ (replica) │      │ (replica) │   (replicaCount or HPA)
           └─────┬─────┘      └─────┬─────┘      └─────┬─────┘
                 │                  │                   │
                 └──────────────────┼──────────────────┘
                                    │
                  ┌─────────────────┼─────────────────┐
                  │                 │                  │
           ┌──────▼──────┐   ┌──────▼──────┐    ┌──────▼──────┐
           │  RDS for     │   │ ElastiCache │    │     S3      │
           │  PostgreSQL  │   │ (Multi-AZ)  │    │  (bucket)   │
           │  (Multi-AZ)  │   └─────────────┘    └─────────────┘
           └──────────────┘

           ┌───────────────┐
           │  worker pod   │   single replica — owns Sidekiq's
           │  (singleton)  │   cron/scheduler, not part of the
           └───────────────┘   web/DB uptime boundary above

Everything above the dotted line between "web" and the datastores is what this chart manages directly. RDS, ElastiCache, and S3 are AWS-managed services the chart points at (externalDatabase.*, externalRedis.*, storage.type: s3) — it doesn't provision or operate them.

Web tier

Scale web.replicaCount manually, or turn on web.autoscaling.enabled to let a HorizontalPodAutoscaler do it based on CPU (and optionally memory) utilization:

web:
  replicaCount: 3
  # or:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 10

A few things that matter once you're running more than one web pod:

storage.type must be s3 if you turn on autoscaling — the chart refuses to render web.autoscaling.enabled: true with storage.type: pvc, because there's no way to confirm at install time that your StorageClass actually supports ReadWriteMany across nodes, and a silent failure there (uploads invisible across pods) is worse than a loud one at install. Use real S3, not the bundled MinIO (storage.s3.useLocalS3: true) — see "Object storage" below.
A PodDisruptionBudget renders automatically once web.replicaCount > 1 or autoscaling is on (web.podDisruptionBudget, default minAvailable: 1) — this is what keeps a node drain or cluster upgrade from evicting every web pod at once.
Spread replicas across AZs. The chart doesn't do this for you by default — set web.affinity with a podAntiAffinity rule (there's a commented example in values.yaml) so the scheduler prefers placing web pods on different nodes/zones.
If you're changing web.autoscaling.enabled on a live release, read the README's "Autoscaling" section first — flipping it in either direction causes a one-time pod-count collapse if you don't set minReplicas/ replicaCount to match your current pod count before the upgrade that flips it.

Worker/scheduler

worker.replicaCount must stay at 1 — the chart's template hard-fails the render if you set it to anything else:

worker.replicaCount must be 1 — the worker process owns Sidekiq's cron/scheduler; running more than one replica duplicates scheduled jobs.

There's also no PodDisruptionBudget on the worker — with only one replica to protect, a minAvailable: 1 budget would just block every voluntary node drain outright, for no HA benefit.

In practice: treat the worker as a component you can't make redundant yet, not as something you've misconfigured. If it goes down, background jobs queue up in Redis and process once it's back — that's a delay, not data loss, and it doesn't affect whether Hudu itself is reachable. A worker pod that (re)starts while Redis is unreachable is a separate, worse case — see the Redis section's callout below.

Database

Disable the bundled Postgres and point at a managed, highly-available instance instead:

postgresql:
  enabled: false
externalDatabase:
  host: my-hudu-db.cluster-xxxxxxxxxx.us-east-1.rds.amazonaws.com
  port: 5432
  username: hudu
  database: hudu_production
  existingSecret: hudu-db-credentials
  existingSecretPasswordKey: password

On AWS, that's Amazon RDS for PostgreSQL in Multi-AZ mode, or Aurora PostgreSQL if you want faster failover and read replicas. Either gives you the automatic primary/replica failover the bundled single-replica StatefulSet can't — the chart doesn't implement Postgres streaming replication or a connection router itself, so this is deliberately left to your managed database rather than something the chart takes on.

Prefer existingSecret over a plaintext externalDatabase.password — point it at a Secret populated from AWS Secrets Manager (e.g. via the External Secrets Operator) rather than committing the RDS password anywhere. The chart never reads the secret value itself at render time, only the Secret name/key, so this works the same way regardless of how the Secret gets populated.

Redis

Same pattern — disable the bundled Redis, point at a managed instance:

redis:
  enabled: false
externalRedis:
  host: my-hudu-cache.xxxxxx.ng.0001.use1.cache.amazonaws.com
  port: 6379
  existingSecret: hudu-redis-credentials
  existingSecretUrlKey: REDIS_URL

On AWS, that's Amazon ElastiCache — either the Redis OSS or Valkey engine, in a Multi-AZ replication group with automatic failover enabled. The chart has no Sentinel/cluster-mode integration of its own; it expects externalRedis.host to already be a single, reliable endpoint, which is exactly what an ElastiCache Multi-AZ primary endpoint gives you.

Redis doesn't need a backup plan. It holds Sidekiq's queues and ActionCable state — not customer data. If it's lost, whatever background jobs were in flight get dropped and need to be retriggered; nothing a client created is lost, and there's nothing to restore.

Multi-AZ failover minimizes a Redis outage — it doesn't eliminate the app's exposure to one. As of this app version, nothing that talks to Redis directly has a rescue or fallback around it: a Sidekiq enqueue call made from a web request (uploading a photo, refreshing a website check, starting a Hudini AI chat) raises straight out to a 500 if Redis is unreachable at that moment, rather than degrading gracefully or falling back to the database. /healthcheck also only checks Postgres, so pods stay in rotation and keep taking traffic through a Redis blip. Worse, a worker pod that (re)starts during that window will crash-loop — a startup hook enqueues a job unconditionally, and an unreachable Redis there kills the process outright rather than skipping and continuing. ElastiCache Multi-AZ still meaningfully shortens how long this window is (automatic failover vs. manual recovery), but plan for brief, real user-facing errors during it rather than assuming HA Redis makes this transparent.

Object storage

Use real S3, not the bundled evaluation-only MinIO:

storage:
  type: s3
  s3:
    bucket: my-hudu-uploads
    region: us-east-1

If you're on EKS, leave storage.s3.accessKeyId/secretAccessKey/existingSecret all blank and annotate the ServiceAccount with an IRSA role instead:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/hudu-s3-access

The chart detects this and lets the AWS SDK's default credential chain resolve the role via the EKS Pod Identity webhook — no long-lived access keys in your values file. Turn on S3 versioning on the bucket if you want protection against accidental overwrite/delete; that's an S3 setting, not a chart one.

Migrations & upgrade safety

Migrations run in a dedicated Job (post-install,pre-upgrade Helm hook), not at container boot — this is what lets multiple web replicas start together without racing each other on db:prepare. On helm upgrade, the hook runs and completes before the new web/worker pods roll out, retrying up to migrate.backoffLimit times if the database isn't reachable yet.

What this means for a low-downtime upgrade: the chart's ordering guarantees the schema is migrated before new code starts, but it does not give you an automatic expand/contract pattern — the previous app version's web pods are still serving traffic against the database while the hook runs (a few seconds to a couple of minutes, depending on the migration). Write migrations that stay backward-compatible with the currently-running app version — add columns instead of renaming them in place, deploy code that tolerates both old and new schema, that kind of thing — the same discipline you'd want on any live database. If a migration fails outright, the hook fails and blocks the upgrade before any new pods roll, so a bad migration can't take down currently-running pods.

Ingress / load balancing / TLS

Nothing about HA changes the Ingress/TLS setup covered in "Deploying Hudu with the Helm chart" — the AWS Load Balancer Controller (ingress.className: alb) or your ingress controller of choice already handles routing across whichever web replicas are healthy, and across AZs, on its own. Multi-AZ here is a property of the load balancer and your EKS node groups, not something this chart configures.

Backups & disaster recovery

The database is the thing worth planning a real recovery process around — it's where the data is:

RDS/Aurora: enable automated backups and set a retention window that matches your recovery requirements; both support point-in-time recovery out of the box. This replaces the "back up the bundled Postgres PVC" advice in the general setup article — you're no longer running the bundled StatefulSet at all.
S3: versioning (see above) covers accidental overwrite/delete; the bucket itself doesn't need a separate backup process for HA purposes.
Redis: no backup needed — see "Redis" above.
The generated-secrets Secret (<release>-hudu-generated): still applies exactly as in the general setup article — back it up once, it's not reproducible from anything else.

Monitoring & health

Most of "Debugging" in the general setup article still applies unchanged. The HA-specific things worth alerting on:

HPA showing <unknown> for its target metrics — means web.resources.requests got cleared; it renders fine but never scales.
PodDisruptionBudget blocking a drain — expected behavior if a drain would take you below minAvailable, not a bug; make sure you have enough replicas that a single node drain doesn't stall on it.
The migrate Job failing (kubectl logs job/<release>-hudu-migrate) — blocks the whole upgrade; check this first if a helm upgrade hangs.
RDS/ElastiCache failover events — watch these through CloudWatch alarms on your instances, not through the chart/cluster; a failover is usually a short connection blip the app retries through, but it's worth knowing when one happens.

Full example

Combining everything above into one values file:

config:
  domain: hudu.example.com

postgresql:
  enabled: false
externalDatabase:
  host: my-hudu-db.cluster-xxxxxxxxxx.us-east-1.rds.amazonaws.com
  port: 5432
  username: hudu
  database: hudu_production
  existingSecret: hudu-db-credentials
  existingSecretPasswordKey: password

redis:
  enabled: false
externalRedis:
  host: my-hudu-cache.xxxxxx.ng.0001.use1.cache.amazonaws.com
  port: 6379
  existingSecret: hudu-redis-credentials
  existingSecretUrlKey: REDIS_URL

storage:
  type: s3
  s3:
    bucket: my-hudu-uploads
    region: us-east-1

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/hudu-s3-access

web:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 10

ingress:
  host: hudu.example.com
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
  tls:
    enabled: true

helm upgrade --install my-hudu oci://public.ecr.aws/q4n0g0t7/hudu-helm-chart \
  -f values-ha.yaml \
  --set secrets.secretKeyBase=$(openssl rand -hex 64) \
  --set secrets.passwordKey=$(openssl rand -hex 16) \
  --set secrets.twoFactorKey=$(openssl rand -hex 16)