Backup & Recovery

The default overlay ships a pg_dump-based backup CronJob that runs every 6 hours, writes a gzipped logical dump to a backup PVC, and (optionally) syncs that PVC to an offsite remote via rclone.

This is intentionally simple — no operators, no S3 SDKs in the cluster, no WAL archiving. If you need point-in-time recovery (PITR) or block-level base backups, see Upgrade path: barmanObjectStore below.

Current state

Offsite sync is live on both clusters as of 2026-09-04, to Backblaze B2 — commstate-uat/postgres and commstate-live/postgres, separate buckets with separate application keys so neither cluster's credential reaches the other's dumps. Before that date the offsite job had never once succeeded.

RCLONE_REMOTE and RCLONE_PATH now ship set in the overlays, so an apply restores them. The rclone.conf itself is the one piece created out of band — see Enabling offsite sync.

Still outstanding: .spec.backup is empty on both CNPG clusters, so there is no WAL archiving and no point-in-time recovery. The recovery point remains up to 6 hours.

Three things a database backup alone will not restore

APP_KEY. Several columns are encrypted with it — plugin credentials, module settings, reseller bank details. A dump restored against a different APP_KEY leaves those permanently unreadable, and the restore reports success while doing it. Back up commstate-secrets alongside the dump.

passport-keys. The OAuth signing keypair. Regenerating it invalidates every issued token, so users and API clients are logged out on restore.

The redis-data volume. It holds the Horizon queue. It is durable now (PVC plus AOF) but is not covered by any backup job — a restore starts with an empty queue.

# store these off-cluster, alongside the dump
kubectl --context=commstate-live get secret commstate-secrets passport-keys \
  -n commstate -o yaml > commstate-live-secrets.yaml

Architecture

CNPG cluster (commstate-db-r service)
        │
        ▼  pg_dump --format=plain | gzip
   /backups/commstate-YYYYMMDD-HHMMSS.sql.gz   (PVC: backup-pvc, 20Gi)
        │
        ▼  rclone sync (optional, every 6h, +15min after the dump)
   <remote>:commstate-backups/<env>/           (B2, R2, S3, GDrive — anything rclone speaks)

Layer	What	RPO	Schedule
Local dump	pg_dump to backup-pvc	up to 6h	every 6h on the hour
Offsite sync	rclone to your remote	up to 6h 15m	every 6h at :15
Retention	dumps older than LOCAL_RETENTION_DAYS are deleted	—	each run

RPO (Recovery Point Objective): up to 6 hours with the default schedule. Shorten by editing the CronJob schedule if you need tighter.

What ships out of the box

Everything below is part of k8s/overlays/default/backup.yaml and applies on the same kubectl apply -k you use for the rest of the stack:

• PersistentVolumeClaim/backup-pvc (20 GiB)
• ConfigMap/backup-config — LOCAL_RETENTION_DAYS=14, plus RCLONE_REMOTE/RCLONE_PATH (b2 and a per-cluster prefix; the live overlay patches the path to its own)
• CronJob/postgres-backup-local — runs pg_dump → /backups/commstate-<stamp>.sql.gz
• CronJob/postgres-backup-offsite — runs rclone sync /backups <remote>:<path>, and fails rather than no-ops when the remote or the rclone.conf is missing

There is deliberately no Secret/backup-offsite in the manifests. An empty placeholder used to ship there, and on 2026-09-04 an apply of the overlay overwrote UAT's real credentials with "" and took its offsite copies down until it was noticed. A placeholder for a secret is not a safe default — it is a scheduled deletion of whatever is actually configured. The secret volume is optional: true, so a cluster without one still schedules the job and gets a loud failure instead of a Pending pod.

• NetworkPolicy/allow-postgres-backup — egress to Postgres on 5432, DNS, and 443 (rclone)

The local dump uses the commstate-db-app secret (created during CNPG setup) and connects to the commstate-db-r service, which prefers replicas — so dumps don't load the primary.

Manual / on-demand dump

Run an immediate backup outside the schedule:

kubectl -n commstate create job --from=cronjob/postgres-backup-local \
  manual-$(date -u +%Y%m%d-%H%M%S)
kubectl -n commstate wait --for=condition=complete --timeout=180s \
  job -l job-name=manual-...

Inspect what's on the PVC:

kubectl -n commstate run --rm -i ls-backups --image=alpine --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"x","image":"alpine","command":["ls","-lh","/backups"],"volumeMounts":[{"name":"b","mountPath":"/backups"}]}],"volumes":[{"name":"b","persistentVolumeClaim":{"claimName":"backup-pvc"}}]}}'

Restore

The default backup is a logical (pg_dump --format=plain) gzipped SQL file. Restore it into the running CNPG cluster by streaming through psql:

# 1. Pick the dump
DUMP=commstate-20260501-080635.sql.gz

# 2. Stream from the backup PVC into the primary
kubectl -n commstate run --rm -i restore-helper \
  --image=postgres:16-alpine --restart=Never \
  --env="PGHOST=commstate-db-rw" \
  --env="PGUSER=postgres" \
  --env="PGPASSWORD=$(kubectl -n commstate get secret commstate-db-superuser -o jsonpath='{.data.password}' | base64 -d)" \
  --env="PGDATABASE=commstate" \
  --overrides='{"spec":{"containers":[{"name":"x","image":"postgres:16-alpine","command":["sh","-c","gunzip -c /backups/'"$DUMP"' | psql"],"volumeMounts":[{"name":"b","mountPath":"/backups"}]}],"volumes":[{"name":"b","persistentVolumeClaim":{"claimName":"backup-pvc"}}]}}'

The dump is taken with --clean --if-exists, so it drops and recreates each object — running it on a non-empty database will overwrite existing data.

After restore, fix table ownership (the dump preserves no owner info):

kubectl -n commstate exec commstate-db-1 -c postgres -- \
  psql -U postgres -d commstate -c "
    DO \$\$ DECLARE r record; BEGIN
      FOR r IN SELECT 'ALTER TABLE \"' || tablename || '\" OWNER TO commstate;' AS sql FROM pg_tables WHERE schemaname='public' LOOP EXECUTE r.sql; END LOOP;
      FOR r IN SELECT 'ALTER SEQUENCE \"' || sequencename || '\" OWNER TO commstate;' AS sql FROM pg_sequences WHERE schemaname='public' LOOP EXECUTE r.sql; END LOOP;
    END \$\$;
  "

Enabling offsite sync

RCLONE_REMOTE and RCLONE_PATH ship in the overlays. The credential does not, and must be created once per cluster — including on any cluster rebuilt from scratch, where this is the step that is easy to forget:

# The remote is reached over the S3 API rather than B2's native one, so the
# same key works for both rclone and the application's `s3` filesystem disk.
cat > /tmp/rclone.conf <<'EOF'
[b2]
type = s3
provider = Other
access_key_id = <application-key-id>
secret_access_key = <application-key>
endpoint = https://s3.us-west-004.backblazeb2.com
region = us-west-004
acl = private
EOF

kubectl -n commstate create secret generic backup-offsite \
  --from-file=rclone.conf=/tmp/rclone.conf \
  --dry-run=client -o yaml | kubectl apply -f -

rm /tmp/rclone.conf

Use each cluster's own application key — the one already in its commstate-s3 secret — so UAT cannot write into the live bucket.

Verify without waiting six hours for the schedule:

kubectl -n commstate create job offsite-check --from=cronjob/postgres-backup-offsite
kubectl -n commstate wait --for=condition=complete --timeout=180s job/offsite-check
kubectl -n commstate logs job/offsite-check   # ends with a listing of the remote
kubectl -n commstate delete job offsite-check

Any rclone backend works — swap the [b2] section for r2, s3, gdrive. The CronJob uses rclone sync --checksum, so dumps deleted locally by retention are deleted remotely too; the remote is a mirror, not an archive.

Tuning

Setting	Where	Default	Notes
Backup schedule	CronJob/postgres-backup-local .spec.schedule	0 */6 * * *	every 6h on the hour (UTC)
Offsite schedule	CronJob/postgres-backup-offsite .spec.schedule	15 */6 * * *	15 min after each local dump
Local retention	ConfigMap/backup-config LOCAL_RETENTION_DAYS	14	deleted by find -mtime +N -delete
Backup PVC size	PersistentVolumeClaim/backup-pvc .resources.requests.storage	20Gi	bump if dumps grow
Source endpoint	hard-coded in CronJob env (PGHOST=commstate-db-r)	replicas-preferred	keeps load off the primary

Disaster recovery scenarios

Database corruption / accidental drop

1. Pick the most recent good commstate-*.sql.gz from the backup PVC (or your offsite remote).
2. Restore via the recipe above.
3. Reissue any writes since the dump from app logs / event store if you have one.

Whole CNPG cluster lost

1. Recreate the cluster (kubectl apply -k k8s/overlays/default) and the bootstrap secrets (Database: CloudNativePG).
2. Wait for the cluster to reach READY 3/3.
3. Restore the latest dump.

Whole node lost (single-node k3s)

The local-path PVCs live on the node — they're gone with the node. Restore from your offsite copy. This is the case offsite sync exists for; if you haven't enabled it yet, do it now.

Upgrade path: barmanObjectStore

When you outgrow logical dumps and want continuous WAL archiving + PITR, switch the cluster to CNPG's native barmanObjectStore backups:

# In k8s/overlays/default/postgres.yaml, add to the Cluster spec:
spec:
  backup:
    barmanObjectStore:
      destinationPath: s3://commstate-backups/
      endpointURL: https://<r2-or-s3-endpoint>
      s3Credentials:
        accessKeyId:     { name: backup-s3-creds, key: ACCESS_KEY }
        secretAccessKey: { name: backup-s3-creds, key: SECRET_KEY }
    retentionPolicy: "30d"

# And a ScheduledBackup CRD for daily base backups:
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: commstate-db-daily
spec:
  schedule: "0 0 2 * * *"   # 02:00 daily
  cluster: { name: commstate-db }
  method: barmanObjectStore

That gives you second-level RPO and lets you bootstrap a brand-new cluster from any point in time using bootstrap.recovery.recoveryTarget.targetTime. The pg_dump CronJob can stay (belt-and-braces) or be removed once barman is verified.