Commstate Docs

Environments & Clusters

Commstate runs on two independent k3s clusters. They share no control plane, no storage and no failure domain — the only thing that crosses between them is telemetry.

live uat
host ranger endurance
arch x86_64 arm64
spec 8 cores / 15 GB (dedicated) 14 cores / 31 GB (shared vCPU)
domains commstate.app autocom.wexron.io
k3s v1.36.3 v1.31.5
also hosts nothing else GitLab, CI runners, monitoring, other projects

The asymmetry is deliberate. endurance carries GitLab, its CI runners and the monitoring stack, so it runs at a high load average with noticeable CPU steal. ranger is a dedicated root server with zero steal and carries production alone.

Reaching them

Both clusters are reached over the tailnet, never over their public IPs — the Kubernetes API is not exposed to the internet on either host.

kubectl config get-contexts
#   commstate-live   commstate-live   commstate-live   commstate
#   commstate-uat    commstate-uat    commstate-uat    commstate

kubectl --context=commstate-live get pods
k9s --context=commstate-uat

Contexts, clusters and users all follow commstate-<env>, and both default to the commstate namespace.

If a new cluster's API rejects your certificate

k3s only puts the node IP, localhost and the hostname in its API server certificate. Reaching it over a tailnet address fails TLS verification until that address is added:

# /etc/rancher/k3s/config.yaml
tls-san:
  - 100.82.39.108   # this node's tailnet address
  - ranger

Then systemctl restart k3s — the certificate is regenerated on start.

Overlays

k8s/overlays/
  default/    UAT — the base every other overlay builds on
  live/       production; layers on ../default
  rings/      module ring pinning (stable / canary / edge)

live deliberately layers on default rather than duplicating it, so the two environments cannot drift structurally. It overrides exactly three things:

  • hostnames — commstate.app instead of autocom.wexron.io
  • image tags — immutable v2.0.0 instead of :latest
  • TLS — cert-manager HTTP-01 at ingress-nginx

Everything else is the same manifest set.

Why live pins image tags and UAT does not

:latest cannot be rolled back. kubectl rollout undo re-pulls the same moving tag, and it cannot answer "what is running right now". UAT tolerates that in exchange for fast iteration; production does not.

NEXT_PUBLIC_API_URL is overridden explicitly

Next.js inlines NEXT_PUBLIC_* at build time. Without an explicit override in the live overlay, the production frontend loads perfectly and talks to the UAT API — a failure that surfaces only when data turns up somewhere unexpected. The value is substituted at container start, so overriding it is a config change rather than a rebuild.

Adding an environment

Adding dev is a values file and a namespace, not a server. Each environment costs roughly 2.5 GB of memory and 0.3 CPU cores measured against the running clusters, so a third fits on either host without new hardware.

  1. Create k8s/overlays/dev/kustomization.yaml layering on ../default, overriding hostnames and image tags.
  2. Create the namespace and its out-of-band secrets (below).
  3. Apply the CNPG cluster first and wait for it, then everything else.
  4. Add a commstate-dev kubeconfig context following the same naming.

Put non-production environments on endurance and keep ranger for production alone. The constraint is blast radius, not capacity: a dev mistake that fills a disk or exhausts memory should not sit one namespace away from live.

Secrets are deliberately not in Git

Five secrets are created out-of-band per environment:

secret contents
commstate-secrets APP_KEY, DB_PASSWORD, REDIS_PASSWORD, REVERB_*, AGENT_SHARED_SECRET
commstate-db-app CNPG application role
commstate-db-superuser CNPG superuser
passport-keys OAuth RSA keypair
gitlab-registry image pull credential

REDIS_PASSWORD must be empty. The Redis manifest has no --requirepass, so a non-empty value makes every client fail with ERR AUTH called without any password configured.

Back these up somewhere off the cluster. Several database columns are encrypted with APP_KEY — plugin credentials, module settings, reseller bank details. A dump restored against a different APP_KEY leaves those columns permanently unreadable, and the restore reports success while doing it.

Two things that bite on a fresh cluster

cert-manager cannot issue anything

The namespace runs default-deny-ingress, and cert-manager creates its HTTP-01 solver pods on the fly, so no app-scoped policy covers them. Challenges sit pending, reporting:

Waiting for HTTP-01 challenge propagation: wrong status code '502'

A 502 rather than a connection error is the tell — ingress-nginx routes to the solver correctly and the policy drops the packet, so the solver looks healthy throughout and cert-manager looks at fault. allow-acme-solver in k8s/base/per-ring/network-policies.yaml permits ingress-nginx to reach solver pods on 8089. This affects renewals as well as first issuance, so a cluster that issued certificates before the policy existed will fail silently 90 days later.

Image pushes fail with MANIFEST_BLOB_UNKNOWN

Docker 29 defaults to the containerd image store, which pushes OCI manifests that GitLab's registry (v4.38) rejects. On any host used as a build runner:

// /etc/docker/daemon.json
{ "features": { "containerd-snapshotter": false } }

Restart Docker afterwards. The storage driver should read overlay2, not overlayfs.