Observability

One monitoring stack serves both environments. It lives on endurance alongside UAT; ranger runs collectors only and ships to it.

live (ranger)                          uat (endurance)
  promtail          ──tailnet──▶  Loki      ┐
  prometheus agent  ──tailnet──▶  Prometheus├──▶ Grafana
  (no TSDB, no UI)                Alertmanager┘

The alternative — a full stack per cluster — gives two half-populated panes of glass and doubles the thing you have to maintain. Live has no Grafana, no Alertmanager and no local storage; its Prometheus runs in agent mode, which scrapes and forwards without a TSDB.

Transport

Both feeds cross the tailnet: private, encrypted, and measured at ~2 ms between the two hosts.

Neither backend is published as a NodePort. endurance has no host firewall (INPUT ACCEPT) and already exposes six NodePorts, and neither Loki OSS nor the Prometheus remote-write receiver has any authentication. A NodePort would put an open log-ingest and an open metric-write endpoint on the public internet.

Two systemd units bind them to the Tailscale address only:

unit	binds	forwards to
`tailnet-loki-bridge`	`100.100.50.50:3100`	Loki ClusterIP
`tailnet-prom-bridge`	`100.100.50.50:9090`	Prometheus ClusterIP

Both resolve the ClusterIP at start rather than baking it in, so recreating a Service does not silently break shipping. Verified reachable from ranger and refused from the public IP.

Switching environment

Environment is chosen by datasource, not by typing a label into every query. Each dashboard carries an Environment variable; changing it swaps every panel.

datasource	backend	mechanism
`Prometheus (Live)`	`prom-proxy-live:8080`	`prom-label-proxy` pins `env=live`
`Prometheus (UAT)`	`prom-proxy-uat:8080`	`prom-label-proxy` pins `env=uat`
`Loki (Live)`	`loki:3100`	`X-Scope-OrgID: live`
`Loki (UAT)`	`loki:3100`	`X-Scope-OrgID: uat`

The two backends need different mechanisms:

Prometheus has no tenancy and Grafana OSS cannot force a label matcher on a datasource, so a proxy pins it server-side. -error-on-replace is set, so a query carrying its own env matcher is rejected rather than silently rewritten.
Loki has native multi-tenancy, and prom-label-proxy does not route /loki/api/v1/* paths — it returns 404. promtail stamps tenant_id on ingest; the datasource carries the header.

There is also a plain Prometheus datasource that returns both environments merged. It is excluded from the dashboard variable by regex, and is only useful for deliberately cross-environment queries.

Why both clusters must use the same chart

Live originally ran prometheus-community/prometheus while UAT ran kube-prometheus-stack. They name the same targets differently:

live (old)	uat
`kubernetes-nodes`	`kubelet`
`kubernetes-api-servers`	`apiserver`
`kubernetes-service-endpoints`	`coredns`, `node-exporter`, …

Twelve UAT jobs, six live jobs, zero overlapping names. No dashboard could work across both regardless of how they were labelled. Live now runs kube-prometheus-stack in agent mode and the conventions line up.

Labelling

env is the origin label — not cluster. CloudNativePG already publishes a cluster label carrying the Postgres cluster name (commstate-db, deplio-pg), so reusing it collides with real series and makes both meanings unqueryable.

Live stamps env=live through its agent's externalLabels on remote-write. UAT cannot do the same: externalLabels apply only to remote-write, federation and alerts, never to locally stored series, and Prometheus Operator has no global scrape-relabel hook. Each chart-managed monitor is therefore relabelled individually in prometheus-values.yaml, and the database PodMonitor through CNPG's podMonitorRelabelings — the operator reverts direct patches to the PodMonitor it generates.

The kubelet monitor needs all four of its relabel keys (relabelings, cAdvisorRelabelings, probesRelabelings, resourceRelabelings). Setting only the first leaves cadvisor, probes and resource metrics unstamped, which shows up as job="kubelet" appearing both with and without an env label.

Alerting

26 rules across seven groups in k8s/monitoring/alerting-rules.yaml, covering pod health, resources, API, database, Redis, HPA and platform.

Alerts currently route to a receiver named "null" — the chart default, which discards everything. Rules evaluate correctly and notify nobody. Adding a real receiver to alertmanager.config in prometheus-values.yaml is the single highest-value change left in this stack.

Alert rules can be silently dead

An alert referencing a metric that does not exist never fires and never reports an error. Eight such rules were found in this repo, including both backup alerts — which referenced cnpg_pg_last_successful_backup_timestamp, a series CNPG only publishes once backups are configured. The alerts meant to warn about missing backups were silenced by the exact condition they guarded against.

Before trusting a new rule, confirm its metric exists:

kubectl exec -n monitoring deploy/monitoring-grafana -c grafana -- \
  wget -qO- 'http://monitoring-kube-prometheus-prometheus:9090/api/v1/label/__name__/values' \
  | grep -o 'your_metric_name'

and check rule health after applying:

curl -s http://<prometheus>/api/v1/rules | jq '.data.groups[].rules[] | select(.health != "ok")'

Retention

store	retention	volume
Loki	168h (7 days)	5 Gi
Prometheus	7 days	5 Gi

Loki ingests roughly 120k lines/day across both environments, comfortably inside its volume. Prometheus was raised to a 2 Gi memory limit — 512 Mi was sized to scrape one cluster and OOMKilled within minutes of the remote-write receiver being enabled, because ingesting a second cluster roughly doubles the head block.

Enabling Loki's auth_enabled moved existing logs under an implicit fake tenant that the per-environment datasources cannot see. With 7-day retention they age out rather than needing migration.