Alerts

The Platform Health pack, the alarm channels and silences are the same as for a single node: see Alarms and notifications. Set a channel first: a cluster that fails over quietly also fails quietly.

printf '%s\n' "https://hooks.example.com/stratum" | \
  stratumctl admin observability operator-webhook set --url-stdin

Console: Admin › Substrate › Alerts, with Firing, Definitions and Silences. The same list is GET /v1/observability/admin/alerts. Every node runs its own alert evaluation and Alertmanager, and the Alertmanagers gossip, so losing a node does not stop alarms.

Alarms to know on a cluster

Alarm	Fires when
control-plane-down	Fewer control planes are alive than expected, on a cluster of three or more. Pages within about 3 minutes.
cluster-not-ready	A blocking readiness finding has been open for 10 minutes.
ceph-bootstrap-failed	A requested Ceph bootstrap failed.
backup-rpo-exceeded, backup-attempt-failing	No good platform backup within twice its cadence, or the last attempt failed.
upgrade-failed, node-rolled-back, upgrade-stuck	An upgrade failed, a node switched itself back, or one stayed in one state for two hours.
host-binaries-stale	A node’s host binaries are not those of the release the cluster runs, for 30 minutes while no upgrade runs.
pki-cluster-ca-expiring	The cluster CA expires within 90 days.
ovn-raft-not-member, ovn-chassis-missing, ovn-no-active-northd, ovsdb-down, ovs-bridge-admin-down	The network’s own health: an OVN database member left its cluster, fewer chassis than nodes, no active northd, or Open vSwitch down on a node. Each condition is one Platform Health pack alarm, so it makes one alert.

A node that you shut down or reboot on purpose pages nobody: the control-plane-down and OVN chassis alarms do not count it. A second, unplanned loss still does.

While the cluster settles after a start, an upgrade or a reboot, firing alarms are held back, so a restart does not page anyone.