The cluster at a glance

Start with readiness, then the members, then who leads what.

Question	Command	Console
Can the cluster serve?	stratumctl admin readiness	Admin › Substrate › Overview
Which nodes, in what state?	stratumctl nodes list	Admin › Metal
Who is a member?	stratumctl admin cluster members list	Admin › Substrate › Cluster
Is every part healthy?	stratumctl admin selfcheck
Is every reconciler working?	stratumctl admin reconcilers fleet	Admin › Substrate › Reconcilers
Is Ceph healthy?	GET /v1/admin/storage/ceph/status	Admin › Substrate › Ceph

Admin › Substrate › Cluster: members, roles and who leads each service.

Members

A member is active, or on its way: joining, failed (a join that did not finish), removing, or decommissioned (kept for the record, and refused by the other nodes). Every membership change raises a generation, and each node reports the generation it applied. A node more than 10 minutes behind shows as MembershipLagging.

Who leads what

Each service elects its own leader, so they can sit on different nodes. Each service address follows its leader.

curl -s "$API/v1/ha/status" -H "Authorization: Bearer $TOKEN" | jq '{clusterState, controlPlaneLeader, vips}'

In the answer	Healthy
clusterState	healthy: one Patroni leader, every member streaming. degraded: Postgres unreachable or a member not streaming.
controlPlaneLeader	A node id. It runs the singleton workers, and the API address follows it.
members[]	Postgres: one leader (the primary), one sync_standby, the rest replica, lag near 0.
openbao	Every member reachable and not sealed, one isLeader. A sealed member is a red flag: nodes unseal themselves at boot.
ovn	The northbound and southbound databases each have one leader, and may differ.
vips[]	Every address has a holder, and it is that service’s leader.

During a failover the lost node shows reachable: false, and for a second or two a service can show no leader or two. Read it again. The raw view is on a node:

sudo podman exec postgres-ha patronictl -c /etc/patroni/patroni.yml list
sudo podman exec openbao bao operator raft list-peers
sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound

Reconcilers

Reconcilers drive the cloud toward its desired state: stratumctl admin reconcilers fleet shows each as the cluster sees it, whichever node answers. Prefer it to reconcilers list, which shows only the node that answered (admin reconcilers).

Health	Means
converged	The last pass found nothing to do.
degraded, failed	The pass found drift it could not fix, or errored.
never_ran	No pass has run yet.
unknown	No node reports it recently, or none leads it.
disabled	Its service is switched off, or waits for something it needs. Not a fault.

A leader-gated reconciler takes the leader’s health: the other nodes skip the pass, and their never_ran means nothing. A per-node reconciler takes the worst node’s. A node that stopped reporting is STALE and left out of the rollup.

disabled on a new cluster is normal for the object storage and managed database reconcilers. Object storage waits for Ceph and its object gateway, and managed databases need object storage. They start by themselves once the gateway is deployed. The note column says which one they wait for.

Next: alerts.