When a node or the leader dies

A cluster of three survives losing one node. Each service fails over on its own, and each service address follows its leader.

Lost	What happens	Typical time
The Postgres primary	Patroni promotes the synchronous standby, with no data loss. The Postgres address follows it.	About 8 seconds
The OpenBao leader	The members elect another. Reads and writes resume.	5 to 10 seconds
An OVN database leader	The members elect another. Existing network flows keep running, because traffic is switched on the nodes and not through OVN central. New network changes resume.	5 to 15 seconds
The control-plane leader	Another node takes the leader lock and runs the singleton workers. The API address moves with it.
A whole node	All of the above for what it led, the API address moves to another node, and its instances on Ceph restart elsewhere.	The address about 3 seconds, instances about 70 seconds

Times are typical, not promises. The control plane never needs you to start these.

Instances on a lost node

An instance whose root volume is on a Ceph class (ceph-rbd) restarts on a surviving node:

1. The node stops heartbeating, and after about 60 seconds is marked unknown.
2. Its Ceph client is fenced, so a node that comes back cannot write to the volume.
3. The volume is re-attached on a survivor and the instance boots there.

This needs a ready survivor with room, and a healthy majority: more ready nodes than dead ones, which guards against moving instances during a network split. Instances on a node-local pool cannot move, since their disk is on the dead node: they come back when the node does. A control plane that has just started judges no node until it has watched heartbeats for about 2 minutes.

Readiness warns EvacuationNotArmed when the cluster would not evacuate: the evacuation is in dry-run, or nothing is on Ceph.

What you do

1. Look. stratumctl admin readiness, stratumctl nodes list, and GET /v1/ha/status (at a glance). Give it a few minutes: the cluster converges by itself, and most transient breakage fixes itself.
2. Bring the node back. Power it on. It rejoins with no action. Its instances that restarted elsewhere are not started twice, and the copies it held are cleaned up.
3. If Ceph reports CEPHADM_REFRESH_FAILED after the node returns, and the node has processes stuck in D state (ps -eo stat | grep -c '^D'), reboot that node once: a fenced device with dead I/O clears only on a reboot.
4. If it will not come back, remove it.

Losing a second node of three stops Postgres, OpenBao, OVN and Ceph’s monitors from forming a majority. Bring one back. If a service stays wedged after a failed join or a power cut, see Recovery.