Recovery

Most breakage heals itself: give it a few minutes and watch readiness. Use this page for what does not. Run the commands as root on a control-plane node.

What heals itself

The raft doctor (stratum-raft-doctor.timer, every minute, on every control-plane node) repairs two cases, and only when its guards hold:

• A raft with two voters and no leader for 3 minutes, when the other voter is not an active member: a joiner that crashed mid-join, whose member is joining, failed or gone.
• A node whose Patroni raft files were damaged by a power cut (an empty dump file, and Patroni logging DCS is not accessible), while a partner has a running leader. It copies the raft files to /var/lib/stratum/raft-doctor/backups/ (the newest 5 are kept), clears them and restarts Postgres. Its journal says PATRONI RAFT SELF-HEAL.

It also restarts Patroni once, wiping nothing, when it waits on the raft although a majority answers. It changes nothing on a healthy cluster.

Two active members that cannot reach each other (a dead cable, a down NIC) are left alone. If one of them is really gone, decommission it first: that makes the doctor treat it as not active. Decommissioning does not take the node out of the rafts, so then use the steps below.

A wedged raft

A failed join can leave a dead voter in the first node’s Postgres, OpenBao and OVN rafts. A two-member raft with one dead member cannot elect a leader, and removing the dead member needs a leader.

Symptoms: GET /readyz on the API address hangs or fails and /startupz sits in postgres-wait; the Postgres and API addresses are not held (ip -br addr); bao status shows HA Mode: standby with no active node; Patroni shows a stopped member at the dead node’s address; OVN’s cluster/status shows Role: candidate, Leader: unknown.

Caution

Last resort, for a cluster left wedged by a failed install, not for a node that is still settling after a reboot. Switch the failed joiner off first, or reinstall it: recovering the first node while it is half up adds the dead voter again. Recover all three rafts even if one shows symptoms.

OpenBao. Write a single-voter peers.json naming only the surviving node, then unseal with the recovery key:

OBRAFT=/var/lib/containers/storage/volumes/stratum-openbao-data/_data/raft
sudo systemctl stop openbao.service
sudo tee "$OBRAFT/peers.json" >/dev/null <<'JSON'
[{"id":"n-4k2x9m1q","address":"192.0.2.11:8201","non_voter":false}]
JSON
sudo systemctl start openbao.service
sudo stratum-bootstrap recover openbao-key | sudo stratum-bootstrap recover openbao-unseal --no-reseal
sudo podman exec openbao bao status | grep 'HA Mode'        # active

The node id is the node_id in /etc/stratum/openbao/*.hcl.

Postgres (Patroni). Stop it completely, delete its raft files, start it. The database itself is in another volume and is untouched.

PRAFT=/var/lib/containers/storage/volumes/stratum-patroni-data/_data/raft
sudo systemctl stop postgres-ha.service
sudo podman rm -f postgres-ha
sudo find "$PRAFT" -type f \( -name '*.dump' -o -name '*.journal' -o -name '*.journal.meta' \) -delete
sudo systemctl start postgres-ha.service
curl -s http://127.0.0.1:8008/patroni | jq -r '.role,.state'    # primary running

OVN. Delete the clustered database files, so a fresh one-member raft starts:

OVN=/var/lib/containers/storage/volumes/stratum-ovn-data/_data
sudo systemctl stop ovn-central.service
sudo podman rm -f ovn-central
sudo rm -f "$OVN"/ovnnb_db.db "$OVN"/ovnsb_db.db "$OVN"/.ovn{nb,sb}_db.db.~lock~
sudo systemctl start ovn-central.service
sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound | grep -E 'Role|Leader'

The control plane was probably stopped by its startup watchdog. Start it again once the Postgres address is back:

sudo systemctl reset-failed stratum-controlplane.service
sudo systemctl start --no-block stratum-controlplane.service

Then join the failed node again, from a new join code.

OpenBao is sealed

Nodes unseal themselves at boot from the TPM. A sealed member of a healthy cluster is a red flag (GET /v1/ha/status, openbao). Check systemctl status stratum-openbao-unseal. When the TPM cannot give the key back, or the node has no TPM and was restarted:

sudo stratum-bootstrap recover openbao-key | sudo stratum-bootstrap recover openbao-unseal

recover openbao-key prints the recovery key from the node, and recover openbao-unseal unseals and seals the key to the node’s TPM again. Keep a copy of the recovery key outside the cluster.

Nobody can sign in

sudo stratum-bootstrap recover admin-password --yes     # a new one-time password, printed once
sudo stratum-bootstrap recover admin-mfa                # clear a lost second factor

Both work from any control-plane node, using that node’s own key. The next sign-in must choose a new password. Prevent this: give a second person platform-admin, and readiness stops warning with SingleInstallAdmin.