Remove or revoke a node

Removing a node means moving its work off, taking its disks out of Ceph, switching it off, decommissioning it and taking it out of the rafts. Decommissioning is the part that matters for security: it makes the other nodes refuse the node at the transport, even though its certificate stays valid.

Caution

A cluster of three that loses a node is a cluster of two: it runs, but losing either survivor stops writes. Ceph with three copies of each object runs degraded on two nodes. Replace the node with a new one, added before or right after.

Planned removal

1. Check the cluster is healthy: stratumctl admin readiness.

2. Move its instances. The plan moves nothing and says which instances would be stopped:

   stratumctl nodes evacuation-plan n-4k2x9m1q
   stratumctl nodes drain n-4k2x9m1q

   Instances on Ceph move live. An instance whose root volume is on the node’s local pool cannot move: --allow-disruption stops it.

3. Take it out of Ceph, when it has OSDs. Drain the host, then remove it once the OSDs are gone:

   curl -X POST "$API/v1/admin/storage/ceph/hosts/node-3/drain" \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"zapOsds": true}'
   curl -s "$API/v1/admin/storage/ceph/osd-removals" -H "Authorization: Bearer $TOKEN"
   curl -X DELETE "$API/v1/admin/storage/ceph/hosts/node-3" -H "Authorization: Bearer $TOKEN"

   Ceph needs room to keep three copies on the nodes that stay. On a cluster with three OSDs a drain cannot finish cleanly: add capacity first.

4. Switch it off.

   stratumctl nodes shutdown n-4k2x9m1q --reason "retiring node-3"

5. Decommission it.

   stratumctl admin cluster members decommission n-4k2x9m1q --reason "retired"
   stratumctl admin cluster members list

   --reason is required and recorded (members decommission). The member becomes decommissioned: the other nodes stop accepting it at the node transport (cold migration, console and session relay) within about 30 seconds, and it leaves the peer lists. The row stays, as the audit trail. Decommissioning the node you are talking to is allowed, and readmit from any node undoes it.

6. Take it out of the rafts. Decommissioning does not remove the node from the Postgres, OpenBao or OVN clusters, and those services do not consult membership: they trust the same certificate. The cluster removes a node from them by itself only after a failed join. See Take a node out of the rafts.

7. Wipe it, or keep it switched off, before it leaves your hands. Revoking is not enough to cut it off from those three services.

Take a node out of the rafts

On a node that stays, with the removed node’s id and management address (stratumctl admin cluster members list, stratumctl nodes get). Check that each raft still lists it before removing it.

# OpenBao: remove the peer by node id (a token with the right to do it: STRATUM_OPENBAO_TOKEN in /etc/stratum/secrets/controlplane.env)
sudo podman exec --env VAULT_ADDR=http://127.0.0.1:8200 --env VAULT_TOKEN="$BAO_TOKEN" \
  openbao bao operator raft remove-peer n-4k2x9m1q

# OVN northbound and southbound: list the servers, then kick the removed one by address
sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/kick OVN_Northbound tcp:192.0.2.13:6643
sudo ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound tcp:192.0.2.13:6644

Caution

cluster/kick of an address the raft does not list makes the server you ask leave its own cluster. Run cluster/status first and kick only an address it names as another member.

Patroni’s own raft also lists the node: remove its address (port 2222) from that raft as well. There is no documented operator command for that step yet, and none of this has been run end to end on a real cluster.

When the node is lost or compromised

Decommission it at once, before anything else: the point is to revoke it.

stratumctl admin cluster members decommission n-4k2x9m1q --reason "disk stolen"

It is refused only when it would leave fewer than two active nodes on a multi-node cluster, because Postgres, OpenBao and OVN need a majority. Instances on Ceph restart on other nodes by themselves; see when a node dies. Instances on a local pool stay down until the node returns. Then take it out of the rafts, as above. Until you do, the dead voter counts toward their majority.

Peers refuse the revoked node. If one of them still receives attempts from it, the cluster CA finding names which node is trying: GET /v1/admin/pki/cluster-ca, the findings field. It clears 15 minutes after the last attempt.

A node decommissioned during a database outage keeps its access until the membership view refreshes.

Bring a node back

stratumctl admin cluster members readmit n-4k2x9m1q

Readmitting is explicit: neither a restart nor a reinstall undoes a decommission. It returns the member to active and restores its transport access within about 30 seconds. It does not add the node to any raft, because decommissioning never removed it from one: a node that was wiped, or taken out of the rafts by hand, does not get that membership back. Such a machine should join again with a new join code.