Skip to content
StratumStratum Docs

Shut down, reboot and cold start

Planned shutdowns are commands, not manual steps: Ceph’s flags and the instances’ shutdown are handled for you.

You want to Do
Reboot or power off one node stratumctl nodes reboot <node-id>, or stratumctl nodes shutdown <node-id>
Keep its instances running meanwhile stratumctl nodes drain <node-id> first
Power off the whole cluster stratumctl admin cluster shutdown
Stop everything, power off yourself stratumctl admin cluster shutdown --prepare-only
See where it is stratumctl admin cluster shutdown status
Undo it before the nodes power off stratumctl admin cluster shutdown cancel

One node

Section titled “One node”
Terminal window
stratumctl nodes reboot n-4k2x9m1q --reason "kernel update"
stratumctl nodes shutdown n-4k2x9m1q --reason "replace a fan"

See nodes reboot and nodes shutdown. Console: Admin › Metal, the node, Shut down or Reboot. The cluster holds the node in maintenance, so nothing is placed on it and no failover starts. About 15 seconds later the node powers off or reboots itself. On the way down it:

  1. sets Ceph’s noout on its own OSDs only, so Ceph does not rebuild their data while the node is away;
  2. asks every instance to shut down (ACPI), asks again every 20 seconds, and powers off the ones still running after 90 seconds;
  3. then stops the platform’s services.

On the way back the instances it stopped start again (unless their owner set them to stopped), and noout is cleared once the node’s OSDs are up and in. If an OSD is not back within 20 minutes the flag is cleared anyway.

A shut-down node stays off until you power it on. Its power button, or Wake-on-LAN to its management NIC where the firmware allows it.

A plain systemctl reboot on the node does the same drain, without the checks below.

Refused when

Section titled “Refused when”
Code Because
QUORUM_AT_RISK The cluster would be left without a majority of control-plane nodes: a second node while one is down. --force overrides this one.
UPGRADE_IN_PROGRESS, CLUSTER_SHUTDOWN_IN_PROGRESS Another operation is running.
NODE_POWER_ACTION_IN_PROGRESS The node already has one.
MIGRATION_IN_PROGRESS An instance is moving to or from it.
NODE_NOT_REPORTING Its heartbeat lapsed: it is probably already off.

Reboot one node at a time, and wait for stratumctl admin readiness to say READY before the next. Two nodes down lose the Postgres, OpenBao, OVN and Ceph monitor majorities. Readiness shows NodePowerAction while a node is down on purpose, and a warning if a reboot has not come back after 20 minutes.

The whole cluster

Section titled “The whole cluster”
Terminal window
stratumctl admin cluster shutdown --reason "power work in the rack"

Confirm by typing the cluster id, or pass --confirm <cluster-id> (admin cluster shutdown). The cluster:

  1. sets the Ceph flags noout, norebalance, nobackfill and norecover (flags already set are left alone, and not cleared afterwards);
  2. stops every running instance, each with 60 seconds to shut down, and keeps its desired state;
  3. powers every node off, the one the command runs on last.

With --prepare-only it stops after step 2: power the nodes off in any order, or cancel. status follows it, and readiness shows ClusterShutdown. Console: Admin › Substrate › Cluster › Shutdown.

Cancel works until the nodes start powering off: it clears the flags and starts the instances. After that, power the nodes on.

Cold start

Section titled “Cold start”

Power the nodes on, in any order. Nothing else is needed.

  1. Each node’s services start by themselves. OpenBao unseals from the node’s TPM, and Postgres, OpenBao and OVN re-form their clusters and elect leaders.
  2. The cluster waits for every node that was up at shutdown to report, and for every OSD to be up and in.
  3. It clears exactly the Ceph flags the shutdown set, and starts the instances that were running.

A node or OSD still missing after 30 minutes does not hold the flags: they are cleared anyway, and the message names what is missing.

A control plane that has just started judges no node dead until it has watched heartbeats itself for about 2 minutes, so a node that is merely still booting is not failed over. A cluster powered off without a shutdown comes back the same way.

To watch a node start: curl -s http://192.0.2.11:8443/startupz names the phase it is in, and how long it has been there.