Upgrade a cluster

Not yet run on a real cluster

An upgrade is proven end to end on a single node. The rolling upgrade of several nodes, with its health checks between them, is covered by unit tests only: no real cluster has run it. Take the first one as untested: have a backup, upgrade at a quiet time, and watch it.

Planning, starting, following and rolling back are the same commands as on one node: see Upgrade, Rollback, Offline upgrade and the upgrade paths, which the plan checks. This page covers what a cluster changes.

See admin upgrade for every command.

stratumctl admin upgrade plan 0.7.0          # nothing changes
stratumctl admin upgrade start 0.7.0 --wait

What the plan shows

The plan lists every node, with the kind of upgrade the release’s changes set for it. The kind is never a choice.

Kind	The release changes	What happens	Instances
containers	Container images only	The changed containers restart.	Keep running
host	The host binaries	The containers restart, then the binaries switch.	Keep running
os	The base OS or kernel	The node reboots.	Shut down, then start again

It also says where the backup goes, how the upgrade could be undone, and every precheck. The upgrade is refused while one fails. For a cluster the prechecks include: the cluster is READY, every node is ready and has reported what it runs in the last 10 minutes, no cluster shutdown or other upgrade runs, and the release is a permitted step from the release you run.

How it rolls

1. Every node fetches the release, and nothing that runs changes.

2. A backup is taken: to the backup target, or on the node when none is set. From here until the end, changes through the API are refused (503 UPGRADE_IN_PROGRESS); reads and signing in still work.

3. The nodes upgrade one at a time. The node the control-plane leader runs on goes last, so leadership moves once.

4. After each node, the next waits for at least a 2-minute settle time and all of:

   • the node reports the new release;
   • the cluster is READY and settled, with the OVN databases serving and one Postgres primary;
   • every Patroni member runs, one leads, and no replica lags by more than 16 MB;
   • Ceph is no worse than before that node began: health, OSDs up and in, placement groups active and clean;
   • OpenBao is initialized, unsealed and has a leader.

   The upgrade’s message says what it waits for.

5. A gate that does not pass within 30 minutes ends the upgrade failed, naming the node, the reason and the nodes not started. The nodes already upgraded stay on the new release. Fix the cause and start the upgrade again, or roll back.

An os upgrade reboots the node: its instances are shut down by the node’s drain on the way down, and Ceph’s noout is set on that node’s OSDs so it does not rebuild their data. To keep instances running, drain the node first. The Ceph daemons themselves are not upgraded: an upgrade moves only the host’s Ceph packages.

While an upgrade runs, readiness shows UpgradeInProgress, new joins are refused (try again later), and node reboots and shutdowns are refused.

Keep the release in place

A release given to the cluster as a file (an uploaded bundle, an attached medium or an import directory) must stay where it is until the upgrade has finished, and until any machine upgrading itself from an older ISO has joined. stratumctl admin upgrade prune removes what the cluster no longer needs, never while an upgrade runs.

If it goes wrong

• A node whose new OS does not come up falls back to the old OS by itself after three failed boots. The upgrade ends failed, the node rolled_back, and readiness shows NodeRolledBack. Acknowledge it with stratumctl admin upgrade acknowledge.
• stratumctl admin upgrade rollback returns to the previous release, when its control plane can read the database as it is. --restore, which restores the pre-upgrade backup in place, is for a single node only. On a cluster, a database that has moved on means a restore from the backup.

OpenBao policies after an upgrade

If readiness shows OpenBaoPoliciesStale after an upgrade (a warning; every token keeps working), each node’s OpenBao must restart once to take the release’s policies. Do it one node at a time, and wait for the node to unseal and the cluster to show an OpenBao leader before the next:

sudo systemctl restart openbao.service        # it unseals again by itself
sudo stratum-bootstrap openbao-tokens --apply

openbao-tokens without --apply says what the node waits for.