Rollback and restore

A rollback is an upgrade to the release the cloud ran before its latest upgrade, done or failed. Containers go back to that release’s images, the host binaries switch back, and after an OS upgrade the node reboots into its previous OS.

stratumctl admin upgrade rollback --dry-run     # the way back; changes nothing
stratumctl admin upgrade rollback

Rollback of upg-4k2j9x0qpa: 0.3.0 back to 0.2.0 (containers)
Steps:       restart
Impact:      VMs keep running. The API is briefly unavailable while the control plane restarts.
Go back to 0.2.0? [y/N] y
Rollback upg-8a2n4fq0zt recorded: back to 0.2.0.

The previous release must still be on the node (an upgrade keeps it), and its control plane must be able to run on the database as it is now. Before the first upgrade, the previous release is the one the node was installed with.

Rollback with restore

Single node only

--restore restores the backup in place on one node. On a cluster, a database the previous release cannot read means a restore from the platform backup.

When the upgrade changed the database in a way the previous release cannot read, rollback refuses and asks for --restore. That first restores the upgrade’s own pre-upgrade backup, then starts the previous release on it:

stratumctl admin upgrade rollback upg-4k2j9x0qpa --restore

• Everything changed after the backup is lost, sessions included. The upgrade refused changes through the API from the backup on, so nothing accepted through the API is lost.
• The command shows the backup and asks to type the upgrade’s ID, or takes --confirm <id>.
• With --wait at a terminal, it asks for the password again once the restore has signed the session out, and follows the rollback.

A backup encrypted to the backup recipient needs the matching age identity on the node first, root only:

sudo install -m 0600 /dev/stdin /var/lib/stratum/upgrade/restore-identity.txt < backup-identity.txt

The restore removes it when done.

After a failed upgrade

A failed OS upgrade whose new OS does not come up falls back to the previous OS by itself. Read why with journalctl -u greenboot-healthcheck -b -1 on the node, then acknowledge the upgrade:

stratumctl admin upgrade acknowledge upg-4k2j9x0qpa