Backup and restore

Two things can be backed up:

Scope	Holds	Restores
Platform state	The control plane’s database, the secrets store and its unseal key, the cluster’s CA, and the node credentials, in one encrypted bundle	Every resource record, user, policy and secret, onto a fresh install
Volume data	The contents of individual volumes	One volume

Platform state

On a cluster

The backup works the same way. For the directory target on several nodes and for which restore a given loss needs, see Back up and restore the cluster.

The platform writes a bundle on a schedule, encrypts it with an age recipient, and ships it to a target off the node. Keep two things outside the cloud, or a restore is impossible:

• the age identity (private key) that matches the recipient;
• the target’s location and credentials.

Configure

Console: Admin › Substrate › Backup & DR. Or the API, as a platform admin:

age-keygen -o backup-identity.txt          # keep this file off the node
curl -X PUT "$API/v1/backup/admin/config" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{
    "enabled": true,
    "cadenceSeconds": 21600,
    "retentionCount": 14,
    "ageRecipient": "age1...",
    "targetKind": "s3",
    "targetPath": "",
    "targetOptions": {
      "s3Endpoint": "https://s3.example.com",
      "s3Bucket": "stratum-backups",
      "s3Prefix": "node-1",
      "s3Region": "auto",
      "s3AccessKey": "...",
      "s3SecretKey": "..."
    }
  }'
curl -X POST "$API/v1/backup/admin/run" -H "Authorization: Bearer $TOKEN"   # one now

Target	Notes
s3	Any S3-compatible store. The bucket must exist; scope the key to that bucket.
directory	A path under /var/lib/stratum, usually /var/lib/stratum/backups. Only useful off the node: mount an NFS export there before the control plane starts, and set targetOptions.requireMountpoint: true so an unmounted path is refused rather than filled.

The newest retentionCount good bundles are kept. Two alarms watch it: backup-rpo-exceeded (no good bundle within twice the cadence) and backup-attempt-failing. stratumctl admin selfcheck reports backup-freshness.

Caution

Without a platform backup, the cluster’s CA exists in one place. Losing it means no new certificate can be issued and the cloud must be reinstalled. Configure a backup on every install that matters.

Restore

Note

Restore onto a fresh install of the same hostname and configuration, from a release at least as new as the one that wrote the bundle.

1. Install the node again from the ISO, with the same node.yaml.

2. Copy the bundle and the age identity onto the node, then:

   sudo stratum-bootstrap restore \
     --bundle stratum-backup-<stamp>.tar.age \
     --identity backup-identity.txt \
     --yes

3. Sign in with the original admin password; the fresh install’s no longer works.

4. Run stratumctl admin selfcheck.

Resource records come back as they were. Volume contents are not in the bundle: instances come back stopped, and volumes whose data was lost come back empty or in error.

Volume data

Projects protect volumes with backup plans: a schedule, a retention count, and the volumes to include (by ID or by tag), written to a vault.

stratumctl backup vault create --name main --project p-web
stratumctl backup plan create --name nightly --vault bv-abc123 --schedule "0 3 * * *" \
  --retention 7 --volume vol-abc123 --project p-web
stratumctl backup recovery-point list --project p-web
stratumctl backup restore rp-abc123 --mode newVolume --project p-web

Caution

Volume backups need a storage driver that can export a volume off the node. Today that is Ceph RBD, on a cluster. Volumes on LVM-thin pools, which is every volume on a single node, are refused. Back up the data from inside the instance until then.

Before an upgrade

Every upgrade takes its own backup first, to the backup target or, without one, on the node. A rollback with --restore uses it; see Rollback.