Bobay Stack

Ops & troubleshooting / 2026-11-02

Before you upgrade, prove you can roll back

A backup is not a restore. Take a backup, run a real restore drill, then set a change window — only then is an upgrade reversible.

4 MIN READ BY BOBAY.TOP Updated 2026-11-08 ↖ All notes
Before you upgrade, prove you can roll back

I took a backup before every upgrade, but twice, when something actually went wrong, the backup was useless: once the backup file sat on the same disk, and when the disk died both went with it; once the backup was missing the configs under /etc, and what came back was a system that would boot but couldn't reach the network.

A backup has to meet three conditions

  • Off-site: at least one copy that isn't on the same physical machine or the same disk.
  • Verifiable: there's a checksum, or you can list the contents — not "the file size looks right".
  • Includes configuration: not just data volumes, but the service configs you changed under /etc.

On PVE, the least-effort starting point is vzdump to remote storage:

vzdump <vmid> --mode snapshot --storage <remote storage> --compress zstd

Container and VM snapshots are "roll back to a minute ago", not "a backup" — they live with the original disk, so they guard against mistakes, not against hardware.

For file-level backups I use restic: the first run is full, everything after is incremental, so it's cheap:

restic -r sftp:<remote>:<path> init
restic -r sftp:<remote>:<path> backup /etc /root /opt --exclude-caches
restic -r sftp:<remote>:<path> snapshots

The step that actually matters: a restore drill

A backup script going green doesn't mean you can restore. I do a real restore on an idle machine, recovering only the three things that matter most:

  1. A minimal container snapshot, to confirm it boots.
  2. The network and SSH configs under /etc, to confirm I can get in remotely after the restore.
  3. A service data directory, to confirm file permissions and ownership are correct.

This step regularly exposes the "it's in the backup but the restore script doesn't handle it" problems — for example, after restic restore the owner is root instead of the original service user.

Change window

The upgrade itself isn't the risk; an upgrade stacked on another change is. When I set a window, I do three things:

  • Change one layer at a time. Kernel upgrade, panel upgrade, network config change — don't put them on the same night.
  • Record the pre-upgrade version numbers and a diff of the key configs; when something breaks, that's the only baseline you have to compare against.
  • Leave an hour of observation time with nothing else scheduled. dmesg and systemctl --failed are the two things in the logs you must check.

The order I follow now

  1. vzdump to remote + restic backup /etc
  2. Pull one backup and run a restore drill (monthly, not on every upgrade)
  3. Tag the current version
  4. Upgrade, checking only systemctl --failed and the key service ports
  5. Watch for an hour; only then move on to the next machine

The sequence isn't complicated, but it turns "upgrading" from a gamble into something you can back out of. For a self-hosted environment, being able to go back matters more than never having an incident.

Backup is not copying

Most people's "backup" is cp -r or a tar. A backup you can actually restore has to meet three conditions: verifiable, off-site, and rollable back to a specific point in time.

Three layers of backup

LayerFrequencyMediumPurpose
SnapshotBefore a changeLocal LVM/ZFSMinute-level rollback
FullDailyAnother diskRecovery from accidental deletion
Off-siteWeeklyAnother machine / object storageHardware failure

The three layers have recovery times of minutes, hours, and more than hours respectively. Only the first layer saves you from the "I just broke it" scenario.

The right vzdump parameters

vzdump 101 --mode snapshot --compress zstd --storage backup-nfs --prune-backups keep-daily=7,keep-weekly=4

--mode snapshot doesn't stop the guest; keep-daily=7 keeps 7 daily backups and keep-weekly=4 keeps 4 weekly ones. Leave out the prune parameters and the disk fills up within two months.

The restore drill you have to do

qmrestore /mnt/backup/vzdump-qemu-101-2026_10_18.vma.zst 999 --storage local-lvm
qm start 999

Restore to a temporary ID, boot it, verify the services, then delete it. A backup you've never drilled is just "a file you have".

A common misjudgement

A backup file with a normal size doesn't mean the contents are complete. When a zstd archive is corrupted, vzdump still exits successfully, but qmrestore reports unexpected EOF. So I run a zstd -t check after every backup; it costs a few seconds.

backupsresticLXCchange management

References

Proxmox backup and restore ↗ restic documentation ↗