Ops & troubleshooting / 2026-11-02
Before you upgrade, prove you can roll back
A backup is not a restore. Take a backup, run a real restore drill, then set a change window — only then is an upgrade reversible.
I took a backup before every upgrade, but twice, when something actually went wrong, the backup was useless: once the backup file sat on the same disk, and when the disk died both went with it; once the backup was missing the configs under /etc, and what came back was a system that would boot but couldn't reach the network.
A backup has to meet three conditions
- Off-site: at least one copy that isn't on the same physical machine or the same disk.
- Verifiable: there's a checksum, or you can list the contents — not "the file size looks right".
- Includes configuration: not just data volumes, but the service configs you changed under
/etc.
On PVE, the least-effort starting point is vzdump to remote storage:
vzdump <vmid> --mode snapshot --storage <remote storage> --compress zstd
Container and VM snapshots are "roll back to a minute ago", not "a backup" — they live with the original disk, so they guard against mistakes, not against hardware.
For file-level backups I use restic: the first run is full, everything after is incremental, so it's cheap:
restic -r sftp:<remote>:<path> init
restic -r sftp:<remote>:<path> backup /etc /root /opt --exclude-caches
restic -r sftp:<remote>:<path> snapshots
The step that actually matters: a restore drill
A backup script going green doesn't mean you can restore. I do a real restore on an idle machine, recovering only the three things that matter most:
- A minimal container snapshot, to confirm it boots.
- The network and SSH configs under
/etc, to confirm I can get in remotely after the restore. - A service data directory, to confirm file permissions and ownership are correct.
This step regularly exposes the "it's in the backup but the restore script doesn't handle it" problems — for example, after restic restore the owner is root instead of the original service user.
Change window
The upgrade itself isn't the risk; an upgrade stacked on another change is. When I set a window, I do three things:
- Change one layer at a time. Kernel upgrade, panel upgrade, network config change — don't put them on the same night.
- Record the pre-upgrade version numbers and a diff of the key configs; when something breaks, that's the only baseline you have to compare against.
- Leave an hour of observation time with nothing else scheduled.
dmesgandsystemctl --failedare the two things in the logs you must check.
The order I follow now
vzdumpto remote +restic backup /etc- Pull one backup and run a restore drill (monthly, not on every upgrade)
- Tag the current version
- Upgrade, checking only
systemctl --failedand the key service ports - Watch for an hour; only then move on to the next machine
The sequence isn't complicated, but it turns "upgrading" from a gamble into something you can back out of. For a self-hosted environment, being able to go back matters more than never having an incident.
Backup is not copying
Most people's "backup" is cp -r or a tar. A backup you can actually restore has to meet three conditions: verifiable, off-site, and rollable back to a specific point in time.
Three layers of backup
| Layer | Frequency | Medium | Purpose |
|---|---|---|---|
| Snapshot | Before a change | Local LVM/ZFS | Minute-level rollback |
| Full | Daily | Another disk | Recovery from accidental deletion |
| Off-site | Weekly | Another machine / object storage | Hardware failure |
The three layers have recovery times of minutes, hours, and more than hours respectively. Only the first layer saves you from the "I just broke it" scenario.
The right vzdump parameters
vzdump 101 --mode snapshot --compress zstd --storage backup-nfs --prune-backups keep-daily=7,keep-weekly=4
--mode snapshot doesn't stop the guest; keep-daily=7 keeps 7 daily backups and keep-weekly=4 keeps 4 weekly ones. Leave out the prune parameters and the disk fills up within two months.
The restore drill you have to do
qmrestore /mnt/backup/vzdump-qemu-101-2026_10_18.vma.zst 999 --storage local-lvm
qm start 999
Restore to a temporary ID, boot it, verify the services, then delete it. A backup you've never drilled is just "a file you have".
A common misjudgement
A backup file with a normal size doesn't mean the contents are complete. When a zstd archive is corrupted, vzdump still exits successfully, but qmrestore reports unexpected EOF. So I run a zstd -t check after every backup; it costs a few seconds.