Bobay Stack

Virtualization / 2026-09-08

VM and Container Migration: The Bandwidth and Time Cost of a Migration

`qm migrate` and `pct migrate` look like one command, but the real cost depends on disk and network. Estimate first, then pick a time window.

4 MIN READ BY BOBAY.TOP Updated 2026-09-14 ↖ All notes
VM and Container Migration: The Bandwidth and Time Cost of a Migration

On a single host, "migration" means moving files; in a cluster, it means moving memory state. The two costs are completely different, and mixing them up will saturate your internal network when you migrate at midnight.

Two kinds of migration

TypeCommandWhat gets moved
Offline migration (shutdown)qm migrate 101 node2 --offlineDisk
Online migration (live)qm migrate 101 node2 --onlineDisk + memory state

The advantage of online migration is no service interruption; the cost is that the more memory you have, the slower it gets, and it keeps occupying the migration network the whole time.

Estimate the time first

Rough formula: disk size ÷ effective bandwidth × 1.3 (overhead factor).

  • 40G disk + gigabit network (measured 110MB/s) ≈ 6–8 minutes
  • 40G disk + 10-gigabit network ≈ 1 minute
  • Online migration of 8G of memory adds another 1–2 minutes for memory synchronization

Doing online migration over gigabit is painful: the memory sync phase may fail to converge repeatedly and eventually fall back to a brief downtime. If you really have to migrate in a gigabit environment, pick a nightly shutdown migration.

Storage type determines the path

  • Shared storage (NFS / Ceph): only memory is moved, finishes in seconds
  • Local storage (LVM / ZFS): the disk has to move too, use the formula above

This is also why "use shared storage uniformly across the cluster" is the right direction — it turns migration from minutes into seconds.

One thing before and one thing after

Before migrating: confirm the target node has enough memory and disk, or it will fail to start after the move.

qm config 101 | grep -E "memory|scsi"

After migrating: check that the bridge the NIC is bound to exists on the target node. A mismatched bridge name is the most common cause of "it migrated but has no network."

A real lesson

I once migrated a VM with 8G of memory and a 40G disk from one gigabit node to another gigabit node. The online migration ran for 26 minutes and still ended with 3 seconds of downtime to complete the switchover. Later I switched to a shutdown migration, and the same data took only 7 minutes.

Conclusion: online migration is not always the better option; it depends on bandwidth and memory size.

Where migration is actually slow

The bottleneck in qm migrate is almost always disk transfer, not memory. A 100G VM should theoretically take 15 minutes over gigabit, but in practice 40+ minutes is common, because:

  1. It goes over an SSH channel, with encryption overhead
  2. Small random writes, so it never reaches line rate
  3. The target-side storage is slow and back-pressures the transfer

Three migration methods

MethodCommandDowntimeWhen to use
Online migrationqm migrate 101 node2 --onlineSecondsSame CPU generation
Offline migrationShut down, then migrateThe whole timeCross-generation CPU, large disks
Backup and restorevzdump + qmrestoreThe whole timeAcross clusters, across data centers

Online migration requires compatible CPU features on source and target. Across generations (say EPYC to Intel) it often crashes at the moment of migration with cpu model mismatch.

Ways to speed it up

# 1. Use a faster channel: direct 10-gigabit link
qm migrate 101 node2 --online --with-local-disks --bwlimit 0

# 2. Compress large disks before transferring (offline case)
vzdump 101 --mode stop --compress zstd --dumpdir /mnt/fast

--bwlimit 0 removes the rate limit. By default PVE limits to around 100MB/s, and on a gigabit network that value actually slows you down.

An easy-to-miss point

Migration temporarily writes the VM config to the target node. If the target node's disk is full, the migration fails at the very last step — the data has been transferred but the config can't be written, and the VM is stuck in "migrating" state. Run df -h before you migrate.

What about across data centers?

Don't use qm migrate. Use vzdump to produce a backup, rsync to transfer it, and qmrestore to import it. The transfer can be resumed if interrupted, which is more reliable than the migration channel.

One-line summary: writing "where things go wrong" together with "how to verify" is more useful than writing only the configuration — configuration can be copied, but the order of judgment cannot.

MigrationClusterBandwidthPVE

References

PVE migration documentation ↗