Virtualization / 2026-07-15
GPU Passthrough in LXC: Three Approaches and How They Actually Differ
Containers aren't VMs, so GPU passthrough works differently: device nodes, driver mounts, and when to drop LXC for a VM.
Containers share the host kernel, so "passthrough" in LXC isn't handing a PCI device to a virtual machine — it's exposing device nodes and drivers to the container. Once that clicks, every pitfall below becomes easy to explain.
Approach 1: Device node passthrough
The most direct method is to add device nodes and mount entries to the container config:
# Confirm the device node on the host
ls -l /dev/nvidia*
# Append to /etc/pve/lxc/<vmid>.conf
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
Key points:
- The major device number must match the actual value on your machine (NVIDIA is usually 195, AMD's
renderD*is 226). - The userspace driver libraries (
libnvidia-*.so) do not automatically enter the container. Either mount the host's library directory or install the same driver version inside the container. The versions must match exactly, otherwisenvidia-smiimmediately reports "Failed to initialize NVML". - The fact that
nvidia-smiruns inside the container doesn't mean CUDA will; you also need/dev/nvidia-uvm.
Approach 2: The nvidia-container-toolkit idea
This path suits Docker containers better and helps LXC only marginally. LXC has no runtime hook like Docker's, so you have to mount devices and libraries in by hand. Hence the common pattern: install the driver on the host, mount devices + mount libraries in the LXC, and install no driver inside the container.
Approach 3: Just use a VM
If the goal is to give one card to model inference and you need a complete driver stack, CUDA, or even driver reinstalls, a VM actually costs you less. Once the PCI device is passed through to QEMU, the guest is an independent machine — if you break the driver, you can just roll back a snapshot, with no impact on the host.
The criterion is simple:
| Requirement | Recommendation |
|---|---|
Just nvidia-smi monitoring or light encoding | LXC device passthrough is enough |
| Full CUDA / inference service, frequent driver upgrades | VM PCI passthrough |
| Multiple containers sharing one card | LXC mounts + quotas, watch for VRAM contention |
| Need vGPU partitioning | Depends on the card; consumer cards generally can't |
Two common failures
- Container fails to start: wrong major device number in
lxc.cgroup2.devices.allow, or a mount entry path that doesn't exist. Checkjournalctl -u pve-container@<vmid>. - Container breaks after a host driver update: library and driver versions no longer match. The container's libraries are mounted in, so when the host upgrades the driver, the container still has the old ones — restart the container or remount.
One last thing: passing a GPU to a container means the card can no longer be allocated on demand to other containers on the host. In multi-tenant setups, do proper quotas — otherwise one container that fills up VRAM will OOM everyone else's jobs.
The three passthrough paths
There are three ways to run a GPU with LXC, and they differ completely in cost and compatibility:
| Method | Prerequisite | Isolation | Fits |
|---|---|---|---|
Device passthrough /dev/nvidia* | Driver installed on host | Weak (shared kernel) | Single-container inference |
nvidia-container-toolkit + CDI | Docker/OCI ecosystem | Medium | Containerized apps |
| VM + PCIe passthrough | Separate IOMMU group | Strong | Need an independent driver version |
LXC passthrough is "mapping device nodes into the container", not virtualization, so don't install the driver inside the container — it will fight with the host's version.
Minimal config
# /etc/pve/lxc/101.conf
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
195 is the nvidia major device number, 511 is nvidia-uvm. The typical symptom of forgetting to map nvidia-uvm: nvidia-smi runs fine, but CUDA initialization crashes outright.
Verification
# Inside the container
ls -l /dev/nvidia*
nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"
If nvidia-smi works but torch returns False, it's almost certainly a mismatch between the driver version and the CUDA runtime libraries inside the container — not a passthrough failure.
A security reminder
Passing through a GPU means handing that device to the container. Processes inside the container can read GPU memory and change clocks. Don't use LXC passthrough in multi-tenant setups — use VM passthrough.
To sum up in one line: writing "where things break" next to "how to verify" is more useful than writing the config alone — configs can be copied, the order of diagnosis cannot.