Bobay Stack

Virtualization / 2026-07-15

GPU Passthrough in LXC: Three Approaches and How They Actually Differ

Containers aren't VMs, so GPU passthrough works differently: device nodes, driver mounts, and when to drop LXC for a VM.

4 MIN READ BY BOBAY.TOP Updated 2026-07-20 ↖ All notes
GPU Passthrough in LXC: Three Approaches and How They Actually Differ

Containers share the host kernel, so "passthrough" in LXC isn't handing a PCI device to a virtual machine — it's exposing device nodes and drivers to the container. Once that clicks, every pitfall below becomes easy to explain.

Approach 1: Device node passthrough

The most direct method is to add device nodes and mount entries to the container config:

# Confirm the device node on the host
ls -l /dev/nvidia*

# Append to /etc/pve/lxc/<vmid>.conf
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file

Key points:

  • The major device number must match the actual value on your machine (NVIDIA is usually 195, AMD's renderD* is 226).
  • The userspace driver libraries (libnvidia-*.so) do not automatically enter the container. Either mount the host's library directory or install the same driver version inside the container. The versions must match exactly, otherwise nvidia-smi immediately reports "Failed to initialize NVML".
  • The fact that nvidia-smi runs inside the container doesn't mean CUDA will; you also need /dev/nvidia-uvm.

Approach 2: The nvidia-container-toolkit idea

This path suits Docker containers better and helps LXC only marginally. LXC has no runtime hook like Docker's, so you have to mount devices and libraries in by hand. Hence the common pattern: install the driver on the host, mount devices + mount libraries in the LXC, and install no driver inside the container.

Approach 3: Just use a VM

If the goal is to give one card to model inference and you need a complete driver stack, CUDA, or even driver reinstalls, a VM actually costs you less. Once the PCI device is passed through to QEMU, the guest is an independent machine — if you break the driver, you can just roll back a snapshot, with no impact on the host.

The criterion is simple:

RequirementRecommendation
Just nvidia-smi monitoring or light encodingLXC device passthrough is enough
Full CUDA / inference service, frequent driver upgradesVM PCI passthrough
Multiple containers sharing one cardLXC mounts + quotas, watch for VRAM contention
Need vGPU partitioningDepends on the card; consumer cards generally can't

Two common failures

  • Container fails to start: wrong major device number in lxc.cgroup2.devices.allow, or a mount entry path that doesn't exist. Check journalctl -u pve-container@<vmid>.
  • Container breaks after a host driver update: library and driver versions no longer match. The container's libraries are mounted in, so when the host upgrades the driver, the container still has the old ones — restart the container or remount.

One last thing: passing a GPU to a container means the card can no longer be allocated on demand to other containers on the host. In multi-tenant setups, do proper quotas — otherwise one container that fills up VRAM will OOM everyone else's jobs.

The three passthrough paths

There are three ways to run a GPU with LXC, and they differ completely in cost and compatibility:

MethodPrerequisiteIsolationFits
Device passthrough /dev/nvidia*Driver installed on hostWeak (shared kernel)Single-container inference
nvidia-container-toolkit + CDIDocker/OCI ecosystemMediumContainerized apps
VM + PCIe passthroughSeparate IOMMU groupStrongNeed an independent driver version

LXC passthrough is "mapping device nodes into the container", not virtualization, so don't install the driver inside the container — it will fight with the host's version.

Minimal config

# /etc/pve/lxc/101.conf
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file

195 is the nvidia major device number, 511 is nvidia-uvm. The typical symptom of forgetting to map nvidia-uvm: nvidia-smi runs fine, but CUDA initialization crashes outright.

Verification

# Inside the container
ls -l /dev/nvidia*
nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"

If nvidia-smi works but torch returns False, it's almost certainly a mismatch between the driver version and the CUDA runtime libraries inside the container — not a passthrough failure.

A security reminder

Passing through a GPU means handing that device to the container. Processes inside the container can read GPU memory and change clocks. Don't use LXC passthrough in multi-tenant setups — use VM passthrough.

To sum up in one line: writing "where things break" next to "how to verify" is more useful than writing the config alone — configs can be copied, the order of diagnosis cannot.

LXCGPUpassthroughPVE

References

PVE LXC Documentation ↗ NVIDIA Container Toolkit ↗