Optional Components#

32 bit (i686) packages for Linux x86_64#

These packages provide 32-bit (i686) driver libraries needed for things such as Steam (popular game app store/launcher), older video games, and some compute applications.

Debian 12#

# dpkg --add-architecture i386
# apt update
# apt install nvidia-driver-libs:i386

By adding the architecture before installing the driver, the i386 packages are installed automatically.

Ubuntu 22.04/24.04#

# dpkg --add-architecture i386
# apt update
# apt install \
   libnvidia-compute-<branch>:i386 libnvidia-decode-<branch>:i386 \
   libnvidia-encode-<branch>:i386 libnvidia-extra-<branch>:i386 \
   libnvidia-fbc1-<branch>:i386 libnvidia-gl-<branch>:i386

Where <branch> is the driver version, for example 590.

By adding the architecture before installing the driver, the i386 packages are installed automatically.

Red Hat Enterprise Linux 8/9, Rocky Linux 8/9, Oracle Linux 8/9, Fedora 42#

# dnf install nvidia-driver-cuda-libs.i686 nvidia-driver-libs.i686 libnvidia-ml.i686 libnvidia-fbc.i686

SUSE Enterprise Linux Server 15, OpenSUSE Leap 15#

# zypper install nvidia-compute-G07-32bit nvidia-gl-G07-32bit nvidia-video-G07-32bit

GPUDirect Storage#

Install NVIDIA Filesystem.

Red Hat Enterprise Linux 8/9

# dnf install nvidia-fs

Note

The GPUDirect storage module is shipped only in DKMS format; this means it requires the module source to be available. Starting with version 570 of the drivers, a mix of Precompiled streams and DKMS modules is no longer supported.

Having a mix of precompiled and DKMS modules makes every single benefit of the precompiled modules disappear completely:

Development packages and headers required to compile modules are still required.
Secure Boot can not be used without a custom MOK.
nvidia-peermem can not be recompiled to leverage the OFED installations as it’s missing from the precompiled packages.
It requires two extra packages containing the source of the precompiled modules to be installed along with the binary modules, making the installation more error-prone and complicated.

Until driver version 565, the source profile can be installed with this command to install the additional packages containing the source of the precompiled modules:

# dnf module install nvidia-driver:$stream/src

Ubuntu 22.04/24.04, Debian 12/13

# apt install nvidia-fs

Using `nvidia-peermem`#

The NVIDIA driver packages provide a kernel module, nvidia-peermem, which provides NVIDIA InfiniBand based HCAs (Host Channel Adapters) direct peer-to-peer read and write access to the NVIDIA GPU’s video memory. It allows GPUDirect RDMA-based applications to use GPU computing power with the RDMA interconnect without needing to copy data to host memory.

This capability is supported with NVIDIA ConnectX®-3 VPI or newer adapters. It works with both InfiniBand and RoCE (RDMA over Converged Ethernet) technologies.

NVIDIA OFED (Open Fabrics Enterprise Distribution), or MLNX_OFED, introduces an API between the InfiniBand Core and peer memory clients such as NVIDIA GPUs. The nvidia-peermem module registers the NVIDIA GPU with the InfiniBand subsystem by using peer-to-peer APIs provided by the NVIDIA GPU driver.

The kernel must have the required support for RDMA peer memory either through additional patches to the kernel or via MLNX_OFED as a prerequisite for loading and using nvidia-peermem. So unless the MLNX_OFED distribution is installed prior to installing the NVIDIA driver, the nvidia-peermem module will simply be built as a stub. Also whenever the OFED distribution changes, the module must be rebuilt to match the updated OFED kernel modules.

Automatic rebuild of `nvidia-peermem`#

DKMS has been extended to support automatic rebuilding of dependent kernel modules explicitly for this use case, so a specific configuration can be added to the system which requires nvidia-peermem with OFED support. It’s sufficient to add the following snippet to /etc/dkms/nvidia.conf prior or after installing the GPU driver or the OFED components:

BUILD_DEPENDS[4]="mlnx-ofed-kernel"
BUILD_DEPENDS_REBUILD="yes"

The first line refers to the nvidia-peermem module as defined in the /usr/src/nvidia-*/dkms.conf file; the number between square brackets must match with the module ID in that file. The second part is the DKMS module (so the DKMS “module” as a whole) on which it depends on.

The second line instructs DKMS to make sure the module ID declared in the first line changes if DKMS tracks a dependency change (update, rebuild for a new kernel, etc).

If the OFED distribution has been installed after the driver, a simple DKMS rebuild should suffice. An example with driver 590 on Fedora.

First, we remove the installed modules, so in case of a reinstall, these are not backed up:

# dkms status -m nvidia
nvidia/590.44.01, 6.17.11-300.fc42.x86_64, x86_64: installed
# dkms uninstall -m nvidia/590.44.01
Module nvidia/590.44.01 for kernel 6.17.11-300.fc42.x86_64 (x86_64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia.ko.xz
Deleting /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-modeset.ko.xz
Deleting /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-drm.ko.xz
Deleting /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-uvm.ko.xz
Deleting /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-peermem.ko.xz
Running depmod..... done.
Executing post-transaction command.......... done.
# dkms status -m nvidia
nvidia/590.44.01, 6.17.11-300.fc42.x86_64, x86_64: built

Then, we force a rebuild, so nvidia-peermem is rebuilt and correctly linked to OFED:

# sudo dkms build -m nvidia/590.44.01 --force
Sign command: /lib/modules/6.17.11-300.fc42.x86_64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module(s).................... done.
Signing module /var/lib/dkms/nvidia/590.44.01/build/kernel-open/nvidia.ko
Signing module /var/lib/dkms/nvidia/590.44.01/build/kernel-open/nvidia-modeset.ko
Signing module /var/lib/dkms/nvidia/590.44.01/build/kernel-open/nvidia-drm.ko
Signing module /var/lib/dkms/nvidia/590.44.01/build/kernel-open/nvidia-uvm.ko
Signing module /var/lib/dkms/nvidia/590.44.01/build/kernel-open/nvidia-peermem.ko

And then we install it again:

# dkms install -m nvidia/590.44.01
Installing /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia.ko.xz
Installing /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-modeset.ko.xz
Installing /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-drm.ko.xz
Installing /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-uvm.ko.xz
Installing /lib/modules/6.17.11-300.fc42.x86_64/extra/nvidia-peermem.ko.xz
Running depmod...... done.
Executing post-transaction command.......... done.

NVSwitch#

To install Fabric Manager, NSCQ, NVSDM, IMEX:

Red Hat Enterprise Linux 8/9, Rocky Linux 8/9, Oracle Linux 8/9, Amazon Linux 2023, Kylin 10

# dnf module install nvidia-driver:$stream/fm

Red Hat Enterprise Linux 10, Rocky Linux 10, Fedora 42

# dnf install nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex

SUSE Enterprise Linux Server 15, openSUSE Leap 15

# zypper install nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex

Azure Linux 3

# tdnf install nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex