From 68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- docs/BUILD_NVIDIA_OPENRM.md | 116 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 docs/BUILD_NVIDIA_OPENRM.md (limited to 'docs/BUILD_NVIDIA_OPENRM.md') diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md new file mode 100644 index 0000000..5699036 --- /dev/null +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -0,0 +1,116 @@ +# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols + +## Overview + +The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: +- `nvidia_vgpu_vfio_probe` +- `nvidia_vgpu_vfio_remove` +- `nvidia_vgpu_vfio_get_ops` +- `nvidia_vgpu_vfio_set_ops` +- `nv_vgpu_mdev_register_parent` +- `nv_vgpu_mdev_unregister_parent` +- And all GSP RPC integration symbols + +These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. + +## Build Order (Critical) + +``` +1. Build open-iov-vfio.ko → Generates kernel/Module.symvers +2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers +3. Load nvidia.ko first +4. Load open-iov-vfio.ko +``` + +## Automated Build (Recommended) + +Use the helper script: + +```bash +cd open-iov +./scripts/build_nvidia_openrm.sh +``` + +This script: +1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) +2. Cleans previous NVIDIA OpenRM build +3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set +4. Verifies vGPU symbols are resolved + +## Manual Build + +```bash +# Step 1: Build Open-IOV kernel module +cd $OPEN_IOV_ROOT +make kernel + +# Step 2: Build NVIDIA OpenRM with extra symbols +cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Step 3: Verify symbols resolved +grep -i vgpu Module.symvers +``` + +## Verification + +After building, verify the symbols are resolved: + +```bash +# Check Module.symvers has vGPU symbols +grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers + +# Check nvidia.ko modinfo +modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | +| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | +| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | +| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | + +## Module Loading Order + +```bash +# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Copy firmware +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ + +# 4. Create mdev device +./scripts/create_mdev.sh + +# 5. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh +``` + +## Key Files + +| File | Purpose | +|------|---------| +| `scripts/build_nvidia_openrm.sh` | Automated build script | +| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | +| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | +| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | + +## Troubleshooting + +If symbols are still unresolved after rebuild: + +```bash +# Full clean rebuild +cd nvidia-open-gpu-kernel-modules/kernel-open +make clean +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Check modpost output for unresolved symbols +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" +``` \ No newline at end of file -- cgit v1.2.3