diff options
Diffstat (limited to 'docs/BUILD_NVIDIA_OPENRM.md')
| -rw-r--r-- | docs/BUILD_NVIDIA_OPENRM.md | 116 |
1 files changed, 116 insertions, 0 deletions
diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md new file mode 100644 index 0000000..5699036 --- /dev/null +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -0,0 +1,116 @@ +# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols + +## Overview + +The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: +- `nvidia_vgpu_vfio_probe` +- `nvidia_vgpu_vfio_remove` +- `nvidia_vgpu_vfio_get_ops` +- `nvidia_vgpu_vfio_set_ops` +- `nv_vgpu_mdev_register_parent` +- `nv_vgpu_mdev_unregister_parent` +- And all GSP RPC integration symbols + +These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. + +## Build Order (Critical) + +``` +1. Build open-iov-vfio.ko → Generates kernel/Module.symvers +2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers +3. Load nvidia.ko first +4. Load open-iov-vfio.ko +``` + +## Automated Build (Recommended) + +Use the helper script: + +```bash +cd open-iov +./scripts/build_nvidia_openrm.sh +``` + +This script: +1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) +2. Cleans previous NVIDIA OpenRM build +3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set +4. Verifies vGPU symbols are resolved + +## Manual Build + +```bash +# Step 1: Build Open-IOV kernel module +cd $OPEN_IOV_ROOT +make kernel + +# Step 2: Build NVIDIA OpenRM with extra symbols +cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Step 3: Verify symbols resolved +grep -i vgpu Module.symvers +``` + +## Verification + +After building, verify the symbols are resolved: + +```bash +# Check Module.symvers has vGPU symbols +grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers + +# Check nvidia.ko modinfo +modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | +| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | +| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | +| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | + +## Module Loading Order + +```bash +# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Copy firmware +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ + +# 4. Create mdev device +./scripts/create_mdev.sh + +# 5. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh +``` + +## Key Files + +| File | Purpose | +|------|---------| +| `scripts/build_nvidia_openrm.sh` | Automated build script | +| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | +| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | +| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | + +## Troubleshooting + +If symbols are still unresolved after rebuild: + +```bash +# Full clean rebuild +cd nvidia-open-gpu-kernel-modules/kernel-open +make clean +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Check modpost output for unresolved symbols +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" +```
\ No newline at end of file |