From 5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60 Mon Sep 17 00:00:00 2001
From: Arthur Hanson Rasmusson <arthur@vgpu.io>
Date: Mon, 14 Sep 2026 06:43:11 -0400
Subject: VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl
 handlers

- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create()
- Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions
- Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP
- Added config space read/write handlers at offset 0x70000000000
- Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers)
- Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL)
- Added udev rules for vfio group permissions
- Created stack_manager.sh unified tool
- Fixed mdev remove crash with deferred cleanup in vfio_device_release callback
- Kernel 6.8+ IOMMUFD compatibility
- QEMU launches successfully with vGPU passthrough
---
 docs/BUILD_NVIDIA_OPENRM.md | 116 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
 create mode 100644 docs/BUILD_NVIDIA_OPENRM.md

(limited to 'docs/BUILD_NVIDIA_OPENRM.md')

diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md
new file mode 100644
index 0000000..5699036
--- /dev/null
+++ b/docs/BUILD_NVIDIA_OPENRM.md
@@ -0,0 +1,116 @@
+# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols
+
+## Overview
+
+The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`:
+- `nvidia_vgpu_vfio_probe`
+- `nvidia_vgpu_vfio_remove`
+- `nvidia_vgpu_vfio_get_ops`
+- `nvidia_vgpu_vfio_set_ops`
+- `nv_vgpu_mdev_register_parent`
+- `nv_vgpu_mdev_unregister_parent`
+- And all GSP RPC integration symbols
+
+These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`.
+
+## Build Order (Critical)
+
+```
+1. Build open-iov-vfio.ko          → Generates kernel/Module.symvers
+2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers
+3. Load nvidia.ko first
+4. Load open-iov-vfio.ko
+```
+
+## Automated Build (Recommended)
+
+Use the helper script:
+
+```bash
+cd open-iov
+./scripts/build_nvidia_openrm.sh
+```
+
+This script:
+1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`)
+2. Cleans previous NVIDIA OpenRM build
+3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set
+4. Verifies vGPU symbols are resolved
+
+## Manual Build
+
+```bash
+# Step 1: Build Open-IOV kernel module
+cd $OPEN_IOV_ROOT
+make kernel
+
+# Step 2: Build NVIDIA OpenRM with extra symbols
+cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
+KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules
+
+# Step 3: Verify symbols resolved
+grep -i vgpu Module.symvers
+```
+
+## Verification
+
+After building, verify the symbols are resolved:
+
+```bash
+# Check Module.symvers has vGPU symbols
+grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers
+
+# Check nvidia.ko modinfo
+modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu
+```
+
+## Common Issues
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` |
+| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first |
+| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild |
+| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config |
+
+## Module Loading Order
+
+```bash
+# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols)
+sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
+
+# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko)
+sudo insmod kernel/open-iov-vfio.ko
+
+# 3. Copy firmware
+sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
+
+# 4. Create mdev device
+./scripts/create_mdev.sh
+
+# 5. Launch QEMU with vGPU
+./scripts/launch_qemu_vgpu.sh
+```
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `scripts/build_nvidia_openrm.sh` | Automated build script |
+| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko |
+| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment |
+| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild |
+
+## Troubleshooting
+
+If symbols are still unresolved after rebuild:
+
+```bash
+# Full clean rebuild
+cd nvidia-open-gpu-kernel-modules/kernel-open
+make clean
+KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules
+
+# Check modpost output for unresolved symbols
+KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu"
+```
\ No newline at end of file
-- 
cgit v1.2.3