diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 13:48:07 -0400 |
| commit | fade84f8b5b781d29daad40699d07928dedea550 (patch) | |
| tree | e4156b3d0a6a1c794c90ef6834b949321cdfb5cb /docs/BUILD_NVIDIA_OPENRM.md | |
| parent | c587eac377c1c5b720142edadebaace5cb1b17c0 (diff) | |
| download | open-iov-fade84f8b5b781d29daad40699d07928dedea550.tar.gz open-iov-fade84f8b5b781d29daad40699d07928dedea550.zip | |
docs: merge the legacy flat documents, add the CUDA deep-dive and link checks
* Merge and remove the pre-reboot flat documents, preserving their still-useful
material in the structured tree: debugging workflows and the EPT-livelock
case study (05-operations/02), performance microbenchmarks and A/B
methodology (05-operations/01), the version-upgrade runbook
(05-operations/02 section 4, with corrected host/guest framing), kernel-code
pitfalls (07-developers) and the OpenRM build/load order
(03-architecture/06 section 7). The remaining legacy files were verified
redundant and removed.
* docs/README now records where each migrated topic lives.
* Portability: the config YAML files are called out as design targets that no
code reads yet, with what adding support actually requires today.
* Getting started: the OpenRM step is marked optional (native vGPU-management
workflows only; the emulated path uses the distro driver).
* tools docs: rmtool/rmtop marked as planned, pointing at the implemented
tools/rmq subset.
* wiki: new CUDA-Userspace-Driver deep-dive page (public-safe, links the
canonical userspace/cuda analysis) and index update.
* scripts/check-docs-links.py plus a CI step so relative links cannot rot.
* STATE.md references to removed documents repointed.
Diffstat (limited to 'docs/BUILD_NVIDIA_OPENRM.md')
| -rw-r--r-- | docs/BUILD_NVIDIA_OPENRM.md | 123 |
1 files changed, 0 insertions, 123 deletions
diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md deleted file mode 100644 index 309a64f..0000000 --- a/docs/BUILD_NVIDIA_OPENRM.md +++ /dev/null @@ -1,123 +0,0 @@ -# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols - -> **Status: historical build notes.** These notes predate the current workflow. -> For the current build/load model see -> [03-architecture/06-host-module-and-hybrid-model.md](03-architecture/06-host-module-and-hybrid-model.md) -> and [02-getting-started/01-install-and-first-run.md](02-getting-started/01-install-and-first-run.md). - -## Overview - -The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: -- `nvidia_vgpu_vfio_probe` -- `nvidia_vgpu_vfio_remove` -- `nvidia_vgpu_vfio_get_ops` -- `nvidia_vgpu_vfio_set_ops` -- `nv_vgpu_mdev_register_parent` -- `nv_vgpu_mdev_unregister_parent` -- And all GSP RPC integration symbols - -These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. - -## Build Order (Critical) - -``` -1. Build open-iov-vfio.ko → Generates kernel/Module.symvers -2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers -3. Load open-iov-vfio.ko first (exports the vGPU VFIO symbols) -4. Optionally load nvidia.ko (imports those symbols) -``` - -## Automated Build (Recommended) - -Use the helper script: - -```bash -cd open-iov -./scripts/build_nvidia_openrm.sh -``` - -This script: -1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) -2. Cleans previous NVIDIA OpenRM build -3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set -4. Verifies vGPU symbols are resolved - -## Manual Build - -```bash -# Step 1: Build Open-IOV kernel module -cd $OPEN_IOV_ROOT -make kernel - -# Step 2: Build NVIDIA OpenRM with extra symbols -cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules - -# Step 3: Verify symbols resolved -grep -i vgpu Module.symvers -``` - -## Verification - -After building, verify the symbols are resolved: - -```bash -# Check Module.symvers has vGPU symbols -grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers - -# Check nvidia.ko modinfo -modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu -``` - -## Common Issues - -| Issue | Cause | Fix | -|-------|-------|-----| -| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | -| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | -| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | -| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | - -## Module Loading Order - -```bash -# 1. Load Open-IOV VFIO driver FIRST (exports the vGPU VFIO symbols and -# registers the mdev parent on the physical GPU) -sudo insmod kernel/open-iov-vfio.ko - -# 2. Load the optional NVIDIA OpenRM (imports the vGPU VFIO symbols; build it -# with KBUILD_EXTRA_SYMBOLS as above) -sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 3. Copy firmware -sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ - -# 4. Create mdev device -./scripts/create_mdev.sh - -# 5. Launch QEMU with vGPU -./scripts/launch_qemu_vgpu.sh -``` - -## Key Files - -| File | Purpose | -|------|---------| -| `scripts/build_nvidia_openrm.sh` | Automated build script | -| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | -| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | -| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | - -## Troubleshooting - -If symbols are still unresolved after rebuild: - -```bash -# Full clean rebuild -cd nvidia-open-gpu-kernel-modules/kernel-open -make clean -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules - -# Check modpost output for unresolved symbols -KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" -```
\ No newline at end of file |
