# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols

> **Status: historical build notes.** These notes predate the current workflow.
> For the current build/load model see
> [03-architecture/06-host-module-and-hybrid-model.md](03-architecture/06-host-module-and-hybrid-model.md)
> and [02-getting-started/01-install-and-first-run.md](02-getting-started/01-install-and-first-run.md).

## Overview

The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`:
- `nvidia_vgpu_vfio_probe`
- `nvidia_vgpu_vfio_remove`
- `nvidia_vgpu_vfio_get_ops`
- `nvidia_vgpu_vfio_set_ops`
- `nv_vgpu_mdev_register_parent`
- `nv_vgpu_mdev_unregister_parent`
- And all GSP RPC integration symbols

These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`.

## Build Order (Critical)

```
1. Build open-iov-vfio.ko          → Generates kernel/Module.symvers
2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers
3. Load open-iov-vfio.ko first (exports the vGPU VFIO symbols)
4. Optionally load nvidia.ko (imports those symbols)
```

## Automated Build (Recommended)

Use the helper script:

```bash
cd open-iov
./scripts/build_nvidia_openrm.sh
```

This script:
1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`)
2. Cleans previous NVIDIA OpenRM build
3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set
4. Verifies vGPU symbols are resolved

## Manual Build

```bash
# Step 1: Build Open-IOV kernel module
cd $OPEN_IOV_ROOT
make kernel

# Step 2: Build NVIDIA OpenRM with extra symbols
cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules

# Step 3: Verify symbols resolved
grep -i vgpu Module.symvers
```

## Verification

After building, verify the symbols are resolved:

```bash
# Check Module.symvers has vGPU symbols
grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers

# Check nvidia.ko modinfo
modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu
```

## Common Issues

| Issue | Cause | Fix |
|-------|-------|-----|
| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` |
| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first |
| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild |
| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config |

## Module Loading Order

```bash
# 1. Load Open-IOV VFIO driver FIRST (exports the vGPU VFIO symbols and
#    registers the mdev parent on the physical GPU)
sudo insmod kernel/open-iov-vfio.ko

# 2. Load the optional NVIDIA OpenRM (imports the vGPU VFIO symbols; build it
#    with KBUILD_EXTRA_SYMBOLS as above)
sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko

# 3. Copy firmware
sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/

# 4. Create mdev device
./scripts/create_mdev.sh

# 5. Launch QEMU with vGPU
./scripts/launch_qemu_vgpu.sh
```

## Key Files

| File | Purpose |
|------|---------|
| `scripts/build_nvidia_openrm.sh` | Automated build script |
| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko |
| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment |
| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild |

## Troubleshooting

If symbols are still unresolved after rebuild:

```bash
# Full clean rebuild
cd nvidia-open-gpu-kernel-modules/kernel-open
make clean
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules

# Check modpost output for unresolved symbols
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu"
```