blob: 56990360b77ea05dada0b48f64ed7af9d736e9a3 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
|
# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols
## Overview
The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`:
- `nvidia_vgpu_vfio_probe`
- `nvidia_vgpu_vfio_remove`
- `nvidia_vgpu_vfio_get_ops`
- `nvidia_vgpu_vfio_set_ops`
- `nv_vgpu_mdev_register_parent`
- `nv_vgpu_mdev_unregister_parent`
- And all GSP RPC integration symbols
These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`.
## Build Order (Critical)
```
1. Build open-iov-vfio.ko → Generates kernel/Module.symvers
2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers
3. Load nvidia.ko first
4. Load open-iov-vfio.ko
```
## Automated Build (Recommended)
Use the helper script:
```bash
cd open-iov
./scripts/build_nvidia_openrm.sh
```
This script:
1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`)
2. Cleans previous NVIDIA OpenRM build
3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set
4. Verifies vGPU symbols are resolved
## Manual Build
```bash
# Step 1: Build Open-IOV kernel module
cd $OPEN_IOV_ROOT
make kernel
# Step 2: Build NVIDIA OpenRM with extra symbols
cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules
# Step 3: Verify symbols resolved
grep -i vgpu Module.symvers
```
## Verification
After building, verify the symbols are resolved:
```bash
# Check Module.symvers has vGPU symbols
grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers
# Check nvidia.ko modinfo
modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu
```
## Common Issues
| Issue | Cause | Fix |
|-------|-------|-----|
| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` |
| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first |
| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild |
| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config |
## Module Loading Order
```bash
# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols)
sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko)
sudo insmod kernel/open-iov-vfio.ko
# 3. Copy firmware
sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/
# 4. Create mdev device
./scripts/create_mdev.sh
# 5. Launch QEMU with vGPU
./scripts/launch_qemu_vgpu.sh
```
## Key Files
| File | Purpose |
|------|---------|
| `scripts/build_nvidia_openrm.sh` | Automated build script |
| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko |
| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment |
| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild |
## Troubleshooting
If symbols are still unresolved after rebuild:
```bash
# Full clean rebuild
cd nvidia-open-gpu-kernel-modules/kernel-open
make clean
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules
# Check modpost output for unresolved symbols
KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu"
```
|
