1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
# Open-IOV Development Tools
Collection of scripts and utilities for vGPU development, debugging, and testing.
## Directory Structure
```
tools/
├── module_ops/ # Kernel module load/unload/reload utilities
├── debug/ # Debugging and diagnostic tools
├── gr-regs/ # Read-only GR/FECS/SKED register sampler + ground truth
├── test/ # Test runners and validation scripts
├── vm/ # VM/QEMU management for vGPU testing
├── firmware/ # GSP firmware management
└── monitoring/ # Real-time monitoring utilities
```
## Quick Start
```bash
# Load modules in converged mode (host + vGPU)
sudo ./module_ops/load_converged.sh
# Unload all modules cleanly
sudo ./module_ops/unload_all.sh
# Run dry-run validation
sudo ./module_ops/dry_run.sh
# Monitor kernel logs in real-time
./monitoring/watch_dmesg.sh
# Verify vGPU sysfs interface
./debug/verify_vgpu_sysfs.sh
```
## Module Operations
### `module_ops/load_converged.sh`
Loads the complete vGPU stack in converged mode:
1. `mdev` core module
2. `nvidia.ko` (patched open-gpu-kernel-modules with NV_VGPU_KVM_BUILD)
3. `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko`
4. `nvidia-vgpu-vfio.ko` (open-iov kernel module)
### `module_ops/unload_all.sh`
Cleanly unloads all modules in reverse order with proper cleanup.
### `module_ops/dry_run.sh`
Runs nvidia-vgpu-vfio.ko in dry-run mode to validate firmware, BAR layout, IRQ caps, IOMMU without hardware changes.
### `module_ops/reload_vgpu.sh`
Reloads only the vGPU kernel module (nvidia-vgpu-vfio.ko) while keeping host nvidia.ko loaded.
## Debug Tools
### `debug/verify_vgpu_sysfs.sh`
Verifies mdev parent registration and vGPU types in sysfs.
### `debug/check_module_state.sh`
Checks module initstate, refcnt, and holder information.
### `debug/dump_vgpu_regs.sh`
Dumps vGPU device registers via debugfs (if available).
### `debug/capture_crash_logs.sh`
Captures kernel logs, dmesg, and module state after a crash.
## Test Scripts
### `test/run_integration_tests.sh`
Runs the integration test suite (host_init, guest_init).
### `test/test_mdev_create.sh`
Creates an mdev device and verifies basic functionality.
### `test/benchmark_vgpu.sh`
Runs performance benchmarks on vGPU.
## VM Management
### `vm/create_test_vm.sh`
Creates a test VM with vGPU passthrough using QEMU.
### `vm/launch_guest_driver.sh`
Launches guest with NVIDIA driver installed for validation.
## Firmware Tools
### `firmware/verify_gsp_firmware.sh`
Verifies GSP firmware files exist and are correct version.
### `firmware/install_tu104_firmware.sh`
Installs TU104 GSP firmware to /lib/firmware/nvidia/tu104/gsp/
## Monitoring
### `monitoring/watch_dmesg.sh`
Real-time kernel log monitoring with filtering for vGPU/NVIDIA messages.
### `monitoring/watch_module_params.sh`
Monitors module parameters and sysfs attributes.
## Environment Variables
```bash
export OPEN_IOV_ROOT=$OPEN_IOV_ROOT
export NVIDIA_OPEN_KERNEL=$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
export VGPU_KERNEL=$OPEN_IOV_ROOT/kernel
export VGPU_USERSPACE=$OPEN_IOV_ROOT/userspace
```
## Module Parameters for Debugging
| Parameter | Description | Values |
|-----------|-------------|--------|
| `NVreg_RmMsg` | RM debug logging | `":"` (verbose), `"0"` (none) |
| `NVreg_RegisterPCIDriver` | PCI driver registration | `0` (disable), `1` (enable) |
| `NVreg_GpuInitOnProbe` | GPU init at probe | `0` (defer), `1` (immediate) |
| `NVreg_PreserveVideoMemoryAllocations` | Video memory preservation | `0` (disable), `1` (enable), `2` (auto) |
| `NVreg_UseKernelSuspendNotifiers` | Suspend notifiers | `0` (disable), `1` (enable) |
| `dyndbg` | Dynamic debug | `'+p'` (all), `'+pf'` (probe+fail) |
## Converged Mode Architecture
```
Host nvidia.ko (patched) nvidia-vgpu-vfio.ko (open-iov)
├── mdev_register_parent() ←── mdev_register_driver() only
├── EXPORT nvidia_vgpu_vfio_set_ops
├── EXPORT nvidia_vgpu_vfio_get_ops
└── Implements RM vGPU ops Calls RM via get_ops, provides VFIO ops via set_ops
```
## Troubleshooting
### Module stuck in "going" state
```bash
cat /sys/module/nvidia-vgpu-vfio/initstate
cat /sys/module/nvidia-vgpu-vfio/refcnt
# Requires reboot to clear
```
### mdev parent not registered
```bash
ls /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/
# Should show nvidia-vgpu-* directories with create/available_instances
```
### Symbol version mismatch
```bash
# Rebuild open-gpu-kernel-modules with VGX_KVM_BUILD=1
cd $NVIDIA_OPEN_KERNEL && make clean && make -j$(nproc) VGX_KVM_BUILD=1
```
### IOMMU/VT-d not enabled
```bash
cat /proc/cmdline | grep -E "iommu|intel_iommu"
# Add: intel_iommu=on iommu=pt to kernel cmdline
```
|
