diff options
Diffstat (limited to 'tools/README.md')
| -rw-r--r-- | tools/README.md | 158 |
1 files changed, 158 insertions, 0 deletions
diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..f9e0018 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,158 @@ +# Open-IOV Development Tools + +Collection of scripts and utilities for vGPU development, debugging, and testing. + +## Directory Structure + +``` +tools/ +├── module_ops/ # Kernel module load/unload/reload utilities +├── debug/ # Debugging and diagnostic tools +├── test/ # Test runners and validation scripts +├── vm/ # VM/QEMU management for vGPU testing +├── firmware/ # GSP firmware management +└── monitoring/ # Real-time monitoring utilities +``` + +## Quick Start + +```bash +# Load modules in converged mode (host + vGPU) +sudo ./module_ops/load_converged.sh + +# Unload all modules cleanly +sudo ./module_ops/unload_all.sh + +# Run dry-run validation +sudo ./module_ops/dry_run.sh + +# Monitor kernel logs in real-time +./monitoring/watch_dmesg.sh + +# Verify vGPU sysfs interface +./debug/verify_vgpu_sysfs.sh +``` + +## Module Operations + +### `module_ops/load_converged.sh` +Loads the complete vGPU stack in converged mode: +1. `mdev` core module +2. `nvidia.ko` (patched open-gpu-kernel-modules with NV_VGPU_KVM_BUILD) +3. `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko` +4. `nvidia-vgpu-vfio.ko` (open-iov kernel module) + +### `module_ops/unload_all.sh` +Cleanly unloads all modules in reverse order with proper cleanup. + +### `module_ops/dry_run.sh` +Runs nvidia-vgpu-vfio.ko in dry-run mode to validate firmware, BAR layout, IRQ caps, IOMMU without hardware changes. + +### `module_ops/reload_vgpu.sh` +Reloads only the vGPU kernel module (nvidia-vgpu-vfio.ko) while keeping host nvidia.ko loaded. + +## Debug Tools + +### `debug/verify_vgpu_sysfs.sh` +Verifies mdev parent registration and vGPU types in sysfs. + +### `debug/check_module_state.sh` +Checks module initstate, refcnt, and holder information. + +### `debug/dump_vgpu_regs.sh` +Dumps vGPU device registers via debugfs (if available). + +### `debug/capture_crash_logs.sh` +Captures kernel logs, dmesg, and module state after a crash. + +## Test Scripts + +### `test/run_integration_tests.sh` +Runs the integration test suite (host_init, guest_init). + +### `test/test_mdev_create.sh` +Creates an mdev device and verifies basic functionality. + +### `test/benchmark_vgpu.sh` +Runs performance benchmarks on vGPU. + +## VM Management + +### `vm/create_test_vm.sh` +Creates a test VM with vGPU passthrough using QEMU. + +### `vm/launch_guest_driver.sh` +Launches guest with NVIDIA driver installed for validation. + +## Firmware Tools + +### `firmware/verify_gsp_firmware.sh` +Verifies GSP firmware files exist and are correct version. + +### `firmware/install_tu104_firmware.sh` +Installs TU104 GSP firmware to /lib/firmware/nvidia/tu104/gsp/ + +## Monitoring + +### `monitoring/watch_dmesg.sh` +Real-time kernel log monitoring with filtering for vGPU/NVIDIA messages. + +### `monitoring/watch_module_params.sh` +Monitors module parameters and sysfs attributes. + +## Environment Variables + +```bash +export OPEN_IOV_ROOT=$OPEN_IOV_ROOT +export NVIDIA_OPEN_KERNEL=$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +export VGPU_KERNEL=$OPEN_IOV_ROOT/kernel +export VGPU_USERSPACE=$OPEN_IOV_ROOT/userspace +``` + +## Module Parameters for Debugging + +| Parameter | Description | Values | +|-----------|-------------|--------| +| `NVreg_RmMsg` | RM debug logging | `":"` (verbose), `"0"` (none) | +| `NVreg_RegisterPCIDriver` | PCI driver registration | `0` (disable), `1` (enable) | +| `NVreg_GpuInitOnProbe` | GPU init at probe | `0` (defer), `1` (immediate) | +| `NVreg_PreserveVideoMemoryAllocations` | Video memory preservation | `0` (disable), `1` (enable), `2` (auto) | +| `NVreg_UseKernelSuspendNotifiers` | Suspend notifiers | `0` (disable), `1` (enable) | +| `dyndbg` | Dynamic debug | `'+p'` (all), `'+pf'` (probe+fail) | + +## Converged Mode Architecture + +``` +Host nvidia.ko (patched) nvidia-vgpu-vfio.ko (open-iov) +├── mdev_register_parent() ←── mdev_register_driver() only +├── EXPORT nvidia_vgpu_vfio_set_ops +├── EXPORT nvidia_vgpu_vfio_get_ops +└── Implements RM vGPU ops Calls RM via get_ops, provides VFIO ops via set_ops +``` + +## Troubleshooting + +### Module stuck in "going" state +```bash +cat /sys/module/nvidia-vgpu-vfio/initstate +cat /sys/module/nvidia-vgpu-vfio/refcnt +# Requires reboot to clear +``` + +### mdev parent not registered +```bash +ls /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/ +# Should show nvidia-vgpu-* directories with create/available_instances +``` + +### Symbol version mismatch +```bash +# Rebuild open-gpu-kernel-modules with VGX_KVM_BUILD=1 +cd $NVIDIA_OPEN_KERNEL && make clean && make -j$(nproc) VGX_KVM_BUILD=1 +``` + +### IOMMU/VT-d not enabled +```bash +cat /proc/cmdline | grep -E "iommu|intel_iommu" +# Add: intel_iommu=on iommu=pt to kernel cmdline +```
\ No newline at end of file |