# Open-IOV Development Tools

Collection of scripts and utilities for vGPU development, debugging, and testing.

## Directory Structure

```
tools/
├── module_ops/           # Kernel module load/unload/reload utilities
├── debug/                # Debugging and diagnostic tools
├── gr-regs/              # Read-only GR/FECS/SKED register sampler + ground truth
├── test/                 # Test runners and validation scripts
├── vm/                   # VM/QEMU management for vGPU testing
├── firmware/             # GSP firmware management
└── monitoring/           # Real-time monitoring utilities
```

## Quick Start

```bash
# Load modules in converged mode (host + vGPU)
sudo ./module_ops/load_converged.sh

# Unload all modules cleanly
sudo ./module_ops/unload_all.sh

# Run dry-run validation
sudo ./module_ops/dry_run.sh

# Monitor kernel logs in real-time
./monitoring/watch_dmesg.sh

# Verify vGPU sysfs interface
./debug/verify_vgpu_sysfs.sh
```

## Module Operations

### `module_ops/load_converged.sh`
Loads the complete vGPU stack in converged mode:
1. `mdev` core module
2. `nvidia.ko` (patched open-gpu-kernel-modules with NV_VGPU_KVM_BUILD)
3. `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko`
4. `nvidia-vgpu-vfio.ko` (open-iov kernel module)

### `module_ops/unload_all.sh`
Cleanly unloads all modules in reverse order with proper cleanup.

### `module_ops/dry_run.sh`
Runs nvidia-vgpu-vfio.ko in dry-run mode to validate firmware, BAR layout, IRQ caps, IOMMU without hardware changes.

### `module_ops/reload_vgpu.sh`
Reloads only the vGPU kernel module (nvidia-vgpu-vfio.ko) while keeping host nvidia.ko loaded.

## Debug Tools

### `debug/verify_vgpu_sysfs.sh`
Verifies mdev parent registration and vGPU types in sysfs.

### `debug/check_module_state.sh`
Checks module initstate, refcnt, and holder information.

### `debug/dump_vgpu_regs.sh`
Dumps vGPU device registers via debugfs (if available).

### `debug/capture_crash_logs.sh`
Captures kernel logs, dmesg, and module state after a crash.

## Test Scripts

### `test/run_integration_tests.sh`
Runs the integration test suite (host_init, guest_init).

### `test/test_mdev_create.sh`
Creates an mdev device and verifies basic functionality.

### `test/benchmark_vgpu.sh`
Runs performance benchmarks on vGPU.

## VM Management

### `vm/create_test_vm.sh`
Creates a test VM with vGPU passthrough using QEMU.

### `vm/launch_guest_driver.sh`
Launches guest with NVIDIA driver installed for validation.

## Firmware Tools

### `firmware/verify_gsp_firmware.sh`
Verifies GSP firmware files exist and are correct version.

### `firmware/install_tu104_firmware.sh`
Installs TU104 GSP firmware to /lib/firmware/nvidia/tu104/gsp/

## Monitoring

### `monitoring/watch_dmesg.sh`
Real-time kernel log monitoring with filtering for vGPU/NVIDIA messages.

### `monitoring/watch_module_params.sh`
Monitors module parameters and sysfs attributes.

## Environment Variables

```bash
export OPEN_IOV_ROOT=$OPEN_IOV_ROOT
export NVIDIA_OPEN_KERNEL=$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
export VGPU_KERNEL=$OPEN_IOV_ROOT/kernel
export VGPU_USERSPACE=$OPEN_IOV_ROOT/userspace
```

## Module Parameters for Debugging

| Parameter | Description | Values |
|-----------|-------------|--------|
| `NVreg_RmMsg` | RM debug logging | `":"` (verbose), `"0"` (none) |
| `NVreg_RegisterPCIDriver` | PCI driver registration | `0` (disable), `1` (enable) |
| `NVreg_GpuInitOnProbe` | GPU init at probe | `0` (defer), `1` (immediate) |
| `NVreg_PreserveVideoMemoryAllocations` | Video memory preservation | `0` (disable), `1` (enable), `2` (auto) |
| `NVreg_UseKernelSuspendNotifiers` | Suspend notifiers | `0` (disable), `1` (enable) |
| `dyndbg` | Dynamic debug | `'+p'` (all), `'+pf'` (probe+fail) |

## Converged Mode Architecture

```
Host nvidia.ko (patched)          nvidia-vgpu-vfio.ko (open-iov)
├── mdev_register_parent()  ←──  mdev_register_driver() only
├── EXPORT nvidia_vgpu_vfio_set_ops
├── EXPORT nvidia_vgpu_vfio_get_ops
└── Implements RM vGPU ops       Calls RM via get_ops, provides VFIO ops via set_ops
```

## Troubleshooting

### Module stuck in "going" state
```bash
cat /sys/module/nvidia-vgpu-vfio/initstate
cat /sys/module/nvidia-vgpu-vfio/refcnt
# Requires reboot to clear
```

### mdev parent not registered
```bash
ls /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/
# Should show nvidia-vgpu-* directories with create/available_instances
```

### Symbol version mismatch
```bash
# Rebuild open-gpu-kernel-modules with VGX_KVM_BUILD=1
cd $NVIDIA_OPEN_KERNEL && make clean && make -j$(nproc) VGX_KVM_BUILD=1
```

### IOMMU/VT-d not enabled
```bash
cat /proc/cmdline | grep -E "iommu|intel_iommu"
# Add: intel_iommu=on iommu=pt to kernel cmdline
```