summaryrefslogtreecommitdiffstats
path: root/tools
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-13 18:36:21 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-13 18:36:21 -0400
commit00e54c13c7c9030c4d5f7151467f2d85a3730e29 (patch)
tree17a45239df8d2e4912ff3697ed06e85a10b969de /tools
downloadopen-iov-00e54c13c7c9030c4d5f7151467f2d85a3730e29.tar.gz
open-iov-00e54c13c7c9030c4d5f7151467f2d85a3730e29.zip
Initial commit: Open-IOV vGPU-IO complete stack
- Firmware decompilation: GSP 570.144W (28.5MB ELF, RV64IMAC), 132 decompiled C files - Kernel modules: nvidia.ko (34MB, OpenRM), open-iov-vfio.ko (2.8MB, VFIO mdev), gsp-vgpu.ko (1.8MB, guest driver) - Userspace: libiov.so/libvgpu.so (76KB), example_vgpu (28KB), rpc_emulator.c - PCI config space emulation (256B with PM, MSI, MSI-X, PCIe caps) - 12 RPC types fully mapped from firmware decompilation - BAR trap, WPR, scheduling, notification, migration emulation - Helper scripts: load_vgpu_stack, create_mdev, launch_qemu_vgpu, test_vgpu_stack, setup_guest_driver, unload_vgpu_stack - Architecture documentation - Wiki: 10 technical documents - Git LFS for large binaries (.ko, .bin, .iso, .img, .o, ghidra repos) - nvidia-open-gpu-kernel-modules as submodule
Diffstat (limited to 'tools')
-rw-r--r--tools/README.md158
-rw-r--r--tools/debug/nv2080_ctrl.c531
-rwxr-xr-xtools/debug/nv2080_ctrl.sh168
-rwxr-xr-xtools/firmware/verify_gsp_firmware.sh60
-rw-r--r--tools/ghidra/analyze_fwimage.py3
-rwxr-xr-xtools/ghidra/analyze_gsp_firmware.sh3
-rw-r--r--tools/ghidra/create_gsp_project.py3
-rw-r--r--tools/ghidra/diff_gsp_firmware.py3
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr0
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java3
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java3
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java3
-rw-r--r--tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py3
-rw-r--r--tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr0
-rw-r--r--tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~0
-rw-r--r--tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java3
-rw-r--r--tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py3
-rw-r--r--tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py3
-rw-r--r--tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py3
-rw-r--r--tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py3
-rwxr-xr-xtools/module_ops/dry_run.sh47
-rwxr-xr-xtools/module_ops/load_converged.sh95
-rwxr-xr-xtools/module_ops/unload_all.sh42
-rwxr-xr-xtools/test/run_integration_tests.sh182
-rw-r--r--tools/vm/cloud-init.yaml26
-rwxr-xr-xtools/vm/launch_vgpu_qemu.sh65
-rwxr-xr-xtools/vm/setup_mdev.sh65
42 files changed, 1478 insertions, 0 deletions
diff --git a/tools/README.md b/tools/README.md
new file mode 100644
index 0000000..f9e0018
--- /dev/null
+++ b/tools/README.md
@@ -0,0 +1,158 @@
+# Open-IOV Development Tools
+
+Collection of scripts and utilities for vGPU development, debugging, and testing.
+
+## Directory Structure
+
+```
+tools/
+├── module_ops/ # Kernel module load/unload/reload utilities
+├── debug/ # Debugging and diagnostic tools
+├── test/ # Test runners and validation scripts
+├── vm/ # VM/QEMU management for vGPU testing
+├── firmware/ # GSP firmware management
+└── monitoring/ # Real-time monitoring utilities
+```
+
+## Quick Start
+
+```bash
+# Load modules in converged mode (host + vGPU)
+sudo ./module_ops/load_converged.sh
+
+# Unload all modules cleanly
+sudo ./module_ops/unload_all.sh
+
+# Run dry-run validation
+sudo ./module_ops/dry_run.sh
+
+# Monitor kernel logs in real-time
+./monitoring/watch_dmesg.sh
+
+# Verify vGPU sysfs interface
+./debug/verify_vgpu_sysfs.sh
+```
+
+## Module Operations
+
+### `module_ops/load_converged.sh`
+Loads the complete vGPU stack in converged mode:
+1. `mdev` core module
+2. `nvidia.ko` (patched open-gpu-kernel-modules with NV_VGPU_KVM_BUILD)
+3. `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko`
+4. `nvidia-vgpu-vfio.ko` (open-iov kernel module)
+
+### `module_ops/unload_all.sh`
+Cleanly unloads all modules in reverse order with proper cleanup.
+
+### `module_ops/dry_run.sh`
+Runs nvidia-vgpu-vfio.ko in dry-run mode to validate firmware, BAR layout, IRQ caps, IOMMU without hardware changes.
+
+### `module_ops/reload_vgpu.sh`
+Reloads only the vGPU kernel module (nvidia-vgpu-vfio.ko) while keeping host nvidia.ko loaded.
+
+## Debug Tools
+
+### `debug/verify_vgpu_sysfs.sh`
+Verifies mdev parent registration and vGPU types in sysfs.
+
+### `debug/check_module_state.sh`
+Checks module initstate, refcnt, and holder information.
+
+### `debug/dump_vgpu_regs.sh`
+Dumps vGPU device registers via debugfs (if available).
+
+### `debug/capture_crash_logs.sh`
+Captures kernel logs, dmesg, and module state after a crash.
+
+## Test Scripts
+
+### `test/run_integration_tests.sh`
+Runs the integration test suite (host_init, guest_init).
+
+### `test/test_mdev_create.sh`
+Creates an mdev device and verifies basic functionality.
+
+### `test/benchmark_vgpu.sh`
+Runs performance benchmarks on vGPU.
+
+## VM Management
+
+### `vm/create_test_vm.sh`
+Creates a test VM with vGPU passthrough using QEMU.
+
+### `vm/launch_guest_driver.sh`
+Launches guest with NVIDIA driver installed for validation.
+
+## Firmware Tools
+
+### `firmware/verify_gsp_firmware.sh`
+Verifies GSP firmware files exist and are correct version.
+
+### `firmware/install_tu104_firmware.sh`
+Installs TU104 GSP firmware to /lib/firmware/nvidia/tu104/gsp/
+
+## Monitoring
+
+### `monitoring/watch_dmesg.sh`
+Real-time kernel log monitoring with filtering for vGPU/NVIDIA messages.
+
+### `monitoring/watch_module_params.sh`
+Monitors module parameters and sysfs attributes.
+
+## Environment Variables
+
+```bash
+export OPEN_IOV_ROOT=$OPEN_IOV_ROOT
+export NVIDIA_OPEN_KERNEL=$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open
+export VGPU_KERNEL=$OPEN_IOV_ROOT/kernel
+export VGPU_USERSPACE=$OPEN_IOV_ROOT/userspace
+```
+
+## Module Parameters for Debugging
+
+| Parameter | Description | Values |
+|-----------|-------------|--------|
+| `NVreg_RmMsg` | RM debug logging | `":"` (verbose), `"0"` (none) |
+| `NVreg_RegisterPCIDriver` | PCI driver registration | `0` (disable), `1` (enable) |
+| `NVreg_GpuInitOnProbe` | GPU init at probe | `0` (defer), `1` (immediate) |
+| `NVreg_PreserveVideoMemoryAllocations` | Video memory preservation | `0` (disable), `1` (enable), `2` (auto) |
+| `NVreg_UseKernelSuspendNotifiers` | Suspend notifiers | `0` (disable), `1` (enable) |
+| `dyndbg` | Dynamic debug | `'+p'` (all), `'+pf'` (probe+fail) |
+
+## Converged Mode Architecture
+
+```
+Host nvidia.ko (patched) nvidia-vgpu-vfio.ko (open-iov)
+├── mdev_register_parent() ←── mdev_register_driver() only
+├── EXPORT nvidia_vgpu_vfio_set_ops
+├── EXPORT nvidia_vgpu_vfio_get_ops
+└── Implements RM vGPU ops Calls RM via get_ops, provides VFIO ops via set_ops
+```
+
+## Troubleshooting
+
+### Module stuck in "going" state
+```bash
+cat /sys/module/nvidia-vgpu-vfio/initstate
+cat /sys/module/nvidia-vgpu-vfio/refcnt
+# Requires reboot to clear
+```
+
+### mdev parent not registered
+```bash
+ls /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/
+# Should show nvidia-vgpu-* directories with create/available_instances
+```
+
+### Symbol version mismatch
+```bash
+# Rebuild open-gpu-kernel-modules with VGX_KVM_BUILD=1
+cd $NVIDIA_OPEN_KERNEL && make clean && make -j$(nproc) VGX_KVM_BUILD=1
+```
+
+### IOMMU/VT-d not enabled
+```bash
+cat /proc/cmdline | grep -E "iommu|intel_iommu"
+# Add: intel_iommu=on iommu=pt to kernel cmdline
+``` \ No newline at end of file
diff --git a/tools/debug/nv2080_ctrl.c b/tools/debug/nv2080_ctrl.c
new file mode 100644
index 0000000..46ade02
--- /dev/null
+++ b/tools/debug/nv2080_ctrl.c
@@ -0,0 +1,531 @@
+/*
+ * NV2080 Control Command Sender
+ * C program to send NV2080 RM control commands via ioctl
+ * Compile: gcc -o nv2080_ctrl nv2080_ctrl.c
+ * Usage: sudo ./nv2080_ctrl /dev/nvidia0 <command> [args...]
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include <errno.h>
+
+// NV_IOCTL definitions from nv-ioctl.h
+#define NV_IOCTL_MAGIC 0x46
+#define NV_ESC_RM_CONTROL 0x54
+#define NV_ESC_CHECK_VERSION_STR 0x32
+#define NV_ESC_RM_ALLOC 0x21
+#define NV_ESC_RM_FREE 0x00
+#define NV_ESC_ALLOC_OS_EVENT 0x24
+
+// IOCTL macros
+#define _IOC_NRBITS 8
+#define _IOC_TYPEBITS 8
+#define _IOC_SIZEBITS 14
+#define _IOC_DIRBITS 2
+
+#define _IOC_NRSHIFT 0
+#define _IOC_TYPESHIFT (_IOC_NRSHIFT + _IOC_NRBITS)
+#define _IOC_SIZESHIFT (_IOC_TYPESHIFT + _IOC_TYPEBITS)
+#define _IOC_DIRSHIFT (_IOC_SIZESHIFT + _IOC_SIZEBITS)
+
+#define _IOC_NONE 0U
+#define _IOC_WRITE 1U
+#define _IOC_READ 2U
+
+#define _IOC(dir, type, nr, size) \
+ (((dir) << _IOC_DIRSHIFT) | \
+ ((type) << _IOC_TYPESHIFT) | \
+ ((nr) << _IOC_NRSHIFT) | \
+ ((size) << _IOC_SIZESHIFT))
+
+#define _IOWR(type, nr, size) _IOC(_IOC_READ | _IOC_WRITE, type, nr, size)
+
+// NV_CTRL_RES = _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, NVOS54_PARAMETERS)
+#define NV_CTRL_RES _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, 128)
+
+// NV2080 Control Commands (from ctrl2080unix.h and ctrl2080vgpumgrinternal.h)
+#define NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT 0x20803d01
+#define NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF 0x20803d02
+#define NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER 0x20803d03
+#define NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS 0x20803d04
+#define NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER 0x20803d05
+#define NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER 0x20803d06
+#define NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS 0x20803d07
+#define NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS 0x20803d08
+
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK 0x20804001
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK 0x20804002
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE 0x20804003
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU 0x20804004
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO 0x20804005
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE 0x20804006
+#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY 0x20804007
+#define NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP 0x20804008
+
+// Action values
+#define NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC 1
+#define NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC 2
+#define NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW 1
+#define NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW 2
+
+// NVOS54_PARAMETERS structure (from nv-ioctl.h)
+typedef struct {
+ uint32_t hClient;
+ uint32_t hObject;
+ uint32_t cmd;
+ uint32_t flags;
+ void *params;
+ uint32_t paramsSize;
+ uint32_t status;
+} NVOS54_PARAMETERS;
+
+// NVOS21_PARAMETERS for RM_ALLOC
+typedef struct {
+ uint32_t hRoot;
+ uint32_t hObjectParent;
+ uint32_t hObjectNew;
+ uint32_t hClass;
+ void *pAllocParms;
+ uint32_t status;
+} NVOS21_PARAMETERS;
+
+// NVOS00_PARAMETERS for RM_FREE
+typedef struct {
+ uint32_t hRoot;
+ uint32_t hObjectParent;
+ uint32_t hObjectOld;
+ uint32_t status;
+} NVOS00_PARAMETERS;
+
+// Version check structure
+typedef struct {
+ uint32_t cmd;
+ char versionString[256];
+ uint32_t reply;
+} nv_ioctl_rm_api_version_t;
+
+// OS Event allocation
+typedef struct {
+ uint32_t hClient;
+ uint32_t hDevice;
+ int fd;
+ uint32_t Status;
+} nv_ioctl_alloc_os_event_t;
+
+// GC6 Blocker Refcnt Params
+typedef struct {
+ uint32_t action;
+} NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS;
+
+// Allow/Disallow GCOFF Params
+typedef struct {
+ uint32_t action;
+} NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS;
+
+// Audio Dynamic Power Params
+typedef struct {
+ uint32_t bEnter;
+} NV2080_CTRL_OS_UNIX_AUDIO_DYNAMIC_POWER_PARAMS;
+
+// Vidmem Persistence Status Params
+typedef struct {
+ uint32_t bVidmemPersistent;
+} NV2080_CTRL_OS_UNIX_VIDMEM_PERSISTENCE_STATUS_PARAMS;
+
+// TGP Status Params
+typedef struct {
+ uint32_t bUpdateTGP;
+} NV2080_CTRL_OS_UNIX_UPDATE_TGP_STATUS_PARAMS;
+
+// VGPU Bootload Params (simplified)
+typedef struct {
+ uint32_t dbdf;
+ uint32_t gfid;
+ uint32_t vgpuType;
+ uint32_t vmPid;
+ uint32_t swizzId;
+ uint32_t numChannels;
+ uint32_t numPluginChannels;
+ uint32_t bDisableDefaultSmcExecPartRestore;
+ uint32_t numGuestFbSegments;
+ uint64_t guestFbPhysAddrList[384];
+ uint64_t guestFbLengthList[384];
+ uint64_t pluginHeapMemoryPhysAddr;
+ uint64_t pluginHeapMemoryLength;
+ uint64_t ctrlBuffOffset;
+} NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS;
+
+// VGPU Shutdown Params
+typedef struct {
+ uint32_t gfid;
+} NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS;
+
+// VGPU Add Type Params
+typedef struct {
+ uint32_t discardVgpuTypes;
+ uint64_t vgpuInfo; // NVA081_CTRL_VGPU_INFO
+} NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS;
+
+// VGPU Enumerate Params
+typedef struct {
+ uint32_t numVgpu;
+ uint64_t vgpuGuest[32]; // NV2080_VGPU_GUEST array
+} NV2080_CTRL_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU_PARAMS;
+
+// VGPU Clear Guest Params
+typedef struct {
+ uint32_t gfid;
+} NV2080_CTRL_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO_PARAMS;
+
+// VGPU FB Usage Params
+typedef struct {
+ uint32_t vgpuCount;
+ struct {
+ uint32_t gfid;
+ uint64_t fbUsed;
+ } vgpuFbUsage[32];
+} NV2080_CTRL_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE_PARAMS;
+
+// VGPU Encoder Capacity Params
+typedef struct {
+ uint32_t gfid;
+ uint32_t encoderCapacity;
+} NV2080_CTRL_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY_PARAMS;
+
+// VGPU Plugin Cleanup Params
+typedef struct {
+ uint32_t gfid;
+} NV2080_CTRL_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS;
+
+// Send RM Control command
+int send_rm_control(int fd, uint32_t client, uint32_t object, uint32_t cmd, void *params, uint32_t params_size) {
+ NVOS54_PARAMETERS ctrl = {
+ .hClient = client,
+ .hObject = object,
+ .cmd = cmd,
+ .flags = 0,
+ .params = params,
+ .paramsSize = params_size,
+ .status = 0
+ };
+
+ int ret = ioctl(fd, NV_CTRL_RES, &ctrl);
+ if (ret == -1) {
+ perror("ioctl NV_CTRL_RES failed");
+ return -1;
+ }
+
+ if (ctrl.status != 0) {
+ fprintf(stderr, "RM Control failed: cmd=0x%x, status=0x%x\n", cmd, ctrl.status);
+ return ctrl.status;
+ }
+
+ return 0;
+}
+
+// Version check
+int version_check(int fd, const char *version) {
+ nv_ioctl_rm_api_version_t vc = {0};
+ vc.cmd = 0x32; // NV_ESC_CHECK_VERSION_STR
+ if (version) {
+ strncpy(vc.versionString, version, sizeof(vc.versionString) - 1);
+ }
+
+ int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_CHECK_VERSION_STR, sizeof(vc)), &vc);
+ if (ret == -1) {
+ perror("ioctl version check failed");
+ return -1;
+ }
+
+ printf("Version check: reply=%u, version=%s\n", vc.reply, vc.versionString);
+ return vc.reply == 1 ? 0 : -1;
+}
+
+// Allocate RM resource
+int rm_alloc(int fd, uint32_t root, uint32_t parent, uint32_t object, uint32_t class, void *data) {
+ NVOS21_PARAMETERS alloc = {
+ .hRoot = root,
+ .hObjectParent = parent,
+ .hObjectNew = object,
+ .hClass = class,
+ .pAllocParms = data,
+ .status = 0
+ };
+
+ int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_ALLOC, sizeof(alloc)), &alloc);
+ if (ret == -1) {
+ perror("ioctl RM_ALLOC failed");
+ return -1;
+ }
+
+ if (alloc.status != 0) {
+ fprintf(stderr, "RM_ALLOC failed: status=0x%x\n", alloc.status);
+ return alloc.status;
+ }
+
+ printf("Allocated object 0x%x (class 0x%x)\n", alloc.hObjectNew, class);
+ return alloc.hObjectNew;
+}
+
+// Free RM resource
+int rm_free(int fd, uint32_t root, uint32_t parent, uint32_t object) {
+ NVOS00_PARAMETERS free_params = {
+ .hRoot = root,
+ .hObjectParent = parent,
+ .hObjectOld = object,
+ .status = 0
+ };
+
+ int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_FREE, sizeof(free_params)), &free_params);
+ if (ret == -1) {
+ perror("ioctl RM_FREE failed");
+ return -1;
+ }
+
+ return free_params.status == 0 ? 0 : free_params.status;
+}
+
+// Print usage
+void print_usage(const char *prog) {
+ fprintf(stderr, "Usage: %s <device> <command> [args...]\n", prog);
+ fprintf(stderr, "\nDevice: /dev/nvidia0, /dev/nvidiactl, etc.\n");
+ fprintf(stderr, "\nCommands:\n");
+ fprintf(stderr, " version-check [version] - Check RM API version\n");
+ fprintf(stderr, " gc6-inc <client> <object> - Increment GC6 blocker refcount\n");
+ fprintf(stderr, " gc6-dec <client> <object> - Decrement GC6 blocker refcount\n");
+ fprintf(stderr, " gcoff-allow <client> <object> - Allow GCOFF\n");
+ fprintf(stderr, " gcoff-disallow <client> <object> - Disallow GCOFF\n");
+ fprintf(stderr, " audio-power <client> <object> <enter> - Audio dynamic power (1=enter, 0=exit)\n");
+ fprintf(stderr, " vidmem-persistence <client> <object> - Query vidmem persistence\n");
+ fprintf(stderr, " tgp-status <client> <object> <update> - Update TGP status (1=update, 0=don't)\n");
+ fprintf(stderr, " vgpu-bootload <client> <object> <dbdf> <gfid> <vgpuType> <vmPid> ... - Bootload GSP VGPU plugin\n");
+ fprintf(stderr, " vgpu-shutdown <client> <object> <gfid> - Shutdown GSP VGPU plugin\n");
+ fprintf(stderr, " vgpu-add-type <client> <object> <discard> <vgpuInfo> - Add vGPU type\n");
+ fprintf(stderr, " vgpu-enumerate <client> <object> - Enumerate vGPUs per pGPU\n");
+ fprintf(stderr, " vgpu-clear-guest <client> <object> <gfid> - Clear guest VM info\n");
+ fprintf(stderr, " vgpu-fb-usage <client> <object> - Get vGPU FB usage\n");
+ fprintf(stderr, " vgpu-encoder-cap <client> <object> <gfid> <capacity> - Set encoder capacity (0-100)\n");
+ fprintf(stderr, " vgpu-plugin-cleanup <client> <object> <gfid> - Cleanup VGPU plugin\n");
+ fprintf(stderr, " alloc <root> <parent> <object> <class> - Allocate RM resource\n");
+ fprintf(stderr, " free <root> <parent> <object> - Free RM resource\n");
+}
+
+int main(int argc, char *argv[]) {
+ if (argc < 3) {
+ print_usage(argv[0]);
+ return 1;
+ }
+
+ const char *device = argv[1];
+ const char *command = argv[2];
+
+ int fd = open(device, O_RDWR);
+ if (fd < 0) {
+ perror("Failed to open device");
+ return 1;
+ }
+
+ int ret = 0;
+
+ if (strcmp(command, "version-check") == 0) {
+ const char *version = argc > 3 ? argv[3] : NULL;
+ ret = version_check(fd, version);
+ }
+ else if (strcmp(command, "gc6-inc") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "gc6-dec") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "gcoff-allow") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "gcoff-disallow") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "audio-power") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t enter = strtoul(argv[5], NULL, 0);
+ NV2080_CTRL_OS_UNIX_AUDIO_DYNAMIC_POWER_PARAMS params = { .bEnter = enter };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vidmem-persistence") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_OS_UNIX_VIDMEM_PERSISTENCE_STATUS_PARAMS params = {0};
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS, &params, sizeof(params));
+ if (ret == 0) {
+ printf("Vidmem persistent: %s\n", params.bVidmemPersistent ? "yes" : "no");
+ }
+ }
+ }
+ else if (strcmp(command, "tgp-status") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t update = strtoul(argv[5], NULL, 0);
+ NV2080_CTRL_OS_UNIX_UPDATE_TGP_STATUS_PARAMS params = { .bUpdateTGP = update };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-bootload") == 0) {
+ if (argc < 12) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS params = {0};
+ params.dbdf = strtoul(argv[5], NULL, 0);
+ params.gfid = strtoul(argv[6], NULL, 0);
+ params.vgpuType = strtoul(argv[7], NULL, 0);
+ params.vmPid = strtoul(argv[8], NULL, 0);
+ params.swizzId = strtoul(argv[9], NULL, 0);
+ params.numChannels = strtoul(argv[10], NULL, 0);
+ params.numPluginChannels = strtoul(argv[11], NULL, 0);
+ // Additional params would need more args
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-shutdown") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t gfid = strtoul(argv[5], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS params = { .gfid = gfid };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-add-type") == 0) {
+ if (argc < 7) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t discard = strtoul(argv[5], NULL, 0);
+ uint64_t vgpuInfo = strtoull(argv[6], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS params = { .discardVgpuTypes = discard, .vgpuInfo = vgpuInfo };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-enumerate") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU_PARAMS params = {0};
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU, &params, sizeof(params));
+ if (ret == 0) {
+ printf("Enumerated %u vGPUs\n", params.numVgpu);
+ }
+ }
+ }
+ else if (strcmp(command, "vgpu-clear-guest") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t gfid = strtoul(argv[5], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO_PARAMS params = { .gfid = gfid };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-fb-usage") == 0) {
+ if (argc < 5) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE_PARAMS params = {0};
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE, &params, sizeof(params));
+ if (ret == 0) {
+ printf("vGPU count: %u\n", params.vgpuCount);
+ for (uint32_t i = 0; i < params.vgpuCount; i++) {
+ printf(" GFID 0x%x: FB used %lu bytes\n", params.vgpuFbUsage[i].gfid, params.vgpuFbUsage[i].fbUsed);
+ }
+ }
+ }
+ }
+ else if (strcmp(command, "vgpu-encoder-cap") == 0) {
+ if (argc < 7) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t gfid = strtoul(argv[5], NULL, 0);
+ uint32_t capacity = strtoul(argv[6], NULL, 0);
+ NV2080_CTRL_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY_PARAMS params = { .gfid = gfid, .encoderCapacity = capacity };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "vgpu-plugin-cleanup") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t client = strtoul(argv[3], NULL, 0);
+ uint32_t object = strtoul(argv[4], NULL, 0);
+ uint32_t gfid = strtoul(argv[5], NULL, 0);
+ NV2080_CTRL_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS params = { .gfid = gfid };
+ ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP, &params, sizeof(params));
+ }
+ }
+ else if (strcmp(command, "alloc") == 0) {
+ if (argc < 7) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t root = strtoul(argv[3], NULL, 0);
+ uint32_t parent = strtoul(argv[4], NULL, 0);
+ uint32_t object = strtoul(argv[5], NULL, 0);
+ uint32_t class = strtoul(argv[6], NULL, 0);
+ ret = rm_alloc(fd, root, parent, object, class, NULL);
+ }
+ }
+ else if (strcmp(command, "free") == 0) {
+ if (argc < 6) { print_usage(argv[0]); ret = 1; }
+ else {
+ uint32_t root = strtoul(argv[3], NULL, 0);
+ uint32_t parent = strtoul(argv[4], NULL, 0);
+ uint32_t object = strtoul(argv[5], NULL, 0);
+ ret = rm_free(fd, root, parent, object);
+ }
+ }
+ else {
+ fprintf(stderr, "Unknown command: %s\n", command);
+ print_usage(argv[0]);
+ ret = 1;
+ }
+
+ close(fd);
+ return ret;
+} \ No newline at end of file
diff --git a/tools/debug/nv2080_ctrl.sh b/tools/debug/nv2080_ctrl.sh
new file mode 100755
index 0000000..22c7117
--- /dev/null
+++ b/tools/debug/nv2080_ctrl.sh
@@ -0,0 +1,168 @@
+#!/bin/bash
+# NV2080 Control Command Tool
+# Sends NV2080 RM control commands to the NVIDIA kernel module via ioctl
+# Usage: ./nv2080_ctrl.sh <device_fd> <command> [params...]
+
+set -euo pipefail
+
+# NV2080 Control Command Definitions (from ctrl2080unix.h and ctrl2080vgpumgrinternal.h)
+# These are the same commands used by GVM userspace via NV_CTRL_RES ioctl
+
+# NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT (0x20803d01)
+# Params: action (INC=1, DEC=2)
+NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT=0x20803d01
+NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC=1
+NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC=2
+
+# NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF (0x20803d02)
+# Params: action (ALLOW=1, DISALLOW=2)
+NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF=0x20803d02
+NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW=1
+NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW=2
+
+# NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER (0x20803d03)
+# Params: bEnter (true/false)
+NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER=0x20803d03
+
+# NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS (0x20803d04)
+NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS=0x20803d04
+
+# NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER (0x20803d05)
+NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER=0x20803d05
+
+# NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER (0x20803d06)
+NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER=0x20803d06
+
+# NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS (0x20803d07)
+NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS=0x20803d07
+
+# NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS (0x20803d08)
+NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS=0x20803d08
+
+# VGPU Manager Internal Commands
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK=0x20804001
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK=0x20804002
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE=0x20804003
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU=0x20804004
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO=0x20804005
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE=0x20804006
+NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY=0x20804007
+NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP=0x20804008
+
+# NV_IOCTL_MAGIC and NV_ESC_RM_CONTROL
+NV_IOCTL_MAGIC=0x46
+NV_ESC_RM_CONTROL=0x54
+
+# NV_CTRL_RES = _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, NVOS54_PARAMETERS)
+NV_CTRL_RES=$(printf "0x%x" $(( (2 << 30) | (NV_IOCTL_MAGIC << 8) | (NV_ESC_RM_CONTROL) | (128 << 16) )))
+
+# Helper to send ioctl
+send_ioctl() {
+ local fd=$1
+ local cmd=$2
+ local data=$3
+ local size=$4
+
+ # This would need a C program to actually send the ioctl
+ # For now, we document the commands
+ echo "Command: 0x$(printf "%x" $cmd)"
+ echo "Data size: $size"
+}
+
+# Main command dispatcher
+case "${1:-help}" in
+ gc6-inc)
+ echo "Increment GC6 blocker refcount"
+ echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, action: $NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC})"
+ ;;
+ gc6-dec)
+ echo "Decrement GC6 blocker refcount"
+ echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, action: $NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC})"
+ ;;
+ gcoff-allow)
+ echo "Allow GCOFF"
+ echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, action: $NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW})"
+ ;;
+ gcoff-disallow)
+ echo "Disallow GCOFF"
+ echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, action: $NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW})"
+ ;;
+ audio-power)
+ echo "Audio dynamic power: $2"
+ ;;
+ profiler-install)
+ echo "Install profiler hooks"
+ ;;
+ profiler-flush)
+ echo "Flush snapshot buffer"
+ ;;
+ profiler-stop)
+ echo "Stop profiler"
+ ;;
+ vidmem-persistence)
+ echo "Query vidmem persistence status"
+ ;;
+ tgp-status)
+ echo "Update TGP status: $2"
+ ;;
+ vgpu-bootload)
+ echo "Bootload GSP VGPU plugin task"
+ echo "Params: dbdf, gfid, vgpuType, vmPid, swizzId, numChannels, numPluginChannels, bDisableSmcPartitionRestore, guestFbPhysAddrList, guestFbLengthList, pluginHeapMemoryPhysAddr, pluginHeapMemoryLength, ctrlBuffOffset"
+ ;;
+ vgpu-shutdown)
+ echo "Shutdown GSP VGPU plugin task"
+ echo "Params: gfid"
+ ;;
+ vgpu-add-type)
+ echo "Add vGPU type to pGPU"
+ echo "Params: discardVgpuTypes, vgpuInfo"
+ ;;
+ vgpu-enumerate)
+ echo "Enumerate vGPUs per pGPU"
+ ;;
+ vgpu-clear-guest)
+ echo "Clear guest VM info"
+ echo "Params: gfid"
+ ;;
+ vgpu-fb-usage)
+ echo "Get vGPU FB usage"
+ ;;
+ vgpu-encoder-cap)
+ echo "Set vGPU encoder capacity"
+ echo "Params: gfid, encoderCapacity (0-100)"
+ ;;
+ vgpu-plugin-cleanup)
+ echo "Cleanup VGPU plugin"
+ echo "Params: gfid"
+ ;;
+ list)
+ echo "Available NV2080 Control Commands:"
+ echo " OS Unix Commands:"
+ echo " gc6-inc - Increment GC6 blocker refcount"
+ echo " gc6-dec - Decrement GC6 blocker refcount"
+ echo " gcoff-allow - Allow GCOFF"
+ echo " gcoff-disallow - Disallow GCOFF"
+ echo " audio-power - Audio dynamic power (enter/exit)"
+ echo " profiler-install - Install profiler hooks (HWPM)"
+ echo " profiler-flush - Flush snapshot buffer"
+ echo " profiler-stop - Stop profiler"
+ echo " vidmem-persistence- Query vidmem persistence status"
+ echo " tgp-status - Update TGP status"
+ echo ""
+ echo " VGPU Manager Internal Commands:"
+ echo " vgpu-bootload - Bootload GSP VGPU plugin task"
+ echo " vgpu-shutdown - Shutdown GSP VGPU plugin task"
+ echo " vgpu-add-type - Add vGPU type to pGPU"
+ echo " vgpu-enumerate - Enumerate vGPUs per pGPU"
+ echo " vgpu-clear-guest - Clear guest VM info"
+ echo " vgpu-fb-usage - Get vGPU FB usage"
+ echo " vgpu-encoder-cap - Set vGPU encoder capacity"
+ echo " vgpu-plugin-cleanup - Cleanup VGPU plugin"
+ ;;
+ help|*)
+ echo "NV2080 Control Command Tool"
+ echo "Usage: $0 <command> [args...]"
+ echo ""
+ echo "Run '$0 list' to see all available commands"
+ ;;
+esac \ No newline at end of file
diff --git a/tools/firmware/verify_gsp_firmware.sh b/tools/firmware/verify_gsp_firmware.sh
new file mode 100755
index 0000000..79f474c
--- /dev/null
+++ b/tools/firmware/verify_gsp_firmware.sh
@@ -0,0 +1,60 @@
+#!/bin/bash
+# Verify GSP firmware for TU104
+# Usage: ./verify_gsp_firmware.sh
+
+set -euo pipefail
+
+FIRMWARE_DIR="/lib/firmware/nvidia/tu104/gsp"
+
+echo "=== GSP Firmware Verification (TU104) ==="
+echo ""
+
+if [[ ! -d "$FIRMWARE_DIR" ]]; then
+ echo "ERROR: Firmware directory not found: $FIRMWARE_DIR"
+ echo "Install firmware with: rzsh nvidia vgpu firmware install"
+ exit 1
+fi
+
+echo "Firmware directory: $FIRMWARE_DIR"
+echo ""
+
+# Check required files
+required_files=(
+ "gsp-535.113.01.bin"
+ "gsp-570.144.bin"
+ "booter_load-535.113.01.bin"
+ "booter_load-570.144.bin"
+ "booter_unload-535.113.01.bin"
+ "booter_unload-570.144.bin"
+ "bootloader-535.113.01.bin"
+ "bootloader-570.144.bin"
+)
+
+all_ok=true
+for file in "${required_files[@]}"; do
+ if [[ -f "$FIRMWARE_DIR/$file" ]]; then
+ size=$(stat -c%s "$FIRMWARE_DIR/$file")
+ echo " ✓ $file ($size bytes)"
+ else
+ echo " ✗ $file MISSING"
+ all_ok=false
+ fi
+done
+
+echo ""
+if $all_ok; then
+ echo "✓ All required firmware files present"
+else
+ echo "✗ Some firmware files missing"
+ echo ""
+ echo "To install firmware:"
+ echo " 1. Download from NVIDIA or copy from nvidia-linux-firmware repo"
+ echo " 2. sudo cp -r nvidia/tu104/gsp/* /lib/firmware/nvidia/tu104/gsp/"
+ echo " 3. sudo update-initramfs -u"
+fi
+
+echo ""
+echo "Current kernel module version:"
+if lsmod | grep -q nvidia; then
+ cat /proc/driver/nvidia/version 2>/dev/null | head -1
+fi \ No newline at end of file
diff --git a/tools/ghidra/analyze_fwimage.py b/tools/ghidra/analyze_fwimage.py
new file mode 100644
index 0000000..ea0f87f
--- /dev/null
+++ b/tools/ghidra/analyze_fwimage.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:557f20be1c03606549eacab7c7e524abd30cc1826e9cac3c583da51206520447
+size 1886
diff --git a/tools/ghidra/analyze_gsp_firmware.sh b/tools/ghidra/analyze_gsp_firmware.sh
new file mode 100755
index 0000000..911e0d2
--- /dev/null
+++ b/tools/ghidra/analyze_gsp_firmware.sh
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d4f65bee3ce0c98e749ab318318dabea3d2268eeec30c1c4c4e5c7e758487bb
+size 3960
diff --git a/tools/ghidra/create_gsp_project.py b/tools/ghidra/create_gsp_project.py
new file mode 100644
index 0000000..454a3e3
--- /dev/null
+++ b/tools/ghidra/create_gsp_project.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2cb538468bd58ebec94432eb58ae79a478e75f8a508f177192533f648dc007c
+size 1916
diff --git a/tools/ghidra/diff_gsp_firmware.py b/tools/ghidra/diff_gsp_firmware.py
new file mode 100644
index 0000000..ea804ec
--- /dev/null
+++ b/tools/ghidra/diff_gsp_firmware.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44976b0275737468689f718b96037540699defc010019793a8e58bf509ddf2a3
+size 7816
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr
diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java b/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java
new file mode 100644
index 0000000..ef338af
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8beabb120fb5e74202ca22087b2c4baa8fc81834c120d93a673b5fd74230f12b
+size 6940
diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java b/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java
new file mode 100644
index 0000000..4b8e165
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaca1bea9dd6d70970d9b629b2d5667614407992f2a85b92157c27d0c18c7908
+size 3354
diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java b/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java
new file mode 100644
index 0000000..aadc1db
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ceda696aad0ec35e26e1fcc02ed6aa3c7ad32faf2eff25cc390620b377776b67
+size 5187
diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py b/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py
new file mode 100644
index 0000000..ebb1e63
--- /dev/null
+++ b/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d35f85f13506e8491bced83697cf44a0708f03e413280831a3f584c829f5cd1e
+size 3869
diff --git a/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr
diff --git a/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~
diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java b/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java
new file mode 100644
index 0000000..1be91a5
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ff08f6647f5f1d1c678344223226fd00ca605bca903a5fdb44e39ad3aa982a0
+size 5725
diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py b/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py
new file mode 100644
index 0000000..431a813
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86e458aa7968cfc1362837ba0278af410c0dbe70bdec97bce413225f98640bd0
+size 7025
diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py b/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py
new file mode 100644
index 0000000..ebb1e63
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d35f85f13506e8491bced83697cf44a0708f03e413280831a3f584c829f5cd1e
+size 3869
diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py b/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py
new file mode 100644
index 0000000..fb828c0
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39970014d9ba8f37b9fec29643ca2b2194dbd9f6e5288c20ce6191c6a8e98cc6
+size 1923
diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py b/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py
new file mode 100644
index 0000000..8d02ac1
--- /dev/null
+++ b/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae3f536d9a82c9cd0e829d28ae9ebe4a3142004ba0ac3c8ca76438a69e5a32ce
+size 1062
diff --git a/tools/module_ops/dry_run.sh b/tools/module_ops/dry_run.sh
new file mode 100755
index 0000000..8bda820
--- /dev/null
+++ b/tools/module_ops/dry_run.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+# Run vGPU dry-run validation
+# Usage: sudo ./dry_run.sh
+
+set -euo pipefail
+
+VGPU_KERNEL="$OPEN_IOV_ROOT/kernel"
+
+echo "=== Open-IOV vGPU Dry-Run Validation ==="
+echo ""
+
+# Check if host nvidia.ko is loaded
+if ! lsmod | grep -q "^nvidia "; then
+ echo "ERROR: Host nvidia.ko not loaded. Load it first with:"
+ echo " rzsh nvidia vgpu load"
+ exit 1
+fi
+
+echo "Host nvidia.ko: LOADED ✓"
+echo ""
+
+# Load vGPU module in dry-run mode
+echo "Loading nvidia-vgpu-vfio.ko in dry-run mode..."
+sudo insmod "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" dyndbg='+p' dry_run=1
+
+# Check result
+sleep 2
+if dmesg -T | grep -q "DRY-RUN MODE: validating init without hardware changes"; then
+ echo ""
+ echo "✓ Dry-run validation PASSED"
+ echo " Firmware, BAR layout, IRQ caps, IOMMU validated"
+else
+ echo ""
+ echo "✗ Dry-run validation FAILED"
+ echo "Check dmesg for errors:"
+ dmesg -T | grep -i -e nvidia-vgpu -e vgpu | tail -20
+fi
+
+# Unload
+echo ""
+echo "Unloading dry-run module..."
+sudo rmmod nvidia-vgpu-vfio
+
+echo ""
+echo "=== Dry-Run Complete ==="
+echo "If validation passed, load for real with:"
+echo " sudo insmod $VGPU_KERNEL/nvidia-vgpu-vfio.ko dyndbg='+p'" \ No newline at end of file
diff --git a/tools/module_ops/load_converged.sh b/tools/module_ops/load_converged.sh
new file mode 100755
index 0000000..77cde09
--- /dev/null
+++ b/tools/module_ops/load_converged.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+# Load vGPU stack in converged mode
+# Usage: sudo ./load_converged.sh [dry_run]
+
+set -euo pipefail
+
+OPEN_IOV_ROOT="$OPEN_IOV_ROOT"
+NVIDIA_OPEN_KERNEL="$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open"
+VGPU_KERNEL="$OPEN_IOV_ROOT/kernel"
+
+DRY_RUN="${1:-false}"
+
+echo "=== Open-IOV Converged Mode Module Load ==="
+echo "Open-IOV Root: $OPEN_IOV_ROOT"
+echo "Dry-run mode: $DRY_RUN"
+echo ""
+
+# Function to check if module is loaded
+is_loaded() {
+ lsmod | grep -q "^$1 "
+}
+
+# Function to load module with error checking
+load_module() {
+ local module_path=$1
+ local module_name=$(basename $module_path .ko)
+ local params="${2:-}"
+
+ if is_loaded "$module_name"; then
+ echo "Module $module_name already loaded"
+ return 0
+ fi
+
+ echo "Loading $module_name..."
+ if sudo insmod "$module_path" $params; then
+ echo " ✓ $module_name loaded successfully"
+ return 0
+ else
+ echo " ✗ Failed to load $module_name"
+ return 1
+ fi
+}
+
+# 1. Load mdev core
+echo "Step 1: Loading mdev core module..."
+sudo modprobe mdev
+echo " ✓ mdev loaded"
+echo ""
+
+# 2. Load host nvidia.ko with vGPU support
+echo "Step 2: Loading host nvidia.ko (patched open-gpu-kernel-modules)..."
+load_module "$NVIDIA_OPEN_KERNEL/nvidia.ko" "NVreg_RegisterPCIDriver=1 NVreg_RmMsg=:"
+echo ""
+
+# 3. Load dependent modules
+echo "Step 3: Loading dependent modules..."
+load_module "$NVIDIA_OPEN_KERNEL/nvidia-modeset.ko"
+load_module "$NVIDIA_OPEN_KERNEL/nvidia-uvm.ko"
+load_module "$NVIDIA_OPEN_KERNEL/nvidia-drm.ko"
+load_module "$NVIDIA_OPEN_KERNEL/nvidia-peermem.ko"
+echo ""
+
+# 4. Load nvidia-vgpu-vfio.ko
+echo "Step 4: Loading nvidia-vgpu-vfio.ko..."
+if [ "$DRY_RUN" = "dry_run" ] || [ "$DRY_RUN" = "true" ]; then
+ echo " Running in DRY-RUN mode first..."
+ load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p' dry_run=1"
+ echo " Dry-run successful, unloading..."
+ sudo rmmod nvidia-vgpu-vfio
+ echo " Loading for real..."
+ load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p'"
+else
+ load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p'"
+fi
+echo ""
+
+# 5. Verify
+echo "Step 5: Verifying vGPU registration..."
+sleep 2
+if [ -d "/sys/bus/pci/devices/0000:01:00.0/mdev_supported_types" ]; then
+ echo " ✓ mdev_supported_types directory exists"
+ ls -la /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/
+else
+ echo " ✗ mdev_supported_types NOT found"
+ echo " Check dmesg for errors: dmesg -T | grep -i vgpu"
+ exit 1
+fi
+
+echo ""
+echo "=== Converged Mode Load Complete ==="
+echo ""
+echo "Next steps:"
+echo " - Create mdev device: echo <uuid> | sudo tee /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/nvidia-vgpu-turing/create"
+echo " - Run QEMU with vGPU passthrough"
+echo " - Run tests: $OPEN_IOV_ROOT/tools/test/run_integration_tests.sh" \ No newline at end of file
diff --git a/tools/module_ops/unload_all.sh b/tools/module_ops/unload_all.sh
new file mode 100755
index 0000000..4c73a0b
--- /dev/null
+++ b/tools/module_ops/unload_all.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+# Unload all vGPU modules cleanly
+# Usage: sudo ./unload_all.sh
+
+set -euo pipefail
+
+echo "=== Unloading Open-IOV vGPU Stack ==="
+
+# Unload in reverse order of loading
+modules=(
+ "nvidia-vgpu-vfio"
+ "nvidia-drm"
+ "nvidia-modeset"
+ "nvidia-uvm"
+ "nvidia-peermem"
+ "nvidia"
+ "mdev"
+)
+
+for module in "${modules[@]}"; do
+ if lsmod | grep -q "^$module "; then
+ echo "Unloading $module..."
+ if sudo rmmod "$module" 2>/dev/null; then
+ echo " ✓ $module unloaded"
+ else
+ echo " ⚠ $module could not be unloaded (may be in use)"
+ # Check refcnt
+ if [[ -f /sys/module/$module/refcnt ]]; then
+ echo " refcnt: $(cat /sys/module/$module/refcnt)"
+ fi
+ if [[ -f /sys/module/$module/initstate ]]; then
+ echo " initstate: $(cat /sys/module/$module/initstate)"
+ fi
+ fi
+ else
+ echo " - $module not loaded"
+ fi
+done
+
+echo ""
+echo "=== Unload Complete ==="
+lsmod | grep -E 'nvidia|mdev' || echo "No NVIDIA/mdev modules loaded" \ No newline at end of file
diff --git a/tools/test/run_integration_tests.sh b/tools/test/run_integration_tests.sh
new file mode 100755
index 0000000..06e309c
--- /dev/null
+++ b/tools/test/run_integration_tests.sh
@@ -0,0 +1,182 @@
+#!/bin/bash
+# GSP vGPU-IO Integration Test Runner
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+KERNEL_TEST_DIR="$PROJECT_ROOT/kernel/tests"
+DRIVER_DIR="$PROJECT_ROOT/driver"
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
+log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
+log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
+
+# Check if running as root
+check_root() {
+ if [[ $EUID -ne 0 ]]; then
+ log_error "This script must be run as root"
+ exit 1
+ fi
+}
+
+# Build kernel test module
+build_test_module() {
+ log_info "Building kernel test module..."
+ cd "$KERNEL_TEST_DIR"
+ make clean
+ make
+ if [[ $? -ne 0 ]]; then
+ log_error "Failed to build test module"
+ exit 1
+ fi
+ log_info "Test module built successfully"
+}
+
+# Load test module and capture output
+run_test_module() {
+ log_info "Loading test module..."
+ cd "$KERNEL_TEST_DIR"
+
+ # Clear dmesg
+ dmesg -C
+
+ # Load module
+ insmod gsp_vgpu_comprehensive_test.ko 2>&1 || {
+ log_error "Failed to load test module"
+ dmesg -T | tail -30
+ return 1
+ }
+
+ # Wait for tests to complete
+ sleep 2
+
+ # Get test results from dmesg
+ dmesg -T | grep -E "(PASS|FAIL|Test Results|TEST SUITE)" | tail -100
+
+ # Check if tests passed
+ if dmesg | grep -q "TEST SUITE PASSED"; then
+ log_info "All tests PASSED!"
+ return 0
+ elif dmesg | grep -q "TEST SUITE FAILED"; then
+ log_error "Tests FAILED!"
+ return 1
+ else
+ log_warn "Could not determine test result"
+ return 1
+ fi
+}
+
+# Unload test module
+unload_test_module() {
+ log_info "Unloading test module..."
+ rmmod gsp_vgpu_comprehensive_test 2>/dev/null || true
+ sleep 1
+}
+
+# Build driver
+build_driver() {
+ log_info "Building vGPU-IO driver..."
+ cd "$DRIVER_DIR"
+ make clean
+ make
+ if [[ $? -ne 0 ]]; then
+ log_error "Failed to build driver"
+ exit 1
+ fi
+ log_info "Driver built successfully"
+}
+
+# Load driver
+load_driver() {
+ log_info "Loading vGPU-IO driver..."
+ cd "$DRIVER_DIR"
+ insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin 2>&1 || {
+ log_error "Failed to load driver"
+ dmesg -T | tail -30
+ return 1
+ }
+
+ # Check if device created
+ sleep 2
+ if ls /dev/gsp-vgpu* 2>/dev/null; then
+ log_info "Driver loaded, device created"
+ return 0
+ else
+ log_warn "Driver loaded but no device node found"
+ dmesg -T | tail -20
+ fi
+}
+
+# Unload driver
+unload_driver() {
+ log_info "Unloading vGPU-IO driver..."
+ rmmod gsp-vgpu 2>/dev/null || true
+ sleep 1
+}
+
+# Run VF creation test
+test_vf_creation() {
+ log_info "Testing VF creation via debugfs..."
+
+ # Check debugfs
+ if [[ -d /sys/kernel/debug/gsp-vgpu ]]; then
+ log_info "Debugfs interface available"
+ cat /sys/kernel/debug/gsp-vgpu/vfs
+ else
+ log_warn "Debugfs not mounted or driver not loaded"
+ return 1
+ fi
+}
+
+# Run RPC test via debugfs
+test_rpc_stats() {
+ log_info "Checking RPC statistics..."
+ if [[ -f /sys/kernel/debug/gsp-vgpu/rpc ]]; then
+ cat /sys/kernel/debug/gsp-vgpu/rpc
+ fi
+}
+
+# Main test runner
+main() {
+ local test_type="${1:-all}"
+
+ check_root
+
+ case "$test_type" in
+ "unit")
+ build_test_module
+ run_test_module
+ unload_test_module
+ ;;
+ "driver")
+ build_driver
+ load_driver
+ test_vf_creation
+ test_rpc_stats
+ unload_driver
+ ;;
+ "all"|*)
+ log_info "Running ALL tests..."
+ build_test_module
+ run_test_module
+ unload_test_module
+
+ build_driver
+ load_driver
+ test_vf_creation
+ test_rpc_stats
+ unload_driver
+ ;;
+ esac
+
+ log_info "Test run completed"
+}
+
+main "$@" \ No newline at end of file
diff --git a/tools/vm/cloud-init.yaml b/tools/vm/cloud-init.yaml
new file mode 100644
index 0000000..05f11b5
--- /dev/null
+++ b/tools/vm/cloud-init.yaml
@@ -0,0 +1,26 @@
+#cloud-config
+users:
+ - name: ubuntu
+ sudo: ALL=(ALL) NOPASSWD:ALL
+ shell: /bin/bash
+ lock_passwd: false
+ passwd: $6$rounds=4096$ubuntu$ubuntu
+ ssh_authorized_keys: []
+
+package_update: true
+package_upgrade: true
+packages:
+ - linux-headers-generic
+ - build-essential
+ - dkms
+ - git
+ - wget
+ - curl
+ - vim
+ - pciutils
+ - lshw
+
+runcmd:
+ - echo "ubuntu:ubuntu" | chpasswd
+ - systemctl enable ssh
+ - systemctl start ssh
diff --git a/tools/vm/launch_vgpu_qemu.sh b/tools/vm/launch_vgpu_qemu.sh
new file mode 100755
index 0000000..0355ef1
--- /dev/null
+++ b/tools/vm/launch_vgpu_qemu.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# QEMU launch script for NVIDIA vGPU testing
+# This script assumes:
+# 1. nvidia-vgpu-vfio.ko is loaded on host (with dry_run=1 for testing)
+# 2. An mdev device has been created via sysfs
+# 3. Guest OS has NVIDIA vGPU guest driver installed
+
+set -euo pipefail
+
+# Configuration
+MDEV_UUID="${MDEV_UUID:-$(cat /proc/sys/kernel/random/uuid)}"
+VGPU_TYPE="${VGPU_TYPE:-nvidia-vgpu-turing}" # or ampere, hopper, pascal
+QEMU_BIN="${QEMU_BIN:-qemu-system-x86_64}"
+GUEST_IMG="${GUEST_IMG:-/var/lib/libvirt/images/ubuntu-vgpu.qcow2}"
+GUEST_KERNEL="${GUEST_KERNEL:-/boot/vmlinuz-$(uname -r)}"
+GUEST_INITRD="${GUEST_INITRD:-/boot/initrd.img-$(uname -r)}"
+
+# PCI address of the NVIDIA GPU (TU104: 10de:1ed3)
+GPU_PCI_ADDR="${GPU_PCI_ADDR:-0000:01:00.0}"
+
+echo "=== NVIDIA vGPU QEMU Launch Script ==="
+echo "MDEV UUID: $MDEV_UUID"
+echo "VGPU Type: $VGPU_TYPE"
+echo "GPU PCI: $GPU_PCI_ADDR"
+echo ""
+
+# Check if mdev device exists
+MDEV_PATH="/sys/bus/mdev/devices/$MDEV_UUID"
+if [[ ! -d "$MDEV_PATH" ]]; then
+ echo "ERROR: mdev device $MDEV_UUID not found"
+ echo ""
+ echo "To create an mdev device (requires NVIDIA vGPU host driver):"
+ echo " echo $MDEV_UUID > /sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE/create"
+ echo ""
+ echo "For dry-run testing without host driver, this script will not work for actual QEMU launch."
+ echo "The kernel module loads in dry-run mode but cannot create real mdev devices."
+ exit 1
+fi
+
+# Get the mdev device's VFIO group
+VFIO_GROUP=$(basename $(readlink -f "$MDEV_PATH/iommu_group"))
+VFIO_DEV="/dev/vfio/$VFIO_GROUP"
+
+if [[ ! -e "$VFIO_DEV" ]]; then
+ echo "ERROR: VFIO device $VFIO_DEV not found"
+ exit 1
+fi
+
+echo "Using VFIO device: $VFIO_DEV"
+echo ""
+
+# QEMU command
+exec $QEMU_BIN \
+ -name "vgpu-guest" \
+ -enable-kvm \
+ -cpu host,host-passthrough=on \
+ -smp 4,sockets=1,cores=4,threads=1 \
+ -m 8G \
+ -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$MDEV_UUID,display=on,x-igd-opregion=on \
+ -device virtio-net-pci,netdev=net0 \
+ -netdev user,id=net0,hostfwd=tcp::2222-:22 \
+ -drive file=$GUEST_IMG,if=virtio,format=qcow2 \
+ -display gtk,gl=on \
+ -vga none \
+ "$@"
diff --git a/tools/vm/setup_mdev.sh b/tools/vm/setup_mdev.sh
new file mode 100755
index 0000000..ef31f46
--- /dev/null
+++ b/tools/vm/setup_mdev.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# Setup script for NVIDIA vGPU mdev device
+# This script creates an mdev device for QEMU testing
+
+set -euo pipefail
+
+GPU_PCI_ADDR="${GPU_PCI_ADDR:-0000:01:00.0}"
+VGPU_TYPE="${VGPU_TYPE:-nvidia-vgpu-turing}"
+MDEV_UUID="${MDEV_UUID:-$(cat /proc/sys/kernel/random/uuid)}"
+
+MDEV_CREATE_PATH="/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE/create"
+
+echo "=== NVIDIA vGPU mdev Device Setup ==="
+echo "GPU PCI: $GPU_PCI_ADDR"
+echo "VGPU Type: $VGPU_TYPE"
+echo "MDEV UUID: $MDEV_UUID"
+echo ""
+
+# Check if GPU device exists
+if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR" ]]; then
+ echo "ERROR: GPU device $GPU_PCI_ADDR not found"
+ echo "Available NVIDIA GPUs:"
+ lspci -nn -d 10de: | grep -E "VGA|3D"
+ exit 1
+fi
+
+# Check if mdev_supported_types exists (requires NVIDIA vGPU host driver)
+if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types" ]]; then
+ echo "ERROR: mdev_supported_types not found for $GPU_PCI_ADDR"
+ echo ""
+ echo "This requires the NVIDIA vGPU host driver (licensed product for data center GPUs)."
+ echo "The regular NVIDIA driver does not provide vGPU mdev support."
+ echo ""
+ echo "For testing on consumer GPUs (TU104, etc.), use dry-run mode:"
+ echo " sudo insmod /path/to/nvidia-vgpu-vfio.ko dry_run=1"
+ echo ""
+ echo "In dry-run mode, the module loads but cannot create real mdev devices."
+ exit 1
+fi
+
+# Check if requested vGPU type exists
+if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE" ]]; then
+ echo "ERROR: vGPU type $VGPU_TYPE not supported"
+ echo "Available types:"
+ ls "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/"
+ exit 1
+fi
+
+# Create mdev device
+echo "Creating mdev device..."
+echo "$MDEV_UUID" | sudo tee "$MDEV_CREATE_PATH" > /dev/null
+
+if [[ -d "/sys/bus/mdev/devices/$MDEV_UUID" ]]; then
+ echo "SUCCESS: mdev device created at /sys/bus/mdev/devices/$MDEV_UUID"
+ echo ""
+ echo "Device info:"
+ cat "/sys/bus/mdev/devices/$MDEV_UUID/description" 2>/dev/null || echo " (no description)"
+ echo ""
+ echo "To use with QEMU:"
+ echo " export MDEV_UUID=$MDEV_UUID"
+ echo " /path/to/launch_vgpu_qemu.sh"
+else
+ echo "ERROR: Failed to create mdev device"
+ exit 1
+fi