diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-13 18:36:21 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-13 18:36:21 -0400 |
| commit | 00e54c13c7c9030c4d5f7151467f2d85a3730e29 (patch) | |
| tree | 17a45239df8d2e4912ff3697ed06e85a10b969de /tools | |
| download | open-iov-00e54c13c7c9030c4d5f7151467f2d85a3730e29.tar.gz open-iov-00e54c13c7c9030c4d5f7151467f2d85a3730e29.zip | |
Initial commit: Open-IOV vGPU-IO complete stack
- Firmware decompilation: GSP 570.144W (28.5MB ELF, RV64IMAC), 132 decompiled C files
- Kernel modules: nvidia.ko (34MB, OpenRM), open-iov-vfio.ko (2.8MB, VFIO mdev), gsp-vgpu.ko (1.8MB, guest driver)
- Userspace: libiov.so/libvgpu.so (76KB), example_vgpu (28KB), rpc_emulator.c
- PCI config space emulation (256B with PM, MSI, MSI-X, PCIe caps)
- 12 RPC types fully mapped from firmware decompilation
- BAR trap, WPR, scheduling, notification, migration emulation
- Helper scripts: load_vgpu_stack, create_mdev, launch_qemu_vgpu, test_vgpu_stack, setup_guest_driver, unload_vgpu_stack
- Architecture documentation
- Wiki: 10 technical documents
- Git LFS for large binaries (.ko, .bin, .iso, .img, .o, ghidra repos)
- nvidia-open-gpu-kernel-modules as submodule
Diffstat (limited to 'tools')
42 files changed, 1478 insertions, 0 deletions
diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..f9e0018 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,158 @@ +# Open-IOV Development Tools + +Collection of scripts and utilities for vGPU development, debugging, and testing. + +## Directory Structure + +``` +tools/ +├── module_ops/ # Kernel module load/unload/reload utilities +├── debug/ # Debugging and diagnostic tools +├── test/ # Test runners and validation scripts +├── vm/ # VM/QEMU management for vGPU testing +├── firmware/ # GSP firmware management +└── monitoring/ # Real-time monitoring utilities +``` + +## Quick Start + +```bash +# Load modules in converged mode (host + vGPU) +sudo ./module_ops/load_converged.sh + +# Unload all modules cleanly +sudo ./module_ops/unload_all.sh + +# Run dry-run validation +sudo ./module_ops/dry_run.sh + +# Monitor kernel logs in real-time +./monitoring/watch_dmesg.sh + +# Verify vGPU sysfs interface +./debug/verify_vgpu_sysfs.sh +``` + +## Module Operations + +### `module_ops/load_converged.sh` +Loads the complete vGPU stack in converged mode: +1. `mdev` core module +2. `nvidia.ko` (patched open-gpu-kernel-modules with NV_VGPU_KVM_BUILD) +3. `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko` +4. `nvidia-vgpu-vfio.ko` (open-iov kernel module) + +### `module_ops/unload_all.sh` +Cleanly unloads all modules in reverse order with proper cleanup. + +### `module_ops/dry_run.sh` +Runs nvidia-vgpu-vfio.ko in dry-run mode to validate firmware, BAR layout, IRQ caps, IOMMU without hardware changes. + +### `module_ops/reload_vgpu.sh` +Reloads only the vGPU kernel module (nvidia-vgpu-vfio.ko) while keeping host nvidia.ko loaded. + +## Debug Tools + +### `debug/verify_vgpu_sysfs.sh` +Verifies mdev parent registration and vGPU types in sysfs. + +### `debug/check_module_state.sh` +Checks module initstate, refcnt, and holder information. + +### `debug/dump_vgpu_regs.sh` +Dumps vGPU device registers via debugfs (if available). + +### `debug/capture_crash_logs.sh` +Captures kernel logs, dmesg, and module state after a crash. + +## Test Scripts + +### `test/run_integration_tests.sh` +Runs the integration test suite (host_init, guest_init). + +### `test/test_mdev_create.sh` +Creates an mdev device and verifies basic functionality. + +### `test/benchmark_vgpu.sh` +Runs performance benchmarks on vGPU. + +## VM Management + +### `vm/create_test_vm.sh` +Creates a test VM with vGPU passthrough using QEMU. + +### `vm/launch_guest_driver.sh` +Launches guest with NVIDIA driver installed for validation. + +## Firmware Tools + +### `firmware/verify_gsp_firmware.sh` +Verifies GSP firmware files exist and are correct version. + +### `firmware/install_tu104_firmware.sh` +Installs TU104 GSP firmware to /lib/firmware/nvidia/tu104/gsp/ + +## Monitoring + +### `monitoring/watch_dmesg.sh` +Real-time kernel log monitoring with filtering for vGPU/NVIDIA messages. + +### `monitoring/watch_module_params.sh` +Monitors module parameters and sysfs attributes. + +## Environment Variables + +```bash +export OPEN_IOV_ROOT=$OPEN_IOV_ROOT +export NVIDIA_OPEN_KERNEL=$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +export VGPU_KERNEL=$OPEN_IOV_ROOT/kernel +export VGPU_USERSPACE=$OPEN_IOV_ROOT/userspace +``` + +## Module Parameters for Debugging + +| Parameter | Description | Values | +|-----------|-------------|--------| +| `NVreg_RmMsg` | RM debug logging | `":"` (verbose), `"0"` (none) | +| `NVreg_RegisterPCIDriver` | PCI driver registration | `0` (disable), `1` (enable) | +| `NVreg_GpuInitOnProbe` | GPU init at probe | `0` (defer), `1` (immediate) | +| `NVreg_PreserveVideoMemoryAllocations` | Video memory preservation | `0` (disable), `1` (enable), `2` (auto) | +| `NVreg_UseKernelSuspendNotifiers` | Suspend notifiers | `0` (disable), `1` (enable) | +| `dyndbg` | Dynamic debug | `'+p'` (all), `'+pf'` (probe+fail) | + +## Converged Mode Architecture + +``` +Host nvidia.ko (patched) nvidia-vgpu-vfio.ko (open-iov) +├── mdev_register_parent() ←── mdev_register_driver() only +├── EXPORT nvidia_vgpu_vfio_set_ops +├── EXPORT nvidia_vgpu_vfio_get_ops +└── Implements RM vGPU ops Calls RM via get_ops, provides VFIO ops via set_ops +``` + +## Troubleshooting + +### Module stuck in "going" state +```bash +cat /sys/module/nvidia-vgpu-vfio/initstate +cat /sys/module/nvidia-vgpu-vfio/refcnt +# Requires reboot to clear +``` + +### mdev parent not registered +```bash +ls /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/ +# Should show nvidia-vgpu-* directories with create/available_instances +``` + +### Symbol version mismatch +```bash +# Rebuild open-gpu-kernel-modules with VGX_KVM_BUILD=1 +cd $NVIDIA_OPEN_KERNEL && make clean && make -j$(nproc) VGX_KVM_BUILD=1 +``` + +### IOMMU/VT-d not enabled +```bash +cat /proc/cmdline | grep -E "iommu|intel_iommu" +# Add: intel_iommu=on iommu=pt to kernel cmdline +```
\ No newline at end of file diff --git a/tools/debug/nv2080_ctrl.c b/tools/debug/nv2080_ctrl.c new file mode 100644 index 0000000..46ade02 --- /dev/null +++ b/tools/debug/nv2080_ctrl.c @@ -0,0 +1,531 @@ +/* + * NV2080 Control Command Sender + * C program to send NV2080 RM control commands via ioctl + * Compile: gcc -o nv2080_ctrl nv2080_ctrl.c + * Usage: sudo ./nv2080_ctrl /dev/nvidia0 <command> [args...] + */ + +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <sys/ioctl.h> +#include <errno.h> + +// NV_IOCTL definitions from nv-ioctl.h +#define NV_IOCTL_MAGIC 0x46 +#define NV_ESC_RM_CONTROL 0x54 +#define NV_ESC_CHECK_VERSION_STR 0x32 +#define NV_ESC_RM_ALLOC 0x21 +#define NV_ESC_RM_FREE 0x00 +#define NV_ESC_ALLOC_OS_EVENT 0x24 + +// IOCTL macros +#define _IOC_NRBITS 8 +#define _IOC_TYPEBITS 8 +#define _IOC_SIZEBITS 14 +#define _IOC_DIRBITS 2 + +#define _IOC_NRSHIFT 0 +#define _IOC_TYPESHIFT (_IOC_NRSHIFT + _IOC_NRBITS) +#define _IOC_SIZESHIFT (_IOC_TYPESHIFT + _IOC_TYPEBITS) +#define _IOC_DIRSHIFT (_IOC_SIZESHIFT + _IOC_SIZEBITS) + +#define _IOC_NONE 0U +#define _IOC_WRITE 1U +#define _IOC_READ 2U + +#define _IOC(dir, type, nr, size) \ + (((dir) << _IOC_DIRSHIFT) | \ + ((type) << _IOC_TYPESHIFT) | \ + ((nr) << _IOC_NRSHIFT) | \ + ((size) << _IOC_SIZESHIFT)) + +#define _IOWR(type, nr, size) _IOC(_IOC_READ | _IOC_WRITE, type, nr, size) + +// NV_CTRL_RES = _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, NVOS54_PARAMETERS) +#define NV_CTRL_RES _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, 128) + +// NV2080 Control Commands (from ctrl2080unix.h and ctrl2080vgpumgrinternal.h) +#define NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT 0x20803d01 +#define NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF 0x20803d02 +#define NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER 0x20803d03 +#define NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS 0x20803d04 +#define NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER 0x20803d05 +#define NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER 0x20803d06 +#define NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS 0x20803d07 +#define NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS 0x20803d08 + +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK 0x20804001 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK 0x20804002 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE 0x20804003 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU 0x20804004 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO 0x20804005 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE 0x20804006 +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY 0x20804007 +#define NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP 0x20804008 + +// Action values +#define NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC 1 +#define NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC 2 +#define NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW 1 +#define NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW 2 + +// NVOS54_PARAMETERS structure (from nv-ioctl.h) +typedef struct { + uint32_t hClient; + uint32_t hObject; + uint32_t cmd; + uint32_t flags; + void *params; + uint32_t paramsSize; + uint32_t status; +} NVOS54_PARAMETERS; + +// NVOS21_PARAMETERS for RM_ALLOC +typedef struct { + uint32_t hRoot; + uint32_t hObjectParent; + uint32_t hObjectNew; + uint32_t hClass; + void *pAllocParms; + uint32_t status; +} NVOS21_PARAMETERS; + +// NVOS00_PARAMETERS for RM_FREE +typedef struct { + uint32_t hRoot; + uint32_t hObjectParent; + uint32_t hObjectOld; + uint32_t status; +} NVOS00_PARAMETERS; + +// Version check structure +typedef struct { + uint32_t cmd; + char versionString[256]; + uint32_t reply; +} nv_ioctl_rm_api_version_t; + +// OS Event allocation +typedef struct { + uint32_t hClient; + uint32_t hDevice; + int fd; + uint32_t Status; +} nv_ioctl_alloc_os_event_t; + +// GC6 Blocker Refcnt Params +typedef struct { + uint32_t action; +} NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS; + +// Allow/Disallow GCOFF Params +typedef struct { + uint32_t action; +} NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS; + +// Audio Dynamic Power Params +typedef struct { + uint32_t bEnter; +} NV2080_CTRL_OS_UNIX_AUDIO_DYNAMIC_POWER_PARAMS; + +// Vidmem Persistence Status Params +typedef struct { + uint32_t bVidmemPersistent; +} NV2080_CTRL_OS_UNIX_VIDMEM_PERSISTENCE_STATUS_PARAMS; + +// TGP Status Params +typedef struct { + uint32_t bUpdateTGP; +} NV2080_CTRL_OS_UNIX_UPDATE_TGP_STATUS_PARAMS; + +// VGPU Bootload Params (simplified) +typedef struct { + uint32_t dbdf; + uint32_t gfid; + uint32_t vgpuType; + uint32_t vmPid; + uint32_t swizzId; + uint32_t numChannels; + uint32_t numPluginChannels; + uint32_t bDisableDefaultSmcExecPartRestore; + uint32_t numGuestFbSegments; + uint64_t guestFbPhysAddrList[384]; + uint64_t guestFbLengthList[384]; + uint64_t pluginHeapMemoryPhysAddr; + uint64_t pluginHeapMemoryLength; + uint64_t ctrlBuffOffset; +} NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS; + +// VGPU Shutdown Params +typedef struct { + uint32_t gfid; +} NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS; + +// VGPU Add Type Params +typedef struct { + uint32_t discardVgpuTypes; + uint64_t vgpuInfo; // NVA081_CTRL_VGPU_INFO +} NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS; + +// VGPU Enumerate Params +typedef struct { + uint32_t numVgpu; + uint64_t vgpuGuest[32]; // NV2080_VGPU_GUEST array +} NV2080_CTRL_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU_PARAMS; + +// VGPU Clear Guest Params +typedef struct { + uint32_t gfid; +} NV2080_CTRL_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO_PARAMS; + +// VGPU FB Usage Params +typedef struct { + uint32_t vgpuCount; + struct { + uint32_t gfid; + uint64_t fbUsed; + } vgpuFbUsage[32]; +} NV2080_CTRL_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE_PARAMS; + +// VGPU Encoder Capacity Params +typedef struct { + uint32_t gfid; + uint32_t encoderCapacity; +} NV2080_CTRL_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY_PARAMS; + +// VGPU Plugin Cleanup Params +typedef struct { + uint32_t gfid; +} NV2080_CTRL_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS; + +// Send RM Control command +int send_rm_control(int fd, uint32_t client, uint32_t object, uint32_t cmd, void *params, uint32_t params_size) { + NVOS54_PARAMETERS ctrl = { + .hClient = client, + .hObject = object, + .cmd = cmd, + .flags = 0, + .params = params, + .paramsSize = params_size, + .status = 0 + }; + + int ret = ioctl(fd, NV_CTRL_RES, &ctrl); + if (ret == -1) { + perror("ioctl NV_CTRL_RES failed"); + return -1; + } + + if (ctrl.status != 0) { + fprintf(stderr, "RM Control failed: cmd=0x%x, status=0x%x\n", cmd, ctrl.status); + return ctrl.status; + } + + return 0; +} + +// Version check +int version_check(int fd, const char *version) { + nv_ioctl_rm_api_version_t vc = {0}; + vc.cmd = 0x32; // NV_ESC_CHECK_VERSION_STR + if (version) { + strncpy(vc.versionString, version, sizeof(vc.versionString) - 1); + } + + int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_CHECK_VERSION_STR, sizeof(vc)), &vc); + if (ret == -1) { + perror("ioctl version check failed"); + return -1; + } + + printf("Version check: reply=%u, version=%s\n", vc.reply, vc.versionString); + return vc.reply == 1 ? 0 : -1; +} + +// Allocate RM resource +int rm_alloc(int fd, uint32_t root, uint32_t parent, uint32_t object, uint32_t class, void *data) { + NVOS21_PARAMETERS alloc = { + .hRoot = root, + .hObjectParent = parent, + .hObjectNew = object, + .hClass = class, + .pAllocParms = data, + .status = 0 + }; + + int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_ALLOC, sizeof(alloc)), &alloc); + if (ret == -1) { + perror("ioctl RM_ALLOC failed"); + return -1; + } + + if (alloc.status != 0) { + fprintf(stderr, "RM_ALLOC failed: status=0x%x\n", alloc.status); + return alloc.status; + } + + printf("Allocated object 0x%x (class 0x%x)\n", alloc.hObjectNew, class); + return alloc.hObjectNew; +} + +// Free RM resource +int rm_free(int fd, uint32_t root, uint32_t parent, uint32_t object) { + NVOS00_PARAMETERS free_params = { + .hRoot = root, + .hObjectParent = parent, + .hObjectOld = object, + .status = 0 + }; + + int ret = ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_FREE, sizeof(free_params)), &free_params); + if (ret == -1) { + perror("ioctl RM_FREE failed"); + return -1; + } + + return free_params.status == 0 ? 0 : free_params.status; +} + +// Print usage +void print_usage(const char *prog) { + fprintf(stderr, "Usage: %s <device> <command> [args...]\n", prog); + fprintf(stderr, "\nDevice: /dev/nvidia0, /dev/nvidiactl, etc.\n"); + fprintf(stderr, "\nCommands:\n"); + fprintf(stderr, " version-check [version] - Check RM API version\n"); + fprintf(stderr, " gc6-inc <client> <object> - Increment GC6 blocker refcount\n"); + fprintf(stderr, " gc6-dec <client> <object> - Decrement GC6 blocker refcount\n"); + fprintf(stderr, " gcoff-allow <client> <object> - Allow GCOFF\n"); + fprintf(stderr, " gcoff-disallow <client> <object> - Disallow GCOFF\n"); + fprintf(stderr, " audio-power <client> <object> <enter> - Audio dynamic power (1=enter, 0=exit)\n"); + fprintf(stderr, " vidmem-persistence <client> <object> - Query vidmem persistence\n"); + fprintf(stderr, " tgp-status <client> <object> <update> - Update TGP status (1=update, 0=don't)\n"); + fprintf(stderr, " vgpu-bootload <client> <object> <dbdf> <gfid> <vgpuType> <vmPid> ... - Bootload GSP VGPU plugin\n"); + fprintf(stderr, " vgpu-shutdown <client> <object> <gfid> - Shutdown GSP VGPU plugin\n"); + fprintf(stderr, " vgpu-add-type <client> <object> <discard> <vgpuInfo> - Add vGPU type\n"); + fprintf(stderr, " vgpu-enumerate <client> <object> - Enumerate vGPUs per pGPU\n"); + fprintf(stderr, " vgpu-clear-guest <client> <object> <gfid> - Clear guest VM info\n"); + fprintf(stderr, " vgpu-fb-usage <client> <object> - Get vGPU FB usage\n"); + fprintf(stderr, " vgpu-encoder-cap <client> <object> <gfid> <capacity> - Set encoder capacity (0-100)\n"); + fprintf(stderr, " vgpu-plugin-cleanup <client> <object> <gfid> - Cleanup VGPU plugin\n"); + fprintf(stderr, " alloc <root> <parent> <object> <class> - Allocate RM resource\n"); + fprintf(stderr, " free <root> <parent> <object> - Free RM resource\n"); +} + +int main(int argc, char *argv[]) { + if (argc < 3) { + print_usage(argv[0]); + return 1; + } + + const char *device = argv[1]; + const char *command = argv[2]; + + int fd = open(device, O_RDWR); + if (fd < 0) { + perror("Failed to open device"); + return 1; + } + + int ret = 0; + + if (strcmp(command, "version-check") == 0) { + const char *version = argc > 3 ? argv[3] : NULL; + ret = version_check(fd, version); + } + else if (strcmp(command, "gc6-inc") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "gc6-dec") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "gcoff-allow") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "gcoff-disallow") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_PARAMS params = { .action = NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "audio-power") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t enter = strtoul(argv[5], NULL, 0); + NV2080_CTRL_OS_UNIX_AUDIO_DYNAMIC_POWER_PARAMS params = { .bEnter = enter }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vidmem-persistence") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_OS_UNIX_VIDMEM_PERSISTENCE_STATUS_PARAMS params = {0}; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS, ¶ms, sizeof(params)); + if (ret == 0) { + printf("Vidmem persistent: %s\n", params.bVidmemPersistent ? "yes" : "no"); + } + } + } + else if (strcmp(command, "tgp-status") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t update = strtoul(argv[5], NULL, 0); + NV2080_CTRL_OS_UNIX_UPDATE_TGP_STATUS_PARAMS params = { .bUpdateTGP = update }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-bootload") == 0) { + if (argc < 12) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS params = {0}; + params.dbdf = strtoul(argv[5], NULL, 0); + params.gfid = strtoul(argv[6], NULL, 0); + params.vgpuType = strtoul(argv[7], NULL, 0); + params.vmPid = strtoul(argv[8], NULL, 0); + params.swizzId = strtoul(argv[9], NULL, 0); + params.numChannels = strtoul(argv[10], NULL, 0); + params.numPluginChannels = strtoul(argv[11], NULL, 0); + // Additional params would need more args + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-shutdown") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t gfid = strtoul(argv[5], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS params = { .gfid = gfid }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-add-type") == 0) { + if (argc < 7) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t discard = strtoul(argv[5], NULL, 0); + uint64_t vgpuInfo = strtoull(argv[6], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS params = { .discardVgpuTypes = discard, .vgpuInfo = vgpuInfo }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-enumerate") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU_PARAMS params = {0}; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU, ¶ms, sizeof(params)); + if (ret == 0) { + printf("Enumerated %u vGPUs\n", params.numVgpu); + } + } + } + else if (strcmp(command, "vgpu-clear-guest") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t gfid = strtoul(argv[5], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO_PARAMS params = { .gfid = gfid }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-fb-usage") == 0) { + if (argc < 5) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE_PARAMS params = {0}; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE, ¶ms, sizeof(params)); + if (ret == 0) { + printf("vGPU count: %u\n", params.vgpuCount); + for (uint32_t i = 0; i < params.vgpuCount; i++) { + printf(" GFID 0x%x: FB used %lu bytes\n", params.vgpuFbUsage[i].gfid, params.vgpuFbUsage[i].fbUsed); + } + } + } + } + else if (strcmp(command, "vgpu-encoder-cap") == 0) { + if (argc < 7) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t gfid = strtoul(argv[5], NULL, 0); + uint32_t capacity = strtoul(argv[6], NULL, 0); + NV2080_CTRL_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY_PARAMS params = { .gfid = gfid, .encoderCapacity = capacity }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "vgpu-plugin-cleanup") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t client = strtoul(argv[3], NULL, 0); + uint32_t object = strtoul(argv[4], NULL, 0); + uint32_t gfid = strtoul(argv[5], NULL, 0); + NV2080_CTRL_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS params = { .gfid = gfid }; + ret = send_rm_control(fd, client, object, NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP, ¶ms, sizeof(params)); + } + } + else if (strcmp(command, "alloc") == 0) { + if (argc < 7) { print_usage(argv[0]); ret = 1; } + else { + uint32_t root = strtoul(argv[3], NULL, 0); + uint32_t parent = strtoul(argv[4], NULL, 0); + uint32_t object = strtoul(argv[5], NULL, 0); + uint32_t class = strtoul(argv[6], NULL, 0); + ret = rm_alloc(fd, root, parent, object, class, NULL); + } + } + else if (strcmp(command, "free") == 0) { + if (argc < 6) { print_usage(argv[0]); ret = 1; } + else { + uint32_t root = strtoul(argv[3], NULL, 0); + uint32_t parent = strtoul(argv[4], NULL, 0); + uint32_t object = strtoul(argv[5], NULL, 0); + ret = rm_free(fd, root, parent, object); + } + } + else { + fprintf(stderr, "Unknown command: %s\n", command); + print_usage(argv[0]); + ret = 1; + } + + close(fd); + return ret; +}
\ No newline at end of file diff --git a/tools/debug/nv2080_ctrl.sh b/tools/debug/nv2080_ctrl.sh new file mode 100755 index 0000000..22c7117 --- /dev/null +++ b/tools/debug/nv2080_ctrl.sh @@ -0,0 +1,168 @@ +#!/bin/bash +# NV2080 Control Command Tool +# Sends NV2080 RM control commands to the NVIDIA kernel module via ioctl +# Usage: ./nv2080_ctrl.sh <device_fd> <command> [params...] + +set -euo pipefail + +# NV2080 Control Command Definitions (from ctrl2080unix.h and ctrl2080vgpumgrinternal.h) +# These are the same commands used by GVM userspace via NV_CTRL_RES ioctl + +# NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT (0x20803d01) +# Params: action (INC=1, DEC=2) +NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT=0x20803d01 +NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC=1 +NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC=2 + +# NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF (0x20803d02) +# Params: action (ALLOW=1, DISALLOW=2) +NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF=0x20803d02 +NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW=1 +NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW=2 + +# NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER (0x20803d03) +# Params: bEnter (true/false) +NV2080_CTRL_CMD_OS_UNIX_AUDIO_DYNAMIC_POWER=0x20803d03 + +# NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS (0x20803d04) +NV2080_CTRL_CMD_OS_UNIX_INSTALL_PROFILER_HOOKS=0x20803d04 + +# NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER (0x20803d05) +NV2080_CTRL_CMD_OS_UNIX_FLUSH_SNAPSHOT_BUFFER=0x20803d05 + +# NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER (0x20803d06) +NV2080_CTRL_CMD_OS_UNIX_STOP_PROFILER=0x20803d06 + +# NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS (0x20803d07) +NV2080_CTRL_CMD_OS_UNIX_VIDMEM_PERSISTENCE_STATUS=0x20803d07 + +# NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS (0x20803d08) +NV2080_CTRL_CMD_OS_UNIX_UPDATE_TGP_STATUS=0x20803d08 + +# VGPU Manager Internal Commands +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK=0x20804001 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK=0x20804002 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE=0x20804003 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_ENUMERATE_VGPU_PER_PGPU=0x20804004 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_CLEAR_GUEST_VM_INFO=0x20804005 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_GET_VGPU_FB_USAGE=0x20804006 +NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SET_VGPU_ENCODER_CAPACITY=0x20804007 +NV2080_CTRL_CMD_INTERNAL_VGPU_PLUGIN_CLEANUP=0x20804008 + +# NV_IOCTL_MAGIC and NV_ESC_RM_CONTROL +NV_IOCTL_MAGIC=0x46 +NV_ESC_RM_CONTROL=0x54 + +# NV_CTRL_RES = _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, NVOS54_PARAMETERS) +NV_CTRL_RES=$(printf "0x%x" $(( (2 << 30) | (NV_IOCTL_MAGIC << 8) | (NV_ESC_RM_CONTROL) | (128 << 16) ))) + +# Helper to send ioctl +send_ioctl() { + local fd=$1 + local cmd=$2 + local data=$3 + local size=$4 + + # This would need a C program to actually send the ioctl + # For now, we document the commands + echo "Command: 0x$(printf "%x" $cmd)" + echo "Data size: $size" +} + +# Main command dispatcher +case "${1:-help}" in + gc6-inc) + echo "Increment GC6 blocker refcount" + echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, action: $NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_INC})" + ;; + gc6-dec) + echo "Decrement GC6 blocker refcount" + echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_GC6_BLOCKER_REFCNT, action: $NV2080_CTRL_OS_UNIX_GC6_BLOCKER_REFCNT_DEC})" + ;; + gcoff-allow) + echo "Allow GCOFF" + echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, action: $NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_ALLOW})" + ;; + gcoff-disallow) + echo "Disallow GCOFF" + echo "ioctl(fd, NV_CTRL_RES, {cmd: $NV2080_CTRL_CMD_OS_UNIX_ALLOW_DISALLOW_GCOFF, action: $NV2080_CTRL_OS_UNIX_ALLOW_DISALLOW_GCOFF_DISALLOW})" + ;; + audio-power) + echo "Audio dynamic power: $2" + ;; + profiler-install) + echo "Install profiler hooks" + ;; + profiler-flush) + echo "Flush snapshot buffer" + ;; + profiler-stop) + echo "Stop profiler" + ;; + vidmem-persistence) + echo "Query vidmem persistence status" + ;; + tgp-status) + echo "Update TGP status: $2" + ;; + vgpu-bootload) + echo "Bootload GSP VGPU plugin task" + echo "Params: dbdf, gfid, vgpuType, vmPid, swizzId, numChannels, numPluginChannels, bDisableSmcPartitionRestore, guestFbPhysAddrList, guestFbLengthList, pluginHeapMemoryPhysAddr, pluginHeapMemoryLength, ctrlBuffOffset" + ;; + vgpu-shutdown) + echo "Shutdown GSP VGPU plugin task" + echo "Params: gfid" + ;; + vgpu-add-type) + echo "Add vGPU type to pGPU" + echo "Params: discardVgpuTypes, vgpuInfo" + ;; + vgpu-enumerate) + echo "Enumerate vGPUs per pGPU" + ;; + vgpu-clear-guest) + echo "Clear guest VM info" + echo "Params: gfid" + ;; + vgpu-fb-usage) + echo "Get vGPU FB usage" + ;; + vgpu-encoder-cap) + echo "Set vGPU encoder capacity" + echo "Params: gfid, encoderCapacity (0-100)" + ;; + vgpu-plugin-cleanup) + echo "Cleanup VGPU plugin" + echo "Params: gfid" + ;; + list) + echo "Available NV2080 Control Commands:" + echo " OS Unix Commands:" + echo " gc6-inc - Increment GC6 blocker refcount" + echo " gc6-dec - Decrement GC6 blocker refcount" + echo " gcoff-allow - Allow GCOFF" + echo " gcoff-disallow - Disallow GCOFF" + echo " audio-power - Audio dynamic power (enter/exit)" + echo " profiler-install - Install profiler hooks (HWPM)" + echo " profiler-flush - Flush snapshot buffer" + echo " profiler-stop - Stop profiler" + echo " vidmem-persistence- Query vidmem persistence status" + echo " tgp-status - Update TGP status" + echo "" + echo " VGPU Manager Internal Commands:" + echo " vgpu-bootload - Bootload GSP VGPU plugin task" + echo " vgpu-shutdown - Shutdown GSP VGPU plugin task" + echo " vgpu-add-type - Add vGPU type to pGPU" + echo " vgpu-enumerate - Enumerate vGPUs per pGPU" + echo " vgpu-clear-guest - Clear guest VM info" + echo " vgpu-fb-usage - Get vGPU FB usage" + echo " vgpu-encoder-cap - Set vGPU encoder capacity" + echo " vgpu-plugin-cleanup - Cleanup VGPU plugin" + ;; + help|*) + echo "NV2080 Control Command Tool" + echo "Usage: $0 <command> [args...]" + echo "" + echo "Run '$0 list' to see all available commands" + ;; +esac
\ No newline at end of file diff --git a/tools/firmware/verify_gsp_firmware.sh b/tools/firmware/verify_gsp_firmware.sh new file mode 100755 index 0000000..79f474c --- /dev/null +++ b/tools/firmware/verify_gsp_firmware.sh @@ -0,0 +1,60 @@ +#!/bin/bash +# Verify GSP firmware for TU104 +# Usage: ./verify_gsp_firmware.sh + +set -euo pipefail + +FIRMWARE_DIR="/lib/firmware/nvidia/tu104/gsp" + +echo "=== GSP Firmware Verification (TU104) ===" +echo "" + +if [[ ! -d "$FIRMWARE_DIR" ]]; then + echo "ERROR: Firmware directory not found: $FIRMWARE_DIR" + echo "Install firmware with: rzsh nvidia vgpu firmware install" + exit 1 +fi + +echo "Firmware directory: $FIRMWARE_DIR" +echo "" + +# Check required files +required_files=( + "gsp-535.113.01.bin" + "gsp-570.144.bin" + "booter_load-535.113.01.bin" + "booter_load-570.144.bin" + "booter_unload-535.113.01.bin" + "booter_unload-570.144.bin" + "bootloader-535.113.01.bin" + "bootloader-570.144.bin" +) + +all_ok=true +for file in "${required_files[@]}"; do + if [[ -f "$FIRMWARE_DIR/$file" ]]; then + size=$(stat -c%s "$FIRMWARE_DIR/$file") + echo " ✓ $file ($size bytes)" + else + echo " ✗ $file MISSING" + all_ok=false + fi +done + +echo "" +if $all_ok; then + echo "✓ All required firmware files present" +else + echo "✗ Some firmware files missing" + echo "" + echo "To install firmware:" + echo " 1. Download from NVIDIA or copy from nvidia-linux-firmware repo" + echo " 2. sudo cp -r nvidia/tu104/gsp/* /lib/firmware/nvidia/tu104/gsp/" + echo " 3. sudo update-initramfs -u" +fi + +echo "" +echo "Current kernel module version:" +if lsmod | grep -q nvidia; then + cat /proc/driver/nvidia/version 2>/dev/null | head -1 +fi
\ No newline at end of file diff --git a/tools/ghidra/analyze_fwimage.py b/tools/ghidra/analyze_fwimage.py new file mode 100644 index 0000000..ea0f87f --- /dev/null +++ b/tools/ghidra/analyze_fwimage.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:557f20be1c03606549eacab7c7e524abd30cc1826e9cac3c583da51206520447 +size 1886 diff --git a/tools/ghidra/analyze_gsp_firmware.sh b/tools/ghidra/analyze_gsp_firmware.sh new file mode 100755 index 0000000..911e0d2 --- /dev/null +++ b/tools/ghidra/analyze_gsp_firmware.sh @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5d4f65bee3ce0c98e749ab318318dabea3d2268eeec30c1c4c4e5c7e758487bb +size 3960 diff --git a/tools/ghidra/create_gsp_project.py b/tools/ghidra/create_gsp_project.py new file mode 100644 index 0000000..454a3e3 --- /dev/null +++ b/tools/ghidra/create_gsp_project.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a2cb538468bd58ebec94432eb58ae79a478e75f8a508f177192533f648dc007c +size 1916 diff --git a/tools/ghidra/diff_gsp_firmware.py b/tools/ghidra/diff_gsp_firmware.py new file mode 100644 index 0000000..ea804ec --- /dev/null +++ b/tools/ghidra/diff_gsp_firmware.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:44976b0275737468689f718b96037540699defc010019793a8e58bf509ddf2a3 +size 7816 diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-analysis.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fn2.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-bar-fns.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~ new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-fwimage-570.144.lock~ diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-handlers2.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-mig-tables.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-rpc-handlers2.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds2.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-subcmds3.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-targeted2.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr b/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/gsp-wpr.gpr diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java b/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java new file mode 100644 index 0000000..ef338af --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/scripts/BasicAnalysis.java @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8beabb120fb5e74202ca22087b2c4baa8fc81834c120d93a673b5fd74230f12b +size 6940 diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java b/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java new file mode 100644 index 0000000..4b8e165 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/scripts/DecompileFunctions.java @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eaca1bea9dd6d70970d9b629b2d5667614407992f2a85b92157c27d0c18c7908 +size 3354 diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java b/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java new file mode 100644 index 0000000..aadc1db --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/scripts/ExtractFunctions.java @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ceda696aad0ec35e26e1fcc02ed6aa3c7ad32faf2eff25cc390620b377776b67 +size 5187 diff --git a/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py b/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py new file mode 100644 index 0000000..ebb1e63 --- /dev/null +++ b/tools/ghidra/gsp-fwimage-570.144/scripts/basic_analysis.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d35f85f13506e8491bced83697cf44a0708f03e413280831a3f584c829f5cd1e +size 3869 diff --git a/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.gpr diff --git a/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~ new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/gsp-tu104-570.144.lock~ diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java b/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java new file mode 100644 index 0000000..1be91a5 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/scripts/DecompileFunctions.java @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4ff08f6647f5f1d1c678344223226fd00ca605bca903a5fdb44e39ad3aa982a0 +size 5725 diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py b/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py new file mode 100644 index 0000000..431a813 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/scripts/analyze_gsp_fwimage.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:86e458aa7968cfc1362837ba0278af410c0dbe70bdec97bce413225f98640bd0 +size 7025 diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py b/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py new file mode 100644 index 0000000..ebb1e63 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/scripts/basic_analysis.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d35f85f13506e8491bced83697cf44a0708f03e413280831a3f584c829f5cd1e +size 3869 diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py b/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py new file mode 100644 index 0000000..fb828c0 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/scripts/extract_fwimage.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:39970014d9ba8f37b9fec29643ca2b2194dbd9f6e5288c20ce6191c6a8e98cc6 +size 1923 diff --git a/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py b/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py new file mode 100644 index 0000000..8d02ac1 --- /dev/null +++ b/tools/ghidra/gsp-tu104-570.144/scripts/extract_gsp_version.py @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ae3f536d9a82c9cd0e829d28ae9ebe4a3142004ba0ac3c8ca76438a69e5a32ce +size 1062 diff --git a/tools/module_ops/dry_run.sh b/tools/module_ops/dry_run.sh new file mode 100755 index 0000000..8bda820 --- /dev/null +++ b/tools/module_ops/dry_run.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# Run vGPU dry-run validation +# Usage: sudo ./dry_run.sh + +set -euo pipefail + +VGPU_KERNEL="$OPEN_IOV_ROOT/kernel" + +echo "=== Open-IOV vGPU Dry-Run Validation ===" +echo "" + +# Check if host nvidia.ko is loaded +if ! lsmod | grep -q "^nvidia "; then + echo "ERROR: Host nvidia.ko not loaded. Load it first with:" + echo " rzsh nvidia vgpu load" + exit 1 +fi + +echo "Host nvidia.ko: LOADED ✓" +echo "" + +# Load vGPU module in dry-run mode +echo "Loading nvidia-vgpu-vfio.ko in dry-run mode..." +sudo insmod "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" dyndbg='+p' dry_run=1 + +# Check result +sleep 2 +if dmesg -T | grep -q "DRY-RUN MODE: validating init without hardware changes"; then + echo "" + echo "✓ Dry-run validation PASSED" + echo " Firmware, BAR layout, IRQ caps, IOMMU validated" +else + echo "" + echo "✗ Dry-run validation FAILED" + echo "Check dmesg for errors:" + dmesg -T | grep -i -e nvidia-vgpu -e vgpu | tail -20 +fi + +# Unload +echo "" +echo "Unloading dry-run module..." +sudo rmmod nvidia-vgpu-vfio + +echo "" +echo "=== Dry-Run Complete ===" +echo "If validation passed, load for real with:" +echo " sudo insmod $VGPU_KERNEL/nvidia-vgpu-vfio.ko dyndbg='+p'"
\ No newline at end of file diff --git a/tools/module_ops/load_converged.sh b/tools/module_ops/load_converged.sh new file mode 100755 index 0000000..77cde09 --- /dev/null +++ b/tools/module_ops/load_converged.sh @@ -0,0 +1,95 @@ +#!/bin/bash +# Load vGPU stack in converged mode +# Usage: sudo ./load_converged.sh [dry_run] + +set -euo pipefail + +OPEN_IOV_ROOT="$OPEN_IOV_ROOT" +NVIDIA_OPEN_KERNEL="$OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open" +VGPU_KERNEL="$OPEN_IOV_ROOT/kernel" + +DRY_RUN="${1:-false}" + +echo "=== Open-IOV Converged Mode Module Load ===" +echo "Open-IOV Root: $OPEN_IOV_ROOT" +echo "Dry-run mode: $DRY_RUN" +echo "" + +# Function to check if module is loaded +is_loaded() { + lsmod | grep -q "^$1 " +} + +# Function to load module with error checking +load_module() { + local module_path=$1 + local module_name=$(basename $module_path .ko) + local params="${2:-}" + + if is_loaded "$module_name"; then + echo "Module $module_name already loaded" + return 0 + fi + + echo "Loading $module_name..." + if sudo insmod "$module_path" $params; then + echo " ✓ $module_name loaded successfully" + return 0 + else + echo " ✗ Failed to load $module_name" + return 1 + fi +} + +# 1. Load mdev core +echo "Step 1: Loading mdev core module..." +sudo modprobe mdev +echo " ✓ mdev loaded" +echo "" + +# 2. Load host nvidia.ko with vGPU support +echo "Step 2: Loading host nvidia.ko (patched open-gpu-kernel-modules)..." +load_module "$NVIDIA_OPEN_KERNEL/nvidia.ko" "NVreg_RegisterPCIDriver=1 NVreg_RmMsg=:" +echo "" + +# 3. Load dependent modules +echo "Step 3: Loading dependent modules..." +load_module "$NVIDIA_OPEN_KERNEL/nvidia-modeset.ko" +load_module "$NVIDIA_OPEN_KERNEL/nvidia-uvm.ko" +load_module "$NVIDIA_OPEN_KERNEL/nvidia-drm.ko" +load_module "$NVIDIA_OPEN_KERNEL/nvidia-peermem.ko" +echo "" + +# 4. Load nvidia-vgpu-vfio.ko +echo "Step 4: Loading nvidia-vgpu-vfio.ko..." +if [ "$DRY_RUN" = "dry_run" ] || [ "$DRY_RUN" = "true" ]; then + echo " Running in DRY-RUN mode first..." + load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p' dry_run=1" + echo " Dry-run successful, unloading..." + sudo rmmod nvidia-vgpu-vfio + echo " Loading for real..." + load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p'" +else + load_module "$VGPU_KERNEL/nvidia-vgpu-vfio.ko" "dyndbg='+p'" +fi +echo "" + +# 5. Verify +echo "Step 5: Verifying vGPU registration..." +sleep 2 +if [ -d "/sys/bus/pci/devices/0000:01:00.0/mdev_supported_types" ]; then + echo " ✓ mdev_supported_types directory exists" + ls -la /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/ +else + echo " ✗ mdev_supported_types NOT found" + echo " Check dmesg for errors: dmesg -T | grep -i vgpu" + exit 1 +fi + +echo "" +echo "=== Converged Mode Load Complete ===" +echo "" +echo "Next steps:" +echo " - Create mdev device: echo <uuid> | sudo tee /sys/bus/pci/devices/0000:01:00.0/mdev_supported_types/nvidia-vgpu-turing/create" +echo " - Run QEMU with vGPU passthrough" +echo " - Run tests: $OPEN_IOV_ROOT/tools/test/run_integration_tests.sh"
\ No newline at end of file diff --git a/tools/module_ops/unload_all.sh b/tools/module_ops/unload_all.sh new file mode 100755 index 0000000..4c73a0b --- /dev/null +++ b/tools/module_ops/unload_all.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# Unload all vGPU modules cleanly +# Usage: sudo ./unload_all.sh + +set -euo pipefail + +echo "=== Unloading Open-IOV vGPU Stack ===" + +# Unload in reverse order of loading +modules=( + "nvidia-vgpu-vfio" + "nvidia-drm" + "nvidia-modeset" + "nvidia-uvm" + "nvidia-peermem" + "nvidia" + "mdev" +) + +for module in "${modules[@]}"; do + if lsmod | grep -q "^$module "; then + echo "Unloading $module..." + if sudo rmmod "$module" 2>/dev/null; then + echo " ✓ $module unloaded" + else + echo " ⚠ $module could not be unloaded (may be in use)" + # Check refcnt + if [[ -f /sys/module/$module/refcnt ]]; then + echo " refcnt: $(cat /sys/module/$module/refcnt)" + fi + if [[ -f /sys/module/$module/initstate ]]; then + echo " initstate: $(cat /sys/module/$module/initstate)" + fi + fi + else + echo " - $module not loaded" + fi +done + +echo "" +echo "=== Unload Complete ===" +lsmod | grep -E 'nvidia|mdev' || echo "No NVIDIA/mdev modules loaded"
\ No newline at end of file diff --git a/tools/test/run_integration_tests.sh b/tools/test/run_integration_tests.sh new file mode 100755 index 0000000..06e309c --- /dev/null +++ b/tools/test/run_integration_tests.sh @@ -0,0 +1,182 @@ +#!/bin/bash +# GSP vGPU-IO Integration Test Runner + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +KERNEL_TEST_DIR="$PROJECT_ROOT/kernel/tests" +DRIVER_DIR="$PROJECT_ROOT/driver" + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1"; } + +# Check if running as root +check_root() { + if [[ $EUID -ne 0 ]]; then + log_error "This script must be run as root" + exit 1 + fi +} + +# Build kernel test module +build_test_module() { + log_info "Building kernel test module..." + cd "$KERNEL_TEST_DIR" + make clean + make + if [[ $? -ne 0 ]]; then + log_error "Failed to build test module" + exit 1 + fi + log_info "Test module built successfully" +} + +# Load test module and capture output +run_test_module() { + log_info "Loading test module..." + cd "$KERNEL_TEST_DIR" + + # Clear dmesg + dmesg -C + + # Load module + insmod gsp_vgpu_comprehensive_test.ko 2>&1 || { + log_error "Failed to load test module" + dmesg -T | tail -30 + return 1 + } + + # Wait for tests to complete + sleep 2 + + # Get test results from dmesg + dmesg -T | grep -E "(PASS|FAIL|Test Results|TEST SUITE)" | tail -100 + + # Check if tests passed + if dmesg | grep -q "TEST SUITE PASSED"; then + log_info "All tests PASSED!" + return 0 + elif dmesg | grep -q "TEST SUITE FAILED"; then + log_error "Tests FAILED!" + return 1 + else + log_warn "Could not determine test result" + return 1 + fi +} + +# Unload test module +unload_test_module() { + log_info "Unloading test module..." + rmmod gsp_vgpu_comprehensive_test 2>/dev/null || true + sleep 1 +} + +# Build driver +build_driver() { + log_info "Building vGPU-IO driver..." + cd "$DRIVER_DIR" + make clean + make + if [[ $? -ne 0 ]]; then + log_error "Failed to build driver" + exit 1 + fi + log_info "Driver built successfully" +} + +# Load driver +load_driver() { + log_info "Loading vGPU-IO driver..." + cd "$DRIVER_DIR" + insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin 2>&1 || { + log_error "Failed to load driver" + dmesg -T | tail -30 + return 1 + } + + # Check if device created + sleep 2 + if ls /dev/gsp-vgpu* 2>/dev/null; then + log_info "Driver loaded, device created" + return 0 + else + log_warn "Driver loaded but no device node found" + dmesg -T | tail -20 + fi +} + +# Unload driver +unload_driver() { + log_info "Unloading vGPU-IO driver..." + rmmod gsp-vgpu 2>/dev/null || true + sleep 1 +} + +# Run VF creation test +test_vf_creation() { + log_info "Testing VF creation via debugfs..." + + # Check debugfs + if [[ -d /sys/kernel/debug/gsp-vgpu ]]; then + log_info "Debugfs interface available" + cat /sys/kernel/debug/gsp-vgpu/vfs + else + log_warn "Debugfs not mounted or driver not loaded" + return 1 + fi +} + +# Run RPC test via debugfs +test_rpc_stats() { + log_info "Checking RPC statistics..." + if [[ -f /sys/kernel/debug/gsp-vgpu/rpc ]]; then + cat /sys/kernel/debug/gsp-vgpu/rpc + fi +} + +# Main test runner +main() { + local test_type="${1:-all}" + + check_root + + case "$test_type" in + "unit") + build_test_module + run_test_module + unload_test_module + ;; + "driver") + build_driver + load_driver + test_vf_creation + test_rpc_stats + unload_driver + ;; + "all"|*) + log_info "Running ALL tests..." + build_test_module + run_test_module + unload_test_module + + build_driver + load_driver + test_vf_creation + test_rpc_stats + unload_driver + ;; + esac + + log_info "Test run completed" +} + +main "$@"
\ No newline at end of file diff --git a/tools/vm/cloud-init.yaml b/tools/vm/cloud-init.yaml new file mode 100644 index 0000000..05f11b5 --- /dev/null +++ b/tools/vm/cloud-init.yaml @@ -0,0 +1,26 @@ +#cloud-config +users: + - name: ubuntu + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + lock_passwd: false + passwd: $6$rounds=4096$ubuntu$ubuntu + ssh_authorized_keys: [] + +package_update: true +package_upgrade: true +packages: + - linux-headers-generic + - build-essential + - dkms + - git + - wget + - curl + - vim + - pciutils + - lshw + +runcmd: + - echo "ubuntu:ubuntu" | chpasswd + - systemctl enable ssh + - systemctl start ssh diff --git a/tools/vm/launch_vgpu_qemu.sh b/tools/vm/launch_vgpu_qemu.sh new file mode 100755 index 0000000..0355ef1 --- /dev/null +++ b/tools/vm/launch_vgpu_qemu.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# QEMU launch script for NVIDIA vGPU testing +# This script assumes: +# 1. nvidia-vgpu-vfio.ko is loaded on host (with dry_run=1 for testing) +# 2. An mdev device has been created via sysfs +# 3. Guest OS has NVIDIA vGPU guest driver installed + +set -euo pipefail + +# Configuration +MDEV_UUID="${MDEV_UUID:-$(cat /proc/sys/kernel/random/uuid)}" +VGPU_TYPE="${VGPU_TYPE:-nvidia-vgpu-turing}" # or ampere, hopper, pascal +QEMU_BIN="${QEMU_BIN:-qemu-system-x86_64}" +GUEST_IMG="${GUEST_IMG:-/var/lib/libvirt/images/ubuntu-vgpu.qcow2}" +GUEST_KERNEL="${GUEST_KERNEL:-/boot/vmlinuz-$(uname -r)}" +GUEST_INITRD="${GUEST_INITRD:-/boot/initrd.img-$(uname -r)}" + +# PCI address of the NVIDIA GPU (TU104: 10de:1ed3) +GPU_PCI_ADDR="${GPU_PCI_ADDR:-0000:01:00.0}" + +echo "=== NVIDIA vGPU QEMU Launch Script ===" +echo "MDEV UUID: $MDEV_UUID" +echo "VGPU Type: $VGPU_TYPE" +echo "GPU PCI: $GPU_PCI_ADDR" +echo "" + +# Check if mdev device exists +MDEV_PATH="/sys/bus/mdev/devices/$MDEV_UUID" +if [[ ! -d "$MDEV_PATH" ]]; then + echo "ERROR: mdev device $MDEV_UUID not found" + echo "" + echo "To create an mdev device (requires NVIDIA vGPU host driver):" + echo " echo $MDEV_UUID > /sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE/create" + echo "" + echo "For dry-run testing without host driver, this script will not work for actual QEMU launch." + echo "The kernel module loads in dry-run mode but cannot create real mdev devices." + exit 1 +fi + +# Get the mdev device's VFIO group +VFIO_GROUP=$(basename $(readlink -f "$MDEV_PATH/iommu_group")) +VFIO_DEV="/dev/vfio/$VFIO_GROUP" + +if [[ ! -e "$VFIO_DEV" ]]; then + echo "ERROR: VFIO device $VFIO_DEV not found" + exit 1 +fi + +echo "Using VFIO device: $VFIO_DEV" +echo "" + +# QEMU command +exec $QEMU_BIN \ + -name "vgpu-guest" \ + -enable-kvm \ + -cpu host,host-passthrough=on \ + -smp 4,sockets=1,cores=4,threads=1 \ + -m 8G \ + -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$MDEV_UUID,display=on,x-igd-opregion=on \ + -device virtio-net-pci,netdev=net0 \ + -netdev user,id=net0,hostfwd=tcp::2222-:22 \ + -drive file=$GUEST_IMG,if=virtio,format=qcow2 \ + -display gtk,gl=on \ + -vga none \ + "$@" diff --git a/tools/vm/setup_mdev.sh b/tools/vm/setup_mdev.sh new file mode 100755 index 0000000..ef31f46 --- /dev/null +++ b/tools/vm/setup_mdev.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Setup script for NVIDIA vGPU mdev device +# This script creates an mdev device for QEMU testing + +set -euo pipefail + +GPU_PCI_ADDR="${GPU_PCI_ADDR:-0000:01:00.0}" +VGPU_TYPE="${VGPU_TYPE:-nvidia-vgpu-turing}" +MDEV_UUID="${MDEV_UUID:-$(cat /proc/sys/kernel/random/uuid)}" + +MDEV_CREATE_PATH="/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE/create" + +echo "=== NVIDIA vGPU mdev Device Setup ===" +echo "GPU PCI: $GPU_PCI_ADDR" +echo "VGPU Type: $VGPU_TYPE" +echo "MDEV UUID: $MDEV_UUID" +echo "" + +# Check if GPU device exists +if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR" ]]; then + echo "ERROR: GPU device $GPU_PCI_ADDR not found" + echo "Available NVIDIA GPUs:" + lspci -nn -d 10de: | grep -E "VGA|3D" + exit 1 +fi + +# Check if mdev_supported_types exists (requires NVIDIA vGPU host driver) +if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types" ]]; then + echo "ERROR: mdev_supported_types not found for $GPU_PCI_ADDR" + echo "" + echo "This requires the NVIDIA vGPU host driver (licensed product for data center GPUs)." + echo "The regular NVIDIA driver does not provide vGPU mdev support." + echo "" + echo "For testing on consumer GPUs (TU104, etc.), use dry-run mode:" + echo " sudo insmod /path/to/nvidia-vgpu-vfio.ko dry_run=1" + echo "" + echo "In dry-run mode, the module loads but cannot create real mdev devices." + exit 1 +fi + +# Check if requested vGPU type exists +if [[ ! -d "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/$VGPU_TYPE" ]]; then + echo "ERROR: vGPU type $VGPU_TYPE not supported" + echo "Available types:" + ls "/sys/bus/pci/devices/$GPU_PCI_ADDR/mdev_supported_types/" + exit 1 +fi + +# Create mdev device +echo "Creating mdev device..." +echo "$MDEV_UUID" | sudo tee "$MDEV_CREATE_PATH" > /dev/null + +if [[ -d "/sys/bus/mdev/devices/$MDEV_UUID" ]]; then + echo "SUCCESS: mdev device created at /sys/bus/mdev/devices/$MDEV_UUID" + echo "" + echo "Device info:" + cat "/sys/bus/mdev/devices/$MDEV_UUID/description" 2>/dev/null || echo " (no description)" + echo "" + echo "To use with QEMU:" + echo " export MDEV_UUID=$MDEV_UUID" + echo " /path/to/launch_vgpu_qemu.sh" +else + echo "ERROR: Failed to create mdev device" + exit 1 +fi |
