From 00e54c13c7c9030c4d5f7151467f2d85a3730e29 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Sun, 13 Sep 2026 18:36:21 -0400 Subject: Initial commit: Open-IOV vGPU-IO complete stack - Firmware decompilation: GSP 570.144W (28.5MB ELF, RV64IMAC), 132 decompiled C files - Kernel modules: nvidia.ko (34MB, OpenRM), open-iov-vfio.ko (2.8MB, VFIO mdev), gsp-vgpu.ko (1.8MB, guest driver) - Userspace: libiov.so/libvgpu.so (76KB), example_vgpu (28KB), rpc_emulator.c - PCI config space emulation (256B with PM, MSI, MSI-X, PCIe caps) - 12 RPC types fully mapped from firmware decompilation - BAR trap, WPR, scheduling, notification, migration emulation - Helper scripts: load_vgpu_stack, create_mdev, launch_qemu_vgpu, test_vgpu_stack, setup_guest_driver, unload_vgpu_stack - Architecture documentation - Wiki: 10 technical documents - Git LFS for large binaries (.ko, .bin, .iso, .img, .o, ghidra repos) - nvidia-open-gpu-kernel-modules as submodule --- wiki/GSP-RPC-Emulation.md | 227 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 wiki/GSP-RPC-Emulation.md (limited to 'wiki/GSP-RPC-Emulation.md') diff --git a/wiki/GSP-RPC-Emulation.md b/wiki/GSP-RPC-Emulation.md new file mode 100644 index 0000000..2e6849a --- /dev/null +++ b/wiki/GSP-RPC-Emulation.md @@ -0,0 +1,227 @@ +# GSP vGPU RPC Emulation + +## Overview + +Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace. + +## Problem + +The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware: + +| Handler | Address | Size | Purpose | +|---------|---------|------|---------| +| BAR Trap | 0x04CB8188 | 40KB | BAR1/ForceBarAccess/Aperture | +| WPR Handler 1 | 0x04F00B0C | 62KB | Write/Read Protect Regions | +| WPR Handler 2 | 0x04F00C78 | 62KB | Write/Read Protect Regions | +| Migration | 0x04179ED8/0x04179FBC | - | Live migration | + +Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188). + +## Solution: Emulation Mode + +Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Emulation Architecture │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Userspace App (libiov.so) │ +│ │ │ +│ ▼ │ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Kernel IOCTL │────▶│ Emulation Layer │ │ +│ │ Handler │ │ (kernel) │ │ +│ └─────────────────┘ └────────┬────────┘ │ +│ │ │ +│ ┌───────────────┼───────────────┐ │ +│ ▼ ▼ ▼ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ BAR Emu │ │ WPR Emu │ │ Mem Emu │ │ +│ │ (kernel) │ │ (kernel) │ │ (kernel) │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────────────────────────────────┐ │ +│ │ Userspace Emulator (libiov.so) │ │ +│ │ rpc_emu_bar_*(), rpc_emu_wpr_*() │ │ +│ └──────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## Auto-Detection (Default Behavior) + +**Emulation mode is ON by default** and auto-detects at driver load: + +1. Driver initializes RPC communication with GSP +2. Sends test `BAR_TRAP` RPC (non-destructive READ) +3. **If GSP responds** (ret=0) → vGPU handlers present → **emulation disabled** +4. **If GSP times out/errors** (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → **emulation enabled** + +``` +dmesg output: + gsp-vgpu: Auto-detecting GSP vGPU RPC handler support... + gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode + # or + gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode +``` + +## Module Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `emulation_mode` | bool | 1 (ON) | Current state (read-only, shows auto-detected value) | +| `emulation_override` | bool | 0 (auto) | Override: 1=force emulation ON, 0=force emulation OFF | + +```bash +# Default: auto-detect +sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin + +# Force emulation ON (testing with GRID GSP) +sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1 + +# Force emulation OFF (requires GRID GSP) +sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0 +``` + +## Emulated RPC Handlers + +### Kernel Emulation (`driver/`) + +| RPC Type | Handler | Emulation | +|----------|---------|-----------| +| `GSP_RPC_ALLOC` (0x01) | `gsp_vf_mem_alloc()` | Synthetic handles/VAs from local counter | +| `GSP_RPC_PIN` (0x05) | `gsp_vf_mem_pin()` | Synthetic DMA addresses from local counter | +| `GSP_RPC_MAP` (0x03) | `gsp_vf_mem_map()` | Identity map (return requested VA) | +| `GSP_RPC_BAR_TRAP` (0x0A) | `gsp_bar_trap_handler()` | Local aperture/force_access state | +| `GSP_RPC_MAP` (WPR) | `gsp_vf_wpr_protect()` | Local WPR region list | +| `GSP_RPC_UNMAP` (WPR) | `gsp_vf_wpr_unprotect()` | Local WPR region removal | + +### Userspace Emulation (`userspace/lib/rpc_emulator.c`) + +| Function | Purpose | +|----------|---------| +| `rpc_emu_bar_config()` | Local aperture offset/size + backing store | +| `rpc_emu_bar_force_access()` | Local force_access state | +| `rpc_emu_bar_save/restore()` | Migration state serialization | +| `rpc_emu_wpr_protect_write/read()` | Local WPR region tracking | +| `rpc_emu_wpr_unprotect()` | Local WPR region removal | +| `rpc_emu_wpr_set_timeout()` | Local timeout tracking | +| `rpc_emu_wpr_check_access()` | Access violation detection | + +## Emulation Data Structures + +### BAR State (per VF) +```c +struct gsp_bar_state { + uint64_t aperture_offset; // BAR1 aperture base + uint64_t aperture_size; // BAR1 aperture size (max 256KB) + bool force_access; // ForceBarAccess enabled + uint32_t force_vf; // VF holding force access + void *backing_store; // Emulated BAR memory + size_t backing_size; // Backing store size +}; +``` + +### WPR Region List (per VF) +```c +struct gsp_wpr_region { + struct list_head list; + uint64_t base; // Region base address + uint64_t size; // Region size + uint32_t flags; // GSP_WPR_WRITE/READ/EXECUTE + uint32_t vf_id; // Owning VF + void (*completion_cb)(...); // Write/read complete callback + void (*timeout_cb)(...); // Timeout callback + uint64_t timeout_ns; // Timeout in nanoseconds +}; +``` + +### Userspace Emulator State +```c +struct emu_bar_state { + uint64_t aperture_offset; + uint64_t aperture_size; + bool force_access; + uint32_t force_vf; + void *backing_store; + size_t backing_size; + pthread_mutex_t lock; +}; + +struct emu_wpr_state { + struct emu_wpr_region *regions; // Linked list + pthread_mutex_t lock; +}; +``` + +## Fallback Logic + +Userspace library (`libiov.so`) implements fallback: + +```c +vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) { + // 1. Try kernel IOCTL (sends RPC to GSP) + ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args); + + // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator + if (ret < 0 && dev->emulation_enabled && + (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) { + return rpc_emu_bar_config(vf->id, offset, size); + } + + return map_errno(errno); +} +``` + +## Limitations + +| Feature | Emulation Status | Notes | +|---------|-----------------|-------| +| BAR Aperture Config | ✅ Full | Local state + backing store | +| ForceBarAccess | ✅ Full | Per-VF exclusive access | +| BAR Save/Restore | ✅ Full | Migration support | +| WPR Write Protect | ✅ Full | Region tracking + access check | +| WPR Read Protect | ✅ Full | Region tracking + access check | +| WPR Timeout | ✅ Partial | Timeout stored, callbacks not wired | +| Memory Alloc | ✅ Basic | Synthetic handles/VAs | +| Memory Pin | ✅ Basic | Synthetic DMA addresses | +| Memory Map | ✅ Basic | Identity mapping | +| Migration | ⚠️ Partial | BAR/WPR state only | +| Scheduling | ❌ No | Returns success, no actual scheduling | +| Notifications | ❌ No | No async event delivery | + +## Testing Emulation + +```bash +# 1. Load driver (auto-enables emulation on consumer GSP) +sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin + +# 2. Verify emulation enabled in dmesg +dmesg -T | grep -i emulation +# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode + +# 3. Run example - should complete without timeouts +sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu + +# 4. Check debugfs for emulated state +cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state +cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions +``` + +## Future Work + +- [ ] Wire WPR timeout callbacks to kernel timers +- [ ] Implement emulated scheduling with work queues +- [ ] Add async notification support via eventfd +- [ ] Complete migration state for WPR regions +- [ ] Performance optimization (batch RPC emulation) +- [ ] Conformance tests vs GRID GSP behavior + +## Related Documentation + +- [BAR Emulation Specification](BAR-Emulation-Specification.md) +- [WPR Management](WPR-Management.md) +- [RPC Message Format](RPC-Message-Format.md) +- [Integration Guide](vGPU-IO-Integration-Guide.md) \ No newline at end of file -- cgit v1.2.3