# Open-IOV vGPU-IO Refactored Implementation TODO List > **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept > as a record. Current entry point: [docs/README.md](README.md). For the > current architecture and work list see > [03-architecture/01](03-architecture/01-system-overview.md), > [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md) and > [04-implementation/01](04-implementation/01-known-knowns-and-unknowns.md). ## Overview This document tracks the granular implementation steps to refactor Open-IOV from guest-side emulation to **host-side GSP emulation**, enabling unmodified guest NVIDIA drivers to run with full functionality. **Restore Point**: Git commit `f9ed5d8` - complete current stack before refactoring **Target**: Host-only GSP emulation in `open-iov-vfio.ko`, guest runs pure unmodified `nvidia.ko` --- ## Phase 0: Repository & Infrastructure Setup ### 0.1 Git Branching - [ ] `git checkout -b refactor/host-gsp-emulation` from `f9ed5d8` - [ ] Create `docs/old/` with previous ARCHITECTURE.md (DONE) - [ ] Create `docs/refactored-architecture.md` (DONE) - [ ] Create this TODO document ### 0.2 Build System Verification - [ ] Verify current build works: `make clean && make` in open-iov root - [ ] Verify all 3 kernel modules build: `nvidia.ko`, `open-iov-vfio.ko`, `gsp-vgpu.ko` - [ ] Verify userspace builds: `libiov.so`, `example_vgpu` - [ ] Document current build commands for regression testing ### 0.3 Test Infrastructure - [x] Create `scripts/test_gsp_emu.sh` - automated GSP emulation unit tests - [x] Create `scripts/test_guest_driver.sh` - guest driver validation suite - [x] Create `scripts/test_cuda_workloads.sh` - CUDA workload validation - [x] Create `scripts/test_graphics_workloads.sh` - graphics validation - [x] Create `scripts/test_windows_guest.sh` - Windows guest validation - [x] Create `scripts/test_vgpu_stack.sh` - comprehensive test runner - [ ] Set up CI pipeline for incremental testing --- ## Phase 1: Host Kernel - GSP Emulation Engine (Core) ### 1.1 New File: `kernel/gsp_emulation.h` - [x] Define `struct gsp_emu_context` with: - [x] `struct nv_vgpu_device *vgpu` - [x] `struct task_struct *thread` - [x] `void __iomem *bar0_vaddr`, `*doorbell_reg`, `*shared_mem` - [x] `struct gsp_client *rpc_client` - [x] `spinlock_t lock` - [x] `bool running` - [x] Statistics: `u64 processed_rpcs`, `failed_rpcs`, `total_latency_ns` - [x] Function prototypes: - [x] `int gsp_emu_init(struct nv_vgpu_device *vgpu)` - [x] `void gsp_emu_cleanup(struct nv_vgpu_device *vgpu)` - [x] `int gsp_emu_process_rpc(struct gsp_emu_context *ctx)` - [x] `void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id)` ### 1.2 New File: `kernel/gsp_emulation.c` - [x] Implement `gsp_emu_thread_fn()` - per-mdev polling thread - [x] Read doorbell at `bar0_vaddr + 0x110C00` - [x] ACK doorbell immediately (write 0) - [x] Call `gsp_emu_process_rpc()` - [x] Adaptive polling: busy-wait first 100μs, then `usleep_range(50, 100)` - [x] Handle `kthread_should_stop()` for clean shutdown - [x] Implement `gsp_emu_init()` - [x] Allocate `gsp_emu_context` - [x] Map BAR0, locate doorbell reg + shared mem offset (0x1000) - [x] Allocate shared memory (4KB, page-aligned, WC mapping) - [x] Map shared memory into guest BAR0 at 0x1000 - [x] Initialize `gsp_client` via `gsp_client_init()` (reuse existing) - [x] Start kthread `gsp_emu_thread_fn` - [x] Implement `gsp_emu_cleanup()` - [x] Stop kthread - [x] Free shared memory - [x] Call `gsp_client_fini()` - [x] Free context - [x] Implement `gsp_emu_process_rpc()` - dispatch all 12 RPC types - [x] Copy request from shared memory (offset 0x0000) - [x] Validate seq_id != 0 - [x] Switch on `req.type` → call appropriate `gsp_rpc_*` function - [x] Write response to shared memory (offset 0x0080) - [x] Call `gsp_emu_inject_completion_interrupt()` - [x] Implement `gsp_emu_inject_completion_interrupt()` - [x] Write to MSI-X PBA at `bar0_vaddr + 0x3000` (vector 0) - [x] Signal eventfd: `eventfd_signal(vgpu->msix_trigger[0])` ### 1.3 Modify `kernel/main.c` - [x] Add `struct gsp_emu_context *gsp_emu` to `struct nv_vgpu_device` (reused `gsp_comm_ctx`) - [x] In `nv_vgpu_device_init_work()`: call `gsp_emu_init(vgpu)` after `nv_vgpu_gsp_init()` - [x] In `nv_vgpu_device_cleanup_work()`: call `gsp_emu_cleanup(vgpu)` before `nv_vgpu_gsp_cleanup()` - [x] Include `gsp_emulation.h` ### 1.4 Modify `kernel/Makefile` - [x] Add `gsp_emulation.o` to `open-iov-vfio-y` or module objects - [x] Verify build: `make clean && make` ### 1.5 Unit Tests for GSP Emulation (kernel) - [x] Create `kernel/tests/gsp_emulation_test.c` - [x] Create `kernel/tests/Makefile` - [x] Test ALLOC/FREE round-trip - [x] Test MAP/UNMAP with various sizes/alignments - [x] Test PIN/UNPIN returns valid DMA addresses - [x] Test SCHEDULE/YIELD completes - [x] Test BAR_TRAP read/write emulation - [x] Test NOTIFY register/unregister - [x] Test MIGRATE 4-phase state machine - [x] Test concurrent RPCs from multiple threads - [x] Test error handling (invalid seq_id, bad payload) - [x] Add to `kernel/tests/Makefile` --- ## Phase 2: PCI Config Space & BAR Enhancements ### 2.1 Enhance PCI Config Space (`kernel/main.c`) - [x] Verify MSI-X capability at 0x70 points to Table@BAR0+0x2000, PBA@BAR0+0x3000 - [x] Ensure `VFIO_PCI_CONFIG_REGION_INDEX` returns full 256B config space - [x] Add config space write handlers for: - [x] Command register (0x04): Memory/IO/BusMaster enable - [x] MSI-X Message Control (0x72): Enable/disable, table size - [x] MSI-X Table Offset (0x74-0x77): Verify BAR0+0x2000 - [x] MSI-X PBA Offset (0x78-0x7B): Verify BAR0+0x3000 ### 2.2 BAR Region VFIO Mapping (`kernel/bar_emulation.c`) - [x] Implement `VFIO_PCI_BAR0_REGION_INDEX` → 16MB emulated config space - [x] Implement `VFIO_PCI_BAR1_REGION_INDEX` → 256MB VRAM (backed by host allocation) - [x] Implement `VFIO_PCI_BAR2_REGION_INDEX` → 32MB reserved - [x] Implement `VFIO_PCI_BAR3_REGION_INDEX` → 32MB reserved - [x] Implement `VFIO_PCI_MSIX_REGION_INDEX` → MSI-X Table (0x2000) + PBA (0x3000) - [x] Ensure BAR0 mmap includes doorbell @ 0x110C00 and shared mem @ 0x1000 ### 2.3 MSI-X Table/PBA Emulation - [x] Allocate MSI-X table (32 entries × 16 bytes = 512 bytes) in BAR0 at 0x2000 - [x] Allocate PBA (32 bits = 4 bytes, aligned to 64 bytes) in BAR0 at 0x3000 - [x] Guest writes to MSI-X table entries → host stores for interrupt injection - [x] Guest reads PBA → host returns pending bits --- ## Phase 3: GSP RPC Handler Completion ### 3.1 Verify All 12 RPC Handlers in `gsp_rpc_client.c` - [x] ALLOC (0x01): VRAM allocation via host nvidia.ko - [x] FREE (0x02): Free allocation - [x] MAP (0x03): Map to GPU VA - [x] UNMAP (0x04): Unmap GPU VA - [x] PIN (0x05): Pin for DMA, return DMA address - [x] UNPIN (0x06): Unpin pages - [x] SCHEDULE (0x07): Submit work to local workqueue - [x] YIELD (0x08): No-op success - [x] NOTIFY (0x09): Async notification registration - [x] BAR_TRAP (0x0A): Shadow register read/write - [x] MIGRATE (0x0B): 4-phase protocol - [x] DEBUG (0xFF): Firmware introspection ### 3.2 Enhance Handler Implementations - [x] ALLOC: Use host page allocation + DMA mapping - [x] MAP: GPU VA mapping management - [x] PIN: Return DMA address from mapping - [x] BAR_TRAP: Use bar_emulation.c shadow memory - [x] MIGRATE: Use migration.c 4-phase state machine - [x] NOTIFY: Use gsp_rpc_notify_register() + eventfd for guest delivery ### 3.3 Add RPC Validation & Error Handling - [x] Sequence ID tracking (detect duplicates, gaps) - [x] Payload size validation per RPC type - [x] Timeout handling (guest watchdog) - [x] Proper error codes in response.status --- ## Phase 4: Portable Architecture Implementation ### 4.1 Firmware Analyzer (`kernel/firmware_analyzer.c`) - [x] Compute firmware CRC32 and SHA256 signatures - [x] Pattern matching for RPC dispatcher discovery - [x] Pattern matching for jump table discovery - [x] Pattern matching for ALLOC handler discovery - [x] Known firmware signature database - [x] Auto-discovery fallback for unknown firmware - [x] `fw_analyzer_analyze()` - analyze firmware blob - [x] `fw_analyzer_load_and_analyze()` - load and analyze from filesystem - [x] `fw_analyzer_print_result()` - debug output ### 4.2 Dynamic RPC Dispatcher (`kernel/dynamic_rpc_dispatcher.c`) - [x] Version-specific interface registry (550, 560, 570, 580) - [x] Native firmware handler function pointers - [x] Emulation fallback interface - [x] Automatic interface selection based on firmware analysis - [x] Per-RPC-type emulation toggle - [x] Dispatcher API for all 12 RPC types - [x] `dynamic_dispatcher_init()` - initialize with firmware path - [x] `dynamic_dispatcher_get_interface()` - get current interface - [x] `dynamic_dispatcher_use_emulation()` - check if emulation used for type ### 4.3 Hardware Architecture Layer (`kernel/architecture/`) - [x] `architecture.h` - hardware abstraction API - [x] `architecture.c` - architecture implementations - [x] Turing architecture (TU102/TU104) config - [x] Ampere architecture (GA100/GA102/GA104) config - [x] Blackwell architecture (GB100/GB202) config - [x] Architecture lookup by name and PCI ID - [x] BAR configuration queries - [x] Doorbell/shared memory offset queries - [x] MSI-X vector count queries - [x] Feature support queries (MIG, SR-IOV) ### 4.4 Driver Version Adaptation Layer (`kernel/version/`) - [x] `version.h` - version adaptation API - [x] `version.c` - version implementations - [x] Version 550 interface - [x] Version 560 interface - [x] Version 570 interface - [x] Version 580 interface - [x] Feature detection per version - [x] Interface lookup by version string ### 4.5 Declarative Configuration Files - [x] `config/hardware/turing.yaml` - Turing hardware config - [x] `config/hardware/ampere.yaml` - Ampere hardware config - [x] `config/hardware/blackwell.yaml` - Blackwell hardware config - [x] `config/versions/550.yaml` - Driver 550 config - [x] `config/versions/560.yaml` - Driver 560 config - [x] `config/versions/570.yaml` - Driver 570 config - [x] `config/versions/580.yaml` - Driver 580 config --- ## Phase 5: Guest Driver Integration Testing ### 4.1 Host Module Loading Sequence - [ ] `sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko` - [ ] `sudo insmod kernel/open-iov-vfio.ko` - [ ] Verify mdev types appear: `ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/` - [ ] Create mdev: `echo | sudo tee /sys/bus/pci/devices/.../mdev_supported_types/nvidia-vgpu-turing/create` ### 4.2 QEMU Launch with mdev - [ ] `qemu-system-x86_64 -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/,display=off` - [ ] Verify guest sees PCI device: `lspci -d 10de:` → should show 10de:1ed3 - [ ] Verify BARs: `lspci -v -d 10de:` → BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB - [ ] Verify MSI-X: `lspci -v -d 10de:` → MSI-X capability present ### 4.3 Guest NVIDIA Driver Installation - [ ] Install OpenRM 570.144 in guest: `./NVIDIA-Linux-x86_64-570.144.run --silent --dkms` - [ ] Install firmware: Copy `bootloader`, `booter_load`, `gsp` binaries to `/lib/firmware/nvidia/tu102/gsp/` - [ ] Load driver: `sudo modprobe nvidia` - [ ] Verify: `nvidia-smi` shows vGPU (T4 or equivalent) - [ ] Verify: `dmesg | grep -i nvidia` shows successful probe ### 4.4 Basic Functionality Tests - [ ] `nvidia-smi -q` shows all fields populated - [ ] `nvidia-smi dmon` shows utilization counters - [ ] `nvidia-smi mig` shows MIG not supported (expected for vGPU) - [ ] `cat /proc/driver/nvidia/gpus/0000:00:00.0/information` shows correct info --- ## Phase 5: CUDA Workload Validation ### 5.1 CUDA SDK Tests - [ ] Compile and run `vectorAdd` sample - [ ] Compile and run `matrixMul` sample - [ ] Compile and run `simpleCUBLAS` sample - [ ] Compile and run `simpleCUFFT` sample - [ ] Verify numerical correctness vs CPU reference ### 5.2 cuBLAS / cuDNN Tests - [ ] SGEMM/DGEMM various sizes (64x64 to 4096x4096) - [ ] Batched GEMM - [ ] cuDNN convolution forward/backward - [ ] cuDNN RNN/LSTM - [ ] Performance within 10% of native ### 5.3 PyTorch / TensorFlow Integration - [ ] PyTorch: `torch.cuda.is_available()`, tensor ops, autograd - [ ] PyTorch: Training loop (ResNet-50, BERT-small) - [ ] TensorFlow: `tf.config.list_physical_devices('GPU')` - [ ] TensorFlow: Keras model training - [ ] Mixed precision (FP16/BF16) works ### 5.4 Multi-GPU / NCCL Tests - [ ] Create 2+ vGPUs on same physical GPU - [ ] NCCL all-reduce across vGPUs - [ ] NCCL broadcast/reduce - [ ] Multi-process CUDA (MPS) if supported ### 5.5 CUDA Graphs & Streams - [ ] CUDA Graph capture/replay - [ ] Multiple streams concurrent execution - [ ] Stream priorities - [ ] Event synchronization --- ## Phase 6: Graphics Workload Validation ### 6.1 Vulkan Tests - [ ] `vulkaninfo` shows GPU properties - [ ] `vkcube` renders correctly - [ ] `vkcubepp` (Vulkan 1.1+) renders - [ ] Sascha Willems Vulkan samples (deferred, compute, ray tracing if supported) - [ ] VK_KHR_swapchain, VK_KHR_dedicated_allocation work ### 6.2 OpenGL Tests - [ ] `glxinfo | grep -i nvidia` shows correct renderer - [ ] `glxgears` runs at expected FPS - [ ] `glmark2` benchmark completes - [ ] Piglit test suite (subset) passes ### 6.3 Video Encode/Decode - [ ] FFmpeg + NVENC: H.264 encode 1080p60 - [ ] FFmpeg + NVENC: HEVC encode 4K30 - [ ] FFmpeg + NVDEC: H.264 decode 1080p60 - [ ] FFmpeg + NVDEC: HEVC decode 4K30 - [ ] GStreamer nvh264enc/nvh265enc elements - [ ] GStreamer nvh264dec/nvh265dec elements ### 6.4 Windows Guest Graphics (Separate Track) - [ ] Windows 11 guest with NVIDIA 570.144 driver - [ ] Device Manager: "NVIDIA T4" (or equivalent) with no error codes - [ ] GPU-Z / CUDA-Z detect vGPU correctly - [ ] D3D11: `D3D11CreateDevice` succeeds, render triangle - [ ] D3D12: `D3D12CreateDevice` succeeds, render triangle - [ ] Vulkan on Windows: `vkCreateInstance`, render - [ ] DXGI swapchain present works - [ ] Video encode via Media Foundation (MFT) - [ ] Video decode via Media Foundation (MFT) --- ## Phase 7: Performance Benchmarking ### 7.1 Microbenchmarks - [ ] GSP RPC round-trip latency (target < 10μs) - [ ] BAR register access latency (target < 2μs) - [ ] VRAM alloc/free latency (target < 50μs) - [ ] VRAM map/unmap latency (target < 20μs) - [ ] DMA pin/unpin latency (target < 30μs) - [ ] MSI-X injection latency (target < 5μs) ### 7.2 Macrobenchmarks - [ ] CUDA kernel launch overhead (target < 5% vs native) - [ ] cuBLAS SGEMM 4096x4096 (target > 90% native GFLOPS) - [ ] cuDNN ResNet-50 training step (target > 85% native) - [ ] Vulkan frame time (target < 2ms @ 1080p) - [ ] NVENC 1080p60 encode latency (target < 5ms) - [ ] NVDEC 1080p60 decode latency (target < 3ms) ### 7.3 Stress Tests - [ ] 24-hour CUDA compute stress (no errors, no memory leaks) - [ ] 24-hour graphics stress (Unigine Heaven/Valley loop) - [ ] 1000x VF create/destroy cycles - [ ] Concurrent CUDA + graphics workloads - [ ] Memory pressure test (allocate 90% VRAM, verify OOM handling) --- ## Phase 8: Live Migration Validation ### 8.1 Migration Protocol Tests - [ ] QUERY phase: source reports capabilities - [ ] PREPARE phase: destination allocates resources - [ ] TRANSFER phase: iterative VRAM + GSP state copy - [ ] COMMIT phase: cutover, guest resumes on destination - [ ] CLEANUP phase: source releases resources ### 8.2 Migration Under Load - [ ] Migrate idle vGPU (downtime < 50ms) - [ ] Migrate running CUDA kernel (downtime < 100ms) - [ ] Migrate running graphics (no visual artifacts) - [ ] Migrate with pinned DMA buffers - [ ] Verify post-migration CUDA/graphics work correctly ### 8.3 Migration State Verification - [ ] GSP context (registers, PC, stack) restored - [ ] Page tables identical - [ ] VRAM contents identical (checksum) - [ ] BAR register state identical - [ ] MSI-X table/PBA state identical --- ## Phase 9: Windows Guest Validation ### 9.1 Windows Guest Setup - [ ] Windows 10 22H2 / Windows 11 23H2 guest - [ ] VirtIO drivers for network/storage - [ ] QEMU guest agent - [ ] UEFI firmware (OVMF with TPM) ### 9.2 NVIDIA Driver Installation - [ ] Download 570.144 Windows driver - [ ] Install with `setup.exe -s -noreboot` (silent) - [ ] Reboot guest - [ ] Verify in Device Manager: "NVIDIA T4" (Code 0 = working) ### 9.3 Windows Compute Tests - [ ] `nvidia-smi.exe` shows vGPU - [ ] CUDA-Z detects GPU, runs device query - [ ] `vectorAdd.exe` (compiled in guest) executes - [ ] PyTorch/TensorFlow in Windows (WSL2 or native) ### 9.4 Windows Graphics Tests - [ ] `dxdiag` shows correct GPU info - [ ] D3D11 tutorial samples render - [ ] D3D12 tutorial samples render - [ ] Vulkan SDK samples run - [ ] FurMark / 3DMark stress test ### 9.5 Windows Video Tests - [ ] Media Foundation H.264/HEVC encode - [ ] Media Foundation H.264/HEVC decode - [ ] NVENC API direct usage - [ ] NVDEC API direct usage --- ## Phase 10: Reliability & Production Hardening ### 10.1 Error Handling & Recovery - [ ] Guest driver crash → host cleans up mdev state - [ ] Host kernel oops in GSP emulation → watchdog restarts thread - [ ] Physical GPU reset → host reinitializes GSP emulation - [ ] IOMMU fault → proper error to guest - [ ] VRAM OOM → proper CUDA_OUT_OF_MEMORY to guest ### 10.2 Monitoring & Observability - [ ] Debugfs interface: `/sys/kernel/debug/open-iov-vfio/` - [ ] Per-mdev RPC statistics - [ ] Latency histograms - [ ] Error counters - [ ] GSP emulation thread CPU usage - [ ] Prometheus metrics exporter (optional) - [ ] Tracepoints for RPC processing ### 10.3 Security Hardening - [ ] Validate all guest inputs (seq_id, payload sizes, pointers) - [ ] Isolate mdev memory (no cross-VF access) - [ ] Rate-limit doorbell writes (prevent DoS) - [ ] Audit VFIO ioctl permissions ### 10.4 Documentation & Handoff - [ ] Update `docs/refactored-architecture.md` with as-built details - [ ] Create `docs/OPERATIONS.md` for deployment - [ ] Create `docs/TROUBLESHOOTING.md` for common issues - [ ] Record all firmware addresses used - [ ] Document test results for each phase --- ## Success Criteria Checklist ### Minimum Viable Product (Phases 0-4) - [ ] Host loads `open-iov-vfio.ko` with GSP emulation thread - [ ] mdev creates successfully with correct PCI config space - [ ] Guest `nvidia.ko` loads, probes vGPU successfully - [ ] `nvidia-smi` in guest shows vGPU - [ ] Simple CUDA kernel (`vectorAdd`) executes correctly ### Compute Ready (Phases 5-6) - [ ] cuBLAS/cuDNN workloads pass numerical verification - [ ] PyTorch/TensorFlow training works - [ ] Vulkan/OpenGL rendering works - [ ] Video encode/decode works - [ ] Performance > 85% native for compute, > 90% for graphics ### Production Ready (Phases 7-10) - [ ] 24-hour stress tests pass - [ ] Live migration works under load - [ ] Windows guest fully functional - [ ] Monitoring/observability in place - [ ] Security audit passed - [ ] Documentation complete --- ## Dependencies & Prerequisites | Component | Version | Source | |-----------|---------|--------| | Linux Kernel | 6.8+ | Host distro | | NVIDIA OpenRM | 570.144 | `nvidia-open-gpu-kernel-modules` submodule | | QEMU | 8.0+ | Host distro / source | | libvfio-user | Latest | `libvfio-user` repo | | CUDA Toolkit | 12.4+ | Guest (for compilation) | | Vulkan SDK | 1.3+ | Guest | | Windows 10/11 | 22H2/23H2 | Guest ISO | --- ## Tracking | Phase | Status | Start Date | End Date | Notes | |-------|--------|------------|----------|-------| | 0: Setup | ✅ DONE | - | - | Git repo, docs, restore point | | 1: GSP Emulation Engine | ✅ DONE | | | Core implementation complete | | 2: PCI/BAR Enhancements | ✅ DONE | | | Config space, MSI-X, BAR regions | | 3: RPC Handler Completion | ✅ DONE | | | All 12 RPC types with emulation | | 4: Portable Architecture | ✅ DONE | | | Firmware analyzer, dispatcher, arch/version layers, configs | | 5: Guest Driver Integration | 🔄 IN PROGRESS | | | Requires reboot to test | | 6: CUDA Validation | ⏳ PENDING | | | | | 7: Graphics Validation | ⏳ PENDING | | | | | 8: Performance Benchmarks | ⏳ PENDING | | | | | 9: Migration Validation | ⏳ PENDING | | | | | 10: Windows Guest | ⏳ PENDING | | | | | 11: Production Hardening | ⏳ PENDING | | | | | 10: Production Hardening | ⏳ PENDING | | | | --- *TODO List Version: 1.0* *Repository: $OPEN_IOV_ROOT* *Branch: refactor/host-gsp-emulation* *Base Commit: f9ed5d8*