# Open-IOV vGPU-IO Refactored Architecture ## System Design from First Principles ### 1. Problem Statement **Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with full GPU functionality (CUDA, graphics, video encode/decode, AI/ML) on Turing GPUs (TU102/TU104) without requiring: - Proprietary GRID host drivers - Modified guest drivers - Paravirtualization interfaces - Guest-side Open-IOV code **Constraints**: - Guest runs stock `nvidia.ko` + `libcuda.so` + `libnvidia-ml.so` + `libnvidia-opencl.so` - Host runs OpenRM `nvidia.ko` (for physical GPU management) + Open-IOV hypervisor components - No guest kernel modifications, no guest userspace Open-IOV libraries - High performance: near-native GPU utilization for compute and graphics workloads - Support both Linux and Windows guests ### 2. Core Design Principle: Host-Side GSP Emulation The fundamental insight: **The GSP (GPU System Processor) is a RISC-V microcontroller running firmware that implements 12 RPC types. We can emulate this firmware entirely in the host kernel.** ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ FIRST PRINCIPLE: GSP IS JUST AN RPC SERVER │ ├─────────────────────────────────────────────────────────────────────────────────┤ │ │ │ Physical Reality: │ │ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ │ │ Guest RM │ ─────────────────────────────► │ GSP Firmware │ │ │ │ (nvidia.ko)│ ◄───────────────────────────── │ (570.144W, RV64IMAC) │ │ │ └─────────────┘ Responses └─────────────────────────┘ │ │ │ │ │ │ │ 12 RPC Types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │ │ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │ │ │ │ │ │ ▼ │ │ │ Virtualized Reality: │ │ │ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ │ │ Guest RM │ ─────────────────────────────► │ Host Kernel │ │ │ │ (nvidia.ko)│ ◄───────────────────────────── │ (open-iov-vfio.ko) │ │ │ └─────────────┘ Responses │ gsp_rpc_client.c │ │ │ │ │ (12 emulated handlers)│ │ │ │ Guest sees identical └─────────────────────────┘ │ │ │ interface - cannot tell │ │ ▼ │ │ No guest-side code required. │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ ``` ### 3. System Architecture Overview ``` ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ HOST SYSTEM ║ ║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ ║ ║ USERSPACE ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ QEMU/KVM + libvfio-user │ ║ ║ ║ ║ │ • -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/ │ ║ ║ ║ ║ │ • VFIO region mmap, ioctl, eventfd for IRQs │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ mdev management CLI / mdev management daemon (Management Tools) │ ║ ║ ║ ║ │ • Link against: libvgpu.so (host-side) │ ║ ║ ║ ║ │ • VF lifecycle: create/destroy, resource allocation, migration orchestration │ ║ ║ ║ ║ │ • Calls: vgpu_open(), vgpu_vf_create(), vgpu_mem_alloc(), vgpu_bar_config(), │ ║ ║ ║ ║ │ vgpu_wpr_protect(), vgpu_schedule(), vgpu_migrate_start(), etc. │ ║ ║ ║ ║ │ • Communicates via VFIO ioctls on /dev/vfio/ │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ libvgpu.so / libiov.so (HOST-SIDE ONLY) │ ║ ║ ║ ║ │ • Full vGPU management API: vgpu_*() functions │ ║ ║ ║ ║ │ • Implements: VF create/destroy, memory ops, BAR config, WPR, scheduling, │ ║ ║ ║ ║ │ notifications, migration │ ║ ║ ║ ║ │ • rpc_emulator.c: Host-side emulation fallbacks for management operations │ ║ ║ ║ ║ │ (rpc_emu_bar_config/save/restore, rpc_emu_wpr_*, rpc_emu_schedule/yield, │ ║ ║ ║ ║ │ rpc_emu_notify_register/unregister/send, rpc_emu_migrate_*) │ ║ ║ ║ ║ │ • NO GSP RPC emulation here - that lives in kernel │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ ║ ║ ║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ ║ ║ KERNEL SPACE ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ nvidia.ko (Host RM - OpenRM 570.144) │ ║ ║ ║ ║ │ • Boots physical GSP firmware (570.144W) │ ║ ║ ║ ║ │ • Manages physical GPU resources (VRAM, engines, display) │ ║ ║ ║ ║ │ • Exports vGPU VFIO symbols: │ ║ ║ ║ ║ │ - nvidia_vgpu_vfio_get_ops() / nvidia_vgpu_vfio_set_ops() │ ║ ║ ║ ║ │ - nv_vgpu_mdev_register_parent() / nv_vgpu_mdev_unregister_parent() │ ║ ║ ║ ║ │ • GPC callbacks from open-iov-vfio.ko: vgpu_create, vgpu_delete, vgpu_bar_info, │ ║ ║ ║ ║ │ vgpu_start, get_name, get_description, get_instances, get_sparse_mmap, update_request │ ║ ║ ║ ║ │ • Owns physical MSI-X interrupt │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ open-iov-vfio.ko (HYPERVISOR - Core Component) │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ mdev Parent Registration │ │ ║ ║ ║ ║ │ │ • Registers with nvidia.ko via nv_vgpu_mdev_register_parent() │ │ ║ ║ ║ ║ │ │ • Creates mediated device types: nvidia-vgpu-turing, nvidia-vgpu-ampere, etc. │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ PCI Config Space Emulation (256 bytes per mdev) │ │ ║ ║ ║ ║ │ │ • Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU), Class=0x030000 │ │ ║ ║ ║ ║ │ │ • BAR0=16MB (64-bit, prefetchable), BAR1=256MB (64-bit, prefetchable) │ │ ║ ║ ║ ║ │ │ • BAR2=32MB, BAR3=32MB (64-bit) │ │ ║ ║ ║ ║ │ │ • Capabilities: PM (0x40), MSI (0x50), MSI-X (0x70), PCIe (0x80) │ │ ║ ║ ║ ║ │ │ • MSI-X Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ │ ║ ║ ║ ║ │ │ • VFIO_CONFIG_REGION_INDEX mapped to this emulated config space │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ BAR Region Emulation (VFIO Region Indices) │ │ ║ ║ ║ ║ │ │ • VFIO_PCI_BAR0_REGION_INDEX → 16MB emulated config space │ │ ║ ║ ║ ║ │ │ • VFIO_PCI_BAR1_REGION_INDEX → 256MB emulated framebuffer (backed by host VRAM allocation) │ │ ║ ║ ║ ║ │ │ • VFIO_PCI_BAR2_REGION_INDEX → 32MB reserved │ │ ║ ║ ║ ║ │ │ • VFIO_PCI_BAR3_REGION_INDEX → 32MB reserved │ │ ║ ║ ║ ║ │ │ • VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table (0x2000) + PBA (0x3000) in BAR0 │ │ ║ ║ ║ ║ │ │ • BAR trap handling: 64 trapped regions covering all BAR0 registers (bar_emulation.c) │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ GSP EMULATION ENGINE (NEW - Core Innovation) │ │ ║ ║ ║ ║ │ │ │ │ ║ ║ ║ ║ │ │ Per-mdev GSP Emulation Thread: │ │ ║ ║ ║ ║ │ │ ```c │ │ ║ ║ ║ ║ │ │ static int gsp_emu_thread(void *data) { │ │ ║ ║ ║ ║ │ │ struct nv_vgpu_device *vgpu = data; │ │ ║ ║ ║ ║ │ │ while (!kthread_should_stop()) { │ │ ║ ║ ║ ║ │ │ u32 doorbell = readl(vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); │ │ ║ ║ ║ ║ │ │ if (doorbell) { │ │ ║ ║ ║ ║ │ │ writel(0, vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); // ACK │ │ ║ ║ ║ ║ │ │ gsp_emu_process_rpc(vgpu); // Process via gsp_rpc_client.c │ │ ║ ║ ║ ║ │ │ } │ │ ║ ║ ║ ║ │ │ usleep_range(100, 200); // 1-2μs polling interval │ │ ║ ║ ║ ║ │ │ } │ │ ║ ║ ║ ║ │ │ return 0; │ │ ║ ║ ║ ║ │ │ } │ │ ║ ║ ║ ║ │ │ ``` │ │ ║ ║ ║ ║ │ │ │ │ ║ ║ ║ ║ │ │ GSP Shared Memory Management: │ │ ║ ║ ║ ║ │ │ • Allocates contiguous host memory for GSP mailbox (4KB aligned) │ │ ║ ║ �b ║ │ │ • Maps into guest BAR0 at GSP_SHARED_MEM_OFFSET (0x1000) │ │ ║ ║ ║ ║ │ │ • WC (Write-Combining) mapping for performance │ │ ║ ║ ║ ║ │ │ • Cache coherency: clflush on write, memory barriers │ │ ║ ║ ║ ║ │ │ • Matches firmware mailbox layout exactly (from decompilation) │ │ ║ ║ ║ ║ │ │ │ │ ║ ║ ║ ║ │ │ RPC Handler Suite (gsp_rpc_client.c - ALL 12 TYPES): │ │ ║ ║ ║ ║ │ │ ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ │ ║ ║ ║ │ │ │ RPC Type │ Emulated Implementation │ │ │ ║ ║ ║ │ │ ├─────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ │ ║ ║ ║ │ │ │ ALLOC (0x01)│ VRAM allocation via host nvidia.ko, returns GPU VA + handle │ │ │ ║ ║ ║ │ │ │ FREE (0x02) │ Free allocation, unmap pages │ │ │ ║ ║ ║ │ │ │ MAP (0x03) │ Map allocation to GPU VA, update page tables │ │ │ ║ ║ ║ │ │ │ UNMAP (0x04)│ Unmap GPU VA range │ │ │ ║ ║ ║ │ │ │ PIN (0x05) │ Pin pages for DMA, return DMA address │ │ │ ║ ║ ║ │ │ │ UNPIN (0x06)│ Unpin pages │ │ │ ║ ║ ║ │ │ │ SCHEDULE(0x07)│ Submit work to GSP scheduler (local workqueue) │ │ │ ║ ║ ║ │ │ │ YIELD (0x08)│ Timeslice yield (no-op success) │ │ │ ║ ║ ║ │ │ │ NOTIFY(0x09)│ Async notification registration, eventfd signaling │ │ │ ║ ║ ║ │ │ │ BAR_TRAP(0x0A)│ BAR register read/write emulation via shadow memory │ │ │ ║ ║ ║ │ │ │ MIGRATE(0x0B)│ 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP) │ │ │ ║ ║ ║ │ │ │ DEBUG (0x0C)│ Debug interface, firmware introspection │ │ │ ║ ║ ║ │ │ └─────────────┴──────────────────────────────────────────────────────────────────────────────────┘ │ │ ║ ║ ║ │ │ │ │ ║ ║ ║ │ │ Firmware Address Mapping (from decompilation): │ │ ║ ║ ║ │ │ • Primary Dispatcher: 0x057A1AC0 (32 sub-commands) │ │ ║ ║ ║ │ │ • Secondary Dispatcher: 0x04ED39DC │ │ ║ ║ ║ │ │ • BAR Trap Handler: 0x04CB8188 │ │ ║ ║ ║ │ │ • WPR Handlers: 0x04F00B0C, 0x04F00C78 │ │ ║ ║ ║ │ │ • ALLOC Main: 0x057E94EC │ │ ║ ║ ║ │ │ • SCHEDULE Main: 0x051EFBF4 │ │ ║ ║ ║ │ │ • Migration Tables: 0x04179ED8, 0x04179FBC (16 entries each) │ │ ║ ║ ║ │ │ • RPC Jump Table: 0x059BFEB0 (12 handlers) │ │ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ MSI-X Interrupt Injection │ │ ║ ║ ║ ║ │ │ • When emulated GSP completes RPC → write to MSI-X PBA → eventfd_signal() │ │ ║ ║ ║ ║ │ │ • QEMU/KVM delivers MSI-X interrupt to guest nvidia.ko ISR │ │ ║ ║ ║ ║ │ │ • Guest sees normal GSP completion interrupt │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ Memory Management (memory.c) │ │ ║ ║ ║ ║ │ │ • DMA mapping via IOMMU (VFIO DMA API) │ │ ║ ║ ║ ║ │ │ • Pin/unpin for device DMA │ │ ║ ║ ║ ║ │ │ • VRAM allocation via host nvidia.ko │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ │ │ ║ ║ ║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ ║ ║ │ │ Live Migration (migration.c) │ │ ║ ║ ║ ║ │ │ • 4-phase: QUERY → PREPARE → TRANSFER → COMMIT → CLEANUP │ │ ║ ║ ║ ║ │ │ • State save/restore: GSP context, page tables, VRAM contents, BAR state │ │ ║ ║ ║ ║ │ │ • Dual jump tables from firmware (0x04179ED8, 0x04179FBC) │ │ ║ ║ ║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ ║ ║ ║ PHYSICAL GPU: TU102/TU104 (RTX 2080 Ti / T4 / RTX 2080 Super) ║ ║ • REAL GSP Firmware 570.144W running on FALCON RISC-V ║ ║ • Used by host nvidia.ko for physical GPU management ║ ║ • NOT used for guest vGPU RPCs (those are emulated in host kernel) ║ ║ ║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ │ VFIO MEDIATED DEVICE (mdev) │ │ /sys/bus/mdev/devices// │ ▼ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ GUEST VM ║ ║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ ║ ║ USERSPACE ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ Guest Applications │ ║ ║ ║ ║ │ • CUDA Applications (cuBLAS, cuDNN, custom kernels) │ ║ ║ ║ ║ │ • Graphics: OpenGL, Vulkan, DirectX (via WINE/Proton on Linux, native on Windows) │ ║ ║ ║ ║ │ • Video: NVENC/NVDEC (FFmpeg, GStreamer, Media Foundation) │ ║ ║ ║ ║ │ • AI/ML: TensorRT, PyTorch, TensorFlow │ ║ ║ ║ ║ │ • NO Open-IOV libraries, NO modified code │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ Standard NVIDIA Userspace Drivers (UNMODIFIED) │ ║ ║ ║ ║ │ • libcuda.so / libcuda.so.1 (CUDA Driver API) │ ║ ║ ║ ║ │ • libnvidia-ml.so / libnvidia-ml.so.1 (NVML) │ ║ ║ ║ ║ │ • libnvidia-opencl.so / libnvidia-opencl.so.1 (OpenCL) │ ║ ║ ║ ║ │ • libnvidia-vulkan.so / libGLX_nvidia.so (Vulkan/OpenGL) │ ║ ║ ║ ║ │ • libnvidia-encode.so / libnvidia-decode.so (Video) │ ║ ║ ║ ║ │ • nvidia-smi, nvidia-settings, nvidia-debugdump │ ║ ║ ║ ║ │ • ALL communicate with guest nvidia.ko via /dev/nvidia* ioctls │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ ║ ║ ║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ ║ ║ KERNEL SPACE ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ nvidia.ko (Guest RM - UNMODIFIED OpenRM 570.144) │ ║ ║ ║ ║ │ • PCI Probe: detects 10de:1ed3 (TU104 vGPU) from emulated config space │ ║ ║ ║ ║ │ • BAR Mapping: │ ║ ║ ║ ║ │ - BAR0 (16MB): Config registers, GSP doorbell @ 0x110c00, shared mem @ 0x1000 │ ║ ║ ║ ║ │ - BAR1 (256MB): Framebuffer (VRAM) │ ║ ║ ║ ║ │ - BAR2 (32MB), BAR3 (32MB): Reserved │ ║ ║ ║ ║ │ • MSI-X Setup: Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ ║ ║ ║ ║ │ • GSP Client Initialization: │ ║ ║ ║ ║ │ - Writes to doorbell (queue head) to submit RPCs │ ║ ║ ║ ║ │ - Reads/writes shared memory for RPC messages │ ║ ║ ║ ║ │ - Waits for MSI-X interrupt on completion │ ║ ║ ║ ║ │ • RM Subsystems (all functional): │ ║ ║ ║ ║ │ - Memory: alloc, free, map, unmap, pin, unpin via GSP RPC │ ║ ║ ║ ║ │ - Scheduling: work submission, yield, priority via GSP RPC │ ║ ║ ║ ║ │ - BAR: register access via BAR_TRAP RPC │ ║ ║ ║ ║ │ - WPR: protect/unprotect via GSP RPC │ ║ ║ ║ ║ │ - Migration: save/restore via MIGRATE RPC │ ║ ║ ║ ║ │ - Notifications: async events via NOTIFY RPC │ ║ ║ ║ ║ │ • NO Open-IOV code in guest kernel │ ║ ║ ║ ║ │ • NO gsp-vgpu.ko, NO libvgpu.so in guest │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ ║ ║ │ vfio-pci.ko (Guest VFIO Core) │ ║ ║ ║ ║ │ • mmap of BAR regions → host VFIO regions │ ║ ║ ║ ║ │ • MSI-X interrupt handling → eventfd from host │ ║ ║ ║ ║ │ • Config space read/write → host emulated config space │ ║ ║ ║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ ║ ║ ║ VIRTUAL GPU (mdev as seen by Guest): ║ ║ PCI Config (256B) │ BAR0(16MB) │ BAR1(256MB) │ BAR2(32MB) │ BAR3(32MB) │ MSI-X Table/PBA ║ ║ Vendor=10DE │ Config │ Framebuffer │ Reserved │ Reserved │ Table@0x2000 ║ ║ Device=1ED3 │ Doorbell │ (VRAM) │ │ │ PBA@0x3000 ║ ║ Class=030000 │ Queue Heads│ │ │ │ ║ ║ BARs: 64-bit │ Shared Mem │ │ │ │ ║ ║ Caps: PM,MSI, │ @0x110c00 │ │ │ │ ║ ║ MSI-X, PCIe │ @0x1000 │ │ │ │ ║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ``` ### 4. Communication Pathways (Complete) | # | Pathway | Direction | Mechanism | Latency Target | |---|---------|-----------|-----------|----------------| | 1 | **GSP RPC (Guest→Host)** | Guest nvidia.ko → Host open-iov-vfio.ko | Doorbell write (MMIO) → host polling thread → gsp_rpc_client.c | < 5μs | | 2 | **GSP Response (Host→Guest)** | Host open-iov-vfio.ko → Guest nvidia.ko | Write shared memory → MSI-X interrupt injection (eventfd) | < 3μs | | 3 | **BAR Register Access** | Guest nvidia.ko → Host open-iov-vfio.ko | MMIO page fault → bar_emulation.c shadow memory | < 2μs | | 4 | **PCI Config Space** | Guest kernel → Host open-iov-vfio.ko | VFIO_CONFIG_REGION_INDEX ioctl → emulated 256B config | < 1μs | | 5 | **VFIO DMA (VRAM)** | Guest nvidia.ko ↔ Physical VRAM | IOMMU DMA mapping via VFIO DMA API | Near-native | | 6 | **MSI-X Interrupts** | Physical GPU → Guest nvidia.ko | Host nvidia.ko ISR → nv_vgpu_vfio_inject_interrupt → eventfd → QEMU → Guest MSI-X | < 10μs | | 7 | **Management Ops** | mdev management CLI/mgr → Host open-iov-vfio.ko | VFIO ioctls on mdev device → libvgpu.so → rpc_emulator.c | < 100μs | | 8 | **Migration** | Host ↔ Host | Migration protocol over network, state via open-iov-vfio.ko | Seconds | ### 5. GSP Emulation Engine - Detailed Design #### 5.1 Mailbox Layout (from Firmware Decompilation) ``` GSP Shared Memory Mailbox (at BAR0 + 0x1000, 4KB): ┌─────────────────────────────────────────────────────────────────────────────┐ │ Offset │ Size │ Field │ Description │ ├────────┼───────┼──────────────────────────┼─────────────────────────────────┤ │ 0x0000 │ 64B │ RPC Request Header │ type, subcmd, seq_id, payload │ │ 0x0040 │ 64B │ RPC Request Payload[6] │ u64 payload[6] per RPC type │ │ 0x0080 │ 64B │ RPC Response Header │ status, type, seq_id │ │ 0x00C0 │ 64B │ RPC Response Payload[6] │ u64 result[6] │ │ 0x0100 │ 3KB │ Reserved / Extended │ Future use, migration state │ └─────────────────────────────────────────────────────────────────────────────┘ ``` #### 5.2 Doorbell Register (at BAR0 + 0x110C00) ``` Queue Head Register (0x110C00): ┌─────────────────────────────────────────────────────────────────────────────┐ │ Bits │ Field │ Description │ ├───────┼────────────────┼───────────────────────────────────────────────────┤ │ 31:1 │ Reserved │ Must be zero │ │ 0 │ Doorbell Valid │ 1 = New RPC request in mailbox, 0 = Idle │ └─────────────────────────────────────────────────────────────────────────────┘ ``` #### 5.3 Per-mdev GSP Emulation Thread ```c // In open-iov-vfio.ko - new file: gsp_emulation.c struct gsp_emu_context { struct nv_vgpu_device *vgpu; struct task_struct *thread; void __iomem *bar0_vaddr; void __iomem *doorbell_reg; void __iomem *shared_mem; struct gsp_client *rpc_client; // From gsp_rpc_client.c spinlock_t lock; bool running; u64 processed_rpcs; u64 failed_rpcs; }; static int gsp_emu_thread_fn(void *data) { struct gsp_emu_context *ctx = data; while (!kthread_should_stop()) { // Read doorbell (atomic, no lock needed for single reader) u32 doorbell = readl(ctx->doorbell_reg); if (doorbell & 0x1) { // ACK immediately to prevent guest retry writel(0, ctx->doorbell_reg); // Process RPC - this is the hot path int ret = gsp_emu_process_rpc(ctx); if (ret) ctx->failed_rpcs++; else ctx->processed_rpcs++; } // Adaptive polling: busy-wait for first 100μs, then sleep if (ctx->processed_rpcs % 1000 == 0) usleep_range(50, 100); else cpu_relax(); } return 0; } static int gsp_emu_process_rpc(struct gsp_emu_context *ctx) { // Copy request from shared memory (guest → host) struct gsp_rpc_msg req; memcpy_fromio(&req, ctx->shared_mem, sizeof(req)); // Validate sequence ID if (req.seq_id == 0) return -EINVAL; // Dispatch to firmware-decompiled handler struct gsp_rpc_response rsp = {0}; rsp.seq_id = req.seq_id; rsp.type = req.type; int ret = 0; switch (req.type) { case GSP_RPC_ALLOC: ret = gsp_rpc_alloc(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2], req.payload[3], req.payload[4], req.payload[5], &rsp.result[0], &rsp.result[1], &rsp.result[2]); break; case GSP_RPC_FREE: ret = gsp_rpc_free(ctx->rpc_client, req.payload[0]); break; case GSP_RPC_MAP: ret = gsp_rpc_map(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2], req.payload[3], req.payload[4], req.payload[5], &rsp.result[0]); break; case GSP_RPC_UNMAP: ret = gsp_rpc_unmap(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2]); break; case GSP_RPC_PIN: ret = gsp_rpc_pin(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2], req.payload[3], &rsp.result[0], &rsp.result[1]); break; case GSP_RPC_UNPIN: ret = gsp_rpc_unpin(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2]); break; case GSP_RPC_SCHEDULE: ret = gsp_rpc_schedule(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2], req.payload[3], req.payload[4]); break; case GSP_RPC_YIELD: ret = gsp_rpc_yield(ctx->rpc_client); break; case GSP_RPC_NOTIFY: ret = gsp_rpc_notify_register(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2]); break; case GSP_RPC_BAR_TRAP: ret = gsp_rpc_bar_trap(ctx->rpc_client, req.payload[0], req.payload[1], req.payload[2], req.payload[3], req.payload[4], &rsp.result[0]); break; case GSP_RPC_MIGRATE: // Handle migration sub-commands ret = gsp_emu_handle_migrate(ctx, &req, &rsp); break; case GSP_RPC_DEBUG: ret = gsp_emu_handle_debug(ctx, &req, &rsp); break; default: rsp.status = -ENOTTY; ret = -ENOTTY; } // Write response to shared memory (host → guest) rsp.status = ret ? -EIO : 0; memcpy_toio(ctx->shared_mem + 0x80, &rsp, sizeof(rsp)); // Inject MSI-X completion interrupt gsp_emu_inject_completion_interrupt(ctx, req.seq_id); return ret; } ``` #### 5.4 MSI-X Completion Interrupt Injection ```c static void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id) { // Write to MSI-X PBA (Pending Bit Array) at BAR0 + 0x3000 // Vector 0 = GSP completion interrupt void __iomem *pba = ctx->bar0_vaddr + 0x3000; writel(0x1, pba); // Set pending bit for vector 0 // Trigger eventfd for VFIO interrupt injection if (ctx->vgpu->msix_trigger[0]) eventfd_signal(ctx->vgpu->msix_trigger[0], 1); } ``` ### 6. Performance Requirements | Metric | Target | Measurement Method | |--------|--------|-------------------| | GSP RPC round-trip (emulated) | < 10μs | ktime_get_ns() in gsp_emu_thread | | BAR register access | < 2μs | Guest rdtsc around MMIO | | VRAM allocation (ALLOC) | < 50μs | Guest nvidia.ko tracing | | VRAM map (MAP) | < 20μs | Guest nvidia.ko tracing | | DMA pin (PIN) | < 30μs | Guest nvidia.ko tracing | | CUDA kernel launch overhead | < 5% vs native | CUDA events, nvprof | | Graphics frame time (Vulkan) | < 2ms @ 60fps | PresentMon, vk_perf | | Migration downtime | < 100ms | Stopwatch, ping during migration | | MSI-X injection latency | < 5μs | Guest ISR timestamp vs host | ### 7. Testing Strategy - Incremental Validation #### Phase 0: Host Infrastructure (Week 1) - [ ] `open-iov-vfio.ko` loads, registers mdev parent with `nvidia.ko` - [ ] mdev creation via sysfs succeeds - [ ] PCI config space presented correctly (lspci in guest shows 10de:1ed3) - [ ] BAR regions mmap'able in guest - [ ] MSI-X table/PBA accessible in guest BAR0 #### Phase 1: GSP Emulation Core (Week 2) - [ ] GSP emulation thread starts per mdev - [ ] Doorbell polling detects guest writes - [ ] ALLOC RPC: guest allocates VRAM, gets valid GPU VA + handle - [ ] FREE RPC: guest frees allocation, no leaks - [ ] MAP/UNMAP RPC: guest maps/unmaps GPU VA ranges - [ ] PIN/UNPIN RPC: guest pins pages, gets valid DMA addresses #### Phase 2: Advanced RPCs (Week 3) - [ ] SCHEDULE/YIELD: guest submits work, gets completion - [ ] BAR_TRAP: guest reads/writes BAR registers, gets emulated values - [ ] NOTIFY: guest registers for async events, receives them - [ ] MIGRATE: 4-phase migration saves/restores state #### Phase 3: Guest Driver Integration (Week 4) - [ ] Guest `nvidia.ko` loads, probes vGPU successfully - [ ] `nvidia-smi` shows vGPU (T4 / A100 equivalent) - [ ] CUDA context creation succeeds - [ ] Simple CUDA kernel (vectorAdd) executes correctly - [ ] `nvidia-ml` (NVML) queries work (temperature, utilization, memory) #### Phase 4: Compute Workloads (Week 5) - [ ] cuBLAS SGEMM/DGEMM passes numerical verification - [ ] cuDNN convolution forward/backward passes - [ ] PyTorch/TensorFlow training step completes - [ ] Multi-GPU (NCCL) across vGPUs works - [ ] CUDA Graphs capture/replay works #### Phase 5: Graphics Workloads (Week 6) - [ ] Vulkan: `vkCreateDevice` succeeds, render triangle - [ ] OpenGL: `glxinfo` shows renderer, `glxgears` runs - [ ] Video encode: FFmpeg + NVENC H.264/HEVC - [ ] Video decode: FFmpeg + NVDEC H.264/HEVC - [ ] Windows guest: D3D11/D3D12 device creation, render loop #### Phase 6: Stress & Reliability (Week 7) - [ ] 24-hour CUDA stress test (memtest, compute) - [ ] 24-hour graphics stress (Unigine, FurMark equivalent) - [ ] Hot-plug VF create/destroy cycles (1000x) - [ ] Migration under load (live migrate running CUDA kernel) - [ ] Host reboot recovery (mdev persists, guest recovers) #### Phase 7: Windows Guest (Week 8) - [ ] Windows 10/11 guest installs NVIDIA driver - [ ] Device Manager shows "NVIDIA T4" (or equivalent) - [ ] CUDA-Z / GPU-Z detect vGPU correctly - [ ] DirectX 11/12: D3D11CreateDevice, D3D12CreateDevice - [ ] Vulkan on Windows: vkCreateInstance, render - [ ] Video encode/decode via Media Foundation ### 8. Implementation File Map ``` open-iov/ ├── kernel/ │ ├── gsp_emulation.c # NEW: GSP emulation thread + RPC processing │ ├── gsp_emulation.h # NEW: Structures, prototypes │ ├── gsp_rpc_client.c # EXISTING: Firmware-decompiled handlers │ ├── gsp_rpc_integration.c # EXISTING: Integration layer │ ├── bar_emulation.c # EXISTING: BAR trap handling │ ├── memory.c # EXISTING: DMA, VRAM management │ ├── irq.c # EXISTING: MSI-X injection │ ├── migration.c # EXISTING: Live migration │ ├── main.c # MODIFY: Add gsp_emu_context to nv_vgpu_device │ └── Makefile # MODIFY: Add gsp_emulation.o ├── userspace/lib/ │ ├── libvgpu.c # EXISTING: Management API │ ├── rpc_emulator.c # EXISTING: Host-side emulation fallbacks │ └── Makefile # UNCHANGED ├── scripts/ │ ├── test_gsp_emu.sh # NEW: Automated GSP emulation tests │ ├── test_guest_driver.sh # NEW: Guest driver validation │ ├── test_cuda_workloads.sh # NEW: CUDA workload validation │ ├── test_graphics_workloads.sh # NEW: Graphics validation │ └── test_windows_guest.sh # NEW: Windows guest validation └── docs/ ├── refactored-architecture.md # THIS DOCUMENT └── refactored-implementation-to-do.md # TODO LIST ``` ### 9. Risk Mitigation | Risk | Likelihood | Impact | Mitigation | |------|------------|--------|------------| | GSP mailbox layout mismatch | Medium | High | Validate against firmware decompilation; add runtime checksum | | MSI-X injection race | Medium | High | Use atomic PBA updates; test with concurrent RPCs | | Cache coherency (WC memory) | High | Medium | clflush + mfence on every shared mem write; test on AMD/Intel | | Guest driver version skew | Medium | Medium | Test against multiple OpenRM versions (550, 560, 570) | | Windows guest driver issues | Medium | High | Separate Windows test CI; use WHQL driver | | Performance regression vs native | Low | High | Continuous benchmarking; profile hot paths | | Migration state corruption | Low | High | Checksum all migrated state; dry-run validation | --- *Document Version: 1.0 (Refactored Architecture)* *Author: Open-IOV Team* *Date: $(date)* *Repository: $OPEN_IOV_ROOT* *Git Commit: f9ed5d8 (restore point)*