From 8f01fb29256aa1a8ab3088015916425ff910e0b0 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 22 Sep 2026 09:51:51 -0400 Subject: docs: architecture rewrite - emulate vs pass-through model, phased performance roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Clear separation: EMULATE (GSP interface, BARs, config) vs PASS-THROUGH (compute, memory, DMA to physical GPU) - Phased roadmap: Phase 1 (GSP-RM correctness) → Phase 2 (memory model) → Phase 3 (compute forwarding) → Phase 4 (performance) → Phase 5 (graphics) - Host module architecture corrected: apt nvidia (570.211.01) for physical GPU, open-iov-vfio.ko for vGPU emulation, NO 610 submodule - Host stability guarantees documented (BAR3 shadow, submodule skip) - GSP_EMULATION_ARCHITECTURE.md: memory model corrected (BAR1/BAR3 shadows) --- docs/GSP_EMULATION_ARCHITECTURE.md | 25 +- docs/refactored-architecture.md | 831 ++++++++++++------------------------- 2 files changed, 283 insertions(+), 573 deletions(-) (limited to 'docs') diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md index eb7f697..8da8f49 100644 --- a/docs/GSP_EMULATION_ARCHITECTURE.md +++ b/docs/GSP_EMULATION_ARCHITECTURE.md @@ -116,14 +116,23 @@ implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by > RPC loop plus multi-second stalls. Emulation mode now fails these fast with > `pr_warn_ratelimited`. -### Memory model caveat (design note) - -Guest VRAM (BAR1) is mapped **directly to the host GPU's physical BAR1** (gpa). -`gsp_rpc_alloc_emul`, however, services ALLOC with *host system pages* DMA-mapped -to the physical device. These two models are not yet unified: a guest allocation -returning `gpu_va=0x100000000+` is not visible through the guest's BAR1 window. -This is a known design gap to resolve before CUDA workloads; it does not affect -driver probe/GSP boot, which is the current milestone. +### Memory model (corrected 2026-09-22) + +Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer** +(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses +hit our clean host-RAM backing. This is a placeholder — for real CUDA compute, +the memory model must evolve (see `refactored-architecture.md` Phase 2): + +- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption). + Guest driver probes but cannot compute. +- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU + VRAM partition (required for CUDA). +- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU + channels (required for compute). + +Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB, +`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach +the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`). --- diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md index fe17248..9a2f401 100644 --- a/docs/refactored-architecture.md +++ b/docs/refactored-architecture.md @@ -1,595 +1,296 @@ -# Open-IOV vGPU-IO Refactored Architecture +# Open-IOV vGPU-IO Architecture -## System Design from First Principles +> **Status**: Living document. Updated 2026-09-22 post host-stability fix. +> **Supersedes**: Previous refactored-architecture.md (v1.0). -### 1. Problem Statement +--- + +## 1. Problem Statement + +**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with **full GPU functionality** (CUDA, graphics, video, AI/ML) on consumer and data-center GPUs without proprietary GRID licenses. + +**Correctness requirement**: Guest must see a fully functional NVIDIA GPU. `nvidia-smi`, CUDA, OpenGL/Vulkan, and compute must all work. -**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with full GPU functionality (CUDA, graphics, video encode/decode, AI/ML) on Turing GPUs (TU102/TU104) without requiring: -- Proprietary GRID host drivers -- Modified guest drivers -- Paravirtualization interfaces -- Guest-side Open-IOV code +**Performance requirement**: GPU compute must execute on the physical GPU hardware, not in software. Target: < 5% overhead vs native for CUDA kernel launches. **Constraints**: -- Guest runs stock `nvidia.ko` + `libcuda.so` + `libnvidia-ml.so` + `libnvidia-opencl.so` -- Host runs OpenRM `nvidia.ko` (for physical GPU management) + Open-IOV hypervisor components -- No guest kernel modifications, no guest userspace Open-IOV libraries -- High performance: near-native GPU utilization for compute and graphics workloads -- Support both Linux and Windows guests +- Guest runs **stock** `nvidia.ko` + standard NVIDIA userspace (no Open-IOV code in guest) +- Host runs **apt** `nvidia-driver-570` for physical GPU management (570.211.01) +- Host runs **open-iov-vfio.ko** for vGPU emulation (mediated device, mdev) +- No SR-IOV required (works on consumer GPUs like RTX 2080 Super) -### 2. Core Design Principle: Host-Side GSP Emulation +--- + +## 2. Core Architectural Principle: Emulate the Interface, Pass Through the Work -The fundamental insight: **The GSP (GPU System Processor) is a RISC-V microcontroller running firmware that implements 12 RPC types. We can emulate this firmware entirely in the host kernel.** +A vGPU is a **mediated passthrough** device. Some functions must be emulated (the guest expects virtual hardware), others must be passed through to the physical GPU (for performance). ``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ FIRST PRINCIPLE: GSP IS JUST AN RPC SERVER │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Physical Reality: │ -│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ -│ │ Guest RM │ ─────────────────────────────► │ GSP Firmware │ │ -│ │ (nvidia.ko)│ ◄───────────────────────────── │ (570.144W, RV64IMAC) │ │ -│ └─────────────┘ Responses └─────────────────────────┘ │ -│ │ │ │ -│ │ 12 RPC Types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │ -│ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │ -│ │ │ │ -│ ▼ │ │ -│ Virtualized Reality: │ │ -│ ┌─────────────┐ Doorbell + Shared Mem ┌─────────────────────────┐ │ -│ │ Guest RM │ ─────────────────────────────► │ Host Kernel │ │ -│ │ (nvidia.ko)│ ◄───────────────────────────── │ (open-iov-vfio.ko) │ │ -│ └─────────────┘ Responses │ gsp_rpc_client.c │ │ -│ │ │ (12 emulated handlers)│ │ -│ │ Guest sees identical └─────────────────────────┘ │ -│ │ interface - cannot tell │ -│ ▼ │ -│ No guest-side code required. │ -│ │ -└─────────────────────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────────┐ +│ EMULATE (software — guest expects virtual hardware) │ +│ • PCI config space (guest sees virtual TU104 [10de:1ed3]) │ +│ • BAR0 register interface (shadow, not physical GPU BAR0) │ +│ • GSP boot sequence (FWSEC/Booter/RISCV auto-complete) │ +│ • GSP-RM protocol layer (command queue, reply format) │ +│ • Interrupt routing (eventfd → guest MSI-X) │ +│ • Display output (headless / virtual display) │ +│ • BAR3 instance memory window (shadow, not physical GPU) │ +└──────────────────────────────────────────────────────────────────────┘ + +┌──────────────────────────────────────────────────────────────────────┐ +│ PASS-THROUGH (hardware — for performance) │ +│ • GPU compute (CUDA kernels) → execute on physical GPU │ +│ • GPU memory (guest VRAM) → backed by real GPU VRAM or DMA-mapped │ +│ • DMA operations → use real GPU DMA engines │ +│ • Channel/pushbuffer work → physical GPU channels │ +│ • Graphics rendering → physical GPU render pipeline │ +│ • Video encode/decode → physical GPU NVENC/NVDEC engines │ +└──────────────────────────────────────────────────────────────────────┘ ``` -### 3. System Architecture Overview +**Why not full emulation?** Software emulation of CUDA kernels would be 100-1000x slower than native. The physical GPU must do the actual compute work. + +**Why not full passthrough?** The guest driver expects to manage the GSP, boot firmware, and own the GPU. We can't give it direct physical GPU access (the host driver already owns it). The GSP emulation layer translates guest commands into host GPU operations. + +--- + +## 3. Host Module Architecture (Corrected 2026-09-22) ``` -╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ -║ HOST SYSTEM ║ -║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ -║ ║ USERSPACE ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ QEMU/KVM + libvfio-user │ ║ ║ -║ ║ │ • -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/ │ ║ ║ -║ ║ │ • VFIO region mmap, ioctl, eventfd for IRQs │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ║ ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ mdev management CLI / mdev management daemon (Management Tools) │ ║ ║ -║ ║ │ • Link against: libvgpu.so (host-side) │ ║ ║ -║ ║ │ • VF lifecycle: create/destroy, resource allocation, migration orchestration │ ║ ║ -║ ║ │ • Calls: vgpu_open(), vgpu_vf_create(), vgpu_mem_alloc(), vgpu_bar_config(), │ ║ ║ -║ ║ │ vgpu_wpr_protect(), vgpu_schedule(), vgpu_migrate_start(), etc. │ ║ ║ -║ ║ │ • Communicates via VFIO ioctls on /dev/vfio/ │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ║ ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ libvgpu.so / libiov.so (HOST-SIDE ONLY) │ ║ ║ -║ ║ │ • Full vGPU management API: vgpu_*() functions │ ║ ║ -║ ║ │ • Implements: VF create/destroy, memory ops, BAR config, WPR, scheduling, │ ║ ║ -║ ║ │ notifications, migration │ ║ ║ -║ ║ │ • rpc_emulator.c: Host-side emulation fallbacks for management operations │ ║ ║ -║ ║ │ (rpc_emu_bar_config/save/restore, rpc_emu_wpr_*, rpc_emu_schedule/yield, │ ║ ║ -║ ║ │ rpc_emu_notify_register/unregister/send, rpc_emu_migrate_*) │ ║ ║ -║ ║ │ • NO GSP RPC emulation here - that lives in kernel │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ -║ ║ -║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ -║ ║ KERNEL SPACE ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ nvidia.ko (Host RM - OpenRM 570.144) │ ║ ║ -║ ║ │ • Boots physical GSP firmware (570.144W) │ ║ ║ -║ ║ │ • Manages physical GPU resources (VRAM, engines, display) │ ║ ║ -║ ║ │ • Exports vGPU VFIO symbols: │ ║ ║ -║ ║ │ - nvidia_vgpu_vfio_get_ops() / nvidia_vgpu_vfio_set_ops() │ ║ ║ -║ ║ │ - nv_vgpu_mdev_register_parent() / nv_vgpu_mdev_unregister_parent() │ ║ ║ -║ ║ │ • GPC callbacks from open-iov-vfio.ko: vgpu_create, vgpu_delete, vgpu_bar_info, │ ║ ║ -║ ║ │ vgpu_start, get_name, get_description, get_instances, get_sparse_mmap, update_request │ ║ ║ -║ ║ │ • Owns physical MSI-X interrupt │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ║ ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ open-iov-vfio.ko (HYPERVISOR - Core Component) │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ mdev Parent Registration │ │ ║ ║ -║ ║ │ │ • Registers with nvidia.ko via nv_vgpu_mdev_register_parent() │ │ ║ ║ -║ ║ │ │ • Creates mediated device types: nvidia-vgpu-turing, nvidia-vgpu-ampere, etc. │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ PCI Config Space Emulation (256 bytes per mdev) │ │ ║ ║ -║ ║ │ │ • Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU), Class=0x030000 │ │ ║ ║ -║ ║ │ │ • BAR0=16MB (64-bit, prefetchable), BAR1=256MB (64-bit, prefetchable) │ │ ║ ║ -║ ║ │ │ • BAR2=32MB, BAR3=32MB (64-bit) │ │ ║ ║ -║ ║ │ │ • Capabilities: PM (0x40), MSI (0x50), MSI-X (0x70), PCIe (0x80) │ │ ║ ║ -║ ║ │ │ • MSI-X Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ │ ║ ║ -║ ║ │ │ • VFIO_CONFIG_REGION_INDEX mapped to this emulated config space │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ BAR Region Emulation (VFIO Region Indices) │ │ ║ ║ -║ ║ │ │ • VFIO_PCI_BAR0_REGION_INDEX → 16MB emulated config space │ │ ║ ║ -║ ║ │ │ • VFIO_PCI_BAR1_REGION_INDEX → 256MB emulated framebuffer (backed by host VRAM allocation) │ │ ║ ║ -║ ║ │ │ • VFIO_PCI_BAR2_REGION_INDEX → 32MB reserved │ │ ║ ║ -║ ║ │ │ • VFIO_PCI_BAR3_REGION_INDEX → 32MB reserved │ │ ║ ║ -║ ║ │ │ • VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table (0x2000) + PBA (0x3000) in BAR0 │ │ ║ ║ -║ ║ │ │ • BAR trap handling: 64 trapped regions covering all BAR0 registers (bar_emulation.c) │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ GSP EMULATION ENGINE (NEW - Core Innovation) │ │ ║ ║ -║ ║ │ │ │ │ ║ ║ -║ ║ │ │ Per-mdev GSP Emulation Thread: │ │ ║ ║ -║ ║ │ │ ```c │ │ ║ ║ -║ ║ │ │ static int gsp_emu_thread(void *data) { │ │ ║ ║ -║ ║ │ │ struct nv_vgpu_device *vgpu = data; │ │ ║ ║ -║ ║ │ │ while (!kthread_should_stop()) { │ │ ║ ║ -║ ║ │ │ u32 doorbell = readl(vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); │ │ ║ ║ -║ ║ │ │ if (doorbell) { │ │ ║ ║ -║ ║ │ │ writel(0, vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET); // ACK │ │ ║ ║ -║ ║ │ │ gsp_emu_process_rpc(vgpu); // Process via gsp_rpc_client.c │ │ ║ ║ -║ ║ │ │ } │ │ ║ ║ -║ ║ │ │ usleep_range(100, 200); // 1-2μs polling interval │ │ ║ ║ -║ ║ │ │ } │ │ ║ ║ -║ ║ │ │ return 0; │ │ ║ ║ -║ ║ │ │ } │ │ ║ ║ -║ ║ │ │ ``` │ │ ║ ║ -║ ║ │ │ │ │ ║ ║ -║ ║ │ │ GSP Shared Memory Management: │ │ ║ ║ -║ ║ │ │ • Allocates contiguous host memory for GSP mailbox (4KB aligned) │ │ ║ ║ -�b ║ │ │ • Maps into guest BAR0 at GSP_SHARED_MEM_OFFSET (0x1000) │ │ ║ ║ -║ ║ │ │ • WC (Write-Combining) mapping for performance │ │ ║ ║ -║ ║ │ │ • Cache coherency: clflush on write, memory barriers │ │ ║ ║ -║ ║ │ │ • Matches firmware mailbox layout exactly (from decompilation) │ │ ║ ║ -║ ║ │ │ │ │ ║ ║ -║ ║ │ │ RPC Handler Suite (gsp_rpc_client.c - ALL 12 TYPES): │ │ ║ ║ -║ ║ │ │ ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ │ ║ -║ ║ │ │ │ RPC Type │ Emulated Implementation │ │ │ ║ -║ ║ │ │ ├─────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ │ ║ -║ ║ │ │ │ ALLOC (0x01)│ VRAM allocation via host nvidia.ko, returns GPU VA + handle │ │ │ ║ -║ ║ │ │ │ FREE (0x02) │ Free allocation, unmap pages │ │ │ ║ -║ ║ │ │ │ MAP (0x03) │ Map allocation to GPU VA, update page tables │ │ │ ║ -║ ║ │ │ │ UNMAP (0x04)│ Unmap GPU VA range │ │ │ ║ -║ ║ │ │ │ PIN (0x05) │ Pin pages for DMA, return DMA address │ │ │ ║ -║ ║ │ │ │ UNPIN (0x06)│ Unpin pages │ │ │ ║ -║ ║ │ │ │ SCHEDULE(0x07)│ Submit work to GSP scheduler (local workqueue) │ │ │ ║ -║ ║ │ │ │ YIELD (0x08)│ Timeslice yield (no-op success) │ │ │ ║ -║ ║ │ │ │ NOTIFY(0x09)│ Async notification registration, eventfd signaling │ │ │ ║ -║ ║ │ │ │ BAR_TRAP(0x0A)│ BAR register read/write emulation via shadow memory │ │ │ ║ -║ ║ │ │ │ MIGRATE(0x0B)│ 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP) │ │ │ ║ -║ ║ │ │ │ DEBUG (0x0C)│ Debug interface, firmware introspection │ │ │ ║ -║ ║ │ │ └─────────────┴──────────────────────────────────────────────────────────────────────────────────┘ │ │ ║ -║ ║ │ │ │ │ ║ -║ ║ │ │ Firmware Address Mapping (from decompilation): │ │ ║ -║ ║ │ │ • Primary Dispatcher: 0x057A1AC0 (32 sub-commands) │ │ ║ -║ ║ │ │ • Secondary Dispatcher: 0x04ED39DC │ │ ║ -║ ║ │ │ • BAR Trap Handler: 0x04CB8188 │ │ ║ -║ ║ │ │ • WPR Handlers: 0x04F00B0C, 0x04F00C78 │ │ ║ -║ ║ │ │ • ALLOC Main: 0x057E94EC │ │ ║ -║ ║ │ │ • SCHEDULE Main: 0x051EFBF4 │ │ ║ -║ ║ │ │ • Migration Tables: 0x04179ED8, 0x04179FBC (16 entries each) │ │ ║ -║ ║ │ │ • RPC Jump Table: 0x059BFEB0 (12 handlers) │ │ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ MSI-X Interrupt Injection │ │ ║ ║ -║ ║ │ │ • When emulated GSP completes RPC → write to MSI-X PBA → eventfd_signal() │ │ ║ ║ -║ ║ │ │ • QEMU/KVM delivers MSI-X interrupt to guest nvidia.ko ISR │ │ ║ ║ -║ ║ │ │ • Guest sees normal GSP completion interrupt │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ Memory Management (memory.c) │ │ ║ ║ -║ ║ │ │ • DMA mapping via IOMMU (VFIO DMA API) │ │ ║ ║ -║ ║ │ │ • Pin/unpin for device DMA │ │ ║ ║ -║ ║ │ │ • VRAM allocation via host nvidia.ko │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ │ │ ║ ║ -║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ -║ ║ │ │ Live Migration (migration.c) │ │ ║ ║ -║ ║ │ │ • 4-phase: QUERY → PREPARE → TRANSFER → COMMIT → CLEANUP │ │ ║ ║ -║ ║ │ │ • State save/restore: GSP context, page tables, VRAM contents, BAR state │ │ ║ ║ -║ ║ │ │ • Dual jump tables from firmware (0x04179ED8, 0x04179FBC) │ │ ║ ║ -║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ -║ ║ -║ PHYSICAL GPU: TU102/TU104 (RTX 2080 Ti / T4 / RTX 2080 Super) ║ -║ • REAL GSP Firmware 570.144W running on FALCON RISC-V ║ -║ • Used by host nvidia.ko for physical GPU management ║ -║ • NOT used for guest vGPU RPCs (those are emulated in host kernel) ║ -║ ║ -╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ - │ VFIO MEDIATED DEVICE (mdev) │ - │ /sys/bus/mdev/devices// │ - ▼ - -╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ -║ GUEST VM ║ -║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ -║ ║ USERSPACE ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ Guest Applications │ ║ ║ -║ ║ │ • CUDA Applications (cuBLAS, cuDNN, custom kernels) │ ║ ║ -║ ║ │ • Graphics: OpenGL, Vulkan, DirectX (via WINE/Proton on Linux, native on Windows) │ ║ ║ -║ ║ │ • Video: NVENC/NVDEC (FFmpeg, GStreamer, Media Foundation) │ ║ ║ -║ ║ │ • AI/ML: TensorRT, PyTorch, TensorFlow │ ║ ║ -║ ║ │ • NO Open-IOV libraries, NO modified code │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ║ ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ Standard NVIDIA Userspace Drivers (UNMODIFIED) │ ║ ║ -║ ║ │ • libcuda.so / libcuda.so.1 (CUDA Driver API) │ ║ ║ -║ ║ │ • libnvidia-ml.so / libnvidia-ml.so.1 (NVML) │ ║ ║ -║ ║ │ • libnvidia-opencl.so / libnvidia-opencl.so.1 (OpenCL) │ ║ ║ -║ ║ │ • libnvidia-vulkan.so / libGLX_nvidia.so (Vulkan/OpenGL) │ ║ ║ -║ ║ │ • libnvidia-encode.so / libnvidia-decode.so (Video) │ ║ ║ -║ ║ │ • nvidia-smi, nvidia-settings, nvidia-debugdump │ ║ ║ -║ ║ │ • ALL communicate with guest nvidia.ko via /dev/nvidia* ioctls │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ -║ ║ -║ ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ -║ ║ KERNEL SPACE ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ nvidia.ko (Guest RM - UNMODIFIED OpenRM 570.144) │ ║ ║ -║ ║ │ • PCI Probe: detects 10de:1ed3 (TU104 vGPU) from emulated config space │ ║ ║ -║ ║ │ • BAR Mapping: │ ║ ║ -║ ║ │ - BAR0 (16MB): Config registers, GSP doorbell @ 0x110c00, shared mem @ 0x1000 │ ║ ║ -║ ║ │ - BAR1 (256MB): Framebuffer (VRAM) │ ║ ║ -║ ║ │ - BAR2 (32MB), BAR3 (32MB): Reserved │ ║ ║ -║ ║ │ • MSI-X Setup: Table @ BAR0+0x2000, PBA @ BAR0+0x3000 │ ║ ║ -║ ║ │ • GSP Client Initialization: │ ║ ║ -║ ║ │ - Writes to doorbell (queue head) to submit RPCs │ ║ ║ -║ ║ │ - Reads/writes shared memory for RPC messages │ ║ ║ -║ ║ │ - Waits for MSI-X interrupt on completion │ ║ ║ -║ ║ │ • RM Subsystems (all functional): │ ║ ║ -║ ║ │ - Memory: alloc, free, map, unmap, pin, unpin via GSP RPC │ ║ ║ -║ ║ │ - Scheduling: work submission, yield, priority via GSP RPC │ ║ ║ -║ ║ │ - BAR: register access via BAR_TRAP RPC │ ║ ║ -║ ║ │ - WPR: protect/unprotect via GSP RPC │ ║ ║ -║ ║ │ - Migration: save/restore via MIGRATE RPC │ ║ ║ -║ ║ │ - Notifications: async events via NOTIFY RPC │ ║ ║ -║ ║ │ • NO Open-IOV code in guest kernel │ ║ ║ -║ ║ │ • NO gsp-vgpu.ko, NO libvgpu.so in guest │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ║ ║ ║ -║ ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ -║ ║ │ vfio-pci.ko (Guest VFIO Core) │ ║ ║ -║ ║ │ • mmap of BAR regions → host VFIO regions │ ║ ║ -║ ║ │ • MSI-X interrupt handling → eventfd from host │ ║ ║ -║ ║ │ • Config space read/write → host emulated config space │ ║ ║ -║ ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ -║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ -║ ║ -║ VIRTUAL GPU (mdev as seen by Guest): ║ -║ PCI Config (256B) │ BAR0(16MB) │ BAR1(256MB) │ BAR2(32MB) │ BAR3(32MB) │ MSI-X Table/PBA ║ -║ Vendor=10DE │ Config │ Framebuffer │ Reserved │ Reserved │ Table@0x2000 ║ -║ Device=1ED3 │ Doorbell │ (VRAM) │ │ │ PBA@0x3000 ║ -║ Class=030000 │ Queue Heads│ │ │ │ ║ -║ BARs: 64-bit │ Shared Mem │ │ │ │ ║ -║ Caps: PM,MSI, │ @0x110c00 │ │ │ │ ║ -║ MSI-X, PCIe │ @0x1000 │ │ │ │ ║ -╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +╔══════════════════════════════════════════════════════════════════════╗ +║ HOST KERNEL SPACE ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ nvidia.ko (apt 570.211.01) │ ║ +║ │ • Manages physical RTX 2080 Super (GSP firmware booted) │ ║ +║ │ • Provides /dev/nvidia0 for nvidia-smi on host │ ║ +║ │ • Owns physical GPU interrupts, BARs, VRAM │ ║ +║ │ • NOT modified, NOT rebuilt — stock Ubuntu package │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ open-iov-vfio.ko (Open-IOV vGPU VFIO driver) │ ║ +║ │ • Registers mdev parent on the physical GPU's PCI device │ ║ +║ │ • Creates mdev types (nvidia-vgpu-turing, etc.) │ ║ +║ │ • Per-mdev: emulated PCI config, BAR0 shadow, BAR1 vidmem, │ ║ +║ │ BAR3 shadow, MSI-X emulation, GSP emulation thread │ ║ +║ │ • Exports VFIO device ops (read/write/ioctl/mmap) │ ║ +║ │ • Self-contained: depends only on vfio, mdev │ ║ +║ │ • NO dependency on nvidia submodule (removed 2026-09-22) │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ vfio / mdev (kernel prerequisites) │ ║ +║ │ • vfio, vfio_iommu_type1, vfio_pci, mdev │ ║ +║ │ • IOMMU domain management, device isolation │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ PHYSICAL GPU: RTX 2080 Super (TU104, 10de:1ed3) ║ +║ • Managed by apt nvidia driver (host nvidia-smi works) ║ +║ • open-iov-vfio registers mdev parent but does NOT drive the GPU ║ +║ • Guest vGPU is a software construct — physical GPU unaffected ║ +╚══════════════════════════════════════════════════════════════════════╝ + │ VFIO mdev + │ /sys/bus/mdev/devices// + ▼ +╔══════════════════════════════════════════════════════════════════════╗ +║ GUEST VM (QEMU/KVM) ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ nvidia.ko (stock 570.144) │ ║ +║ │ • PCI probe: sees TU104 [10de:1ed3] via emulated config │ ║ +║ │ • BAR0: shadow (vzalloc'd, mapped via VFIO) │ ║ +║ │ • BAR1: emulated vidmem (vzalloc'd host RAM) │ ║ +║ │ • BAR3: shadow (vzalloc'd) │ ║ +║ │ • MSI-X: eventfd → QEMU → guest ISR │ ║ +║ │ • GSP client: msgq RPCs via emulated GSP-RM channel │ ║ +║ │ • NO Open-IOV code in guest │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +╚══════════════════════════════════════════════════════════════════════╝ ``` -### 4. Communication Pathways (Complete) +### Why no nvidia submodule on host? -| # | Pathway | Direction | Mechanism | Latency Target | -|---|---------|-----------|-----------|----------------| -| 1 | **GSP RPC (Guest→Host)** | Guest nvidia.ko → Host open-iov-vfio.ko | Doorbell write (MMIO) → host polling thread → gsp_rpc_client.c | < 5μs | -| 2 | **GSP Response (Host→Guest)** | Host open-iov-vfio.ko → Guest nvidia.ko | Write shared memory → MSI-X interrupt injection (eventfd) | < 3μs | -| 3 | **BAR Register Access** | Guest nvidia.ko → Host open-iov-vfio.ko | MMIO page fault → bar_emulation.c shadow memory | < 2μs | -| 4 | **PCI Config Space** | Guest kernel → Host open-iov-vfio.ko | VFIO_CONFIG_REGION_INDEX ioctl → emulated 256B config | < 1μs | -| 5 | **VFIO DMA (VRAM)** | Guest nvidia.ko ↔ Physical VRAM | IOMMU DMA mapping via VFIO DMA API | Near-native | -| 6 | **MSI-X Interrupts** | Physical GPU → Guest nvidia.ko | Host nvidia.ko ISR → nv_vgpu_vfio_inject_interrupt → eventfd → QEMU → Guest MSI-X | < 10μs | -| 7 | **Management Ops** | mdev management CLI/mgr → Host open-iov-vfio.ko | VFIO ioctls on mdev device → libvgpu.so → rpc_emulator.c | < 100μs | -| 8 | **Migration** | Host ↔ Host | Migration protocol over network, state via open-iov-vfio.ko | Seconds | +The **nvidia-open-gpu-kernel-modules submodule (610.57.04-based)** was originally intended for NVIDIA's *native* vGPU management (RM callbacks like `vgpu_create_callback`). For our QEMU emulated-VFIO path, it's **unnecessary and harmful**: -### 5. GSP Emulation Engine - Detailed Design +1. **Unnecessary**: open-iov-vfio.ko handles all VFIO emulation independently +2. **Harmful**: Loading the 610-based submodule alongside apt 570.211.01 causes driver conflicts that crash the host GSP (WPR2 stuck, `unexpected WPR2 already up`) -#### 5.1 Mailbox Layout (from Firmware Decompilation) +**Root cause fixed 2026-09-22** (`85d0668`): stack_manager skips submodule when apt nvidia is present. -``` -GSP Shared Memory Mailbox (at BAR0 + 0x1000, 4KB): -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Offset │ Size │ Field │ Description │ -├────────┼───────┼──────────────────────────┼─────────────────────────────────┤ -│ 0x0000 │ 64B │ RPC Request Header │ type, subcmd, seq_id, payload │ -│ 0x0040 │ 64B │ RPC Request Payload[6] │ u64 payload[6] per RPC type │ -│ 0x0080 │ 64B │ RPC Response Header │ status, type, seq_id │ -│ 0x00C0 │ 64B │ RPC Response Payload[6] │ u64 result[6] │ -│ 0x0100 │ 3KB │ Reserved / Extended │ Future use, migration state │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` +--- -#### 5.2 Doorbell Register (at BAR0 + 0x110C00) +## 4. Emulated Component Details -``` -Queue Head Register (0x110C00): -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Bits │ Field │ Description │ -├───────┼────────────────┼───────────────────────────────────────────────────┤ -│ 31:1 │ Reserved │ Must be zero │ -│ 0 │ Doorbell Valid │ 1 = New RPC request in mailbox, 0 = Idle │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` +### 4.1 BAR0 Shadow (16 MB vzalloc) -#### 5.3 Per-mdev GSP Emulation Thread - -```c -// In open-iov-vfio.ko - new file: gsp_emulation.c - -struct gsp_emu_context { - struct nv_vgpu_device *vgpu; - struct task_struct *thread; - void __iomem *bar0_vaddr; - void __iomem *doorbell_reg; - void __iomem *shared_mem; - struct gsp_client *rpc_client; // From gsp_rpc_client.c - spinlock_t lock; - bool running; - u64 processed_rpcs; - u64 failed_rpcs; -}; - -static int gsp_emu_thread_fn(void *data) { - struct gsp_emu_context *ctx = data; - - while (!kthread_should_stop()) { - // Read doorbell (atomic, no lock needed for single reader) - u32 doorbell = readl(ctx->doorbell_reg); - - if (doorbell & 0x1) { - // ACK immediately to prevent guest retry - writel(0, ctx->doorbell_reg); - - // Process RPC - this is the hot path - int ret = gsp_emu_process_rpc(ctx); - if (ret) - ctx->failed_rpcs++; - else - ctx->processed_rpcs++; - } - - // Adaptive polling: busy-wait for first 100μs, then sleep - if (ctx->processed_rpcs % 1000 == 0) - usleep_range(50, 100); - else - cpu_relax(); - } - return 0; -} - -static int gsp_emu_process_rpc(struct gsp_emu_context *ctx) { - // Copy request from shared memory (guest → host) - struct gsp_rpc_msg req; - memcpy_fromio(&req, ctx->shared_mem, sizeof(req)); - - // Validate sequence ID - if (req.seq_id == 0) - return -EINVAL; - - // Dispatch to firmware-decompiled handler - struct gsp_rpc_response rsp = {0}; - rsp.seq_id = req.seq_id; - rsp.type = req.type; - - int ret = 0; - switch (req.type) { - case GSP_RPC_ALLOC: - ret = gsp_rpc_alloc(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2], - req.payload[3], req.payload[4], req.payload[5], - &rsp.result[0], &rsp.result[1], &rsp.result[2]); - break; - case GSP_RPC_FREE: - ret = gsp_rpc_free(ctx->rpc_client, req.payload[0]); - break; - case GSP_RPC_MAP: - ret = gsp_rpc_map(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2], - req.payload[3], req.payload[4], req.payload[5], - &rsp.result[0]); - break; - case GSP_RPC_UNMAP: - ret = gsp_rpc_unmap(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2]); - break; - case GSP_RPC_PIN: - ret = gsp_rpc_pin(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2], - req.payload[3], &rsp.result[0], &rsp.result[1]); - break; - case GSP_RPC_UNPIN: - ret = gsp_rpc_unpin(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2]); - break; - case GSP_RPC_SCHEDULE: - ret = gsp_rpc_schedule(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2], - req.payload[3], req.payload[4]); - break; - case GSP_RPC_YIELD: - ret = gsp_rpc_yield(ctx->rpc_client); - break; - case GSP_RPC_NOTIFY: - ret = gsp_rpc_notify_register(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2]); - break; - case GSP_RPC_BAR_TRAP: - ret = gsp_rpc_bar_trap(ctx->rpc_client, - req.payload[0], req.payload[1], req.payload[2], - req.payload[3], req.payload[4], &rsp.result[0]); - break; - case GSP_RPC_MIGRATE: - // Handle migration sub-commands - ret = gsp_emu_handle_migrate(ctx, &req, &rsp); - break; - case GSP_RPC_DEBUG: - ret = gsp_emu_handle_debug(ctx, &req, &rsp); - break; - default: - rsp.status = -ENOTTY; - ret = -ENOTTY; - } - - // Write response to shared memory (host → guest) - rsp.status = ret ? -EIO : 0; - memcpy_toio(ctx->shared_mem + 0x80, &rsp, sizeof(rsp)); - - // Inject MSI-X completion interrupt - gsp_emu_inject_completion_interrupt(ctx, req.seq_id); - - return ret; -} -``` +The guest's BAR0 is a software shadow, NOT the physical GPU's BAR0. -#### 5.4 MSI-X Completion Interrupt Injection - -```c -static void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id) { - // Write to MSI-X PBA (Pending Bit Array) at BAR0 + 0x3000 - // Vector 0 = GSP completion interrupt - void __iomem *pba = ctx->bar0_vaddr + 0x3000; - writel(0x1, pba); // Set pending bit for vector 0 - - // Trigger eventfd for VFIO interrupt injection - if (ctx->vgpu->msix_trigger[0]) - eventfd_signal(ctx->vgpu->msix_trigger[0], 1); -} -``` +| BAR0 offset | Region | Who writes | Who reads | +|-------------------|-------------------------------------|------------|-----------| +| `0x000-0xFFF` | Emulated registers (PMC boot, etc.) | shadow | shadow | +| `0x1000` | RPC request mailbox | guest | emu thread| +| `0x1080` | RPC response mailbox | emu thread | guest | +| `0x2000` | MSI-X table (32 entries) | guest | guest | +| `0x3000` | MSI-X PBA (pending bits) | emu thread | guest | +| `0x110040` | GSP MAILBOX0 (boot args) | guest | emu thread| +| `0x110C00` | GSP doorbell | guest | emu thread| +| `0x300000` | VBIOS (1MB PROM window) | host init | guest | +| rest | Emulated register space | shadow | shadow | -### 6. Performance Requirements - -| Metric | Target | Measurement Method | -|--------|--------|-------------------| -| GSP RPC round-trip (emulated) | < 10μs | ktime_get_ns() in gsp_emu_thread | -| BAR register access | < 2μs | Guest rdtsc around MMIO | -| VRAM allocation (ALLOC) | < 50μs | Guest nvidia.ko tracing | -| VRAM map (MAP) | < 20μs | Guest nvidia.ko tracing | -| DMA pin (PIN) | < 30μs | Guest nvidia.ko tracing | -| CUDA kernel launch overhead | < 5% vs native | CUDA events, nvprof | -| Graphics frame time (Vulkan) | < 2ms @ 60fps | PresentMon, vk_perf | -| Migration downtime | < 100ms | Stopwatch, ping during migration | -| MSI-X injection latency | < 5μs | Guest ISR timestamp vs host | - -### 7. Testing Strategy - Incremental Validation - -#### Phase 0: Host Infrastructure (Week 1) -- [ ] `open-iov-vfio.ko` loads, registers mdev parent with `nvidia.ko` -- [ ] mdev creation via sysfs succeeds -- [ ] PCI config space presented correctly (lspci in guest shows 10de:1ed3) -- [ ] BAR regions mmap'able in guest -- [ ] MSI-X table/PBA accessible in guest BAR0 - -#### Phase 1: GSP Emulation Core (Week 2) -- [ ] GSP emulation thread starts per mdev -- [ ] Doorbell polling detects guest writes -- [ ] ALLOC RPC: guest allocates VRAM, gets valid GPU VA + handle -- [ ] FREE RPC: guest frees allocation, no leaks -- [ ] MAP/UNMAP RPC: guest maps/unmaps GPU VA ranges -- [ ] PIN/UNPIN RPC: guest pins pages, gets valid DMA addresses - -#### Phase 2: Advanced RPCs (Week 3) -- [ ] SCHEDULE/YIELD: guest submits work, gets completion -- [ ] BAR_TRAP: guest reads/writes BAR registers, gets emulated values -- [ ] NOTIFY: guest registers for async events, receives them -- [ ] MIGRATE: 4-phase migration saves/restores state - -#### Phase 3: Guest Driver Integration (Week 4) -- [ ] Guest `nvidia.ko` loads, probes vGPU successfully -- [ ] `nvidia-smi` shows vGPU (T4 / A100 equivalent) -- [ ] CUDA context creation succeeds -- [ ] Simple CUDA kernel (vectorAdd) executes correctly -- [ ] `nvidia-ml` (NVML) queries work (temperature, utilization, memory) - -#### Phase 4: Compute Workloads (Week 5) -- [ ] cuBLAS SGEMM/DGEMM passes numerical verification -- [ ] cuDNN convolution forward/backward passes -- [ ] PyTorch/TensorFlow training step completes -- [ ] Multi-GPU (NCCL) across vGPUs works -- [ ] CUDA Graphs capture/replay works - -#### Phase 5: Graphics Workloads (Week 6) -- [ ] Vulkan: `vkCreateDevice` succeeds, render triangle -- [ ] OpenGL: `glxinfo` shows renderer, `glxgears` runs -- [ ] Video encode: FFmpeg + NVENC H.264/HEVC -- [ ] Video decode: FFmpeg + NVDEC H.264/HEVC -- [ ] Windows guest: D3D11/D3D12 device creation, render loop - -#### Phase 6: Stress & Reliability (Week 7) -- [ ] 24-hour CUDA stress test (memtest, compute) -- [ ] 24-hour graphics stress (Unigine, FurMark equivalent) -- [ ] Hot-plug VF create/destroy cycles (1000x) -- [ ] Migration under load (live migrate running CUDA kernel) -- [ ] Host reboot recovery (mdev persists, guest recovers) - -#### Phase 7: Windows Guest (Week 8) -- [ ] Windows 10/11 guest installs NVIDIA driver -- [ ] Device Manager shows "NVIDIA T4" (or equivalent) -- [ ] CUDA-Z / GPU-Z detect vGPU correctly -- [ ] DirectX 11/12: D3D11CreateDevice, D3D12CreateDevice -- [ ] Vulkan on Windows: vkCreateInstance, render -- [ ] Video encode/decode via Media Foundation - -### 8. Implementation File Map +**Critical**: The GSP emulation thread polls the **shadow** exclusively. Touching the physical GPU's BAR0 (even via `ioremap`) caused the original GSP crash (fixed `f60ae7f`). -``` -open-iov/ -├── kernel/ -│ ├── gsp_emulation.c # NEW: GSP emulation thread + RPC processing -│ ├── gsp_emulation.h # NEW: Structures, prototypes -│ ├── gsp_rpc_client.c # EXISTING: Firmware-decompiled handlers -│ ├── gsp_rpc_integration.c # EXISTING: Integration layer -│ ├── bar_emulation.c # EXISTING: BAR trap handling -│ ├── memory.c # EXISTING: DMA, VRAM management -│ ├── irq.c # EXISTING: MSI-X injection -│ ├── migration.c # EXISTING: Live migration -│ ├── main.c # MODIFY: Add gsp_emu_context to nv_vgpu_device -│ └── Makefile # MODIFY: Add gsp_emulation.o -├── userspace/lib/ -│ ├── libvgpu.c # EXISTING: Management API -│ ├── rpc_emulator.c # EXISTING: Host-side emulation fallbacks -│ └── Makefile # UNCHANGED -├── scripts/ -│ ├── test_gsp_emu.sh # NEW: Automated GSP emulation tests -│ ├── test_guest_driver.sh # NEW: Guest driver validation -│ ├── test_cuda_workloads.sh # NEW: CUDA workload validation -│ ├── test_graphics_workloads.sh # NEW: Graphics validation -│ └── test_windows_guest.sh # NEW: Windows guest validation -└── docs/ - ├── refactored-architecture.md # THIS DOCUMENT - └── refactored-implementation-to-do.md # TODO LIST -``` +### 4.2 BAR1 Emulated Vidmem (256 MB vzalloc) + +Guest VRAM aperture backed by host RAM. This is a **placeholder** — for real compute, this must be backed by physical GPU memory (see Phase 3 below). + +### 4.3 BAR3 Shadow (32 MB vzalloc) + +Instance memory window. **Must be shadowed** — mapping to the physical GPU's BAR3 would let guest writes corrupt host GPU instance memory (fixed `85d0668`). + +### 4.4 GSP Boot Emulation + +The guest driver boots "GSP" through falcon stages: + +1. **FWSEC FRTS** (GSP falcon CPUCTL) → auto-complete: HALTED + VBIOS_SCRATCH[0x0E]=0 + WPR2 ADDR +2. **Booter Load** (SEC2 falcon CPUCTL) → auto-complete: SEC2_MAILBOX0=0 + HALTED + RISCV_STATUS=1 +3. **GSP-RM msgq init** (libos args via MAILBOX0/1) → our emulation parses the queue chain and processes RPCs + +### 4.5 GSP-RM Emulation + +The guest's physical-path driver uses the **real GSP-RM msgq protocol** (not the old PV doorbell/BAR0-shared-mem). Our emulation: +- Parses `LibosMemoryRegionInitArgument` chain from GSP MAILBOX0/1 +- Finds the shared queue buffer in guest sysmem (129 non-contiguous 4K pages) +- Processes commands from the command queue +- Writes replies to the status queue +- Injects MSI-X interrupts + +Key RPC types handled: GSP_INIT_DONE, SET_SYSTEM_INFO, SET_REGISTRY, SET_GUEST_SYSTEM_INFO, GET_GSP_STATIC_INFO, GSP_RM_CONTROL (multiple sub-commands). + +--- + +## 5. Phased Implementation Roadmap + +### Phase 1: Correctness — GSP-RM Emulation Complete (IN PROGRESS) + +**Goal**: `nvidia-smi` works in the guest. Guest driver fully initializes. + +**What's needed**: Complete GSP-RM reply content for the RM init sequence: +- `INTERNAL_GET_DEVICE_INFO_TABLE` (0x20800a40) — 11 device entries +- `FIFO_GET_DEVICE_INFO_TABLE` (0x20801112) — 12 engine entries +- `INTERNAL_INTR_GET_KERNEL_TABLE` (0x20800a5c) — 8 entries + subtreeMap +- `GspStaticConfigInfo` — full TU104 config +- All remaining `GSP_RM_CONTROL` sub-commands + +**Oracle**: The host apt nvidia driver (570.211.01) has a **real GSP** managing the physical GPU. Use `tools/rmq/rmq` or `nvidia-smi` on host to dump ground-truth struct layouts. Compare against `kernel/gsp_rm_tu104.h`. + +**Current blocker**: `RmInitNvDevice` fails with `Cannot initialize the device (0x24:0x72:1099)`. The fn-10 FREE timeout during cleanup is characterized but not fully resolved. + +### Phase 2: Memory Model — Guest VRAM Backed by Real GPU Memory + +**Goal**: Guest CUDA memory allocations land in physical GPU VRAM (or DMA-coherent host memory accessible by the GPU). + +**What's needed**: +- Replace emulated vidmem (BAR1 host RAM) with DMA-mapped region the physical GPU can access +- Implement `SET_PAGE_DIRECTORY`, `UPDATE_PDE_2`, `ALLOC` handlers that map guest allocations to real GPU address space +- Ensure cache coherency between guest CPU and GPU DMA + +**Key challenge**: The guest's GPU VA space must map to physical GPU VRAM addresses. The host driver's GSP manages the physical GPU's memory. We'd need to either: +- (a) Cooperate with host GSP (share VRAM regions) — complex +- (b) Use a dedicated VRAM partition for the guest — requires host driver cooperation +- (c) Use system memory with DMA mapping (slower but simpler) — guest sees "unified memory" + +### Phase 3: Compute Forwarding — Guest Work Executes on Physical GPU + +**Goal**: CUDA kernels launched in the guest execute on the physical GPU. + +**What's needed**: +- Guest channel/pushbuffer writes are intercepted and forwarded to host GPU channels +- GPU context switching between host and guest contexts +- Completion interrupts relayed from host GPU to guest + +**Approach options**: +1. **GSP-RM translation**: Translate guest GSP-RM compute commands to host GSP-RM commands. The host GSP firmware already knows how to schedule work on the GPU. We'd be a "GSP proxy". +2. **Channel forwarding**: Intercept guest channel doorbell writes, translate guest pushbuffer contents, submit to host GPU channels. Requires deep understanding of GPU channel format. +3. **UVM integration**: Use the host's Unified Virtual Memory (UVM) to share CUDA memory between host and guest. Launch kernels via host CUDA APIs on behalf of guest. + +**Most promising**: Option 1 (GSP-RM translation). The guest driver sends GSP-RM commands for compute (ALLOC, MAP, SCHEDULE). Our emulation translates these to equivalent host GSP-RM commands via the host driver. The host GSP firmware executes them on the physical GPU. Results are relayed back. + +### Phase 4: Performance Optimization + +**Goal**: < 5% overhead vs native for CUDA workloads. + +**What's needed**: +- Optimize GSP-RM translation latency (batch commands, avoid unnecessary copies) +- Direct DMA mapping for large memory transfers +- Interrupt coalescing for high-frequency completions +- Profile and eliminate hot-path bottlenecks (doorbell polling, shared memory access) + +### Phase 5: Graphics & Video + +**Goal**: OpenGL/Vulkan rendering, NVENC/NVDEC video. + +**What's needed**: +- Display surface management (virtual display or forwarded framebuffer) +- Graphics pipeline forwarding (similar to compute forwarding) +- Video engine access (NVENC/NVDEC on physical GPU) + +--- + +## 6. Testing Strategy + +### Correctness Testing (Phase 1) +- [x] Host nvidia-smi works with open-iov-vfio loaded +- [x] mdev create/remove lifecycle clean (no hang, no host GPU corruption) +- [x] QEMU guest boots with vGPU attached +- [x] Guest PCI probe detects vGPU (TU104 [10de:1ed3]) +- [x] Guest nvidia driver loads and creates /dev/nvidia0 +- [ ] Guest RmInitAdapter completes (RmInitNvDevice succeeds) +- [ ] Guest nvidia-smi shows GPU +- [ ] Guest CUDA context creation + +### Performance Testing (Phases 2-4) +- [ ] GSP-RM round-trip latency < 10μs +- [ ] CUDA kernel launch overhead < 5% vs native +- [ ] Memory bandwidth within 10% of native +- [ ] Graphics frame time within 5% of native + +--- + +## 7. Key Files + +| File | Role | +|------|------| +| `kernel/gsp_emulation.c/h` | GSP emulation thread, falcon boot auto-complete, MSI-X injection | +| `kernel/gsp_rm_emulation.c/h` | GSP-RM msgq protocol emulation (guest sysmem queues) | +| `kernel/gsp_rm_tu104.h` | TU104 hardware tables (engines, interrupts, falcons, device info) | +| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations | +| `kernel/bar_emulation.c/h` | BAR0 shadow allocation, MSI-X shadow state | +| `kernel/bar_init.c` | BAR init (shadow vidmem, BAR3 shadow, physical ioremap for reference) | +| `kernel/main.c` | VFIO ops, mdev create/remove, PCI config space, VFIO mmap | +| `kernel/memory.c` | DMA mapping, VRAM management | +| `kernel/irq.c` | MSI-X/MSI setup and injection | +| `tools/rmq/rmq.c` | Host RM API client (oracle for GSP-RM ground truth) | + +--- + +## 8. Host Stability Guarantees (2026-09-22) + +After the host-stability fix (`85d0668`), the following are guaranteed: -### 9. Risk Mitigation +1. **open-iov-vfio.ko loads/unloads cleanly** with apt nvidia driver present +2. **Host nvidia-smi works throughout** the full lifecycle (load, mdev, QEMU, unload) +3. **No host GPU corruption** — no WPR2 stuck, no GSP crash, no module hang +4. **Guest driver probe is safe** — even when RmInitNvDevice fails, host is unaffected +5. **mdev create/remove is clean** — gsp_emu thread starts/stops properly -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| GSP mailbox layout mismatch | Medium | High | Validate against firmware decompilation; add runtime checksum | -| MSI-X injection race | Medium | High | Use atomic PBA updates; test with concurrent RPCs | -| Cache coherency (WC memory) | High | Medium | clflush + mfence on every shared mem write; test on AMD/Intel | -| Guest driver version skew | Medium | Medium | Test against multiple OpenRM versions (550, 560, 570) | -| Windows guest driver issues | Medium | High | Separate Windows test CI; use WHQL driver | -| Performance regression vs native | Low | High | Continuous benchmarking; profile hot paths | -| Migration state corruption | Low | High | Checksum all migrated state; dry-run validation | +**Root causes that were fixed**: +- Submodule nvidia.ko conflict with apt nvidia → skip submodule +- BAR3 physical mapping to guest → vzalloc'd shadow +- GSP-RM teardown garbage declaring GSP death → quiet stop polling --- -*Document Version: 1.0 (Refactored Architecture)* -*Author: Open-IOV Team* -*Date: $(date)* -*Repository: $OPEN_IOV_ROOT* -*Git Commit: f9ed5d8 (restore point)* \ No newline at end of file +*Document Version: 2.0 (Emulate vs Pass-Through Architecture)* +*Updated: 2026-09-22 (post host-stability fix)* +*Repository: ~/Git/integdev_gpu_drv/open-iov* +*Git Commit: 85d0668 (host stability fix)* -- cgit v1.2.3