diff options
Diffstat (limited to 'docs/ARCHITECTURE.md')
| -rw-r--r-- | docs/ARCHITECTURE.md | 476 |
1 files changed, 476 insertions, 0 deletions
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..f9b7ae4 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,476 @@ +# Open-IOV vGPU-IO Architecture Document + +## Overview + +Open-IOV vGPU-IO is a complete open-source implementation of NVIDIA vGPU (virtual GPU) technology for Turing architecture GPUs (TU102/TU104), enabling GPU virtualization without proprietary GRID host drivers. The project provides: + +- **Host-side VFIO mdev driver** (`open-iov-vfio.ko`) - Creates mediated devices for QEMU/KVM +- **Guest-side vGPU driver** (`gsp-vgpu.ko`) - Communicates with GSP firmware via RPC +- **Userspace library** (`libiov.so`/`libvgpu.so`) - C API for VF management and GPU operations +- **Firmware decompilation** - Complete reverse engineering of GSP 570.144W firmware + +## System Architecture + +``` +╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ HOST SYSTEM ║ +║ ╔════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ USERSPACE (Host) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌──────────────┐ ┌─────────────────────┐ ┌──────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ QEMU/KVM │ │ libvfio-user │ │ mdev management CLI / mdev management daemon │ │ ║ ║ +║ ║ │ │ │ │ │ │ (GPU Virtualization Management CLI/Manager) │ │ ║ ║ +║ ║ │ │ -device vfio-│ │ - VFIO userspace │ │ │ │ ║ ║ +║ ║ │ │ pci,sysfsdev│ │ library │ │ • mdev create/destroy │ │ ║ ║ +║ ║ │ │ │ │ - ioctl/eventfd │ │ • VF lifecycle management │ │ ║ ║ +║ ║ │ │ ┌──────────┐ │ │ - mmap support │ │ • Migration orchestration │ │ ║ ║ +║ ║ │ │ │ VFIO │ │ │ │ │ • Resource allocation │ │ ║ ║ +║ ║ │ │ │ Region │ │ └──────────┬──────────┘ └──────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ │ (mdev) │ │ │ │ ║ ║ +║ ║ │ │ └────┬─────┘ │ │ │ ║ ║ +║ ║ │ │ │ │ ▼ │ ║ ║ +║ ║ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ │ │ │ libvgpu.so / libiov.so (Host-side) │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_open/close/get_info │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_vf_create/destroy/get_info │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_mem_alloc/free/map/unmap/pin/unpin │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_bar_config/force_access/save/restore │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_schedule/yield │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_register/unregister_event_callback, vgpu_process_events │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_migrate_start/status/cancel │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ ║ ║ +║ ║ │ │ │ │ │ │ rpc_emulator.c (Host-side Emulation) │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_bar_config/save/restore/get_state │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_wpr_protect/unprotect/set_timeout/check_access │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_schedule/yield │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_notify_register/unregister/send │ │ │ ║ ║ +║ ║ │ │ │ │ │ └──────────────────────────────────────────────────────────────────┘ │ │ ║ ║ +║ ║ │ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ └──────┼────────┘ │ ║ ║ +║ ║ │ │ VFIO_DEVICE_GET_REGION_INFO / VFIO_DEVICE_GET_IRQ_INFO / VFIO_DEVICE_RESET │ ║ ║ +║ ║ │ │ mmap() / ioctl() / read() / write() / poll() │ ║ ║ +║ ║ │ ▼ │ ║ ║ +║ ║ └─────────┼──────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ │ ║ ║ +║ ║ ▼ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ KERNEL SPACE (Host) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ nvidia.ko (Host RM Kernel Module) │ │ ║ ║ +║ ║ │ │ (NVIDIA OpenRM 570.144 - Host Driver) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • GSP Firmware Boot & Management │ │ ║ ║ +║ ║ │ │ • Physical GPU Resource Management (VRAM, engines, etc.) │ │ ║ ║ +║ ║ │ │ • vGPU VFIO Symbol Exports: │ │ ║ ║ +║ ║ │ │ - nvidia_vgpu_vfio_get_ops() │ │ ║ ║ +║ ║ │ │ - nvidia_vgpu_vfio_set_ops() │ │ ║ ║ +║ ║ │ │ - nv_vgpu_mdev_register_parent() │ │ ║ ║ +║ ║ │ │ - nv_vgpu_mdev_unregister_parent() │ │ ║ ║ +║ ║ │ │ • GPC Callbacks (from open-iov-vfio.ko): │ │ ║ ║ +║ ║ │ │ - vgpu_create / vgpu_delete / vgpu_bar_info / vgpu_start │ │ ║ ║ +║ ║ │ │ - get_name / get_description / get_instances / get_sparse_mmap │ │ ║ ║ +║ ║ │ │ - update_request │ │ ║ ║ +║ ║ │ │ • Host IRQ/MSI-X Handler (owns physical GPU interrupt) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ open-iov-vfio.ko (Host VFIO mdev Driver) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • mdev_parent registration with nvidia.ko │ │ ║ ║ +║ ║ │ │ • VFIO Device Ops: probe/remove/open/close/ioctl/mmap │ │ ║ ║ +║ ║ │ │ • PCI Config Space Emulation (256 bytes): │ │ ║ ║ +║ ║ │ │ - Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU) │ │ ║ ║ +║ ║ │ │ - BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB (all 64-bit) │ │ ║ ║ +║ ║ │ │ - Capabilities: PM, MSI, MSI-X (Table@BAR0+0x2000), PCIe │ │ ║ ║ +║ ║ │ │ • BAR Trap Emulation (bar_emulation.c): │ │ ║ ║ +║ ║ │ │ - 64 trapped regions covering all BAR0 registers │ │ ║ ║ +║ ║ │ │ - Shadow memory for register state │ │ ║ ║ +║ ║ │ │ • GSP Communication (gsp_comm.c): │ │ ║ ║ +║ ║ │ │ - Polling-based doorbell (queue head @ 0x110c00) │ │ ║ ║ +║ ║ │ │ - Shared memory region for RPC messages │ │ ║ ║ +║ ║ │ │ • Firmware-Decompiled RPC Client (gsp_rpc_client.c + gsp_rpc_integration.c): │ │ ║ ║ +║ ║ │ │ - All 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │ ║ ║ +║ ║ │ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │ ║ ║ +║ ║ │ │ - Real firmware addresses from decompilation │ │ ║ ║ +║ ║ │ │ • Memory Management (memory.c): │ │ ║ ║ +║ ║ │ │ - DMA mapping, pin/unpin, map/unmap │ │ ║ ║ +║ ║ │ │ • IRQ Forwarding (irq.c): │ │ ║ ║ +║ ║ │ │ - MSI-X table/PBA emulation │ │ ║ ║ +║ ║ │ │ - eventfd trigger injection to guest │ │ ║ ║ +║ ║ │ │ • Live Migration (migration.c): │ │ ║ ║ +║ ║ │ │ - 4-phase: QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP │ │ ║ ║ +║ ║ │ │ - Dual jump tables (0x04179ED8, 0x04179FBC) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ vfio-mdev.ko / vfio-pci.ko (Kernel VFIO Core) │ │ ║ ║ +║ ║ │ │ • Mediated device framework │ │ ║ ║ +║ ║ │ │ • VFIO PCI device emulation │ │ ║ ║ +║ ║ │ │ • IOMMU DMA mapping (DMA remapping) │ │ ║ ║ +║ ║ │ │ • Region/IRQ index mapping: │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR0_REGION_INDEX → BAR0 (16MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR1_REGION_INDEX → BAR1 (256MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR2_REGION_INDEX → BAR2 (32MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR3_REGION_INDEX → BAR3 (32MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_CONFIG_REGION_INDEX → PCI Config Space (256B) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table/PBA │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ HARDWARE (Physical GPU) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ NVIDIA TU102/TU104 GPU (RTX 2080 Ti / T4 / etc.) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ ║ ║ +║ ║ │ │ BAR0 │ │ BAR1 │ │ BAR2 │ │ BAR3 │ │ GSP FALCON │ │ ║ ║ +║ ║ │ │ (16 MB) │ │ (256 MB) │ │ (32 MB) │ │ (32 MB) │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ ┌─────────────────┐ │ │ ║ ║ +║ ║ │ │ Config Regs │ │ Framebuffer │ │ Reserved │ │ Reserved │ │ │ REAL GSP │ │ │ ║ ║ +║ ║ │ │ Doorbell │ │ (VRAM) │ │ │ │ │ │ │ FIRMWARE │ │ │ ║ ║ +║ ║ │ │ Queue Heads │ │ │ │ │ │ │ │ │ (570.144W) │ │ │ ║ ║ +║ ║ │ │ Shared Mem │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ RPC Dispatchers │ │ │ ║ ║ +║ ║ │ │ 0x110c00 │ │ │ │ │ │ │ │ │ • Primary │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x057A1AC0 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • Secondary │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04ED39DC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ Specialized │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ Handlers: │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • BAR_TRAP │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04CB8188 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • WPR x2 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04F00B0C │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04F00C78 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • ALLOC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x057E94EC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • SCHEDULE │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x051EFBF4 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • Migration │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04179ED8 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04179FBC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 12 RPC Types: │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ ALLOC, FREE, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ MAP, UNMAP, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ PIN, UNPIN, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ SCHEDULE, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ YIELD, NOTIFY, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ BAR_TRAP, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ MIGRATE, DEBUG │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ └─────────────────┘ │ │ ║ ║ +║ ║ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ Physical IRQ (MSI-X) ──────────────────────────────────────────────► nvidia.ko ISR │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ + + │ VFIO MEDIATED DEVICE (mdev) │ + │ sysfs: /sys/bus/mdev/devices/<UUID>/ │ + │ • PCI Config Space (256B) │ + │ • BAR0 Region (16MB) │ + │ • BAR1 Region (256MB) │ + │ • BAR2 Region (32MB) │ + │ • BAR3 Region (32MB) │ + │ • MSI-X Table/PBA Region │ + │ • IRQs (MSI-X vectors) │ + ▼ + +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ GUEST VM ║ +║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ USERSPACE (Guest) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ Guest Applications │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • CUDA Applications │ │ ║ ║ +║ ║ │ │ • OpenGL/Vulkan Applications │ │ ║ ║ +║ ║ │ │ • Video Encode/Decode (NVENC/NVDEC) │ │ ║ ║ +║ ║ │ │ • AI/ML Workloads (cuDNN, TensorRT) │ │ ║ ║ +║ ║ │ │ • Custom vGPU Management (via libvgpu.so) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ libvgpu.so / libiov.so (Guest-side Userspace Library) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • vgpu_open/close/get_info │ │ ║ ║ +║ ║ │ │ • vgpu_vf_create/destroy/get_info │ │ ║ ║ +║ ║ │ │ • vgpu_mem_alloc/free/map/unmap/pin/unpin │ │ ║ ║ +║ ║ │ │ • vgpu_bar_config/force_access/save/restore │ │ ║ ║ +║ ║ │ │ • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback │ │ ║ ║ +║ ║ │ │ • vgpu_schedule/yield │ │ ║ ║ +║ ║ │ │ • vgpu_register/unregister_event_callback, vgpu_process_events │ │ ║ ║ +║ ║ │ │ • vgpu_migrate_start/status/cancel │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ ┌────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ ║ ║ +║ ║ │ │ │ rpc_emulator.c (Guest-side Emulation Fallbacks) │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_bar_config/save/restore/get_state │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_wpr_protect/unprotect/set_timeout/check_access │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_schedule/yield │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_notify_register/unregister/send │ │ │ ║ ║ +║ ║ │ │ └────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ libnvidia-ml.so / libnvidia-opencl.so / libcuda.so (NVIDIA Userspace Drivers) │ │ ║ ║ +║ ║ │ │ • Standard NVIDIA userspace driver stack (unchanged) │ │ ║ ║ +║ ║ │ │ • Communicates with guest nvidia.ko via ioctl on /dev/nvidia* │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ═════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ KERNEL SPACE (Guest) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ nvidia.ko (Guest RM Kernel Module) │ │ ║ ║ +║ ║ │ │ (NVIDIA OpenRM 570.144 - Guest Driver, UNMODIFIED) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • Detects vGPU via PCI Config Space (Vendor=0x10DE, Device=0x1ED3) │ │ ║ ║ +║ ║ │ │ • Maps BAR0 (config), BAR1 (framebuffer), BAR2, BAR3 │ │ ║ ║ +║ ║ │ │ • Sets up MSI-X interrupts (table @ BAR0+0x2000, PBA @ BAR0+0x3000) │ │ ║ ║ +║ ║ │ │ • GSP Client: │ │ ║ ║ +║ ║ │ │ - Writes to BAR0 doorbell (queue head @ 0x110c00) │ │ ║ ║ +║ ║ │ │ - Reads/writes shared memory for RPC messages │ │ ║ ║ +║ ║ │ │ - Expects GSP firmware with vGPU RPC handlers │ │ ║ ║ +║ ║ │ │ • RM (Resource Manager) for GPU resources │ │ ║ ║ +║ ║ │ │ • nvidia-modeset.ko / nvidia-uvm.ko / nvidia-peermem.ko │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ gsp-vgpu.ko (Guest vGPU Driver - Open-IOV) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • Char Device: /dev/gsp-vgpu (misc device) │ │ ║ ║ +║ ║ │ │ • IOCTL Interface (matches libvgpu.so API): │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_CREATE_VF / DESTROY_VF │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_ALLOC / FREE / MAP / UNMAP / PIN / UNPIN │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_BAR_CONFIG / FORCE / SAVE / RESTORE │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_WPR_PROTECT / UNPROTECT / TIMEOUT │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_SCHEDULE / YIELD │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_MIGRATE │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_GET_INFO │ │ ║ ║ +║ ║ │ │ • RPC Client (rpc/gsp_rpc.c): │ │ ║ ║ +║ ║ │ │ - Doorbell writes to queue head │ │ ║ ║ +║ ║ │ │ - Ring buffer management │ │ ║ ║ +║ ║ │ │ • Emulation Mode (AUTO-DETECTED): │ │ ║ ║ +║ ║ │ │ - Test BAR_TRAP RPC at load; if timeout → enable emulation │ │ ║ ║ +║ ║ │ │ - Module params: emulation_mode (ro), emulation_override (force) │ │ ║ ║ +║ ║ │ │ - In emulation: BAR_TRAP, WPR, ALLOC, PIN, MAP handled locally │ │ ║ ║ +║ ║ │ │ • Debugfs: /sys/kernel/debug/gsp-vgpu/ │ │ ║ ║ +║ ║ │ │ • VF Lifecycle (vf/gsp_vf.c) │ │ ║ ║ +║ ║ │ │ • BAR Emulation (bar/gsp_bar.c) │ │ ║ ║ +║ ║ │ │ • WPR Management (wpr/gsp_wpr.c) │ │ ║ ║ +║ ║ │ │ • Migration (migration/gsp_migration.c) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ VFIO-PCI Core (Guest Kernel) │ │ ║ ║ +║ ║ │ │ • vfio-pci.ko: PCI device emulation for VFIO │ │ ║ ║ +║ ║ │ │ • Handles mmap of BAR regions │ │ ║ ║ +║ ║ │ │ • MSI-X interrupt handling │ │ ║ ║ +║ ║ │ │ • Config space read/write │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ══════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ VIRTUAL GPU (mdev Device as seen by Guest) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ PCI Config Space (256B) │ BAR0 (16MB) │ BAR1 (256MB) │ BAR2 (32MB) │ BAR3 (32MB) │ MSI-X │ ║ ║ +║ ║ │ • Vendor=0x10DE │ Config Regs │ Framebuffer │ Reserved │ Reserved │ Table │ ║ ║ +║ ║ │ • Device=0x1ED3 │ Doorbell │ (VRAM) │ │ │ @0x2000│ ║ ║ +║ ║ │ • Class=0x030000 │ Queue Heads │ │ │ │ PBA │ ║ ║ +║ ║ │ • BARs: 64-bit, prefetch │ Shared Mem │ │ │ │ @0x3000│ ║ ║ +║ ║ │ • Caps: PM, MSI, MSI-X, │ @0x110c00 │ │ │ │ │ ║ ║ +║ ║ │ PCIe │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ╚═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +╚══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +``` + +## Communication Pathways (Host ↔ Guest) + +### 1. **VFIO Region Access (Memory-Mapped I/O)** +``` +Guest App → libvgpu.so → ioctl(/dev/gsp-vgpu) → gsp-vgpu.ko + → Guest vfio-pci.ko → mmap(BAR0/BAR1/BAR2/BAR3) + → Host vfio-pci.ko → open-iov-vfio.ko BAR emulation + → Host nvidia.ko → Physical GPU BARs +``` +**Path**: Guest userspace → Guest kernel → Host kernel → Physical GPU +**Mechanism**: VFIO mmap with IOMMU DMA remapping + +### 2. **PCI Config Space Access** +``` +Guest nvidia.ko → pci_read_config_* / pci_write_config_* + → Guest vfio-pci.ko → VFIO_DEVICE_GET_REGION_INFO (CONFIG_REGION) + → Host open-iov-vfio.ko → nv_vgpu_config_read/write() + → Returns emulated 256-byte config space +``` +**Path**: Guest kernel → Host kernel (via VFIO ioctl) +**Mechanism**: VFIO config space region emulation + +### 3. **GSP RPC Communication (Native Path)** +``` +Guest nvidia.ko → Write BAR0 doorbell (0x110c00) + shared memory + → Physical GPU GSP firmware receives RPC + → GSP processes RPC (ALLOC, MAP, PIN, SCHEDULE, BAR_TRAP, etc.) + → GSP writes response to shared memory + → Guest nvidia.ko reads response +``` +**Path**: Guest kernel → Physical GPU (direct, no host kernel involvement) +**Mechanism**: Doorbell register + shared memory in BAR0 + +### 4. **GSP RPC Communication (Emulation Path - Host Mediated)** +``` +Guest nvidia.ko → Write BAR0 doorbell (0x110c00) + → Host open-iov-vfio.ko (polling) detects doorbell + → gsp_comm.c → gsp_rpc_integration.c → gsp_rpc_client.c + → Firmware-decompiled RPC handlers (emulated in host kernel) + → Response written to shared memory + → Guest nvidia.ko reads response +``` +**Path**: Guest kernel → Host kernel (open-iov-vfio.ko) → Emulated RPC +**Mechanism**: Host polls GSP doorbell, processes RPC in software + +### 5. **BAR Trap Emulation** +``` +Guest nvidia.ko → Write to trapped BAR register (page fault) + → Guest vfio-pci.ko → VFIO page fault handler + → Host open-iov-vfio.ko → nv_vgpu_bar_trap() + → If native: gsp_rpc_bar_trap() → GSP firmware (0x04CB8188) + → If emulated: rpc_emu_bar_trap() in host userspace (libvgpu.so) + → Result written back to guest BAR mapping +``` +**Path**: Guest kernel page fault → Host kernel → Host userspace (emulation) or GSP firmware +**Mechanism**: VFIO page fault + BAR trap emulation + +### 6. **Interrupt Forwarding (Host → Guest)** +``` +Physical GPU → MSI-X IRQ → Host nvidia.ko ISR + → nvidia.ko calls nv_vgpu_vfio_inject_interrupt() (VFIO ops) + → open-iov-vfio.ko → eventfd_signal() + → QEMU/KVM → Guest MSI-X interrupt injected + → Guest nvidia.ko ISR handles interrupt +``` +**Path**: Physical GPU → Host kernel → Host userspace (QEMU) → Guest kernel +**Mechanism**: VFIO interrupt injection via eventfd + +### 7. **Userspace Management (mdev management CLI/mdev management daemon)** +``` +mdev management CLI → libvgpu.so (host) → ioctl(/dev/vfio/<mdev>) + → open-iov-vfio.ko → mdev VFIO ops + → VF creation, BAR config, memory ops, migration +``` +**Path**: Host userspace → Host kernel (VFIO mdev) +**Mechanism**: Standard VFIO ioctl on mdev device + +### 8. **Guest vGPU Driver Communication** +``` +Guest App → libvgpu.so (guest) → ioctl(/dev/gsp-vgpu) + → gsp-vgpu.ko → Internal RPC client + → If emulation: Local handling in gsp-vgpu.ko + → If native: GSP RPC via BAR0 doorbell +``` +**Path**: Guest userspace → Guest kernel (gsp-vgpu.ko) → GSP or emulation + +## Key Component Summary + +| Component | Location | Type | Purpose | +|-----------|----------|------|---------| +| **nvidia.ko (Host)** | Host Kernel | RM Driver | Boots GSP, manages physical GPU, exports vGPU VFIO symbols | +| **open-iov-vfio.ko** | Host Kernel | VFIO mdev Driver | Creates mdev, PCI config space, BAR emulation, RPC integration, IRQ forwarding | +| **gsp-vgpu.ko** | Guest Kernel | vGPU Driver | Char device `/dev/gsp-vgpu`, ioctl API, emulation mode, RPC client | +| **nvidia.ko (Guest)** | Guest Kernel | RM Driver | Standard NVIDIA driver, detects vGPU, communicates with GSP | +| **libvgpu.so / libiov.so (Host)** | Host Userspace | Library | Management API (mdev management CLI/mdev management daemon), emulation fallbacks | +| **libvgpu.so / libiov.so (Guest)** | Guest Userspace | Library | Application API, emulation fallbacks | +| **libnvidia-*.so** | Guest Userspace | NVIDIA Drivers | Standard CUDA/OpenGL/OpenCL userspace stack | +| **mdev management CLI / mdev management daemon** | Host Userspace | Tools | mdev lifecycle, VF management, migration orchestration | +| **REAL GSP Firmware** | Physical GPU | Firmware | 570.144W, 12 RPC types, BAR_TRAP/WPR/SCHEDULE/MIGRATE handlers | +| **EMULATED GSP** | Host Kernel + Userspace | Software | rpc_emulator.c, gsp_rpc_client.c (when GSP lacks vGPU handlers) | + +## Emulation Mode Details + +### When Enabled +- **Auto-detect**: At gsp-vgpu.ko load, sends test BAR_TRAP RPC +- **Timeout/Error** → `emulation_mode = true` +- **Module params**: `emulation_override=1` (force on), `emulation_override=0` (force off) + +### What Gets Emulated +| Operation | Host Kernel (open-iov-vfio.ko) | Host Userspace (libvgpu.so) | Guest Kernel (gsp-vgpu.ko) | +|-----------|-------------------------------|----------------------------|---------------------------| +| BAR_TRAP | nv_vgpu_bar_trap() → rpc_emu_bar_trap() | rpc_emu_bar_config/save/restore | gsp_bar_force_access() | +| WPR | nv_vgpu_wpr_*() | rpc_emu_wpr_protect/unprotect | gsp_vf_wpr_protect/unprotect | +| ALLOC/MAP/PIN | gsp_rpc_client.c (local) | rpc_emu_* fallbacks | gsp_vf_mem_alloc/map/pin | +| SCHEDULE/YIELD | gsp_rpc_schedule/yield (workqueue) | rpc_emu_schedule/yield | gsp_vf_schedule/yield | +| NOTIFY | gsp_rpc_notify_register/unregister | rpc_emu_notify_register/unregister/send | gsp_vf_notify_* | +| MIGRATION | migration.c (4-phase) | rpc_emu_migrate_* | gsp_migration_start | + +## Build Artifacts + +``` +$OPEN_IOV_ROOT/ +├── nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko (34MB) Host RM Driver +├── kernel/open-iov-vfio.ko (2.8MB) Host VFIO mdev Driver +├── driver/gsp-vgpu.ko (1.8MB) Guest vGPU Driver +├── userspace/lib/libiov.so (76KB) Userspace Library (libvgpu.so symlink) +├── userspace/examples/example_vgpu (28KB) Test Application +├── firmware/570-GSP/gsp-570.144.fwimage.bin (28MB) GSP Firmware +├── scripts/load_vgpu_stack.sh Host module loader +├── scripts/create_mdev.sh mdev creation +├── scripts/launch_qemu_vgpu.sh QEMU launcher +├── scripts/test_vgpu_stack.sh Test suite +├── scripts/setup_guest_driver.sh Guest driver setup +├── scripts/unload_vgpu_stack.sh Cleanup +├── docs/ARCHITECTURE.md This document +└── wiki/ 10 technical docs +``` + +## Deployment Sequence + +```bash +# HOST (after reboot to clear module_blacklist) +# 1. Load NVIDIA OpenRM host driver (exports vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO mdev driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Create mediated device for QEMU +./scripts/create_mdev.sh # Creates /sys/bus/mdev/devices/<UUID>/ + +# 4. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh # -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID> + +# GUEST (inside VM) +# 5. Install NVIDIA driver (unmodified OpenRM) +./scripts/setup_guest_driver.sh # Installs nvidia.ko, firmware, loads driver + +# 6. Verify +nvidia-smi # Should show vGPU (T4 / A100 / etc.) +lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) +``` + +--- + +*Architecture Version: 2.0 (with detailed host/guest communication pathways)* +*Last Updated: $(date)* +*Project: $OPEN_IOV_ROOT*
\ No newline at end of file |