Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 476 insertions, 0 deletions

diff --git a/docs/old/ARCHITECTURE.md b/docs/old/ARCHITECTURE.md
new file mode 100644
index 0000000..f9b7ae4
--- /dev/null
+++ b/docs/old/ARCHITECTURE.md
@@ -0,0 +1,476 @@
+# Open-IOV vGPU-IO Architecture Document
+
+## Overview
+
+Open-IOV vGPU-IO is a complete open-source implementation of NVIDIA vGPU (virtual GPU) technology for Turing architecture GPUs (TU102/TU104), enabling GPU virtualization without proprietary GRID host drivers. The project provides:
+
+- **Host-side VFIO mdev driver** (`open-iov-vfio.ko`) - Creates mediated devices for QEMU/KVM
+- **Guest-side vGPU driver** (`gsp-vgpu.ko`) - Communicates with GSP firmware via RPC
+- **Userspace library** (`libiov.so`/`libvgpu.so`) - C API for VF management and GPU operations
+- **Firmware decompilation** - Complete reverse engineering of GSP 570.144W firmware
+
+## System Architecture
+
+```
+╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
+║                                                         HOST SYSTEM                                              ║
+║  ╔════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║                                                                                                            ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │                                    USERSPACE (Host)                                                   │  ║  ║
+║  ║  │                                                                                                      │  ║  ║
+║  ║  │  ┌──────────────┐    ┌─────────────────────┐    ┌──────────────────────────────────────────────┐   │  ║  ║
+║  ║  │  │   QEMU/KVM   │    │    libvfio-user     │    │           mdev management CLI / mdev management daemon                   │   │  ║  ║
+║  ║  │  │              │    │                     │    │  (GPU Virtualization Management CLI/Manager)   │   │  ║  ║
+║  ║  │  │ -device vfio-│    │ - VFIO userspace    │    │                                              │   │  ║  ║
+║  ║  │  │  pci,sysfsdev│    │   library           │    │  • mdev create/destroy                       │   │  ║  ║
+║  ║  │  │              │    │ - ioctl/eventfd     │    │  • VF lifecycle management                   │   │  ║  ║
+║  ║  │  │ ┌──────────┐ │    │ - mmap support      │    │  • Migration orchestration                   │   │  ║  ║
+║  ║  │  │ │  VFIO    │ │    │                     │    │  • Resource allocation                       │   │  ║  ║
+║  ║  │  │ │  Region  │ │    └──────────┬──────────┘    └──────────────────────────────────────────────┘   │  ║  ║
+║  ║  │  │ │  (mdev)  │ │               │                                                                     │  ║  ║
+║  ║  │  │ └────┬─────┘ │               │                                                                     │  ║  ║
+║  ║  │  │      │       │               ▼                                                                     │  ║  ║
+║  ║  │  │      │       │    ┌──────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║  │  │      │       │    │                    libvgpu.so / libiov.so (Host-side)                     │   │  ║  ║
+║  ║  │  │      │       │    │                                                                          │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_open/close/get_info                                              │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_vf_create/destroy/get_info                                       │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_mem_alloc/free/map/unmap/pin/unpin                               │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_bar_config/force_access/save/restore                             │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback   │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_schedule/yield                                                   │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_register/unregister_event_callback, vgpu_process_events          │   │  ║  ║
+║  ║  │  │      │       │    │  • vgpu_migrate_start/status/cancel                                      │   │  ║  ║
+║  ║  │  │      │       │    │                                                                          │   │  ║  ║
+║  ║  │  │      │       │    │  ┌──────────────────────────────────────────────────────────────────┐   │   │  ║  ║
+║  ║  │  │      │       │    │  │              rpc_emulator.c (Host-side Emulation)                 │   │   │  ║  ║
+║  ║  │  │      │       │    │  │                                                                  │   │   │  ║  ║
+║  ║  │  │      │       │    │  │  • rpc_emu_bar_config/save/restore/get_state                     │   │   │  ║  ║
+║  ║  │  │      │       │    │  │  • rpc_emu_wpr_protect/unprotect/set_timeout/check_access        │   │   │  ║  ║
+║  ║  │  │      │       │    │  │  • rpc_emu_schedule/yield                                        │   │   │  ║  ║
+║  ║  │  │      │       │    │  │  • rpc_emu_notify_register/unregister/send                       │   │   │  ║  ║
+║  ║  │  │      │       │    │  └──────────────────────────────────────────────────────────────────┘   │   │  ║  ║
+║  ║  │  │      │       │    └──────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║  │  └──────┼────────┘                                                                                   │  ║  ║
+║  ║  │         │ VFIO_DEVICE_GET_REGION_INFO / VFIO_DEVICE_GET_IRQ_INFO / VFIO_DEVICE_RESET              │  ║  ║
+║  ║  │         │ mmap() / ioctl() / read() / write() / poll()                                             │  ║  ║
+║  ║  │         ▼                                                                                          │  ║  ║
+║  ║  └─────────┼──────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║            │                                                                                           ║  ║
+║  ║            ▼                                                                                           ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │                                    KERNEL SPACE (Host)                                               │  ║  ║
+║  ║  │                                                                                                      │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                         nvidia.ko (Host RM Kernel Module)                                     │  │  ║  ║
+║  ║  │  │  (NVIDIA OpenRM 570.144 - Host Driver)                                                        │  │  ║  ║
+║  ║  │  │                                                                                                 │  │  ║  ║
+║  ║  │  │  • GSP Firmware Boot & Management                                                             │  │  ║  ║
+║  ║  │  │  • Physical GPU Resource Management (VRAM, engines, etc.)                                     │  │  ║  ║
+║  ║  │  │  • vGPU VFIO Symbol Exports:                                                                  │  │  ║  ║
+║  ║  │  │    - nvidia_vgpu_vfio_get_ops()                                                               │  │  ║  ║
+║  ║  │  │    - nvidia_vgpu_vfio_set_ops()                                                               │  │  ║  ║
+║  ║  │  │    - nv_vgpu_mdev_register_parent()                                                           │  │  ║  ║
+║  ║  │  │    - nv_vgpu_mdev_unregister_parent()                                                         │  │  ║  ║
+║  ║  │  │  • GPC Callbacks (from open-iov-vfio.ko):                                                     │  │  ║  ║
+║  ║  │  │    - vgpu_create / vgpu_delete / vgpu_bar_info / vgpu_start                                   │  │  ║  ║
+║  ║  │  │    - get_name / get_description / get_instances / get_sparse_mmap                             │  │  ║  ║
+║  ║  │  │    - update_request                                                                           │  │  ║  ║
+║  ║  │  │  • Host IRQ/MSI-X Handler (owns physical GPU interrupt)                                       │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                      │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    open-iov-vfio.ko (Host VFIO mdev Driver)                                    │  │  ║  ║
+║  ║  │  │                                                                                                 │  │  ║  ║
+║  ║  │  │  • mdev_parent registration with nvidia.ko                                                    │  │  ║  ║
+║  ║  │  │  • VFIO Device Ops: probe/remove/open/close/ioctl/mmap                                        │  │  ║  ║
+║  ║  │  │  • PCI Config Space Emulation (256 bytes):                                                    │  │  ║  ║
+║  ║  │  │    - Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU)                                                │  │  ║  ║
+║  ║  │  │    - BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB (all 64-bit)                                │  │  ║  ║
+║  ║  │  │    - Capabilities: PM, MSI, MSI-X (Table@BAR0+0x2000), PCIe                                   │  │  ║  ║
+║  ║  │  │  • BAR Trap Emulation (bar_emulation.c):                                                      │  │  ║  ║
+║  ║  │  │    - 64 trapped regions covering all BAR0 registers                                           │  │  ║  ║
+║  ║  │  │    - Shadow memory for register state                                                         │  │  ║  ║
+║  ║  │  │  • GSP Communication (gsp_comm.c):                                                            │  │  ║  ║
+║  ║  │  │    - Polling-based doorbell (queue head @ 0x110c00)                                           │  │  ║  ║
+║  ║  │  │    - Shared memory region for RPC messages                                                    │  │  ║  ║
+║  ║  │  │  • Firmware-Decompiled RPC Client (gsp_rpc_client.c + gsp_rpc_integration.c):               │  │  ║  ║
+║  ║  │  │    - All 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN,                                  │  │  ║  ║
+║  ║  │  │      SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG                                       │  │  ║  ║
+║  ║  │  │    - Real firmware addresses from decompilation                                               │  │  ║  ║
+║  ║  │  │  • Memory Management (memory.c):                                                              │  │  ║  ║
+║  ║  │  │    - DMA mapping, pin/unpin, map/unmap                                                        │  │  ║  ║
+║  ║  │  │  • IRQ Forwarding (irq.c):                                                                    │  │  ║  ║
+║  ║  │  │    - MSI-X table/PBA emulation                                                                │  │  ║  ║
+║  ║  │  │    - eventfd trigger injection to guest                                                       │  │  ║  ║
+║  ║  │  │  • Live Migration (migration.c):                                                              │  │  ║  ║
+║  ║  │  │    - 4-phase: QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP                                          │  │  ║  ║
+║  ║  │  │    - Dual jump tables (0x04179ED8, 0x04179FBC)                                               │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                      │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                         vfio-mdev.ko / vfio-pci.ko (Kernel VFIO Core)                         │  │  ║  ║
+║  ║  │  │  • Mediated device framework                                                                    │  │  ║  ║
+║  ║  │  │  • VFIO PCI device emulation                                                                    │  │  ║  ║
+║  ║  │  │  • IOMMU DMA mapping (DMA remapping)                                                            │  │  ║  ║
+║  ║  │  │  • Region/IRQ index mapping:                                                                    │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_BAR0_REGION_INDEX → BAR0 (16MB)                                                 │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_BAR1_REGION_INDEX → BAR1 (256MB)                                                │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_BAR2_REGION_INDEX → BAR2 (32MB)                                                 │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_BAR3_REGION_INDEX → BAR3 (32MB)                                                 │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_CONFIG_REGION_INDEX → PCI Config Space (256B)                                   │  │  ║  ║
+║  ║  │  │    - VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table/PBA                                             │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                      │  ║  ║
+║  ║  └──────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                            ║  ║
+║  ║  ════════════════════════════════════════════════════════════════════════════════════════════════════════════  ║  ║
+║  ║                                    HARDWARE (Physical GPU)                                                  ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │                         NVIDIA TU102/TU104 GPU (RTX 2080 Ti / T4 / etc.)                              │  ║  ║
+║  ║  │                                                                                                       │  ║  ║
+║  ║  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐   │  ║  ║
+║  ║  │  │    BAR0     │  │    BAR1     │  │    BAR2     │  │    BAR3     │  │      GSP FALCON       │   │  ║  ║
+║  ║  │  │  (16 MB)    │  │  (256 MB)   │  │  (32 MB)    │  │  (32 MB)    │  │                       │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  ┌─────────────────┐  │   │  ║  ║
+║  ║  │  │ Config Regs │  │ Framebuffer │  │  Reserved   │  │  Reserved   │  │  │  REAL GSP       │  │   │  ║  ║
+║  ║  │  │ Doorbell    │  │   (VRAM)    │  │             │  │             │  │  │  FIRMWARE       │  │   │  ║  ║
+║  ║  │  │ Queue Heads │  │             │  │             │  │             │  │  │  (570.144W)     │  │   │  ║  ║
+║  ║  │  │ Shared Mem  │  │             │  │             │  │             │  │  │                 │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ RPC Dispatchers │  │   │  ║  ║
+║  ║  │  │ 0x110c00    │  │             │  │             │  │             │  │  │  • Primary     │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x057A1AC0  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • Secondary   │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04ED39DC  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │                 │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ Specialized    │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ Handlers:      │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • BAR_TRAP    │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04CB8188  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • WPR x2      │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04F00B0C  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04F00C78  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • ALLOC       │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x057E94EC  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • SCHEDULE    │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x051EFBF4  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │  • Migration   │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04179ED8  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │    0x04179FBC  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │                 │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ 12 RPC Types:  │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ ALLOC, FREE,   │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ MAP, UNMAP,    │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ PIN, UNPIN,    │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ SCHEDULE,      │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ YIELD, NOTIFY, │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ BAR_TRAP,      │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  │ MIGRATE, DEBUG │  │   │  ║  ║
+║  ║  │  │             │  │             │  │             │  │             │  │  └─────────────────┘  │   │  ║  ║
+║  ║  │  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────────────────┘   │  ║  ║
+║  ║  │                                                                                                       │  ║  ║
+║  ║  │  Physical IRQ (MSI-X) ──────────────────────────────────────────────► nvidia.ko ISR                 │  ║  ║
+║  ║  └──────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                            ║  ║
+║  ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+║                                                                                                                  ║
+╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
+
+                                    │ VFIO MEDIATED DEVICE (mdev) │
+                                    │  sysfs: /sys/bus/mdev/devices/<UUID>/ │
+                                    │  • PCI Config Space (256B)            │
+                                    │  • BAR0 Region (16MB)                 │
+                                    │  • BAR1 Region (256MB)                │
+                                    │  • BAR2 Region (32MB)                 │
+                                    │  • BAR3 Region (32MB)                 │
+                                    │  • MSI-X Table/PBA Region             │
+                                    │  • IRQs (MSI-X vectors)               │
+                                    ▼
+
+╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
+║                                                         GUEST VM                                                   ║
+║  ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║                                                                                                                  ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │                                    USERSPACE (Guest)                                                      │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                              Guest Applications                                                       │  │  ║  ║
+║  ║  │  │                                                                                                       │  │  ║  ║
+║  ║  │  │  • CUDA Applications                                                                                 │  │  ║  ║
+║  ║  │  │  • OpenGL/Vulkan Applications                                                                        │  │  ║  ║
+║  ║  │  │  • Video Encode/Decode (NVENC/NVDEC)                                                                 │  │  ║  ║
+║  ║  │  │  • AI/ML Workloads (cuDNN, TensorRT)                                                                 │  │  ║  ║
+║  ║  │  │  • Custom vGPU Management (via libvgpu.so)                                                           │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    libvgpu.so / libiov.so (Guest-side Userspace Library)                           │  │  ║  ║
+║  ║  │  │                                                                                                     │  │  ║  ║
+║  ║  │  │  • vgpu_open/close/get_info                                                                        │  │  ║  ║
+║  ║  │  │  • vgpu_vf_create/destroy/get_info                                                                 │  │  ║  ║
+║  ║  │  │  • vgpu_mem_alloc/free/map/unmap/pin/unpin                                                         │  │  ║  ║
+║  ║  │  │  • vgpu_bar_config/force_access/save/restore                                                       │  │  ║  ║
+║  ║  │  │  • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback                             │  │  ║  ║
+║  ║  │  │  • vgpu_schedule/yield                                                                             │  │  ║  ║
+║  ║  │  │  • vgpu_register/unregister_event_callback, vgpu_process_events                                    │  │  ║  ║
+║  ║  │  │  • vgpu_migrate_start/status/cancel                                                                │  │  ║  ║
+║  ║  │  │                                                                                                     │  │  ║  ║
+║  ║  │  │  ┌────────────────────────────────────────────────────────────────────────────────────────────┐   │  │  ║  ║
+║  ║  │  │  │              rpc_emulator.c (Guest-side Emulation Fallbacks)                                │   │  │  ║  ║
+║  ║  │  │  │                                                                                             │   │  │  ║  ║
+║  ║  │  │  │  • rpc_emu_bar_config/save/restore/get_state                                                │   │  │  ║  ║
+║  ║  │  │  │  • rpc_emu_wpr_protect/unprotect/set_timeout/check_access                                   │   │  │  ║  ║
+║  ║  │  │  │  • rpc_emu_schedule/yield                                                                   │   │  │  ║  ║
+║  ║  │  │  │  • rpc_emu_notify_register/unregister/send                                                  │   │  │  ║  ║
+║  ║  │  │  └────────────────────────────────────────────────────────────────────────────────────────────┘   │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    libnvidia-ml.so / libnvidia-opencl.so / libcuda.so (NVIDIA Userspace Drivers)   │  │  ║  ║
+║  ║  │  │  • Standard NVIDIA userspace driver stack (unchanged)                                              │  │  ║  ║
+║  ║  │  │  • Communicates with guest nvidia.ko via ioctl on /dev/nvidia*                                     │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                  ║  ║
+║  ║  ═════════════════════════════════════════════════════════════════════════════════════════════════════════════  ║  ║
+║  ║                                    KERNEL SPACE (Guest)                                                          ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    nvidia.ko (Guest RM Kernel Module)                                              │  │  ║  ║
+║  ║  │  │  (NVIDIA OpenRM 570.144 - Guest Driver, UNMODIFIED)                                              │  │  ║  ║
+║  ║  │  │                                                                                                     │  │  ║  ║
+║  ║  │  │  • Detects vGPU via PCI Config Space (Vendor=0x10DE, Device=0x1ED3)                              │  │  ║  ║
+║  ║  │  │  • Maps BAR0 (config), BAR1 (framebuffer), BAR2, BAR3                                             │  │  ║  ║
+║  ║  │  │  • Sets up MSI-X interrupts (table @ BAR0+0x2000, PBA @ BAR0+0x3000)                             │  │  ║  ║
+║  ║  │  │  • GSP Client:                                                                                     │  │  ║  ║
+║  ║  │  │    - Writes to BAR0 doorbell (queue head @ 0x110c00)                                             │  │  ║  ║
+║  ║  │  │    - Reads/writes shared memory for RPC messages                                                 │  │  ║  ║
+║  ║  │  │    - Expects GSP firmware with vGPU RPC handlers                                                 │  │  ║  ║
+║  ║  │  │  • RM (Resource Manager) for GPU resources                                                       │  │  ║  ║
+║  ║  │  │  • nvidia-modeset.ko / nvidia-uvm.ko / nvidia-peermem.ko                                         │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    gsp-vgpu.ko (Guest vGPU Driver - Open-IOV)                                     │  │  ║  ║
+║  ║  │  │                                                                                                     │  │  ║  ║
+║  ║  │  │  • Char Device: /dev/gsp-vgpu (misc device)                                                        │  │  ║  ║
+║  ║  │  │  • IOCTL Interface (matches libvgpu.so API):                                                       │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_CREATE_VF / DESTROY_VF                                                             │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_ALLOC / FREE / MAP / UNMAP / PIN / UNPIN                                           │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_BAR_CONFIG / FORCE / SAVE / RESTORE                                                │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_WPR_PROTECT / UNPROTECT / TIMEOUT                                                  │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_SCHEDULE / YIELD                                                                    │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_MIGRATE                                                                             │  │  ║  ║
+║  ║  │  │    - GSP_IOCTL_GET_INFO                                                                            │  │  ║  ║
+║  ║  │  │  • RPC Client (rpc/gsp_rpc.c):                                                                     │  │  ║  ║
+║  ║  │  │    - Doorbell writes to queue head                                                                 │  │  ║  ║
+║  ║  │  │    - Ring buffer management                                                                        │  │  ║  ║
+║  ║  │  │  • Emulation Mode (AUTO-DETECTED):                                                                 │  │  ║  ║
+║  ║  │  │    - Test BAR_TRAP RPC at load; if timeout → enable emulation                                     │  │  ║  ║
+║  ║  │  │    - Module params: emulation_mode (ro), emulation_override (force)                               │  │  ║  ║
+║  ║  │  │    - In emulation: BAR_TRAP, WPR, ALLOC, PIN, MAP handled locally                                 │  │  ║  ║
+║  ║  │  │  • Debugfs: /sys/kernel/debug/gsp-vgpu/                                                            │  │  ║  ║
+║  ║  │  │  • VF Lifecycle (vf/gsp_vf.c)                                                                      │  │  ║  ║
+║  ║  │  │  • BAR Emulation (bar/gsp_bar.c)                                                                   │  │  ║  ║
+║  ║  │  │  • WPR Management (wpr/gsp_wpr.c)                                                                  │  │  ║  ║
+║  ║  │  │  • Migration (migration/gsp_migration.c)                                                           │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐  │  ║  ║
+║  ║  │  │                    VFIO-PCI Core (Guest Kernel)                                                    │  │  ║  ║
+║  ║  │  │  • vfio-pci.ko: PCI device emulation for VFIO                                                      │  │  ║  ║
+║  ║  │  │  • Handles mmap of BAR regions                                                                      │  │  ║  ║
+║  ║  │  │  • MSI-X interrupt handling                                                                         │  │  ║  ║
+║  ║  │  │  • Config space read/write                                                                          │  │  ║  ║
+║  ║  │  └────────────────────────────────────────────────────────────────────────────────────────────────────┘  │  ║  ║
+║  ║  │                                                                                                          │  ║  ║
+║  ║  └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                  ║  ║
+║  ║  ══════════════════════════════════════════════════════════════════════════════════════════════════════════════  ║  ║
+║  ║                                    VIRTUAL GPU (mdev Device as seen by Guest)                                    ║  ║
+║  ║  ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║  │  PCI Config Space (256B)  │  BAR0 (16MB)  │  BAR1 (256MB)  │  BAR2 (32MB)  │  BAR3 (32MB)  │  MSI-X  │  ║  ║
+║  ║  │  • Vendor=0x10DE          │  Config Regs  │  Framebuffer │  Reserved     │  Reserved     │  Table  │  ║  ║
+║  ║  │  • Device=0x1ED3          │  Doorbell     │  (VRAM)      │               │               │  @0x2000│  ║  ║
+║  ║  │  • Class=0x030000         │  Queue Heads  │              │               │               │  PBA    │  ║  ║
+║  ║  │  • BARs: 64-bit, prefetch │  Shared Mem   │              │               │               │  @0x3000│  ║  ║
+║  ║  │  • Caps: PM, MSI, MSI-X,  │  @0x110c00    │              │               │               │         │  ║  ║
+║  ║  │    PCIe                                                                     │  ║  ║
+║  ║  └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                  ║  ║
+║  ╚═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+╚══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
+```
+
+## Communication Pathways (Host ↔ Guest)
+
+### 1. **VFIO Region Access (Memory-Mapped I/O)**
+```
+Guest App → libvgpu.so → ioctl(/dev/gsp-vgpu) → gsp-vgpu.ko
+    → Guest vfio-pci.ko → mmap(BAR0/BAR1/BAR2/BAR3)
+    → Host vfio-pci.ko → open-iov-vfio.ko BAR emulation
+    → Host nvidia.ko → Physical GPU BARs
+```
+**Path**: Guest userspace → Guest kernel → Host kernel → Physical GPU
+**Mechanism**: VFIO mmap with IOMMU DMA remapping
+
+### 2. **PCI Config Space Access**
+```
+Guest nvidia.ko → pci_read_config_* / pci_write_config_*
+    → Guest vfio-pci.ko → VFIO_DEVICE_GET_REGION_INFO (CONFIG_REGION)
+    → Host open-iov-vfio.ko → nv_vgpu_config_read/write()
+    → Returns emulated 256-byte config space
+```
+**Path**: Guest kernel → Host kernel (via VFIO ioctl)
+**Mechanism**: VFIO config space region emulation
+
+### 3. **GSP RPC Communication (Native Path)**
+```
+Guest nvidia.ko → Write BAR0 doorbell (0x110c00) + shared memory
+    → Physical GPU GSP firmware receives RPC
+    → GSP processes RPC (ALLOC, MAP, PIN, SCHEDULE, BAR_TRAP, etc.)
+    → GSP writes response to shared memory
+    → Guest nvidia.ko reads response
+```
+**Path**: Guest kernel → Physical GPU (direct, no host kernel involvement)
+**Mechanism**: Doorbell register + shared memory in BAR0
+
+### 4. **GSP RPC Communication (Emulation Path - Host Mediated)**
+```
+Guest nvidia.ko → Write BAR0 doorbell (0x110c00)
+    → Host open-iov-vfio.ko (polling) detects doorbell
+    → gsp_comm.c → gsp_rpc_integration.c → gsp_rpc_client.c
+    → Firmware-decompiled RPC handlers (emulated in host kernel)
+    → Response written to shared memory
+    → Guest nvidia.ko reads response
+```
+**Path**: Guest kernel → Host kernel (open-iov-vfio.ko) → Emulated RPC
+**Mechanism**: Host polls GSP doorbell, processes RPC in software
+
+### 5. **BAR Trap Emulation**
+```
+Guest nvidia.ko → Write to trapped BAR register (page fault)
+    → Guest vfio-pci.ko → VFIO page fault handler
+    → Host open-iov-vfio.ko → nv_vgpu_bar_trap()
+    → If native: gsp_rpc_bar_trap() → GSP firmware (0x04CB8188)
+    → If emulated: rpc_emu_bar_trap() in host userspace (libvgpu.so)
+    → Result written back to guest BAR mapping
+```
+**Path**: Guest kernel page fault → Host kernel → Host userspace (emulation) or GSP firmware
+**Mechanism**: VFIO page fault + BAR trap emulation
+
+### 6. **Interrupt Forwarding (Host → Guest)**
+```
+Physical GPU → MSI-X IRQ → Host nvidia.ko ISR
+    → nvidia.ko calls nv_vgpu_vfio_inject_interrupt() (VFIO ops)
+    → open-iov-vfio.ko → eventfd_signal()
+    → QEMU/KVM → Guest MSI-X interrupt injected
+    → Guest nvidia.ko ISR handles interrupt
+```
+**Path**: Physical GPU → Host kernel → Host userspace (QEMU) → Guest kernel
+**Mechanism**: VFIO interrupt injection via eventfd
+
+### 7. **Userspace Management (mdev management CLI/mdev management daemon)**
+```
+mdev management CLI → libvgpu.so (host) → ioctl(/dev/vfio/<mdev>)
+    → open-iov-vfio.ko → mdev VFIO ops
+    → VF creation, BAR config, memory ops, migration
+```
+**Path**: Host userspace → Host kernel (VFIO mdev)
+**Mechanism**: Standard VFIO ioctl on mdev device
+
+### 8. **Guest vGPU Driver Communication**
+```
+Guest App → libvgpu.so (guest) → ioctl(/dev/gsp-vgpu)
+    → gsp-vgpu.ko → Internal RPC client
+    → If emulation: Local handling in gsp-vgpu.ko
+    → If native: GSP RPC via BAR0 doorbell
+```
+**Path**: Guest userspace → Guest kernel (gsp-vgpu.ko) → GSP or emulation
+
+## Key Component Summary
+
+| Component | Location | Type | Purpose |
+|-----------|----------|------|---------|
+| **nvidia.ko (Host)** | Host Kernel | RM Driver | Boots GSP, manages physical GPU, exports vGPU VFIO symbols |
+| **open-iov-vfio.ko** | Host Kernel | VFIO mdev Driver | Creates mdev, PCI config space, BAR emulation, RPC integration, IRQ forwarding |
+| **gsp-vgpu.ko** | Guest Kernel | vGPU Driver | Char device `/dev/gsp-vgpu`, ioctl API, emulation mode, RPC client |
+| **nvidia.ko (Guest)** | Guest Kernel | RM Driver | Standard NVIDIA driver, detects vGPU, communicates with GSP |
+| **libvgpu.so / libiov.so (Host)** | Host Userspace | Library | Management API (mdev management CLI/mdev management daemon), emulation fallbacks |
+| **libvgpu.so / libiov.so (Guest)** | Guest Userspace | Library | Application API, emulation fallbacks |
+| **libnvidia-*.so** | Guest Userspace | NVIDIA Drivers | Standard CUDA/OpenGL/OpenCL userspace stack |
+| **mdev management CLI / mdev management daemon** | Host Userspace | Tools | mdev lifecycle, VF management, migration orchestration |
+| **REAL GSP Firmware** | Physical GPU | Firmware | 570.144W, 12 RPC types, BAR_TRAP/WPR/SCHEDULE/MIGRATE handlers |
+| **EMULATED GSP** | Host Kernel + Userspace | Software | rpc_emulator.c, gsp_rpc_client.c (when GSP lacks vGPU handlers) |
+
+## Emulation Mode Details
+
+### When Enabled
+- **Auto-detect**: At gsp-vgpu.ko load, sends test BAR_TRAP RPC
+- **Timeout/Error** → `emulation_mode = true`
+- **Module params**: `emulation_override=1` (force on), `emulation_override=0` (force off)
+
+### What Gets Emulated
+| Operation | Host Kernel (open-iov-vfio.ko) | Host Userspace (libvgpu.so) | Guest Kernel (gsp-vgpu.ko) |
+|-----------|-------------------------------|----------------------------|---------------------------|
+| BAR_TRAP | nv_vgpu_bar_trap() → rpc_emu_bar_trap() | rpc_emu_bar_config/save/restore | gsp_bar_force_access() |
+| WPR | nv_vgpu_wpr_*() | rpc_emu_wpr_protect/unprotect | gsp_vf_wpr_protect/unprotect |
+| ALLOC/MAP/PIN | gsp_rpc_client.c (local) | rpc_emu_* fallbacks | gsp_vf_mem_alloc/map/pin |
+| SCHEDULE/YIELD | gsp_rpc_schedule/yield (workqueue) | rpc_emu_schedule/yield | gsp_vf_schedule/yield |
+| NOTIFY | gsp_rpc_notify_register/unregister | rpc_emu_notify_register/unregister/send | gsp_vf_notify_* |
+| MIGRATION | migration.c (4-phase) | rpc_emu_migrate_* | gsp_migration_start |
+
+## Build Artifacts
+
+```
+$OPEN_IOV_ROOT/
+├── nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko      (34MB)  Host RM Driver
+├── kernel/open-iov-vfio.ko                                    (2.8MB)  Host VFIO mdev Driver
+├── driver/gsp-vgpu.ko                                         (1.8MB)  Guest vGPU Driver
+├── userspace/lib/libiov.so                                    (76KB)   Userspace Library (libvgpu.so symlink)
+├── userspace/examples/example_vgpu                            (28KB)   Test Application
+├── firmware/570-GSP/gsp-570.144.fwimage.bin                   (28MB)   GSP Firmware
+├── scripts/load_vgpu_stack.sh                                 Host module loader
+├── scripts/create_mdev.sh                                     mdev creation
+├── scripts/launch_qemu_vgpu.sh                                QEMU launcher
+├── scripts/test_vgpu_stack.sh                                 Test suite
+├── scripts/setup_guest_driver.sh                              Guest driver setup
+├── scripts/unload_vgpu_stack.sh                               Cleanup
+├── docs/ARCHITECTURE.md                                       This document
+└── wiki/                                                      10 technical docs
+```
+
+## Deployment Sequence
+
+```bash
+# HOST (after reboot to clear module_blacklist)
+# 1. Load NVIDIA OpenRM host driver (exports vGPU VFIO symbols)
+sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko
+
+# 2. Load Open-IOV VFIO mdev driver (registers mdev parent with nvidia.ko)
+sudo insmod kernel/open-iov-vfio.ko
+
+# 3. Create mediated device for QEMU
+./scripts/create_mdev.sh          # Creates /sys/bus/mdev/devices/<UUID>/
+
+# 4. Launch QEMU with vGPU
+./scripts/launch_qemu_vgpu.sh     # -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>
+
+# GUEST (inside VM)
+# 5. Install NVIDIA driver (unmodified OpenRM)
+./scripts/setup_guest_driver.sh   # Installs nvidia.ko, firmware, loads driver
+
+# 6. Verify
+nvidia-smi                        # Should show vGPU (T4 / A100 / etc.)
+lspci -d 10de:                    # Should show 10de:1ed3 (TU104 vGPU)
+```
+
+---
+
+*Architecture Version: 2.0 (with detailed host/guest communication pathways)*
+*Last Updated: $(date)*
+*Project: $OPEN_IOV_ROOT*
\ No newline at end of file