VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers

- Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create()
- Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions
- Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP
- Added config space read/write handlers at offset 0x70000000000
- Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers)
- Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL)
- Added udev rules for vfio group permissions
- Created stack_manager.sh unified tool
- Fixed mdev remove crash with deferred cleanup in vfio_device_release callback
- Kernel 6.8+ IOMMUFD compatibility
- QEMU launches successfully with vGPU passthrough

1 files changed, 595 insertions, 0 deletions

diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md
new file mode 100644
index 0000000..fe17248
--- /dev/null
+++ b/docs/refactored-architecture.md
@@ -0,0 +1,595 @@
+# Open-IOV vGPU-IO Refactored Architecture
+
+## System Design from First Principles
+
+### 1. Problem Statement
+
+**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with full GPU functionality (CUDA, graphics, video encode/decode, AI/ML) on Turing GPUs (TU102/TU104) without requiring:
+- Proprietary GRID host drivers
+- Modified guest drivers
+- Paravirtualization interfaces
+- Guest-side Open-IOV code
+
+**Constraints**:
+- Guest runs stock `nvidia.ko` + `libcuda.so` + `libnvidia-ml.so` + `libnvidia-opencl.so`
+- Host runs OpenRM `nvidia.ko` (for physical GPU management) + Open-IOV hypervisor components
+- No guest kernel modifications, no guest userspace Open-IOV libraries
+- High performance: near-native GPU utilization for compute and graphics workloads
+- Support both Linux and Windows guests
+
+### 2. Core Design Principle: Host-Side GSP Emulation
+
+The fundamental insight: **The GSP (GPU System Processor) is a RISC-V microcontroller running firmware that implements 12 RPC types. We can emulate this firmware entirely in the host kernel.**
+
+```
+┌─────────────────────────────────────────────────────────────────────────────────┐
+│                         FIRST PRINCIPLE: GSP IS JUST AN RPC SERVER              │
+├─────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                 │
+│  Physical Reality:                                                              │
+│  ┌─────────────┐     Doorbell + Shared Mem      ┌─────────────────────────┐   │
+│  │  Guest RM   │ ─────────────────────────────► │   GSP Firmware          │   │
+│  │  (nvidia.ko)│ ◄───────────────────────────── │   (570.144W, RV64IMAC)  │   │
+│  └─────────────┘     Responses                   └─────────────────────────┘   │
+│       │                                                                    │   │
+│       │ 12 RPC Types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN,                │   │
+│       │             SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG      │   │
+│       │                                                                    │   │
+│       ▼                                                                    │   │
+│  Virtualized Reality:                                                       │   │
+│  ┌─────────────┐     Doorbell + Shared Mem      ┌─────────────────────────┐   │
+│  │  Guest RM   │ ─────────────────────────────► │   Host Kernel           │   │
+│  │  (nvidia.ko)│ ◄───────────────────────────── │   (open-iov-vfio.ko)    │   │
+│  └─────────────┘     Responses                   │   gsp_rpc_client.c      │   │
+│       │                                          │   (12 emulated handlers)│   │
+│       │  Guest sees identical                    └─────────────────────────┘   │
+│       │  interface - cannot tell                                               │
+│       ▼                                                                        │
+│  No guest-side code required.                                                 │
+│                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────┘
+```
+
+### 3. System Architecture Overview
+
+```
+╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
+║                                                         HOST SYSTEM                                                      ║
+║  ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║ USERSPACE                                                                                                           ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║ │ QEMU/KVM + libvfio-user                                                                                         │  ║  ║
+║  ║ │  • -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>                                                      │  ║  ║
+║  ║ │  • VFIO region mmap, ioctl, eventfd for IRQs                                                                   │  ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                     ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║ │ mdev management CLI / mdev management daemon (Management Tools)                                                                            │  ║  ║
+║  ║ │  • Link against: libvgpu.so (host-side)                                                                        │  ║  ║
+║  ║ │  • VF lifecycle: create/destroy, resource allocation, migration orchestration                                  │  ║  ║
+║  ║ │  • Calls: vgpu_open(), vgpu_vf_create(), vgpu_mem_alloc(), vgpu_bar_config(),                                 │  ║  ║
+║  ║ │         vgpu_wpr_protect(), vgpu_schedule(), vgpu_migrate_start(), etc.                                        │  ║  ║
+║  ║ │  • Communicates via VFIO ioctls on /dev/vfio/<mdev>                                                            │  ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                     ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║ │ libvgpu.so / libiov.so (HOST-SIDE ONLY)                                                                        │  ║  ║
+║  ║ │  • Full vGPU management API: vgpu_*() functions                                                               │  ║  ║
+║  ║ │  • Implements: VF create/destroy, memory ops, BAR config, WPR, scheduling,                                   │  ║  ║
+║  ║ │    notifications, migration                                                                                    │  ║  ║
+║  ║ │  • rpc_emulator.c: Host-side emulation fallbacks for management operations                                   │  ║  ║
+║  ║ │    (rpc_emu_bar_config/save/restore, rpc_emu_wpr_*, rpc_emu_schedule/yield,                                 │  ║  ║
+║  ║ │     rpc_emu_notify_register/unregister/send, rpc_emu_migrate_*)                                              │  ║  ║
+║  ║ │  • NO GSP RPC emulation here - that lives in kernel                                                           │  ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+║                                                                                                                         ║
+║  ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║ KERNEL SPACE                                                                                                         ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║ │ nvidia.ko (Host RM - OpenRM 570.144)                                                                            │  ║  ║
+║  ║ │  • Boots physical GSP firmware (570.144W)                                                                       │  ║  ║
+║  ║ │  • Manages physical GPU resources (VRAM, engines, display)                                                     │  ║  ║
+║  ║ │  • Exports vGPU VFIO symbols:                                                                                   │  ║  ║
+║  ║ │    - nvidia_vgpu_vfio_get_ops() / nvidia_vgpu_vfio_set_ops()                                                   │  ║  ║
+║  ║ │    - nv_vgpu_mdev_register_parent() / nv_vgpu_mdev_unregister_parent()                                         │  ║  ║
+║  ║ │  • GPC callbacks from open-iov-vfio.ko: vgpu_create, vgpu_delete, vgpu_bar_info,                              │  ║  ║
+║  ║ │    vgpu_start, get_name, get_description, get_instances, get_sparse_mmap, update_request                     │  ║  ║
+║  ║ │  • Owns physical MSI-X interrupt                                                                                │  ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ║                                                                                                                     ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐  ║  ║
+║  ║ │ open-iov-vfio.ko (HYPERVISOR - Core Component)                                                                 │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ mdev Parent Registration                                                                               │   │  ║  ║
+║  ║ │  │  • Registers with nvidia.ko via nv_vgpu_mdev_register_parent()                                        │   │  ║  ║
+║  ║ │  │  • Creates mediated device types: nvidia-vgpu-turing, nvidia-vgpu-ampere, etc.                        │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ PCI Config Space Emulation (256 bytes per mdev)                                                        │   │  ║  ║
+║  ║ │  │  • Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU), Class=0x030000                                          │   │  ║  ║
+║  ║ │  │  • BAR0=16MB (64-bit, prefetchable), BAR1=256MB (64-bit, prefetchable)                                │   │  ║  ║
+║  ║ │  │  • BAR2=32MB, BAR3=32MB (64-bit)                                                                       │   │  ║  ║
+║  ║ │  │  • Capabilities: PM (0x40), MSI (0x50), MSI-X (0x70), PCIe (0x80)                                     │   │  ║  ║
+║  ║ │  │  • MSI-X Table @ BAR0+0x2000, PBA @ BAR0+0x3000                                                        │   │  ║  ║
+║  ║ │  │  • VFIO_CONFIG_REGION_INDEX mapped to this emulated config space                                      │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ BAR Region Emulation (VFIO Region Indices)                                                             │   │  ║  ║
+║  ║ │  │  • VFIO_PCI_BAR0_REGION_INDEX → 16MB emulated config space                                            │   │  ║  ║
+║  ║ │  │  • VFIO_PCI_BAR1_REGION_INDEX → 256MB emulated framebuffer (backed by host VRAM allocation)          │   │  ║  ║
+║  ║ │  │  • VFIO_PCI_BAR2_REGION_INDEX → 32MB reserved                                                          │   │  ║  ║
+║  ║ │  │  • VFIO_PCI_BAR3_REGION_INDEX → 32MB reserved                                                          │   │  ║  ║
+║  ║ │  │  • VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table (0x2000) + PBA (0x3000) in BAR0                           │   │  ║  ║
+║  ║ │  │  • BAR trap handling: 64 trapped regions covering all BAR0 registers (bar_emulation.c)               │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ GSP EMULATION ENGINE (NEW - Core Innovation)                                                           │   │  ║  ║
+║  ║ │  │                                                                                                         │   │  ║  ║
+║  ║ │  │  Per-mdev GSP Emulation Thread:                                                                        │   │  ║  ║
+║  ║ │  │  ```c                                                                                                   │   │  ║  ║
+║  ║ │  │  static int gsp_emu_thread(void *data) {                                                               │   │  ║  ║
+║  ║ │  │      struct nv_vgpu_device *vgpu = data;                                                               │   │  ║  ║
+║  ║ │  │      while (!kthread_should_stop()) {                                                                  │   │  ║  ║
+║  ║ │  │          u32 doorbell = readl(vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET);                                │   │  ║  ║
+║  ║ │  │          if (doorbell) {                                                                               │   │  ║  ║
+║  ║ │  │              writel(0, vgpu->bar0_vaddr + GSP_DOORBELL_OFFSET);  // ACK                               │   │  ║  ║
+║  ║ │  │              gsp_emu_process_rpc(vgpu);  // Process via gsp_rpc_client.c                             │   │  ║  ║
+║  ║ │  │          }                                                                                             │   │  ║  ║
+║  ║ │  │          usleep_range(100, 200);  // 1-2μs polling interval                                           │   │  ║  ║
+║  ║ │  │      }                                                                                                 │   │  ║  ║
+║  ║ │  │      return 0;                                                                                         │   │  ║  ║
+║  ║ │  │  }                                                                                                     │   │  ║  ║
+║  ║ │  │  ```                                                                                                   │   │  ║  ║
+║  ║ │  │                                                                                                         │   │  ║  ║
+║  ║ │  │  GSP Shared Memory Management:                                                                         │   │  ║  ║
+║  ║ │  │  • Allocates contiguous host memory for GSP mailbox (4KB aligned)                                     │   │  ║  ║
+�b  ║ │  │  • Maps into guest BAR0 at GSP_SHARED_MEM_OFFSET (0x1000)                                             │   │  ║  ║
+║  ║ │  │  • WC (Write-Combining) mapping for performance                                                        │   │  ║  ║
+║  ║ │  │  • Cache coherency: clflush on write, memory barriers                                                 │   │  ║  ║
+║  ║ │  │  • Matches firmware mailbox layout exactly (from decompilation)                                       │   │  ║  ║
+║  ║ │  │                                                                                                         │   │  ║  ║
+║  ║ │  │  RPC Handler Suite (gsp_rpc_client.c - ALL 12 TYPES):                                                 │   │  ║  ║
+║  ║ │  │  ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────┐   │   │  ║
+║  ║ │  │  │ RPC Type    │ Emulated Implementation                                                           │   │   │  ║
+║  ║ │  │  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────┤   │   │  ║
+║  ║ │  │  │ ALLOC (0x01)│ VRAM allocation via host nvidia.ko, returns GPU VA + handle                      │   │   │  ║
+║  ║ │  │  │ FREE (0x02) │ Free allocation, unmap pages                                                      │   │   │  ║
+║  ║ │  │  │ MAP (0x03)  │ Map allocation to GPU VA, update page tables                                      │   │   │  ║
+║  ║ │  │  │ UNMAP (0x04)│ Unmap GPU VA range                                                                │   │   │  ║
+║  ║ │  │  │ PIN (0x05)  │ Pin pages for DMA, return DMA address                                             │   │   │  ║
+║  ║ │  │  │ UNPIN (0x06)│ Unpin pages                                                                       │   │   │  ║
+║  ║ │  │  │ SCHEDULE(0x07)│ Submit work to GSP scheduler (local workqueue)                                 │   │   │  ║
+║  ║ │  │  │ YIELD (0x08)│ Timeslice yield (no-op success)                                                   │   │   │  ║
+║  ║ │  │  │ NOTIFY(0x09)│ Async notification registration, eventfd signaling                                │   │   │  ║
+║  ║ │  │  │ BAR_TRAP(0x0A)│ BAR register read/write emulation via shadow memory                            │   │   │  ║
+║  ║ │  │  │ MIGRATE(0x0B)│ 4-phase live migration (QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP)                 │   │   │  ║
+║  ║ │  │  │ DEBUG (0x0C)│ Debug interface, firmware introspection                                           │   │   │  ║
+║  ║ │  │  └─────────────┴──────────────────────────────────────────────────────────────────────────────────┘   │   │  ║
+║  ║ │  │                                                                                                         │   │  ║
+║  ║ │  │  Firmware Address Mapping (from decompilation):                                                        │   │  ║
+║  ║ │  │  • Primary Dispatcher: 0x057A1AC0 (32 sub-commands)                                                    │   │  ║
+║  ║ │  │  • Secondary Dispatcher: 0x04ED39DC                                                                    │   │  ║
+║  ║ │  │  • BAR Trap Handler: 0x04CB8188                                                                        │   │  ║
+║  ║ │  │  • WPR Handlers: 0x04F00B0C, 0x04F00C78                                                               │   │  ║
+║  ║ │  │  • ALLOC Main: 0x057E94EC                                                                              │   │  ║
+║  ║ │  │  • SCHEDULE Main: 0x051EFBF4                                                                           │   │  ║
+║  ║ │  │  • Migration Tables: 0x04179ED8, 0x04179FBC (16 entries each)                                        │   │  ║
+║  ║ │  │  • RPC Jump Table: 0x059BFEB0 (12 handlers)                                                           │   │  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ MSI-X Interrupt Injection                                                                                │   │  ║  ║
+║  ║ │  │  • When emulated GSP completes RPC → write to MSI-X PBA → eventfd_signal()                            │   │  ║  ║
+║  ║ │  │  • QEMU/KVM delivers MSI-X interrupt to guest nvidia.ko ISR                                           │   │  ║  ║
+║  ║ │  │  • Guest sees normal GSP completion interrupt                                                           │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ Memory Management (memory.c)                                                                             │   │  ║  ║
+║  ║ │  │  • DMA mapping via IOMMU (VFIO DMA API)                                                                  │   │  ║  ║
+║  ║ │  │  • Pin/unpin for device DMA                                                                              │   │  ║  ║
+║  ║ │  │  • VRAM allocation via host nvidia.ko                                                                    │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ │                                                                                                                 │  ║  ║
+║  ║ │  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐   │  ║  ║
+║  ║ │  │ Live Migration (migration.c)                                                                             │   │  ║  ║
+║  ║ │  │  • 4-phase: QUERY → PREPARE → TRANSFER → COMMIT → CLEANUP                                               │   │  ║  ║
+║  ║ │  │  • State save/restore: GSP context, page tables, VRAM contents, BAR state                             │   │  ║  ║
+║  ║ │  │  • Dual jump tables from firmware (0x04179ED8, 0x04179FBC)                                             │   │  ║  ║
+║  ║ │  └────────────────────────────────────────────────────────────────────────────────────────────────────────┘   │  ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘  ║  ║
+║  ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+║                                                                                                                         ║
+║ PHYSICAL GPU: TU102/TU104 (RTX 2080 Ti / T4 / RTX 2080 Super)                                                          ║
+║  • REAL GSP Firmware 570.144W running on FALCON RISC-V                                                                  ║
+║  • Used by host nvidia.ko for physical GPU management                                                                  ║
+║  • NOT used for guest vGPU RPCs (those are emulated in host kernel)                                                    ║
+║                                                                                                                         ║
+╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
+                                    │ VFIO MEDIATED DEVICE (mdev) │
+                                    │  /sys/bus/mdev/devices/<UUID>/ │
+                                    ▼
+
+╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
+║                                                         GUEST VM                                                          ║
+║  ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║ USERSPACE                                                                                                              ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐   ║  ║
+║  ║ │ Guest Applications                                                                                                 │   ║  ║
+║  ║ │  • CUDA Applications (cuBLAS, cuDNN, custom kernels)                                                             │   ║  ║
+║  ║ │  • Graphics: OpenGL, Vulkan, DirectX (via WINE/Proton on Linux, native on Windows)                              │   ║  ║
+║  ║ │  • Video: NVENC/NVDEC (FFmpeg, GStreamer, Media Foundation)                                                      │   ║  ║
+║  ║ │  • AI/ML: TensorRT, PyTorch, TensorFlow                                                                          │   ║  ║
+║  ║ │  • NO Open-IOV libraries, NO modified code                                                                       │   ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘   ║  ║
+║  ║                                                                                                                       ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐   ║  ║
+║  ║ │ Standard NVIDIA Userspace Drivers (UNMODIFIED)                                                                   │   ║  ║
+║  ║ │  • libcuda.so / libcuda.so.1 (CUDA Driver API)                                                                   │   ║  ║
+║  ║ │  • libnvidia-ml.so / libnvidia-ml.so.1 (NVML)                                                                    │   ║  ║
+║  ║ │  • libnvidia-opencl.so / libnvidia-opencl.so.1 (OpenCL)                                                          │   ║  ║
+║  ║ │  • libnvidia-vulkan.so / libGLX_nvidia.so (Vulkan/OpenGL)                                                        │   ║  ║
+║  ║ │  • libnvidia-encode.so / libnvidia-decode.so (Video)                                                             │   ║  ║
+║  ║ │  • nvidia-smi, nvidia-settings, nvidia-debugdump                                                                 │   ║  ║
+║  ║ │  • ALL communicate with guest nvidia.ko via /dev/nvidia* ioctls                                                  │   ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘   ║  ║
+║  ╚═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+║                                                                                                                         ║
+║  ╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗  ║
+║  ║ KERNEL SPACE                                                                                                          ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐   ║  ║
+║  ║ │ nvidia.ko (Guest RM - UNMODIFIED OpenRM 570.144)                                                                 │   ║  ║
+║  ║ │  • PCI Probe: detects 10de:1ed3 (TU104 vGPU) from emulated config space                                        │   ║  ║
+║  ║ │  • BAR Mapping:                                                                                                  │   ║  ║
+║  ║ │    - BAR0 (16MB): Config registers, GSP doorbell @ 0x110c00, shared mem @ 0x1000                               │   ║  ║
+║  ║ │    - BAR1 (256MB): Framebuffer (VRAM)                                                                            │   ║  ║
+║  ║ │    - BAR2 (32MB), BAR3 (32MB): Reserved                                                                          │   ║  ║
+║  ║ │  • MSI-X Setup: Table @ BAR0+0x2000, PBA @ BAR0+0x3000                                                          │   ║  ║
+║  ║ │  • GSP Client Initialization:                                                                                    │   ║  ║
+║  ║ │    - Writes to doorbell (queue head) to submit RPCs                                                             │   ║  ║
+║  ║ │    - Reads/writes shared memory for RPC messages                                                                 │   ║  ║
+║  ║ │    - Waits for MSI-X interrupt on completion                                                                     │   ║  ║
+║  ║ │  • RM Subsystems (all functional):                                                                               │   ║  ║
+║  ║ │    - Memory: alloc, free, map, unmap, pin, unpin via GSP RPC                                                   │   ║  ║
+║  ║ │    - Scheduling: work submission, yield, priority via GSP RPC                                                  │   ║  ║
+║  ║ │    - BAR: register access via BAR_TRAP RPC                                                                       │   ║  ║
+║  ║ │    - WPR: protect/unprotect via GSP RPC                                                                          │   ║  ║
+║  ║ │    - Migration: save/restore via MIGRATE RPC                                                                     │   ║  ║
+║  ║ │    - Notifications: async events via NOTIFY RPC                                                                  │   ║  ║
+║  ║ │  • NO Open-IOV code in guest kernel                                                                              │   ║  ║
+║  ║ │  • NO gsp-vgpu.ko, NO libvgpu.so in guest                                                                        │   ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘   ║  ║
+║  ║                                                                                                                       ║  ║
+║  ║ ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐   ║  ║
+║  ║ │ vfio-pci.ko (Guest VFIO Core)                                                                                    │   ║  ║
+║  ║ │  • mmap of BAR regions → host VFIO regions                                                                       │   ║  ║
+║  ║ │  • MSI-X interrupt handling → eventfd from host                                                                  │   ║  ║
+║  ║ │  • Config space read/write → host emulated config space                                                          │   ║  ║
+║  ║ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘   ║  ║
+║  ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝  ║
+║                                                                                                                         ║
+║ VIRTUAL GPU (mdev as seen by Guest):                                                                                     ║
+║  PCI Config (256B) │ BAR0(16MB) │ BAR1(256MB) │ BAR2(32MB) │ BAR3(32MB) │ MSI-X Table/PBA                            ║
+║  Vendor=10DE       │ Config     │ Framebuffer │ Reserved   │ Reserved   │ Table@0x2000                               ║
+║  Device=1ED3       │ Doorbell   │ (VRAM)      │            │            │ PBA@0x3000                                 ║
+║  Class=030000      │ Queue Heads│             │            │            │                                            ║
+║  BARs: 64-bit      │ Shared Mem │             │            │            │                                            ║
+║  Caps: PM,MSI,     │ @0x110c00  │             │            │            │                                            ║
+║  MSI-X, PCIe       │ @0x1000    │             │            │            │                                            ║
+╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
+```
+
+### 4. Communication Pathways (Complete)
+
+| # | Pathway | Direction | Mechanism | Latency Target |
+|---|---------|-----------|-----------|----------------|
+| 1 | **GSP RPC (Guest→Host)** | Guest nvidia.ko → Host open-iov-vfio.ko | Doorbell write (MMIO) → host polling thread → gsp_rpc_client.c | < 5μs |
+| 2 | **GSP Response (Host→Guest)** | Host open-iov-vfio.ko → Guest nvidia.ko | Write shared memory → MSI-X interrupt injection (eventfd) | < 3μs |
+| 3 | **BAR Register Access** | Guest nvidia.ko → Host open-iov-vfio.ko | MMIO page fault → bar_emulation.c shadow memory | < 2μs |
+| 4 | **PCI Config Space** | Guest kernel → Host open-iov-vfio.ko | VFIO_CONFIG_REGION_INDEX ioctl → emulated 256B config | < 1μs |
+| 5 | **VFIO DMA (VRAM)** | Guest nvidia.ko ↔ Physical VRAM | IOMMU DMA mapping via VFIO DMA API | Near-native |
+| 6 | **MSI-X Interrupts** | Physical GPU → Guest nvidia.ko | Host nvidia.ko ISR → nv_vgpu_vfio_inject_interrupt → eventfd → QEMU → Guest MSI-X | < 10μs |
+| 7 | **Management Ops** | mdev management CLI/mgr → Host open-iov-vfio.ko | VFIO ioctls on mdev device → libvgpu.so → rpc_emulator.c | < 100μs |
+| 8 | **Migration** | Host ↔ Host | Migration protocol over network, state via open-iov-vfio.ko | Seconds |
+
+### 5. GSP Emulation Engine - Detailed Design
+
+#### 5.1 Mailbox Layout (from Firmware Decompilation)
+
+```
+GSP Shared Memory Mailbox (at BAR0 + 0x1000, 4KB):
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ Offset │ Size  │ Field                    │ Description                     │
+├────────┼───────┼──────────────────────────┼─────────────────────────────────┤
+│ 0x0000 │ 64B   │ RPC Request Header       │ type, subcmd, seq_id, payload  │
+│ 0x0040 │ 64B   │ RPC Request Payload[6]   │ u64 payload[6] per RPC type    │
+│ 0x0080 │ 64B   │ RPC Response Header      │ status, type, seq_id           │
+│ 0x00C0 │ 64B   │ RPC Response Payload[6]  │ u64 result[6]                  │
+│ 0x0100 │ 3KB   │ Reserved / Extended      │ Future use, migration state    │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+#### 5.2 Doorbell Register (at BAR0 + 0x110C00)
+
+```
+Queue Head Register (0x110C00):
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ Bits  │ Field          │ Description                                       │
+├───────┼────────────────┼───────────────────────────────────────────────────┤
+│ 31:1  │ Reserved       │ Must be zero                                      │
+│ 0     │ Doorbell Valid │ 1 = New RPC request in mailbox, 0 = Idle         │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+#### 5.3 Per-mdev GSP Emulation Thread
+
+```c
+// In open-iov-vfio.ko - new file: gsp_emulation.c
+
+struct gsp_emu_context {
+    struct nv_vgpu_device *vgpu;
+    struct task_struct *thread;
+    void __iomem *bar0_vaddr;
+    void __iomem *doorbell_reg;
+    void __iomem *shared_mem;
+    struct gsp_client *rpc_client;  // From gsp_rpc_client.c
+    spinlock_t lock;
+    bool running;
+    u64 processed_rpcs;
+    u64 failed_rpcs;
+};
+
+static int gsp_emu_thread_fn(void *data) {
+    struct gsp_emu_context *ctx = data;
+    
+    while (!kthread_should_stop()) {
+        // Read doorbell (atomic, no lock needed for single reader)
+        u32 doorbell = readl(ctx->doorbell_reg);
+        
+        if (doorbell & 0x1) {
+            // ACK immediately to prevent guest retry
+            writel(0, ctx->doorbell_reg);
+            
+            // Process RPC - this is the hot path
+            int ret = gsp_emu_process_rpc(ctx);
+            if (ret)
+                ctx->failed_rpcs++;
+            else
+                ctx->processed_rpcs++;
+        }
+        
+        // Adaptive polling: busy-wait for first 100μs, then sleep
+        if (ctx->processed_rpcs % 1000 == 0)
+            usleep_range(50, 100);
+        else
+            cpu_relax();
+    }
+    return 0;
+}
+
+static int gsp_emu_process_rpc(struct gsp_emu_context *ctx) {
+    // Copy request from shared memory (guest → host)
+    struct gsp_rpc_msg req;
+    memcpy_fromio(&req, ctx->shared_mem, sizeof(req));
+    
+    // Validate sequence ID
+    if (req.seq_id == 0)
+        return -EINVAL;
+    
+    // Dispatch to firmware-decompiled handler
+    struct gsp_rpc_response rsp = {0};
+    rsp.seq_id = req.seq_id;
+    rsp.type = req.type;
+    
+    int ret = 0;
+    switch (req.type) {
+        case GSP_RPC_ALLOC:
+            ret = gsp_rpc_alloc(ctx->rpc_client, 
+                req.payload[0], req.payload[1], req.payload[2],
+                req.payload[3], req.payload[4], req.payload[5],
+                &rsp.result[0], &rsp.result[1], &rsp.result[2]);
+            break;
+        case GSP_RPC_FREE:
+            ret = gsp_rpc_free(ctx->rpc_client, req.payload[0]);
+            break;
+        case GSP_RPC_MAP:
+            ret = gsp_rpc_map(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2],
+                req.payload[3], req.payload[4], req.payload[5],
+                &rsp.result[0]);
+            break;
+        case GSP_RPC_UNMAP:
+            ret = gsp_rpc_unmap(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2]);
+            break;
+        case GSP_RPC_PIN:
+            ret = gsp_rpc_pin(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2],
+                req.payload[3], &rsp.result[0], &rsp.result[1]);
+            break;
+        case GSP_RPC_UNPIN:
+            ret = gsp_rpc_unpin(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2]);
+            break;
+        case GSP_RPC_SCHEDULE:
+            ret = gsp_rpc_schedule(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2],
+                req.payload[3], req.payload[4]);
+            break;
+        case GSP_RPC_YIELD:
+            ret = gsp_rpc_yield(ctx->rpc_client);
+            break;
+        case GSP_RPC_NOTIFY:
+            ret = gsp_rpc_notify_register(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2]);
+            break;
+        case GSP_RPC_BAR_TRAP:
+            ret = gsp_rpc_bar_trap(ctx->rpc_client,
+                req.payload[0], req.payload[1], req.payload[2],
+                req.payload[3], req.payload[4], &rsp.result[0]);
+            break;
+        case GSP_RPC_MIGRATE:
+            // Handle migration sub-commands
+            ret = gsp_emu_handle_migrate(ctx, &req, &rsp);
+            break;
+        case GSP_RPC_DEBUG:
+            ret = gsp_emu_handle_debug(ctx, &req, &rsp);
+            break;
+        default:
+            rsp.status = -ENOTTY;
+            ret = -ENOTTY;
+    }
+    
+    // Write response to shared memory (host → guest)
+    rsp.status = ret ? -EIO : 0;
+    memcpy_toio(ctx->shared_mem + 0x80, &rsp, sizeof(rsp));
+    
+    // Inject MSI-X completion interrupt
+    gsp_emu_inject_completion_interrupt(ctx, req.seq_id);
+    
+    return ret;
+}
+```
+
+#### 5.4 MSI-X Completion Interrupt Injection
+
+```c
+static void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id) {
+    // Write to MSI-X PBA (Pending Bit Array) at BAR0 + 0x3000
+    // Vector 0 = GSP completion interrupt
+    void __iomem *pba = ctx->bar0_vaddr + 0x3000;
+    writel(0x1, pba);  // Set pending bit for vector 0
+    
+    // Trigger eventfd for VFIO interrupt injection
+    if (ctx->vgpu->msix_trigger[0])
+        eventfd_signal(ctx->vgpu->msix_trigger[0], 1);
+}
+```
+
+### 6. Performance Requirements
+
+| Metric | Target | Measurement Method |
+|--------|--------|-------------------|
+| GSP RPC round-trip (emulated) | < 10μs | ktime_get_ns() in gsp_emu_thread |
+| BAR register access | < 2μs | Guest rdtsc around MMIO |
+| VRAM allocation (ALLOC) | < 50μs | Guest nvidia.ko tracing |
+| VRAM map (MAP) | < 20μs | Guest nvidia.ko tracing |
+| DMA pin (PIN) | < 30μs | Guest nvidia.ko tracing |
+| CUDA kernel launch overhead | < 5% vs native | CUDA events, nvprof |
+| Graphics frame time (Vulkan) | < 2ms @ 60fps | PresentMon, vk_perf |
+| Migration downtime | < 100ms | Stopwatch, ping during migration |
+| MSI-X injection latency | < 5μs | Guest ISR timestamp vs host |
+
+### 7. Testing Strategy - Incremental Validation
+
+#### Phase 0: Host Infrastructure (Week 1)
+- [ ] `open-iov-vfio.ko` loads, registers mdev parent with `nvidia.ko`
+- [ ] mdev creation via sysfs succeeds
+- [ ] PCI config space presented correctly (lspci in guest shows 10de:1ed3)
+- [ ] BAR regions mmap'able in guest
+- [ ] MSI-X table/PBA accessible in guest BAR0
+
+#### Phase 1: GSP Emulation Core (Week 2)
+- [ ] GSP emulation thread starts per mdev
+- [ ] Doorbell polling detects guest writes
+- [ ] ALLOC RPC: guest allocates VRAM, gets valid GPU VA + handle
+- [ ] FREE RPC: guest frees allocation, no leaks
+- [ ] MAP/UNMAP RPC: guest maps/unmaps GPU VA ranges
+- [ ] PIN/UNPIN RPC: guest pins pages, gets valid DMA addresses
+
+#### Phase 2: Advanced RPCs (Week 3)
+- [ ] SCHEDULE/YIELD: guest submits work, gets completion
+- [ ] BAR_TRAP: guest reads/writes BAR registers, gets emulated values
+- [ ] NOTIFY: guest registers for async events, receives them
+- [ ] MIGRATE: 4-phase migration saves/restores state
+
+#### Phase 3: Guest Driver Integration (Week 4)
+- [ ] Guest `nvidia.ko` loads, probes vGPU successfully
+- [ ] `nvidia-smi` shows vGPU (T4 / A100 equivalent)
+- [ ] CUDA context creation succeeds
+- [ ] Simple CUDA kernel (vectorAdd) executes correctly
+- [ ] `nvidia-ml` (NVML) queries work (temperature, utilization, memory)
+
+#### Phase 4: Compute Workloads (Week 5)
+- [ ] cuBLAS SGEMM/DGEMM passes numerical verification
+- [ ] cuDNN convolution forward/backward passes
+- [ ] PyTorch/TensorFlow training step completes
+- [ ] Multi-GPU (NCCL) across vGPUs works
+- [ ] CUDA Graphs capture/replay works
+
+#### Phase 5: Graphics Workloads (Week 6)
+- [ ] Vulkan: `vkCreateDevice` succeeds, render triangle
+- [ ] OpenGL: `glxinfo` shows renderer, `glxgears` runs
+- [ ] Video encode: FFmpeg + NVENC H.264/HEVC
+- [ ] Video decode: FFmpeg + NVDEC H.264/HEVC
+- [ ] Windows guest: D3D11/D3D12 device creation, render loop
+
+#### Phase 6: Stress & Reliability (Week 7)
+- [ ] 24-hour CUDA stress test (memtest, compute)
+- [ ] 24-hour graphics stress (Unigine, FurMark equivalent)
+- [ ] Hot-plug VF create/destroy cycles (1000x)
+- [ ] Migration under load (live migrate running CUDA kernel)
+- [ ] Host reboot recovery (mdev persists, guest recovers)
+
+#### Phase 7: Windows Guest (Week 8)
+- [ ] Windows 10/11 guest installs NVIDIA driver
+- [ ] Device Manager shows "NVIDIA T4" (or equivalent)
+- [ ] CUDA-Z / GPU-Z detect vGPU correctly
+- [ ] DirectX 11/12: D3D11CreateDevice, D3D12CreateDevice
+- [ ] Vulkan on Windows: vkCreateInstance, render
+- [ ] Video encode/decode via Media Foundation
+
+### 8. Implementation File Map
+
+```
+open-iov/
+├── kernel/
+│   ├── gsp_emulation.c          # NEW: GSP emulation thread + RPC processing
+│   ├── gsp_emulation.h          # NEW: Structures, prototypes
+│   ├── gsp_rpc_client.c         # EXISTING: Firmware-decompiled handlers
+│   ├── gsp_rpc_integration.c    # EXISTING: Integration layer
+│   ├── bar_emulation.c          # EXISTING: BAR trap handling
+│   ├── memory.c                 # EXISTING: DMA, VRAM management
+│   ├── irq.c                    # EXISTING: MSI-X injection
+│   ├── migration.c              # EXISTING: Live migration
+│   ├── main.c                   # MODIFY: Add gsp_emu_context to nv_vgpu_device
+│   └── Makefile                 # MODIFY: Add gsp_emulation.o
+├── userspace/lib/
+│   ├── libvgpu.c                # EXISTING: Management API
+│   ├── rpc_emulator.c           # EXISTING: Host-side emulation fallbacks
+│   └── Makefile                 # UNCHANGED
+├── scripts/
+│   ├── test_gsp_emu.sh          # NEW: Automated GSP emulation tests
+│   ├── test_guest_driver.sh     # NEW: Guest driver validation
+│   ├── test_cuda_workloads.sh   # NEW: CUDA workload validation
+│   ├── test_graphics_workloads.sh # NEW: Graphics validation
+│   └── test_windows_guest.sh    # NEW: Windows guest validation
+└── docs/
+    ├── refactored-architecture.md     # THIS DOCUMENT
+    └── refactored-implementation-to-do.md # TODO LIST
+```
+
+### 9. Risk Mitigation
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|------------|--------|------------|
+| GSP mailbox layout mismatch | Medium | High | Validate against firmware decompilation; add runtime checksum |
+| MSI-X injection race | Medium | High | Use atomic PBA updates; test with concurrent RPCs |
+| Cache coherency (WC memory) | High | Medium | clflush + mfence on every shared mem write; test on AMD/Intel |
+| Guest driver version skew | Medium | Medium | Test against multiple OpenRM versions (550, 560, 570) |
+| Windows guest driver issues | Medium | High | Separate Windows test CI; use WHQL driver |
+| Performance regression vs native | Low | High | Continuous benchmarking; profile hot paths |
+| Migration state corruption | Low | High | Checksum all migrated state; dry-run validation |
+
+---
+
+*Document Version: 1.0 (Refactored Architecture)*
+*Author: Open-IOV Team*
+*Date: $(date)*
+*Repository: $OPEN_IOV_ROOT*
+*Git Commit: f9ed5d8 (restore point)*
\ No newline at end of file