Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 227 insertions, 0 deletions

diff --git a/wiki/GSP-RPC-Emulation.md b/wiki/GSP-RPC-Emulation.md
new file mode 100644
index 0000000..2e6849a
--- /dev/null
+++ b/wiki/GSP-RPC-Emulation.md
@@ -0,0 +1,227 @@
+# GSP vGPU RPC Emulation
+
+## Overview
+
+Open-IOV includes **emulation mode** that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.
+
+## Problem
+
+The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:
+
+| Handler | Address | Size | Purpose |
+|---------|---------|------|---------|
+| BAR Trap | 0x04CB8188 | 40KB | BAR1/ForceBarAccess/Aperture |
+| WPR Handler 1 | 0x04F00B0C | 62KB | Write/Read Protect Regions |
+| WPR Handler 2 | 0x04F00C78 | 62KB | Write/Read Protect Regions |
+| Migration | 0x04179ED8/0x04179FBC | - | Live migration |
+
+Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).
+
+## Solution: Emulation Mode
+
+Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    Emulation Architecture                        │
+├─────────────────────────────────────────────────────────────────┤
+│                                                                  │
+│  Userspace App (libiov.so)                                       │
+│       │                                                          │
+│       ▼                                                          │
+│  ┌─────────────────┐     ┌─────────────────┐                   │
+│  │ Kernel IOCTL    │────▶│ Emulation Layer │                   │
+│  │ Handler         │     │ (kernel)        │                   │
+│  └─────────────────┘     └────────┬────────┘                   │
+│                                    │                            │
+│                    ┌───────────────┼───────────────┐           │
+│                    ▼               ▼               ▼           │
+│             ┌──────────┐    ┌──────────┐    ┌──────────┐      │
+│             │ BAR Emu  │    │ WPR Emu  │    │ Mem Emu  │      │
+│             │ (kernel) │    │ (kernel) │    │ (kernel) │      │
+│             └──────────┘    └──────────┘    └──────────┘      │
+│                    │               │               │           │
+│                    ▼               ▼               ▼           │
+│             ┌──────────────────────────────────────────┐       │
+│             │         Userspace Emulator (libiov.so)   │       │
+│             │  rpc_emu_bar_*(), rpc_emu_wpr_*()        │       │
+│             └──────────────────────────────────────────┘       │
+│                                                                  │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Auto-Detection (Default Behavior)
+
+**Emulation mode is ON by default** and auto-detects at driver load:
+
+1. Driver initializes RPC communication with GSP
+2. Sends test `BAR_TRAP` RPC (non-destructive READ)
+3. **If GSP responds** (ret=0) → vGPU handlers present → **emulation disabled**
+4. **If GSP times out/errors** (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → **emulation enabled**
+
+```
+dmesg output:
+  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
+  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
+  # or
+  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
+```
+
+## Module Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `emulation_mode` | bool | 1 (ON) | Current state (read-only, shows auto-detected value) |
+| `emulation_override` | bool | 0 (auto) | Override: 1=force emulation ON, 0=force emulation OFF |
+
+```bash
+# Default: auto-detect
+sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
+
+# Force emulation ON (testing with GRID GSP)
+sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1
+
+# Force emulation OFF (requires GRID GSP)
+sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0
+```
+
+## Emulated RPC Handlers
+
+### Kernel Emulation (`driver/`)
+
+| RPC Type | Handler | Emulation |
+|----------|---------|-----------|
+| `GSP_RPC_ALLOC` (0x01) | `gsp_vf_mem_alloc()` | Synthetic handles/VAs from local counter |
+| `GSP_RPC_PIN` (0x05) | `gsp_vf_mem_pin()` | Synthetic DMA addresses from local counter |
+| `GSP_RPC_MAP` (0x03) | `gsp_vf_mem_map()` | Identity map (return requested VA) |
+| `GSP_RPC_BAR_TRAP` (0x0A) | `gsp_bar_trap_handler()` | Local aperture/force_access state |
+| `GSP_RPC_MAP` (WPR) | `gsp_vf_wpr_protect()` | Local WPR region list |
+| `GSP_RPC_UNMAP` (WPR) | `gsp_vf_wpr_unprotect()` | Local WPR region removal |
+
+### Userspace Emulation (`userspace/lib/rpc_emulator.c`)
+
+| Function | Purpose |
+|----------|---------|
+| `rpc_emu_bar_config()` | Local aperture offset/size + backing store |
+| `rpc_emu_bar_force_access()` | Local force_access state |
+| `rpc_emu_bar_save/restore()` | Migration state serialization |
+| `rpc_emu_wpr_protect_write/read()` | Local WPR region tracking |
+| `rpc_emu_wpr_unprotect()` | Local WPR region removal |
+| `rpc_emu_wpr_set_timeout()` | Local timeout tracking |
+| `rpc_emu_wpr_check_access()` | Access violation detection |
+
+## Emulation Data Structures
+
+### BAR State (per VF)
+```c
+struct gsp_bar_state {
+    uint64_t aperture_offset;   // BAR1 aperture base
+    uint64_t aperture_size;     // BAR1 aperture size (max 256KB)
+    bool force_access;          // ForceBarAccess enabled
+    uint32_t force_vf;          // VF holding force access
+    void *backing_store;        // Emulated BAR memory
+    size_t backing_size;        // Backing store size
+};
+```
+
+### WPR Region List (per VF)
+```c
+struct gsp_wpr_region {
+    struct list_head list;
+    uint64_t base;              // Region base address
+    uint64_t size;              // Region size
+    uint32_t flags;             // GSP_WPR_WRITE/READ/EXECUTE
+    uint32_t vf_id;             // Owning VF
+    void (*completion_cb)(...); // Write/read complete callback
+    void (*timeout_cb)(...);    // Timeout callback
+    uint64_t timeout_ns;        // Timeout in nanoseconds
+};
+```
+
+### Userspace Emulator State
+```c
+struct emu_bar_state {
+    uint64_t aperture_offset;
+    uint64_t aperture_size;
+    bool force_access;
+    uint32_t force_vf;
+    void *backing_store;
+    size_t backing_size;
+    pthread_mutex_t lock;
+};
+
+struct emu_wpr_state {
+    struct emu_wpr_region *regions;  // Linked list
+    pthread_mutex_t lock;
+};
+```
+
+## Fallback Logic
+
+Userspace library (`libiov.so`) implements fallback:
+
+```c
+vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
+    // 1. Try kernel IOCTL (sends RPC to GSP)
+    ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
+    
+    // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
+    if (ret < 0 && dev->emulation_enabled && 
+        (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
+        return rpc_emu_bar_config(vf->id, offset, size);
+    }
+    
+    return map_errno(errno);
+}
+```
+
+## Limitations
+
+| Feature | Emulation Status | Notes |
+|---------|-----------------|-------|
+| BAR Aperture Config | ✅ Full | Local state + backing store |
+| ForceBarAccess | ✅ Full | Per-VF exclusive access |
+| BAR Save/Restore | ✅ Full | Migration support |
+| WPR Write Protect | ✅ Full | Region tracking + access check |
+| WPR Read Protect | ✅ Full | Region tracking + access check |
+| WPR Timeout | ✅ Partial | Timeout stored, callbacks not wired |
+| Memory Alloc | ✅ Basic | Synthetic handles/VAs |
+| Memory Pin | ✅ Basic | Synthetic DMA addresses |
+| Memory Map | ✅ Basic | Identity mapping |
+| Migration | ⚠️ Partial | BAR/WPR state only |
+| Scheduling | ❌ No | Returns success, no actual scheduling |
+| Notifications | ❌ No | No async event delivery |
+
+## Testing Emulation
+
+```bash
+# 1. Load driver (auto-enables emulation on consumer GSP)
+sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin
+
+# 2. Verify emulation enabled in dmesg
+dmesg -T | grep -i emulation
+# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
+
+# 3. Run example - should complete without timeouts
+sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu
+
+# 4. Check debugfs for emulated state
+cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
+cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions
+```
+
+## Future Work
+
+- [ ] Wire WPR timeout callbacks to kernel timers
+- [ ] Implement emulated scheduling with work queues
+- [ ] Add async notification support via eventfd
+- [ ] Complete migration state for WPR regions
+- [ ] Performance optimization (batch RPC emulation)
+- [ ] Conformance tests vs GRID GSP behavior
+
+## Related Documentation
+
+- [BAR Emulation Specification](BAR-Emulation-Specification.md)
+- [WPR Management](WPR-Management.md)
+- [RPC Message Format](RPC-Message-Format.md)
+- [Integration Guide](vGPU-IO-Integration-Guide.md)
\ No newline at end of file