Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 305 insertions, 0 deletions

diff --git a/wiki/GSP-570.144W-Firmware-Analysis.md b/wiki/GSP-570.144W-Firmware-Analysis.md
new file mode 100644
index 0000000..d079b0b
--- /dev/null
+++ b/wiki/GSP-570.144W-Firmware-Analysis.md
@@ -0,0 +1,305 @@
+# NVIDIA GSP 570.144W Firmware Analysis (TU102/TU104)
+
+## Binary Information
+- **File**: `gsp-570.144.fwimage.bin`
+- **Size**: 28.5 MB
+- **Format**: ELF, RV64IMAC (RISC-V 64-bit)
+- **Base Address**: 0x04000000
+- **MD5**: `13ce12a392a31eeb79a2e6ebd3a4a93a`
+
+## Architecture Overview
+
+### Memory Map
+```
+0x04000000 - 0x05FFFFFF  : Code + Read-only data (~32 MB)
+0x04296040               : Global state structure (cRam0000000004296048)
+0x0429B750               : Error/status registers
+0x041DC350               : Stack canary / frame guard
+0x429B000 - 0x429BFFF    : Memory allocator arena (0x429B450 = alloc entry)
+```
+
+### Calling Convention (RISC-V RV64IMAC)
+| Register | Purpose |
+|----------|---------|
+| s0 (x8) | Frame pointer |
+| s1 (x9), s4 (x12), s10 (x20) | RA save locations |
+| s1-s11 | Callee-saved (pushed to stack) |
+| a0-a7 | Arguments / return values |
+| sp | Stack pointer (grows down, frames 96-560 bytes) |
+
+---
+
+## RPC System
+
+### 12 RPC Types (Jump Table at 0x059BFEB0)
+
+| Type | Name | Handler A | Handler B | Dispatcher |
+|------|------|-----------|-----------|------------|
+| 0x01 | ALLOC | — | — | Primary |
+| 0x02 | FREE | — | — | Primary |
+| 0x03 | MAP | 0x05766AB7 | 0x05778A37 | Secondary |
+| 0x04 | UNMAP | 0x057669B7 | 0x05779937 | Primary + Secondary |
+| 0x05 | PIN | 0x057664B7 | 0x0577A3B7 | Secondary |
+| 0x06 | UNPIN | 0x057682B7 | 0x05779FB7 | Secondary |
+| 0x07 | SCHEDULE | 0x05767F37 | 0x0575AEB7 | Primary + Secondary |
+| 0x08 | YIELD | 0x0575AE37 | 0x05767337 | Secondary |
+| 0x09 | NOTIFY | 0x0577C8B7 | 0x05767837 | Secondary |
+| 0x0A | BAR_TRAP | 0x057675B7 | — | Secondary |
+| 0x0B | MIGRATE | — | — | Primary |
+| 0x0C | DEBUG | — | — | — |
+
+### Primary Dispatcher (0x057A1AC0)
+```
+Entry: FUN_ram_057a1ac0(param_1, param_2, param_3, param_4, *out_id, *out_a, *out_b, *out_c)
+```
+- Extracts RPC type from `param_1`
+- Validates against max type (bgeu x12, x13 checks for types 11 and 13)
+- MIGRATE (0x0B) branches at 0x057A1BDC and 0x057A1D54
+- Indexes into 32-entry jump table at **0x057A3AD8**
+
+#### Primary Dispatcher Sub-Command Jump Table (0x057A3AD8)
+
+| Subcmd | Target | Decompiled | Description |
+|--------|--------|------------|-------------|
+| 0 | 0x057A1DB4 | ✅ | Returns ID=0x40 |
+| 1 | 0x057A1C18 | ✅ | |
+| 2 | 0x057A39FC | ✅ | |
+| 3 | 0x057A1BCC | ✅ | |
+| 4 | 0x057A1EF4 | ✅ | |
+| 5 | 0x057A1E94 | ✅ | |
+| 6 | 0x057A3A28 | ✅ | |
+| 7 | 0x057A3A60 | ✅ | |
+| 8 | 0x057A1C6C | ✅ | |
+| 9 | 0x057A3A48 | ✅ | |
+| 10 | 0x057A1F6C | ✅ | |
+| 11 | 0x057A3AC4 | ✅ | |
+| 12 | 0x057A1E10 | ✅ | |
+| 13 | 0x057A3A90 | ✅ | |
+| 14 | 0x057A39BC | ✅ | |
+| 15 | 0x057A3AAC | ✅ | |
+| 16 | 0x057A3A78 | ✅ | |
+| 17 | 0x057A1CCC | ✅ | |
+| 18 | 0x057A39E0 | ✅ | |
+| 19 | 0x057A1FA8 | ✅ | |
+| 20 | 0x057A2094 | ✅ | |
+| 21 | 0x057A2074 | ✅ | |
+| 22 | 0x057A2044 | ✅ | |
+| 23 | 0x057A20D4 | ✅ | |
+| 24 | 0x057A20B4 | ✅ | |
+| 25 | 0x057A1FC8 | ✅ | |
+| 26 | 0x057A2008 | ✅ | |
+| 27 | 0x057A1FE8 | ✅ | |
+| 28 | 0x057A2024 | ✅ | |
+| 29 | 0x057A219C | ✅ | |
+| 30 | 0x057A20FC | ✅ | |
+| 31 | 0x057A214C | ✅ | |
+
+**Pattern**: Each sub-command returns a constant ID in `*out_id` (e.g., 0x10, 0x14, 0x40) and echoes input params to output pointers.
+
+### Secondary Dispatcher (0x04ED39DC)
+- Branch-based dispatch for ALL 11 RPC types
+- 141 KB decompiled — handles MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP
+- Entry: 0x04ED34B8 (PIN handler from secondary)
+
+### Specialized Main Handlers
+
+| Function | Address | Size | Role |
+|----------|---------|------|------|
+| `rpc_alloc_main` | 0x057E94EC | 40 KB | Core ALLOC implementation |
+| `rpc_schedule_main` | 0x051EFBF4 | 1.8 KB | Core SCHEDULE implementation |
+| `rpc_bar_trap_handler` | 0x04CB8188 | 40 KB | BAR1 emulation |
+| `rpc_pin_handler` | 0x04ED34B8 | — | PIN from secondary dispatcher |
+
+---
+
+## BAR Emulation (RPC 0x0A)
+
+### Handler: 0x04CB8188 (40 KB)
+**Strings Referenced**:
+- `BAR1` at 0x0406570E, 0x04085DE0
+- `Bar1Rest` at 0x040656DA
+- `ForceBarAccess` at 0x040656F2
+- `Bar1Aperture` at 0x04065722
+
+**Key Operations**:
+- Memory allocation via `UNK_ram_058A1A9C` at arena 0x429B450
+- Buffer management with 0x429B428 (cache/flush)
+- Error codes: 0x51 (OOM), 0x1E (timeout), 0x1D, 0x1F, 0x25, 0x37, 0x56
+- Uses `ebreak` for fatal errors with logging at 0x05AB7DA0
+- Stack canary check against 0x041DC350
+
+**Integration Point for vGPU-IO**:
+```c
+// Register BAR trap handler at 0x04CB8188
+// Monitor BAR1, Bar1Rest, ForceBarAccess, Bar1Aperture registers
+// Emulate BAR1 aperture/size for guest VF
+```
+
+---
+
+## WPR Management
+
+### Handlers: 0x04F00B0C, 0x04F00C78 (62 KB each)
+
+**String References (60 locations)**:
+- `WPR_WRITE_COMPLETE`, `WPR_READ_COMPLETE`
+- `WPR_WRITE_TIMEOUT`, `WPR_READ_TIMEOUT`
+- `NV_WPR_WRITE_COMPLETE`, `NV_WPR_READ_COMPLETE`
+- `NV_WPR_WRITE_TIMEOUT`, `NV_WPR_READ_TIMEOUT`
+
+**Key Operations**:
+- VBIOS parsing (checks for `0xAA55`, `0x57464F4E` "WFON", `0x5746564E` "WFVN")
+- Memory allocation at 0x429B450
+- DMA/copy via `UNK_ram_0588F9C4`, `UNK_ram_05AEC86C`
+- State machine with completion/timeout handling
+- Error codes: 0x1E-0x25, 0x37, 0x51, 0x56, 0xFFFF
+
+**Integration Point for vGPU-IO**:
+```c
+// Hook WPR completion callbacks from string reference locations
+// Implement NV_WPR_WRITE/READ_COMPLETE/TIMEOUT notifications
+// Manage WPR regions for guest VF memory isolation
+```
+
+---
+
+## Migration Protocol (RPC 0x0B)
+
+### Dual Jump Tables
+
+#### Table 1: 0x04179ED8 (Primary dispatcher MIGRATE branch at 0x057A1BDC)
+| Index | Target | Decompiled |
+|-------|--------|------------|
+| 0 | 0x057A2344 | ✅ Returns ID=0x14, reads flag at `in_t1+0x7F9` |
+| 1 | 0x057A2320 | ✅ |
+| 2 | 0x057A230C | ✅ |
+| 3 | 0x057A22F8 | ✅ Returns ID=0x10 |
+| 4 | 0x057A2388 | ✅ |
+| 5 | 0x057A23D8 | ✅ |
+| 6 | 0x057A23AC | ✅ |
+| 7 | 0x057A2478 | ✅ |
+| 8 | 0x057A2428 | ✅ |
+| 9 | 0x057A2454 | ✅ |
+| 10 | 0x057A2404 | ✅ |
+| 11 | 0x057A2364 | ✅ |
+| 12 | 0x057A1F3C | (dup) |
+| 13 | 0x057A299C | |
+| 14 | 0x057A1F3C | (dup) |
+| 15 | 0x057A1F3C | (dup) |
+
+#### Table 2: 0x04179FBC (Primary dispatcher MIGRATE branch at 0x057A1D5C)
+| Index | Target | Decompiled |
+|-------|--------|------------|
+| 0 | 0x057A257C | ✅ |
+| 1 | 0x057A25E4 | ✅ |
+| 2 | 0x057A24EC | ✅ |
+| 3 | 0x057A2608 | ✅ |
+| 4 | 0x057A2534 | ✅ |
+| 5 | 0x057A25C0 | ✅ |
+| 6 | 0x057A24A4 | ✅ |
+| 7 | 0x057A259C | ✅ |
+| 8 | 0x057A2510 | ✅ |
+| 9 | 0x057A2558 | ✅ |
+| 10 | 0x057A24C8 | ✅ |
+| 11 | 0x057A262C | ✅ |
+| 12 | 0x057A345C | ✅ |
+| 13 | 0x057A36D8 | ✅ |
+| 14 | 0x057A35D8 | ✅ |
+| 15 | 0x057A3658 | ✅ |
+
+**Pattern**: Migration sub-commands follow same descriptor pattern as primary dispatcher — return constant ID, echo params.
+
+---
+
+## Entry Point & Boot Sequence
+
+### 0x04009000 (573 bytes)
+- Initializes global state at 0x04296040
+- Sets up stack canary at 0x041DC350
+- Configures memory allocator arena at 0x429B450
+- Registers RPC dispatchers
+- Jumps to main scheduler loop
+
+---
+
+## Decompiled Output Directory
+
+```
+$OPEN_IOV_ROOT/firmware/570-GSP/decompiled/
+├── entry_ram_04009000.c                    # Boot entry
+├── FUN_ram_04cb8188_ram_04cb8188.c         # BAR trap handler (40 KB)
+├── FUN_ram_04cc58f8_ram_04cc58f8.c         # Secondary dispatcher (141 KB)
+├── FUN_ram_04d00a18_ram_04d00a18.c         # ALLOC-dominant (142 KB)
+├── FUN_ram_04ed39dc_ram_04ed39dc.c         # Secondary dispatcher (141 KB)
+├── FUN_ram_04f00b0c_ram_04f00b0c.c         # WPR handler 1 (62 KB)
+├── FUN_ram_04f00c78_ram_04f00c78.c         # WPR handler 2 (62 KB)
+├── FUN_ram_051efbf4_ram_051efbf4.c         # SCHEDULE main (1.8 KB)
+├── FUN_ram_054a7de0_ram_054a7de0.c         # ALLOC-dominant (67 KB)
+├── FUN_ram_05723ef4_ram_05723ef4.c         # Tertiary dispatcher (13 KB)
+├── FUN_ram_057a1ac0_ram_057a1ac0.c         # Primary dispatcher (558 B)
+├── FUN_ram_057a1bcc_ram_057a1bcc.c ...     # 32 sub-commands (~550 B each)
+├── FUN_ram_057e94ec_ram_057e94ec.c         # ALLOC main (40 KB)
+├── FUN_ram_059be7f4_ram_059be7f4.c         # RPC jump table (4.9 KB)
+├── FUN_ram_057664b7_ram_057664b7.c ...     # MAP/PIN/UNPIN/SCHED/YIELD/NOTIFY handlers
+└── FUN_ram_057a22f8_ram_057a22f8.c ...     # 28 migration handlers
+```
+
+---
+
+## vGPU-IO Integration Checklist
+
+### Memory Management (ALLOC/FREE/MAP/UNMAP)
+- [ ] Implement kernel RPC client calling 0x057A1AC0
+- [ ] Parse sub-command descriptors from 0x057A3AD8 jump table
+- [ ] Hook ALLOC main at 0x057E94EC for vGPU memory carve-out
+
+### Page Pinning (PIN/UNPIN)
+- [ ] Call secondary dispatcher PIN at 0x04ED34B8
+- [ ] Implement UNPIN via 0x057682B7 / 0x05779FB7
+
+### Scheduling (SCHEDULE/YIELD)
+- [ ] Use 0x05767F37 / 0x0575AEB7 / 0x051EFBF4 for vCPU scheduling
+- [ ] Implement YIELD at 0x0575AE37 / 0x05767337
+
+### BAR Emulation (BAR_TRAP)
+- [ ] Register handler at 0x04CB8188
+- [ ] Emulate BAR1/Bar1Rest/ForceBarAccess/Bar1Aperture for VF
+- [ ] Monitor 0x429B450 allocator for BAR backing store
+
+### WPR Management
+- [ ] Hook completion callbacks from string refs (0x04F00B0C, 0x04F00C78)
+- [ ] Implement NV_WPR_* notification forwarding to guest
+- [ ] Manage WPR regions per-VF for isolation
+
+### Migration (MIGRATE)
+- [ ] Implement dual jump table dispatch (0x04179ED8, 0x04179FBC)
+- [ ] Handle 28 migration sub-commands
+- [ ] Coordinate with guest VF live migration protocol
+
+---
+
+## Tools & Scripts
+
+| Script | Purpose |
+|--------|---------|
+| `DecompileFunctions.java` | Ghidra headless decompilation |
+| `trace_all_subsystems.py` | Binary analysis: WPR/BAR/Migration strings, jump tables |
+| `gsp-fwimage-570.144/` | Ghidra project directory |
+
+### Ghidra Usage
+```bash
+/snap/ghidra/current/ghidra/support/analyzeHeadless \
+  /path/to/project gsp-analysis \
+  -import gsp-570.144.fwimage.bin \
+  -noanalysis \
+  -postScript DecompileFunctions.java \
+  0xADDRESS1,0xADDRESS2,...
+```
+
+---
+
+## References
+- Handoff: `$OPEN_IOV_ROOT/../Handoff-vGPU-IO.md`
+- RPC Map: `$OPEN_IOV_ROOT/firmware/570-GSP/RPC_HANDLER_MAP.md`
+- Final Report: `$OPEN_IOV_ROOT/firmware/570-GSP/FINAL_REPORT.md`
+- Analysis Results: `$OPEN_IOV_ROOT/firmware/570-GSP/analysis_results.json`
\ No newline at end of file