Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 190 insertions, 0 deletions

diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md
new file mode 100644
index 0000000..dc95ed7
--- /dev/null
+++ b/docs/DEBUGGING.md
@@ -0,0 +1,190 @@
+# Open-IOV Debugging Guide
+
+> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest
+> driver failures, and validating a fully working setup. Built from real
+> debugging sessions (including the 2026-09-21 EPT-livelock hunt).
+
+---
+
+## 1. Debug surfaces (know them all)
+
+| Surface | Where | What it shows |
+|---------|-------|---------------|
+| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns |
+| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead |
+| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** |
+| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status |
+| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** |
+| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer |
+| GDB on host kernel | (optional) | host-side driver state |
+
+The  VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain:
+
+```yaml
+serialLog: "/tmp/vgpu-guest-serial.log"
+gdbPort: 1234
+gdbFreeze: false      # true = halt at reset vector (boot debugging)
+```
+
+---
+
+## 2. The golden rule of failure triage
+
+**When the guest agent dies mid-test, the guest is wedged in kernel mode.
+Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.**
+
+Triage order:
+1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages
+2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words
+3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck?
+
+---
+
+## 3. QEMU GDB stub workflow
+
+Attach (no symbols needed for triage):
+
+```bash
+gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \
+    -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \
+    -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit'
+```
+
+Reading the result:
+
+| Observation | Meaning |
+|-------------|---------|
+| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine |
+| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver |
+| RIPs moving between samples | guest alive and progressing |
+
+For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical
+`$rip` **and** `$rsp` every time = the *same instruction* faults/retries
+(EPT-violation livelock) rather than a loop making progress.
+
+With guest symbols (`vmlinux` + module `.ko` with debug info), add
+`add-symbol-file` and get real function names.
+
+### QMP NMI (soft alternative)
+
+```bash
+(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \
+ echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \
+ | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock
+```
+
+Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU
+(spinning with a faulting MMIO instruction) may not print anything — use the
+GDB stub instead.
+
+---
+
+## 4. Case study: the 2026-09-21 EPT livelock
+
+Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive.
+
+1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with
+   `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN.
+2. **GDB stub**: vCPU2 frozen at one address in module text
+   (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in
+   `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that
+   can never complete.
+3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0
+   shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM
+   retried forever.
+4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see
+   `docs/GSP_EMULATION_ARCHITECTURE.md` §2.
+
+Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from
+your VFIO mmap handler.** Always question `virt_to_phys()` on anything that
+isn't lowmem.
+
+---
+
+## 5. What a *successful* driver bring-up looks like (validation checklist)
+
+After `stack_manager.sh load-stack` on the host:
+
+- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers
+- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1**
+- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg
+
+After `vgpu_vm.py start vgpu-linux`:
+
+- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>`
+- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started`
+- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored)
+- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped)
+- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root`
+
+When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`):
+
+- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first)
+- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment)
+- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam
+- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found")
+- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done
+- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144**
+
+Teardown regression:
+
+- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1**
+- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread)
+- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone
+
+Kernel unit test:
+
+- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED`
+- [ ] `sudo rmmod gsp_emulation_test` → clean
+
+---
+
+## 6. Failure taxonomy (symptom → likely cause)
+
+| Symptom | Likely cause | First check |
+|---------|--------------|-------------|
+| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` |
+| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` |
+| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` |
+| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN |
+| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes |
+| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL |
+| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) |
+| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? |
+| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs |
+| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot |
+
+---
+
+## 7. Host-guest version matrix discipline
+
+Probe failures caused by **version mismatch** look identical to emulation bugs.
+Before deep debugging, verify the matrix:
+
+| Component | Must be | Check |
+|-----------|---------|-------|
+| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) |
+| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) |
+| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` |
+| Emulation target | 570.144W | decompiled firmware basis |
+
+Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs
+570.144 emulation cost a full session before the real mmap bug was found —
+the mismatch masked the signal.)
+
+---
+
+## 8. Quick reference
+
+```bash
+# Stack
+./scripts/stack_manager.sh status|load-stack|unload|dmesg
+# VM (harness ONLY - never raw /qemu)
+./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell
+# Debug
+tail -f /tmp/vgpu-guest-serial.log &
+gdb -ex 'target remote localhost:1234'
+sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov'
+# Recover wedged VM
+./scripts/vgpu_vm.py stop <uuid>   # or: sudo kill -9 <qemu pid>
+```