diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/DEBUGGING.md | 190 | ||||
| -rw-r--r-- | docs/DEVELOPMENT_WORKFLOW.md | 39 | ||||
| -rw-r--r-- | docs/GSP_EMULATION_ARCHITECTURE.md | 192 | ||||
| -rw-r--r-- | docs/ONBOARDING.md | 166 | ||||
| -rw-r--r-- | docs/PERFORMANCE.md | 98 | ||||
| -rw-r--r-- | docs/VERSION_UPGRADE.md | 137 |
6 files changed, 808 insertions, 14 deletions
diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md new file mode 100644 index 0000000..dc95ed7 --- /dev/null +++ b/docs/DEBUGGING.md @@ -0,0 +1,190 @@ +# Open-IOV Debugging Guide + +> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest +> driver failures, and validating a fully working setup. Built from real +> debugging sessions (including the 2026-09-21 EPT-livelock hunt). + +--- + +## 1. Debug surfaces (know them all) + +| Surface | Where | What it shows | +|---------|-------|---------------| +| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns | +| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead | +| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** | +| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status | +| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** | +| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer | +| GDB on host kernel | (optional) | host-side driver state | + +The VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain: + +```yaml +serialLog: "/tmp/vgpu-guest-serial.log" +gdbPort: 1234 +gdbFreeze: false # true = halt at reset vector (boot debugging) +``` + +--- + +## 2. The golden rule of failure triage + +**When the guest agent dies mid-test, the guest is wedged in kernel mode. +Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.** + +Triage order: +1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages +2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words +3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck? + +--- + +## 3. QEMU GDB stub workflow + +Attach (no symbols needed for triage): + +```bash +gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ + -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ + -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' +``` + +Reading the result: + +| Observation | Meaning | +|-------------|---------| +| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine | +| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver | +| RIPs moving between samples | guest alive and progressing | + +For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical +`$rip` **and** `$rsp` every time = the *same instruction* faults/retries +(EPT-violation livelock) rather than a loop making progress. + +With guest symbols (`vmlinux` + module `.ko` with debug info), add +`add-symbol-file` and get real function names. + +### QMP NMI (soft alternative) + +```bash +(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ + echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ + | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock +``` + +Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU +(spinning with a faulting MMIO instruction) may not print anything — use the +GDB stub instead. + +--- + +## 4. Case study: the 2026-09-21 EPT livelock + +Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive. + +1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with + `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. +2. **GDB stub**: vCPU2 frozen at one address in module text + (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in + `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that + can never complete. +3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0 + shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM + retried forever. +4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see + `docs/GSP_EMULATION_ARCHITECTURE.md` §2. + +Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from +your VFIO mmap handler.** Always question `virt_to_phys()` on anything that +isn't lowmem. + +--- + +## 5. What a *successful* driver bring-up looks like (validation checklist) + +After `stack_manager.sh load-stack` on the host: + +- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers +- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1** +- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg + +After `vgpu_vm.py start vgpu-linux`: + +- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>` +- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started` +- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored) +- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped) +- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root` + +When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`): + +- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first) +- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment) +- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam +- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found") +- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done +- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144** + +Teardown regression: + +- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1** +- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread) +- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone + +Kernel unit test: + +- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED` +- [ ] `sudo rmmod gsp_emulation_test` → clean + +--- + +## 6. Failure taxonomy (symptom → likely cause) + +| Symptom | Likely cause | First check | +|---------|--------------|-------------| +| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | +| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | +| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` | +| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | +| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes | +| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | +| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) | +| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | +| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | +| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot | + +--- + +## 7. Host-guest version matrix discipline + +Probe failures caused by **version mismatch** look identical to emulation bugs. +Before deep debugging, verify the matrix: + +| Component | Must be | Check | +|-----------|---------|-------| +| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) | +| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) | +| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` | +| Emulation target | 570.144W | decompiled firmware basis | + +Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs +570.144 emulation cost a full session before the real mmap bug was found — +the mismatch masked the signal.) + +--- + +## 8. Quick reference + +```bash +# Stack +./scripts/stack_manager.sh status|load-stack|unload|dmesg +# VM (harness ONLY - never raw /qemu) +./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell +# Debug +tail -f /tmp/vgpu-guest-serial.log & +gdb -ex 'target remote localhost:1234' +sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov' +# Recover wedged VM +./scripts/vgpu_vm.py stop <uuid> # or: sudo kill -9 <qemu pid> +``` diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md index 2263af6..9019ac1 100644 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -17,7 +17,7 @@ $OPEN_IOV_ROOT/ # Main monorepo (git + LFS) | Directory | Purpose | |-----------|---------| | `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) | -| `driver/` | Guest vGPU driver (`gsp-vgpu.ko`) - **to be removed** | +| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) | | `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` | | `userspace/examples/` | `example_vgpu` test app | | `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files | @@ -212,8 +212,7 @@ oiv-clean | Artifact | Location | Size | |----------|----------|------| | `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB | -| `open-iov-vfio.ko` | `kernel/` | 2.8MB | -| `gsp-vgpu.ko` | `driver/` | 1.8MB | +| `open-iov-vfio.ko` | `kernel/` | ~3.8MB | | `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB | | `example_vgpu` | `userspace/examples/` | 27KB | @@ -222,32 +221,44 @@ oiv-clean ## 4. Kernel Module Loading ### Prerequisites -- **REBOOT REQUIRED** if kernel cmdline has `module_blacklist=nvidia,...` -- Check: `cat /proc/cmdline | grep module_blacklist` -- GRUB must be updated: `/etc/default/grub` → `update-grub` → `update-initramfs -u` → reboot +- The **stock Ubuntu nvidia driver must not be loaded** when you want our + OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on + desktop boots (udev) and is held open by the GUI. + `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this + machine) prevents that auto-load after reboot; `insmod` is unaffected. +- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia + (compares `/sys/module/nvidia/version` with our build) and refuses with + guidance instead of silently skipping. ### Load Order (Critical) ```bash -# 1. NVIDIA OpenRM host driver (exports vGPU VFIO symbols) -sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko +# 1. VFIO/mdev prerequisites +sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev -# 2. Open-IOV VFIO mdev driver (depends on nvidia.ko symbols) +# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia +# imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the +# mdev parent on the physical GPU. sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko -# 3. Guest vGPU driver (standalone, works without above) -sudo insmod $OPEN_IOV_ROOT/driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin +# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols +# (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers) +sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 4. GSP firmware for the host GPU +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ ``` ### Using Helper Script ```bash -oiv-load # Runs all three in order +./scripts/stack_manager.sh load-stack # prereqs → open-iov-vfio → nvidia → firmware ``` ### Verify Load ```bash -lsmod | grep -E "nvidia|open_iov|gsp_vgpu" +lsmod | grep -E "nvidia|open_iov" dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -ls -la /dev/gsp-vgpu /dev/nvidia* +ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/ +cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances ``` ### Unload diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md new file mode 100644 index 0000000..eb7f697 --- /dev/null +++ b/docs/GSP_EMULATION_ARCHITECTURE.md @@ -0,0 +1,192 @@ +# Open-IOV GSP Emulation Engine — Architecture + +> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, +> `kernel/bar_emulation.c`, or debugging guest driver bring-up. +> +> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21 +> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`; +> for the original refactor plan see `refactored-architecture.md`. + +--- + +## 1. Purpose + +An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects +to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC +messages to a mailbox in BAR0, rings a doorbell, and waits for a completion +interrupt plus a response written back into the mailbox. + +Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the +host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the +host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes +RPCs, performs them in software (or declines them), writes responses back, and +injects MSI-X interrupts. + +``` +┌──────────────────────────── Guest VM ────────────────────────────┐ +│ stock nvidia.ko (570.144) │ +│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │ +│ ▼ │ +│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │ +└──────────────────────────────────────────────────────────────────┘ + │ (16 MB vzalloc'd shadow, uncached) +┌──────────────────────────── Host ────────────────────────────────┐ +│ gsp_emu thread (open-iov-vfio.ko) │ +│ poll shadow+0x110C00 (doorbell) │ +│ → read RPC msg from shadow+0x1000 │ +│ → dispatch (12 RPC types, emulation implementations) │ +│ → write response to shadow+0x1080 │ +│ → set PBA bit at shadow+0x3000 │ +│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │ +└──────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 2. The single source of truth: `bar0_shadow` + +The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software +shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()` +(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler +(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`). + +Everything the guest driver does with BAR0 lands in this shadow: + +| BAR0 offset | Region | Guest writes go to | Emulation reads from | +|------------------|-----------------------------------|--------------------|----------------------| +| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow | +| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` | +| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads | +| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) | +| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads | +| `0x110C00` | **GSP doorbell** | shadow | thread polls this | +| rest | emulated register space | shadow | shadow | + +> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled +> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real +> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer — +> it could never see guest traffic and even wrote PBA bits into the real GPU's +> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`. +> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so +> cleanup never `dma_free`s it. + +### 2.1 Transport protocol (as emulated) + +1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000` + (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`). +2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`. +3. The emulation thread sees the doorbell, ACKs it (writes 0), validates + `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`. +4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`, + `result[6]`, `seq_id`) written to `BAR0+0x1080`. +5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired + eventfd → QEMU delivers MSI-X to the guest. +6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and + accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the + public `gsp_emu_process_rpc()` entry does not bump counters). + +--- + +## 3. RPC dispatch + +`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls +the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation +implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by +`gsp_emu_init`). + +| Type (val) | Handler path in emulation | Notes | +|----------------|--------------------------------------------------------|-------| +| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list | +| FREE (0x02) | `gsp_rpc_free_emul` | | +| MAP (0x03) | `gsp_rpc_map_emul` | | +| UNMAP (0x04) | `gsp_rpc_unmap_emul` | | +| PIN (0x05) | `gsp_rpc_pin_emul` | | +| UNPIN (0x06) | `gsp_rpc_unpin_emul` | | +| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission | +| YIELD (0x08) | no-op success | | +| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet | +| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO | +| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO | +| DEBUG (0xFF) | log-only in dispatch | | +| unknown | `-ENOTSUPP` | | + +> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to +> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox +> and **rings the same doorbell the thread polls** — a self-triggered infinite +> RPC loop plus multi-second stalls. Emulation mode now fails these fast with +> `pr_warn_ratelimited`. + +### Memory model caveat (design note) + +Guest VRAM (BAR1) is mapped **directly to the host GPU's physical BAR1** (gpa). +`gsp_rpc_alloc_emul`, however, services ALLOC with *host system pages* DMA-mapped +to the physical device. These two models are not yet unified: a guest allocation +returning `gpu_va=0x100000000+` is not visible through the guest's BAR1 window. +This is a known design gap to resolve before CUDA workloads; it does not affect +driver probe/GSP boot, which is the current milestone. + +--- + +## 4. Interrupt delivery (MSI-X) + +1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`. + Our ioctl handler takes `eventfd_ctx_fdget()` references into + `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs). +2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`: + sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger + from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after* + `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a + real bug fixed 2026-09-21). + +INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator +currently fires vector 0 for GSP completion. + +--- + +## 5. Lifecycle + +``` +mdev create (nv_vgpu_mdev_create) + └─ schedule init_work + └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow) + └─ nv_vgpu_irq_init / nv_vgpu_memory_init + └─ gsp_emu_init + ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases) + ├─ gsp_client_init_emulation (RPC client, emulation_mode=true) + ├─ gsp_rpc_emulation_set(true) + └─ kthread_run(gsp_emu_thread_fn) + +mdev remove (nv_vgpu_mdev_remove) + └─ cancel init_work, wait for VFIO fds to close + └─ vfio_unregister_group_dev + └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion) + → memory → irq → BAR (frees bar0_shadow) + └─ list_del, pci_dev_put, vfio_put_device +``` + +--- + +## 6. Debugging the emulator + +- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'` +- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on + the thread path only) +- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub + `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation + checklist. +- Unit test: `kernel/tests/gsp_emulation_test.ko` drives + `gsp_emu_process_rpc()` with crafted mailbox messages (requires + `open-iov-vfio.ko` loaded; `depends:` is set accordingly). + +--- + +## 7. Key files + +| File | Role | +|------|------| +| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection | +| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs | +| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state | +| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space | +| `kernel/memory.c` | simple mapping/pin tracking (host management API) | +| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) | diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md new file mode 100644 index 0000000..1d0a4f7 --- /dev/null +++ b/docs/ONBOARDING.md @@ -0,0 +1,166 @@ +# Open-IOV Developer Onboarding + +> Welcome. This guide gets you from zero to a productive contributor in ~30 +> minutes: what the project is, how the repo is laid out, how to build and +> test, and the standards we hold the code to. + +--- + +## 1. What this project is (60 seconds) + +**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver** +runs in a VM with full GPU functionality — no GRID license. The trick: the +host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the +firmware processor the driver talks to) in software, so the guest's stock +`nvidia.ko` can't tell it isn't running on real hardware. + +Everything guest-facing is **host-side only**. There is no Open-IOV code in +the guest. + +How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W +firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types +mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md` +for how the emulator works. + +## 2. Repository tour + +``` +open-iov/ +├── Makefile # top-level: all/kernel/userspace/examples/tests/clean +├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load) +├── README.md # project intro + quick start +├── STATE.md # live project state (works/limitations/bugs) +├── kernel/ # ★ the host driver (open-iov-vfio.ko) +│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config +│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X) +│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs +│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state +│ ├── bar_init.c # physical BAR mapping (host side) +│ ├── memory.c # simple mapping/pin tracking +│ ├── irq.c # IRQ/eventfd plumbing +│ ├── ioctl.c # host management ioctls (libiov consumers) +│ ├── migration.c # 4-phase live-migration skeleton +│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis +│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch +│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell) +│ ├── version/ # driver-version adaptation (550/560/570/580) +│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one) +│ └── tests/ # kernel unit tests (gsp_emulation_test) +├── userspace/ +│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API +│ ├── examples/ # example_vgpu test app +│ └── tests/ +├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built) +├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs) +├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko) +├── patches/nvidia/ # our delta vs upstream OpenRM tag +├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers +├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra +├── config/ # declarative YAML: hardware/, versions/ +├── docs/ # ★ all developer documentation (start here) +└── wiki/ # firmware analysis pages (RPC format, addresses, ...) +``` + +## 3. Your first build & test + +```bash +cd open-iov +./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS) +sudo ./scripts/stack_manager.sh load-stack +sudo ./scripts/stack_manager.sh mdev-create +./scripts/stack_manager.sh status +``` + +Rules that will save your week (also enforced in `stack_manager.sh`): + +1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`. + nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`. +2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware. +3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`. +4. **Version matrix**: guest driver == host OpenRM == host GSP firmware == + emulation target (all **570.144** today). Mismatches look exactly like + emulation bugs — check versions first (`docs/DEBUGGING.md` §7). +5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness). + Never raw `` or `qemu-system-*` commands. + +Run the unit tests (host, needs the module loaded): + +```bash +make tests +sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED +sudo rmmod gsp_emulation_test +``` + +## 4. Coding standards + +We follow the **Linux kernel coding style** for kernel code and common-sense +C for userspace. The practical checklist: + +### Must +- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the + surrounding file. +- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate + `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). +- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use + `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT` + (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a + whole session — it is called out in code review. +- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32` + (IOMMU IOVAs can exceed 32 bits). +- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`, + `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`, + module refs. Unbalanced gets become leaks that block unload. +- **Remove paths must fully unwind init paths** (we had a remove that forgot + `list_del` + kthread stop + VFIO unregister — leaked everything). +- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation + (it rings the doorbell our own thread polls → self-trigger livelock). + Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. +- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant); + rate-limit anything a guest can spam. +- Comments explain **why**, not what — especially around hardware/firmware + behavior discovered by reverse engineering (cite the wiki/address). +- **Signed-off-by** style commit messages: short imperative summary line, + then the *why* (root cause), the *what*, and the user-visible effect. + Reference real dmesg/GDB evidence where relevant. + +### Must not +- No guest-side Open-IOV code. Ever. The guest is stock. +- No reboots/system-affecting commands from scripts without explicit operator + action (see the project's hard rule). +- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone; + nvidia.ko imports *our* symbols, not vice versa. +- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`. +- Don't widen an API's hardware assumptions silently: if a function needs + `vgpu->pdev`, say so in the comment (unit tests can't call those paths). + +### Kernel module specifics +- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`). +- Exported symbols: only what other components genuinely need + (`EXPORT_SYMBOL` right after the definition). +- New per-device resources belong in `struct nv_vgpu_device` with matching + init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain. + +## 5. Documentation expectations + +- New subsystem → new doc in `docs/` + a link from `README.md` and this file. +- Changed behavior → update the affected doc in the **same commit**. +- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table. +- Firmware addresses/structures → `wiki/Address-Reference.md` and friends. + +## 6. Where to read next (in order) + +1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works +2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist +3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design +4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end +5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal +6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference +7. `wiki/` — firmware reverse-engineering canon + +## 7. Getting help / escalating + +- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`. +- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first. +- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test. +- Found a new failure mode? Debug it per the guide, then **write it down** in + the taxonomy table so the next person doesn't rediscover it. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 0000000..6077920 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,98 @@ +# Open-IOV Performance Testing & A/B Methodology + +> How we measure vGPU performance against bare metal, interpret the results, +> and keep overhead on the critical path honest. **Run correctness first** +> (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise. + +--- + +## 1. What overhead is acceptable (targets) + +| Path | Target | Why | +|------|--------|-----| +| GSP RPC round-trip (doorbell→response→MSI-X) | < 50 µs (stretch: < 10 µs) | every driver op goes through here | +| MSI-X injection (eventfd → guest handler) | < 5 µs | completion latency dominates | +| Guest BAR0 MMIO read/write | < 2 µs (shadow RAM) | hot path for register polling | +| CUDA kernel launch (added overhead vs bare metal) | < 10% | user-visible | +| Memory alloc/map RPCs | < 100 µs | setup-time, amortized | +| CUDA compute throughput | ≥ 95% of bare metal | GPU-bound, should be ~free | +| Graphics (frame rate) | ≥ 90% of bare metal | swapchain + submission overhead | + +## 2. The A/B rule + +Every measurement runs **twice on the same machine, same boot if possible**: + +- **A (bare metal)**: workload on the physical GPU via the host's stock + driver (or our OpenRM nvidia.ko directly on the hardware). +- **B (vGPU)**: identical workload in the guest on the emulated vGPU. + +Same binary, same input sizes, same environment (governor, clocks, hugepages). +Report **B/A ratio %** — never absolute numbers alone. + +One variable at a time. If B regresses, you changed the stack, the workload, +or the machine — find which before optimizing. + +## 3. Microbenchmarks (build these first) + +1. **RPC round-trip**: from the guest, submit a DEBUG/NOOP-ish RPC in a loop; + time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` + gives `total_latency_ns` and counts — divide for per-RPC host cost. + (Add `trace_printk` in `gsp_emu_dispatch_rpc` entry/exit for distribution; + read via `/sys/kernel/debug/tracing/trace`.) +2. **MSI-X latency**: timestamp (guest TSC) at RPC submit → in the guest + driver's interrupt handler. Compare with the host eventfd_signal timestamp. +3. **BAR0 shadow read latency**: guest loops reading a shadow register; + expect ~memory speed (uncached, so ~100-300 ns) — anything ≥ 1 µs means + you're trapping somewhere you shouldn't. +4. **Doorbell polling cost**: host `top`/`perf` on `gsp_emu_N` — the adaptive + poll (busy → 50-100 µs sleep) should sit at ~0% CPU when idle, and react + within one sleep quantum under load. Tune `GSP_EMU_POLL_*` only with data. + +## 4. Workload benchmarks + +Runner: `tools/benchmark_infrastructure.sh` (extend it; keep the A/B pattern). +In-guest execution goes through the harness (`vgpu_vm.py exec`) or the guest +test agent (`tools/guest_agent/`). + +| Suite | What | Pass criterion | +|-------|------|----------------| +| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | +| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | +| `nvbandwidth` (if available) | H2D/D2H/device copy GB/s | report; DMA path validation | +| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | +| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | +| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | + +Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` +both sides — throttling skews everything), and the command lines. + +## 5. Where the overhead lives (profile in this order) + +1. **RPC dispatch** — the single biggest lever. Everything the guest driver + does funnels through the emulation thread. +2. **Doorbell polling cadence** — too slow = latency; too hot = host CPU burn. +3. **eventfd/MSI-X delivery** — measure, don't assume. +4. **BAR0 uncached semantics** — shadow is UC for the guest; that's + MMIO-correct but slow for large reads. If the driver bulk-reads shadow, + consider caching-safe sub-regions. +5. **Memory model gap** (ALLOC host-pages vs BAR1 direct VRAM) — see + `docs/GSP_EMULATION_ARCHITECTURE.md` §3. Until unified, DMA-heavy + workloads are not representative. + +## 6. Optimizing safely + +- Change one thing, re-run the A/B pair, keep the numbers with the commit. +- Never optimize off bare-metal-only data: the goal is B/A, not A. +- If a change makes B faster but breaks the §5 validation checklist in + `docs/DEBUGGING.md`, it doesn't count. +- Record results in `STATE.md` (date, versions, B/A ratios) so regressions + are visible across commits. + +## 7. Reporting template + +``` +Date / commit / versions (guest=host=firmware=emulation): +Machine / GPU / clocks locked?: +Test, unit, A (bare), B (vGPU), B/A %: +Notes (what changed vs last run): +``` diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md new file mode 100644 index 0000000..94c7d68 --- /dev/null +++ b/docs/VERSION_UPGRADE.md @@ -0,0 +1,137 @@ +# Open-IOV Version Upgrade Guide + +> How to support a new NVIDIA driver version (e.g. moving from 570.144 to a +> 610-series release) end-to-end: firmware acquisition → reverse engineering → +> config → host driver → guest validation. The goal is that a version bump is +> **configuration + analysis**, not a rewrite. + +--- + +## 0. Version discipline (read first) + +Four components must stay in lock-step; a mismatch between any two produces +failures that are indistinguishable from emulation bugs: + +``` +guest nvidia driver == host OpenRM nvidia.ko == host GSP firmware == emulation target (decompiled GSP) +``` + +Today that anchor is **570.144** everywhere. When you bump one, you bump all +four. Verify with `docs/DEBUGGING.md` §7 before any deep debugging. + +--- + +## 1. Acquire the new firmware + matching OpenRM tag + +1. Download the Linux x86_64 runfile for the target version: + `https://us.download.nvidia.com/XFree86/Linux-x86_64/<VER>/NVIDIA-Linux-x86_64-<VER>.run` + (the `tesla/` path 404s for consumer releases). +2. Extract the GSP firmware (`gsp_tu10x.bin` / arch-specific) — or fetch the + versioned blob from `linux-firmware.git`: + `nvidia/<VER>/gsp_tu10x.bin`. +3. Extract the `.fwimage` section (host boot image) and record the MD5: + ```bash + objcopy --only-section=.fwimage gsp_tu10x.bin gsp-<VER>.fwimage.bin + md5sum gsp-<VER>.fwimage.bin + ``` +4. Note the matching OpenRM tag at + `github.com/NVIDIA/open-gpu-kernel-modules` (e.g. `570.144`). + +## 2. Lay down versioned directories (never overwrite a working version) + +``` +firmware/<SERIES>-GSP/<VER>/ # e.g. firmware/61X-GSP/610.XX/ + gsp-<VER>.fwimage.bin + decompiled/ # ghidra output lands here +config/versions/<MAJOR>.yaml # e.g. config/versions/610.yaml +``` + +Copy the analysis scripts from the previous version dir +(`firmware/57X-GSP/570.144/*.py`) into the new one — they are script-relative. + +## 3. Reverse-engineer the new firmware (Ghidra headless) + +```bash +cd tools/ghidra +python3 analyze_fwimage.py \ + ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \ + /tmp/ghidra_project_<VER> "gsp_<VER>" +cd ../../firmware/<SERIES>-GSP/<VER> +python3 analyze_firmware.py gsp-<VER>.fwimage.bin +``` + +What to extract (compare against `wiki/Address-Reference.md` for 570.144W): + +| Artifact | Why it matters | +|----------|----------------| +| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | +| BAR trap / WPR handlers | BAR emulation behavior | +| Entry/boot sequence | GSP boot RPC ordering the guest driver expects | +| Migration tables | migration protocol (if enabled) | +| RPC message layouts | struct changes break the emulator silently | +| Version string/signature | runtime firmware matching | + +Sanity checks: still RV64IMAC? still the same 12 top-level RPC types? new +subcommands? Confidential-computing/TEE additions? + +## 4. Produce the version config + +Update `config/versions/<MAJOR>.yaml` with: +- `version.name/major/minor` and `firmware.gsp_version` +- discovered dispatcher/handler addresses (if tracked in yaml) +- `features` toggles per what the firmware actually has +- `symbols` (the nvidia.ko vGPU VFIO interface — verify against the new + OpenRM tag; renamed exports break the nvidia build) + +## 5. Update the runtime signature database + +`kernel/firmware_analyzer.c` matches firmware by CRC32/SHA256 + patterns. +Add the new entry to `known_signatures[]` (version string, hashes, discovered +addresses). This is what lets one `open-iov-vfio.ko` serve multiple firmware +versions safely. + +## 6. Rebuild host + guest against the new anchor + +1. Point `patches/nvidia/` at the new OpenRM tag (rebase our delta; verify + `modinfo nvidia.ko | grep depends` still shows `open-iov-vfio`). +2. `./scripts/stack_manager.sh build-all` (kernel → nvidia → userspace). +3. Install the new host firmware to `/lib/firmware/` + (`stack_manager.sh install-firmware`). +4. **Guest**: install the *matching* stock driver in the VM (runfile for the + exact version), replacing the old one. Keep the guest able to boot without + nvidia auto-loading during the swap (temporarily move dkms-built modules + aside — see `docs/DEBUGGING.md`). + +## 7. Validate (same checklist every version) + +Run the full bring-up checklist in `docs/DEBUGGING.md` §5: +module load → mdev → VM boot → guest probe → `nvidia-smi` → teardown +regression → unit test. Then the workload tests (CUDA vectorAdd, graphics). +Only after the checklist is green do performance comparisons +(`docs/PERFORMANCE.md`). + +## 8. Rollback plan + +Keep the previous version's directory + config untouched. Rollback = +`git checkout` the repo tag/commit for the old anchor + reinstall the old +guest driver. Because versions live in parallel directories, the working +setup is never destroyed by a forward attempt. + +## 9. Multi-version testing matrix (target state) + +For each supported anchor version × each supported GPU family +(`config/hardware/*.yaml`), the CI matrix should run the §7 checklist. When +adding a version, extend the matrix rather than replacing the old column — +regressions across versions are the cheapest bugs to catch. + +## 10. Common pitfalls (learned the hard way) + +- **Runfile version drift**: Ubuntu's `nvidia-driver-XXX-open` metapackages + can resolve to a *newer* major (570-open pulled 580). Always check the + candidate version before installing; prefer the exact-version runfile. +- **GSP firmware filename**: guest driver requests `nvidia/<VER>/gsp_tu10x.bin`; + a wrong-version firmware file = probe failure that looks like an emulation bug. +- **Kernel text addresses differ per build**: re-derive Ghidra addresses per + firmware; never carry 570.144 addresses into a 610 config. +- **Symbol renames across OpenRM tags**: `nvidia_vgpu_vfio_*` renames break + the nvidia build with `Unknown symbol` — check `kernel-open` diffs. |