From 2302fc09080bad51c73f293294e19fd4db464eb0 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- .../02-debugging-and-maintenance-workflows.md | 157 +++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 docs/05-operations/02-debugging-and-maintenance-workflows.md (limited to 'docs/05-operations/02-debugging-and-maintenance-workflows.md') diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md new file mode 100644 index 0000000..bc38a4e --- /dev/null +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -0,0 +1,157 @@ +# Debugging and Maintenance Workflows + +> The practical book: how to see what is happening, how to find the cause, and +> how to add a new firmware, driver, hypervisor or hardware version without +> breaking anything. + +## 1. The observability model + +Three layers, in order of usefulness: + +1. **Guest diagnostics** — the guest driver is the best oracle. It prints + precisely what it expected and what it got (e.g. UVM's "Tracker entry for + value N … channel_init()"). Always start here. +2. **Host module trace** — every RPC and every emulation decision is logged + with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred + lines that reconstruct the object tree and the control battery. +3. **Memory observation** — when the problem is in memory state (mappings, + semaphores, pushes), the module's debug parameters read and write guest + physical memory directly. + +## 2. Standard captures for a bug report + +```bash +# Host +sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log +./scripts/stack_manager.sh status > status.log +# Guest +sudo dmesg > guest-dmesg.log # include full dmesg +sudo lspci -vvv -d 10de: > guest-lspci.log # config space of the virtual GPU +cat /proc/cmdline > guest-cmdline.log +nvidia-smi -q > nvidia-smi-q.log +``` + +Plus: the emulated driver profile (GPU type, firmware image hash, driver +version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh` +collects some of this automatically. + +## 3. Live debugging toolbox + +### 3.1 Kernel logs + +```bash +sudo dmesg -w | grep -iE 'gsp-resman-vfio' # live emulation trace +sudo dmesg -T | grep gsp-resman-vfio | tail -100 # recent +# capture to a file when the ring buffer is small and traffic is heavy: +sudo stdbuf -oL dmesg -w > run.log +``` + +### 3.2 Module debug parameters + +See the host-module doc for the full list. Canonical examples: + +```bash +# Find where a value lives in guest RAM (e.g. a semaphore VA): +echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram + +# Find a decoded method sequence in guest RAM: +echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq + +# Write a value to a guest physical address (completing a semaphore release): +echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest +``` + +**Caution:** all guest-RAM operations run in a kernel thread context by design; +calling them from a normal process context fails with `-EFAULT` (the +hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at +all). This is not a bug. + +### 3.3 QEMU gdbstub (guest virtual memory) + +The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can +read guest *virtual* memory through the vCPU's page tables and write to it — +the single most powerful trick we have when the guest's own kernel address +matters: + +```bash +gdb -batch -ex 'target remote :1234' \ + -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1' +``` + +Workflow that made the memory model tractable: + +1. Find the guest's `page_offset_base` (direct-map base) from + `/proc/kallsyms` in the guest. +2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`. +3. Use `poke_guest` to write the physical address from the host module, or use + the gdbstub to write the virtual address directly. + +### 3.4 Guest-side tracing + +* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw + syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with + `PTRACE_GET_SYSCALL_INFO` does. +* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` + (messages appear as `nvidia-uvm:`). +* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them. + +### 3.5 Failure decision trees + +When a failure is understood, write the decision tree into this document tree +— the tree is what makes the next failure cheap. The field guide's tables are +the result of that discipline so far. + +## 4. Workflows for new inputs + +### 4.1 New firmware version + +1. Add the image under `firmware/-GSP//` (tracked by LFS). +2. Compute signatures (CRC32/SHA256) with the firmware analyzer. +3. Add a firmware profile entry (addresses/signatures) so runtime analysis can + identify it. +4. Run the driver against a guest using that firmware's matching driver; diff + the RPC/control inventory against the baseline. +5. Add/adjust handlers only where the diff requires it; re-run the ladder. + +### 4.2 New guest driver version + +1. Install the stock driver in a guest (no patches). +2. Capture the full init trace and diff function/control IDs against the + previous version. +3. For each difference: decode against our decompiled study notes for that version + (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +4. Add a version-keyed handler or a capability table entry. +5. Re-run rungs 1–2; then the CUDA corpus. + +### 4.3 New hardware generation + +1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list). +2. Run the ladder on that hardware with the closest driver family. +3. Fix differences by configuration first; only add code when an interface + genuinely differs. +4. Publish the profile and the test evidence. + +### 4.4 New hypervisor + +1. Implement the transport boundary (device registration, guest-memory access, + interrupt injection) — the boundary is small by design. +2. Re-run the ladder; the emulation core must not need changes. +3. Document any hypervisor-specific quirk in the debugging docs. + +## 5. Crash and hang recovery + +| Situation | Action | +|---|---| +| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) | +| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed | +| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first | +| Kernel oops in the host module | capture the full oops + `dmesg` before reloading | + +## 6. Release engineering + +* Versions are tagged; each tag carries the rung/matrix table and the + performance table for the platforms tested. +* Configuration profiles are versioned with the code; a profile bump is a + reviewed change with evidence. +* The docs in this tree are part of the release: a release that changes + behaviour without updating them is incomplete. -- cgit v1.2.3