Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 157 insertions, 0 deletions

diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
new file mode 100644
index 0000000..bc38a4e
--- /dev/null
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -0,0 +1,157 @@
+# Debugging and Maintenance Workflows
+
+> The practical book: how to see what is happening, how to find the cause, and
+> how to add a new firmware, driver, hypervisor or hardware version without
+> breaking anything.
+
+## 1. The observability model
+
+Three layers, in order of usefulness:
+
+1. **Guest diagnostics** — the guest driver is the best oracle. It prints
+   precisely what it expected and what it got (e.g. UVM's "Tracker entry for
+   value N … channel_init()"). Always start here.
+2. **Host module trace** — every RPC and every emulation decision is logged
+   with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
+   lines that reconstruct the object tree and the control battery.
+3. **Memory observation** — when the problem is in memory state (mappings,
+   semaphores, pushes), the module's debug parameters read and write guest
+   physical memory directly.
+
+## 2. Standard captures for a bug report
+
+```bash
+# Host
+sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
+./scripts/stack_manager.sh status > status.log
+# Guest
+sudo dmesg > guest-dmesg.log                 # include full dmesg
+sudo lspci -vvv -d 10de: > guest-lspci.log   # config space of the virtual GPU
+cat /proc/cmdline > guest-cmdline.log
+nvidia-smi -q > nvidia-smi-q.log
+```
+
+Plus: the emulated driver profile (GPU type, firmware image hash, driver
+version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
+collects some of this automatically.
+
+## 3. Live debugging toolbox
+
+### 3.1 Kernel logs
+
+```bash
+sudo dmesg -w | grep -iE 'gsp-resman-vfio'          # live emulation trace
+sudo dmesg -T | grep gsp-resman-vfio | tail -100    # recent
+# capture to a file when the ring buffer is small and traffic is heavy:
+sudo stdbuf -oL dmesg -w > run.log
+```
+
+### 3.2 Module debug parameters
+
+See the host-module doc for the full list. Canonical examples:
+
+```bash
+# Find where a value lives in guest RAM (e.g. a semaphore VA):
+echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram
+
+# Find a decoded method sequence in guest RAM:
+echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq
+
+# Write a value to a guest physical address (completing a semaphore release):
+echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
+```
+
+**Caution:** all guest-RAM operations run in a kernel thread context by design;
+calling them from a normal process context fails with `-EFAULT` (the
+hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
+all). This is not a bug.
+
+### 3.3 QEMU gdbstub (guest virtual memory)
+
+The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
+read guest *virtual* memory through the vCPU's page tables and write to it —
+the single most powerful trick we have when the guest's own kernel address
+matters:
+
+```bash
+gdb -batch -ex 'target remote :1234' \
+    -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
+```
+
+Workflow that made the memory model tractable:
+
+1. Find the guest's `page_offset_base` (direct-map base) from
+   `/proc/kallsyms` in the guest.
+2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
+3. Use `poke_guest` to write the physical address from the host module, or use
+   the gdbstub to write the virtual address directly.
+
+### 3.4 Guest-side tracing
+
+* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw
+  syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
+  `PTRACE_GET_SYSCALL_INFO` does.
+* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
+  (messages appear as `nvidia-uvm:`).
+* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.
+
+### 3.5 Failure decision trees
+
+When a failure is understood, write the decision tree into this document tree
+— the tree is what makes the next failure cheap. The field guide's tables are
+the result of that discipline so far.
+
+## 4. Workflows for new inputs
+
+### 4.1 New firmware version
+
+1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS).
+2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
+3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
+   identify it.
+4. Run the driver against a guest using that firmware's matching driver; diff
+   the RPC/control inventory against the baseline.
+5. Add/adjust handlers only where the diff requires it; re-run the ladder.
+
+### 4.2 New guest driver version
+
+1. Install the stock driver in a guest (no patches).
+2. Capture the full init trace and diff function/control IDs against the
+   previous version.
+3. For each difference: decode against our decompiled study notes for that version
+   (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers).
+4. Add a version-keyed handler or a capability table entry.
+5. Re-run rungs 1–2; then the CUDA corpus.
+
+### 4.3 New hardware generation
+
+1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
+2. Run the ladder on that hardware with the closest driver family.
+3. Fix differences by configuration first; only add code when an interface
+   genuinely differs.
+4. Publish the profile and the test evidence.
+
+### 4.4 New hypervisor
+
+1. Implement the transport boundary (device registration, guest-memory access,
+   interrupt injection) — the boundary is small by design.
+2. Re-run the ladder; the emulation core must not need changes.
+3. Document any hypervisor-specific quirk in the debugging docs.
+
+## 5. Crash and hang recovery
+
+| Situation | Action |
+|---|---|
+| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
+| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
+| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
+| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |
+
+## 6. Release engineering
+
+* Versions are tagged; each tag carries the rung/matrix table and the
+  performance table for the platforms tested.
+* Configuration profiles are versioned with the code; a profile bump is a
+  reviewed change with evidence.
+* The docs in this tree are part of the release: a release that changes
+  behaviour without updating them is incomplete.