Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 103 insertions, 0 deletions

diff --git a/tools/guest-diag/README.md b/tools/guest-diag/README.md
new file mode 100644
index 0000000..618663b
--- /dev/null
+++ b/tools/guest-diag/README.md
@@ -0,0 +1,103 @@
+# guest-diag - tracing tools for the emulated vGPU guest
+
+Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up.
+All are tiny and dependency-free (libc only).
+
+## ptrace_ioctl.c - ioctl/result tracer (the important one)
+
+libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace
+never see its ioctls. This ptrace tracer follows a child with
+PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps:
+
+- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit
+  (`rmStatus` is the high dword of the 5th qword) - this is how the current
+  failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED`
+- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status
+- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero
+- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27
+
+Build/run **in the guest**:
+
+    gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c
+    /tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^'
+
+Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct
+has `pad[3], arch, instruction_pointer, stack_pointer` before the union.
+
+## sysc_shim.c - known dead end
+
+An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline
+syscall instruction). Kept only as documentation of that fact.
+
+## vmcycle.sh - reload cycle
+
+pkill QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a fresh
+`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into
+launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s.
+
+## launch_qemu_rootport.sh - QEMU command line with pcie-root-port
+
+Snapshot of the working launch command (derived from 's argv of a stopped
+VM) plus the root-port fix. **Requires** the mdev UUID to be substituted and:
+
+    mkdir -p /tmp/sockets/<domain-uuid>      # else QMP bind fails
+    rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock
+
+Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's
+`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex
+Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads
+`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed".
+Behind a `pcie-root-port` QEMU leaves the capability alone.
+
+## rmq_probe.c - legacy-ioctl RM control probe (guest or host)
+
+Minimal RM client (client alloc + device alloc + NV_ESC_RM_CONTROL) to query
+device/subdevice controls from userspace, e.g.
+`0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`,
+`0x800280 GET_NUM_SUBDEVICES`, `0x20800123 GPU_GET_ENGINES`.
+NOTE: the device alloc currently fails (0x40 in guest, 0x1b on host) - the
+device allocation sequence needs the proper NV0080_ALLOC_PARAMETERS handling
+(or the NVOS64 path that tools/rmq/rmq.c uses); fix before relying on it.
+
+## kret.sh - kretprobe attempt (blocked)
+
+Tries to kretprobe the nvGpuOps*/nvUvmInterface* chain to read return values.
+Blocked on this kernel: "trace_kprobe: Could not probe notrace function
+<name> [nvidia]" - kprobes are not available for nvidia module text.
+Kept for documentation.
+
+## push_guest_tools.sh - rebuild the guest-side binaries from these sources
+
+The guest's /tmp is wiped on boot; this pushes+compiles ptio/ge/rmqp/cdt into
+the guest from the sources kept here. Run it after every VM start.
+
+## guest_engines.c - client RM introspection (used with the ptrace tracer)
+
+Queries the guest client RM for exactly the data UVM's register path uses:
+virtualization mode (0x800289), subdevice count (0x800280), CLASS LIST V2
+(0x800292), ENGINE LIST V2 (0x20800170), CE caps (0x20802a03) and CE->PCE
+masks (0x20802a02). Needs the NVOS64 alloc path; see rmq_probe.c note.
+
+## Guest-side UVM logging (critical for diagnosis!)
+
+UVM's messages are silent until enabled:
+
+    echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints
+
+Then run /tmp/cdt and `dmesg | grep nvidia-uvm`. This produced the current
+blocker's exact message:
+`nvidia-uvm: uvm_gpu.c:1574 init_gpu[pid:..] Failed to initialize the
+channel manager: Call not supported [NV_ERR_NOT_SUPPORTED]`
+(also useful: uvm_enable_debug_procfs, uvm_debug_enable_push_desc,
+uvm_release_asserts, uvm_release_asserts_dump_stack)
+
+## launch_qemu_rootport.sh / .run.sh - QEMU command line
+
+Template (root-port fix) + runtime copy with the live mdev UUID, patched by
+vmcycle.sh. Sockets still live under /tmp because scripts/vgpu_vm.py talks to
+/tmp/qemu-guest-agent.sock by design.
+
+## dbg_seq.sh / dbg_seq2.sh - control-sequence helpers (guest, needs ptio)
+
+Print the RM control sequence immediately before/after the failing
+UVM_REGISTER_GPU ioctl.