From 68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- tools/guest-diag/README.md | 103 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 tools/guest-diag/README.md (limited to 'tools/guest-diag/README.md') diff --git a/tools/guest-diag/README.md b/tools/guest-diag/README.md new file mode 100644 index 0000000..618663b --- /dev/null +++ b/tools/guest-diag/README.md @@ -0,0 +1,103 @@ +# guest-diag - tracing tools for the emulated vGPU guest + +Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up. +All are tiny and dependency-free (libc only). + +## ptrace_ioctl.c - ioctl/result tracer (the important one) + +libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace +never see its ioctls. This ptrace tracer follows a child with +PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps: + +- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit + (`rmStatus` is the high dword of the 5th qword) - this is how the current + failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED` +- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status +- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero +- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27 + +Build/run **in the guest**: + + gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c + /tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^' + +Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct +has `pad[3], arch, instruction_pointer, stack_pointer` before the union. + +## sysc_shim.c - known dead end + +An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline +syscall instruction). Kept only as documentation of that fact. + +## vmcycle.sh - reload cycle + +pkill QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a fresh +`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into +launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s. + +## launch_qemu_rootport.sh - QEMU command line with pcie-root-port + +Snapshot of the working launch command (derived from 's argv of a stopped +VM) plus the root-port fix. **Requires** the mdev UUID to be substituted and: + + mkdir -p /tmp/sockets/ # else QMP bind fails + rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock + +Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's +`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex +Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads +`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed". +Behind a `pcie-root-port` QEMU leaves the capability alone. + +## rmq_probe.c - legacy-ioctl RM control probe (guest or host) + +Minimal RM client (client alloc + device alloc + NV_ESC_RM_CONTROL) to query +device/subdevice controls from userspace, e.g. +`0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`, +`0x800280 GET_NUM_SUBDEVICES`, `0x20800123 GPU_GET_ENGINES`. +NOTE: the device alloc currently fails (0x40 in guest, 0x1b on host) - the +device allocation sequence needs the proper NV0080_ALLOC_PARAMETERS handling +(or the NVOS64 path that tools/rmq/rmq.c uses); fix before relying on it. + +## kret.sh - kretprobe attempt (blocked) + +Tries to kretprobe the nvGpuOps*/nvUvmInterface* chain to read return values. +Blocked on this kernel: "trace_kprobe: Could not probe notrace function + [nvidia]" - kprobes are not available for nvidia module text. +Kept for documentation. + +## push_guest_tools.sh - rebuild the guest-side binaries from these sources + +The guest's /tmp is wiped on boot; this pushes+compiles ptio/ge/rmqp/cdt into +the guest from the sources kept here. Run it after every VM start. + +## guest_engines.c - client RM introspection (used with the ptrace tracer) + +Queries the guest client RM for exactly the data UVM's register path uses: +virtualization mode (0x800289), subdevice count (0x800280), CLASS LIST V2 +(0x800292), ENGINE LIST V2 (0x20800170), CE caps (0x20802a03) and CE->PCE +masks (0x20802a02). Needs the NVOS64 alloc path; see rmq_probe.c note. + +## Guest-side UVM logging (critical for diagnosis!) + +UVM's messages are silent until enabled: + + echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints + +Then run /tmp/cdt and `dmesg | grep nvidia-uvm`. This produced the current +blocker's exact message: +`nvidia-uvm: uvm_gpu.c:1574 init_gpu[pid:..] Failed to initialize the +channel manager: Call not supported [NV_ERR_NOT_SUPPORTED]` +(also useful: uvm_enable_debug_procfs, uvm_debug_enable_push_desc, +uvm_release_asserts, uvm_release_asserts_dump_stack) + +## launch_qemu_rootport.sh / .run.sh - QEMU command line + +Template (root-port fix) + runtime copy with the live mdev UUID, patched by +vmcycle.sh. Sockets still live under /tmp because scripts/vgpu_vm.py talks to +/tmp/qemu-guest-agent.sock by design. + +## dbg_seq.sh / dbg_seq2.sh - control-sequence helpers (guest, needs ptio) + +Print the RM control sequence immediately before/after the failing +UVM_REGISTER_GPU ioctl. -- cgit v1.2.3