diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 00:51:55 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 00:51:55 -0400 |
| commit | 7897fe79ba6a00cc59b7a357903ecfc70480ce97 (patch) | |
| tree | 394a47a5d22b2d6072e19c92e9ced1e667817788 /tools/guest-diag/README.md | |
| parent | 7aef82a5112ee72ee2fc5a430db072085933e927 (diff) | |
| download | open-iov-7897fe79ba6a00cc59b7a357903ecfc70480ce97.tar.gz open-iov-7897fe79ba6a00cc59b7a357903ecfc70480ce97.zip | |
tools: add guest-diag (ptrace ioctl tracer, vmcycle, root-port QEMU cmdline)
The ptrace tracer resolves the current CUDA blocker: libcuda issues raw
syscall instructions, so LD_PRELOAD shims and ltrace see nothing; following
the child with PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO does. It found
UVM_REGISTER_GPU (ioctl 0x25) returning rmStatus=0x56 NV_ERR_NOT_SUPPORTED,
with inputs identical to the working host run (rmCtrlFd=-1, hClient=0,
hSmcPartRef=0). Also dumps NV_ESC_RM_CONTROL cmd/status and RM alloc/free
statuses (all zero) and the UVM ioctl set (guest stops after
0x27,0x1,0x25,0x2 vs 141 ioctls on the host).
launch_qemu_rootport.sh is the working QEMU command line with the
pcie-root-port fix (see its header + the handoff PART 4); vmcycle.sh drives
the module/mdev/VM reload cycle around it.
Diffstat (limited to 'tools/guest-diag/README.md')
| -rw-r--r-- | tools/guest-diag/README.md | 50 |
1 files changed, 50 insertions, 0 deletions
diff --git a/tools/guest-diag/README.md b/tools/guest-diag/README.md new file mode 100644 index 0000000..e113ea5 --- /dev/null +++ b/tools/guest-diag/README.md @@ -0,0 +1,50 @@ +# guest-diag - tracing tools for the emulated vGPU guest + +Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up. +All are tiny and dependency-free (libc only). + +## ptrace_ioctl.c - ioctl/result tracer (the important one) + +libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace +never see its ioctls. This ptrace tracer follows a child with +PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps: + +- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit + (`rmStatus` is the high dword of the 5th qword) - this is how the current + failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED` +- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status +- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero +- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27 + +Build/run **in the guest**: + + gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c + /tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^' + +Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct +has `pad[3], arch, instruction_pointer, stack_pointer` before the union. + +## sysc_shim.c - known dead end + +An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline +syscall instruction). Kept only as documentation of that fact. + +## vmcycle.sh - reload cycle + +pkill QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a fresh +`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into +launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s. + +## launch_qemu_rootport.sh - QEMU command line with pcie-root-port + +Snapshot of the working launch command (derived from 's argv of a stopped +VM) plus the root-port fix. **Requires** the mdev UUID to be substituted and: + + mkdir -p /tmp/sockets/<domain-uuid> # else QMP bind fails + rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock + +Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's +`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex +Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads +`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed". +Behind a `pcie-root-port` QEMU leaves the capability alone. |
