1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
|
# guest-diag - tracing tools for the emulated vGPU guest
Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up.
All are tiny and dependency-free (libc only).
## ptrace_ioctl.c - ioctl/result tracer (the important one)
libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace
never see its ioctls. This ptrace tracer follows a child with
PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps:
- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit
(`rmStatus` is the high dword of the 5th qword) - this is how the current
failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED`
- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status
- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero
- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27
Build/run **in the guest**:
gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c
/tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^'
Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct
has `pad[3], arch, instruction_pointer, stack_pointer` before the union.
## sysc_shim.c - known dead end
An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline
syscall instruction). Kept only as documentation of that fact.
## vmcycle.sh - reload cycle
pkill QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a fresh
`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into
launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s.
## launch_qemu_rootport.sh - QEMU command line with pcie-root-port
Snapshot of the working launch command (captured from a running VM's QEMU
process) plus the root-port fix. **Requires** the mdev UUID to be substituted and:
mkdir -p /tmp/sockets/<domain-uuid> # else QMP bind fails
rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock
Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's
`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex
Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads
`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed".
Behind a `pcie-root-port` QEMU leaves the capability alone.
## rmq_probe.c - legacy-ioctl RM control probe (guest or host)
Minimal RM client (client alloc + device alloc + NV_ESC_RM_CONTROL) to query
device/subdevice controls from userspace, e.g.
`0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`,
`0x800280 GET_NUM_SUBDEVICES`, `0x20800123 GPU_GET_ENGINES`.
NOTE: the device alloc currently fails (0x40 in guest, 0x1b on host) - the
device allocation sequence needs the proper NV0080_ALLOC_PARAMETERS handling
(or the NVOS64 path that tools/rmq/rmq.c uses); fix before relying on it.
## kret.sh - kretprobe attempt (blocked)
Tries to kretprobe the nvGpuOps*/nvUvmInterface* chain to read return values.
Blocked on this kernel: "trace_kprobe: Could not probe notrace function
<name> [nvidia]" - kprobes are not available for nvidia module text.
Kept for documentation.
## push_guest_tools.sh - rebuild the guest-side binaries from these sources
The guest's /tmp is wiped on boot; this pushes+compiles ptio/ge/rmqp/cdt into
the guest from the sources kept here. Run it after every VM start.
## guest_engines.c - client RM introspection (used with the ptrace tracer)
Queries the guest client RM for exactly the data UVM's register path uses:
virtualization mode (0x800289), subdevice count (0x800280), CLASS LIST V2
(0x800292), ENGINE LIST V2 (0x20800170), CE caps (0x20802a03) and CE->PCE
masks (0x20802a02). Needs the NVOS64 alloc path; see rmq_probe.c note.
## Guest-side UVM logging (critical for diagnosis!)
UVM's messages are silent until enabled:
echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints
Then run /tmp/cdt and `dmesg | grep nvidia-uvm`. This produced the current
blocker's exact message:
`nvidia-uvm: uvm_gpu.c:1574 init_gpu[pid:..] Failed to initialize the
channel manager: Call not supported [NV_ERR_NOT_SUPPORTED]`
(also useful: uvm_enable_debug_procfs, uvm_debug_enable_push_desc,
uvm_release_asserts, uvm_release_asserts_dump_stack)
## launch_qemu_rootport.sh / .run.sh - QEMU command line
Template (root-port fix) + runtime copy with the live mdev UUID, patched by
vmcycle.sh. Sockets still live under /tmp because scripts/vgpu_vm.py talks to
/tmp/qemu-guest-agent.sock by design.
## dbg_seq.sh / dbg_seq2.sh - control-sequence helpers (guest, needs ptio)
Print the RM control sequence immediately before/after the failing
UVM_REGISTER_GPU ioctl.
|
