1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
# libcuda 570.144 `cuCtxCreate` segfault - analysis (2026-09-23)
First analysis note for the CUDA userspace driver under Open-IOV. Companion to
`../README.md` (tools) and the host/guest A/B harness (`tools/cuda-ab/`).
## Status summary
| Phase | Result |
|---|---|
| `cuInit(0)` | **OK** (previous sessions: failed) |
| `cuDeviceGetCount` / `cuDeviceGet` | **OK** (device 0) |
| `cuCtxCreate` | **SIGSEGV inside libcuda** - the current blocker |
| `cuMemAlloc` / copies / `ALL OK` | not reached yet |
Guest: Ubuntu 24.04, driver **570.144** (open kernel modules + libcuda
570.144), emulated TU104 `10de:1ed3` via `open-iov-vfio.ko`.
Host oracle: RTX 2080 Super (cc 7.5), driver 570.211.01, CUDA toolkit 12.0.
## The crash, precisely
Guest kernel log:
```
cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90 ... error 5
in libcuda.so.570.144[735cfcb79000+d1c000]
```
* fault address `0xffffffffffffffc8` = **NULL - 0x38**
* file vaddr of the fault: `0x453f90` (dmesg's `[...]` base is the first
*executable* mapping, not the load base - see `scripts/`; the ELF exec
segment starts at vaddr `0x179000`, so the offset must be corrected:
`vaddr = ip - mapping_base + 0x179000`... here it lands in function
range `0x453ef0..0x45401b` found in `.eh_frame`).
* gdb registers at the fault: `rbp=0`, `rax=0`, `rbx=0x7ffff1035010`,
`rsi=0xc030462b` (an NV ioctl request = `_IOWR('F', 0x2b, 48)` =
`NV_ESC_RM_ALLOC`), `rdi=0xffffffff`.
* The faulting instruction reloads a stack slot relative to `rbp`:
`mov rax, [rbp-0x38]` - with `rbp = 0` this reads NULL-0x38.
So a callee in the call chain ended up with a **clobbered/zero frame
pointer**; the function expects `rbp` to be its frame.
## What libcuda is doing when it crashes (from the ioctl trace)
`tools/guest-diag/ioctl_trace.c` (ptrace; libcuda uses raw syscalls) shows
`cuCtxCreate` performing, in order:
```
alloc class=0x0 (NV01_ROOT)
alloc class=0x80 (NV01_DEVICE_0)
alloc class=0x2080 (NV20_SUBDEVICE_0)
alloc class=0x40 (NV01_EVENT x several)
alloc class=0xa06c (KEPLER_CHANNEL_GROUP_A)
alloc class=0x9067 (channel group / runlist object)
alloc class=0x3e (NV01_MEMORY_*)
alloc class=0xc46f (TURING_CHANNEL_GPFIFO_A) <- the channel
control cmd=0x80170d (NV0080_CTRL_FIFO_GET_CHANNELLIST) <- driver-local
alloc class=0xc5c0 (TURING_COMPUTE_A) <- last alloc
[crash]
```
Host-side GSP-RM emulation log for the same window:
```
CE_GET_FAULT_METHOD_BUFFER_SIZE -> 12288 (handled)
GPU_PROMOTE_CTX (0x2080012b, 560 B) -> SUCCESS (generic fallback)
TURING_CHANNEL_GPFIFO_A alloc, TURING_COMPUTE_A alloc
```
## The crashing function (Ghidra, full-analysis project)
`FUN_00453ef0` (disassembly is authoritative; Ghidra also flags it as using
the caller's frame - i.e. a local subroutine of the enclosing FDE range
`0x453ef0..0x45401b`):
```
rdi = object, rsi = per-engine descriptor (from caller)
r14d = [rsi]; if (r14d == 1) r14d = 0
rax = [[rdi]+8]; rax = [rax+0x40]; [rbp-0x38] = rax (spill a table)
call 0x3a9c20(global 4413d60) (returns eax)
if (eax != 0) {
rax = global 4413ed8; rax = [rax+0x40]
rsi = [rdi+0x9488]; esi = *rsi
edi = global 4412f40; edx = r15d (the status); rcx = descriptor
call [rax+0x530] (indirect: log hook)
if (eax == 0) goto 0x453f90 (crash site)
...
}
```
Callers: only two, both in `FUN_0045a670` (`0x45afc1` and `0x45b02b`),
which walks engines and calls this helper per engine with
`rsi = device + 0x2fec`, `device + 0x300c + (idx-1)*0x14`, ... - i.e. it is
a **per-engine context check/report** path.
## Current hypothesis
The crash happens *after* an engine-related check failed: the helper calls a
global logging/dispatch hook (`call [rax+0x530]`) and the frame is corrupted
on the way back, or the hook pointer itself is not a valid function because
the dispatch table was never (fully) initialised in this environment. Two
sub-hypotheses, both testable:
1. **Emulation data gap**: our GSP static info advertises 11 engines (GR,
5x CE, 2x NVDEC, NVENC, SEC2, NVJPG) but only the CE caps are emulated;
a real GSP-RM would let libcuda/driver bring each engine up (or fail in a
handled way). If the per-engine status/descriptor data we provide is
inconsistent, libcuda takes this error path.
2. **Log hook table**: the hook reached through two global pointer
dereferences (`[global+0x40]+0x530`) is not initialised, so the "callee"
is not a real function and clobbers `rbp`.
`GPU_PROMOTE_CTX` returning generic-SUCCESS is also suspicious (on TU104 with
no NVLink/vGPU this may be expected to fail in a specific way); it is the
last *control* before the crash window.
## Register trace (gdb, breakpoints in the live guest)
Sequence observed (single helper invocation):
```
HELPER-ENTRY rbp=0x7fffffffdbf0 rsp=0x7fffffffd5d8 rax=1
HOOK-ENTRY rbp=0x7fffffffd5d0 rsp=0x7fffffffd588 rax=<heap ptr>
HOOK-LEAVE-OK rbp=0x7fffffffd580 rsp=0x7fffffffd570 rax=0
AFTER-HOOK rbp=0x0 rsp=0x7fffffffd590 rax=0 <- return from hook
CRASH rbp=0x0 rsp=0x7fffffffd590
```
* The helper entered with a valid frame, and the hook's own frame is also
valid (`push rbp; mov rbp,rsp` ... `leave; ret` - the hook is at file
vaddr `0x4676f0` and does **not** by itself clobber rbp).
* Yet immediately after the hook returns (`0x453f65`), **rbp is 0**. The
saved frame pointer on the hook's stack must therefore have been
overwritten inside the hook call chain, or the hook returned through a
different path than its `leave`.
* The hook calls `0x481360` with `ecx = [descriptor+0xc]` (an engine class/id
switch over values 0x80, 0x79, 0x5, ...) and `r8 = &local`; that function
also has a normal frame. The descriptor is built by the caller from
`device+0x2fec` / `device+0x300c+(idx-1)*0x14`.
* Table values read live: `[global 4413ed8] = 0x7ffff7896d80`,
`[[global+0x40]] = 0x7ffff3a5f6c0` (libcuda+0x45f6c0),
`[[[[global]+0x40]+0x530]] = 0x7ffff3a676f0` (libcuda+0x4676f0, the hook
above) - i.e. the dispatch table *is* populated with real libcuda code.
So the immediate mechanism is: **the engine-check path runs with an
unexpected engine descriptor/class and loses the caller's frame while
reporting it**. The engine descriptor contents (and what the emulation
advertised for that engine) are the next thing to inspect.
## What is NOT (yet) analysed
* No full auto-analysis output was reviewed beyond the functions above;
the Ghidra project exists (`run/ghidra-proj`) for cheap further
decompilation (`scripts/ghidra_decompile_project.sh <vaddr>`).
* The engine descriptor contents at the hook (`[descriptor+0x0/0x8/0xc]`)
were not yet dumped and compared with the expected engine table semantics.
* No comparison of our GSP static info engine table against a real TU104
vGPU dump yet.
## Reproduction
```
tools/guest-diag/vmcycle.sh
tools/guest-diag/push_guest_tools.sh
# in guest: sudo modprobe nvidia_uvm; setsid /tmp/cdt >/tmp/cdt.out 2>&1 &
```
`/tmp/cdt` prints `-> <call>` before each libcuda entry point (see
`tools/test/cuda_drv_test.c`), so the last marker identifies the crashing
call. Add `strace`-like ioctl detail with `/tmp/iot /tmp/cdt`.
|
