# libcuda 570.144 `cuCtxCreate` segfault - analysis (2026-09-23)

> **Status: historical (2026-09-23 PART 9).** The segfault described here was
> fixed in PART 10 (the RM allocation reply now echoes the request parameters,
> which prevents a 16-byte class-params copy-back from zeroing the caller's
> saved frame pointer). The engine-topology blocker that followed was also
> since resolved (the async copy engine is included); the current frontier is
> the completion/execution path. For the corrected,
> current analysis see
> [`../../docs/05-570.144-decompilation.md`](../../docs/05-570.144-decompilation.md)
> and [`../../READ-ME-FIRST.md`](../../READ-ME-FIRST.md). The runtime
> observations below (gdb traces, ioctl sequences) remain valid, and note that
> the decompiled files it references were produced at the wrong Ghidra image
> base (see PART 9's own address note and `../../docs/05` section 1.1).


First analysis note for the CUDA userspace driver under Open-IOV. Companion to
`../README.md` (tools) and the host/guest A/B harness (`tools/cuda-ab/`).

## Status summary

| Phase | Result |
|---|---|
| `cuInit(0)` | **OK** (previous sessions: failed) |
| `cuDeviceGetCount` / `cuDeviceGet` | **OK** (device 0) |
| `cuCtxCreate` | **SIGSEGV inside libcuda** - the blocker at this snapshot (since fixed) |
| `cuMemAlloc` / copies / `ALL OK` | not reached yet |

Guest: Ubuntu 24.04, driver **570.144** (open kernel modules + libcuda
570.144), emulated TU104 `10de:1ed3` via `vgsp.ko`.
Host oracle: RTX 2080 Super (cc 7.5), driver 570.211.01, CUDA toolkit 12.0.

## The crash, precisely

Guest kernel log:

```
cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90 ... error 5
           in libcuda.so.570.144[735cfcb79000+d1c000]
```

* fault address `0xffffffffffffffc8` = **NULL - 0x38**
* file vaddr of the fault: `0x453f90` (dmesg's `[...]` base is the first
  *executable* mapping, not the load base - see `scripts/`; the ELF exec
  segment starts at vaddr `0x179000`, so the offset must be corrected:
  `vaddr = ip - mapping_base + 0x179000`... here it lands in function
  range `0x453ef0..0x45401b` found in `.eh_frame`).
* gdb registers at the fault: `rbp=0`, `rax=0`, `rbx=0x7ffff1035010`,
  `rsi=0xc030462b` (an NV ioctl request = `_IOWR('F', 0x2b, 48)` =
  `NV_ESC_RM_ALLOC`), `rdi=0xffffffff`.
* The faulting instruction reloads a stack slot relative to `rbp`:
  `mov rax, [rbp-0x38]` - with `rbp = 0` this reads NULL-0x38.

So a callee in the call chain ended up with a **clobbered/zero frame
pointer**; the function expects `rbp` to be its frame.

## What libcuda is doing when it crashes (from the ioctl trace)

`tools/guest-diag/ioctl_trace.c` (ptrace; libcuda uses raw syscalls) shows
`cuCtxCreate` performing, in order:

```
alloc class=0x0      (NV01_ROOT)
alloc class=0x80     (NV01_DEVICE_0)
alloc class=0x2080   (NV20_SUBDEVICE_0)
alloc class=0x40     (NV01_EVENT x several)
alloc class=0xa06c   (KEPLER_CHANNEL_GROUP_A)
alloc class=0x9067   (channel group / runlist object)
alloc class=0x3e     (NV01_MEMORY_*)
alloc class=0xc46f   (TURING_CHANNEL_GPFIFO_A)          <- the channel
control cmd=0x80170d (NV0080_CTRL_FIFO_GET_CHANNELLIST) <- driver-local
alloc class=0xc5c0   (TURING_COMPUTE_A)                 <- last alloc
[crash]
```

Host-side GSP-RM emulation log for the same window:

```
CE_GET_FAULT_METHOD_BUFFER_SIZE -> 12288          (handled)
GPU_PROMOTE_CTX (0x2080012b, 560 B) -> SUCCESS    (generic fallback)
TURING_CHANNEL_GPFIFO_A alloc, TURING_COMPUTE_A alloc
```

## The crashing function (Ghidra, full-analysis project)

`FUN_00453ef0` (disassembly is authoritative; Ghidra also flags it as using
the caller's frame - i.e. a local subroutine of the enclosing FDE range
`0x453ef0..0x45401b`):

```
rdi = object, rsi = per-engine descriptor (from caller)
r14d = [rsi]; if (r14d == 1) r14d = 0
rax = [[rdi]+8]; rax = [rax+0x40]; [rbp-0x38] = rax     (spill a table)
call 0x3a9c20(global 4413d60)                           (returns eax)
if (eax != 0) {
    rax = global 4413ed8; rax = [rax+0x40]
    rsi = [rdi+0x9488]; esi = *rsi
    edi = global 4412f40; edx = r15d (the status); rcx = descriptor
    call [rax+0x530]                                    (indirect: log hook)
    if (eax == 0) goto 0x453f90                         (crash site)
    ...
}
```

Callers: only two, both in `FUN_0045a670` (`0x45afc1` and `0x45b02b`),
which walks engines and calls this helper per engine with
`rsi = device + 0x2fec`, `device + 0x300c + (idx-1)*0x14`, ... - i.e. it is
a **per-engine context check/report** path.

## Current hypothesis

The crash happens *after* an engine-related check failed: the helper calls a
global logging/dispatch hook (`call [rax+0x530]`) and the frame is corrupted
on the way back, or the hook pointer itself is not a valid function because
the dispatch table was never (fully) initialised in this environment. Two
sub-hypotheses, both testable:

1. **Emulation data gap**: our GSP static info advertises 11 engines (GR,
   5x CE, 2x NVDEC, NVENC, SEC2, NVJPG) but only the CE caps are emulated;
   a real GSP-RM would let libcuda/driver bring each engine up (or fail in a
   handled way). If the per-engine status/descriptor data we provide is
   inconsistent, libcuda takes this error path.
2. **Log hook table**: the hook reached through two global pointer
   dereferences (`[global+0x40]+0x530`) is not initialised, so the "callee"
   is not a real function and clobbers `rbp`.

`GPU_PROMOTE_CTX` returning generic-SUCCESS is also suspicious (on TU104 with
no NVLink/vGPU this may be expected to fail in a specific way); it is the
last *control* before the crash window.

## Register trace (gdb, breakpoints in the live guest)

Sequence observed (single helper invocation):

```
HELPER-ENTRY   rbp=0x7fffffffdbf0 rsp=0x7fffffffd5d8 rax=1
HOOK-ENTRY     rbp=0x7fffffffd5d0 rsp=0x7fffffffd588 rax=<heap ptr>
HOOK-LEAVE-OK  rbp=0x7fffffffd580 rsp=0x7fffffffd570 rax=0
AFTER-HOOK     rbp=0x0            rsp=0x7fffffffd590 rax=0   <- return from hook
CRASH          rbp=0x0            rsp=0x7fffffffd590
```

* The helper entered with a valid frame, and the hook's own frame is also
  valid (`push rbp; mov rbp,rsp` ... `leave; ret` - the hook is at file
  vaddr `0x4676f0` and does **not** by itself clobber rbp).
* Yet immediately after the hook returns (`0x453f65`), **rbp is 0**. The
  saved frame pointer on the hook's stack must therefore have been
  overwritten inside the hook call chain, or the hook returned through a
  different path than its `leave`.
* The hook calls `0x481360` with `ecx = [descriptor+0xc]` (an engine class/id
  switch over values 0x80, 0x79, 0x5, ...) and `r8 = &local`; that function
  also has a normal frame. The descriptor is built by the caller from
  `device+0x2fec` / `device+0x300c+(idx-1)*0x14`.
* Table values read live: `[global 4413ed8] = 0x7ffff7896d80`,
  `[[global+0x40]] = 0x7ffff3a5f6c0` (libcuda+0x45f6c0),
  `[[[[global]+0x40]+0x530]] = 0x7ffff3a676f0` (libcuda+0x4676f0, the hook
  above) - i.e. the dispatch table *is* populated with real libcuda code.

So the immediate mechanism is: **the engine-check path runs with an
unexpected engine descriptor/class and loses the caller's frame while
reporting it**. The engine descriptor contents (and what the emulation
advertised for that engine) are the next thing to inspect.

## The engine-list path (summary)

The engine-list construction path for this library is summarised here:

* libcuda builds an **async copy-engine list** from per-engine records
  (engine kind, runlist, class) it receives with the device information.
* The device information is derived from the GSP static info engine table
  this project emulates (`tu104_engine_table`: 11 hardware engines,
  runlists 0=GR, 1..5=CE0..4, 6..7=NVDEC, 8=NVENC, 9=SEC2, 10=NVJPG).
* Only copy-engine records with a positive runlist become async channels;
  the runlist value is reused when a channel is created (0 for GR).

## What is NOT (yet) analysed

* No full auto-analysis output was reviewed beyond the functions above;
  the Ghidra project exists (`run/ghidra-proj`) for cheap further
  decompilation (`scripts/ghidra_decompile_project.sh <vaddr>`).
* The engine descriptor contents at the hook (`[descriptor+0x0/0x8/0xc]`)
  were not yet dumped and compared with the expected engine table semantics.
* No comparison of our GSP static info engine table against a real TU104
  vGPU dump yet.

## Reproduction

```
tools/guest-diag/vmcycle.sh
tools/guest-diag/push_guest_tools.sh
# in guest: sudo modprobe nvidia_uvm; setsid /tmp/cdt >/tmp/cdt.out 2>&1 &
```

`/tmp/cdt` prints `-> <call>` before each libcuda entry point (see
`tools/test/cuda_drv_test.c`), so the last marker identifies the crashing
call. Add `strace`-like ioctl detail with `/tmp/iot /tmp/cdt`.