1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
|
# CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive
> **Status (2026-09-23, honest).** The analysis covers the whole library for
> driver 570.144 and the tooling is version-agnostic. The *control plane* gets
> the stock guest stack through `cuInit`, device enumeration and into
> `cuCtxCreate`, which is blocked at `rc=999` because the emulated engine list
> is missing the async copy engine `COPY14`. The *data plane* is at the
> software memop bridge: the host executes the memsets, copies and semaphore
> releases it decodes from guest push buffers, and **no CUDA kernel has been
> forwarded yet**. Canonical detail: [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md)
> through `08`; reading order
> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md);
> front door [`../docs/README.md`](../docs/README.md).
## 1. What it is
`libcuda.so.1.1` is the userspace half of the CUDA driver: the library an
application or language runtime loads to talk to a GPU. For 570.144 it is a
**single, fully stripped, position-independent shared object of about 71 MB**:
| Property | Value |
|---|---|
| Exports | 659 (public `cu*` API plus internal/legacy entry points) |
| Imports | 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path |
| `.text` | `0x179ec0`–`0xe94c63` (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables) |
| `.rodata` | ~53 MB — error strings, dispatch/version tables, **embedded device code** |
| CUDA note | CUDA 12.8; supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` |
Two entry surfaces coexist: the **public driver API** (`cuInit`,
`cuCtxCreate_v2`, `cuMemAlloc_v2`, `cuLaunchKernel`, …) and the
**entry-table API** — `cuGetExportTable`, a 128-bit-UUID-keyed lookup that
hands internal tables to the CUDA runtime (`libcudart`), tooling and
interposition libraries. Tables are versioned by a leading `struct_size`
field and are append-only.
The library embeds **device code** — cubins for memcpy/memset kernels, the
graph scheduler, syscall trampolines and preemption-handler assembly — and
loads it through its own module machinery, so a stripped `libcuda.so` still
contains many small ELF images in `.rodata`. The CUDA *runtime* (`libcudart`,
the `cuda*` API) is a separate stock library that bottoms out in `libcuda`; it
also calls through the entry tables, so a guest failure in a `cuda*` call may
reach driver code through a different door than the public `cu*` entry point.
**Open-IOV requires this library unmodified.** In the guest it links against
the guest's stock `nvidia.ko` / `nvidia-uvm.ko` exactly as on bare metal; there
is no `LD_PRELOAD` shim and no patch. Instrumentation uses `ptrace` or
kernel-side tracing, because the RM client issues raw `syscall(SYS_ioctl)`
rather than libc `ioctl`.
## 2. Layer model
The library is strictly layered, and the boundaries are where the emulation
interfaces live (condensed from `docs/01` and `docs/02`):
```
application / libcudart
| public cu* API or UUID entry tables
v
generated wrappers -> API layer -> CUDA internal engine
|-- driver-model layer (RM) -> /dev/nvidia* ioctls
|-- hardware layer (QMD/method encoding)
| -> GPU queues, doorbells
|-- UVM userspace (managed memory) -> /dev/nvidia-uvm
'-- JIT/ELF (PTX->SASS, cubin loading)
kernel driver + GSP-RM + GPU
```
* **Generated wrappers and API layer.** A master export list (name, platform
flags, CUDA version introduced) generates the exported symbols, wrappers,
tool callbacks and `cuGetProcAddress` tables (`_vN`/`_ptsz` variants are part
of that data). Below it, the hand-written layer reads the per-thread context
stack, validates state, takes one context lock, calls one engine function and
returns a `CUresult` — the semantic work is always one layer down.
* **Internal engine.** Owns the object graph: process globals (init state,
kernel-client handle, device list), device (chip/caps/limits, vtables),
context (memory/channel/stream managers, QMD pools, ISR thread), stream,
module/function, memory objects, events/graphs/libraries.
* **Driver-model layer.** Selected at initialization behind one function
table: the **RM backend** (the one Open-IOV cares about) plus SoC, Amodel,
MPS and WDDM. It turns engine operations into RM controls and object
allocations over `/dev/nvidia*`, and owns the **driver-ABI adaptation**
layer — one instance per supported kernel-driver ABI, chosen at `cuInit`.
* **Hardware layer.** Chip-specific method/QMD/semaphore/doorbell encoders
selected by compute capability. The userspace driver *builds* the GPU
command streams; the kernel driver allocates, maps, schedules and services
interrupts.
* **Synchronization model.** Three primitives explain most of it: the
**tracking semaphore** (per-channel counter of completed pushes), the
**marker** ("all work before this point") and the **QMD semaphore**
(released when a grid completes). Blocking waits use a per-context ISR
thread and a DPC queue.
## 3. What it touches in a guest
This is the emulation contract; the full checklist is `docs/03` §9.
**Device nodes.** `/dev/nvidiactl` (process-wide control: version negotiation,
client creation, GPU attach, OS events, exported-object fds), `/dev/nvidiaN`
(per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation),
`/dev/nvidia-uvm` (managed memory and page migration) and `/dev/nvidia-vgpuN`
(vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths).
**ioctls.** The control device uses `_IOWR('F', cmd, …)`: `200`
`NV_ESC_CARD_INFO`, `210` `NV_ESC_CHECK_VERSION_STR`, `212`
`NV_ESC_ATTACH_GPUS_TO_FD`. RM escapes include `0x2b` `NV_ESC_RM_ALLOC`
(**the main object-allocation path**), `0x2a` `NV_ESC_RM_CONTROL`, `0x29`
`NV_ESC_RM_FREE`, `0x4e`/`0x4f` `NV_ESC_RM_MAP_MEMORY`/`UNMAP_MEMORY`,
`0x57`/`0x58` `NV_ESC_RM_MAP_MEMORY_DMA`/`UNMAP_MEMORY_DMA` and `0x4a`
`NV_ESC_RM_VID_HEAP_CONTROL`. The request word `rsi=0xc030462b` in a crash
dump decodes to "allocating an RM object with a 48-byte payload".
**RM objects/classes.** Every RM call names a class. Observed for 570.144 on
an emulated TU104: `NV01_ROOT` (`0x0`), `NV01_DEVICE_0` (`0x80`),
`NV20_SUBDEVICE_0` (`0x2080`), `NV01_EVENT` (`0x40`),
`KEPLER_CHANNEL_GROUP_A` (`0xa06c`), a channel-group/runlist object (`0x9067`),
`TURING_CHANNEL_GPFIFO_A` (`0xc46f`, the channel), `TURING_COMPUTE_A`
(`0xc5c0`, the compute engine object), plus memory (`0x3e`) and VA-space
classes. The observed `cuCtxCreate` order is root → device → subdevice →
events → channel group → channel-group object → memory → channel → control
`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` → compute object → engine registration.
**Controls.** Device/GPU info (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`,
`GPU_GET_VIRTUALIZATION_MODE`, `FB_GET_INFO`, `CE_GET_CAPS`, …), FIFO/channels
(`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` `0x80170d`, GPFIFO scheduling,
timeslices, `GET_CLASS_ENGINEID`), GR/context switches, memory, events and
interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates
unknown-but-successful controls; it only fails when a *specific reply field*
is consumed and wrong.
**Memory.** VA reservation (`NV_ESC_RM_VID_HEAP_CONTROL`/`NVOS32`), backing
allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and
mapping (`NVOS33` into the process, `NVOS46` into a device VA space). All
guest memory CUDA uses is either an RM allocation backed by emulated
VRAM/BAR1 or pinned guest RAM the GPU must DMA to.
**Channels, GPFIFO, USERD, doorbells.** A channel is a GPFIFO (array of
pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO,
error notifier and (Volta and later) the USERD itself, passing the handles to
the channel allocation; submission writes the GPFIFO put pointer into USERD
and rings a doorbell token returned by the channel allocation. Pushbuffers are
method streams terminated by a tracking-semaphore release and a new GPFIFO
entry; the host emulation decodes them as such.
**Synchronization below the API.** Semaphores (plain GPU-visible memory
values), 4 KB per-channel error notifiers, OS events (kernel objects used to
wake blocked threads), interrupts routed through the kernel driver, and a
trap handler for debuggers and `device-side assert`.
**UVM and GSP-RM.** When UVM is available the driver registers GPUs, VA
spaces, channels and managed allocations and pushes UVM semaphores into the
same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing
from userspace — it still issues RM ioctls; inside the kernel driver controls
become GSP-RM RPCs, which Open-IOV's host module answers.
## 4. Why forwarding matters, and where we are
The performance budget is dominated by **push-buffer execution**,
**synchronization latency** and (once, at setup) **control-plane latency**
(`docs/04` §4). Only forwarding *real work* to the physical GPU meets the
first two; software execution of state/memory ops is a correctness bridge, not
the data plane (`docs/08`).
* **Stated architecture:** *emulate the interface, pass through the work,
never patch anyone.* Prototype the host-RM/GSP proxy first for correctness,
measure, then move the hot path to direct submission if the budgets demand
it — work path abstracted from day one.
* **Control plane today:** GSP-RM emulation, RPC handling, BAR/PCI/MSI-X
emulation and object/control replies are implemented; the stock guest stack
gets through `cuInit` and device enumeration into `cuCtxCreate`.
* **Current blocker:** `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`)
because the class-filtered engine list is missing the async copy engine
`COPY14` (descriptor `0x793ceb0e`, instance 14, client type `0x38`, class
`0xc9b5`). The oracle list has 7 engines (GR, CE0..CE4, `COPY14`); the guest
list has 6. The driver silently skips engines with no per-(class, instance)
database entry, then reads a zeroed secondary record and rolls back; what
builds that database entry is the open question (`docs/05` §3.6–3.7).
* **Data plane today:** the host module decodes guest push buffers as method
streams and **executes the small work it understands itself** — memsets,
copies and semaphore releases. **No CUDA kernel has been forwarded yet.**
## 5. The crash-path lesson: replies are consumed data
The old deterministic `cuCtxCreate` segfault at `libcuda+0x453f90` shows why
reply coherence matters (`docs/05` §3.5, `docs/04` §6). The engine-registration
hook allocates a `TURING_COMPUTE_A` object with `pAllocParms` pointing at the
caller's own **8-byte stack local** and `paramsSize = 0`; the guest driver does
not honour `paramsSize`, so it reads **16-byte** class params starting there
(the upper 8 bytes are the hook's saved `rbp`) and copies 16 bytes of reply
params back over the same buffer. With the old zeroed reply that copy-back
zeroed the saved `rbp`; the function's epilogue then loaded `rbp = 0` and its
next access — reloading a pointer from `[rbp-0x38]` — faulted at
`0xffffffffffffffc8` (the store was the kernel's `copy_to_user`).
**The fix:** echo the request's params back, forcing only `status` to `NV_OK`;
the copy-back then writes exactly the bytes it read, and the fault disappears —
the same thing the real GSP-RM does for these in/out params.
**The general rule:** the userspace library is a *consumer of emulated data*.
A wrong reply is worse than a slow one. Work up the reply-fidelity ladder —
status-only → coherent zeroed params → coherent values for consumed fields →
exact values — and validate each rung with the A/B oracle.
## 6. How to look things up
The toolchain is version-agnostic (`docs/06`):
```
tools/cudatool.py info|exports|functions|nearest|crash|delta <lib>
tools/callwalk.py <callgraph.tsv> callers|callees|chain ...
```
* **`cudatool.py`** — build id/sections/exports/ABI note (`info`), symbol map
(`exports`), function ranges from `.eh_frame_hdr` (`functions`), containing
function and nearest export for an address (`nearest`), dmesg `ip` to file
vaddr (`crash`), and an interface-level diff (`delta`).
* **`callwalk.py`** — caller/callee trees and shortest call paths over the
callgraph index.
* **Annotated corpus** — `570.144/annotations/names.tsv` (our name map; exports
exact, internals grow over time) and `subsystems.tsv`; corpus headers carry
the recovered name, file address and direct callers/callees. The generated corpus
is not distributed.
**Address convention (read this before citing an offset).** Public docs cite
**file virtual addresses** as `libcuda+0x...`. Ghidra loads these binaries at
image base `0x100000`, so `program address = file vaddr + 0x100000`; the
`tools/` wrappers translate automatically, raw Ghidra scripts do not. An
earlier analysis passed file vaddrs straight to Ghidra and decompiled functions
0x100000 too low, so its decompiled-code conclusions were about the wrong
functions (its live traces stayed valid). The same applies when transferring a
name map to a new driver version.
## 7. Where the detail lives
| Document | What it gives you |
|---|---|
| [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | the reading order, fast recipes, and the two findings that change how you debug |
| [`../userspace/cuda/docs/01-architecture.md`](../userspace/cuda/docs/01-architecture.md) | the full layer/object/sync/memory model |
| [`../userspace/cuda/docs/02-build-and-execution.md`](../userspace/cuda/docs/02-build-and-execution.md) | build model, embedded device code, init/threading, what it touches |
| [`../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../userspace/cuda/docs/03-driver-and-gpu-interface.md) | the emulation contract: nodes, ioctls, classes, controls, channels, UVM |
| [`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) | NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security |
| [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md) | the binary map, the crash resolution, and the current engine-topology blocker |
| [`../userspace/cuda/docs/06-tooling.md`](../userspace/cuda/docs/06-tooling.md) | every tool, its usage and its cost |
| [`../userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) | re-running the analysis on a new driver version |
| [`../userspace/cuda/docs/08-virtualization-approaches.md`](../userspace/cuda/docs/08-virtualization-approaches.md) | forwarding options and the honest current stage |
| [`../docs/README.md`](../docs/README.md) | the project documentation front door |
|
