1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
# 02 — How the CUDA userspace driver is built and how it executes
This document explains the compilation model behind `libcuda.so` and the
runtime model of the resulting binary: what is generated, what is compiled in,
what the loader does, and what the driver does from process start to first
kernel launch. It is written to be re-derivable on any driver version with the
tooling in `tools/`.
---
## 1. Build model
### 1.1 One library, many components
The driver is one shared object assembled from several thousand translation
units. In architecture terms the inputs are:
| Input group | Nature | Notes |
|---|---|---|
| API wrappers | **generated** C | one exported function per API name/version variant; adds destroyed-state check, optional serialization, tracing/interposition dispatch |
| API implementation | hand-written C | one module per API domain |
| CUDA internal engine | hand-written C | the bulk of the driver |
| Driver-model backends | hand-written C/C++ | RM/SoC/simulator/MPS/WDDM behind one function-table interface |
| Hardware layer | hand-written C | per-architecture method/QMD encoders |
| Device code | **assembly and CUDA C** compiled to cubins, embedded as byte arrays | memcpy/memset kernels, syscall trampolines, trap/exit handlers, preemption, graph scheduler |
| Export lists | text | symbol name + platform flags + the CUDA version the symbol appeared in |
| ABI adaptation | **one compiled instance per kernel-driver ABI** | same source compiled repeatedly with different ABI headers and symbol localization |
| Generators | Perl/Python/Java | produce wrappers, proc-address tables, entry tables, interposition tables, error tables, abstraction-layer headers |
Two consequences matter for analysis:
* **The public ABI is data-driven.** Adding an API function is mostly an entry
in a list plus a hand-written implementation; the exported symbol,
the tracing callback ID, the `cuGetProcAddress` entry and the interposition
slot are generated from that list. A new driver version usually moves this
data, not the architecture.
* **Versioning is explicit.** Symbols carry `_vN` suffixes and a
"since CUDA X.Y" attribute; per-thread default stream variants (`_ptsz`)
exist for many calls; `cuGetProcAddress` selects by requested version.
### 1.2 Embedded device code
The library carries its own GPU code. Device assembly sources are preprocessed,
assembled per architecture, converted to C arrays and linked into the library;
CUDA C device sources are compiled with the toolchain's device compiler. At
runtime the driver treats them like user modules: it extracts the image for
the current architecture and loads it through its own module machinery. This
is why a stripped `libcuda.so` still contains many small ELF images and
`.nv.*` section names in `.rodata` — those are cubins, not host code.
### 1.3 Build flavors
* `release` separates debug info into a `.debug` file (which NVIDIA does not
ship in the driver packages we analyse, hence the fully stripped binary).
* `develop`/`debug` add logging and assertions; some entry tables (test
interfaces) are present only outside public-release builds.
* A "safety" configuration removes the tooling dispatch entirely (exports call
the implementation directly).
* The kernel-module build of the same sources exists but is not what runs in
a guest.
### 1.4 What the produced binary looks like
For 570.144 (`cudatool.py info`):
```
build id : c0e1e77ea249944ac4fd43faebed9498a729606a
exec segment base: 0x179000
.text : 0x179ec0 .. 0xe94c64 (13,741,476 bytes)
exports : 659 imports: 243 (libc/libpthread/libdl/librt)
cuda note : CUDA 12.8, driver ABI [470,535,550,560,565,570], device [1,2,7..14]
debuglink : libcuda.so.1.1.debug (not shipped)
```
Sections of interest:
| Section | Why it matters |
|---|---|
| `.dynsym`/`.dynstr` | the 659 exports and 243 imports; no local symbols (stripped) |
| `.text` | all host code |
| `.rodata` (53 MB) | error strings, dispatch/version tables, **embedded cubins** |
| `.data.rel.ro` (~900 KB) | relocated pointer tables (entry tables, vtables, function tables) |
| `.eh_frame`/`.eh_frame_hdr` | unwind info; also the best stripped-binary function map (28,925 FDEs) |
| `.note.cuda.fwd_compatibility` | CUDA version + supported kernel-driver ABI list |
| `.gnu_debuglink` | name/CRC of the (absent) debug file |
## 2. Execution model
### 2.1 Load and initialization
1. The dynamic loader maps the four PT_LOAD segments and relocates
`.rela.dyn`/`.rela.plt`. The library has no NVIDIA userspace dependencies
for the CUDA path; everything below it is a device file.
2. C++-style constructors registered in `.init_array` run. They set up global
mutexes and install an `atexit`/destructor that marks the driver destroyed.
3. The first `cuInit` runs the initialization state machine:
* initialize TLS and globals; select the driver-model backend;
* open the kernel driver's control device and create the process-wide
**kernel client** (one per process, not per context);
* enumerate and attach GPUs, build the device list (chip, caps, limits,
engine lists, memory sizes, BAR info);
* initialize the UVM userspace layer if the platform supports it;
* load embedded device code caches lazily.
4. Subsequent API calls use the per-thread context stack; the first
`cuCtxCreate` builds the heavy per-context state.
An `LD_PRELOAD` shim cannot observe the driver's kernel traffic: the RM client
issues **raw `syscall(SYS_ioctl)`**, not libc `ioctl`. Instrumentation is done
with `ptrace` (see `tools/guest-diag/ioctl_trace.c` in the parent tree) or
with kernel-side tracing.
### 2.2 Interposition and tooling dispatch
Every exported call goes through a global function table before reaching the
implementation. A tool library can register an interposition layer *before*
`cuInit` and receive the original pointers to chain to. This is the sanctioned
mechanism used by profilers/debuggers, and it is also a clean hook point for
A/B instrumentation in the guest.
### 2.3 Entry tables
`cuGetExportTable(uuid)` returns pointers to internal tables for:
* the CUDA runtime library (fatbin loading, primary-context initialization,
hidden module-load paths, kernel launch with internal parameters);
* tooling (profiling, debugging, memory checking, launch tracking);
* debugger support (context-local storage, trap handlers);
* interop and special platforms (video, NVN, OpenCL bridges).
Tables are identified by 128-bit UUIDs and start with a size field, so old
clients and new drivers coexist. This is a key fact for emulation: the guest
runtime/tools may call *through these tables* instead of the public API, and
all of it still resolves to the same internal engine.
### 2.4 Threading and process model
* API calls are caller-threaded; each context has a recursive RW lock.
* One ISR thread per context handles interrupts/DPC callbacks.
* One boost thread per context handles drop-to-idle clock policy.
* MPS (multi-process service) inserts a client/server IPC layer *below* the
normal API: clients share the server's channels; the wire format is a
private socket protocol, not the public API.
* Inter-process memory sharing uses kernel-driver exported handles and
(optionally) POSIX file descriptors; CUDA's own IPC mem handles are an RM
object plus a key.
### 2.5 What the driver touches at runtime (overview)
| Resource | Access | Purpose |
|---|---|---|
| `/dev/nvidiactl` | ioctl | control device: version negotiation, client creation, GPU attach, OS events, exported-object fds |
| `/dev/nvidiaN` | open/ioctl/mmap | per-GPU: object alloc/free, controls, memory alloc/map, interrupts |
| `/dev/nvidia-uvm` | open/ioctl | managed memory, page migration, channel registration (via UVM userspace API) |
| BAR regions | mmap | user-mode doorbells (USERD), interrupt tree/usermode region, framebuffer apertures |
| sysmem | mmap/alloc | command buffers, semaphores, page tables (when the driver owns them) |
| shared memory / sockets | IPC | MPS, IPC events/memory |
The exact ioctl vocabulary observed from the 570.144 guest is enumerated in
`docs/03-driver-and-gpu-interface.md`.
## 3. Running in a guest (what Open-IOV sees)
* The library is **unmodified** in the guest; it links against the guest's
kernel driver (`nvidia.ko` + `nvidia-uvm.ko`) exactly as on bare metal.
* The kernel driver is also unmodified; it talks to the emulated PCI function
and the GSP-RM interface that Open-IOV emulates.
* Therefore, from libcuda's point of view, the guest is a normal NVIDIA
machine whose kernel driver has certain capabilities and whose GPU answers
RM controls. The crash class we chase (`cuCtxCreate` segfault) is a
*userspace* consequence of *kernel/GSP* replies that differ from bare metal.
* The library's own state is therefore the best available oracle: when the
driver misbehaves, comparing its state (or its control traffic) against a
bare-metal A/B run localizes the emulation gap. The A/B harness in
`tools/cuda-ab/` does this at the workload level; `docs/05` does it at the
function level.
## 4. Reproducing the build/execution analysis for a new version
```
tools/cudatool.py info <libcuda.so.NEW> # size, exports, ABI note
tools/cudatool.py exports <libcuda.so.NEW> # symbol map
tools/cudatool.py functions <libcuda.so.NEW> # FDE function map
tools/cudatool.py delta <old> <new> # interface-level diff
tools/ghidra_full_analysis.sh <libcuda.so.NEW> # full auto-analysis
tools/ghidra_export_index.sh ... # function/call/string index
tools/ghidra_batch_decompile.sh ... # decompiled corpus
```
See `docs/06-tooling.md` for details and `docs/07-version-upgrade-playbook.md`
for the full procedure.
|
