# 02 — How the CUDA userspace driver is built and how it executes

This document explains the compilation model behind `libcuda.so` and the
runtime model of the resulting binary: what is generated, what is compiled in,
what the loader does, and what the driver does from process start to first
kernel launch. It is written to be re-derivable on any driver version with the
tooling in `tools/`.

---

## 1. Build model

### 1.1 One library, many components

The driver is one shared object assembled from several thousand translation
units. In architecture terms the inputs are:

| Input group | Nature | Notes |
|---|---|---|
| API wrappers | **generated** C | one exported function per API name/version variant; adds destroyed-state check, optional serialization, tracing/interposition dispatch |
| API implementation | hand-written C | one module per API domain |
| CUDA internal engine | hand-written C | the bulk of the driver |
| Driver-model backends | hand-written C/C++ | RM/SoC/simulator/MPS/WDDM behind one function-table interface |
| Hardware layer | hand-written C | per-architecture method/QMD encoders |
| Device code | **assembly and CUDA C** compiled to cubins, embedded as byte arrays | memcpy/memset kernels, syscall trampolines, trap/exit handlers, preemption, graph scheduler |
| Export lists | text | symbol name + platform flags + the CUDA version the symbol appeared in |
| ABI adaptation | **one compiled instance per kernel-driver ABI** | same source compiled repeatedly with different ABI headers and symbol localization |
| Generators | Perl/Python/Java | produce wrappers, proc-address tables, entry tables, interposition tables, error tables, abstraction-layer headers |

Two consequences matter for analysis:

* **The public ABI is data-driven.** Adding an API function is mostly an entry
  in a list plus a hand-written implementation; the exported symbol,
  the tracing callback ID, the `cuGetProcAddress` entry and the interposition
  slot are generated from that list. A new driver version usually moves this
  data, not the architecture.
* **Versioning is explicit.** Symbols carry `_vN` suffixes and a
  "since CUDA X.Y" attribute; per-thread default stream variants (`_ptsz`)
  exist for many calls; `cuGetProcAddress` selects by requested version.

### 1.2 Embedded device code

The library carries its own GPU code. Device assembly sources are preprocessed,
assembled per architecture, converted to C arrays and linked into the library;
CUDA C device sources are compiled with the toolchain's device compiler. At
runtime the driver treats them like user modules: it extracts the image for
the current architecture and loads it through its own module machinery. This
is why a stripped `libcuda.so` still contains many small ELF images and
`.nv.*` section names in `.rodata` — those are cubins, not host code.

### 1.3 Build flavors

* `release` separates debug info into a `.debug` file (which NVIDIA does not
  ship in the driver packages we analyse, hence the fully stripped binary).
* `develop`/`debug` add logging and assertions; some entry tables (test
  interfaces) are present only outside public-release builds.
* A "safety" configuration removes the tooling dispatch entirely (exports call
  the implementation directly).
* The kernel-module build of the same sources exists but is not what runs in
  a guest.

### 1.4 What the produced binary looks like

For 570.144 (`cudatool.py info`):

```
build id        : c0e1e77ea249944ac4fd43faebed9498a729606a
exec segment base: 0x179000
.text           : 0x179ec0 .. 0xe94c64 (13,741,476 bytes)
exports         : 659        imports: 243 (libc/libpthread/libdl/librt)
cuda note       : CUDA 12.8, driver ABI [470,535,550,560,565,570], device [1,2,7..14]
debuglink       : libcuda.so.1.1.debug (not shipped)
```

Sections of interest:

| Section | Why it matters |
|---|---|
| `.dynsym`/`.dynstr` | the 659 exports and 243 imports; no local symbols (stripped) |
| `.text` | all host code |
| `.rodata` (53 MB) | error strings, dispatch/version tables, **embedded cubins** |
| `.data.rel.ro` (~900 KB) | relocated pointer tables (entry tables, vtables, function tables) |
| `.eh_frame`/`.eh_frame_hdr` | unwind info; also the best stripped-binary function map (28,925 FDEs) |
| `.note.cuda.fwd_compatibility` | CUDA version + supported kernel-driver ABI list |
| `.gnu_debuglink` | name/CRC of the (absent) debug file |

## 2. Execution model

### 2.1 Load and initialization

1. The dynamic loader maps the four PT_LOAD segments and relocates
   `.rela.dyn`/`.rela.plt`. The library has no NVIDIA userspace dependencies
   for the CUDA path; everything below it is a device file.
2. C++-style constructors registered in `.init_array` run. They set up global
   mutexes and install an `atexit`/destructor that marks the driver destroyed.
3. The first `cuInit` runs the initialization state machine:
   * initialize TLS and globals; select the driver-model backend;
   * open the kernel driver's control device and create the process-wide
     **kernel client** (one per process, not per context);
   * enumerate and attach GPUs, build the device list (chip, caps, limits,
     engine lists, memory sizes, BAR info);
   * initialize the UVM userspace layer if the platform supports it;
   * load embedded device code caches lazily.
4. Subsequent API calls use the per-thread context stack; the first
   `cuCtxCreate` builds the heavy per-context state.

An `LD_PRELOAD` shim cannot observe the driver's kernel traffic: the RM client
issues **raw `syscall(SYS_ioctl)`**, not libc `ioctl`. Instrumentation is done
with `ptrace` (see `tools/guest-diag/ioctl_trace.c` in the parent tree) or
with kernel-side tracing.

### 2.2 Interposition and tooling dispatch

Every exported call goes through a global function table before reaching the
implementation. A tool library can register an interposition layer *before*
`cuInit` and receive the original pointers to chain to. This is the sanctioned
mechanism used by profilers/debuggers, and it is also a clean hook point for
A/B instrumentation in the guest.

### 2.3 Entry tables

`cuGetExportTable(uuid)` returns pointers to internal tables for:

* the CUDA runtime library (fatbin loading, primary-context initialization,
  hidden module-load paths, kernel launch with internal parameters);
* tooling (profiling, debugging, memory checking, launch tracking);
* debugger support (context-local storage, trap handlers);
* interop and special platforms (video, NVN, OpenCL bridges).

Tables are identified by 128-bit UUIDs and start with a size field, so old
clients and new drivers coexist. This is a key fact for emulation: the guest
runtime/tools may call *through these tables* instead of the public API, and
all of it still resolves to the same internal engine.

### 2.4 Threading and process model

* API calls are caller-threaded; each context has a recursive RW lock.
* One ISR thread per context handles interrupts/DPC callbacks.
* One boost thread per context handles drop-to-idle clock policy.
* MPS (multi-process service) inserts a client/server IPC layer *below* the
  normal API: clients share the server's channels; the wire format is a
  private socket protocol, not the public API.
* Inter-process memory sharing uses kernel-driver exported handles and
  (optionally) POSIX file descriptors; CUDA's own IPC mem handles are an RM
  object plus a key.

### 2.5 What the driver touches at runtime (overview)

| Resource | Access | Purpose |
|---|---|---|
| `/dev/nvidiactl` | ioctl | control device: version negotiation, client creation, GPU attach, OS events, exported-object fds |
| `/dev/nvidiaN` | open/ioctl/mmap | per-GPU: object alloc/free, controls, memory alloc/map, interrupts |
| `/dev/nvidia-uvm` | open/ioctl | managed memory, page migration, channel registration (via UVM userspace API) |
| BAR regions | mmap | user-mode doorbells (USERD), interrupt tree/usermode region, framebuffer apertures |
| sysmem | mmap/alloc | command buffers, semaphores, page tables (when the driver owns them) |
| shared memory / sockets | IPC | MPS, IPC events/memory |

The exact ioctl vocabulary observed from the 570.144 guest is enumerated in
`docs/03-driver-and-gpu-interface.md`.

## 3. Running in a guest (what Open-IOV sees)

* The library is **unmodified** in the guest; it links against the guest's
  kernel driver (`nvidia.ko` + `nvidia-uvm.ko`) exactly as on bare metal.
* The kernel driver is also unmodified; it talks to the emulated PCI function
  and the GSP-RM interface that Open-IOV emulates.
* Therefore, from libcuda's point of view, the guest is a normal NVIDIA
  machine whose kernel driver has certain capabilities and whose GPU answers
  RM controls. The crash class we chase (`cuCtxCreate` segfault) is a
  *userspace* consequence of *kernel/GSP* replies that differ from bare metal.
* The library's own state is therefore the best available oracle: when the
  driver misbehaves, comparing its state (or its control traffic) against a
  bare-metal A/B run localizes the emulation gap. The A/B harness in
  `tools/cuda-ab/` does this at the workload level; `docs/05` does it at the
  function level.

## 4. Reproducing the build/execution analysis for a new version

```
tools/cudatool.py info       <libcuda.so.NEW>     # size, exports, ABI note
tools/cudatool.py exports    <libcuda.so.NEW>     # symbol map
tools/cudatool.py functions  <libcuda.so.NEW>     # FDE function map
tools/cudatool.py delta <old> <new>               # interface-level diff
tools/ghidra_full_analysis.sh <libcuda.so.NEW>    # full auto-analysis
tools/ghidra_export_index.sh ...                  # function/call/string index
tools/ghidra_batch_decompile.sh ...               # decompiled corpus
```

See `docs/06-tooling.md` for details and `docs/07-version-upgrade-playbook.md`
for the full procedure.