# CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive

> **Status (2026-09-23, honest).** The analysis covers the whole library for
> driver 570.144 and the tooling is version-agnostic. The *control plane* gets
> the stock guest stack through `cuInit`, device enumeration and into
> `cuCtxCreate`, which is blocked at `rc=999` because the emulated engine list
> is missing the async copy engine `COPY14`. The *data plane* is at the
> software memop bridge: the host executes the memsets, copies and semaphore
> releases it decodes from guest push buffers, and **no CUDA kernel has been
> forwarded yet**. Canonical detail: [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md)
> through `08`; reading order
> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md);
> front door [`../docs/README.md`](../docs/README.md).

## 1. What it is

`libcuda.so.1.1` is the userspace half of the CUDA driver: the library an
application or language runtime loads to talk to a GPU. For 570.144 it is a
**single, fully stripped, position-independent shared object of about 71 MB**:

| Property | Value |
|---|---|
| Exports | 659 (public `cu*` API plus internal/legacy entry points) |
| Imports | 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path |
| `.text` | `0x179ec0`–`0xe94c63` (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables) |
| `.rodata` | ~53 MB — error strings, dispatch/version tables, **embedded device code** |
| CUDA note | CUDA 12.8; supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` |

Two entry surfaces coexist: the **public driver API** (`cuInit`,
`cuCtxCreate_v2`, `cuMemAlloc_v2`, `cuLaunchKernel`, …) and the
**entry-table API** — `cuGetExportTable`, a 128-bit-UUID-keyed lookup that
hands internal tables to the CUDA runtime (`libcudart`), tooling and
interposition libraries. Tables are versioned by a leading `struct_size`
field and are append-only.

The library embeds **device code** — cubins for memcpy/memset kernels, the
graph scheduler, syscall trampolines and preemption-handler assembly — and
loads it through its own module machinery, so a stripped `libcuda.so` still
contains many small ELF images in `.rodata`. The CUDA *runtime* (`libcudart`,
the `cuda*` API) is a separate stock library that bottoms out in `libcuda`; it
also calls through the entry tables, so a guest failure in a `cuda*` call may
reach driver code through a different door than the public `cu*` entry point.

**Open-IOV requires this library unmodified.** In the guest it links against
the guest's stock `nvidia.ko` / `nvidia-uvm.ko` exactly as on bare metal; there
is no `LD_PRELOAD` shim and no patch. Instrumentation uses `ptrace` or
kernel-side tracing, because the RM client issues raw `syscall(SYS_ioctl)`
rather than libc `ioctl`.

## 2. Layer model

The library is strictly layered, and the boundaries are where the emulation
interfaces live (condensed from `docs/01` and `docs/02`):

```
application / libcudart
      | public cu* API or UUID entry tables
      v
generated wrappers -> API layer -> CUDA internal engine
                                      |-- driver-model layer (RM) -> /dev/nvidia* ioctls
                                      |-- hardware layer (QMD/method encoding)
                                      |     -> GPU queues, doorbells
                                      |-- UVM userspace (managed memory) -> /dev/nvidia-uvm
                                      '-- JIT/ELF (PTX->SASS, cubin loading)
kernel driver + GSP-RM + GPU
```

* **Generated wrappers and API layer.** A master export list (name, platform
  flags, CUDA version introduced) generates the exported symbols, wrappers,
  tool callbacks and `cuGetProcAddress` tables (`_vN`/`_ptsz` variants are part
  of that data). Below it, the hand-written layer reads the per-thread context
  stack, validates state, takes one context lock, calls one engine function and
  returns a `CUresult` — the semantic work is always one layer down.
* **Internal engine.** Owns the object graph: process globals (init state,
  kernel-client handle, device list), device (chip/caps/limits, vtables),
  context (memory/channel/stream managers, QMD pools, ISR thread), stream,
  module/function, memory objects, events/graphs/libraries.
* **Driver-model layer.** Selected at initialization behind one function
  table: the **RM backend** (the one Open-IOV cares about) plus SoC, Amodel,
  MPS and WDDM. It turns engine operations into RM controls and object
  allocations over `/dev/nvidia*`, and owns the **driver-ABI adaptation**
  layer — one instance per supported kernel-driver ABI, chosen at `cuInit`.
* **Hardware layer.** Chip-specific method/QMD/semaphore/doorbell encoders
  selected by compute capability. The userspace driver *builds* the GPU
  command streams; the kernel driver allocates, maps, schedules and services
  interrupts.
* **Synchronization model.** Three primitives explain most of it: the
  **tracking semaphore** (per-channel counter of completed pushes), the
  **marker** ("all work before this point") and the **QMD semaphore**
  (released when a grid completes). Blocking waits use a per-context ISR
  thread and a DPC queue.

## 3. What it touches in a guest

This is the emulation contract; the full checklist is `docs/03` §9.

**Device nodes.** `/dev/nvidiactl` (process-wide control: version negotiation,
client creation, GPU attach, OS events, exported-object fds), `/dev/nvidiaN`
(per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation),
`/dev/nvidia-uvm` (managed memory and page migration) and `/dev/nvidia-vgpuN`
(vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths).

**ioctls.** The control device uses `_IOWR('F', cmd, …)`: `200`
`NV_ESC_CARD_INFO`, `210` `NV_ESC_CHECK_VERSION_STR`, `212`
`NV_ESC_ATTACH_GPUS_TO_FD`. RM escapes include `0x2b` `NV_ESC_RM_ALLOC`
(**the main object-allocation path**), `0x2a` `NV_ESC_RM_CONTROL`, `0x29`
`NV_ESC_RM_FREE`, `0x4e`/`0x4f` `NV_ESC_RM_MAP_MEMORY`/`UNMAP_MEMORY`,
`0x57`/`0x58` `NV_ESC_RM_MAP_MEMORY_DMA`/`UNMAP_MEMORY_DMA` and `0x4a`
`NV_ESC_RM_VID_HEAP_CONTROL`. The request word `rsi=0xc030462b` in a crash
dump decodes to "allocating an RM object with a 48-byte payload".

**RM objects/classes.** Every RM call names a class. Observed for 570.144 on
an emulated TU104: `NV01_ROOT` (`0x0`), `NV01_DEVICE_0` (`0x80`),
`NV20_SUBDEVICE_0` (`0x2080`), `NV01_EVENT` (`0x40`),
`KEPLER_CHANNEL_GROUP_A` (`0xa06c`), a channel-group/runlist object (`0x9067`),
`TURING_CHANNEL_GPFIFO_A` (`0xc46f`, the channel), `TURING_COMPUTE_A`
(`0xc5c0`, the compute engine object), plus memory (`0x3e`) and VA-space
classes. The observed `cuCtxCreate` order is root → device → subdevice →
events → channel group → channel-group object → memory → channel → control
`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` → compute object → engine registration.

**Controls.** Device/GPU info (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`,
`GPU_GET_VIRTUALIZATION_MODE`, `FB_GET_INFO`, `CE_GET_CAPS`, …), FIFO/channels
(`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` `0x80170d`, GPFIFO scheduling,
timeslices, `GET_CLASS_ENGINEID`), GR/context switches, memory, events and
interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates
unknown-but-successful controls; it only fails when a *specific reply field*
is consumed and wrong.

**Memory.** VA reservation (`NV_ESC_RM_VID_HEAP_CONTROL`/`NVOS32`), backing
allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and
mapping (`NVOS33` into the process, `NVOS46` into a device VA space). All
guest memory CUDA uses is either an RM allocation backed by emulated
VRAM/BAR1 or pinned guest RAM the GPU must DMA to.

**Channels, GPFIFO, USERD, doorbells.** A channel is a GPFIFO (array of
pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO,
error notifier and (Volta and later) the USERD itself, passing the handles to
the channel allocation; submission writes the GPFIFO put pointer into USERD
and rings a doorbell token returned by the channel allocation. Pushbuffers are
method streams terminated by a tracking-semaphore release and a new GPFIFO
entry; the host emulation decodes them as such.

**Synchronization below the API.** Semaphores (plain GPU-visible memory
values), 4 KB per-channel error notifiers, OS events (kernel objects used to
wake blocked threads), interrupts routed through the kernel driver, and a
trap handler for debuggers and `device-side assert`.

**UVM and GSP-RM.** When UVM is available the driver registers GPUs, VA
spaces, channels and managed allocations and pushes UVM semaphores into the
same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing
from userspace — it still issues RM ioctls; inside the kernel driver controls
become GSP-RM RPCs, which Open-IOV's host module answers.

## 4. Why forwarding matters, and where we are

The performance budget is dominated by **push-buffer execution**,
**synchronization latency** and (once, at setup) **control-plane latency**
(`docs/04` §4). Only forwarding *real work* to the physical GPU meets the
first two; software execution of state/memory ops is a correctness bridge, not
the data plane (`docs/08`).

* **Stated architecture:** *emulate the interface, pass through the work,
  never patch anyone.* Prototype the host-RM/GSP proxy first for correctness,
  measure, then move the hot path to direct submission if the budgets demand
  it — work path abstracted from day one.
* **Control plane today:** GSP-RM emulation, RPC handling, BAR/PCI/MSI-X
  emulation and object/control replies are implemented; the stock guest stack
  gets through `cuInit` and device enumeration into `cuCtxCreate`.
* **Current blocker:** `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`)
  because the class-filtered engine list is missing the async copy engine
  `COPY14` (descriptor `0x793ceb0e`, instance 14, client type `0x38`, class
  `0xc9b5`). The oracle list has 7 engines (GR, CE0..CE4, `COPY14`); the guest
  list has 6. The driver silently skips engines with no per-(class, instance)
  database entry, then reads a zeroed secondary record and rolls back; what
  builds that database entry is the open question (`docs/05` §3.6–3.7).
* **Data plane today:** the host module decodes guest push buffers as method
  streams and **executes the small work it understands itself** — memsets,
  copies and semaphore releases. **No CUDA kernel has been forwarded yet.**

## 5. The crash-path lesson: replies are consumed data

The old deterministic `cuCtxCreate` segfault at `libcuda+0x453f90` shows why
reply coherence matters (`docs/05` §3.5, `docs/04` §6). The engine-registration
hook allocates a `TURING_COMPUTE_A` object with `pAllocParms` pointing at the
caller's own **8-byte stack local** and `paramsSize = 0`; the guest driver does
not honour `paramsSize`, so it reads **16-byte** class params starting there
(the upper 8 bytes are the hook's saved `rbp`) and copies 16 bytes of reply
params back over the same buffer. With the old zeroed reply that copy-back
zeroed the saved `rbp`; the function's epilogue then loaded `rbp = 0` and its
next access — reloading a pointer from `[rbp-0x38]` — faulted at
`0xffffffffffffffc8` (the store was the kernel's `copy_to_user`).

**The fix:** echo the request's params back, forcing only `status` to `NV_OK`;
the copy-back then writes exactly the bytes it read, and the fault disappears —
the same thing the real GSP-RM does for these in/out params.

**The general rule:** the userspace library is a *consumer of emulated data*.
A wrong reply is worse than a slow one. Work up the reply-fidelity ladder —
status-only → coherent zeroed params → coherent values for consumed fields →
exact values — and validate each rung with the A/B oracle.

## 6. How to look things up

The toolchain is version-agnostic (`docs/06`):

```
tools/cudatool.py info|exports|functions|nearest|crash|delta <lib>
tools/callwalk.py <callgraph.tsv> callers|callees|chain ...
```

* **`cudatool.py`** — build id/sections/exports/ABI note (`info`), symbol map
  (`exports`), function ranges from `.eh_frame_hdr` (`functions`), containing
  function and nearest export for an address (`nearest`), dmesg `ip` to file
  vaddr (`crash`), and an interface-level diff (`delta`).
* **`callwalk.py`** — caller/callee trees and shortest call paths over the
  callgraph index.
* **Annotated corpus** — `570.144/annotations/names.tsv` (our name map; exports
  exact, internals grow over time) and `subsystems.tsv`; corpus headers carry
  the recovered name, file address and direct callers/callees. The generated corpus
  is not distributed.

**Address convention (read this before citing an offset).** Public docs cite
**file virtual addresses** as `libcuda+0x...`. Ghidra loads these binaries at
image base `0x100000`, so `program address = file vaddr + 0x100000`; the
`tools/` wrappers translate automatically, raw Ghidra scripts do not. An
earlier analysis passed file vaddrs straight to Ghidra and decompiled functions
0x100000 too low, so its decompiled-code conclusions were about the wrong
functions (its live traces stayed valid). The same applies when transferring a
name map to a new driver version.

## 7. Where the detail lives

| Document | What it gives you |
|---|---|
| [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | the reading order, fast recipes, and the two findings that change how you debug |
| [`../userspace/cuda/docs/01-architecture.md`](../userspace/cuda/docs/01-architecture.md) | the full layer/object/sync/memory model |
| [`../userspace/cuda/docs/02-build-and-execution.md`](../userspace/cuda/docs/02-build-and-execution.md) | build model, embedded device code, init/threading, what it touches |
| [`../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../userspace/cuda/docs/03-driver-and-gpu-interface.md) | the emulation contract: nodes, ioctls, classes, controls, channels, UVM |
| [`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) | NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security |
| [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md) | the binary map, the crash resolution, and the current engine-topology blocker |
| [`../userspace/cuda/docs/06-tooling.md`](../userspace/cuda/docs/06-tooling.md) | every tool, its usage and its cost |
| [`../userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) | re-running the analysis on a new driver version |
| [`../userspace/cuda/docs/08-virtualization-approaches.md`](../userspace/cuda/docs/08-virtualization-approaches.md) | forwarding options and the honest current stage |
| [`../docs/README.md`](../docs/README.md) | the project documentation front door |