path: root/userspace/cuda/docs/01-architecture.md

# 01 — CUDA userspace driver architecture

This document describes the *userspace* CUDA driver (`libcuda.so`), the layer
that applications and language runtimes talk to. Kernel drivers, GSP firmware
and the GPU are covered only as the interfaces this library consumes.

Everything below is established from decompilation of shipped binaries,
open-source study and live traces. Addresses are file virtual addresses
(`libcuda+0x...`) for driver **570.144** (CUDA 12.8, build id
`c0e1e77ea249944ac4fd43faebed9498a729606a`, md5
`ed5b2fc0044df6052df1d5113754c499`).

---

## 1. What the library is

`libcuda.so.1.1` is a single ~71 MB, fully stripped position-independent
shared object. For 570.144:

| Property | Value |
|---|---|
| Exported functions | 659 (public CUDA driver API plus internal/legacy entry points) |
| Imported functions | 243 (libc/libpthread/libdl/librt only — no other NVIDIA userspace dependency for the CUDA path) |
| `.text` | `0x179ec0 .. 0xe94c63` (13.7 MB) |
| Functions found by full analysis | 29,494 (28,925 have unwind tables) |
| Static data | `.rodata` 53 MB (error strings, tables, embedded device code), `.data`/`.bss` ~1.4 MB |
| Build id | `c0e1e7…`; a matching `libcuda.so.1.1.debug` is referenced but not shipped |
| CUDA version note | `.note.cuda.fwd_compatibility` says CUDA 12.8, supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` |

Two entry surfaces coexist in the same library:

1. **The public driver API** — `cu*` symbols (`cuInit libcuda+0x310060`,
   `cuCtxCreate_v2 libcuda+0x310360`, `cuMemAlloc_v2 libcuda+0x310b20`,
   `cuLaunchKernel libcuda+0x36f420`, …).
2. **The entry-table API** — `cuGetExportTable` (`libcuda+0x350910`),
   a UUID-keyed lookup that hands internal function tables to the CUDA
   runtime (`libcudart`), tooling (profiler/debugger/memcheck) and
   interposition libraries. These tables are versioned by a leading
   `struct_size` field and are append-only.

The library embeds device code (cubin images for memcpy/memset kernels,
graph scheduler, syscall trampolines and preemption-handler assembly) in its
read-only data; at initialization it uploads them through the normal module
machinery.

### 1.1 The CUDA runtime (`libcudart`) and the driver

The CUDA *runtime* library (`libcudart`) is a separate userspace library that
applications normally link against; it implements the `cuda*` API (as opposed
to the driver's `cu*` API) and is statically linked in many test programs
(including the project's `vector_add` A/B binary). There is no kernel
interface for it: everything it does bottoms out in `libcuda`. Two facts
matter for analysis and debugging:

* **`libcudart` calls the driver through the entry tables as well as the
  public API.** Fatbin loading, primary-context initialization and some
  launch paths use the UUID tables (see §2.3) for speed and to pass internal
  parameters the public API cannot express. So when a guest misbehaves in
  `cuda*` calls, the relevant driver code may be reached through a different
  door than the public `cu*` entry point.
* **`libcudart` version and driver version are negotiated.** The driver
  reports its CUDA version (`cuDriverGetVersion`) and exposes versioned
  entry points; the runtime selects compatible paths. A guest crash can
  therefore involve a runtime→driver version negotiation rather than a
  single API call.

In the Open-IOV guest the runtime is **stock and not our concern to modify**;
it is, however, part of the A/B surface: the same statically linked binary
run on the host (real GPU, apt driver) and in the guest isolates whether a
difference is in the runtime or below it.

## 2. Layer map

The library is strictly layered. Each layer has one job, and the boundary
between layers is where the interesting interfaces live:

```
application / libcudart
        │  public cu* API  or  UUID entry tables
        ▼
public API wrappers            generated: per-thread/global checks, tracing,
        │                      interposition hook table, errors → CUresult
        ▼
API layer                      hand-written per domain: TLS context stack,
        │                      init/state validation, argument checks, one
        │                      context lock, one call into the engine
        ▼
CUDA internal engine            the actual driver: contexts, devices, memory,
        │                      streams, events, modules, graphs, sync
        ├───────────────┬───────────────────────────┬───────────────┐
        ▼               ▼                           ▼               ▼
driver-model layer    hardware layer            UVM userspace    JIT/ELF
(channels, allocs,    (pushbuffer/QMD/method    (managed memory, (PTX→SASS,
 controls; RM client)  encoding per chip)        page migration)  cubin loading)
        │                       │                     │
        ▼                       ▼                     ▼
/dev/nvidia* ioctls      GPU queues/doorbells    /dev/nvidia-uvm
        │
        ▼
kernel driver + GSP-RM + GPU
```

The names of the layers are descriptive working names.

### 2.1 API layer

The `cu*` exports are **generated**, not hand-written. A master export list
(name, platform flags, the CUDA version in which the symbol was introduced)
is turned into:

* the exported symbol table (`cuInit`, `cuCtxCreate`, `cuCtxCreate_v2`,
  `_ptsz`/`_ptds` per-thread-stream variants, …);
* a wrapper per function that performs the "driver destroyed" check, optional
  API serialization, then dispatches through a **global interposition table**
  so tools can replace any entry point before initialization;
* a table of tool callbacks (enter/exit events keyed by stable numeric IDs)
  consumed by the profiling/debugging entry tables;
* the `cuGetProcAddress` lookup tables, which resolve
  `(name, cudaVersion, flags)` to the right symbol variant.

The hand-written API layer below the wrappers is uniform:

1. read the per-thread context stack (the TLS "current context");
2. verify the driver is initialized (`CUDA_ERROR_NOT_INITIALIZED` /
   `_DEINITIALIZED`) and that the object handles are valid;
3. validate arguments and version gates;
4. take exactly one context lock (read or write; recursive RW lock);
5. call one engine function;
6. return a `CUresult`; sticky asynchronous errors are checked here too.

This uniformity is what makes the API layer cheap to map: the semantic work
always happens one layer down, in the engine.

### 2.2 CUDA internal engine

The engine owns all driver state. Its main object graph:

* **Process globals** — initialization state, the single kernel-client handle
  ("RM client") for the process, device list, handle pools, caches, per-thread
  context stack.
* **Device** — chip/arch identity, capability bitfields, limits, hardware
  layer vtable, driver-model vtable, primary context. One per visible GPU.
* **Context** — an application's CUDA context: its own memory manager, channel
  manager, stream manager, semaphore pools, QMD pools, trap handler, ISR
  thread, profiler state. A context is a *userspace* object; at the kernel
  level it materializes as channel groups + channels + a VA space, not as a
  distinct kernel object.
* **Stream** — an ordered work queue. Each stream owns a small pool of QMDs
  (launch descriptors) and a marker (see §2.5).
* **Module/Function** — a loaded cubin image and its kernels; functions carry
  the compiled metadata needed to build QMDs and constant banks.
* **Memory objects** — a hierarchy: *memblock* (an allocation from the
  kernel driver, possibly VA-reserved and mapped), *memobj* (a suballocation
  of a memblock or a standalone object), plus the pointer/VA heaps that
  resolve device pointers and unified addresses back to objects.
* **Events, graphs, libraries** — user-visible objects layered on the same
  marker/memory machinery.

### 2.3 Driver-model layer

Everything that depends on *which* kernel driver is present goes through a
function table selected at initialization:

* the **RM backend** (Liberty/vGPU, Linux and Windows datacenter) — the one
  relevant to Open-IOV;
* a SoC backend for Tegra-class `nvrm_gpu` stacks;
* an Amodel backend (in-process simulator);
* an MPS backend (multi-process service client/server);
* a WDDM backend (Windows).

The RM backend translates engine operations into *RM controls* and object
allocations over `/dev/nvidiactl`/`/dev/nvidiaN`. It also owns the
**driver-ABI adaptation** layer: the userspace driver can talk to kernel
drivers from several ABI generations (570.144 advertises 470…570); one
adaptation instance per ABI is compiled into the library and the matching one
is selected at `cuInit` by brute-force version negotiation.

### 2.4 Hardware layer

Chip-specific behavior — method encodings, QMD layout, semaphore release
encodings, memcpy engine commands, doorbell format — lives behind a
per-architecture vtable selected from the device's compute capability. The
userspace driver *builds* the GPU command streams itself; the kernel driver
mainly allocates objects, maps memory, schedules channels and services
interrupts.

### 2.5 Synchronization model (the core mental model)

Three primitives explain almost all CUDA synchronization:

1. **Tracking semaphore** — each channel has a GPU-visible memory location
   whose value counts completed pushes. The CPU tracks three counters:
   issued-by-CPU, issued-to-GPU and finished-by-GPU. Channel completion is
   "semaphore reached value N".
2. **Marker** — a small set of entries (tracking-semaphore values, QMD launch
   IDs, CPU semaphores, UVM semaphores) that together describe "all work
   before this point". Every stream operation appends to / waits on markers;
   stream sync, event record and stream-wait-event are marker operations.
3. **QMD semaphore** — each launch descriptor (QMD) carries its own completion
   semaphore, released by the GPU when the grid completes. Markers reference
   `(QMD, launch id)` and the driver compares the QMD semaphore payload.

Blocking waits are implemented with an ISR thread per context plus a DPC
queue; a "blocking sync" fast path lets the driver park a thread on a GPU
event rather than spin.

### 2.6 Memory model

* Device pointers come from a per-context address-space manager with four
  views: 32-bit/40-bit device pointers, full device VA, host VA and unified
  VA (UVA). Unified addresses are resolved by a process-global registry, so
  any context can look up a pointer allocated by another.
* Allocation is a two-step design: reserve VA, then back it. Backing comes
  from kernel allocations (vidmem/sysmem), imported handles (IPC), host
  registrations, or the UVM driver (managed memory).
* Managed memory and pageable-memory access are delegated to the UVM
  userspace library/driver, with CUDA maintaining a stream-dependency DAG so
  UVM can migrate pages correctly (see §2.7).
* Copies are dispatched by a policy engine: host↔host in the CPU, small
  host→device inline in the compute pushbuffer, otherwise staged through
  pinned bounce buffers or sent to a copy engine channel. The choice depends
  on size, alignment, engine availability and whether UVM is active.

### 2.7 UVM integration

Managed memory is a collaboration: CUDA owns streams and knows the dependency
order; UVM owns page placement and migration. CUDA registers each channel and
each managed allocation with UVM, tracks stream dependencies in a DAG
(`stream → node → event`), and converts user synchronizations
(`cuStreamSynchronize`, event waits) into UVM DAG syncs. If UVM is
unavailable or unlicensed (vGPU profiles without UVM), the driver falls back
to zero-copy-style behavior.

### 2.8 JIT and module loading

`cuModuleLoad*` accepts fatbins, PTX, cubins and (newer) libraries. The path
is: extract the right device image for the target ISA; JIT-compile PTX if
needed (the embedded PTX compiler is loaded as a separate library); parse and
relocate the ELF; register kernels; apply workarounds; make constant banks and
texture/surface bindings; then upload through the channel machinery. Kernel
launch subsequently builds a QMD referencing the loaded code and constant
banks, and submits it with a PCAS ("prepare/switch") method stream on a
compute channel.

## 3. Execution model inside the process

* **Threads.** Every API call runs on the caller thread. The ISR thread per
  context services interrupts/DPCs; a per-context boost thread handles
  drop-to-idle clock management. Lock ordering is globally defined and
  enforced in debug builds, which is why deadlock analysis needs the lock
  order list rather than guesswork.
* **Context stack.** The "current context" is per thread, with a stack
  (`cuCtxPushCurrent`/`PopCurrent`). Most calls implicitly use the top of the
  stack; the primary context is retained on first use.
* **Lazy initialization.** `cuInit` initializes globals and enumerates
  devices but creates no context. Device enumeration itself performs RM
  queries; the first context creates the bulk of the resources (channels,
  semaphores, QMD pools, ISR thread, UVM registration).
* **Async errors.** GPU faults surface asynchronously: the driver checks
  channel error notifiers when synchronizing or when a marker reports error,
  and returns sticky errors from subsequent API calls.

## 4. The 570.144 binary in one screen

| Object | Export | File vaddr |
|---|---|---|
| initialization | `cuInit` | `0x310060` |
| device enumeration | `cuDeviceGet` / `cuDeviceGetCount` | `0x3100a0` / `0x3100c0` |
| context | `cuCtxCreate_v2` / `_v3` | `0x310360` / `0x310380` |
| module load | `cuModuleLoadData` / `Ex` | `0x3106e0` / `0x310700` |
| library API | `cuLibraryLoadData` | `0x3108e0` |
| memory | `cuMemAlloc_v2` / `cuMemFree_v2` | `0x310b20` / `0x310b60` |
| VMM API | `cuMemAddressReserve` / `cuMemCreate` / `cuMemMap` / `cuMemSetAccess` | `0x330e10`…`0x330ef0` |
| stream/event | `cuStreamCreate` / `cuStreamSynchronize` / `cuEventRecord` | `0x331370` / `0x36f3c0` / `0x36f3e0` |
| launch | `cuLaunchKernel` / `_ptsz` | `0x36f420` / `0x331a50` |
| copy | `cuMemcpyHtoD_v2` / `cuMemcpyDtoH_v2` | `0x36eda0` / `0x36edc0` |
| async alloc | `cuMemAllocAsync` | `0x36fa80` |
| dispatch | `cuGetProcAddress` / `cuGetExportTable` | `0x36fb20` / `0x350910` |

The exports are only 24–30-byte stubs; the work happens in the internal
functions reached from them. Full call chains for the core paths (init,
device attach, context create, memory, module load, launch, sync, export
tables) are in `docs/05-570.144-decompilation.md` and the machine-readable
index `570.144/run/analysis/elf-index/callgraph.tsv`.

## 5. Why this architecture is good news for Open-IOV

1. **One narrow kernel interface.** Everything below the engine funnels
   through the RM backend: object allocations, controls, memory mappings and
   the `/dev/nvidia*` ioctls. That is the surface Open-IOV emulates — not the
   whole API.
2. **The driver builds its own command streams.** Kernel launches, copies and
   syncs are encoded by *userspace* into pushbuffers, GPFIFO entries and QMDs.
   The kernel driver does not generate commands that must be faithfully
   emulated; it allocates, maps, schedules and services interrupts.
3. **Stable, versioned side channels.** Entry tables, `cuGetProcAddress` and
   the driver-ABI adaptation layer are explicitly versioned, so a version
   change moves details, not the shape.
4. **The object model is explicit and enumerable.** The exact kernel objects a
   context needs are visible in the decompiled call order (see `docs/05`),
   which is what makes a host-side emulation able to satisfy the driver
   without touching it.