# 01 — CUDA userspace driver architecture This document describes the *userspace* CUDA driver (`libcuda.so`), the layer that applications and language runtimes talk to. Kernel drivers, GSP firmware and the GPU are covered only as the interfaces this library consumes. Everything below is established from decompilation of shipped binaries, open-source study and live traces. Addresses are file virtual addresses (`libcuda+0x...`) for driver **570.144** (CUDA 12.8, build id `c0e1e77ea249944ac4fd43faebed9498a729606a`, md5 `ed5b2fc0044df6052df1d5113754c499`). --- ## 1. What the library is `libcuda.so.1.1` is a single ~71 MB, fully stripped position-independent shared object. For 570.144: | Property | Value | |---|---| | Exported functions | 659 (public CUDA driver API plus internal/legacy entry points) | | Imported functions | 243 (libc/libpthread/libdl/librt only — no other NVIDIA userspace dependency for the CUDA path) | | `.text` | `0x179ec0 .. 0xe94c63` (13.7 MB) | | Functions found by full analysis | 29,494 (28,925 have unwind tables) | | Static data | `.rodata` 53 MB (error strings, tables, embedded device code), `.data`/`.bss` ~1.4 MB | | Build id | `c0e1e7…`; a matching `libcuda.so.1.1.debug` is referenced but not shipped | | CUDA version note | `.note.cuda.fwd_compatibility` says CUDA 12.8, supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` | Two entry surfaces coexist in the same library: 1. **The public driver API** — `cu*` symbols (`cuInit libcuda+0x310060`, `cuCtxCreate_v2 libcuda+0x310360`, `cuMemAlloc_v2 libcuda+0x310b20`, `cuLaunchKernel libcuda+0x36f420`, …). 2. **The entry-table API** — `cuGetExportTable` (`libcuda+0x350910`), a UUID-keyed lookup that hands internal function tables to the CUDA runtime (`libcudart`), tooling (profiler/debugger/memcheck) and interposition libraries. These tables are versioned by a leading `struct_size` field and are append-only. The library embeds device code (cubin images for memcpy/memset kernels, graph scheduler, syscall trampolines and preemption-handler assembly) in its read-only data; at initialization it uploads them through the normal module machinery. ### 1.1 The CUDA runtime (`libcudart`) and the driver The CUDA *runtime* library (`libcudart`) is a separate userspace library that applications normally link against; it implements the `cuda*` API (as opposed to the driver's `cu*` API) and is statically linked in many test programs (including the project's `vector_add` A/B binary). There is no kernel interface for it: everything it does bottoms out in `libcuda`. Two facts matter for analysis and debugging: * **`libcudart` calls the driver through the entry tables as well as the public API.** Fatbin loading, primary-context initialization and some launch paths use the UUID tables (see §2.3) for speed and to pass internal parameters the public API cannot express. So when a guest misbehaves in `cuda*` calls, the relevant driver code may be reached through a different door than the public `cu*` entry point. * **`libcudart` version and driver version are negotiated.** The driver reports its CUDA version (`cuDriverGetVersion`) and exposes versioned entry points; the runtime selects compatible paths. A guest crash can therefore involve a runtime→driver version negotiation rather than a single API call. In the Open-IOV guest the runtime is **stock and not our concern to modify**; it is, however, part of the A/B surface: the same statically linked binary run on the host (real GPU, apt driver) and in the guest isolates whether a difference is in the runtime or below it. ## 2. Layer map The library is strictly layered. Each layer has one job, and the boundary between layers is where the interesting interfaces live: ``` application / libcudart │ public cu* API or UUID entry tables ▼ public API wrappers generated: per-thread/global checks, tracing, │ interposition hook table, errors → CUresult ▼ API layer hand-written per domain: TLS context stack, │ init/state validation, argument checks, one │ context lock, one call into the engine ▼ CUDA internal engine the actual driver: contexts, devices, memory, │ streams, events, modules, graphs, sync ├───────────────┬───────────────────────────┬───────────────┐ ▼ ▼ ▼ ▼ driver-model layer hardware layer UVM userspace JIT/ELF (channels, allocs, (pushbuffer/QMD/method (managed memory, (PTX→SASS, controls; RM client) encoding per chip) page migration) cubin loading) │ │ │ ▼ ▼ ▼ /dev/nvidia* ioctls GPU queues/doorbells /dev/nvidia-uvm │ ▼ kernel driver + GSP-RM + GPU ``` The names of the layers are descriptive working names. ### 2.1 API layer The `cu*` exports are **generated**, not hand-written. A master export list (name, platform flags, the CUDA version in which the symbol was introduced) is turned into: * the exported symbol table (`cuInit`, `cuCtxCreate`, `cuCtxCreate_v2`, `_ptsz`/`_ptds` per-thread-stream variants, …); * a wrapper per function that performs the "driver destroyed" check, optional API serialization, then dispatches through a **global interposition table** so tools can replace any entry point before initialization; * a table of tool callbacks (enter/exit events keyed by stable numeric IDs) consumed by the profiling/debugging entry tables; * the `cuGetProcAddress` lookup tables, which resolve `(name, cudaVersion, flags)` to the right symbol variant. The hand-written API layer below the wrappers is uniform: 1. read the per-thread context stack (the TLS "current context"); 2. verify the driver is initialized (`CUDA_ERROR_NOT_INITIALIZED` / `_DEINITIALIZED`) and that the object handles are valid; 3. validate arguments and version gates; 4. take exactly one context lock (read or write; recursive RW lock); 5. call one engine function; 6. return a `CUresult`; sticky asynchronous errors are checked here too. This uniformity is what makes the API layer cheap to map: the semantic work always happens one layer down, in the engine. ### 2.2 CUDA internal engine The engine owns all driver state. Its main object graph: * **Process globals** — initialization state, the single kernel-client handle ("RM client") for the process, device list, handle pools, caches, per-thread context stack. * **Device** — chip/arch identity, capability bitfields, limits, hardware layer vtable, driver-model vtable, primary context. One per visible GPU. * **Context** — an application's CUDA context: its own memory manager, channel manager, stream manager, semaphore pools, QMD pools, trap handler, ISR thread, profiler state. A context is a *userspace* object; at the kernel level it materializes as channel groups + channels + a VA space, not as a distinct kernel object. * **Stream** — an ordered work queue. Each stream owns a small pool of QMDs (launch descriptors) and a marker (see §2.5). * **Module/Function** — a loaded cubin image and its kernels; functions carry the compiled metadata needed to build QMDs and constant banks. * **Memory objects** — a hierarchy: *memblock* (an allocation from the kernel driver, possibly VA-reserved and mapped), *memobj* (a suballocation of a memblock or a standalone object), plus the pointer/VA heaps that resolve device pointers and unified addresses back to objects. * **Events, graphs, libraries** — user-visible objects layered on the same marker/memory machinery. ### 2.3 Driver-model layer Everything that depends on *which* kernel driver is present goes through a function table selected at initialization: * the **RM backend** (Liberty/vGPU, Linux and Windows datacenter) — the one relevant to Open-IOV; * a SoC backend for Tegra-class `nvrm_gpu` stacks; * an Amodel backend (in-process simulator); * an MPS backend (multi-process service client/server); * a WDDM backend (Windows). The RM backend translates engine operations into *RM controls* and object allocations over `/dev/nvidiactl`/`/dev/nvidiaN`. It also owns the **driver-ABI adaptation** layer: the userspace driver can talk to kernel drivers from several ABI generations (570.144 advertises 470…570); one adaptation instance per ABI is compiled into the library and the matching one is selected at `cuInit` by brute-force version negotiation. ### 2.4 Hardware layer Chip-specific behavior — method encodings, QMD layout, semaphore release encodings, memcpy engine commands, doorbell format — lives behind a per-architecture vtable selected from the device's compute capability. The userspace driver *builds* the GPU command streams itself; the kernel driver mainly allocates objects, maps memory, schedules channels and services interrupts. ### 2.5 Synchronization model (the core mental model) Three primitives explain almost all CUDA synchronization: 1. **Tracking semaphore** — each channel has a GPU-visible memory location whose value counts completed pushes. The CPU tracks three counters: issued-by-CPU, issued-to-GPU and finished-by-GPU. Channel completion is "semaphore reached value N". 2. **Marker** — a small set of entries (tracking-semaphore values, QMD launch IDs, CPU semaphores, UVM semaphores) that together describe "all work before this point". Every stream operation appends to / waits on markers; stream sync, event record and stream-wait-event are marker operations. 3. **QMD semaphore** — each launch descriptor (QMD) carries its own completion semaphore, released by the GPU when the grid completes. Markers reference `(QMD, launch id)` and the driver compares the QMD semaphore payload. Blocking waits are implemented with an ISR thread per context plus a DPC queue; a "blocking sync" fast path lets the driver park a thread on a GPU event rather than spin. ### 2.6 Memory model * Device pointers come from a per-context address-space manager with four views: 32-bit/40-bit device pointers, full device VA, host VA and unified VA (UVA). Unified addresses are resolved by a process-global registry, so any context can look up a pointer allocated by another. * Allocation is a two-step design: reserve VA, then back it. Backing comes from kernel allocations (vidmem/sysmem), imported handles (IPC), host registrations, or the UVM driver (managed memory). * Managed memory and pageable-memory access are delegated to the UVM userspace library/driver, with CUDA maintaining a stream-dependency DAG so UVM can migrate pages correctly (see §2.7). * Copies are dispatched by a policy engine: host↔host in the CPU, small host→device inline in the compute pushbuffer, otherwise staged through pinned bounce buffers or sent to a copy engine channel. The choice depends on size, alignment, engine availability and whether UVM is active. ### 2.7 UVM integration Managed memory is a collaboration: CUDA owns streams and knows the dependency order; UVM owns page placement and migration. CUDA registers each channel and each managed allocation with UVM, tracks stream dependencies in a DAG (`stream → node → event`), and converts user synchronizations (`cuStreamSynchronize`, event waits) into UVM DAG syncs. If UVM is unavailable or unlicensed (vGPU profiles without UVM), the driver falls back to zero-copy-style behavior. ### 2.8 JIT and module loading `cuModuleLoad*` accepts fatbins, PTX, cubins and (newer) libraries. The path is: extract the right device image for the target ISA; JIT-compile PTX if needed (the embedded PTX compiler is loaded as a separate library); parse and relocate the ELF; register kernels; apply workarounds; make constant banks and texture/surface bindings; then upload through the channel machinery. Kernel launch subsequently builds a QMD referencing the loaded code and constant banks, and submits it with a PCAS ("prepare/switch") method stream on a compute channel. ## 3. Execution model inside the process * **Threads.** Every API call runs on the caller thread. The ISR thread per context services interrupts/DPCs; a per-context boost thread handles drop-to-idle clock management. Lock ordering is globally defined and enforced in debug builds, which is why deadlock analysis needs the lock order list rather than guesswork. * **Context stack.** The "current context" is per thread, with a stack (`cuCtxPushCurrent`/`PopCurrent`). Most calls implicitly use the top of the stack; the primary context is retained on first use. * **Lazy initialization.** `cuInit` initializes globals and enumerates devices but creates no context. Device enumeration itself performs RM queries; the first context creates the bulk of the resources (channels, semaphores, QMD pools, ISR thread, UVM registration). * **Async errors.** GPU faults surface asynchronously: the driver checks channel error notifiers when synchronizing or when a marker reports error, and returns sticky errors from subsequent API calls. ## 4. The 570.144 binary in one screen | Object | Export | File vaddr | |---|---|---| | initialization | `cuInit` | `0x310060` | | device enumeration | `cuDeviceGet` / `cuDeviceGetCount` | `0x3100a0` / `0x3100c0` | | context | `cuCtxCreate_v2` / `_v3` | `0x310360` / `0x310380` | | module load | `cuModuleLoadData` / `Ex` | `0x3106e0` / `0x310700` | | library API | `cuLibraryLoadData` | `0x3108e0` | | memory | `cuMemAlloc_v2` / `cuMemFree_v2` | `0x310b20` / `0x310b60` | | VMM API | `cuMemAddressReserve` / `cuMemCreate` / `cuMemMap` / `cuMemSetAccess` | `0x330e10`…`0x330ef0` | | stream/event | `cuStreamCreate` / `cuStreamSynchronize` / `cuEventRecord` | `0x331370` / `0x36f3c0` / `0x36f3e0` | | launch | `cuLaunchKernel` / `_ptsz` | `0x36f420` / `0x331a50` | | copy | `cuMemcpyHtoD_v2` / `cuMemcpyDtoH_v2` | `0x36eda0` / `0x36edc0` | | async alloc | `cuMemAllocAsync` | `0x36fa80` | | dispatch | `cuGetProcAddress` / `cuGetExportTable` | `0x36fb20` / `0x350910` | The exports are only 24–30-byte stubs; the work happens in the internal functions reached from them. Full call chains for the core paths (init, device attach, context create, memory, module load, launch, sync, export tables) are in `docs/05-570.144-decompilation.md` and the machine-readable index `570.144/run/analysis/elf-index/callgraph.tsv`. ## 5. Why this architecture is good news for Open-IOV 1. **One narrow kernel interface.** Everything below the engine funnels through the RM backend: object allocations, controls, memory mappings and the `/dev/nvidia*` ioctls. That is the surface Open-IOV emulates — not the whole API. 2. **The driver builds its own command streams.** Kernel launches, copies and syncs are encoded by *userspace* into pushbuffers, GPFIFO entries and QMDs. The kernel driver does not generate commands that must be faithfully emulated; it allocates, maps, schedules and services interrupts. 3. **Stable, versioned side channels.** Entry tables, `cuGetProcAddress` and the driver-ABI adaptation layer are explicitly versioned, so a version change moves details, not the shape. 4. **The object model is explicit and enumerable.** The exact kernel objects a context needs are visible in the decompiled call order (see `docs/05`), which is what makes a host-side emulation able to satisfy the driver without touching it.