diff options
Diffstat (limited to 'userspace/cuda/docs/01-architecture.md')
| -rw-r--r-- | userspace/cuda/docs/01-architecture.md | 302 |
1 files changed, 302 insertions, 0 deletions
diff --git a/userspace/cuda/docs/01-architecture.md b/userspace/cuda/docs/01-architecture.md new file mode 100644 index 0000000..2cde42f --- /dev/null +++ b/userspace/cuda/docs/01-architecture.md @@ -0,0 +1,302 @@ +# 01 — CUDA userspace driver architecture + +This document describes the *userspace* CUDA driver (`libcuda.so`), the layer +that applications and language runtimes talk to. Kernel drivers, GSP firmware +and the GPU are covered only as the interfaces this library consumes. + +Everything below is established from decompilation of shipped binaries, +open-source study and live traces. Addresses are file virtual addresses +(`libcuda+0x...`) for driver **570.144** (CUDA 12.8, build id +`c0e1e77ea249944ac4fd43faebed9498a729606a`, md5 +`ed5b2fc0044df6052df1d5113754c499`). + +--- + +## 1. What the library is + +`libcuda.so.1.1` is a single ~71 MB, fully stripped position-independent +shared object. For 570.144: + +| Property | Value | +|---|---| +| Exported functions | 659 (public CUDA driver API plus internal/legacy entry points) | +| Imported functions | 243 (libc/libpthread/libdl/librt only — no other NVIDIA userspace dependency for the CUDA path) | +| `.text` | `0x179ec0 .. 0xe94c63` (13.7 MB) | +| Functions found by full analysis | 29,494 (28,925 have unwind tables) | +| Static data | `.rodata` 53 MB (error strings, tables, embedded device code), `.data`/`.bss` ~1.4 MB | +| Build id | `c0e1e7…`; a matching `libcuda.so.1.1.debug` is referenced but not shipped | +| CUDA version note | `.note.cuda.fwd_compatibility` says CUDA 12.8, supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` | + +Two entry surfaces coexist in the same library: + +1. **The public driver API** — `cu*` symbols (`cuInit libcuda+0x310060`, + `cuCtxCreate_v2 libcuda+0x310360`, `cuMemAlloc_v2 libcuda+0x310b20`, + `cuLaunchKernel libcuda+0x36f420`, …). +2. **The entry-table API** — `cuGetExportTable` (`libcuda+0x350910`), + a UUID-keyed lookup that hands internal function tables to the CUDA + runtime (`libcudart`), tooling (profiler/debugger/memcheck) and + interposition libraries. These tables are versioned by a leading + `struct_size` field and are append-only. + +The library embeds device code (cubin images for memcpy/memset kernels, +graph scheduler, syscall trampolines and preemption-handler assembly) in its +read-only data; at initialization it uploads them through the normal module +machinery. + +### 1.1 The CUDA runtime (`libcudart`) and the driver + +The CUDA *runtime* library (`libcudart`) is a separate userspace library that +applications normally link against; it implements the `cuda*` API (as opposed +to the driver's `cu*` API) and is statically linked in many test programs +(including the project's `vector_add` A/B binary). There is no kernel +interface for it: everything it does bottoms out in `libcuda`. Two facts +matter for analysis and debugging: + +* **`libcudart` calls the driver through the entry tables as well as the + public API.** Fatbin loading, primary-context initialization and some + launch paths use the UUID tables (see §2.3) for speed and to pass internal + parameters the public API cannot express. So when a guest misbehaves in + `cuda*` calls, the relevant driver code may be reached through a different + door than the public `cu*` entry point. +* **`libcudart` version and driver version are negotiated.** The driver + reports its CUDA version (`cuDriverGetVersion`) and exposes versioned + entry points; the runtime selects compatible paths. A guest crash can + therefore involve a runtime→driver version negotiation rather than a + single API call. + +In the Open-IOV guest the runtime is **stock and not our concern to modify**; +it is, however, part of the A/B surface: the same statically linked binary +run on the host (real GPU, apt driver) and in the guest isolates whether a +difference is in the runtime or below it. + +## 2. Layer map + +The library is strictly layered. Each layer has one job, and the boundary +between layers is where the interesting interfaces live: + +``` +application / libcudart + │ public cu* API or UUID entry tables + ▼ +public API wrappers generated: per-thread/global checks, tracing, + │ interposition hook table, errors → CUresult + ▼ +API layer hand-written per domain: TLS context stack, + │ init/state validation, argument checks, one + │ context lock, one call into the engine + ▼ +CUDA internal engine the actual driver: contexts, devices, memory, + │ streams, events, modules, graphs, sync + ├───────────────┬───────────────────────────┬───────────────┐ + ▼ ▼ ▼ ▼ +driver-model layer hardware layer UVM userspace JIT/ELF +(channels, allocs, (pushbuffer/QMD/method (managed memory, (PTX→SASS, + controls; RM client) encoding per chip) page migration) cubin loading) + │ │ │ + ▼ ▼ ▼ +/dev/nvidia* ioctls GPU queues/doorbells /dev/nvidia-uvm + │ + ▼ +kernel driver + GSP-RM + GPU +``` + +The names of the layers are descriptive working names. + +### 2.1 API layer + +The `cu*` exports are **generated**, not hand-written. A master export list +(name, platform flags, the CUDA version in which the symbol was introduced) +is turned into: + +* the exported symbol table (`cuInit`, `cuCtxCreate`, `cuCtxCreate_v2`, + `_ptsz`/`_ptds` per-thread-stream variants, …); +* a wrapper per function that performs the "driver destroyed" check, optional + API serialization, then dispatches through a **global interposition table** + so tools can replace any entry point before initialization; +* a table of tool callbacks (enter/exit events keyed by stable numeric IDs) + consumed by the profiling/debugging entry tables; +* the `cuGetProcAddress` lookup tables, which resolve + `(name, cudaVersion, flags)` to the right symbol variant. + +The hand-written API layer below the wrappers is uniform: + +1. read the per-thread context stack (the TLS "current context"); +2. verify the driver is initialized (`CUDA_ERROR_NOT_INITIALIZED` / + `_DEINITIALIZED`) and that the object handles are valid; +3. validate arguments and version gates; +4. take exactly one context lock (read or write; recursive RW lock); +5. call one engine function; +6. return a `CUresult`; sticky asynchronous errors are checked here too. + +This uniformity is what makes the API layer cheap to map: the semantic work +always happens one layer down, in the engine. + +### 2.2 CUDA internal engine + +The engine owns all driver state. Its main object graph: + +* **Process globals** — initialization state, the single kernel-client handle + ("RM client") for the process, device list, handle pools, caches, per-thread + context stack. +* **Device** — chip/arch identity, capability bitfields, limits, hardware + layer vtable, driver-model vtable, primary context. One per visible GPU. +* **Context** — an application's CUDA context: its own memory manager, channel + manager, stream manager, semaphore pools, QMD pools, trap handler, ISR + thread, profiler state. A context is a *userspace* object; at the kernel + level it materializes as channel groups + channels + a VA space, not as a + distinct kernel object. +* **Stream** — an ordered work queue. Each stream owns a small pool of QMDs + (launch descriptors) and a marker (see §2.5). +* **Module/Function** — a loaded cubin image and its kernels; functions carry + the compiled metadata needed to build QMDs and constant banks. +* **Memory objects** — a hierarchy: *memblock* (an allocation from the + kernel driver, possibly VA-reserved and mapped), *memobj* (a suballocation + of a memblock or a standalone object), plus the pointer/VA heaps that + resolve device pointers and unified addresses back to objects. +* **Events, graphs, libraries** — user-visible objects layered on the same + marker/memory machinery. + +### 2.3 Driver-model layer + +Everything that depends on *which* kernel driver is present goes through a +function table selected at initialization: + +* the **RM backend** (Liberty/vGPU, Linux and Windows datacenter) — the one + relevant to Open-IOV; +* a SoC backend for Tegra-class `nvrm_gpu` stacks; +* an Amodel backend (in-process simulator); +* an MPS backend (multi-process service client/server); +* a WDDM backend (Windows). + +The RM backend translates engine operations into *RM controls* and object +allocations over `/dev/nvidiactl`/`/dev/nvidiaN`. It also owns the +**driver-ABI adaptation** layer: the userspace driver can talk to kernel +drivers from several ABI generations (570.144 advertises 470…570); one +adaptation instance per ABI is compiled into the library and the matching one +is selected at `cuInit` by brute-force version negotiation. + +### 2.4 Hardware layer + +Chip-specific behavior — method encodings, QMD layout, semaphore release +encodings, memcpy engine commands, doorbell format — lives behind a +per-architecture vtable selected from the device's compute capability. The +userspace driver *builds* the GPU command streams itself; the kernel driver +mainly allocates objects, maps memory, schedules channels and services +interrupts. + +### 2.5 Synchronization model (the core mental model) + +Three primitives explain almost all CUDA synchronization: + +1. **Tracking semaphore** — each channel has a GPU-visible memory location + whose value counts completed pushes. The CPU tracks three counters: + issued-by-CPU, issued-to-GPU and finished-by-GPU. Channel completion is + "semaphore reached value N". +2. **Marker** — a small set of entries (tracking-semaphore values, QMD launch + IDs, CPU semaphores, UVM semaphores) that together describe "all work + before this point". Every stream operation appends to / waits on markers; + stream sync, event record and stream-wait-event are marker operations. +3. **QMD semaphore** — each launch descriptor (QMD) carries its own completion + semaphore, released by the GPU when the grid completes. Markers reference + `(QMD, launch id)` and the driver compares the QMD semaphore payload. + +Blocking waits are implemented with an ISR thread per context plus a DPC +queue; a "blocking sync" fast path lets the driver park a thread on a GPU +event rather than spin. + +### 2.6 Memory model + +* Device pointers come from a per-context address-space manager with four + views: 32-bit/40-bit device pointers, full device VA, host VA and unified + VA (UVA). Unified addresses are resolved by a process-global registry, so + any context can look up a pointer allocated by another. +* Allocation is a two-step design: reserve VA, then back it. Backing comes + from kernel allocations (vidmem/sysmem), imported handles (IPC), host + registrations, or the UVM driver (managed memory). +* Managed memory and pageable-memory access are delegated to the UVM + userspace library/driver, with CUDA maintaining a stream-dependency DAG so + UVM can migrate pages correctly (see §2.7). +* Copies are dispatched by a policy engine: host↔host in the CPU, small + host→device inline in the compute pushbuffer, otherwise staged through + pinned bounce buffers or sent to a copy engine channel. The choice depends + on size, alignment, engine availability and whether UVM is active. + +### 2.7 UVM integration + +Managed memory is a collaboration: CUDA owns streams and knows the dependency +order; UVM owns page placement and migration. CUDA registers each channel and +each managed allocation with UVM, tracks stream dependencies in a DAG +(`stream → node → event`), and converts user synchronizations +(`cuStreamSynchronize`, event waits) into UVM DAG syncs. If UVM is +unavailable or unlicensed (vGPU profiles without UVM), the driver falls back +to zero-copy-style behavior. + +### 2.8 JIT and module loading + +`cuModuleLoad*` accepts fatbins, PTX, cubins and (newer) libraries. The path +is: extract the right device image for the target ISA; JIT-compile PTX if +needed (the embedded PTX compiler is loaded as a separate library); parse and +relocate the ELF; register kernels; apply workarounds; make constant banks and +texture/surface bindings; then upload through the channel machinery. Kernel +launch subsequently builds a QMD referencing the loaded code and constant +banks, and submits it with a PCAS ("prepare/switch") method stream on a +compute channel. + +## 3. Execution model inside the process + +* **Threads.** Every API call runs on the caller thread. The ISR thread per + context services interrupts/DPCs; a per-context boost thread handles + drop-to-idle clock management. Lock ordering is globally defined and + enforced in debug builds, which is why deadlock analysis needs the lock + order list rather than guesswork. +* **Context stack.** The "current context" is per thread, with a stack + (`cuCtxPushCurrent`/`PopCurrent`). Most calls implicitly use the top of the + stack; the primary context is retained on first use. +* **Lazy initialization.** `cuInit` initializes globals and enumerates + devices but creates no context. Device enumeration itself performs RM + queries; the first context creates the bulk of the resources (channels, + semaphores, QMD pools, ISR thread, UVM registration). +* **Async errors.** GPU faults surface asynchronously: the driver checks + channel error notifiers when synchronizing or when a marker reports error, + and returns sticky errors from subsequent API calls. + +## 4. The 570.144 binary in one screen + +| Object | Export | File vaddr | +|---|---|---| +| initialization | `cuInit` | `0x310060` | +| device enumeration | `cuDeviceGet` / `cuDeviceGetCount` | `0x3100a0` / `0x3100c0` | +| context | `cuCtxCreate_v2` / `_v3` | `0x310360` / `0x310380` | +| module load | `cuModuleLoadData` / `Ex` | `0x3106e0` / `0x310700` | +| library API | `cuLibraryLoadData` | `0x3108e0` | +| memory | `cuMemAlloc_v2` / `cuMemFree_v2` | `0x310b20` / `0x310b60` | +| VMM API | `cuMemAddressReserve` / `cuMemCreate` / `cuMemMap` / `cuMemSetAccess` | `0x330e10`…`0x330ef0` | +| stream/event | `cuStreamCreate` / `cuStreamSynchronize` / `cuEventRecord` | `0x331370` / `0x36f3c0` / `0x36f3e0` | +| launch | `cuLaunchKernel` / `_ptsz` | `0x36f420` / `0x331a50` | +| copy | `cuMemcpyHtoD_v2` / `cuMemcpyDtoH_v2` | `0x36eda0` / `0x36edc0` | +| async alloc | `cuMemAllocAsync` | `0x36fa80` | +| dispatch | `cuGetProcAddress` / `cuGetExportTable` | `0x36fb20` / `0x350910` | + +The exports are only 24–30-byte stubs; the work happens in the internal +functions reached from them. Full call chains for the core paths (init, +device attach, context create, memory, module load, launch, sync, export +tables) are in `docs/05-570.144-decompilation.md` and the machine-readable +index `570.144/run/analysis/elf-index/callgraph.tsv`. + +## 5. Why this architecture is good news for Open-IOV + +1. **One narrow kernel interface.** Everything below the engine funnels + through the RM backend: object allocations, controls, memory mappings and + the `/dev/nvidia*` ioctls. That is the surface Open-IOV emulates — not the + whole API. +2. **The driver builds its own command streams.** Kernel launches, copies and + syncs are encoded by *userspace* into pushbuffers, GPFIFO entries and QMDs. + The kernel driver does not generate commands that must be faithfully + emulated; it allocates, maps, schedules and services interrupts. +3. **Stable, versioned side channels.** Entry tables, `cuGetProcAddress` and + the driver-ABI adaptation layer are explicitly versioned, so a version + change moves details, not the shape. +4. **The object model is explicit and enumerable.** The exact kernel objects a + context needs are visible in the decompiled call order (see `docs/05`), + which is what makes a host-side emulation able to satisfy the driver + without touching it. |