diff options
Diffstat (limited to 'wiki/CUDA-Userspace-Driver.md')
| -rw-r--r-- | wiki/CUDA-Userspace-Driver.md | 245 |
1 files changed, 245 insertions, 0 deletions
diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md new file mode 100644 index 0000000..3fa5d41 --- /dev/null +++ b/wiki/CUDA-Userspace-Driver.md @@ -0,0 +1,245 @@ +# CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive + +> **Status (2026-09-23, honest).** The analysis covers the whole library for +> driver 570.144 and the tooling is version-agnostic. The *control plane* gets +> the stock guest stack through `cuInit`, device enumeration and into +> `cuCtxCreate`, which is blocked at `rc=999` because the emulated engine list +> is missing the async copy engine `COPY14`. The *data plane* is at the +> software memop bridge: the host executes the memsets, copies and semaphore +> releases it decodes from guest push buffers, and **no CUDA kernel has been +> forwarded yet**. Canonical detail: [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) +> through `08`; reading order +> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); +> front door [`../docs/README.md`](../docs/README.md). + +## 1. What it is + +`libcuda.so.1.1` is the userspace half of the CUDA driver: the library an +application or language runtime loads to talk to a GPU. For 570.144 it is a +**single, fully stripped, position-independent shared object of about 71 MB**: + +| Property | Value | +|---|---| +| Exports | 659 (public `cu*` API plus internal/legacy entry points) | +| Imports | 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path | +| `.text` | `0x179ec0`–`0xe94c63` (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables) | +| `.rodata` | ~53 MB — error strings, dispatch/version tables, **embedded device code** | +| CUDA note | CUDA 12.8; supported kernel-driver ABIs `[470, 535, 550, 560, 565, 570]` | + +Two entry surfaces coexist: the **public driver API** (`cuInit`, +`cuCtxCreate_v2`, `cuMemAlloc_v2`, `cuLaunchKernel`, …) and the +**entry-table API** — `cuGetExportTable`, a 128-bit-UUID-keyed lookup that +hands internal tables to the CUDA runtime (`libcudart`), tooling and +interposition libraries. Tables are versioned by a leading `struct_size` +field and are append-only. + +The library embeds **device code** — cubins for memcpy/memset kernels, the +graph scheduler, syscall trampolines and preemption-handler assembly — and +loads it through its own module machinery, so a stripped `libcuda.so` still +contains many small ELF images in `.rodata`. The CUDA *runtime* (`libcudart`, +the `cuda*` API) is a separate stock library that bottoms out in `libcuda`; it +also calls through the entry tables, so a guest failure in a `cuda*` call may +reach driver code through a different door than the public `cu*` entry point. + +**Open-IOV requires this library unmodified.** In the guest it links against +the guest's stock `nvidia.ko` / `nvidia-uvm.ko` exactly as on bare metal; there +is no `LD_PRELOAD` shim and no patch. Instrumentation uses `ptrace` or +kernel-side tracing, because the RM client issues raw `syscall(SYS_ioctl)` +rather than libc `ioctl`. + +## 2. Layer model + +The library is strictly layered, and the boundaries are where the emulation +interfaces live (condensed from `docs/01` and `docs/02`): + +``` +application / libcudart + | public cu* API or UUID entry tables + v +generated wrappers -> API layer -> CUDA internal engine + |-- driver-model layer (RM) -> /dev/nvidia* ioctls + |-- hardware layer (QMD/method encoding) + | -> GPU queues, doorbells + |-- UVM userspace (managed memory) -> /dev/nvidia-uvm + '-- JIT/ELF (PTX->SASS, cubin loading) +kernel driver + GSP-RM + GPU +``` + +* **Generated wrappers and API layer.** A master export list (name, platform + flags, CUDA version introduced) generates the exported symbols, wrappers, + tool callbacks and `cuGetProcAddress` tables (`_vN`/`_ptsz` variants are part + of that data). Below it, the hand-written layer reads the per-thread context + stack, validates state, takes one context lock, calls one engine function and + returns a `CUresult` — the semantic work is always one layer down. +* **Internal engine.** Owns the object graph: process globals (init state, + kernel-client handle, device list), device (chip/caps/limits, vtables), + context (memory/channel/stream managers, QMD pools, ISR thread), stream, + module/function, memory objects, events/graphs/libraries. +* **Driver-model layer.** Selected at initialization behind one function + table: the **RM backend** (the one Open-IOV cares about) plus SoC, Amodel, + MPS and WDDM. It turns engine operations into RM controls and object + allocations over `/dev/nvidia*`, and owns the **driver-ABI adaptation** + layer — one instance per supported kernel-driver ABI, chosen at `cuInit`. +* **Hardware layer.** Chip-specific method/QMD/semaphore/doorbell encoders + selected by compute capability. The userspace driver *builds* the GPU + command streams; the kernel driver allocates, maps, schedules and services + interrupts. +* **Synchronization model.** Three primitives explain most of it: the + **tracking semaphore** (per-channel counter of completed pushes), the + **marker** ("all work before this point") and the **QMD semaphore** + (released when a grid completes). Blocking waits use a per-context ISR + thread and a DPC queue. + +## 3. What it touches in a guest + +This is the emulation contract; the full checklist is `docs/03` §9. + +**Device nodes.** `/dev/nvidiactl` (process-wide control: version negotiation, +client creation, GPU attach, OS events, exported-object fds), `/dev/nvidiaN` +(per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation), +`/dev/nvidia-uvm` (managed memory and page migration) and `/dev/nvidia-vgpuN` +(vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths). + +**ioctls.** The control device uses `_IOWR('F', cmd, …)`: `200` +`NV_ESC_CARD_INFO`, `210` `NV_ESC_CHECK_VERSION_STR`, `212` +`NV_ESC_ATTACH_GPUS_TO_FD`. RM escapes include `0x2b` `NV_ESC_RM_ALLOC` +(**the main object-allocation path**), `0x2a` `NV_ESC_RM_CONTROL`, `0x29` +`NV_ESC_RM_FREE`, `0x4e`/`0x4f` `NV_ESC_RM_MAP_MEMORY`/`UNMAP_MEMORY`, +`0x57`/`0x58` `NV_ESC_RM_MAP_MEMORY_DMA`/`UNMAP_MEMORY_DMA` and `0x4a` +`NV_ESC_RM_VID_HEAP_CONTROL`. The request word `rsi=0xc030462b` in a crash +dump decodes to "allocating an RM object with a 48-byte payload". + +**RM objects/classes.** Every RM call names a class. Observed for 570.144 on +an emulated TU104: `NV01_ROOT` (`0x0`), `NV01_DEVICE_0` (`0x80`), +`NV20_SUBDEVICE_0` (`0x2080`), `NV01_EVENT` (`0x40`), +`KEPLER_CHANNEL_GROUP_A` (`0xa06c`), a channel-group/runlist object (`0x9067`), +`TURING_CHANNEL_GPFIFO_A` (`0xc46f`, the channel), `TURING_COMPUTE_A` +(`0xc5c0`, the compute engine object), plus memory (`0x3e`) and VA-space +classes. The observed `cuCtxCreate` order is root → device → subdevice → +events → channel group → channel-group object → memory → channel → control +`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` → compute object → engine registration. + +**Controls.** Device/GPU info (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`, +`GPU_GET_VIRTUALIZATION_MODE`, `FB_GET_INFO`, `CE_GET_CAPS`, …), FIFO/channels +(`NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` `0x80170d`, GPFIFO scheduling, +timeslices, `GET_CLASS_ENGINEID`), GR/context switches, memory, events and +interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates +unknown-but-successful controls; it only fails when a *specific reply field* +is consumed and wrong. + +**Memory.** VA reservation (`NV_ESC_RM_VID_HEAP_CONTROL`/`NVOS32`), backing +allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and +mapping (`NVOS33` into the process, `NVOS46` into a device VA space). All +guest memory CUDA uses is either an RM allocation backed by emulated +VRAM/BAR1 or pinned guest RAM the GPU must DMA to. + +**Channels, GPFIFO, USERD, doorbells.** A channel is a GPFIFO (array of +pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO, +error notifier and (Volta and later) the USERD itself, passing the handles to +the channel allocation; submission writes the GPFIFO put pointer into USERD +and rings a doorbell token returned by the channel allocation. Pushbuffers are +method streams terminated by a tracking-semaphore release and a new GPFIFO +entry; the host emulation decodes them as such. + +**Synchronization below the API.** Semaphores (plain GPU-visible memory +values), 4 KB per-channel error notifiers, OS events (kernel objects used to +wake blocked threads), interrupts routed through the kernel driver, and a +trap handler for debuggers and `device-side assert`. + +**UVM and GSP-RM.** When UVM is available the driver registers GPUs, VA +spaces, channels and managed allocations and pushes UVM semaphores into the +same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing +from userspace — it still issues RM ioctls; inside the kernel driver controls +become GSP-RM RPCs, which Open-IOV's host module answers. + +## 4. Why forwarding matters, and where we are + +The performance budget is dominated by **push-buffer execution**, +**synchronization latency** and (once, at setup) **control-plane latency** +(`docs/04` §4). Only forwarding *real work* to the physical GPU meets the +first two; software execution of state/memory ops is a correctness bridge, not +the data plane (`docs/08`). + +* **Stated architecture:** *emulate the interface, pass through the work, + never patch anyone.* Prototype the host-RM/GSP proxy first for correctness, + measure, then move the hot path to direct submission if the budgets demand + it — work path abstracted from day one. +* **Control plane today:** GSP-RM emulation, RPC handling, BAR/PCI/MSI-X + emulation and object/control replies are implemented; the stock guest stack + gets through `cuInit` and device enumeration into `cuCtxCreate`. +* **Current blocker:** `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) + because the class-filtered engine list is missing the async copy engine + `COPY14` (descriptor `0x793ceb0e`, instance 14, client type `0x38`, class + `0xc9b5`). The oracle list has 7 engines (GR, CE0..CE4, `COPY14`); the guest + list has 6. The driver silently skips engines with no per-(class, instance) + database entry, then reads a zeroed secondary record and rolls back; what + builds that database entry is the open question (`docs/05` §3.6–3.7). +* **Data plane today:** the host module decodes guest push buffers as method + streams and **executes the small work it understands itself** — memsets, + copies and semaphore releases. **No CUDA kernel has been forwarded yet.** + +## 5. The crash-path lesson: replies are consumed data + +The old deterministic `cuCtxCreate` segfault at `libcuda+0x453f90` shows why +reply coherence matters (`docs/05` §3.5, `docs/04` §6). The engine-registration +hook allocates a `TURING_COMPUTE_A` object with `pAllocParms` pointing at the +caller's own **8-byte stack local** and `paramsSize = 0`; the guest driver does +not honour `paramsSize`, so it reads **16-byte** class params starting there +(the upper 8 bytes are the hook's saved `rbp`) and copies 16 bytes of reply +params back over the same buffer. With the old zeroed reply that copy-back +zeroed the saved `rbp`; the function's epilogue then loaded `rbp = 0` and its +next access — reloading a pointer from `[rbp-0x38]` — faulted at +`0xffffffffffffffc8` (the store was the kernel's `copy_to_user`). + +**The fix:** echo the request's params back, forcing only `status` to `NV_OK`; +the copy-back then writes exactly the bytes it read, and the fault disappears — +the same thing the real GSP-RM does for these in/out params. + +**The general rule:** the userspace library is a *consumer of emulated data*. +A wrong reply is worse than a slow one. Work up the reply-fidelity ladder — +status-only → coherent zeroed params → coherent values for consumed fields → +exact values — and validate each rung with the A/B oracle. + +## 6. How to look things up + +The toolchain is version-agnostic (`docs/06`): + +``` +tools/cudatool.py info|exports|functions|nearest|crash|delta <lib> +tools/callwalk.py <callgraph.tsv> callers|callees|chain ... +``` + +* **`cudatool.py`** — build id/sections/exports/ABI note (`info`), symbol map + (`exports`), function ranges from `.eh_frame_hdr` (`functions`), containing + function and nearest export for an address (`nearest`), dmesg `ip` to file + vaddr (`crash`), and an interface-level diff (`delta`). +* **`callwalk.py`** — caller/callee trees and shortest call paths over the + callgraph index. +* **Annotated corpus** — `570.144/annotations/names.tsv` (our name map; exports + exact, internals grow over time) and `subsystems.tsv`; corpus headers carry + the recovered name, file address and direct callers/callees. The generated corpus + is not distributed. + +**Address convention (read this before citing an offset).** Public docs cite +**file virtual addresses** as `libcuda+0x...`. Ghidra loads these binaries at +image base `0x100000`, so `program address = file vaddr + 0x100000`; the +`tools/` wrappers translate automatically, raw Ghidra scripts do not. An +earlier analysis passed file vaddrs straight to Ghidra and decompiled functions +0x100000 too low, so its decompiled-code conclusions were about the wrong +functions (its live traces stayed valid). The same applies when transferring a +name map to a new driver version. + +## 7. Where the detail lives + +| Document | What it gives you | +|---|---| +| [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | the reading order, fast recipes, and the two findings that change how you debug | +| [`../userspace/cuda/docs/01-architecture.md`](../userspace/cuda/docs/01-architecture.md) | the full layer/object/sync/memory model | +| [`../userspace/cuda/docs/02-build-and-execution.md`](../userspace/cuda/docs/02-build-and-execution.md) | build model, embedded device code, init/threading, what it touches | +| [`../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../userspace/cuda/docs/03-driver-and-gpu-interface.md) | the emulation contract: nodes, ioctls, classes, controls, channels, UVM | +| [`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) | NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security | +| [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md) | the binary map, the crash resolution, and the current engine-topology blocker | +| [`../userspace/cuda/docs/06-tooling.md`](../userspace/cuda/docs/06-tooling.md) | every tool, its usage and its cost | +| [`../userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) | re-running the analysis on a new driver version | +| [`../userspace/cuda/docs/08-virtualization-approaches.md`](../userspace/cuda/docs/08-virtualization-approaches.md) | forwarding options and the honest current stage | +| [`../docs/README.md`](../docs/README.md) | the project documentation front door | |