From 21b8ebc9db1d0ae649d6406087ea22b54e3b5775 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Wed, 23 Sep 2026 12:11:04 -0400 Subject: userspace/cuda: add CUDA userspace driver analysis, tooling and name maps Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree). --- userspace/cuda/docs/02-build-and-execution.md | 190 ++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 userspace/cuda/docs/02-build-and-execution.md (limited to 'userspace/cuda/docs/02-build-and-execution.md') diff --git a/userspace/cuda/docs/02-build-and-execution.md b/userspace/cuda/docs/02-build-and-execution.md new file mode 100644 index 0000000..9fc2f02 --- /dev/null +++ b/userspace/cuda/docs/02-build-and-execution.md @@ -0,0 +1,190 @@ +# 02 — How the CUDA userspace driver is built and how it executes + +This document explains the compilation model behind `libcuda.so` and the +runtime model of the resulting binary: what is generated, what is compiled in, +what the loader does, and what the driver does from process start to first +kernel launch. It is written to be re-derivable on any driver version with the +tooling in `tools/`. + +--- + +## 1. Build model + +### 1.1 One library, many components + +The driver is one shared object assembled from several thousand translation +units. In architecture terms the inputs are: + +| Input group | Nature | Notes | +|---|---|---| +| API wrappers | **generated** C | one exported function per API name/version variant; adds destroyed-state check, optional serialization, tracing/interposition dispatch | +| API implementation | hand-written C | one module per API domain | +| CUDA internal engine | hand-written C | the bulk of the driver | +| Driver-model backends | hand-written C/C++ | RM/SoC/simulator/MPS/WDDM behind one function-table interface | +| Hardware layer | hand-written C | per-architecture method/QMD encoders | +| Device code | **assembly and CUDA C** compiled to cubins, embedded as byte arrays | memcpy/memset kernels, syscall trampolines, trap/exit handlers, preemption, graph scheduler | +| Export lists | text | symbol name + platform flags + the CUDA version the symbol appeared in | +| ABI adaptation | **one compiled instance per kernel-driver ABI** | same source compiled repeatedly with different ABI headers and symbol localization | +| Generators | Perl/Python/Java | produce wrappers, proc-address tables, entry tables, interposition tables, error tables, abstraction-layer headers | + +Two consequences matter for analysis: + +* **The public ABI is data-driven.** Adding an API function is mostly an entry + in a list plus a hand-written implementation; the exported symbol, + the tracing callback ID, the `cuGetProcAddress` entry and the interposition + slot are generated from that list. A new driver version usually moves this + data, not the architecture. +* **Versioning is explicit.** Symbols carry `_vN` suffixes and a + "since CUDA X.Y" attribute; per-thread default stream variants (`_ptsz`) + exist for many calls; `cuGetProcAddress` selects by requested version. + +### 1.2 Embedded device code + +The library carries its own GPU code. Device assembly sources are preprocessed, +assembled per architecture, converted to C arrays and linked into the library; +CUDA C device sources are compiled with the toolchain's device compiler. At +runtime the driver treats them like user modules: it extracts the image for +the current architecture and loads it through its own module machinery. This +is why a stripped `libcuda.so` still contains many small ELF images and +`.nv.*` section names in `.rodata` — those are cubins, not host code. + +### 1.3 Build flavors + +* `release` separates debug info into a `.debug` file (which NVIDIA does not + ship in the driver packages we analyse, hence the fully stripped binary). +* `develop`/`debug` add logging and assertions; some entry tables (test + interfaces) are present only outside public-release builds. +* A "safety" configuration removes the tooling dispatch entirely (exports call + the implementation directly). +* The kernel-module build of the same sources exists but is not what runs in + a guest. + +### 1.4 What the produced binary looks like + +For 570.144 (`cudatool.py info`): + +``` +build id : c0e1e77ea249944ac4fd43faebed9498a729606a +exec segment base: 0x179000 +.text : 0x179ec0 .. 0xe94c64 (13,741,476 bytes) +exports : 659 imports: 243 (libc/libpthread/libdl/librt) +cuda note : CUDA 12.8, driver ABI [470,535,550,560,565,570], device [1,2,7..14] +debuglink : libcuda.so.1.1.debug (not shipped) +``` + +Sections of interest: + +| Section | Why it matters | +|---|---| +| `.dynsym`/`.dynstr` | the 659 exports and 243 imports; no local symbols (stripped) | +| `.text` | all host code | +| `.rodata` (53 MB) | error strings, dispatch/version tables, **embedded cubins** | +| `.data.rel.ro` (~900 KB) | relocated pointer tables (entry tables, vtables, function tables) | +| `.eh_frame`/`.eh_frame_hdr` | unwind info; also the best stripped-binary function map (28,925 FDEs) | +| `.note.cuda.fwd_compatibility` | CUDA version + supported kernel-driver ABI list | +| `.gnu_debuglink` | name/CRC of the (absent) debug file | + +## 2. Execution model + +### 2.1 Load and initialization + +1. The dynamic loader maps the four PT_LOAD segments and relocates + `.rela.dyn`/`.rela.plt`. The library has no NVIDIA userspace dependencies + for the CUDA path; everything below it is a device file. +2. C++-style constructors registered in `.init_array` run. They set up global + mutexes and install an `atexit`/destructor that marks the driver destroyed. +3. The first `cuInit` runs the initialization state machine: + * initialize TLS and globals; select the driver-model backend; + * open the kernel driver's control device and create the process-wide + **kernel client** (one per process, not per context); + * enumerate and attach GPUs, build the device list (chip, caps, limits, + engine lists, memory sizes, BAR info); + * initialize the UVM userspace layer if the platform supports it; + * load embedded device code caches lazily. +4. Subsequent API calls use the per-thread context stack; the first + `cuCtxCreate` builds the heavy per-context state. + +An `LD_PRELOAD` shim cannot observe the driver's kernel traffic: the RM client +issues **raw `syscall(SYS_ioctl)`**, not libc `ioctl`. Instrumentation is done +with `ptrace` (see `tools/guest-diag/ioctl_trace.c` in the parent tree) or +with kernel-side tracing. + +### 2.2 Interposition and tooling dispatch + +Every exported call goes through a global function table before reaching the +implementation. A tool library can register an interposition layer *before* +`cuInit` and receive the original pointers to chain to. This is the sanctioned +mechanism used by profilers/debuggers, and it is also a clean hook point for +A/B instrumentation in the guest. + +### 2.3 Entry tables + +`cuGetExportTable(uuid)` returns pointers to internal tables for: + +* the CUDA runtime library (fatbin loading, primary-context initialization, + hidden module-load paths, kernel launch with internal parameters); +* tooling (profiling, debugging, memory checking, launch tracking); +* debugger support (context-local storage, trap handlers); +* interop and special platforms (video, NVN, OpenCL bridges). + +Tables are identified by 128-bit UUIDs and start with a size field, so old +clients and new drivers coexist. This is a key fact for emulation: the guest +runtime/tools may call *through these tables* instead of the public API, and +all of it still resolves to the same internal engine. + +### 2.4 Threading and process model + +* API calls are caller-threaded; each context has a recursive RW lock. +* One ISR thread per context handles interrupts/DPC callbacks. +* One boost thread per context handles drop-to-idle clock policy. +* MPS (multi-process service) inserts a client/server IPC layer *below* the + normal API: clients share the server's channels; the wire format is a + private socket protocol, not the public API. +* Inter-process memory sharing uses kernel-driver exported handles and + (optionally) POSIX file descriptors; CUDA's own IPC mem handles are an RM + object plus a key. + +### 2.5 What the driver touches at runtime (overview) + +| Resource | Access | Purpose | +|---|---|---| +| `/dev/nvidiactl` | ioctl | control device: version negotiation, client creation, GPU attach, OS events, exported-object fds | +| `/dev/nvidiaN` | open/ioctl/mmap | per-GPU: object alloc/free, controls, memory alloc/map, interrupts | +| `/dev/nvidia-uvm` | open/ioctl | managed memory, page migration, channel registration (via UVM userspace API) | +| BAR regions | mmap | user-mode doorbells (USERD), interrupt tree/usermode region, framebuffer apertures | +| sysmem | mmap/alloc | command buffers, semaphores, page tables (when the driver owns them) | +| shared memory / sockets | IPC | MPS, IPC events/memory | + +The exact ioctl vocabulary observed from the 570.144 guest is enumerated in +`docs/03-driver-and-gpu-interface.md`. + +## 3. Running in a guest (what Open-IOV sees) + +* The library is **unmodified** in the guest; it links against the guest's + kernel driver (`nvidia.ko` + `nvidia-uvm.ko`) exactly as on bare metal. +* The kernel driver is also unmodified; it talks to the emulated PCI function + and the GSP-RM interface that Open-IOV emulates. +* Therefore, from libcuda's point of view, the guest is a normal NVIDIA + machine whose kernel driver has certain capabilities and whose GPU answers + RM controls. The crash class we chase (`cuCtxCreate` segfault) is a + *userspace* consequence of *kernel/GSP* replies that differ from bare metal. +* The library's own state is therefore the best available oracle: when the + driver misbehaves, comparing its state (or its control traffic) against a + bare-metal A/B run localizes the emulation gap. The A/B harness in + `tools/cuda-ab/` does this at the workload level; `docs/05` does it at the + function level. + +## 4. Reproducing the build/execution analysis for a new version + +``` +tools/cudatool.py info # size, exports, ABI note +tools/cudatool.py exports # symbol map +tools/cudatool.py functions # FDE function map +tools/cudatool.py delta # interface-level diff +tools/ghidra_full_analysis.sh # full auto-analysis +tools/ghidra_export_index.sh ... # function/call/string index +tools/ghidra_batch_decompile.sh ... # decompiled corpus +``` + +See `docs/06-tooling.md` for details and `docs/07-version-upgrade-playbook.md` +for the full procedure. -- cgit v1.2.3