diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
| commit | 21b8ebc9db1d0ae649d6406087ea22b54e3b5775 (patch) | |
| tree | 192aa5d3c76e0cf3a785936d9902c6848609d37e /userspace/cuda/README.md | |
| parent | 631bd20c63a4058775a4fa549b9473915c918020 (diff) | |
| download | open-iov-21b8ebc9db1d0ae649d6406087ea22b54e3b5775.tar.gz open-iov-21b8ebc9db1d0ae649d6406087ea22b54e3b5775.zip | |
userspace/cuda: add CUDA userspace driver analysis, tooling and name maps
Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs:
* docs/01-08: architecture (layers, object model, threads, sync, memory,
channels, launch), build and execution model, the driver/kernel/GPU
interface contract, virtualization approaches, the 570.144 decompilation
map with the corrected libcuda crash path and the current engine-topology
blocker, tooling, and a version-upgrade playbook.
* tools/: version-agnostic toolchain - stdlib ELF/interface toolkit
(exports, unwind-table function map, wrapper resolution, version delta,
crash address decode), callgraph walker, corpus annotator, and Ghidra
analysis/decompilation/name-map scripts with resumable parallel drivers.
* 570.144/annotations/: our name map (exports + resolved wrappers + verified
internals) and subsystem index, tracked so naming work accumulates.
* READ-ME-FIRST.md: reading guide with the fast lookup recipes.
The full decompiled corpus and annotated corpus stay untracked (vendor-
derived, gitignored under the version run/ tree).
Diffstat (limited to 'userspace/cuda/README.md')
| -rw-r--r-- | userspace/cuda/README.md | 124 |
1 files changed, 124 insertions, 0 deletions
diff --git a/userspace/cuda/README.md b/userspace/cuda/README.md new file mode 100644 index 0000000..e949d0e --- /dev/null +++ b/userspace/cuda/README.md @@ -0,0 +1,124 @@ +# CUDA userspace driver analysis (Open-IOV) + +This directory contains the Open-IOV analysis of the NVIDIA CUDA userspace +driver (`libcuda.so.<version>`): what it is, how it is built, what it asks of +the kernel driver and the GPU, how that maps onto Open-IOV's emulation and +passthrough model, and the tooling to repeat the analysis on future driver +versions. + +It exists to make virtualized CUDA **work correctly and fast** in Open-IOV +VMs and to keep working across driver/firmware/hardware changes. It is **not** +a CUDA reimplementation and not a way to produce an open-source CUDA: we +disassemble and decompile the vendor userspace driver to understand the +interface it expects, and we publish our own descriptions, tables and tools. +Decompiled output is never committed. + +## Start here + +> **New here (agent or human)? Read [`READ-ME-FIRST.md`](READ-ME-FIRST.md) +> first** — it is a reading guide: what to read in what order, what each +> document gives you, fast lookup recipes, and the two findings that change +> how you debug (Ghidra image base and the `cuCtxCreate` path — crash solved, +> engine topology blocking). + +| Document | What it covers | +|---|---| +| [`docs/01-architecture.md`](docs/01-architecture.md) | the CUDA internal architecture: layers, object model, threads, memory, channels, launches, sync | +| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) | +| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts | +| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security | +| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker | +| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas | +| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version | +| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands | + +Per-version material lives in version directories (`570.144/`), including the +crash-focused debug helpers (`570.144/scripts/`) and the original +`cuCtxCreate` write-up (`570.144/docs/ANALYSIS-cuctxcreate-segv.md`). + +## Status (honest) + +* **570.144 decompilation: complete for the whole library.** A full Ghidra + auto-analysis produced 29,494 functions, 659 exported functions, 77,742 call edges and + a complete decompiled C corpus; the corpus is machine-generated analysis + output and is therefore kept out of git (see "Working with decompiled + output" below). +* **Name recovery: partial.** Exports are exact; internal function names are + recovered from call structure, string usage and cross-version matching. The + name map is deliberately a separate, reviewable artifact + (`570.144/annotations/names.tsv`) so it accumulates across versions. +* **Crash path (`cuCtxCreate`): solved.** The alloc reply must echo the + request's params: the guest driver copies 16 bytes of class params back over + the caller's 8-byte local, which zeroed the engine hook's saved `rbp` and + faulted at `libcuda+0x453f90`. Echoing the request (only `status` forced to + `NV_OK`) makes that copy-back idempotent; `cuCtxCreate` no longer segfaults + (see `docs/05-...` §3.5). +* **Current blocker: engine topology.** `cuCtxCreate` now returns `rc=999` + (`CUDA_ERROR_UNKNOWN`); the guest's class-filtered engine list has 6 engines + (GR + CE0..CE4) where the oracle has 7 — the async copy engine `COPY14` + (descriptor `0x793ceb0e`, instance 14, client type `0x38`) is missing, so the + walker reads a zeroed secondary record, the alloc is rejected (class `0x41`, + st `0x19`) and the context is rolled back (`docs/05` §3.6). +* **Open-IOV relevance: documented.** The emulation boundary, the controls the + driver issues, and the anti-fragility strategy are in `docs/03`/`docs/04`. + +## Layout + +``` +userspace/cuda/ +├── README.md this file (index + status) +├── READ-ME-FIRST.md reading guide for agents/debuggers +├── WORKING-HERE.md coordination note for parallel agents +├── docs/ public analysis documents +├── tools/ reusable, version-agnostic tooling +│ ├── cudatool.py ELF/export/FDE/wrapper/version-delta toolkit +│ ├── callwalk.py callgraph callee/caller/path walker +│ ├── annotate_corpus.py annotate + rename the decompiled corpus +│ ├── ghidra_full_analysis.sh +│ ├── ghidra_export_index.sh +│ ├── ghidra_batch_decompile.sh +│ ├── ghidra_index_merge.sh +│ ├── ghidra_decompile_at.sh +│ └── ghidra/ Ghidra Java scripts (analysis, decompile, name map) +└── 570.144/ + ├── README.md version-specific debug tooling (existing) + ├── docs/ version-specific write-ups + ├── scripts/ version-specific helpers + ├── ghidra/ version-specific Ghidra script + ├── annotations/ our name map + subsystem index (tracked) + └── run/ analysis scratch: extracted binary, Ghidra projects, + decompiled corpus (all gitignored) +``` + +## Working with decompiled output + +The decompiled corpus for 570.144 is under +`570.144/run/analysis/decompiled/` (gitignored): + +* `buckets/bucket_<start>_<end>.c` — decompiled C for functions in that + program-address range, in address order, each with a header comment giving + the function address and `libcuda+0x...` file offset; +* `index/bucket_<start>_<end>.tsv` + merged `decompiled_index.tsv` — + `addr, file_addr, size, name, file, line, status` for every function. + +Search the corpus by address or by name: + +``` +grep -n "libcuda+0x453ef0" 570.144/run/analysis/decompiled/buckets/*.c +awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv +``` + +**Address convention (important).** Public docs cite *file virtual addresses* +(`libcuda+0x...`). Ghidra loads these binaries at image base `0x100000`, so +program addresses inside Ghidra/Ghidra output are `file vaddr + 0x100000`. +The tooling translates automatically; if you use raw Ghidra scripts, check +with `tools/ghidra/PrintProgramInfo.java`. An earlier analysis in this tree +did not account for this and decompiled addresses 0x100000 too low — see +`docs/05-570.144-decompilation.md`. + +## Rules + +* Tracked docs describe components and behaviour only. +* Decompiled/disassembled vendor output is never committed (see + `.gitignore`); we cite `libcuda+0x...` offsets and describe behaviour. +* Every finding lands in `docs/` (structured) and, when deep, `wiki/`. |
