summaryrefslogtreecommitdiffstats
path: root/userspace/cuda/README.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:04 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:04 -0400
commitf874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (patch)
tree2d07208f789e402686ac960e5ad7456b6fd21cfd /userspace/cuda/README.md
parentae18e08d02500579f084616874d8f2ae57f8281b (diff)
downloadopen-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.tar.gz
open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.zip
userspace/cuda: add CUDA userspace driver analysis, tooling and name maps
Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree).
Diffstat (limited to 'userspace/cuda/README.md')
-rw-r--r--userspace/cuda/README.md124
1 files changed, 124 insertions, 0 deletions
diff --git a/userspace/cuda/README.md b/userspace/cuda/README.md
new file mode 100644
index 0000000..e949d0e
--- /dev/null
+++ b/userspace/cuda/README.md
@@ -0,0 +1,124 @@
+# CUDA userspace driver analysis (Open-IOV)
+
+This directory contains the Open-IOV analysis of the NVIDIA CUDA userspace
+driver (`libcuda.so.<version>`): what it is, how it is built, what it asks of
+the kernel driver and the GPU, how that maps onto Open-IOV's emulation and
+passthrough model, and the tooling to repeat the analysis on future driver
+versions.
+
+It exists to make virtualized CUDA **work correctly and fast** in Open-IOV
+VMs and to keep working across driver/firmware/hardware changes. It is **not**
+a CUDA reimplementation and not a way to produce an open-source CUDA: we
+disassemble and decompile the vendor userspace driver to understand the
+interface it expects, and we publish our own descriptions, tables and tools.
+Decompiled output is never committed.
+
+## Start here
+
+> **New here (agent or human)? Read [`READ-ME-FIRST.md`](READ-ME-FIRST.md)
+> first** — it is a reading guide: what to read in what order, what each
+> document gives you, fast lookup recipes, and the two findings that change
+> how you debug (Ghidra image base and the `cuCtxCreate` path — crash solved,
+> engine topology blocking).
+
+| Document | What it covers |
+|---|---|
+| [`docs/01-architecture.md`](docs/01-architecture.md) | the CUDA internal architecture: layers, object model, threads, memory, channels, launches, sync |
+| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) |
+| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts |
+| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security |
+| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker |
+| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas |
+| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version |
+| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands |
+
+Per-version material lives in version directories (`570.144/`), including the
+crash-focused debug helpers (`570.144/scripts/`) and the original
+`cuCtxCreate` write-up (`570.144/docs/ANALYSIS-cuctxcreate-segv.md`).
+
+## Status (honest)
+
+* **570.144 decompilation: complete for the whole library.** A full Ghidra
+ auto-analysis produced 29,494 functions, 659 exported functions, 77,742 call edges and
+ a complete decompiled C corpus; the corpus is machine-generated analysis
+ output and is therefore kept out of git (see "Working with decompiled
+ output" below).
+* **Name recovery: partial.** Exports are exact; internal function names are
+ recovered from call structure, string usage and cross-version matching. The
+ name map is deliberately a separate, reviewable artifact
+ (`570.144/annotations/names.tsv`) so it accumulates across versions.
+* **Crash path (`cuCtxCreate`): solved.** The alloc reply must echo the
+ request's params: the guest driver copies 16 bytes of class params back over
+ the caller's 8-byte local, which zeroed the engine hook's saved `rbp` and
+ faulted at `libcuda+0x453f90`. Echoing the request (only `status` forced to
+ `NV_OK`) makes that copy-back idempotent; `cuCtxCreate` no longer segfaults
+ (see `docs/05-...` §3.5).
+* **Current blocker: engine topology.** `cuCtxCreate` now returns `rc=999`
+ (`CUDA_ERROR_UNKNOWN`); the guest's class-filtered engine list has 6 engines
+ (GR + CE0..CE4) where the oracle has 7 — the async copy engine `COPY14`
+ (descriptor `0x793ceb0e`, instance 14, client type `0x38`) is missing, so the
+ walker reads a zeroed secondary record, the alloc is rejected (class `0x41`,
+ st `0x19`) and the context is rolled back (`docs/05` §3.6).
+* **Open-IOV relevance: documented.** The emulation boundary, the controls the
+ driver issues, and the anti-fragility strategy are in `docs/03`/`docs/04`.
+
+## Layout
+
+```
+userspace/cuda/
+├── README.md this file (index + status)
+├── READ-ME-FIRST.md reading guide for agents/debuggers
+├── WORKING-HERE.md coordination note for parallel agents
+├── docs/ public analysis documents
+├── tools/ reusable, version-agnostic tooling
+│ ├── cudatool.py ELF/export/FDE/wrapper/version-delta toolkit
+│ ├── callwalk.py callgraph callee/caller/path walker
+│ ├── annotate_corpus.py annotate + rename the decompiled corpus
+│ ├── ghidra_full_analysis.sh
+│ ├── ghidra_export_index.sh
+│ ├── ghidra_batch_decompile.sh
+│ ├── ghidra_index_merge.sh
+│ ├── ghidra_decompile_at.sh
+│ └── ghidra/ Ghidra Java scripts (analysis, decompile, name map)
+└── 570.144/
+ ├── README.md version-specific debug tooling (existing)
+ ├── docs/ version-specific write-ups
+ ├── scripts/ version-specific helpers
+ ├── ghidra/ version-specific Ghidra script
+ ├── annotations/ our name map + subsystem index (tracked)
+ └── run/ analysis scratch: extracted binary, Ghidra projects,
+ decompiled corpus (all gitignored)
+```
+
+## Working with decompiled output
+
+The decompiled corpus for 570.144 is under
+`570.144/run/analysis/decompiled/` (gitignored):
+
+* `buckets/bucket_<start>_<end>.c` — decompiled C for functions in that
+ program-address range, in address order, each with a header comment giving
+ the function address and `libcuda+0x...` file offset;
+* `index/bucket_<start>_<end>.tsv` + merged `decompiled_index.tsv` —
+ `addr, file_addr, size, name, file, line, status` for every function.
+
+Search the corpus by address or by name:
+
+```
+grep -n "libcuda+0x453ef0" 570.144/run/analysis/decompiled/buckets/*.c
+awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv
+```
+
+**Address convention (important).** Public docs cite *file virtual addresses*
+(`libcuda+0x...`). Ghidra loads these binaries at image base `0x100000`, so
+program addresses inside Ghidra/Ghidra output are `file vaddr + 0x100000`.
+The tooling translates automatically; if you use raw Ghidra scripts, check
+with `tools/ghidra/PrintProgramInfo.java`. An earlier analysis in this tree
+did not account for this and decompiled addresses 0x100000 too low — see
+`docs/05-570.144-decompilation.md`.
+
+## Rules
+
+* Tracked docs describe components and behaviour only.
+* Decompiled/disassembled vendor output is never committed (see
+ `.gitignore`); we cite `libcuda+0x...` offsets and describe behaviour.
+* Every finding lands in `docs/` (structured) and, when deep, `wiki/`.