1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
|
# CUDA userspace driver analysis (Open-IOV)
This directory contains the Open-IOV analysis of the NVIDIA CUDA userspace
driver (`libcuda.so.<version>`): what it is, how it is built, what it asks of
the kernel driver and the GPU, how that maps onto Open-IOV's emulation and
passthrough model, and the tooling to repeat the analysis on future driver
versions.
It exists to make virtualized CUDA **work correctly and fast** in Open-IOV
VMs and to keep working across driver/firmware/hardware changes. It is **not**
a CUDA reimplementation and not a way to produce an open-source CUDA: we
disassemble and decompile the vendor userspace driver to understand the
interface it expects, and we publish our own descriptions, tables and tools.
Decompiled output is never committed.
## Start here
> **New here (agent or human)? Read [`READ-ME-FIRST.md`](READ-ME-FIRST.md)
> first** — it is a reading guide: what to read in what order, what each
> document gives you, fast lookup recipes, and the two findings that change
> how you debug (Ghidra image base and the `cuCtxCreate` path — crash solved,
> engine topology blocking).
| Document | What it covers |
|---|---|
| [`docs/01-architecture.md`](docs/01-architecture.md) | the CUDA internal architecture: layers, object model, threads, memory, channels, launches, sync |
| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) |
| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts |
| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security |
| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker |
| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas |
| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version |
| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands |
Per-version material lives in version directories (`570.144/`), including the
crash-focused debug helpers (`570.144/scripts/`) and the original
`cuCtxCreate` write-up (`570.144/docs/ANALYSIS-cuctxcreate-segv.md`).
## Status (honest)
* **570.144 decompilation: complete for the whole library.** A full Ghidra
auto-analysis produced 29,494 functions, 659 exported functions, 77,742 call edges and
a complete decompiled C corpus; the corpus is machine-generated analysis
output and is therefore kept out of git (see "Working with decompiled
output" below).
* **Name recovery: partial.** Exports are exact; internal function names are
recovered from call structure, string usage and cross-version matching. The
name map is deliberately a separate, reviewable artifact
(`570.144/annotations/names.tsv`) so it accumulates across versions.
* **Crash path (`cuCtxCreate`): solved.** The alloc reply must echo the
request's params: the guest driver copies 16 bytes of class params back over
the caller's 8-byte local, which zeroed the engine hook's saved `rbp` and
faulted at `libcuda+0x453f90`. Echoing the request (only `status` forced to
`NV_OK`) makes that copy-back idempotent; `cuCtxCreate` no longer segfaults
(see `docs/05-...` §3.5).
* **Current blocker: engine topology.** `cuCtxCreate` now returns `rc=999`
(`CUDA_ERROR_UNKNOWN`); the guest's class-filtered engine list has 6 engines
(GR + CE0..CE4) where the oracle has 7 — the async copy engine `COPY14`
(descriptor `0x793ceb0e`, instance 14, client type `0x38`) is missing, so the
walker reads a zeroed secondary record, the alloc is rejected (class `0x41`,
st `0x19`) and the context is rolled back (`docs/05` §3.6).
* **Open-IOV relevance: documented.** The emulation boundary, the controls the
driver issues, and the anti-fragility strategy are in `docs/03`/`docs/04`.
## Layout
```
userspace/cuda/
├── README.md this file (index + status)
├── READ-ME-FIRST.md reading guide for agents/debuggers
├── WORKING-HERE.md coordination note for parallel agents
├── docs/ public analysis documents
├── tools/ reusable, version-agnostic tooling
│ ├── cudatool.py ELF/export/FDE/wrapper/version-delta toolkit
│ ├── callwalk.py callgraph callee/caller/path walker
│ ├── annotate_corpus.py annotate + rename the decompiled corpus
│ ├── ghidra_full_analysis.sh
│ ├── ghidra_export_index.sh
│ ├── ghidra_batch_decompile.sh
│ ├── ghidra_index_merge.sh
│ ├── ghidra_decompile_at.sh
│ └── ghidra/ Ghidra Java scripts (analysis, decompile, name map)
└── 570.144/
├── README.md version-specific debug tooling (existing)
├── docs/ version-specific write-ups
├── scripts/ version-specific helpers
├── ghidra/ version-specific Ghidra script
├── annotations/ our name map + subsystem index (tracked)
└── run/ analysis scratch: extracted binary, Ghidra projects,
decompiled corpus (all gitignored)
```
## Working with decompiled output
The decompiled corpus for 570.144 is under
`570.144/run/analysis/decompiled/` (gitignored):
* `buckets/bucket_<start>_<end>.c` — decompiled C for functions in that
program-address range, in address order, each with a header comment giving
the function address and `libcuda+0x...` file offset;
* `index/bucket_<start>_<end>.tsv` + merged `decompiled_index.tsv` —
`addr, file_addr, size, name, file, line, status` for every function.
Search the corpus by address or by name:
```
grep -n "libcuda+0x453ef0" 570.144/run/analysis/decompiled/buckets/*.c
awk -F'\t' '$4=="FUN_0055a670"' 570.144/run/analysis/decompiled/decompiled_index.tsv
```
**Address convention (important).** Public docs cite *file virtual addresses*
(`libcuda+0x...`). Ghidra loads these binaries at image base `0x100000`, so
program addresses inside Ghidra/Ghidra output are `file vaddr + 0x100000`.
The tooling translates automatically; if you use raw Ghidra scripts, check
with `tools/ghidra/PrintProgramInfo.java`. An earlier analysis in this tree
did not account for this and decompiled addresses 0x100000 too low — see
`docs/05-570.144-decompilation.md`.
## Rules
* Tracked docs describe components and behaviour only.
* Decompiled/disassembled vendor output is never committed (see
`.gitignore`); we cite `libcuda+0x...` offsets and describe behaviour.
* Every finding lands in `docs/` (structured) and, when deep, `wiki/`.
|
