1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
# READ-ME-FIRST — CUDA analysis guide for agents
You are being pointed here because you are working on CUDA bring-up,
debugging or emulation for Open-IOV and there is now a body of analysis that
will save you hours. This file tells you **what to read, in what order, and
why it matters**, plus the fastest recipes for the questions you are likely to
have.
Everything referenced here lives under `open-iov/userspace/cuda/` and was
written by the CUDA-decompilation analysis (2026-09). Nothing in this area
modifies the guest, the kernel module or the emulation; it is analysis plus
tooling.
---
## 0. Read these two things first (5 minutes, they change how you debug)
1. **`docs/05-570.144-decompilation.md` §1.1 — the address convention.**
Ghidra loads these binaries at **image base `0x100000`**. The original
per-version helper passed file vaddrs straight to Ghidra, so the old
decompiled files in `570.144/run/` (`helper_453ef0.c`, `fn_0x481360.c`, …)
are **the wrong functions** (0x100000 too low). The runtime observations in
`570.144/docs/ANALYSIS-cuctxcreate-segv.md` (gdb traces, live table values,
ioctl traces) are still valid; any conclusion drawn from its decompiled
code must be re-derived. The new tooling translates addresses for you:
`tools/ghidra_decompile_at.sh` and the annotated corpus.
2. **`docs/05` §3 — the `cuCtxCreate` path: crash and engine topology
solved, completion is the frontier.** The old fault at
`libcuda+0x453f90` (helper at `+0x453ef0`) is solved: the alloc reply must
**echo the request's params**, because the driver copies 16 bytes of class
params back over the caller's 8-byte local (the hook's saved `rbp`) — a
zeroed reply is what made `rbp=0` (§3.5). The engine inventory is now
complete (the async copy engine is included); context creation reaches a
completion wait that currently ends in `rc=999` (§3.6 carries the
historical analysis and a status note). §3.3 gives the engine-record layout
(`base+0x2fec`,
`+0x300c+(idx-1)*0x14`, `+0x350c`, `+0x351c+(idx-0x16)*0x14`; fields
+0/+8/+0xc) and §3.7 lists exactly what the emulation should check first.
If you read nothing else, read those two.
## 1. The five-document core (30–45 minutes total)
Read in this order; each builds on the previous one.
| # | Document | Time | What you gain |
|---|---|---|---|
| 1 | `docs/01-architecture.md` | 15 min | The mental model: layers (API → engine → driver-model → hardware), context/device/stream/channel/memory objects, the marker + tracking-semaphore sync model, launch/QMD flow, UVM integration, threading. This is the vocabulary the rest of the docs use. |
| 2 | `docs/03-driver-and-gpu-interface.md` | 15 min | The emulation contract: device nodes and every ioctl CUDA issues, RM object classes, control families, memory/VA/mapping, channels/GPFIFO/USERD/doorbells, semaphores/notifiers/interrupts, UVM, GSP-RM. Ends with an 8-point emulation checklist. |
| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and engine-topology fix, the completion/execution frontier, and how to look anything up in the corpus. |
| 4 | `docs/04-virtualization-and-emulation.md` | 15 min | Design doc: how NVIDIA's vGPU split works, how Open-IOV differs (emulate the interface, execute the work), and the anti-fragility/performance/portability/security principles that constrain any fix you make. |
| 5 | `docs/02-build-and-execution.md` | 10 min | How the library is built (generated API wrappers, entry tables, embedded device code) and how it executes in a process (init sequence, threads, interposition, `/dev` access, why LD_PRELOAD can't see its syscalls). |
## 2. Reference material (use when you need it)
* `docs/06-tooling.md` — every tool, its usage and its cost. The recipes below
are extracts.
* `docs/07-version-upgrade-playbook.md` — how to redo the analysis when the
driver/CUDA/GSP changes. Read §1 if you only need the cheap `delta` triage.
* `docs/08-virtualization-approaches.md` — one page on the plausible execution
approaches (software emulation vs host-RM proxy vs direct submission vs
hybrid), their trade-offs, and the honest current stage of the project.
* `570.144/annotations/README.md` + `names.tsv` + `subsystems.tsv` — our name
map (659 exports, 628 resolved wrappers, verified internals) and subsystem
index. The map is applied to the annotated corpus already.
## 3. Fast recipes (the 90% cases)
**"What does the driver do with the reply we just gave it?"**
→ Find the ioctl/control in the guest trace, then locate the consuming code:
`docs/03` maps controls to subsystems; `docs/05` §4 shows corpus searches.
The annotated corpus headers list callers/callees with names, so you can walk
the path.
**"Where is `libcuda+0xNNNNNN`?"**
```
tools/cudatool.py nearest 570.144/run/libcuda.so.570.144 0xNNNNNN
tools/ghidra_decompile_at.sh 570.144/run/ghidra-full cuda-full \
libcuda.so.570.144 0xNNNNNN /tmp/fn.c
grep -n "libcuda+0xNNNNNN" 570.144/run/analysis/decompiled-annotated/buckets/*.c
```
**"A guest segfault gives me `lib[base+size] ip` — what is it?"**
```
tools/cudatool.py crash 570.144/run/libcuda.so.570.144 <base> <ip>
```
It prints the file vaddr, containing function start, and nearest export.
**"Which functions call X / does X call?"**
```
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callers 0xNNNNNN
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callees 0xNNNNNN 2
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv chain 0xA 0xB
```
**"What changed between driver versions?"**
```
tools/cudatool.py delta <old libcuda> <new libcuda>
```
Same-generation builds (e.g. 570.144 vs 570.211.01) show the same 659 exports
with small moves — the name map transfers; new/removed exports flag feature
changes.
**"I need the whole picture of a subsystem."**
→ `docs/01` for the model, then `570.144/annotations/subsystems.tsv` for the
known addresses, then the annotated corpus. `docs/05` §5 lists what the
decompiler does *not* recover (types, indirect targets).
## 4. What is where on disk
```
userspace/cuda/
├── READ-ME-FIRST.md ← you are here
├── README.md index + honest status
├── docs/01..07 public analysis (tracked)
├── tools/ reusable tooling (tracked)
│ ├── cudatool.py ELF exports/FDEs/wrappers/delta/fdematch
│ ├── callwalk.py callee/caller/path walker
│ ├── name_candidates.py API-implementation name candidates
│ ├── subsystem_digest.py per-subsystem call-tree digest
│ ├── diff_corpus.py corpus diff across driver versions
│ └── ghidra_*.sh, ghidra/ Ghidra analysis/decompile/name-map scripts
└── 570.144/
├── README.md, docs/, scripts/ version-specific helpers + old write-up
├── annotations/ name map + subsystem index (tracked)
└── run/
├── libcuda.so.570.144 the guest binary (md5 ed5b2fc0…)
├── ghidra-full/ analyzed project (29,494 functions)
├── analysis/elf-index/ functions/exports/callgraph/strings TSVs
├── analysis/decompiled/ raw corpus (29,254 functions, 0 failures)
├── analysis/decompiled-annotated/ corpus + names + call context
└── analysis/subsystem-digest.txt, fdematch-*.txt generated reports
```
## 5. Rules while you work in this area
* **Do not modify anything outside `userspace/cuda/` from analysis work.**
Emulation fixes go through the kernel module and the main handoff.
* **Do not clean `run/`** — it holds hours of analysis. You may
`rm -rf 570.144/run/analysis/decompiled/wk` to reclaim ~2 GB (per-chunk
project copies; the batch decompiler recreates them).
* **Decompiled output is never committed.** It is vendor-derived; the
gitignores in `open-iov/` and `570.144/` cover the binary, projects and
corpus. Tracked artifacts are our own words, names, tables and tools.
* **Write the public description first.** Tracked docs describe components
and behaviour; keep implementation details in the code and its comments.
* **When you fix the emulation**, update `docs/03` (contract) and, if the
interface understanding changed, `docs/05`; record the delta for the
version.
## 6. The shortest possible version
1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash and
engine topology solved; completion/execution frontier) (read now).
2. `docs/01` and `docs/03` — the model and the contract.
3. `docs/04` — the design principles your fix must respect.
4. `docs/06` — the recipes; then `tools/`.
|
