# READ-ME-FIRST — CUDA analysis guide for agents
You are being pointed here because you are working on CUDA bring-up,
debugging or emulation for Open-IOV and there is now a body of analysis that
will save you hours. This file tells you **what to read, in what order, and
why it matters**, plus the fastest recipes for the questions you are likely to
have.
Everything referenced here lives under `open-iov/userspace/cuda/` and was
written by the CUDA-decompilation analysis (2026-09). Nothing in this area
modifies the guest, the kernel module or the emulation; it is analysis plus
tooling.
---
## 0. Read these two things first (5 minutes, they change how you debug)
1. **`docs/05-570.144-decompilation.md` §1.1 — the address convention.**
Ghidra loads these binaries at **image base `0x100000`**. The original
per-version helper passed file vaddrs straight to Ghidra, so the old
decompiled files in `570.144/run/` (`helper_453ef0.c`, `fn_0x481360.c`, …)
are **the wrong functions** (0x100000 too low). The runtime observations in
`570.144/docs/ANALYSIS-cuctxcreate-segv.md` (gdb traces, live table values,
ioctl traces) are still valid; any conclusion drawn from its decompiled
code must be re-derived. The new tooling translates addresses for you:
`tools/ghidra_decompile_at.sh` and the annotated corpus.
2. **`docs/05` §3 — the `cuCtxCreate` path: crash and engine topology
solved, completion is the frontier.** The old fault at
`libcuda+0x453f90` (helper at `+0x453ef0`) is solved: the alloc reply must
**echo the request's params**, because the driver copies 16 bytes of class
params back over the caller's 8-byte local (the hook's saved `rbp`) — a
zeroed reply is what made `rbp=0` (§3.5). The engine inventory is now
complete (the async copy engine is included); context creation reaches a
completion wait that currently ends in `rc=999` (§3.6 carries the
historical analysis and a status note). §3.3 gives the engine-record layout
(`base+0x2fec`,
`+0x300c+(idx-1)*0x14`, `+0x350c`, `+0x351c+(idx-0x16)*0x14`; fields
+0/+8/+0xc) and §3.7 lists exactly what the emulation should check first.
If you read nothing else, read those two.
## 1. The five-document core (30–45 minutes total)
Read in this order; each builds on the previous one.
| # | Document | Time | What you gain |
|---|---|---|---|
| 1 | `docs/01-architecture.md` | 15 min | The mental model: layers (API → engine → driver-model → hardware), context/device/stream/channel/memory objects, the marker + tracking-semaphore sync model, launch/QMD flow, UVM integration, threading. This is the vocabulary the rest of the docs use. |
| 2 | `docs/03-driver-and-gpu-interface.md` | 15 min | The emulation contract: device nodes and every ioctl CUDA issues, RM object classes, control families, memory/VA/mapping, channels/GPFIFO/USERD/doorbells, semaphores/notifiers/interrupts, UVM, GSP-RM. Ends with an 8-point emulation checklist. |
| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and engine-topology fix, the completion/execution frontier, and how to look anything up in the corpus. |
| 4 | `docs/04-virtualization-and-emulation.md` | 15 min | Design doc: how NVIDIA's vGPU split works, how Open-IOV differs (emulate the interface, execute the work), and the anti-fragility/performance/portability/security principles that constrain any fix you make. |
| 5 | `docs/02-build-and-execution.md` | 10 min | How the library is built (generated API wrappers, entry tables, embedded device code) and how it executes in a process (init sequence, threads, interposition, `/dev` access, why LD_PRELOAD can't see its syscalls). |
## 2. Reference material (use when you need it)
* `docs/06-tooling.md` — every tool, its usage and its cost. The recipes below
are extracts.
* `docs/07-version-upgrade-playbook.md` — how to redo the analysis when the
driver/CUDA/GSP changes. Read §1 if you only need the cheap `delta` triage.
* `docs/08-virtualization-approaches.md` — one page on the plausible execution
approaches (software emulation vs host-RM proxy vs direct submission vs
hybrid), their trade-offs, and the honest current stage of the project.
* `570.144/annotations/README.md` + `names.tsv` + `subsystems.tsv` — our name
map (659 exports, 628 resolved wrappers, verified internals) and subsystem
index. The map is applied to the annotated corpus already.
## 3. Fast recipes (the 90% cases)
**"What does the driver do with the reply we just gave it?"**
→ Find the ioctl/control in the guest trace, then locate the consuming code:
`docs/03` maps controls to subsystems; `docs/05` §4 shows corpus searches.
The annotated corpus headers list callers/callees with names, so you can walk
the path.
**"Where is `libcuda+0xNNNNNN`?"**
```
tools/cudatool.py nearest 570.144/run/libcuda.so.570.144 0xNNNNNN
tools/ghidra_decompile_at.sh 570.144/run/ghidra-full cuda-full \
libcuda.so.570.144 0xNNNNNN /tmp/fn.c
grep -n "libcuda+0xNNNNNN" 570.144/run/analysis/decompiled-annotated/buckets/*.c
```
**"A guest segfault gives me `lib[base+size] ip` — what is it?"**
```
tools/cudatool.py crash 570.144/run/libcuda.so.570.144
```
It prints the file vaddr, containing function start, and nearest export.
**"Which functions call X / does X call?"**
```
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callers 0xNNNNNN
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv callees 0xNNNNNN 2
tools/callwalk.py 570.144/run/analysis/elf-index/callgraph.tsv chain 0xA 0xB
```
**"What changed between driver versions?"**
```
tools/cudatool.py delta
```
Same-generation builds (e.g. 570.144 vs 570.211.01) show the same 659 exports
with small moves — the name map transfers; new/removed exports flag feature
changes.
**"I need the whole picture of a subsystem."**
→ `docs/01` for the model, then `570.144/annotations/subsystems.tsv` for the
known addresses, then the annotated corpus. `docs/05` §5 lists what the
decompiler does *not* recover (types, indirect targets).
## 4. What is where on disk
```
userspace/cuda/
├── READ-ME-FIRST.md ← you are here
├── README.md index + honest status
├── docs/01..07 public analysis (tracked)
├── tools/ reusable tooling (tracked)
│ ├── cudatool.py ELF exports/FDEs/wrappers/delta/fdematch
│ ├── callwalk.py callee/caller/path walker
│ ├── name_candidates.py API-implementation name candidates
│ ├── subsystem_digest.py per-subsystem call-tree digest
│ ├── diff_corpus.py corpus diff across driver versions
│ └── ghidra_*.sh, ghidra/ Ghidra analysis/decompile/name-map scripts
└── 570.144/
├── README.md, docs/, scripts/ version-specific helpers + old write-up
├── annotations/ name map + subsystem index (tracked)
└── run/
├── libcuda.so.570.144 the guest binary (md5 ed5b2fc0…)
├── ghidra-full/ analyzed project (29,494 functions)
├── analysis/elf-index/ functions/exports/callgraph/strings TSVs
├── analysis/decompiled/ raw corpus (29,254 functions, 0 failures)
├── analysis/decompiled-annotated/ corpus + names + call context
└── analysis/subsystem-digest.txt, fdematch-*.txt generated reports
```
## 5. Rules while you work in this area
* **Do not modify anything outside `userspace/cuda/` from analysis work.**
Emulation fixes go through the kernel module and the main handoff.
* **Do not clean `run/`** — it holds hours of analysis. You may
`rm -rf 570.144/run/analysis/decompiled/wk` to reclaim ~2 GB (per-chunk
project copies; the batch decompiler recreates them).
* **Decompiled output is never committed.** It is vendor-derived; the
gitignores in `open-iov/` and `570.144/` cover the binary, projects and
corpus. Tracked artifacts are our own words, names, tables and tools.
* **Write the public description first.** Tracked docs describe components
and behaviour; keep implementation details in the code and its comments.
* **When you fix the emulation**, update `docs/03` (contract) and, if the
interface understanding changed, `docs/05`; record the delta for the
version.
## 6. The shortest possible version
1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash and
engine topology solved; completion/execution frontier) (read now).
2. `docs/01` and `docs/03` — the model and the contract.
3. `docs/04` — the design principles your fix must respect.
4. `docs/06` — the recipes; then `tools/`.