From 21b8ebc9db1d0ae649d6406087ea22b54e3b5775 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Wed, 23 Sep 2026 12:11:04 -0400 Subject: userspace/cuda: add CUDA userspace driver analysis, tooling and name maps Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree). --- userspace/cuda/docs/06-tooling.md | 165 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 userspace/cuda/docs/06-tooling.md (limited to 'userspace/cuda/docs/06-tooling.md') diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md new file mode 100644 index 0000000..85d1c8c --- /dev/null +++ b/userspace/cuda/docs/06-tooling.md @@ -0,0 +1,165 @@ +# 06 — Tooling: how to analyse a CUDA userspace driver + +The toolchain in `tools/` is version-agnostic. Point it at any +`libcuda.so.` and it produces the same artifacts as for 570.144. All +tools are stdlib-only or Ghidra scripts; nothing needs network access. + +``` +userspace/cuda/tools/ +├── cudatool.py ELF/interface toolkit (python3, no deps) +├── callwalk.py callee/caller trees and shortest call path +├── annotate_corpus.py annotated + name-substituted corpus +├── ghidra_full_analysis.sh import + full auto-analysis into a project +├── ghidra_export_index.sh functions/exports/callgraph/strings index +├── ghidra_batch_decompile.sh parallel, resumable full decompilation +├── ghidra_decompile_at.sh decompile one function by file vaddr +├── ghidra_index_merge.sh merge batch indexes +└── ghidra/ + ├── ConfigureAnalysis.java analyzer tuning for big stripped binaries + ├── ExportFunctionIndex.java index exporter + ├── BatchDecompile.java chunk decompiler + ├── DecompileAt.java single function (file-vaddr aware) + ├── ApplyNameMap.java apply/transfer a name map + └── PrintProgramInfo.java address convention + stats +``` + +## 1. Fast facts without Ghidra — `cudatool.py` + +``` +tools/cudatool.py info # build id, sections, exports, ABI note +tools/cudatool.py exports # addr size name (file vaddrs) +tools/cudatool.py functions # function ranges from .eh_frame_hdr +tools/cudatool.py nearest # function start + nearest export +tools/cudatool.py crash # dmesg -> file vaddr +tools/cudatool.py strings [--grep RE] +tools/cudatool.py note # .note.cuda.fwd_compatibility JSON +tools/cudatool.py delta # see §4 +``` + +Useful for: checking what changed between two driver package versions before +spending time on decompilation; converting a guest dmesg crash to a file vaddr +without gdb; getting a function list in a second. + +### Address conventions + +* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used + everywhere in this directory. +* Ghidra may load the same binary at a non-zero image base. For 570.144 it is + `0x100000`; check with `PrintProgramInfo.java`. The Ghidra wrappers here + translate file vaddr → program address automatically; raw Ghidra scripts do + not. + +## 2. Full analysis — Ghidra + +``` +# 1. import + analyze (keep the project; can take minutes to hours) +tools/ghidra_full_analysis.sh + +# 2. export the index (fast, read-only) +tools/ghidra_export_index.sh + +# 3. decompile everything (parallel, resumable; safe to re-run) +tools/ghidra_batch_decompile.sh \ + [chunk_hex] [jobs] [timeout_s] +tools/ghidra_index_merge.sh +``` + +Notes: + +* `ConfigureAnalysis.java` disables the two expensive identifier analyzers + (parameter-ID, function-ID) and keeps everything needed for a stripped + binary. Full analysis of 71 MB then takes minutes, not hours. +* `ghidra_batch_decompile.sh` gives **every chunk its own project copy** under + `/wk/` because Ghidra takes an exclusive project lock even in + read-only mode; completed chunks are skipped on re-run. Delete `wk/` to + reclaim disk. +* The program name passed to Ghidra is the binary's file name + (`libcuda.so.570.144`). +* Do **not** pass `-analysisTimeoutPerFile 0`: in this Ghidra build that means + "abort analysis immediately" (the original per-version helper in + `570.144/scripts/` did this and produced an analysed-looking project with no + analysis at all). + +## 2b. Annotating the corpus + +``` +tools/annotate_corpus.py +``` + +Rewrites every function header with the recovered name, file address, size and +direct callers/callees (named where the map has them), and substitutes +recovered names for `FUN_` references in the bodies. Runs in seconds for +30k functions; the output is gitignored like the raw corpus. + +## 2c. Walking the callgraph without Ghidra + +``` +tools/callwalk.py /callgraph.tsv callees 0x31b7f0 2 10 +tools/callwalk.py /callgraph.tsv callers 0x453ef0 +tools/callwalk.py /callgraph.tsv chain 0x31b7f0 0x2cb940 +``` + +Addresses are file vaddrs; `names.tsv` next to the callgraph is used for +labels. + +## 3. Looking things up + +``` +# one function by file vaddr +tools/ghidra_decompile_at.sh 0x45a670 out.c + +# corpus search by file address (headers are annotated) +grep -n "libcuda+0x45a670" /buckets/*.c + +# index lookups +awk -F'\t' '$2=="0x45a670"' /decompiled_index.tsv +grep -P "^0x0055a670\t" /callgraph.tsv # direct callers/callees + +# exports/strings +grep cuLaunchKernel /exports.tsv +``` + +## 4. Version deltas — `cudatool.py delta` + +``` +tools/cudatool.py delta +``` + +Reports, at the interface level: added/removed/moved/resized exports, import +count changes, `.text` growth, build-id and the CUDA/ABI note. This is the +first thing to run on a new driver release; it tells you immediately whether +the update is a patch (nothing moved), a feature release (new exports) or a +reshuffle (many moves). + +## 5. Name transfer — `ApplyNameMap.java` + +``` +tools/ghidra_decompile_at.sh-free invocation: + analyzeHeadless -process -noanalysis \ + -scriptPath tools/ghidra -postScript ApplyNameMap.java [--dry-run] +``` + +The map is `file_vaddrname` per line (comments allowed). Name maps live +in version directories (`570.144/annotations/names.tsv`) and are tracked, so +naming work accumulates: apply the previous version's map to a new binary, +then fix the misses. `--dry-run` reports how many names would land. + +## 6. Writing new Ghidra scripts + +Follow the existing pattern: a `GhidraScript` subclass in `tools/ghidra/`, +args via `getScriptArgs()`, and file-vaddr ↔ program-address translation via +`currentProgram.getImageBase()` (see `DecompileAt.java`). Keep outputs under +the version's gitignored `run/` tree unless the artifact is our own work +product (name maps, indexes of our findings), which may be tracked. + +## 7. Cost guide (measured on 570.144, 16 cores) + +| Step | Time | Disk | +|---|---|---| +| import alone | ~1 min | 324 MB project | +| full analysis | ~10–15 min | same project | +| index export | ~1–2 min | ~25 MB TSV | +| full decompilation (5 workers) | ~20–30 min | ~3 GB C + 2 GB project copies | + +Notes for a rerun: set `GHIDRA_HEADLESS_MAXMEM` (default 12G for analysis, +4G per batch job) and keep total batch jobs ≤ 5–6 on a 32 GB machine. -- cgit v1.2.3