diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:04 -0400 |
| commit | f874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (patch) | |
| tree | 2d07208f789e402686ac960e5ad7456b6fd21cfd /userspace/cuda/docs/06-tooling.md | |
| parent | ae18e08d02500579f084616874d8f2ae57f8281b (diff) | |
| download | open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.tar.gz open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.zip | |
userspace/cuda: add CUDA userspace driver analysis, tooling and name maps
Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs:
* docs/01-08: architecture (layers, object model, threads, sync, memory,
channels, launch), build and execution model, the driver/kernel/GPU
interface contract, virtualization approaches, the 570.144 decompilation
map with the corrected libcuda crash path and the current engine-topology
blocker, tooling, and a version-upgrade playbook.
* tools/: version-agnostic toolchain - stdlib ELF/interface toolkit
(exports, unwind-table function map, wrapper resolution, version delta,
crash address decode), callgraph walker, corpus annotator, and Ghidra
analysis/decompilation/name-map scripts with resumable parallel drivers.
* 570.144/annotations/: our name map (exports + resolved wrappers + verified
internals) and subsystem index, tracked so naming work accumulates.
* READ-ME-FIRST.md: reading guide with the fast lookup recipes.
The full decompiled corpus and annotated corpus stay untracked (vendor-
derived, gitignored under the version run/ tree).
Diffstat (limited to 'userspace/cuda/docs/06-tooling.md')
| -rw-r--r-- | userspace/cuda/docs/06-tooling.md | 165 |
1 files changed, 165 insertions, 0 deletions
diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md new file mode 100644 index 0000000..85d1c8c --- /dev/null +++ b/userspace/cuda/docs/06-tooling.md @@ -0,0 +1,165 @@ +# 06 — Tooling: how to analyse a CUDA userspace driver + +The toolchain in `tools/` is version-agnostic. Point it at any +`libcuda.so.<version>` and it produces the same artifacts as for 570.144. All +tools are stdlib-only or Ghidra scripts; nothing needs network access. + +``` +userspace/cuda/tools/ +├── cudatool.py ELF/interface toolkit (python3, no deps) +├── callwalk.py callee/caller trees and shortest call path +├── annotate_corpus.py annotated + name-substituted corpus +├── ghidra_full_analysis.sh import + full auto-analysis into a project +├── ghidra_export_index.sh functions/exports/callgraph/strings index +├── ghidra_batch_decompile.sh parallel, resumable full decompilation +├── ghidra_decompile_at.sh decompile one function by file vaddr +├── ghidra_index_merge.sh merge batch indexes +└── ghidra/ + ├── ConfigureAnalysis.java analyzer tuning for big stripped binaries + ├── ExportFunctionIndex.java index exporter + ├── BatchDecompile.java chunk decompiler + ├── DecompileAt.java single function (file-vaddr aware) + ├── ApplyNameMap.java apply/transfer a name map + └── PrintProgramInfo.java address convention + stats +``` + +## 1. Fast facts without Ghidra — `cudatool.py` + +``` +tools/cudatool.py info <lib> # build id, sections, exports, ABI note +tools/cudatool.py exports <lib> # addr size name (file vaddrs) +tools/cudatool.py functions <lib> # function ranges from .eh_frame_hdr +tools/cudatool.py nearest <lib> <addr> # function start + nearest export +tools/cudatool.py crash <lib> <map_base> <ip> # dmesg -> file vaddr +tools/cudatool.py strings <lib> [--grep RE] +tools/cudatool.py note <lib> # .note.cuda.fwd_compatibility JSON +tools/cudatool.py delta <libA> <libB> # see §4 +``` + +Useful for: checking what changed between two driver package versions before +spending time on decompilation; converting a guest dmesg crash to a file vaddr +without gdb; getting a function list in a second. + +### Address conventions + +* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used + everywhere in this directory. +* Ghidra may load the same binary at a non-zero image base. For 570.144 it is + `0x100000`; check with `PrintProgramInfo.java`. The Ghidra wrappers here + translate file vaddr → program address automatically; raw Ghidra scripts do + not. + +## 2. Full analysis — Ghidra + +``` +# 1. import + analyze (keep the project; can take minutes to hours) +tools/ghidra_full_analysis.sh <lib> <project_dir> <project_name> + +# 2. export the index (fast, read-only) +tools/ghidra_export_index.sh <project_dir> <project_name> <program> <outdir> + +# 3. decompile everything (parallel, resumable; safe to re-run) +tools/ghidra_batch_decompile.sh <project_dir> <project_name> <program> <outdir> \ + <start_hex> <end_hex> [chunk_hex] [jobs] [timeout_s] +tools/ghidra_index_merge.sh <outdir> +``` + +Notes: + +* `ConfigureAnalysis.java` disables the two expensive identifier analyzers + (parameter-ID, function-ID) and keeps everything needed for a stripped + binary. Full analysis of 71 MB then takes minutes, not hours. +* `ghidra_batch_decompile.sh` gives **every chunk its own project copy** under + `<outdir>/wk/` because Ghidra takes an exclusive project lock even in + read-only mode; completed chunks are skipped on re-run. Delete `wk/` to + reclaim disk. +* The program name passed to Ghidra is the binary's file name + (`libcuda.so.570.144`). +* Do **not** pass `-analysisTimeoutPerFile 0`: in this Ghidra build that means + "abort analysis immediately" (the original per-version helper in + `570.144/scripts/` did this and produced an analysed-looking project with no + analysis at all). + +## 2b. Annotating the corpus + +``` +tools/annotate_corpus.py <outdir> <indexdir> <names.tsv> <annotated_outdir> +``` + +Rewrites every function header with the recovered name, file address, size and +direct callers/callees (named where the map has them), and substitutes +recovered names for `FUN_<addr>` references in the bodies. Runs in seconds for +30k functions; the output is gitignored like the raw corpus. + +## 2c. Walking the callgraph without Ghidra + +``` +tools/callwalk.py <indexdir>/callgraph.tsv callees 0x31b7f0 2 10 +tools/callwalk.py <indexdir>/callgraph.tsv callers 0x453ef0 +tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940 +``` + +Addresses are file vaddrs; `names.tsv` next to the callgraph is used for +labels. + +## 3. Looking things up + +``` +# one function by file vaddr +tools/ghidra_decompile_at.sh <project_dir> <project_name> <program> 0x45a670 out.c + +# corpus search by file address (headers are annotated) +grep -n "libcuda+0x45a670" <outdir>/buckets/*.c + +# index lookups +awk -F'\t' '$2=="0x45a670"' <outdir>/decompiled_index.tsv +grep -P "^0x0055a670\t" <indexdir>/callgraph.tsv # direct callers/callees + +# exports/strings +grep cuLaunchKernel <indexdir>/exports.tsv +``` + +## 4. Version deltas — `cudatool.py delta` + +``` +tools/cudatool.py delta <libA> <libB> +``` + +Reports, at the interface level: added/removed/moved/resized exports, import +count changes, `.text` growth, build-id and the CUDA/ABI note. This is the +first thing to run on a new driver release; it tells you immediately whether +the update is a patch (nothing moved), a feature release (new exports) or a +reshuffle (many moves). + +## 5. Name transfer — `ApplyNameMap.java` + +``` +tools/ghidra_decompile_at.sh-free invocation: + analyzeHeadless <project_dir> <project_name> -process <program> -noanalysis \ + -scriptPath tools/ghidra -postScript ApplyNameMap.java <names.tsv> [--dry-run] +``` + +The map is `file_vaddr<TAB>name` per line (comments allowed). Name maps live +in version directories (`570.144/annotations/names.tsv`) and are tracked, so +naming work accumulates: apply the previous version's map to a new binary, +then fix the misses. `--dry-run` reports how many names would land. + +## 6. Writing new Ghidra scripts + +Follow the existing pattern: a `GhidraScript` subclass in `tools/ghidra/`, +args via `getScriptArgs()`, and file-vaddr ↔ program-address translation via +`currentProgram.getImageBase()` (see `DecompileAt.java`). Keep outputs under +the version's gitignored `run/` tree unless the artifact is our own work +product (name maps, indexes of our findings), which may be tracked. + +## 7. Cost guide (measured on 570.144, 16 cores) + +| Step | Time | Disk | +|---|---|---| +| import alone | ~1 min | 324 MB project | +| full analysis | ~10–15 min | same project | +| index export | ~1–2 min | ~25 MB TSV | +| full decompilation (5 workers) | ~20–30 min | ~3 GB C + 2 GB project copies | + +Notes for a rerun: set `GHIDRA_HEADLESS_MAXMEM` (default 12G for analysis, +4G per batch job) and keep total batch jobs ≤ 5–6 on a 32 GB machine. |
