summaryrefslogtreecommitdiffstats
path: root/userspace/cuda/docs/06-tooling.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:04 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:04 -0400
commitf874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (patch)
tree2d07208f789e402686ac960e5ad7456b6fd21cfd /userspace/cuda/docs/06-tooling.md
parentae18e08d02500579f084616874d8f2ae57f8281b (diff)
downloadopen-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.tar.gz
open-iov-f874d709e5a45d63c8ee9ab6b0300195aa5d20a5.zip
userspace/cuda: add CUDA userspace driver analysis, tooling and name maps
Add the analysis of the NVIDIA CUDA userspace driver used by Open-IOV VMs: * docs/01-08: architecture (layers, object model, threads, sync, memory, channels, launch), build and execution model, the driver/kernel/GPU interface contract, virtualization approaches, the 570.144 decompilation map with the corrected libcuda crash path and the current engine-topology blocker, tooling, and a version-upgrade playbook. * tools/: version-agnostic toolchain - stdlib ELF/interface toolkit (exports, unwind-table function map, wrapper resolution, version delta, crash address decode), callgraph walker, corpus annotator, and Ghidra analysis/decompilation/name-map scripts with resumable parallel drivers. * 570.144/annotations/: our name map (exports + resolved wrappers + verified internals) and subsystem index, tracked so naming work accumulates. * READ-ME-FIRST.md: reading guide with the fast lookup recipes. The full decompiled corpus and annotated corpus stay untracked (vendor- derived, gitignored under the version run/ tree).
Diffstat (limited to 'userspace/cuda/docs/06-tooling.md')
-rw-r--r--userspace/cuda/docs/06-tooling.md165
1 files changed, 165 insertions, 0 deletions
diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md
new file mode 100644
index 0000000..85d1c8c
--- /dev/null
+++ b/userspace/cuda/docs/06-tooling.md
@@ -0,0 +1,165 @@
+# 06 — Tooling: how to analyse a CUDA userspace driver
+
+The toolchain in `tools/` is version-agnostic. Point it at any
+`libcuda.so.<version>` and it produces the same artifacts as for 570.144. All
+tools are stdlib-only or Ghidra scripts; nothing needs network access.
+
+```
+userspace/cuda/tools/
+├── cudatool.py ELF/interface toolkit (python3, no deps)
+├── callwalk.py callee/caller trees and shortest call path
+├── annotate_corpus.py annotated + name-substituted corpus
+├── ghidra_full_analysis.sh import + full auto-analysis into a project
+├── ghidra_export_index.sh functions/exports/callgraph/strings index
+├── ghidra_batch_decompile.sh parallel, resumable full decompilation
+├── ghidra_decompile_at.sh decompile one function by file vaddr
+├── ghidra_index_merge.sh merge batch indexes
+└── ghidra/
+ ├── ConfigureAnalysis.java analyzer tuning for big stripped binaries
+ ├── ExportFunctionIndex.java index exporter
+ ├── BatchDecompile.java chunk decompiler
+ ├── DecompileAt.java single function (file-vaddr aware)
+ ├── ApplyNameMap.java apply/transfer a name map
+ └── PrintProgramInfo.java address convention + stats
+```
+
+## 1. Fast facts without Ghidra — `cudatool.py`
+
+```
+tools/cudatool.py info <lib> # build id, sections, exports, ABI note
+tools/cudatool.py exports <lib> # addr size name (file vaddrs)
+tools/cudatool.py functions <lib> # function ranges from .eh_frame_hdr
+tools/cudatool.py nearest <lib> <addr> # function start + nearest export
+tools/cudatool.py crash <lib> <map_base> <ip> # dmesg -> file vaddr
+tools/cudatool.py strings <lib> [--grep RE]
+tools/cudatool.py note <lib> # .note.cuda.fwd_compatibility JSON
+tools/cudatool.py delta <libA> <libB> # see §4
+```
+
+Useful for: checking what changed between two driver package versions before
+spending time on decompilation; converting a guest dmesg crash to a file vaddr
+without gdb; getting a function list in a second.
+
+### Address conventions
+
+* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used
+ everywhere in this directory.
+* Ghidra may load the same binary at a non-zero image base. For 570.144 it is
+ `0x100000`; check with `PrintProgramInfo.java`. The Ghidra wrappers here
+ translate file vaddr → program address automatically; raw Ghidra scripts do
+ not.
+
+## 2. Full analysis — Ghidra
+
+```
+# 1. import + analyze (keep the project; can take minutes to hours)
+tools/ghidra_full_analysis.sh <lib> <project_dir> <project_name>
+
+# 2. export the index (fast, read-only)
+tools/ghidra_export_index.sh <project_dir> <project_name> <program> <outdir>
+
+# 3. decompile everything (parallel, resumable; safe to re-run)
+tools/ghidra_batch_decompile.sh <project_dir> <project_name> <program> <outdir> \
+ <start_hex> <end_hex> [chunk_hex] [jobs] [timeout_s]
+tools/ghidra_index_merge.sh <outdir>
+```
+
+Notes:
+
+* `ConfigureAnalysis.java` disables the two expensive identifier analyzers
+ (parameter-ID, function-ID) and keeps everything needed for a stripped
+ binary. Full analysis of 71 MB then takes minutes, not hours.
+* `ghidra_batch_decompile.sh` gives **every chunk its own project copy** under
+ `<outdir>/wk/` because Ghidra takes an exclusive project lock even in
+ read-only mode; completed chunks are skipped on re-run. Delete `wk/` to
+ reclaim disk.
+* The program name passed to Ghidra is the binary's file name
+ (`libcuda.so.570.144`).
+* Do **not** pass `-analysisTimeoutPerFile 0`: in this Ghidra build that means
+ "abort analysis immediately" (the original per-version helper in
+ `570.144/scripts/` did this and produced an analysed-looking project with no
+ analysis at all).
+
+## 2b. Annotating the corpus
+
+```
+tools/annotate_corpus.py <outdir> <indexdir> <names.tsv> <annotated_outdir>
+```
+
+Rewrites every function header with the recovered name, file address, size and
+direct callers/callees (named where the map has them), and substitutes
+recovered names for `FUN_<addr>` references in the bodies. Runs in seconds for
+30k functions; the output is gitignored like the raw corpus.
+
+## 2c. Walking the callgraph without Ghidra
+
+```
+tools/callwalk.py <indexdir>/callgraph.tsv callees 0x31b7f0 2 10
+tools/callwalk.py <indexdir>/callgraph.tsv callers 0x453ef0
+tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940
+```
+
+Addresses are file vaddrs; `names.tsv` next to the callgraph is used for
+labels.
+
+## 3. Looking things up
+
+```
+# one function by file vaddr
+tools/ghidra_decompile_at.sh <project_dir> <project_name> <program> 0x45a670 out.c
+
+# corpus search by file address (headers are annotated)
+grep -n "libcuda+0x45a670" <outdir>/buckets/*.c
+
+# index lookups
+awk -F'\t' '$2=="0x45a670"' <outdir>/decompiled_index.tsv
+grep -P "^0x0055a670\t" <indexdir>/callgraph.tsv # direct callers/callees
+
+# exports/strings
+grep cuLaunchKernel <indexdir>/exports.tsv
+```
+
+## 4. Version deltas — `cudatool.py delta`
+
+```
+tools/cudatool.py delta <libA> <libB>
+```
+
+Reports, at the interface level: added/removed/moved/resized exports, import
+count changes, `.text` growth, build-id and the CUDA/ABI note. This is the
+first thing to run on a new driver release; it tells you immediately whether
+the update is a patch (nothing moved), a feature release (new exports) or a
+reshuffle (many moves).
+
+## 5. Name transfer — `ApplyNameMap.java`
+
+```
+tools/ghidra_decompile_at.sh-free invocation:
+ analyzeHeadless <project_dir> <project_name> -process <program> -noanalysis \
+ -scriptPath tools/ghidra -postScript ApplyNameMap.java <names.tsv> [--dry-run]
+```
+
+The map is `file_vaddr<TAB>name` per line (comments allowed). Name maps live
+in version directories (`570.144/annotations/names.tsv`) and are tracked, so
+naming work accumulates: apply the previous version's map to a new binary,
+then fix the misses. `--dry-run` reports how many names would land.
+
+## 6. Writing new Ghidra scripts
+
+Follow the existing pattern: a `GhidraScript` subclass in `tools/ghidra/`,
+args via `getScriptArgs()`, and file-vaddr ↔ program-address translation via
+`currentProgram.getImageBase()` (see `DecompileAt.java`). Keep outputs under
+the version's gitignored `run/` tree unless the artifact is our own work
+product (name maps, indexes of our findings), which may be tracked.
+
+## 7. Cost guide (measured on 570.144, 16 cores)
+
+| Step | Time | Disk |
+|---|---|---|
+| import alone | ~1 min | 324 MB project |
+| full analysis | ~10–15 min | same project |
+| index export | ~1–2 min | ~25 MB TSV |
+| full decompilation (5 workers) | ~20–30 min | ~3 GB C + 2 GB project copies |
+
+Notes for a rerun: set `GHIDRA_HEADLESS_MAXMEM` (default 12G for analysis,
+4G per batch job) and keep total batch jobs ≤ 5–6 on a 32 GB machine.