path: root/userspace/cuda/docs/06-tooling.md

# 06 — Tooling: how to analyse a CUDA userspace driver

The toolchain in `tools/` is version-agnostic. Point it at any
`libcuda.so.<version>` and it produces the same artifacts as for 570.144. All
tools are stdlib-only or Ghidra scripts; nothing needs network access.

```
userspace/cuda/tools/
├── cudatool.py                     ELF/interface toolkit (python3, no deps)
├── callwalk.py                     callee/caller trees and shortest call path
├── name_candidates.py              propose impl names for wrapper callees
├── subsystem_digest.py             per-subsystem call-tree digest
├── diff_corpus.py                  diff two decompiled corpus indexes
├── annotate_corpus.py              annotated + name-substituted corpus
├── ghidra_full_analysis.sh         import + full auto-analysis into a project
├── ghidra_export_index.sh          functions/exports/callgraph/strings index
├── ghidra_batch_decompile.sh       parallel, resumable full decompilation
├── ghidra_decompile_at.sh          decompile one function by file vaddr
├── ghidra_index_merge.sh           merge batch indexes
└── ghidra/
    ├── ConfigureAnalysis.java      analyzer tuning for big stripped binaries
    ├── ExportFunctionIndex.java    index exporter
    ├── BatchDecompile.java         chunk decompiler
    ├── DecompileAt.java            single function (file-vaddr aware)
    ├── AnalyzeFunctions.java       targeted Decompiler Parameter ID pass
    ├── ApplyNameMap.java           apply/transfer a name map
    └── PrintProgramInfo.java       address convention + stats
```

## 1. Fast facts without Ghidra — `cudatool.py`

```
tools/cudatool.py info      <lib>          # build id, sections, exports, ABI note
tools/cudatool.py exports   <lib>          # addr size name (file vaddrs)
tools/cudatool.py functions <lib>          # function ranges from .eh_frame_hdr
tools/cudatool.py nearest   <lib> <addr>   # function start + nearest export
tools/cudatool.py crash     <lib> <map_base> <ip>   # dmesg -> file vaddr
tools/cudatool.py strings   <lib> [--grep RE]
tools/cudatool.py note      <lib>          # .note.cuda.fwd_compatibility JSON
tools/cudatool.py delta     <libA> <libB>  # see §4
tools/cudatool.py fdematch  <libA> <libB> [--names names.tsv] [--window 0x100]
```

Useful for: checking what changed between two driver package versions before
spending time on decompilation; converting a guest dmesg crash to a file vaddr
without gdb; getting a function list in a second.

### 1b. Cross-version function matching — `fdematch`

```
tools/cudatool.py fdematch <libA> <libB> [--names <names.tsv>] [--window 0x100]
```

`fdematch` matches function ranges from the two `.eh_frame_hdr` tables. A
single global address shift is not enough across patch versions, so it uses
the common exports as **local anchors**: for each A function it looks for a B
function with the same FDE size within `--window` bytes of
`old_addr + delta(nearest common export)`. The report gives exact / window /
unique match counts and the overall match rate, and with `--names` it breaks
the name map down by its `source` column (exports, wrappers, verified,
wrapper-callee), which is exactly what you need to know before transferring
names to a new version. Example result for 570.144 → 570.211.01: 85.5% of
functions match (58.3% uniquely), **all 659 exports transfer exactly, but
only ~1% of the generated wrapper addresses do** — the wrapper region is
regenerated and must be re-resolved with `cudatool.py wrappers`, never by
address arithmetic.

### Address conventions

* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used
  everywhere in this directory.
* Ghidra may load the same binary at a non-zero image base. For 570.144 it is
  `0x100000`; check with `PrintProgramInfo.java`. The Ghidra wrappers here
  translate file vaddr → program address automatically; raw Ghidra scripts do
  not.

## 2. Full analysis — Ghidra

```
# 1. import + analyze (keep the project; can take minutes to hours)
tools/ghidra_full_analysis.sh <lib> <project_dir> <project_name>

# 2. export the index (fast, read-only)
tools/ghidra_export_index.sh <project_dir> <project_name> <program> <outdir>

# 3. decompile everything (parallel, resumable; safe to re-run)
tools/ghidra_batch_decompile.sh <project_dir> <project_name> <program> <outdir> \
    <start_hex> <end_hex> [chunk_hex] [jobs] [timeout_s]
tools/ghidra_index_merge.sh <outdir>
```

Notes:

* `ConfigureAnalysis.java` disables the two expensive identifier analyzers
  (parameter-ID, function-ID) and keeps everything needed for a stripped
  binary. Full analysis of 71 MB then takes minutes, not hours.
* `ghidra_batch_decompile.sh` gives **every chunk its own project copy** under
  `<outdir>/wk/` because Ghidra takes an exclusive project lock even in
  read-only mode; completed chunks are skipped on re-run. Delete `wk/` to
  reclaim disk.
* The program name passed to Ghidra is the binary's file name
  (`libcuda.so.570.144`).
* Do **not** pass `-analysisTimeoutPerFile 0`: in this Ghidra build that means
  "abort analysis immediately" (the original per-version helper in
  `570.144/scripts/` did this and produced an analysed-looking project with no
  analysis at all).

## 2b. Annotating the corpus

```
tools/annotate_corpus.py <outdir> <indexdir> <names.tsv> <annotated_outdir>
```

Rewrites every function header with the recovered name, file address, size and
direct callers/callees (named where the map has them), and substitutes
recovered names for `FUN_<addr>` references in the bodies. Runs in seconds for
30k functions; the output is gitignored like the raw corpus.

## 2c. Walking the callgraph without Ghidra

```
tools/callwalk.py <indexdir>/callgraph.tsv callees 0x31b7f0 2 10
tools/callwalk.py <indexdir>/callgraph.tsv callers 0x453ef0
tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940
```

Addresses are file vaddrs; `names.tsv` next to the callgraph is used for
labels.

## 2d. Targeted type recovery

The full analysis disables the Decompiler Parameter ID analyzer (§2), so
function signatures stay at the default `FUN_<addr>(...)`. `AnalyzeFunctions.java`
re-runs that analyzer over a caller-supplied function list instead of the whole
program: it takes an addresses file (one `0x<file_vaddr>` per line, comments
allowed), translates through the image base like `DecompileAt.java`, resolves
each address to its function and runs the analyzer on the resulting address
set (batched, with a per-function decompiler timeout). Run it against a
project copy, not the canonical one, because it commits parameters into the
project:

```
cp -a <proj_dir> <proj_dir>-typed
analyzeHeadless <proj_dir>-typed <proj_name> -process <program> -noanalysis \
    -scriptPath tools/ghidra \
    -postScript AnalyzeFunctions.java <addresses.txt> \
        --timeout 120 --batch 20 --out <report.tsv>
```

Then re-decompile the affected functions with `ghidra_decompile_at.sh` from the
typed project; the report's `params_before/after` and `GAIN` columns show
which functions gained structure. `--list` resolves the list without running
the analyzer (dry run).

## 2d. Subsystem call-tree digest — `subsystem_digest.py`

```
tools/subsystem_digest.py <subsystems.tsv> <callgraph.tsv> [--out FILE]
                          [--depth 2] [--max-per-node 8] [--subsystem NAME]
```

For every subsystem in `annotations/subsystems.tsv` it prints the annotated
members, the subsystem **entry functions** (members not called by another
member of the same subsystem) and a depth-limited call tree from each entry
whose direct callees are grouped by subsystem. It closes with a
cross-subsystem edge count matrix. This is the quickest way to see a
subsystem's shape and where its boundary with the rest of the driver is,
without opening Ghidra. Example output for 570.144 is saved under
`570.144/run/analysis/subsystem-digest.txt` (gitignored).

## 2e. API-implementation candidates — `name_candidates.py`

```
tools/name_candidates.py <exports.tsv> <callgraph.tsv> <names.tsv> <out.tsv>
                         [--max-outside 2]
```

Exports are generated stubs → generated wrappers → the hand-written
implementation of that API. `name_candidates.py` groups the wrappers by base
name (stripping `_v<N>`, `_ptsz`, `_ptds`), takes the intersection of their
direct callees and filters out helpers that are also called by wrappers
outside the group (the trace/init/lock helpers). A single surviving candidate
is `high` confidence when the group has several variants or the callee is
unique; otherwise the best-ranked candidate is `medium`. Output rows carry
the full candidate set and the outside-wrapper count as evidence, plus a
`status` column that starts as `unverified`. Verification means reading the
body in the corpus and marking `verified`/`corrected`; only verified entries
may be copied into `names.tsv` (source `wrapper-callee`). The 570.144 run is
tracked in `570.144/annotations/name-candidates.tsv`.

## 3. Looking things up

```
# one function by file vaddr
tools/ghidra_decompile_at.sh <project_dir> <project_name> <program> 0x45a670 out.c

# corpus search by file address (headers are annotated)
grep -n "libcuda+0x45a670" <outdir>/buckets/*.c

# index lookups
awk -F'\t' '$2=="0x45a670"' <outdir>/decompiled_index.tsv
grep -P "^0x0055a670\t" <indexdir>/callgraph.tsv     # direct callers/callees

# exports/strings
grep cuLaunchKernel <indexdir>/exports.tsv
```

## 4. Version deltas — `cudatool.py delta`

```
tools/cudatool.py delta <libA> <libB>
```

Reports, at the interface level: added/removed/moved/resized exports, import
count changes, `.text` growth, build-id and the CUDA/ABI note. This is the
first thing to run on a new driver release; it tells you immediately whether
the update is a patch (nothing moved), a feature release (new exports) or a
reshuffle (many moves).

### 4b. Corpus deltas — `diff_corpus.py`

```
tools/diff_corpus.py <indexA.tsv> <indexB.tsv> [--label-a A] [--label-b B]
                     [--out FILE] [--max-list 40] [--json]
```

Diffs two `decompiled_index.tsv` corpora (the merged output of
`ghidra_batch_decompile.sh` + `ghidra_index_merge.sh`) and reports
added/removed/moved/resized functions and a summary. Named functions match by
name; the address shift is estimated from them and anonymous `FUN_*`
functions match at `old_addr + shift` (a size difference there is a resize).
This makes a driver upgrade measurable at the function level: run it after
decompiling the new version to see how much of the previous corpus still
corresponds. It needs a decompiled index for the other version; if only the
binary exists, use `cudatool.py fdematch` (no decompilation required).

## 5. Name transfer — `ApplyNameMap.java`

```
tools/ghidra_decompile_at.sh-free invocation:
  analyzeHeadless <project_dir> <project_name> -process <program> -noanalysis \
      -scriptPath tools/ghidra -postScript ApplyNameMap.java <names.tsv> [--dry-run]
```

The map is `file_vaddr<TAB>name` per line (comments allowed). Name maps live
in version directories (`570.144/annotations/names.tsv`) and are tracked, so
naming work accumulates: apply the previous version's map to a new binary,
then fix the misses. `--dry-run` reports how many names would land.

## 6. Writing new Ghidra scripts

Follow the existing pattern: a `GhidraScript` subclass in `tools/ghidra/`,
args via `getScriptArgs()`, and file-vaddr ↔ program-address translation via
`currentProgram.getImageBase()` (see `DecompileAt.java`). Keep outputs under
the version's gitignored `run/` tree unless the artifact is our own work
product (name maps, indexes of our findings), which may be tracked.

## 7. Cost guide (measured on 570.144, 16 cores)

| Step | Time | Disk |
|---|---|---|
| import alone | ~1 min | 324 MB project |
| full analysis | ~10–15 min | same project |
| index export | ~1–2 min | ~25 MB TSV |
| full decompilation (5 workers) | ~20–30 min | ~3 GB C + 2 GB project copies |

Notes for a rerun: set `GHIDRA_HEADLESS_MAXMEM` (default 12G for analysis,
4G per batch job) and keep total batch jobs ≤ 5–6 on a 32 GB machine.