# CUDA driver analysis (570.144)

Tools and notes for **debugging CUDA execution inside Open-IOV guests**.

> **Full analysis now lives one level up.** See `../README.md` and
> `../docs/` for the architecture documents, the complete 570.144
> decompilation (29,254 functions, annotated corpus under
> `run/analysis/decompiled-annotated/`), the corrected crash-path analysis
> (`../docs/05-570.144-decompilation.md`) and the reusable tooling
> (`../tools/`).
>
> **Address warning:** Ghidra loads this binary at image base `0x100000`.
> `scripts/ghidra_decompile_project.sh` passed file vaddrs straight to Ghidra,
> so decompilations produced with it (e.g. `run/helper_453ef0.c`) are of
> addresses 0x100000 too low. Use `../tools/ghidra_decompile_at.sh`, which
> translates file vaddr → Ghidra address automatically.

## Purpose (read this first)

These tools exist to make virtualized CUDA *work correctly and fast* in
Open-IOV VMs, and to understand the failures we hit:

* debug guest-side CUDA/driver crashes (like the `libcuda` segfault in
  `cuCtxCreate` this tree was created for),
* verify correctness (host/guest A/B comparisons, `tools/cuda-ab/`),
* measure and explain performance,
* later: assess security and isolation of virtualized CUDA.

They are **not** a reimplementation effort and **not** a way to produce an
open-source CUDA. We only disassemble/decompile the vendor userspace driver to
understand what it expects from the kernel driver and the GPU, so the
Open-IOV host emulation can satisfy those expectations. Decompiled output is
never committed (see `.gitignore`).

## Layout

```
userspace/cuda/570.144/
├── README.md            this file
├── .gitignore           decompiled/disassembled output and extracted binaries
├── scripts/
│   ├── fetch_from_guest.sh   copy a library out of a running guest (nc)
│   ├── disasm.sh             objdump helpers (bytes / intel disassembly / symbol)
│   ├── crash_vaddr.py        guest dmesg crash -> file vaddr + nearest symbol
│   └── ghidra_decompile.sh   headless Ghidra: decompile one function by vaddr
├── ghidra/
│   └── DecompileAt.java      post-script used by ghidra_decompile.sh
└── run/                  (generated) extracted libs, decompiled C, logs
```

## Quick start (debug a guest crash)

1. Read the crash from the guest kernel log:

   ```
   sudo dmesg | grep -E 'segfault|RIP:'          # guest
   # e.g. cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90
   #      ... in libcuda.so.570.144[735cfcb79000+d1c000]
   ```

   The library's *executable* segment starts at a nonzero vaddr, so the
   dmesg offset is **not** the file vaddr. Use `scripts/crash_vaddr.py` with
   the library, the reported base and the reported ip (or with gdb's
   `info sharedlibrary` output) to get the file vaddr and nearest exported
   symbol.

2. Disassemble around it:

   ```
   scripts/disasm.sh run/libcuda.so.570.144 <vaddr> [before] [after]
   ```

3. Decompile the containing function with Ghidra:

   ```
   scripts/ghidra_decompile.sh run/libcuda.so.570.144 <vaddr>
   # -> run/libcuda_<vaddr>.c   (gitignored)
   ```

4. Correlate with the driver-side view: which ioctl/control did the driver
   last service before the crash? In the guest:

   ```
   /tmp/iot /tmp/cdt        # ptrace tracer (tools/guest-diag/ioctl_trace.c)
   ```

   and on the host: `dmesg | grep 'RM_CONTROL cmd='` / `'GSP_RM_ALLOC'`.

## Getting the library

From a running guest (guest agent must be up):

```
scripts/fetch_from_guest.sh /usr/lib/x86_64-linux-gnu/libcuda.so.570.144
# -> run/libcuda.so.570.144 (md5-checked)
```

From a driver package: `NVIDIA-Linux-x86_64-570.144.run --extract-only`, the
user-space libraries land in the extracted tree.

## Mapping dmesg ip -> file vaddr, precisely

`dmesg` prints `lib[<mapping base>+<size>]` where `<mapping base>` is the
process address of the first *executable* mapping, not the ELF load base.
Compute it from the ELF program headers:

```
vaddr = ip - mapping_base + first_exec_segment_vaddr_aligned
```

`scripts/crash_vaddr.py` does this (it reads `readelf -lW` output from the
library) and also prints the nearest preceding exported symbol from
`readelf -sW --dyn-syms`.

## Notes

* Host/guest A/B correctness harness lives in `tools/cuda-ab/`.
* Guest-side tracing helpers: `tools/guest-diag/` (`ioctl_trace.c`,
  `ptrace_ioctl.c`, `push_guest_tools.sh`).
* Nothing here replaces the docs rule: tracked docs describe behaviour, never
  vendor file paths; this directory is analysis tooling, so it names the
  artifacts it analyses.