diff options
Diffstat (limited to 'userspace/cuda/570.144/README.md')
| -rw-r--r-- | userspace/cuda/570.144/README.md | 109 |
1 files changed, 109 insertions, 0 deletions
diff --git a/userspace/cuda/570.144/README.md b/userspace/cuda/570.144/README.md new file mode 100644 index 0000000..f32b762 --- /dev/null +++ b/userspace/cuda/570.144/README.md @@ -0,0 +1,109 @@ +# CUDA driver analysis (570.144) + +Tools and notes for **debugging CUDA execution inside Open-IOV guests**. + +## Purpose (read this first) + +These tools exist to make virtualized CUDA *work correctly and fast* in +Open-IOV VMs, and to understand the failures we hit: + +* debug guest-side CUDA/driver crashes (like the `libcuda` segfault in + `cuCtxCreate` this tree was created for), +* verify correctness (host/guest A/B comparisons, `tools/cuda-ab/`), +* measure and explain performance, +* later: assess security and isolation of virtualized CUDA. + +They are **not** a reimplementation effort and **not** a way to produce an +open-source CUDA. We only disassemble/decompile the vendor userspace driver to +understand what it expects from the kernel driver and the GPU, so the +Open-IOV host emulation can satisfy those expectations. Decompiled output is +never committed (see `.gitignore`). + +## Layout + +``` +userspace/cuda/570.144/ +├── README.md this file +├── .gitignore decompiled/disassembled output and extracted binaries +├── scripts/ +│ ├── fetch_from_guest.sh copy a library out of a running guest (nc) +│ ├── disasm.sh objdump helpers (bytes / intel disassembly / symbol) +│ ├── crash_vaddr.py guest dmesg crash -> file vaddr + nearest symbol +│ └── ghidra_decompile.sh headless Ghidra: decompile one function by vaddr +├── ghidra/ +│ └── DecompileAt.java post-script used by ghidra_decompile.sh +└── run/ (generated) extracted libs, decompiled C, logs +``` + +## Quick start (debug a guest crash) + +1. Read the crash from the guest kernel log: + + ``` + sudo dmesg | grep -E 'segfault|RIP:' # guest + # e.g. cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90 + # ... in libcuda.so.570.144[735cfcb79000+d1c000] + ``` + + The library's *executable* segment starts at a nonzero vaddr, so the + dmesg offset is **not** the file vaddr. Use `scripts/crash_vaddr.py` with + the library, the reported base and the reported ip (or with gdb's + `info sharedlibrary` output) to get the file vaddr and nearest exported + symbol. + +2. Disassemble around it: + + ``` + scripts/disasm.sh run/libcuda.so.570.144 <vaddr> [before] [after] + ``` + +3. Decompile the containing function with Ghidra: + + ``` + scripts/ghidra_decompile.sh run/libcuda.so.570.144 <vaddr> + # -> run/libcuda_<vaddr>.c (gitignored) + ``` + +4. Correlate with the driver-side view: which ioctl/control did the driver + last service before the crash? In the guest: + + ``` + /tmp/iot /tmp/cdt # ptrace tracer (tools/guest-diag/ioctl_trace.c) + ``` + + and on the host: `dmesg | grep 'RM_CONTROL cmd='` / `'GSP_RM_ALLOC'`. + +## Getting the library + +From a running guest (guest agent must be up): + +``` +scripts/fetch_from_guest.sh /usr/lib/x86_64-linux-gnu/libcuda.so.570.144 +# -> run/libcuda.so.570.144 (md5-checked) +``` + +From a driver package: `NVIDIA-Linux-x86_64-570.144.run --extract-only`, the +user-space libraries land in the extracted tree. + +## Mapping dmesg ip -> file vaddr, precisely + +`dmesg` prints `lib[<mapping base>+<size>]` where `<mapping base>` is the +process address of the first *executable* mapping, not the ELF load base. +Compute it from the ELF program headers: + +``` +vaddr = ip - mapping_base + first_exec_segment_vaddr_aligned +``` + +`scripts/crash_vaddr.py` does this (it reads `readelf -lW` output from the +library) and also prints the nearest preceding exported symbol from +`readelf -sW --dyn-syms`. + +## Notes + +* Host/guest A/B correctness harness lives in `tools/cuda-ab/`. +* Guest-side tracing helpers: `tools/guest-diag/` (`ioctl_trace.c`, + `ptrace_ioctl.c`, `push_guest_tools.sh`). +* Nothing here replaces the docs rule: tracked docs describe behaviour, never + vendor file paths; this directory is analysis tooling, so it names the + artifacts it analyses. |
