summaryrefslogtreecommitdiffstats
path: root/userspace/cuda/570.144/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'userspace/cuda/570.144/README.md')
-rw-r--r--userspace/cuda/570.144/README.md109
1 files changed, 109 insertions, 0 deletions
diff --git a/userspace/cuda/570.144/README.md b/userspace/cuda/570.144/README.md
new file mode 100644
index 0000000..f32b762
--- /dev/null
+++ b/userspace/cuda/570.144/README.md
@@ -0,0 +1,109 @@
+# CUDA driver analysis (570.144)
+
+Tools and notes for **debugging CUDA execution inside Open-IOV guests**.
+
+## Purpose (read this first)
+
+These tools exist to make virtualized CUDA *work correctly and fast* in
+Open-IOV VMs, and to understand the failures we hit:
+
+* debug guest-side CUDA/driver crashes (like the `libcuda` segfault in
+ `cuCtxCreate` this tree was created for),
+* verify correctness (host/guest A/B comparisons, `tools/cuda-ab/`),
+* measure and explain performance,
+* later: assess security and isolation of virtualized CUDA.
+
+They are **not** a reimplementation effort and **not** a way to produce an
+open-source CUDA. We only disassemble/decompile the vendor userspace driver to
+understand what it expects from the kernel driver and the GPU, so the
+Open-IOV host emulation can satisfy those expectations. Decompiled output is
+never committed (see `.gitignore`).
+
+## Layout
+
+```
+userspace/cuda/570.144/
+├── README.md this file
+├── .gitignore decompiled/disassembled output and extracted binaries
+├── scripts/
+│ ├── fetch_from_guest.sh copy a library out of a running guest (nc)
+│ ├── disasm.sh objdump helpers (bytes / intel disassembly / symbol)
+│ ├── crash_vaddr.py guest dmesg crash -> file vaddr + nearest symbol
+│ └── ghidra_decompile.sh headless Ghidra: decompile one function by vaddr
+├── ghidra/
+│ └── DecompileAt.java post-script used by ghidra_decompile.sh
+└── run/ (generated) extracted libs, decompiled C, logs
+```
+
+## Quick start (debug a guest crash)
+
+1. Read the crash from the guest kernel log:
+
+ ```
+ sudo dmesg | grep -E 'segfault|RIP:' # guest
+ # e.g. cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90
+ # ... in libcuda.so.570.144[735cfcb79000+d1c000]
+ ```
+
+ The library's *executable* segment starts at a nonzero vaddr, so the
+ dmesg offset is **not** the file vaddr. Use `scripts/crash_vaddr.py` with
+ the library, the reported base and the reported ip (or with gdb's
+ `info sharedlibrary` output) to get the file vaddr and nearest exported
+ symbol.
+
+2. Disassemble around it:
+
+ ```
+ scripts/disasm.sh run/libcuda.so.570.144 <vaddr> [before] [after]
+ ```
+
+3. Decompile the containing function with Ghidra:
+
+ ```
+ scripts/ghidra_decompile.sh run/libcuda.so.570.144 <vaddr>
+ # -> run/libcuda_<vaddr>.c (gitignored)
+ ```
+
+4. Correlate with the driver-side view: which ioctl/control did the driver
+ last service before the crash? In the guest:
+
+ ```
+ /tmp/iot /tmp/cdt # ptrace tracer (tools/guest-diag/ioctl_trace.c)
+ ```
+
+ and on the host: `dmesg | grep 'RM_CONTROL cmd='` / `'GSP_RM_ALLOC'`.
+
+## Getting the library
+
+From a running guest (guest agent must be up):
+
+```
+scripts/fetch_from_guest.sh /usr/lib/x86_64-linux-gnu/libcuda.so.570.144
+# -> run/libcuda.so.570.144 (md5-checked)
+```
+
+From a driver package: `NVIDIA-Linux-x86_64-570.144.run --extract-only`, the
+user-space libraries land in the extracted tree.
+
+## Mapping dmesg ip -> file vaddr, precisely
+
+`dmesg` prints `lib[<mapping base>+<size>]` where `<mapping base>` is the
+process address of the first *executable* mapping, not the ELF load base.
+Compute it from the ELF program headers:
+
+```
+vaddr = ip - mapping_base + first_exec_segment_vaddr_aligned
+```
+
+`scripts/crash_vaddr.py` does this (it reads `readelf -lW` output from the
+library) and also prints the nearest preceding exported symbol from
+`readelf -sW --dyn-syms`.
+
+## Notes
+
+* Host/guest A/B correctness harness lives in `tools/cuda-ab/`.
+* Guest-side tracing helpers: `tools/guest-diag/` (`ioctl_trace.c`,
+ `ptrace_ioctl.c`, `push_guest_tools.sh`).
+* Nothing here replaces the docs rule: tracked docs describe behaviour, never
+ vendor file paths; this directory is analysis tooling, so it names the
+ artifacts it analyses.