1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
|
# CUDA driver analysis (570.144)
Tools and notes for **debugging CUDA execution inside Open-IOV guests**.
## Purpose (read this first)
These tools exist to make virtualized CUDA *work correctly and fast* in
Open-IOV VMs, and to understand the failures we hit:
* debug guest-side CUDA/driver crashes (like the `libcuda` segfault in
`cuCtxCreate` this tree was created for),
* verify correctness (host/guest A/B comparisons, `tools/cuda-ab/`),
* measure and explain performance,
* later: assess security and isolation of virtualized CUDA.
They are **not** a reimplementation effort and **not** a way to produce an
open-source CUDA. We only disassemble/decompile the vendor userspace driver to
understand what it expects from the kernel driver and the GPU, so the
Open-IOV host emulation can satisfy those expectations. Decompiled output is
never committed (see `.gitignore`).
## Layout
```
userspace/cuda/570.144/
├── README.md this file
├── .gitignore decompiled/disassembled output and extracted binaries
├── scripts/
│ ├── fetch_from_guest.sh copy a library out of a running guest (nc)
│ ├── disasm.sh objdump helpers (bytes / intel disassembly / symbol)
│ ├── crash_vaddr.py guest dmesg crash -> file vaddr + nearest symbol
│ └── ghidra_decompile.sh headless Ghidra: decompile one function by vaddr
├── ghidra/
│ └── DecompileAt.java post-script used by ghidra_decompile.sh
└── run/ (generated) extracted libs, decompiled C, logs
```
## Quick start (debug a guest crash)
1. Read the crash from the guest kernel log:
```
sudo dmesg | grep -E 'segfault|RIP:' # guest
# e.g. cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90
# ... in libcuda.so.570.144[735cfcb79000+d1c000]
```
The library's *executable* segment starts at a nonzero vaddr, so the
dmesg offset is **not** the file vaddr. Use `scripts/crash_vaddr.py` with
the library, the reported base and the reported ip (or with gdb's
`info sharedlibrary` output) to get the file vaddr and nearest exported
symbol.
2. Disassemble around it:
```
scripts/disasm.sh run/libcuda.so.570.144 <vaddr> [before] [after]
```
3. Decompile the containing function with Ghidra:
```
scripts/ghidra_decompile.sh run/libcuda.so.570.144 <vaddr>
# -> run/libcuda_<vaddr>.c (gitignored)
```
4. Correlate with the driver-side view: which ioctl/control did the driver
last service before the crash? In the guest:
```
/tmp/iot /tmp/cdt # ptrace tracer (tools/guest-diag/ioctl_trace.c)
```
and on the host: `dmesg | grep 'RM_CONTROL cmd='` / `'GSP_RM_ALLOC'`.
## Getting the library
From a running guest (guest agent must be up):
```
scripts/fetch_from_guest.sh /usr/lib/x86_64-linux-gnu/libcuda.so.570.144
# -> run/libcuda.so.570.144 (md5-checked)
```
From a driver package: `NVIDIA-Linux-x86_64-570.144.run --extract-only`, the
user-space libraries land in the extracted tree.
## Mapping dmesg ip -> file vaddr, precisely
`dmesg` prints `lib[<mapping base>+<size>]` where `<mapping base>` is the
process address of the first *executable* mapping, not the ELF load base.
Compute it from the ELF program headers:
```
vaddr = ip - mapping_base + first_exec_segment_vaddr_aligned
```
`scripts/crash_vaddr.py` does this (it reads `readelf -lW` output from the
library) and also prints the nearest preceding exported symbol from
`readelf -sW --dyn-syms`.
## Notes
* Host/guest A/B correctness harness lives in `tools/cuda-ab/`.
* Guest-side tracing helpers: `tools/guest-diag/` (`ioctl_trace.c`,
`ptrace_ioctl.c`, `push_guest_tools.sh`).
* Nothing here replaces the docs rule: tracked docs describe behaviour, never
vendor file paths; this directory is analysis tooling, so it names the
artifacts it analyses.
|
