userspace/cuda: libcuda analysis tree (debug-focused), guest ioctl tracer

Analysis tooling for debugging CUDA execution in Open-IOV guests:
- userspace/cuda/570.144/: fetch a library from a running guest (nc),
  objdump helpers, crash ip -> file vaddr + symbol mapping, and Ghidra
  headless import/analysis/decompile scripts (project kept so single
  functions decompile in seconds). The purpose is debugging correctness
  and performance of CUDA in Open-IOV VMs (not a reimplementation);
  extracted binaries and decompiled output are gitignored.
- tools/guest-diag/ioctl_trace.c: ptrace tracker for the NVIDIA driver
  ioctls of a child (libcuda issues raw syscalls, so LD_PRELOAD sees
  nothing). Decodes RM_ALLOC (class) and RM_CONTROL (cmd) arguments;
  this is how the cuCtxCreate crash sequence was captured.
- tools/test/cuda_drv_test.c: unbuffered + per-call markers so a crash
  identifies the CUDA call it happened in.

8 files changed, 426 insertions, 0 deletions

diff --git a/userspace/cuda/570.144/.gitignore b/userspace/cuda/570.144/.gitignore
new file mode 100644
index 0000000..c5997d3
--- /dev/null
+++ b/userspace/cuda/570.144/.gitignore
@@ -0,0 +1,18 @@
+# Extracted vendor binaries (analysis inputs, never committed)
+run/*.so
+run/*.so.*
+run/libcuda*
+run/*.bin
+run/*.run
+
+# Decompiled / disassembled output
+run/*.c
+run/*.txt
+run/*.asm
+run/*.json
+run/*.log
+
+# Ghidra projects
+run/*.rep/
+run/*.gpr
+run/ghidra-*/
diff --git a/userspace/cuda/570.144/README.md b/userspace/cuda/570.144/README.md
new file mode 100644
index 0000000..f32b762
--- /dev/null
+++ b/userspace/cuda/570.144/README.md
@@ -0,0 +1,109 @@
+# CUDA driver analysis (570.144)
+
+Tools and notes for **debugging CUDA execution inside Open-IOV guests**.
+
+## Purpose (read this first)
+
+These tools exist to make virtualized CUDA *work correctly and fast* in
+Open-IOV VMs, and to understand the failures we hit:
+
+* debug guest-side CUDA/driver crashes (like the `libcuda` segfault in
+  `cuCtxCreate` this tree was created for),
+* verify correctness (host/guest A/B comparisons, `tools/cuda-ab/`),
+* measure and explain performance,
+* later: assess security and isolation of virtualized CUDA.
+
+They are **not** a reimplementation effort and **not** a way to produce an
+open-source CUDA. We only disassemble/decompile the vendor userspace driver to
+understand what it expects from the kernel driver and the GPU, so the
+Open-IOV host emulation can satisfy those expectations. Decompiled output is
+never committed (see `.gitignore`).
+
+## Layout
+
+```
+userspace/cuda/570.144/
+├── README.md            this file
+├── .gitignore           decompiled/disassembled output and extracted binaries
+├── scripts/
+│   ├── fetch_from_guest.sh   copy a library out of a running guest (nc)
+│   ├── disasm.sh             objdump helpers (bytes / intel disassembly / symbol)
+│   ├── crash_vaddr.py        guest dmesg crash -> file vaddr + nearest symbol
+│   └── ghidra_decompile.sh   headless Ghidra: decompile one function by vaddr
+├── ghidra/
+│   └── DecompileAt.java      post-script used by ghidra_decompile.sh
+└── run/                  (generated) extracted libs, decompiled C, logs
+```
+
+## Quick start (debug a guest crash)
+
+1. Read the crash from the guest kernel log:
+
+   ```
+   sudo dmesg | grep -E 'segfault|RIP:'          # guest
+   # e.g. cdt[912]: segfault at ffffffffffffffc8 ip 0000735cfce53f90
+   #      ... in libcuda.so.570.144[735cfcb79000+d1c000]
+   ```
+
+   The library's *executable* segment starts at a nonzero vaddr, so the
+   dmesg offset is **not** the file vaddr. Use `scripts/crash_vaddr.py` with
+   the library, the reported base and the reported ip (or with gdb's
+   `info sharedlibrary` output) to get the file vaddr and nearest exported
+   symbol.
+
+2. Disassemble around it:
+
+   ```
+   scripts/disasm.sh run/libcuda.so.570.144 <vaddr> [before] [after]
+   ```
+
+3. Decompile the containing function with Ghidra:
+
+   ```
+   scripts/ghidra_decompile.sh run/libcuda.so.570.144 <vaddr>
+   # -> run/libcuda_<vaddr>.c   (gitignored)
+   ```
+
+4. Correlate with the driver-side view: which ioctl/control did the driver
+   last service before the crash? In the guest:
+
+   ```
+   /tmp/iot /tmp/cdt        # ptrace tracer (tools/guest-diag/ioctl_trace.c)
+   ```
+
+   and on the host: `dmesg | grep 'RM_CONTROL cmd='` / `'GSP_RM_ALLOC'`.
+
+## Getting the library
+
+From a running guest (guest agent must be up):
+
+```
+scripts/fetch_from_guest.sh /usr/lib/x86_64-linux-gnu/libcuda.so.570.144
+# -> run/libcuda.so.570.144 (md5-checked)
+```
+
+From a driver package: `NVIDIA-Linux-x86_64-570.144.run --extract-only`, the
+user-space libraries land in the extracted tree.
+
+## Mapping dmesg ip -> file vaddr, precisely
+
+`dmesg` prints `lib[<mapping base>+<size>]` where `<mapping base>` is the
+process address of the first *executable* mapping, not the ELF load base.
+Compute it from the ELF program headers:
+
+```
+vaddr = ip - mapping_base + first_exec_segment_vaddr_aligned
+```
+
+`scripts/crash_vaddr.py` does this (it reads `readelf -lW` output from the
+library) and also prints the nearest preceding exported symbol from
+`readelf -sW --dyn-syms`.
+
+## Notes
+
+* Host/guest A/B correctness harness lives in `tools/cuda-ab/`.
+* Guest-side tracing helpers: `tools/guest-diag/` (`ioctl_trace.c`,
+  `ptrace_ioctl.c`, `push_guest_tools.sh`).
+* Nothing here replaces the docs rule: tracked docs describe behaviour, never
+  vendor file paths; this directory is analysis tooling, so it names the
+  artifacts it analyses.
diff --git a/userspace/cuda/570.144/ghidra/DecompileAt.java b/userspace/cuda/570.144/ghidra/DecompileAt.java
new file mode 100644
index 0000000..5092ee9
--- /dev/null
+++ b/userspace/cuda/570.144/ghidra/DecompileAt.java
@@ -0,0 +1,61 @@
+// SPDX-License-Identifier: BSD-3-Clause
+//
+// DecompileAt.java - Ghidra post-script: decompile the function containing a
+// given address (no auto-analysis required).
+//
+// Usage (via scripts/ghidra_decompile.sh):
+//   analyzeHeadless <proj> <name> -import <binary> -noanalysis \
+//     -postScript DecompileAt.java <hex-vaddr> [outfile]
+
+import ghidra.app.script.GhidraScript;
+import ghidra.app.decompiler.DecompInterface;
+import ghidra.app.decompiler.DecompileResults;
+import ghidra.program.model.address.Address;
+import ghidra.program.model.listing.Function;
+import java.io.FileWriter;
+
+public class DecompileAt extends GhidraScript {
+    @Override
+    public void run() throws Exception {
+        String[] args = getScriptArgs();
+        if (args.length < 1) {
+            println("usage: DecompileAt.java <hex-vaddr> [outfile]");
+            return;
+        }
+        long target = Long.parseUnsignedLong(args[0].replaceFirst("^0x", ""), 16);
+        Address addr = toAddr(target);
+
+        Function f = getFunctionContaining(addr);
+        if (f == null) {
+            disassemble(addr);
+            f = createFunction(addr, null);
+        }
+        if (f == null) {
+            println("// no function at 0x" + Long.toHexString(target));
+            return;
+        }
+
+        DecompInterface dif = new DecompInterface();
+        dif.toggleCCode(true);
+        dif.toggleSyntaxTree(true);
+        dif.openProgram(currentProgram);
+
+        DecompileResults res = dif.decompileFunction(f, 120, monitor);
+        StringBuilder sb = new StringBuilder();
+        sb.append("// function ").append(f.getName())
+          .append(" at 0x").append(Long.toHexString(f.getEntryPoint().getOffset()))
+          .append(" containing 0x").append(Long.toHexString(target)).append("\n");
+        if (res != null && res.decompileCompleted()) {
+            sb.append(res.getDecompiledFunction().getC());
+        } else {
+            sb.append("// decompilation failed\n");
+        }
+        String text = sb.toString();
+        if (args.length > 1) {
+            try (FileWriter w = new FileWriter(args[1])) {
+                w.write(text);
+            }
+        }
+        println(text);
+    }
+}
diff --git a/userspace/cuda/570.144/scripts/disasm.sh b/userspace/cuda/570.144/scripts/disasm.sh
new file mode 100755
index 0000000..01fea69
--- /dev/null
+++ b/userspace/cuda/570.144/scripts/disasm.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# disasm.sh - objdump helpers for a library under analysis.
+#
+# Usage:
+#   scripts/disasm.sh <file> <hex-vaddr> [before] [after]
+#       disassemble [vaddr-before, vaddr+after] (default 0x40 each side)
+#   scripts/disasm.sh --bytes <file> <hex-start> <hex-end>
+#       raw bytes in the range
+#   scripts/disasm.sh --symbol <file> <name>
+#       exported symbol address/size (readelf dyn-syms)
+#   scripts/disasm.sh --nearest <file> <hex-vaddr>
+#       nearest preceding exported symbol (FUNC) and the offset into it
+set -e
+
+cmd="${1:?usage: disasm.sh [--bytes|--symbol|--nearest] <file> ...}"
+case "$cmd" in
+--bytes)
+    file="$2"; start="$3"; end="$4"
+    objdump -s --start-address="$start" --stop-address="$end" "$file"
+    ;;
+--symbol)
+    file="$2"; name="$3"
+    readelf -sW --dyn-syms "$file" | awk -v n="$name" '$8 == n {print}'
+    ;;
+--nearest)
+    file="$2"; target="$3"
+    python3 - "$file" "$target" <<'PY'
+import subprocess, sys
+path, t = sys.argv[1], int(sys.argv[2], 16)
+out = subprocess.check_output(["readelf", "-sW", "--dyn-syms", path], text=True)
+syms = []
+for line in out.splitlines():
+    p = line.split()
+    if len(p) >= 8 and p[3] == "FUNC":
+        try:
+            syms.append((int(p[1], 16), int(p[2]), p[7]))
+        except ValueError:
+            pass
+syms.sort()
+prev = None
+for a, s, n in syms:
+    if a <= t:
+        prev = (a, s, n)
+    else:
+        break
+if prev:
+    print("nearest exported: %s at 0x%x size 0x%x (+0x%x)" %
+          (prev[2], prev[0], prev[1], t - prev[0]))
+else:
+    print("no exported symbol before 0x%x" % t)
+PY
+    ;;
+*)
+    file="$1"; vaddr="$2"
+    before="${3:-0x40}"; after="${4:-0x40}"
+    start=$(printf '0x%x' $(( vaddr - before )) 2>/dev/null || echo "$vaddr")
+    end=$(printf '0x%x' $(( vaddr + after )) 2>/dev/null || echo "$vaddr")
+    # allow hex arithmetic with shell
+    start=$(python3 -c "print(hex(int('$vaddr',16)-int('$before',16)))")
+    end=$(python3 -c "print(hex(int('$vaddr',16)+int('$after',16)))")
+    objdump -d -Mintel --start-address="$start" --stop-address="$end" "$file"
+    ;;
+esac
diff --git a/userspace/cuda/570.144/scripts/fetch_from_guest.sh b/userspace/cuda/570.144/scripts/fetch_from_guest.sh
new file mode 100755
index 0000000..6ed24f1
--- /dev/null
+++ b/userspace/cuda/570.144/scripts/fetch_from_guest.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# fetch_from_guest.sh - copy a file out of a running Open-IOV guest.
+#
+# The guest agent is used to start a netcat transfer from the guest to the
+# host (QEMU user networking: the host is 10.0.2.2), which is much faster and
+# more robust than pushing tens of megabytes through the agent in base64
+# chunks.
+#
+# Usage: scripts/fetch_from_guest.sh <guest-path> [guest-bind-ip]
+#        e.g. scripts/fetch_from_guest.sh \
+#               /usr/lib/x86_64-linux-gnu/libcuda.so.570.144
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+RUN="$(cd "$HERE/.." && pwd)/run"
+VMEXEC="$HERE/../../../scripts/vgpu_vm.py"
+PORT="${PORT:-47311}"
+
+GUEST_PATH="${1:?usage: $0 <guest-path> [guest-bind-ip]}"
+HOST_IP="${2:-10.0.2.2}"
+OUT="$RUN/$(basename "$GUEST_PATH")"
+
+mkdir -p "$RUN"
+
+# Guest-side md5 for verification
+GUEST_MD5=$("$VMEXEC" exec "md5sum '$GUEST_PATH'" 2>/dev/null | awk '{print $1}')
+[ -n "$GUEST_MD5" ] || { echo "cannot stat $GUEST_PATH in guest"; exit 1; }
+
+echo "[*] guest md5: $GUEST_MD5"
+echo "[*] listening on :$PORT -> $OUT"
+rm -f "$OUT"
+
+nc -l "$PORT" > "$OUT" &
+NCPID=$!
+sleep 1
+
+"$VMEXEC" exec "setsid sh -c 'nc $HOST_IP $PORT < $GUEST_PATH' >/dev/null 2>&1 &" >/dev/null 2>&1 || true
+
+# Wait for the transfer to settle (size stable for two checks)
+prev=0
+for _ in $(seq 1 60); do
+    sleep 2
+    cur=$(stat -c%s "$OUT" 2>/dev/null || echo 0)
+    [ "$cur" -gt 0 ] && [ "$cur" = "$prev" ] && break
+    prev=$cur
+done
+wait $NCPID 2>/dev/null || true
+
+HOST_MD5=$(md5sum "$OUT" | awk '{print $1}')
+echo "[*] host  md5: $HOST_MD5"
+if [ "$GUEST_MD5" != "$HOST_MD5" ]; then
+    echo "[FAIL] md5 mismatch"
+    exit 1
+fi
+echo "[ok] $OUT ($(stat -c%s "$OUT") bytes)"
diff --git a/userspace/cuda/570.144/scripts/ghidra_analyze.sh b/userspace/cuda/570.144/scripts/ghidra_analyze.sh
new file mode 100755
index 0000000..b844300
--- /dev/null
+++ b/userspace/cuda/570.144/scripts/ghidra_analyze.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# ghidra_analyze.sh - import a library into a Ghidra project and run the full
+# auto-analysis once. The project is kept, so later decompilations are cheap:
+#
+#   scripts/ghidra_decompile_project.sh <hex-vaddr> [outfile]
+#
+# Full analysis of a ~70 MB library can take a long time; run it in the
+# background and log to run/ghidra-analyze.log.
+#
+# Usage: scripts/ghidra_analyze.sh <binary>
+#   env: GHIDRA_HOME (default /snap/ghidra/current/ghidra)
+#        PROJECT_DIR (default ../run/ghidra-proj)
+#        PROJECT_NAME (default cuda-analysis)
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+ROOT="$(cd "$HERE/.." && pwd)"
+RUN="$ROOT/run"
+BIN="${1:?usage: $0 <binary>}"
+
+GHIDRA_HOME="${GHIDRA_HOME:-/snap/ghidra/current/ghidra}"
+ANALYZE="$GHIDRA_HOME/support/analyzeHeadless"
+PROJECT_DIR="${PROJECT_DIR:-$RUN/ghidra-proj}"
+PROJECT_NAME="${PROJECT_NAME:-cuda-analysis}"
+
+[ -x "$ANALYZE" ] || { echo "analyzeHeadless not found at $ANALYZE"; exit 1; }
+mkdir -p "$PROJECT_DIR"
+
+echo "[*] importing + analyzing $BIN (this can take a long time)"
+"$ANALYZE" "$PROJECT_DIR" "$PROJECT_NAME" \
+    -import "$BIN" \
+    -overwrite \
+    -scriptPath "$ROOT/ghidra" \
+    -analysisTimeoutPerFile 0
+echo "[ok] analysis complete"
\ No newline at end of file
diff --git a/userspace/cuda/570.144/scripts/ghidra_decompile.sh b/userspace/cuda/570.144/scripts/ghidra_decompile.sh
new file mode 100755
index 0000000..bf95e8c
--- /dev/null
+++ b/userspace/cuda/570.144/scripts/ghidra_decompile.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# ghidra_decompile.sh - decompile the function containing a file vaddr.
+#
+# Runs Ghidra headless with -noanalysis: the program is imported, the crash
+# address is disassembled and turned into a function on demand, then
+# decompiled. That avoids a full auto-analysis pass over a ~70 MB library.
+#
+# Usage: scripts/ghidra_decompile.sh <binary> <hex-vaddr> [outfile]
+#   env: GHIDRA_HOME (default: /snap/ghidra/current/ghidra)
+#        PROJECT_DIR (default: ../run/ghidra-proj)
+#
+# Output: run/<basename>_<vaddr>.c unless outfile is given.
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+ROOT="$(cd "$HERE/.." && pwd)"
+RUN="$ROOT/run"
+BIN="${1:?usage: $0 <binary> <hex-vaddr> [outfile]}"
+VADDR="${2:?usage: $0 <binary> <hex-vaddr> [outfile]}"
+OUT="${3:-$RUN/$(basename "$BIN")_${VADDR}.c}"
+
+GHIDRA_HOME="${GHIDRA_HOME:-/snap/ghidra/current/ghidra}"
+ANALYZE="$GHIDRA_HOME/support/analyzeHeadless"
+PROJECT_DIR="${PROJECT_DIR:-$RUN/ghidra-proj}"
+PROJECT_NAME="${PROJECT_NAME:-cuda-analysis}"
+
+[ -x "$ANALYZE" ] || { echo "analyzeHeadless not found at $ANALYZE"; exit 1; }
+mkdir -p "$PROJECT_DIR" "$RUN"
+
+echo "[*] decompiling $BIN at $VADDR -> $OUT"
+"$ANALYZE" "$PROJECT_DIR" "$PROJECT_NAME" \
+    -import "$BIN" \
+    -overwrite \
+    -noanalysis \
+    -scriptPath "$ROOT/ghidra" \
+    -postScript DecompileAt.java "$VADDR" "$OUT" \
+    -deleteProject \
+    2>&1 | tail -5
+
+[ -f "$OUT" ] || { echo "[FAIL] no output produced"; exit 1; }
+echo "[ok] $OUT"
diff --git a/userspace/cuda/570.144/scripts/ghidra_decompile_project.sh b/userspace/cuda/570.144/scripts/ghidra_decompile_project.sh
new file mode 100755
index 0000000..66d7c90
--- /dev/null
+++ b/userspace/cuda/570.144/scripts/ghidra_decompile_project.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# ghidra_decompile_project.sh - decompile a function from an already-analyzed
+# Ghidra project (run scripts/ghidra_analyze.sh once first). No re-import and
+# no analysis, so this takes seconds.
+#
+# Usage: scripts/ghidra_decompile_project.sh <hex-vaddr> [outfile] [program]
+#   env: GHIDRA_HOME, PROJECT_DIR, PROJECT_NAME (same as ghidra_analyze.sh)
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+ROOT="$(cd "$HERE/.." && pwd)"
+RUN="$ROOT/run"
+VADDR="${1:?usage: $0 <hex-vaddr> [outfile] [program]}"
+OUT="${2:-$RUN/fn_$VADDR.c}"
+PROGRAM="${3:-libcuda.so.570.144}"
+
+GHIDRA_HOME="${GHIDRA_HOME:-/snap/ghidra/current/ghidra}"
+ANALYZE="$GHIDRA_HOME/support/analyzeHeadless"
+PROJECT_DIR="${PROJECT_DIR:-$RUN/ghidra-proj}"
+PROJECT_NAME="${PROJECT_NAME:-cuda-analysis}"
+
+[ -d "$PROJECT_DIR" ] || { echo "no project at $PROJECT_DIR (run ghidra_analyze.sh)"; exit 1; }
+mkdir -p "$RUN"
+
+echo "[*] decompiling $PROGRAM at $VADDR (project) -> $OUT"
+"$ANALYZE" "$PROJECT_DIR" "$PROJECT_NAME" \
+    -process "$PROGRAM" \
+    -noanalysis \
+    -scriptPath "$ROOT/ghidra" \
+    -postScript DecompileAt.java "$VADDR" "$OUT" \
+    2>&1 | tail -3
+
+[ -f "$OUT" ] || { echo "[FAIL] no output"; exit 1; }
+echo "[ok] $OUT"