tools: arm UVM kprobes deterministically (modprobe+wait+verify) and retry pushbuffer pins

Root cause of the cudaMalloc/UVM-init stall on our bring-up path:
guest_kprobes.sh ran while nvidia_uvm was not yet loaded (it loads lazily
with the first CUDA client), so every probe add failed silently (symbols
not in kallsyms) and the host never got the #sem -> pushbuffer-pin
pipeline. The CE push page (0x120021db8) then never translated and the
guest spun forever at DEVICE.

- guest_kprobes.sh: modprobe nvidia_uvm, wait (bounded 30 s) for the
  uvmt symbol, install, and print KP_ARMED/KP_FAIL for bring-up checks.
- semfix_watch.pin_pushbuffer: retry transient/partial gva2gpa misses
  in-call (bounded passes with a short sleep) instead of aborting on the
  first all-miss block; report unresolved page counts.

Validated: 2/2 cold boots (vmkeep6/7) arm the probes and run
victory_mail green in 15-20 s (MAIL + VADD bad=0 checksum=12204516 +
STATUS OK, oracle-matched); pre-fix boots with unarmed probes stalled
4/4.

1 files changed, 21 insertions, 0 deletions

diff --git a/tools/guest-diag/guest_kprobes.sh b/tools/guest-diag/guest_kprobes.sh
index 75b2ba3..cf34833 100644
--- a/tools/guest-diag/guest_kprobes.sh
+++ b/tools/guest-diag/guest_kprobes.sh
@@ -22,6 +22,19 @@ if [ ! -d "$TR/events" ]; then
         mount -t debugfs debugfs "$TR" 2>/dev/null || true
 fi
 
+# The UVM symbols only exist after nvidia_uvm is loaded. It loads lazily with
+# the first CUDA client, but the probes must be armed BEFORE that client's
+# first CE push - otherwise no #sem reaches the host, the push page never
+# translates and the client stalls in cudaMalloc forever. modprobe it and
+# wait (bounded) for the tracking-symbol to appear.
+modprobe nvidia_uvm 2>/dev/null || true
+i=0
+while [ $i -lt 30 ]; do
+    grep -qw uvm_gpu_tracking_semaphore_update_completed_value /proc/kallsyms 2>/dev/null && break
+    i=$((i+1))
+    sleep 1
+done
+
 # An enabled kprobe event cannot be deleted (EBUSY); disable the group first.
 echo 0 > "$TR/events/kprobes/enable" 2>/dev/null || true
 
@@ -81,3 +94,11 @@ echo 65536 > "$TR/buffer_size_kb" 2>/dev/null || true
 echo > "$TR/trace" 2>/dev/null || true
 echo "--- installed probes ---"
 grep -E 'uvmt|uvch|uvbp|uvpbf|uvtw|uvsv' "$TR/kprobe_events" 2>/dev/null || echo "(none)"
+
+# Arming check: the host recovery pipeline (#sem -> pushbuffer pin) depends on
+# these probes; make success/failure explicit for the bring-up scripts.
+if grep -q "uvmt" "$TR/kprobe_events" 2>/dev/null; then
+    echo "KP_ARMED"
+else
+    echo "KP_FAIL uvmt missing"
+fi