wall-2 session 74: the client's 11s stalls are userspace completion-slot spins; the next wall is the polled RM control 0x2080a097

Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield +
clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting
for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings.
Guest-side filling of the slot flags lets the client proceed immediately, but
vkCreateDevice still fails -4: with the slots filled the client settles into
an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation
answers with the generic all-zero reply, so the poll never completes.

Also captured: sema_push_raw dumps proving the GR channel-init push contains
NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then
zeros), and the negative result that neither channelFillGpFifo nor
nvWriteGpEntry runs for this channel's kick.

New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot
check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot
watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and
vmap_watch2.py (vmap/kcore watchers), README-session74.md.

1 files changed, 30 insertions, 0 deletions

diff --git a/tools/guest-diag/wall2/slotfill.sh b/tools/guest-diag/wall2/slotfill.sh
new file mode 100644
index 0000000..a3416a9
--- /dev/null
+++ b/tools/guest-diag/wall2/slotfill.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+# Watch the vk_frame client and fill the completion-slot u16 (base+0x2e) in
+# every small /dev/nvidiactl rw-s mapping while it waits.
+cd /tmp
+rm -f /tmp/slotfill.log
+(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk76.out 2>&1 &)
+PID=""
+for i in $(seq 1 100); do PID=$(pgrep -x vk_frame | head -1); [ -n "$PID" ] && break; sleep 0.1; done
+echo "pid=$PID" >> /tmp/slotfill.log
+END=$((SECONDS+60))
+while [ $SECONDS -lt $END ]; do
+  [ -d /proc/$PID ] || break
+  for m in $(grep 'nvidiactl' /proc/$PID/maps | awk '$2 ~ /rw-s/ {print $1}'); do
+    lo=${m%-*}; hi=${m#*-}
+    size=$((0x$hi - 0x$lo))
+    [ $size -gt 65536 ] && continue
+    off=$((0x$lo + 0x2e))
+    v=$(dd if=/proc/$PID/mem bs=1 skip=$off count=2 2>/dev/null | xxd -p)
+    if [ "$v" = "ffff" ]; then
+      printf '\x00\x00' | dd of=/proc/$PID/mem bs=1 seek=$off count=2 conv=notrunc 2>/dev/null
+      echo "$(date +%s.%N) FILL $m+0x2e was ffff" >> /tmp/slotfill.log
+    fi
+  done
+  sleep 0.2
+done
+echo "done" >> /tmp/slotfill.log
+echo "=== client ==="
+cat /tmp/vk76.out
+echo "=== fills ==="
+grep -c FILL /tmp/slotfill.log