wall-2 session-77: vkCreateDevice PASSES - r682 0x7E event validated live (registered slot 13, EVPOST rc=0) and the completion word found/driven: the main-thread wait reads a u32 at +0x80 of a 16 KB rw-s /dev/nvidiactl notification mapping (should-wait = word != 1); a host write of 1 via the guest RAM file (PA from pagemap, file offset PA-0x80000000 above 4G) flips the wait and vkCreateDevice returns success (VKDEVICE ok family=0) - reproduced 3x (r683 manual, r684 nudger, r684d probe+single-word). Tools: t79.py (wait-chain resolver), guest_words.sh, nudge.sh; README-session77.md records the recipe, the residual guest event-dispatch gap (records consumed, notification word not written), and the -4/gdb-attach cautions.

4 files changed, 218 insertions, 0 deletions

diff --git a/tools/guest-diag/wall2/README-session77.md b/tools/guest-diag/wall2/README-session77.md
new file mode 100644
index 0000000..cf2450e
--- /dev/null
+++ b/tools/guest-diag/wall2/README-session77.md
@@ -0,0 +1,75 @@
+# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion
+
+## The two-part unblock
+
+1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
+   nvkms/Vulkan path allocates its completion event as
+   NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
+   registers it alongside 0x79. Live proof (r683, sema_dbg=1):
+
+       GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
+           hParent=0xcaf00001 hObject=0xcaf00003
+       event registered hEvent=0xcaf00003 hSrc=0x0
+           notifyIndex=1006632972 (off 8, slot 13)
+           params=1100e0c1000000007e0000000c00003c0000000000000000
+       EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0
+
+   (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
+   NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
+   (`guest_read == our_write` in the rmstate dump), but the guest's
+   event dispatch does NOT write the event's notification word — the
+   residual delivery gap.
+
+2. **The completion word found and driven (the actual unblock)**. The
+   main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
+   a u32 readiness word: `should-wait = (word != 1)`; the word lives at
+   `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
+   notification buffer; holder chain sub -> obj2 -> holder, resolved
+   live by t79.py). The host can write it directly in the guest RAM
+   file:
+
+       PA = pagemap PFN << 12  (+0x80);  file offset = PA - 0x80000000
+       (above-4G RAM; below-4G would be PA itself per the cycle's
+       guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)
+
+   Writing 1 flipped the wait and **vkCreateDevice returned success**
+   (`VKDEVICE ok family=0`) — reproduced three times:
+   * r683: manual write after a t79 probe;
+   * r684: nudger (all 16 KB mapping words) + slotfill restart;
+   * r684d: probe -> single-word write (the reliable recipe).
+
+## The reliable recipe (per run)
+
+1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
+2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
+   slots re-arm, so re-run slotfill if the client is stuck in the
+   sched_yield spin before the word-wait).
+3. Wait for the main thread in `do_poll` (the word-wait).
+4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
+   host write `01 00 00 00` at `PA-0x80000000`.
+5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
+   image/memory/pipeline setup; its next wait is NOT the same word
+   (the word stays 1) — next session: probe the follow-on state
+   (attach fails while the client spins in the post-device phase;
+   use the /proc-only checks + a longer observation window).
+
+## Cautions learned
+
+* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
+  -4 give-up path (an RM error flag word gets written): restrict nudges
+  to the single probed word.
+* Attaching gdb to the client while it is in the tight post-device spin
+  can hang the attach (leave a `timeout`-wrapped gdb, then kill that
+  gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
+  dead tracer otherwise wedges it).
+* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
+  python, not zsh.
+
+## Tools
+
+* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
+  and prints the client's nvidiactl rw-s mapping sizes.
+* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
+  nvidiactl mapping (pagemap; run in the guest as root).
+* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
+  zero word (multi-page variant; prefer the single-word recipe).
diff --git a/tools/guest-diag/wall2/guest_words.sh b/tools/guest-diag/wall2/guest_words.sh
new file mode 100755
index 0000000..6ac200c
--- /dev/null
+++ b/tools/guest-diag/wall2/guest_words.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+# guest_words.sh - print "VA PA" for the +0x80 word of every 16 KB rw-s
+# /dev/nvidiactl mapping of the vk_frame client (pagemap PFN -> PA).
+PID=$(pgrep -x vk_frame | head -1)
+[ -z "$PID" ] && exit 0
+sudo grep -E 'rw-s' /proc/$PID/maps | grep nvidiactl | while read -r line; do
+  RNG=$(echo "$line" | awk '{print $1}')
+  LO=${RNG%-*}; HI=${RNG#*-}
+  SZ=$((0x$HI - 0x$LO))
+  [ "$SZ" -ne 16384 ] && continue
+  VP=$(( (0x$LO + 0x80) / 4096 ))
+  ENT=$(sudo dd if=/proc/$PID/pagemap bs=8 skip=$VP count=1 2>/dev/null | xxd -e -g8 | awk '{print $2}')
+  [ -z "$ENT" ] && continue
+  PFN=$(( 0x$ENT & 0x7fffffffffffff ))
+  PA=$(( PFN * 4096 + 0x80 ))
+  echo "0x$LO 0x$(printf %x $PA)"
+done
diff --git a/tools/guest-diag/wall2/nudge.sh b/tools/guest-diag/wall2/nudge.sh
new file mode 100755
index 0000000..143b1cd
--- /dev/null
+++ b/tools/guest-diag/wall2/nudge.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+# nudge.sh - host-side readiness-word nudger for the vk_frame wait.
+# Polls the guest for the 16 KB nvidiactl mapping words (guest_words.sh),
+# and writes 1 to any that are still 0 in the guest RAM file.
+# Usage: nudge.sh <duration_seconds>
+DUR=${1:-300}
+F=/dev/shm/vgpu-guest-ram
+END=$(( $(date +%s) + DUR ))
+sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/guest_words.sh ubuntu@127.0.0.1:/tmp/guest_words.sh >/dev/null 2>&1
+while [ "$(date +%s)" -lt "$END" ]; do
+  OUT=$(timeout 20 sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 "sudo bash /tmp/guest_words.sh" 2>/dev/null)
+  echo "[$(date +%H:%M:%S)] words:"; echo "$OUT"
+  echo "$OUT" | while read -r VA PA; do
+    [ -z "$PA" ] && continue
+    PAD=$((PA)); FOD=$((PAD - 0x80000000))
+    [ $FOD -lt 0 ] && continue
+    CUR=$(sudo dd if=$F bs=4 skip=$((FOD/4)) count=1 2>/dev/null | xxd -p)
+    if [ "$CUR" = "00000000" ]; then
+      printf '\x01\x00\x00\x00' | sudo dd of=$F bs=1 seek=$FOD count=4 conv=notrunc 2>/dev/null
+      echo "  NUDGE $PA (va=$VA) -> 1"
+    else
+      echo "  skip $PA (va=$VA) = $CUR"
+    fi
+  done
+  sleep 2
+done
+echo NUDGE-DONE
diff --git a/tools/guest-diag/wall2/t79.py b/tools/guest-diag/wall2/t79.py
new file mode 100644
index 0000000..c2fe381
--- /dev/null
+++ b/tools/guest-diag/wall2/t79.py
@@ -0,0 +1,99 @@
+import gdb, sys
+
+inf = gdb.selected_inferior()
+pid = inf.pid
+base = None
+for line in open("/proc/%d/maps" % pid):
+    if "libnvidia-eglcore.so.570.144" in line:
+        parts = line.split()
+        lo, hi = [int(x, 16) for x in parts[0].split('-')]
+        off = int(parts[2], 16)
+        if off == 0:
+            base = lo
+            break
+if base is None:
+    print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1)
+print("EGLBASE 0x%x PID %d" % (base, pid))
+
+maps = []
+for line in open("/proc/%d/maps" % pid):
+    parts = line.split()
+    try:
+        lo, hi = [int(x, 16) for x in parts[0].split('-')]
+        maps.append((lo, hi, parts[1], parts[-1] if len(parts) > 5 else "anon"))
+    except Exception:
+        pass
+
+def resolve(addr):
+    for lo, hi, perm, name in maps:
+        if lo <= addr < hi:
+            return "0x%x-0x%x %s %s" % (lo, hi, perm, name)
+    return "?"
+
+def rd(a, n):
+    return inf.read_memory(a, n).tobytes()
+def u32(a):
+    return int.from_bytes(rd(a, 4), 'little')
+def u64(a):
+    return int.from_bytes(rd(a, 8), 'little')
+
+state = {"done": False}
+
+class B(gdb.Breakpoint):
+    def __init__(self, spec, fn, limit=100000):
+        super().__init__(spec, internal=False)
+        self.fn = fn; self.limit = limit; self.hits = 0
+        self.silent = True
+    def stop(self):
+        self.hits += 1
+        try:
+            stop = bool(self.fn(self))
+        except Exception as e:
+            print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e))
+            stop = False
+        return stop or self.hits >= self.limit
+
+def loophead(bp):
+    if state["done"]:
+        return False
+    state["done"] = True
+    rbx = int(gdb.parse_and_eval("$rbx"))
+    sub = u64(rbx+0x70)
+    print("WAIT rbx=0x%x sub=0x%x" % (rbx, sub))
+    try:
+        obj2 = u64(sub+0x58)
+        off = u32(obj2+0x60)
+        holder = u64(obj2+0x58)
+        base2 = u64(holder+0x58)
+        word_addr = base2 + off
+        val = u32(word_addr)
+        print("WAIT obj2=0x%x off=0x%x holder=0x%x base=0x%x word=0x%x val=0x%x" %
+              (obj2, off, holder, base2, word_addr, val))
+        print("WAIT word map:", resolve(word_addr))
+        print("WAIT holder obj2+0x50=0x%x obj2+0x64=0x%x" % (u64(obj2+0x50), u32(obj2+0x64)))
+        try:
+            hd = rd(holder, 0x30)
+            print("WAIT holder dump:", " ".join("%02x" % b for b in hd))
+        except Exception as e:
+            print("WAIT holder dump err", e)
+    except Exception as e:
+        print("WAIT chain err", e)
+    # dump the client's nvidiactl rw-s mappings
+    print("WAIT nvidiactl maps:")
+    for lo, hi, perm, name in maps:
+        if "nvidiactl" in name and "rw-s" in perm:
+            print("  0x%x-0x%x size=0x%x" % (lo, hi, hi-lo))
+    return True
+
+B("*0x%x" % (base+0xdcebd9), loophead, 1)
+
+try:
+    gdb.execute("continue")
+except gdb.error as e:
+    print("CONT-ERR %s" % e)
+print("TRACE-END")
+try:
+    gdb.execute("detach")
+except gdb.error as e:
+    print("DETACH-ERR %s" % e)
+gdb.execute("quit")