wall-2 session 74: the client's 11s stalls are userspace completion-slot spins; the next wall is the polled RM control 0x2080a097

Root-caused the vk_frame client's exact 11.00s gaps: a pure sched_yield +
clock_gettime userspace spin inside libnvidia-eglcore's EGL/GL init, waiting
for completion slots (u16 at slot+0xe == 0xffff) in /dev/nvidiactl mappings.
Guest-side filling of the slot flags lets the client proceed immediately, but
vkCreateDevice still fails -4: with the slots filled the client settles into
an NVOS54 poll of RM control 0x2080a097 (paramsSize=1168) which our emulation
answers with the generic all-zero reply, so the poll never completes.

Also captured: sema_push_raw dumps proving the GR channel-init push contains
NO release (entry0 decodes 2 methods then zeros; entry2 SET_OBJECT 0xc597 then
zeros), and the negative result that neither channelFillGpFifo nor
nvWriteGpEntry runs for this channel's kick.

New probes: gdbpoll6/gdbu16/gdbfill/gdbwatch/gdbparam/gdbgap3/gdbspin (slot
check, flag fill, NVOS54 param dump, backtraces), slotfill{,2}.sh (guest slot
watcher), stracevk/fastrun{,2}.sh (strace+perf recipes), find_ring_va.py and
vmap_watch2.py (vmap/kcore watchers), README-session74.md.

1 files changed, 66 insertions, 0 deletions

diff --git a/tools/guest-diag/wall2/README-session74.md b/tools/guest-diag/wall2/README-session74.md
new file mode 100644
index 0000000..58587eb
--- /dev/null
+++ b/tools/guest-diag/wall2/README-session74.md
@@ -0,0 +1,66 @@
+# WALL #2 / device-init completion — session 74 findings (r675)
+
+## The client's "11 s stalls" fully root-caused (this is NOT the channel-init release)
+strace (ioctl-only) shows THREE exact 11.00 s gaps with NO ioctl in flight; a
+full strace shows the client doing a pure `sched_yield` spin (~57K/s) +
+`clock_gettime` (perf) inside **libnvidia-eglcore.so.570.144** — i.e. a
+userspace timeout loop. gdb backtraces place the spin inside the Vulkan ICD's
+EGL/GL init (`vkCreateInstance` -> libGLX_nvidia vk_icdNegotiate... ->
+libEGL -> eglcore wait loop).
+
+## The wait target: completion slots in /dev/nvidiactl mappings
+- The wait loop (eglcore file offsets ~0xb284b0 delay, ~0xb33d00 wait loop,
+  ~0xb1c750 slot check) polls a slot table on the client's heap object:
+  table at object+0x1428, count at object+0x22efc.
+- Slot check: `esi` is offset by 0x43; for esi-0x43 > 4 it reads a **u16 at
+  slot+0xe** and treats `0xffff` as "empty/not complete".
+- gdb sampling during the stall: 13164 checks with `u16@slot+0xe = 0xffff`,
+  1 with `0` — the client spins until the flag changes.
+- The slot pointer = a **/dev/nvidiactl rw-s mapping** (page-sized; base+0x20
+  = slot). These are the client's mappings of RM memory (the channel
+  notifier pages: service logs `notif=0x11627f060` per channel).
+
+## Filling the slots (guest-side) WORKS mechanically but is not the -4 cause
+- `slotfill.sh` / `slotfill2.sh`: while vk_frame runs, write 0 to
+  `base+0x2e` of every small /dev/nvidiactl rw-s mapping when it reads 0xffff.
+- Result: the 11 s gaps vanish (5 fills within ~2 s; the client proceeds
+  immediately) — **but vkCreateDevice still fails -4**. So the 11 s waits are
+  not the failure; they are an init-side slowdown the client tolerates.
+
+## The next wall: the polled RM control 0x2080a097
+- With the slots filled, the client settles into **NVOS54 (ioctl nr=0x2a)
+  polling every ~1.25 s** (strace), and the host dmesg shows
+  `GSP_RM_CONTROL cmd=0x2080a097 hClient=0xc1d000bb hObject=0xbeef0004
+  paramsSize=1168 flags=0x0` -> `unknown cmd=0x2080a097 -> generic SUCCESS`
+  (all-zero reply), 36 polls in 45 s.
+- The NVOS54 struct (gdb dump): {hClient=0xc1d000c0, hObject=0xbeef0004,
+  cmd=0x2080a097, flags=0, params=ptr, paramsSize=0x490, status=ptr, ...}.
+- The params buffer = 1168 bytes of 32-byte records with a `0x25` field at
+  +0x10 (dump in /tmp/opencode evidence; gdbparam.sh captures it live).
+- The client polls until the control's status/reply signals completion; our
+  generic all-zero reply never does. **This is the wall that must be solved to
+  get vkCreateDevice past -4** (either a real reply for 0x2080a097 or making
+  the polled status field change).
+
+## Other session-74 results
+- The GR ring-entry writer was NOT found by disassembly-guessing: nvidia.ko
+  `channelFillGpFifo` and nvidia-modeset `nvWriteGpEntry` never fire for this
+  channel's kick (gdb hbreaks); nvidia/modeset kprobes are `notrace`; QEMU gdb
+  hbreak works.
+- `sema_push_raw` dumps (module param) captured the service's reads of the
+  push: entry0 decodes only 2 methods then zeros (`consumed=0x288/1432`),
+  entry2 starts `SET_OBJECT 0xc597` then zeros; **no release in the push**
+  (`have_rel=0` for all three entries) — the channel-init release theory for
+  these waits is dead.
+- `find_ring_va.py` / `vmap_watch2.py`: the ring entries are NOT in any
+  vmallocinfo area during the run; the nv_vm_map_pages areas hold the CE
+  pushes; the GR push content is not found by content searches.
+
+## Files
+- gdbpoll6.gdb, gdbu16.gdb, gdbfill.gdb, gdbwatch.gdb, gdbparam.sh,
+  gdbgap3.sh, gdbspin.sh — the gdb probes (slot check, u16 flag, fill,
+  NVOS54 param dump, backtraces).
+- slotfill.sh / slotfill2.sh — the guest-side slot-fill watcher.
+- stracevk.sh, fastrun.sh, fastrun2.sh — the strace/perf recipes.
+- perfstall5.sh — perf sampling with map-relative offsets.
+- find_ring_va.py, vmap_watch2.py — the vmap/kcore watchers.