wall-2 forensics tools: GR channel-init push localization (session 73)

Tools and findings from the live wall-2 campaign (r674/r675): the vk_frame
client's GR channel-init push (va=0x4480000 len=1432) cannot be translated by
the sema service, so no release is applied and vkCreateDevice fails -4.

Captured: the RM window-record/memdesc layout (ring memdesc PA == the
service's validated ring pa; push memdesc PA injected live and read by the
service without a false-translation line), the CE-only shape of the service
rescue matcher, the notrace status of nvidia/modeset kprobes (QEMU gdb hbreak
works), and the ring-entry encoding from channelFillGpFifo/nvWriteGpEntry
(neither runs for this channel's kick).

New tools: wall2_resolver{2..5}.py, wall2_livecap{1..3}.py, RAM structure
scanners (scan_r673*.py, trace_push674.py), gdb675*.gdb probes, plus a
README with the ranked next steps.

1 files changed, 85 insertions, 0 deletions

diff --git a/tools/guest-diag/wall2/README.md b/tools/guest-diag/wall2/README.md
new file mode 100644
index 0000000..1ef9558
--- /dev/null
+++ b/tools/guest-diag/wall2/README.md
@@ -0,0 +1,85 @@
+# WALL #2 (GR channel-init release) — session 73 findings + tools
+
+## Goal
+The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the
+kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA
+0x4480000) and waits ~11 s per attempt for a completion the sema service
+cannot apply: it cannot translate the push VA ("false translation for push
+va=0x4480000 ... rescuing"), so no release is decoded and the wait times out.
+
+## Findings (r674/r675, 8+ client runs)
+1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on
+   `uvm_hal_turing_host_semaphore_release` never fired during a client run.
+2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events`
+   refuses them ("Could not probe notrace function ... [nvidia]"). UVM module
+   probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine.
+3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1,
+   lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2
+   are consumed as "skipping stale entry ... after ~150 failed rescues";
+   gr_wedge=1.
+4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found
+   NO push data: neither the CE release group (0x20038090) nor the host-class
+   release header (0x20050017) appears outside driver code pages.
+5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and
+   `nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] =
+   ((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on
+   entry[1]).
+6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry`
+   (nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits
+   for the GR channel) -> the GR channel's ring entries are written by an
+   unidentified writer (possibly the emulation/GSP path); the push content
+   was never located in guest RAM.
+7. RM window records + memdescs (host-readable, stable layout):
+   - a page holds the client channel's window records
+     {VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000}
+     (ring), each inside head/tail list nodes;
+   - memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page:
+     the ring memdesc's FIRST_PA == the service's validated ring pa
+     (verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the
+     window's first VA page;
+   - the push memdesc `{0x20000, 0x03008000, PA}` was injected as
+     0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping
+     va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page
+     WITHOUT the "false translation" line - but decoded no release (the page
+     content does not look like the method stream; later candidates read as
+     zero).
+8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function
+   `channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel,
+   channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during
+   a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere.
+
+## Tools (this directory)
+- wall2_resolver2.py — v2: CE release-group content search + tuple arrays
+  (pa&~0xfff, strides 0x20/0x40/0x60). No candidates live.
+- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe.
+- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check
+  (this one injected successfully; rdesc[0] bug fixed in v5).
+- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every
+  0x20000 memdesc PA found in pages holding both window records.
+- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the
+  host release header (both negative; useful for re-checks).
+- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc
+  scans).
+- trace_push674.py — VA-reference tracer with direct-map pointer following.
+- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry
+  probes, channel struct dumps).
+
+## Next steps (ranked)
+1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA;
+   get the VA from the channel struct; find the channel struct via the ring
+   memdesc page's head/tail pointers). The writer's caller holds the push's
+   CPU mapping -> PA -> inject -> service decodes the release.
+2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's
+   kernel stack; the current candidates never fired) and read the semaphore
+   value it polls; bump it host-side via the ring-page PA (the finish payload
+   may live at ring_cpu + channel->0x50).
+3. Or: fix the service's rescue matcher to accept GR-shaped releases once the
+   push content is captured (see item 1).
+
+## Ops
+- pkill -f self-matches: the bracket trick fails when the pattern text appears
+  elsewhere in the agent's own command line; kill by explicit PID from
+  `ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`.
+- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host
+  reboot). gdb674d/gdb675d sessions left attached (never kill gdbs).
+- Never reload the guest nvidia modules; never kill clients mid-flight.