# WALL #2 (GR channel-init release) — session 73 findings + tools

## Goal
The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the
kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA
0x4480000) and waits ~11 s per attempt for a completion the sema service
cannot apply: it cannot translate the push VA ("false translation for push
va=0x4480000 ... rescuing"), so no release is decoded and the wait times out.

## Findings (r674/r675, 8+ client runs)
1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on
   `uvm_hal_turing_host_semaphore_release` never fired during a client run.
2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events`
   refuses them ("Could not probe notrace function ... [nvidia]"). UVM module
   probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine.
3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1,
   lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2
   are consumed as "skipping stale entry ... after ~150 failed rescues";
   gr_wedge=1.
4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found
   NO push data: neither the CE release group (0x20038090) nor the host-class
   release header (0x20050017) appears outside driver code pages.
5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and
   `nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] =
   ((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on
   entry[1]).
6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry`
   (nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits
   for the GR channel) -> the GR channel's ring entries are written by an
   unidentified writer (possibly the emulation/GSP path); the push content
   was never located in guest RAM.
7. RM window records + memdescs (host-readable, stable layout):
   - a page holds the client channel's window records
     {VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000}
     (ring), each inside head/tail list nodes;
   - memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page:
     the ring memdesc's FIRST_PA == the service's validated ring pa
     (verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the
     window's first VA page;
   - the push memdesc `{0x20000, 0x03008000, PA}` was injected as
     0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping
     va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page
     WITHOUT the "false translation" line - but decoded no release (the page
     content does not look like the method stream; later candidates read as
     zero).
8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function
   `channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel,
   channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during
   a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere.

## Tools (this directory)
- wall2_resolver2.py — v2: CE release-group content search + tuple arrays
  (pa&~0xfff, strides 0x20/0x40/0x60). No candidates live.
- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe.
- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check
  (this one injected successfully; rdesc[0] bug fixed in v5).
- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every
  0x20000 memdesc PA found in pages holding both window records.
- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the
  host release header (both negative; useful for re-checks).
- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc
  scans).
- trace_push674.py — VA-reference tracer with direct-map pointer following.
- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry
  probes, channel struct dumps).

## Next steps (ranked)
1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA;
   get the VA from the channel struct; find the channel struct via the ring
   memdesc page's head/tail pointers). The writer's caller holds the push's
   CPU mapping -> PA -> inject -> service decodes the release.
2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's
   kernel stack; the current candidates never fired) and read the semaphore
   value it polls; bump it host-side via the ring-page PA (the finish payload
   may live at ring_cpu + channel->0x50).
3. Or: fix the service's rescue matcher to accept GR-shaped releases once the
   push content is captured (see item 1).

## Ops
- pkill -f self-matches: the bracket trick fails when the pattern text appears
  elsewhere in the agent's own command line; kill by explicit PID from
  `ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`.
- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host
  reboot). gdb674d/gdb675d sessions left attached (never kill gdbs).
- Never reload the guest nvidia modules; never kill clients mid-flight.