path: root/tools/guest-diag/wall2/README-session77.md

# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion

## The two-part unblock

1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
   nvkms/Vulkan path allocates its completion event as
   NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
   registers it alongside 0x79. Live proof (r683, sema_dbg=1):

       GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
           hParent=0xcaf00001 hObject=0xcaf00003
       event registered hEvent=0xcaf00003 hSrc=0x0
           notifyIndex=1006632972 (off 8, slot 13)
           params=1100e0c1000000007e0000000c00003c0000000000000000
       EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0

   (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
   NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
   (`guest_read == our_write` in the rmstate dump), but the guest's
   event dispatch does NOT write the event's notification word — the
   residual delivery gap.

2. **The completion word found and driven (the actual unblock)**. The
   main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
   a u32 readiness word: `should-wait = (word != 1)`; the word lives at
   `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
   notification buffer; holder chain sub -> obj2 -> holder, resolved
   live by t79.py). The host can write it directly in the guest RAM
   file:

       PA = pagemap PFN << 12  (+0x80);  file offset = PA - 0x80000000
       (above-4G RAM; below-4G would be PA itself per the cycle's
       guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)

   Writing 1 flipped the wait and **vkCreateDevice returned success**
   (`VKDEVICE ok family=0`) — reproduced three times:
   * r683: manual write after a t79 probe;
   * r684: nudger (all 16 KB mapping words) + slotfill restart;
   * r684d: probe -> single-word write (the reliable recipe).

## The reliable recipe (per run)

1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
   slots re-arm, so re-run slotfill if the client is stuck in the
   sched_yield spin before the word-wait).
3. Wait for the main thread in `do_poll` (the word-wait).
4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
   host write `01 00 00 00` at `PA-0x80000000`.
5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
   image/memory/pipeline setup; its next wait is NOT the same word
   (the word stays 1) — next session: probe the follow-on state
   (attach fails while the client spins in the post-device phase;
   use the /proc-only checks + a longer observation window).

## Cautions learned

* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
  -4 give-up path (an RM error flag word gets written): restrict nudges
  to the single probed word.
* Attaching gdb to the client while it is in the tight post-device spin
  can hang the attach (leave a `timeout`-wrapped gdb, then kill that
  gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
  dead tracer otherwise wedges it).
* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
  python, not zsh.

## Tools

* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
  and prints the client's nvidiactl rw-s mapping sizes.
* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
  nvidiactl mapping (pagemap; run in the guest as root).
* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
  zero word (multi-page variant; prefer the single-word recipe).

## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6)

* The post-device spin is the **GL module bring-up**: the main thread's
  A8-layer retry loop (eglcore 0xa85d24) works on a work item whose
  embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6
  submit layer (0xd624e0) ORs **0x200000** into its pending word (the
  same engine-pending bit SESSION 75/76 watched on E+0x150), then the
  A8 loop yields via the B284B0 spin helper (sched_yield) waiting for
  the GL ucode init to complete on the GR engine.
* **The push windows ARE now readable** (first time): the resolver6
  recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} ->
  same-page 0x20000 memdesc PA -> `sema_map_add`) injected
  `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d
  68000000` = the 2-method semaphore-like push) and `0x4ca0000 ->
  0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5,
  `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d
  70/68`). The service now decodes the streams (`sema-push ... tail=0dc0
  012000000070...`) where SESSION 73-74 saw "unread".
  CAUTION: the resolver injects every candidate; a zero-page candidate
  injected last replaces the good mapping for that VA (re-inject the
  good page after the scan).
* **Remaining gate**: the service still reports `have_rel=0` for every
  GR push (the CE-shaped release matcher) and — decisively — the
  GR-forward machinery is NOT engaging: `drain held=4 starved=149
  forwarded=0 ... gr_nocap=4`. The GL ucode init work is never
  forwarded/executed, so the completion (and the 0x200000 bit clear)
  never arrives. Next: the GR forward/capture path for these channels
  (operand capture + release decode for the GR shape), then the client
  proceeds to VKFRAME.

## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe

* The GR/GL init releases decode only with the low-aperture acceptance
  (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms,
  commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group
  `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0
  pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ...
  02000001`); `have_rel=1`, `applied` rising, un-translated releases
  queue in svc_pending (417 observed).
* Corrected pair-specific push injections for r686:
  `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc),
  `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus
  `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window).
  CAUTION: resolver7 injects each page's descs for BOTH pairs (union
  loop) - fix it to keep per-pair pages or inject manually from the
  per-pair scan. The window-record layout is {tag, VA, SIZE, 0,
  "tail", ...} stride 0x40 (search the VA u64 with size at +8 and
  "tail" at +0x10).
* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no
  window records in the known record pages; the service cannot resolve
  them (targeted rm_mem miss) so the releases stay pending and the
  NvOgl wait spins. Find their PAs (client mapping pages / RM
  VA-tracker / live poll-watch), inject, and the pending releases
  should apply.

## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying

* **The record scan fix**: the RM window record is 0x40 bytes
  {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is
  stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT
  "tail" (the hex display `7461696c` reads as "tail" only as a number).
  Searching for b'tail' finds nothing; `wall2_recscan.py` searches for
  the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at
  -0x20/-0x18/-0x10. It found **all 75 records** in one pass.
* **The GR init semaphore windows** (r686):
  `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content:
  +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 /
  rel=0x4b17e30 pay=1 payloads) and
  `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the
  release applied there: +0xb10 became 2). Injections:
  `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and
  `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`.
* **State after the injections**: the low-VA releases decode
  (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at
  ~418 (the client re-kicks continuously); the NvOgl wait still spins
  (the A8 loop compares per-entry counter tables r14+0x5e8 vs the
  wanted set - the completion is not just the semaphore value). Next:
  the A8 counter tables' source (what fills r14+0x5e8 / the "wanted"
  table - likely the GR channel's own progress counters) or the GR
  forward/ucode execution (`forwarded=0 gr_nocap=7` still).

## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass

* `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a
  translation for the rank slot VA `0x20400fec0 + (19-r)*0x10`
  (r = gr_index = rank by ascending handle among the guest's GR
  channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live
  from the guest pin stream) or `svc_counter_derive` (the
  vaspaces/discovered/targeted walk chain, `sema_counter_derive=1`
  default). Both miss for the counter page -> no "counter auto" log,
  `pub_calls=0` -> the A8 loop's per-entry counter tables never fill and
  the NvOgl wait spins.
* The client does NOT map the counter pool: `/proc/<pid>/pagemap` for
  0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not
  present) and no VMA covers it - it is a UVM kernel-side object, so the
  pin stream cannot resolve it through the client's page tables. The
  stream's log shows `sem=0` (zero semaphore resolutions) even though
  the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44
  `20400f` entries; the stream's "channel-anchored expansion" for the
  UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the
  machinery that must find them (it is written for the CUDA second-client
  stall - "the only live uvmt event is the spinning sem").
* Bypass available now: the `sema_counter` parameter registers a channel
  slot manually - `echo <h_object>,<pa> > .../parameters/sema_counter`.
  Once the counter page's PA is known (via the guest kcore/UVM walk),
  this wires the publish path without a kernel rebuild.
* The rank slots for the GL client's channels (handles 0xbeec0100,
  0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 /
  0x20400ffb0.

## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported

* The client's own pagemap does NOT contain the counter pool: the
  semfix_uvmt tool's resolver (`guest_pmap_win.py <pid> 0x20400f000
  0x204010000 64`) returns NOTHING, as does the UVM pool range
  (`0x121000000 0x121100000`). The tool's `pin_counter_page` has
  therefore never succeeded for these clients (the STALL WATCHDOG's
  "re-pin counter" attempts across sessions produced no "counter:
  pinned" line).
* VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the
  emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its
  window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow
  there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`);
  poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and
  then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but
  did NOT move the client's wait. The vidmem route is not (or not only)
  the completion path; the A8 loop's tables have another source.
* gdb-attach hazard (again): attaches to the client in its post-device
  spin hang with no output; two gdb chains had to be killed by PID
  (never a pattern) and the client SIGCONT'd. Probe only while the
  client is in do_poll.

## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed)

* The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's
  method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns
  `(hi<<32)|count`. The count comes from **b1bf00**:
  - fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`;
  - slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)**
    which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client
    READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc));
    the loop takes the MIN of those reads.
* Live (r687, object handle 0xbeec0100): the window = VA
  0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0
  (slow path active), obj+0x1078=7 (fast-path field unused).
* **The host CAN write it**: resolving the window's PA via the client's
  pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via
  the RAM file) made the client read it - **obj+0x1070 = 0x100** (the
  method stores the returned count there) and the A8 "actual" table
  entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's
  completion counter can be driven from the host.
* The wait still spins: the A8 loop's first scan (compare
  a85e1d: actual vs want, want observed 0) passes with our value, but
  the loop continues into a second scan (compare a85ec1, driven by
  b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those
  (the second scan's actual source and the rsp+0x38 flag), and whether
  the other objects (entries 0-2 of the A8 table) need the same window
  writes.
* The other objects' windows were not observed (only 0xbeec0100 called
  a82f30 in the sampled window); the nvidia0-side window (obj+0x1478)
  is NOT present in the pagemap (VM_IO; host cannot write it via the
  RAM file).

## ADDENDUM (r687, session 84) — the a86038 want-scan mapped: ALL WANTS PASS with the driven window value; the residual gate is the A8 loop's internal callback/retry machinery

* Live dump of the a86038 scan (BP at 0xa8608e, per-index):
    i=0 want=0x0 actual=0x0 pend=no
    i=1 want=0x0 actual=0x0 pend=no
    i=2 want=0x0 actual=0x0 pend=no
    i=3 want=0x6 actual=0xffffffff00000100 pend=no
  (want from a82fd0(job_obj, i) = the job object's +0x98 array maxed
  against global+0x600; actual from r14+0x5e8 = our driven counter).
  NO pending entries -> the scan completes.
* Writing 0x100 to the ENTIRE readback window (offsets 0x0..0x1ff0 of VA
  0x7827e6c12000) did not clear the wait either.
* The loop still cycles: the bt stays at the a85f08-a85f1a yield call
  (b284b0(*(rsp+0x10), 0xb) - a bounded pause/yield spin of ~0x14
  units, hence the slow cadence), and the exit-attempt path a85f30
  decrements *(rbp) (w0=1) while firing the global callbacks
  (0x206f790/0x206f7a0 vtables: *(%rax), *0x10, *0x18) with args from
  rbp+0x30/0x38/0x40. The remaining gate is this callback/retry
  machinery (or another wait object), not the readback values.
* NEXT: resolve the global callback targets (0x206f790/0x206f7a0
  vtables at runtime) and trace the a85f30 exit path (what the
  callbacks do and what makes *(rbp) reach 0); or find how the real GPU
  writes the readback window (the op-4 methods 0x800000bb/0xbc the
  service skips) and emulate that write so the client's own machinery
  completes.

## ADDENDUM (r687, session 85) — **THE NvOgl WAIT CLEARED** (the session-count gate) and the client advanced to vkAllocateMemory (new wall: -2 OUT_OF_DEVICE_MEMORY)

* The A8 loop's exit requires the session count `*(u32*)(r14+0x5bc)`
  (= lib_base + 0x2070b7c, the global at +0x20705c0+0x5bc) to reach 0;
  the loop head increments `*(rbp)` and the entry scans pass with the
  driven window value, but with count=4 the exit checks
  (a85b48/a85e78: `if (r12d == 0)`) are never taken - the loop waits for
  the engine sessions to complete (count -> 0).
* **DIRECT TEST (succeeded)**: `set {unsigned int}0x<base+0x2070b7c> = 0`
  via gdb -> the loop exited, the NvOgl init completed, and the client
  ADVANCED past the post-device wait for the first time:

      VKDEVICE ok family=0
      VKFAIL vkAllocateMemory(dev, &ma, NULL, &img_mem) -> -2 (line 204)

* **NEW WALL**: `vkAllocateMemory` for the image (DEVICE_LOCAL) returns
  -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY) at vk_frame.c:204 - the guest
  driver's heap/type selection fails before any alloc RPC (no
  GSP_RM_ALLOC in dmesg). Next: the memory-heap reporting
  (FB_GET_INFO_V2 0x20801303 is handled; check the heap-size/type
  queries the driver uses for the DEVICE_LOCAL type, and the emulated
  VRAM size the driver sees).
* Recipe for the next run: slotfill + word nudge -> VKDEVICE ok; then
  write 0 to lib_base+0x2070b7c (gdb; the harness) once the NvOgl wait
  starts (the A8 loop's count gate); then the memory-alloc wall.

## ADDENDUM (r689, session 86) — **THE DEVICE_LOCAL -2 WALL CRACKED**: the 512 MiB emulated VRAM was the gate; the "nvidia-nvidia-vgpu-turing-2g" mdev type (2 GiB) lets the allocation proceed

* Evidence chain: the ICD hooks (t91.py) captured the client's allocation
  `ALLOC sType=5 size=0x4000 typeIndex=1` (a valid 16 KB DEVICE_LOCAL
  request); the standalone memory-properties dump (vk_memprops.py)
  showed device-local heap[0] = 512 MiB with a separate 2936 MiB
  non-local heap. With 512 MiB the driver failed the allocation
  (VK_ERROR_OUT_OF_DEVICE_MEMORY, no RM RPC).
* **FIX TESTED**: the cycle with `VGPU_MDEV_TYPE=nvidia-nvidia-vgpu-turing-2g`
  (start_vm_keep.sh honours the env var) -> `heap[0] size=0x80000000
  (2048 MiB)` (vk_memprops.py) -> the SAME client with the SAME recipe
  (slotfill + word nudge + count auto-clear) reaches `VKDEVICE ok` and
  the allocation NO LONGER fails -2: the client proceeds into
  `os_schedule` (a deeper wait).
* REMAINING in the new state: the client's later pushes (va=0x44852d4 /
  0x4ca52d4, then +0x586c) extend beyond the first window page and read
  `pa=0x0`/zeros (the sema injections covered page 1 only; the window's
  page list is scattered - the memdesc carries only the FIRST page).
  NEXT: extend the window injections to the full scattered page list
  (from the memdesc's page array) or accept 1-page pushes; then the
  allocation completes and the client proceeds to the buffer/pipeline.
* Tools: vk_memprops.py (standalone memory-properties dump; safe
  alongside other clients), vk_alloc_test.py (standalone allocation
  reproducer - NOTE: it hits vkCreateDevice -4 without the harness
  (word nudge + count clear), so it cannot replace the vk_frame run),
  t91.py (auto-clear + ICD memprops/alloc hooks).

## ADDENDUM (r689, session 87) — the window page-list hunt: the pages are scattered; content matches found but the VA<->page relation is unresolved

* The 2 GiB cycle's window records (recscan89): `0x4480000` descs =
  {0x107c47000 (live init stream), 0x103c06000, 0x12ecc2000};
  `0x4ca0000` descs = {0x11dbfa000, 0x12f398000 (live init stream)}.
  The window's later offsets (va+0x52d4, +0x586c, +0x9000) at
  first_PA+offset are ALL ZEROS -> the windows are NOT contiguous
  (scattered pages).
* The memdesc record (page 0x106128000) is {size=0x20000, flags
  0x03008000, PA=0x107c47000, kernel pointers 0xffff8d2e5e21ec50 /
  0x4fe7ec50 / 0x461287e8 ... , "head"/"tail"} — the page list hangs off
  the kernel pointers. A guest kcore walk of those pointers returned
  "unreadable" (stale addresses by the time of the walk; the walk must
  happen on FRESH pointers in the same pass as the dump).
* Content search for the distinctive init-stream prefix (`00 80 01 20
  b5 c5 00 00 04 00 04 20`) found the window pages:
  0x107c47000+0x0 (the injected init page), 0x103c06000+0x858,
  0x11dbfa000+0x858, 0x126d69000+0x59c — the later pages hold the same
  engine-init streams at arbitrary offsets; the VA<->page relation is
  NOT linear (the +0x59c push appears in pages whose +0x0 is zero), so
  the mapping cannot be reconstructed by offset alone.
* NEXT: (a) dump the memdesc + kcore-walk its page-list pointers in ONE
  guest pass (fresh addresses); or (b) inject the content-matched pages
  at trial VAs and watch the service's reads; or (c) the service-side
  decode of the window page lists. The client (2632) blocks in
  os_schedule at the allocation — the unread later pushes are the
  suspected gate.

## Session 88 addendum - the push-page fix, the gdb-kill wedge, the ring-mapping gap

**1. PATH-5 RULE (the push pages).** A GR window's pushes do NOT live at
the window base page: the r689-family windows carry their method streams in
WINDOW PAGE 5 (va offsets 0x52d4/0x559c/0x586c/0x596c all live in
0x5000-0x5fff). Standing injections at the base page resolve pa=0x0 for the
pushes (the service's `sema-pushall ... pa=0x0` lines) and the channel
holds/skips entries as "GR no-capture". The fix: page PA =
match_filoff - (va & 0xfff) + 0x80000000 for a content match of the push's
first8 bytes constrained to the same in-page offset; then
`echo 0x<va&~0xfff>,0x<pa>,0x1000,1 > .../sema_map_add`. Proven on r689:
channel beec0104's pushes at va=0x4d00000 (page -> PA 0x12f398000) went
from first8=00000000 to first8=20012000 0000c5c0, and `fast sweep`
applied 3281 -> 3809 (+30/sweep sustained).

**2. host_resolve.py** automates 1: it polls dmesg for sema-pushall lines
with pa=0x0 and non-zero first8, searches /dev/shm/vgpu-guest-ram
(offset-constrained), and injects the page. The service's 6 s no-capture
hold is the race window - run it BEFORE the client and keep it polling.

**3. NEVER SIGKILL AN ATTACHED GDB.** On r689, killing the stale t91.py
gdb (kill -9 on PID 5334) killed its tracees (client threads
2883/2884/2885/3048); one held the guest RM API rwsem write lock; the lock
leaked and every later RM call blocked (`[vkps] Update blocked for more
than 1228 seconds` in rmapiLockAcquire -> RmGssLegacyRpcCmd ->
_nv04ControlWithSecInfo). The t91 harness design is correct: `timeout 500`
SIGTERMs the gdb and it detaches cleanly. If a gdb must go, use SIGTERM
and wait; SIGKILL only when the guest is about to be cycled anyway. A
leaked kernel rwsem cannot be repaired live.

**4. The r690 gap: the GR channel ring mapping.** Fresh cycle r690 (mdev
f5a796b0): every RM RPC of the client init is answered (GR ctx commit,
channel allocs class 0x9067 hObject 0xcef00000/0xcef00080, EVPOSTs), but
the service reports `gpfifo done=0 nomap=938 pushes=0` - it cannot read
any GPFIFO ring, so no entry decodes, no pushall line appears, and the
guest's completion poll loops on RM control 0x20800a70 until the ~10 s RM
timeout -> vkCreateDevice returns -4 (VK_ERROR_DEVICE_LOST). r689's
service had done=19108 (rings mapped) - so the ring mapping is the
missing ingredient for a fresh client in r690; candidate paths: the FIFO
alloc params (12 bytes, GPFIFO VA inside) are not logged; the scanner saw
the CE window (ipins 4592) but no GR ring this boot. Next: log the 0x9067
alloc params (or scan the guest for the ring VA) and inject the ring page.

**5. gr_walk.py / gr_walk2.py** are the guest-side kcore walkers (fresh-pass
memdesc discovery; 12 records found, kptrs readable in-pass) kept as the
alternate grounding path.

## Session 89 addendum - THE PT WALK WORKS (live); the completion-word gap

**1. GMMU PT walk resolved LIVE (the mission's PT-binding step: DONE).**
The service's own walker math (svc_gmmu_walk: root_bits=ilog2(entries),
shift=49-root_bits; L1 shift38, L2 shift29, PD0 idx=(va>>21)&0xff with 16-byte
dual big/small entries, PT idx=(va>>12)&0x1ff; PDE bit0=0, PTE bit0=1;
pte_addr=(e>>8)<<12) was replicated host-side over /dev/shm/vgpu-guest-ram
(walk.py / live_walk.py). The LIVE published pdb (dmesg: "page directory
... pdb=0x... entries=4" - a NEW pdb per client, e.g. 0x105f0a000 /
0x12a359000 / 0x107012000) resolves every client VA while the client runs:
ring 0x44a0000 -> PA, windows 0x4480000/0x4485000/0x4ca0000..., releases
0x4213000/0x45a0bb0, and the injection via sema_map_add (ap=1 sysmem) makes
the service read them. VALIDATED BY CONTENT: the ring page held exactly the
GPFIFO entries the service logged (entry[1] raw=8000040004480598 va=0x4480598
len=4), and the push page at va=0x4480598 held the method stream
(...0x20020047...). Post-mortem reads fail: the guest frees/zeros the
structures and pdb pages when the client dies - ALL resolution must run
LIVE during the client's ~11 s life (live_walk.py polls dmesg every 0.4 s,
parses pdb + VAs, walks, injects; static family list pre-seeds the
deterministic VAs).

**2. Progress with the walk+inject live: gpfifo done=0 -> 6-9.**
With the live injections the service decoded and CONSUMED the whole GR ring
(entries raw logged, releases decoded: rel=0x4213000 pay=1, rel=0x45a0bb0
pay=3, have_rel=1) and submissions reached 6. The client STILL ends
vkCreateDevice -4 (~11 s, RM 10 s poll of control 0x20800a70) because the
releases are decoded but NOT APPLIED and the client's completion counter
page is unresolvable.

**3. Why releases do not apply (two gates found).**
(a) svc_fwd_defer_arm(): with gsp_phys_gr_forward=1 a GR ring candidate that
is valid but not "offered" keeps defer armed - every GR release goes to the
forwarded-hardware path (which never runs) instead of the inline software
apply. Cycling with gsp_phys_gr_forward=0 removes the arm, but the release
still does not apply - the decode's apply path is additionally gated by
svc.op_tr_pending (a global "operand targeted scan in progress" state) via
svc_opdef_release: while op_tr_pending=1 every release is recorded into an
opdef slot and withheld, silently when the slot already contains it. The
live symptom: `counter page 0x20400f000 unreadable via
vaspaces/discovered/targeted; slot stays pending` repeated at the client's
poll rate plus repeated `targeted resolve va=0x20400f000: full RAM scan
found nothing`.

**4. The counter/notifier page record tables (host-side resolve).**
Searching the RAM for the counter-page VA as u64 finds 16-byte {pa,va}
record arrays, e.g. {0x5FEDC000,0x20400D800},{0x5FEDD000,0x20400E000},
{0x5FEDE000,0x20400E800},{0x5FEDF000,0x20400F000},... (VA step 0x800,
PA step 0x1000) and a second variant ({0x5FEEE000,0x20400E000},
{0x5FEEF000,0x20400F000}). These are how the counter page (and probably
the notifier, notif=0x1xxxxxxx0 per registration) can be resolved WITHOUT
the VA-space walk: search the VA u64, parse the neighbouring u64 as PA,
inject. Must be done live (records vanish post-mortem). NOTE: the service's
own targeted scan reports "found nothing" for this VA - the record layout
the host parses is not what that scan looks for.

**5. Next steps (ranked) for session 90.**
1. Extend live_walk.py with a record-resolve fallback: for any VA the
   service logs as unreadable/pending/stalled (regex set already widens),
   search the RAM for its u64, parse {pa,va} records (both orders), inject
   candidates; feed it counter-page (0x20400f000 family), notifier
   (notif=... per registration) and release VAs BEFORE the decode (static
   list already covers the deterministic ones: add 0x20400f000 with the
   record-resolve because the pdb walk cannot reach the UVM/counter space).
2. Or fix the two release gates kernel-side: (a) don't arm the GR deferral
   when the forwarding engine has no proven channel (offered flag); (b)
   treat op_tr_pending as per-request state (clear when no scan is actually
   in progress for the entry) so a stuck global does not withhold every
   release. Both are small, testable changes (gsp_sema_service.c).
3. Watch for the first live sign: `rel-apply ... written` / `pub-adv` /
   `counter auto/derived` lines on the client's channel; then VKDEVICE
   should pass and the flow proceeds to vkAllocateMemory.

**6. Counter-page record resolve PROVEN LIVE (session 89 end).**
`echo 0x20400f000,0x5FEDF000,0x1000,1 > sema_map_add` (the {pa,va} record
found by the u64 search) made the service learn the slot:
`sema-chan: counter auto h=0xbeef0100 rank=0 pa=0x5fedfff0 (slot
va=0x20400fff0)` and the completion publish went from ZERO to 172 calls
(`counter pub_calls=72 -> 172` over the client's life; previously 0 on every
boot of this session). The client still ends vkCreateDevice -4, so the
remaining unmet wait is the GR ctx-init release word (rel=0x4213000 pay=1,
rel=0x45a0bb0 pay=3 - decoded, have_rel=1, never applied: no rel-apply /
pub-adv line for either). With the counter now advancing, the next
instrumented step is naming the exact GR release-handling branch that
swallows the apply (rebuild with a bounded print at each branch of
svc_rel_apply_906f / svc_opdef_release / svc_rel_apply_launch), then fix
that gate. The gr_forward defer-arm gate is already understood (see 3a) and
gr_forward=0 is the correct test configuration; the remaining suspect is
the service-global svc.op_tr_pending path in svc_opdef_release (it records
the release into an opdef slot and returns true silently when the slot
already contains it).

**7. Session 90: the CURRENT-boot wait loop site (the old recipe no longer matches).**
The client's long spin on this driver/boot sits INSIDE vkCreateInstance (bt:
libEGL -> eglcore -> 0xB2852D helper), a bounded yield loop at
eglcore+0xB2852D: `mov (%r11),%r12d / test $1,%r12b / jne <exit> /
mov 0x4(%r11),%eax / cmp $1,%eax / jne <retry>` - the loop waits for an
IN-PROCESS word pair at r11 (e.g. 0x7ffc2a1c5080, stack/heap; guest-written
only, the host RAM file cannot feed it because it is not a GPU buffer).
Evidence the old site moved: t91's breakpoint at eglcore+0xa85d24 never
fires this boot (`clears=0`) and a read-watchpoint (t80r.py) on the +0x80
words of the two 16 KB rw-s nvidiactl mappings does not fire either (the
16K event-buffer wait of sessions 77/78 is not the live target now; the
spin stack is +0xB2852D <- +0xA5325A <- +0xA539AD <- +0xA47491 <- +0x9FF515
<- +0xA20D04 <- libEGL, NOT the 0xA85F1A/A8 path). A widened multi-word
nudge (nudge2.sh/guest_words2.sh: all rw-s nvidia mappings <=256K, +0x80)
DID fire on 20+ words but the client still returned -4 - the session-77
caution stands: multi-word writes trip the error path; single-word only,
after identifying the true target.
NEXT: (a) rwatch WRITE on r11+4 to name the writer of the in-process word
(the component that must run; if nothing ever writes it, disassemble the
exit branch 0x...ebc9 vs retry 0x...eb93 to see the pass/fail contract);
(b) the same for the 0xA53xxx caller chain - one of these callers is the
A8/NvOgl GL-ucode wait of sessions 78/79 (its completion comes from the
emulated GR engine path); (c) only then re-test the count-clear + single
word nudge recipes.

**8. Session 91: THE vkCreateInstance WALL BREACHED (the marker-clear recipe).**
The client's retry loop (eglcore+0xB33D30) waits until NO event-slot entry has
the pending marker 0xffff: per entry ptr = *(u64*)(obj+0x1428+i*8),
cnt = *(u32*)(obj+0x22efc), the checked slot = ptr + n*0x10 (n = the call's
esi), marker = u16@+0xe == 0xffff (or u32@+0 when (esi-0x43)<=4) via the
check at +0xB1C750 (OR of all pending markers). The guest's event dispatch
NEVER clears it, so the client burns its 10 s -4 timeout. PROVEN FIX:
clear {u16@slot+0xe, u16@slot+0xc, u32@slot+0} to 0 at every hit of
+0xB33D30 (t93 single-shot; t94 auto-clearer, 4000 hits) -> the client
passes vkCreateInstance AND device enumeration FOR THE FIRST TIME
(`VKINSTANCE ok / VKDEVS 2 / VKPICK`, previously the 10 s -4 fired during
this phase). t94 cleared 5+ distinct module objects (obj=0x5d67063a84a0,
0x5d670661ad80, 0x5d670673eed0, 0x5d67067ebb50, ...) over the client's life.
The client then sat in vkCreateDevice's poll(/dev/nvidia0, nfds=1,
timeout=10 ms) loop at eglcore+0xA12DDF..+0xA12DE4 WITHOUT any -4 for 5+
minutes (vs the old ~11 s death) - the longest the client has ever run.

**9. The remaining gap (single): the event-fd signal.** The device-phase wait
is a poll() on /dev/nvidia0 (fd 27) - nvidia_poll wakes only when the guest
RM signals the client's event state; the guest consumes our POST_EVENT
records but never performs the event signaling (session 78's "residual
delivery gap"). Writing to /proc/PID/fd/<nvidia0> is not a signal. The
session-84 count write (base+0x2070b7c, read 4, wrote 0, t97) did not flip
this wait alone. NEXT: fix the host POST_EVENT -> guest dispatch association
(gsp_rm_emulation.c r579 area: the payload fields hClient/hEvent/notifyIndex/
data/info/status; compare against the guest's _kgspRpcPostEvent ->
gpuNotifySubDeviceEvent expectations) so the guest signals the event and
nvidia_poll returns POLLIN; then the same t94-pattern (or a service-side
equivalent) durably unblocks both waits per boot.

**10. Session 92: THE AUTOMATED PIPELINE - fresh boot to VKDEVICE with ZERO manual steps.**
On a fresh r692 boot (cycle params unchanged, gr_forward=0) a single gdb
script (t99.py) carrying two breakpoint handlers drove the client end-to-end:
(a) the instance-phase marker cleaner at eglcore+0xB33D30 (t94 pattern: clear
u16@slot+0xe / u16@slot+0xc / u32@slot to 0 per entry at ptr+n*0x10 for
entries *(obj+0x1428+i*8), cnt *(obj+0x22efc)) - fired CLEAR#1..#5 across
five module objects; (b) the device-phase wait-word chaser at
eglcore+0xDCEBD9 (t79 chain: rbx -> *(rbx+0x70) -> *(sub+0x58)=obj2 ->
  off=*(u32)(obj2+0x60), holder=*(obj2+0x58), base=*(holder+0x58),
word=base+off) - found WORD addr=0x734781a86080 val=0x0 and wrote 1.
Result: `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok family=0` with no
manual intervention (r692 client 1725). The client then sat in the
POST-DEVICE wait (the next phase - allocation/engine-init), which t99 does
not cover. The live_walk PT-walk injector ran alongside and injected the
window/ring VAs (0x4d20000, 0x4923000...; note 0x4b19b10 resolved ap=FB -
the GL init release is vidmem-mapped and skipped by the SYS-only injector).

**11. HARD LESSON (attach discipline, again).** A gdb attached to a client
that never hits its breakpoints wedges: `continue` never returns, the
timeout's SIGTERM does not tear it down, and the ptrace slot blocks all
later attaches (hit 3x this session: 146084, the t91-on-1725 attach, both
left holding). RULE: only attach with breakpoints that are KNOWN to fire
for the client's current phase, keep windows <= ~60 s, and prefer
self-terminating handlers (hit caps). A wedged attach leaves the client
RUNNING (state R) but unattachable; cycle the VM for a clean environment
when more attaches are needed.

**12. Continuing the chain (next session).** (1) Identify the post-device
wait of the VKDEVICE-ok client (bt sample once attachable; likely the
allocation/UVM phase or the A8/NvOgl engine-init with its count gate at
lib_base+0x2070b7c=4 -> 0 and/or the 0xdceb80 wait loop - add the t91
count-clear BP to the t99 bundle if the A8 path is live). (2) Extend t99
into the "durable harness bundle": marker-clean + word-chase + count-clear
+ (once VKFRAME) the release/alloc phases. (3) The bNotifyList=1
(gsp_rm_ev_nlist) path stays armed for the event-signal experiment;
post/EVPOST logging re-budgeted (gsp_rm_ev_log=2000 set).

**13. Session 93: THE ZERO-PTRACE PIPELINE (r693) - VKDEVICE with no gdb-wedge risk.**
Split the harness by risk: (a) ONE gdb (t101.py) for the INSTANCE-phase
marker clears only (base+0xB33D30 handler, clears u16@+0xe/+0xc and u32@+0
of each entry; self-disables after 24 clears; NO other BPs so it cannot
fire constantly); (b) the WAIT WORDS handled by /tmp/unwait.py - a guest
python poller with ZERO ptrace that scans the newest vk_frame's rw-s
nvidia mappings of exactly 0x4000 bytes and writes 1 to any zero u32 at
mapping+0x80 via /proc/PID/mem (the session-77 geometry, confirmed by t79
chains: word = the +0x80 of a 16 KB rw-s nvidiactl mapping). Result on
fresh r693 (mdev 261980af): t101 CLEAR#1..#5 then idle; unwait wrote FIVE
wait words (0x7876ead57080, 0x7876f1816080, 0x7876ea47f080,
0x7876ea81a080, 0x7876ead51080); client reached `VKINSTANCE ok / VKDEVS 2
/ VKPICK / VKDEVICE ok family=0` with NO wedge exposure. live_walk (PT-walk
injector) ran alongside. The client then sat in the post-device wait that
the +0x80 poller does not cover (t102.py = the one-shot safe diagnostic:
attach -> print PC/base/A8-count(base+0x2070b7c)/bt -> detach, NO continue
and NO breakpoints - use it when the slot is free).

**14. THE RM-LOCK WEDGE ROOT CAUSE + RULE.** A gdb killed (SIGTERM) while
the tracee sits AT a breakpoint leaves the tracee STOPPED; if that stop is
inside an RM api critical section (the rwlock held across an RM call!), the
RM write lock stays held and EVERY other client blocks in
os_acquire_rwlock_write (state D) - observed directly: old client 1725
stopped by its wedged t99 gdb -> new client 7967 hung `state=D wchan=
os_acquire_rwlock_write` on ioctl 0xc020462b; killing the gdb + `kill -CONT`
resumed 1725 out of the ptrace stop but it resumed INSIDE the same RM wait,
so the lock stayed held until the VM was cycled. RULES: never let a gdb
timeout fire while a tracee is at a frequent-firing BP; prefer BPs that
fire once or self-disable; prefer no-ptrace pollers for anything with a
repeating pattern; the safe diagnostic is attach -> read -> detach without
continue.

**15. Session 94: THE INSTANCE CLEARS ARE REQUIRED (zero-gdb disproved); t104 is the safe form.**
Pure zero-gdb run (fresh r694, client + unwait.py ONLY, no gdb): unwait
wrote three +0x80 words and the client still ended `VKFAIL vkCreateDevice
... -4` - WITHOUT the instance-phase marker clears the device gate fails
even with the words. With t104.py (single BP at +0xB33D30; clears
u16@+0xe/+0xc/u32@+0; self-exits on cleared>=8 OR chits>=40 OR 25 s since
the last clear -> returns True so the script tail runs and batch gdb exits
cleanly, no outer-timeout dependence) + unwait + live_walk: CLEAR#1..#5,
two words written, client reached `VKINSTANCE ok / VKDEVS 2 / VKPICK /
VKDEVICE ok family=0` (r694 client 3725). The client then sat in the
post-device wait again; the t102 diagnostic could not attach because the
t104 gdb still held the slot (its 25 s-idle exit never fired - no
post-instance B33D30 hits - and its 150 s timeout likely fired at a stop
= the known wedge risk). NEXT FORM: make t104 detach ITSELF from inside
the handler once the clears are done (`gdb.execute("detach")` from the
stop handler; the tracee resumes; the script ends and gdb exits) so NO
gdb is ever held while the client runs - then t102 always has the slot
and the wedge class disappears end-to-end.

**16. The allocation/post-device phase is the remaining gate.** With
VKDEVICE green the client waits in a site that is neither the +0x80 word
pattern nor (likely) the A8 count path; the next session should (a) get
the detach-first t104 live, (b) run t102 immediately after VKDEVICE to
name the wait (read PC/bt/A8 count = base+0x2070b7c), and (c) inspect the
service for this client (pushes decoded / releases applied / counter
pub_calls) to see whether the allocation needs the live injections only
or a service-side gap.

**17. Session 95: THE GDB THREAD-STOP INTERFERENCE + the SIGKILL nuance.**
The t105 attach (t=1.5 s, before the client's worker threads spawned) left
four worker threads in `ptrace_stop.part.0` (futex) - the client's main
thread span waiting on them, so the client froze at the post-device wait
with the gdb holding the trace. TERM/INT/HUP on the wedged gdb were all
ignored (batch gdb stuck in continue). SIGKILL on the gdb (1584) was tried
with the reasoning that all stopped threads sat at futex (NO RM lock held):
result - the tracer died, the tracees DETACHED+RESUMED (client 1518
SURVIVED with its workers back in futex_wait_queue; contrast session 88
where SIGKILLed-tracer tracees were KILLED mid-syscall). So the SIGKILL
outcome depends on the tracee's stop context; the mission's never-SIGKILL
rule stands, but this datapoint matters when judging recoverability.
After the gdb died, 1518 still held the RM write lock from its own
post-device wait (its wait word was no longer polled because unwait.py
tracks the NEWEST vk_frame and a fresh test client 6555 was launched and
blocked D-state on os_acquire_rwlock_write) - the unwait-newest coupling
must be replaced by ALL-clients polling in the next form.

**18. HARNESS DESIGN RULES (accumulated).** (1) A gdb attached across
thread-spawn windows stops worker threads and freezes the client - attach
only AFTER all threads exist, or ensure the gdb explicitly resumes new
threads, or do the work with NO gdb. (2) Any gdb that must run through
thread-spawn windows MUST self-detach quickly (the instance phase spans
~10-20 s and the clears arrive in ~5 bursts) and NEVER rely on outer
timeouts. (3) unwait.py should poll EVERY live vk_frame (not just the
newest). (4) The durable answer for all of this is the service-side
folding: the instance markers and the +0x80 words are guest-RAM writes a
kernel-side service can perform on the same triggers (the marker semantics
and the word semantics are understood) - that removes gdb and /proc
helpers from the loop entirely.

**19. Session 96: THE DETACH-FIRST CLEARER PROVEN (t106) + the orphan-tracer plague.**
t106.py (single +0xB33D30 BP; clears markers; detaches FROM THE HANDLER as
soon as cleared>=5 or chits>=14: `gdb.execute("detach")` + return True so
the batch gdb exits) on fresh r696 (mdev 241af1c1): CLEAR#1..#5,
`BC-DETACHED cleaned=5 chits=5`, `T106-DONE chits=5 cleared=5`,
`TracerPid: 0` - THE SLOT IS LEFT FREE. unwait.py wrote five words and the
client (1549) reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok
family=0`. THE CLEAN FORM OF THE CLEARER EXISTS NOW.

**20. THE ORPHAN-TRACER PLAGUE (root cause of every t102 failure).** Every
INTERRUPTED gdb attempt leaves an attached orphan: (a) piping gdb to
`head` closes the pipe early and the gdb stays attached (hit twice); (b) an
ssh command dying/timing out mid-gdb-add leaves the gdb running (hit
again). Symptoms: later attaches fail with `Could not attach ... already
traced by process N` / `ptrace: Inappropriate ioctl for device` and the
t102 log stays empty. RECOVERY (session-95 evidence): TERM is usually
ignored (wedged in continue); SIGKILL the orphan + `kill -CONT <client>`;
the tracee SURVIVES when it was running/at-futex and detaches cleanly
(TracerPid 0). STRICT DISCIPLINE: never pipe gdb output (redirect to a
file, read it after); never let an ssh command end while a gdb may be
attached (generous timeouts); after every gdb run verify TracerPid 0.

**21. THE NEXT FORM (t107, one gdb, one window).** Combine the proven
clearer and the diagnostic in ONE batch gdb: attach -> clears -> detach ->
`python time.sleep(...)` idle -> RE-ATTACH (gdb can attach again after
detach) -> read PC / bt / A8 count (t102 content) -> detach -> print -
one ssh session, one window, everything self-contained, no orphan risk
(every path ends in an explicit detach; wrap each phase in try/except with
a final detach). Expected: names the post-device wait (allocation/UVM vs
A8 count base+0x2070b7c) so the allocation can be passed and the first
VKFRAME reached.

**22. Session 97: THE DEEP SUBMISSION STATE + THE COUNTER-SLOT WEB.**
The r696 client (1549) reached the deepest state yet: the service reported
`gpfifo done=11576 pushes=10932 counter pub_calls=1356` and climbing
(gpfifo done was 0 in the pre-pipeline boots) - the client submits the full
work stream and the service decodes/processes it (pairs=14395 map=14318,
ipins=9240 from live_walk). The client then sits in the KNOWN userspace
post-device spin (main thread `running` in all samples; one tid in
os_acquire_rwlock_read). NEW: a counter-slot collision spam
(`counter auto ... pa=0x5fedfff0 rejected (slot owned by another channel;
not installed)`, 232+ callbacks suppressed) - the 0x20400f000 injection
resolved the WHOLE counter VA family to ONE page, so both live channels
collided; the manual bypass `echo <h>,<pa> > .../parameters/sema_counter`
(h_object,pa syntax, main.c:2965) was applied for both handles
(0xbeee0100,0xbeec0100 -> 0x5fedfff0) and pub_calls jumped 1356 -> 3739.
The client gate (VKDEVICE -> frames) persists: its remaining wait is the
userspace spin word NOT covered by the +0x80/16K poller (candidate: the
tracker/counter the service now advances - watch whether pub traffic
alone resolves the spin, else t107-GDB the spin site). ENVIRONMENT: one
orphan gdb (10092 on 1549) + stale channels (beec0100) from prior
attempts; the orphan-kill discipline applies (SIGKILL by PID + CONT +
verify TracerPid).

**23. THE IOCTL-ARG SAMPLING AXIS (no ptrace).** /proc/<tid>/syscall shows
the running ioctl's ARGS (fd, cmd, arg) when caught - sampling the main
thread repeatedly during an ioctl-loop phase names the exact stuck RM
control without any gdb. Use it before ptrace-based diagnostics when the
thread does syscalls at a catchable rate.

**24. Session 98: THE POST-DEVICE SPIN NAMED - THE SEC2 SCRUBBER AUTH-TAG WAIT.**
perf (no ptrace!) on the spun client: top symbol `channelReadChannelMemdesc`
24.33% (then scheduler internals) - the guest's nvidia driver hot-loops
reading the CHANNEL MEMDESC. Source decode (nvidia-570.144-reference,
sec2_utils.c): the only reference caller is
`_sec2utilsUpdateGetPtr` -> `channelReadChannelMemdesc(pChannel,
authTagBufSemaOffset)` inside `_sec2utilsGetNextAuthTagSlot`, whose loop
waits for the scrub auth-tag slots to free; `authTagBufSemaOffset =
finishPayloadOffset + CHANNEL_ENGINE_SEMAPHORE_SIZE` (4). So the spin =
the SEC2 MEMORY-SCRUBBER channel waiting for its auth-tag semaphore (a u32
inside the channel memdesc buffer) to advance - the GPU never delivers it
in our emulation, so the client waits forever (each poll = one full memdesc
transfer = the 24% cost). THE FIX DIRECTION: the service must complete the
scrubber's semaphore updates - the scrubber channel registers in the
service (`scrubber channel 0x... detected (gpfifo=... userd=...)`) and its
submits carry semaphore releases targeting the auth-tag VA
(pbGpuVA + authTagBufSemaOffset) in the channel buffer; on decode, write
the release value to that VA's page (the channel buffer is a guest-RAM
allocation the service can resolve/pin; the pending queue holding 17
entries is the suspect pool for the unresolvable ones - inspect
svc_pending[]). Service-side counter to watch after the fix: the scrub
loop exits, the client proceeds to the allocation/pipeline. Tool: perf
record/report on the guest is the no-ptrace spin-pointer (`perf record -p
<pid> -- sleep 3; perf report --stdio --sort=symbol,dso`).

**25. Session 99: THE SCRUBBER GATE CONNECTED TO THE CE CHANNELS' ok=0.**
chan_dump on r696 (client 1549 spun): six channels -
`h=0xbeee0100 eng=0 gpfifo=0x44a0000 userd=0xfee00 ok=1 get=6 sub=6354 rel=0x4213020 crel=0x5fedfff0 notif=0x11cd51060`
`h=0xbeec0100 eng=0 gpfifo=0x4cc0000 userd=0xfee00 ok=1 rel=0x4923020 crel=0x5fedfff0`
`h=0xbeec0104 eng=0 gpfifo=0x4d20000 userd=0xfee00 ok=1 rel=0x4b17e30 crel=0x5fedffe0`
and the CE-class channels `beee0101 (eng=11, gpfifo=0x4500000 userd=0x8f600 as=2 ok=0)`,
`beee0102 (eng=12, gpfifo=0x4560000 userd=0x8f400 ok=0)`, `beed0101 (eng=12,
gpfifo=0x49c0000 userd=0x8f200 ok=0)` - ALL ok=0: the CE userd discovery
(failed probe: EAGAIN 0/0 or EINVAL) means no CE pump, no CE sema applies,
so the SEC2 scrubber channel (a CE-class channel) never sees its auth-tag
semaphore advance and the client spins in _sec2utilsGetNextAuthTagSlot.
The service's candidates (registered PA, ring+0x10000 in both apertures,
8-slot list) all miss for the CE channels. `find_userd` (FB scan for a
live put@0x8c/get@0x88 pair) found ZERO candidates - the CE userds are not
in FB with live puts (either the client's PUT writes go to a page the
service's candidates miss, or the CE userds genuinely idle). A raw host-RAM
shape scan (put in 1..65535, get<=put) is too noisy (24 hits in the first
28KB of BIOS-era low RAM) - needs tighter validation. NEXT APPROACH: find
the REAL userd page by DIFF-WATCH - sample the candidate pages (and the
CE ring pages) twice around a scrub submit and diff for the page whose
+0x88/+0x8c pair changes; the guest kernel writes GP_PUT there, so the
changing page is the true userd. Then fix the discovery (or add a
sema_userd-style injection param) so the CE pump engages and the scrubber
completes. NOTE: the GR channels' userd=0xfee00 (as=2, FB) and ok=1 with
get=6/sub=6354 - the GR machinery runs fine; only the CE/scrubber path is
dark.

**26. Session 100: THE CE USERD HUNT EXHAUSTED THE STANDARD LAYOUTS; THE CIRCULAR WAIT.**
Candidates examined for the CE userds this session: (a) the registered FB
offsets (0x8f600/0x8f400/0x8f200 as=2) - ALL ZERO in the FB shadow
(sema_vidmem_dump 0x8f100,0x600 twice); (b) the ring-PA + 0x10000 linear
guess - random/zero/table garbage (the allocations are NON-CONTIGUOUS);
(c) the PDB walks of gpfifo+0x10000 (0x4501000->pa 0x125e91000,
0x4561000->pa 0x132032000, 0x49c1000->pa 0x11bddd000 via pdb 0x1152f4000)
- ALL THREE PAGES ALL ZERO. The CE userds were never written - and the
deeper reading: the SEC2 scrub flow is a CIRCULAR WAIT: the scrubber waits
for its previous scrub pushes to be CONSUMED (the auth-tag sema advances
only when the GPU runs the scrub) -> the consumption needs the CE pump ->
the CE pump needs a live userd -> the userd is written BY SUBMITS -> the
submits are held by the wait. In a working boot the guest's first scrub
push happens BEFORE the wait loop (the loop only waits when the slots are
full) - so in our boots the FIRST scrub push must have gone into the CE
ring, the service failed to consume it (ok=0/userd dark), and the client
now waits for a completion that will never come.

**27. THE DIRECT BYPASS (next session, ranked).** (1) Write the auth-tag
semaphore in the CHANNEL BUFFER (memdesc) page directly: the scrub loop's
`_sec2utilsUpdateGetPtr` reads `channelReadChannelMemdesc(pChannel,
authTagBufSemaOffset)` = a u32 at memdesc+finishPayloadOffset+4; advancing
that value frees the slots and unblocks the client WITHOUT the CE pump.
The memdesc = the channel buffer allocation (a VA in the client's space);
resolve it via the PDB (the guest's channel struct gives the buffer VA;
the service's channel registration/alloc params carry it) and write the
incremented value from the service (or the host RAM file). (2) OR decode
the scrub pushes the service already sees (check sema-push for the SCRUBBER
channel - its submits carry the auth-tag VA; the release decode should
target it; the service's release applies would write the sema if the VA
resolves). (3) The registered FB userd offsets being zero suggests the
client's kernel maps the userd through a VA whose PA the discovery misses;
the diff-watch (two samples around activity) remains the fallback proof.

**28. Session 101: THE LOOP SEMANTICS + perf trace (NO-PTRACE SYSCALL TRACING).**
The wait loop decoded exactly (sec2_utils.c:420-441):
`while (nextPut == pSec2Utils->authTagGetIndex) { gpuCheckTimeout(); 
authTagGetIndex = _sec2utilsUpdateGetPtr(); osSpinLoop(); }` - it exits
when the memdesc sema (u32 at pChannelBufferMemdesc +
authTagBufSemaOffset; authTagBufSemaOffset = finishPayloadOffset + 4)
advances past nextPut; it also has a GPU timeout that would error out.
perf trace on the spun client (no ptrace!) revealed the live pattern:
tid 1793 = a 10ms-timed futex loop (WAKE val=1 then
WAIT_BITSET|CLOCK_REALTIME 10ms - the paced retry); tid 1788 = repeated
`openat -> read(fd, 299 bytes) -> close -> futex-wait` - a transient-fd
POLLING READER of a small blob (the fd is opened and closed each cycle, so
fd 28 no longer exists; the client's persistent fds: /dev/nvidia-modeset
(10), /dev/nvidia0 (11/12/14/15), GLCache .toc/.bin (16/17), memfd
\"allocation fd\" (18)). The 299-byte read is the likely userland face of
the memdesc read path (the 24% channelReadChannelMemdesc hot loop).
NEXT: (a) catch the transient fd path by sampling `ls -l /proc/<pid>/fd`
every ~0.2 s in a loop around the burst (or perf trace -s during a burst);
(b) if it is a driver read() of the memdesc, resolve its buffer and write
the auth-tag sema value (advance past nextPut) via the RAM file; (c) the
loop timeout (gpuCheckTimeout) may eventually error the client - the write
must land first.

**29. Session 102: fd-sampler miss, no nvidia debugfs; the memdesc hunt narrows.**
The 0.15 s /proc/1549/fd sampler over 20 s caught NO transient path (the
open/read(299)/close reader is bursty or dormant between bursts; the static
fds are /dev/nvidiactl, /dev/nvidia0, /dev/nvidia-modeset, a socket, the
GLCache .toc/.bin pair, and a memfd "allocation fd"). No nvidia debugfs
tree exists in the guest. Remaining ranked paths to the auth-tag sema
(memdesc + finishPayloadOffset + 4): (1) content-locate the scrubber's
channel buffer by its method stream (the C5B5/C56F shapes: SET_OBJECT
0xc5c0/0xc5b5, headers 0x200121ed/0x200120a9 observed in the session-78
window dumps) and read/write the sema inside; (2) the scrubber channel's
own sema-push decodes (its pushes carry the auth-tag VA = pbGpuVA +
authTagBufSemaOffset; grep the boot's sema-pushall for the scrubber
channel id and read the rel= VA - then PDB-walk it); (3) sample the fd
burst narrower (0.02 s) right after a client restart when the burst is
hot. The service-side ideal: on detecting the scrubber channel, resolve
its channel buffer and pre-advance the auth-tag sema (a durable fold).

**30. Session 103: THE SCRUB STREAM FOUND; ONE RELEASE DELIVERED; THE SEMA PAGE STILL AMBIGUOUS.**
Content-located the CE/scrub init method stream (headers 0x200121ed/
0x200120a9 searched): full stream at `pa=0x114e98000` = SET_OBJECT
0xC5C0/0xC5B5 (0x20012000/0x20012080 headers), engine-init methods
0x200121ed(3)/0x200120a9(1)/0x2001209c(0x0c000001), bulk methods
(0x8000xxxx), then TWO REPORT_SEM releases in one buffer:
`0x200426c0 00000000 04b19b10 ...` (va=0x4b19b10) and
`0x200426c0 00000000 04b17e30 ...` (va=0x4b17e30), tail 0x2001c00d
70/68. Twins at pa=0x1026ad000+0x87c and pa=0x12afba000+0x87c (identical
content at identical offsets - multi-mapped copies). Release resolution:
`0x4b17e30 -> pa=0x114efbe30 (SYS) cur=1` - THAT release WAS delivered (the
service applied it, cur=1). `0x4b19b10` resolves via the walker's big-half
2MB branch to FB 0x660b10 holding garbage (0xD9F26F94) - poke_vidmem
tests ([0x660b10]=1 then =0x40000000) did NOT flip the client, and the
linear-block guesses (buffer VA 0x4b00000 <-> stream pa 0x114e98000, sema
candidate 0x114eb1b10) read all zero. NEXT: the auth-tag sema page must be
picked by VALIDATION (the existing value must be a small plausible
GetPtr): walk 0x4b19000 returning ALL chains (small/64K/2MB both
apertures), pick the page whose u32 at +0x1b10 (or the offset the memdesc
actually uses) is a small number; or take the buffer aperture from the
channel's alloc params (the service's registration) and walk with that
aperture exclusively. The fix remains: advance that u32 (write via the
host RAM file if SYS, poke_vidmem if FB).

**31. Session 104: THE SEMA VA 0x4b19b10 IS NOT THE LOOP'S PAGE; THE KPROBE IS THE ANSWER.**
The validated multi-chain walk (small-4KB and big chains BOTH) resolves
VA 0x4b19b10 -> FB 0x660b10 (both chains converge; the guest's own PTs say
the object lives in vidmem). poke_vidmem writes at 0x660b10 (0x1,
0x40000000) and 0x660b14 (0x1, 0x40000000) LANDED (verified via
sema_vidmem_dump: [0x660b10]=0x11112222) yet the client does not flip and
perf still shows channelReadChannelMemdesc at 24.52%. CONCLUSION: the
REPORT_SEM VA 0x4b19b10 (and 0x4b17e30) are NOT the auth-tag sema the loop
reads - they are the GL-init release objects; the real auth-tag sema is at
pChannelBufferMemdesc + finishPayloadOffset + 4 (a VA and page we do not
yet know; finishPayloadOffset is NOT 0x19b10). THE DEFINITIVE INSTRUMENT:
a guest kprobe on channelReadChannelMemdesc (the offset argument = %si =
the authTagBufSemaOffset VALUE; and the pChannel pointer lets us read the
channel struct -> pChannelBufferMemdesc): `echo 'p:crcd
channelReadChannelMemdesc offset=%si' >> /sys/kernel/tracing/kprobe_events`
+ enable the event + read tracing pipe WHILE the loop runs. This names the
offset and the memdesc in one shot, then the sema page = walk(memdesc+off)
and the advance (RAM file if SYS; FB + poke if the converged mapping).

**32. Session 105: KPROBE BLOCKED (notrace); THE BAR1-PATH ANALYSIS; THE CHANNEL-FIELD OFFSETS.**
(1) The kprobe on channelReadChannelMemdesc is IMPOSSIBLE: the guest's
dmesg says `trace_kprobe: Could not probe notrace function
channelReadChannelMemdesc [nvidia]` - the function is annotated notrace
(ftrace/perf-probe/BPF-kprobe all blocked). The tracefs filesystem had to
be mounted first (mount -t tracefs tracefs /sys/kernel/tracing) and it
already carries UVM kprobes from earlier sessions (uvmt/uvch/uvbp/uvtw/
uvsv...). The static sec2utils functions (the callers) are absent from
kallsyms (inlined) - no probe target on that path.
(2) BAR1-path analysis: the guest driver's memdesc read uses
TRANSFER_FLAGS_USE_BAR1|SHADOW_ALLOC|SHADOW_INIT_MEM - a CPU read of the
GUEST'S BAR1 mapping, which the mdev mmap maps onto the HOST's vidmem
shadow (the same buffer poke_vidmem writes). The pokes at FB 0x660b10/
0x660b14 LANDED (verified) yet the loop did not advance, so the memdesc
page is NOT the FB 0x660b10 the PT walk converged on (VA 0x4b19b10 belongs
to the GL-init release object, not the channel-buffer memdesc).
(3) The channel registration parse exposes the guest-visible channel
fields and their alloc-param offsets (gsp_rm_emulation.c
gsp_rm_register_channel: GSPRM_CHAN_GPFIFO_OFFSET/ENTRIES, ENGINE_TYPE,
CID, INSTANCE_MEM, USERD_MEM(+APERTURE), NOTIFIER_MEM(+APERTURE)) - the
channel-buffer/memdesc field (pChannelBufferMemdesc) is not among them;
the next session should dump a CE channel's FULL alloc params (the
service's rate-limited alloc dumps or a fresh boot) and look for the
channel-buffer VA/PA field, then sema = buffer + finishPayloadOffset + 4
(per-channel offset, e.g. the 0x19b10-style for the GL channel).
(4) Fallback if the field is not in the params: per-VA content hunt for
the auth-tag sema VALUE (the retprobe would have given it; without
kprobes: the sema value = a small slot counter - scan the CE channel's
allocation candidates for a plausible counter that CHANGES across two
samples).

**33. Session 106: THE CHANNEL BUFFER IS NOT A CLIENT PARAM; THE TAIL SWEEP AND HASH SCAN NEGATIVE.**
NV_CHANNEL_ALLOC_PARAMS (alloc_channel.h:300-341) layout mapped to the
service offsets: gpFifoOffset@8/entries@16, engineType@128, cid@132,
instanceMem@144, userdMem@168, ramfcMem@192, mthdbufMem@216,
hPhysChannelGroup/internalFlags, errorNotifierMem@248... - THE CHANNEL
BUFFER (pChannelBufferMemdesc) IS NOT IN THE PARAMS: it is a
driver-internal memdesc created by channelAlloc, so the alloc-params path
cannot name it (the kprobe/debuginfo path was the right one; it is
notrace-blocked). The finish-payload tail of the stream page
(0x114e98000 +0xd4..+0x118) was swept with 0x40000000 (18 u32s; the
pre-sweep values: +0xd8=0x68000000 finish marker, +0xe0=0x901025a6,
+0xe4..+0xec=0x8000xxxx methods) - no flip. A 16-page high-entropy scan of
0x114e98000..0x114ea8000 for the auth-tag SHA-256 array: ZERO high-entropy
32-byte blocks (the hash array is elsewhere / another allocation). NEXT:
(1) the memmgrMemDescBeginTransfer SHADOW insight - the read happens from
a FRESH kmalloc'd shadow copied from the memdesc each call; the shadow's
source = the memdesc's backing page; the memdesc page must be found by
TRANSFER-TIME properties (not the VA walk); (2) or a guest perf hardware
watchpoint on the driver's BAR1 ioremap VA + the (unknown) memdesc offset;
(3) or the service-side broad assist: increment a RANGE of plausible FB
sema cells periodically (blind but bounded), watching for the flip.

**34. Session 107: FB ASSIST SWEEPS NEGATIVE - THE READ IS NOT FB-SOURCED; THE TWINS ARE THE NEXT TARGET.**
Two bounded FB "sema assist" sweeps (distinct values per cell,
0x40000000|idx): (a) the walked page 0x660b00..0x661000 (320 cells) - no
flip; (b) the CE-channel allocation cluster 0x8f000..0x90000 (1024 cells,
where the registered userds live) - no flip; perf re-confirmed
channelReadChannelMemdesc 23.13% (the loop still runs). CONCLUSION: the
memdesc read does NOT come from the FB shadow the CPU can poke - the
transfer's SHADOW_INIT must read the memdesc through a path that bypasses
these bytes, or the memdesc backing is SYSTEM memory via an unknown VA.
NEXT (one-minute test): the STREAM TWINS - pa 0x1026ad000 and
0x12afba000 hold BYTE-IDENTICAL copies of the CE method stream (verified at
+0x7c0/+0x880: same 0x68000000 finish marker area, same 0x200120a9/9c
methods) - one of them is very likely the memdesc BACKING in sysmem (the
driver copies the same stream per channel/boot). Sweep BOTH twins'
stream-area pages (reuse the assist code; 1 page each) and watch for the
flip; if it flips, bisect the winning page to the sema cell and make it
durable (host RAM file writes = zero-cost, no poke_vidmem needed for
sysmem). Failing that: content-diff the twins across a driver transition
(the memdesc copy updates during submits; the copy that CHANGES is the
backing).