summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag/wall2/README-session77.md
blob: 22247d6909f2358b661427265843c2fc003892b8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion

## The two-part unblock

1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
   nvkms/Vulkan path allocates its completion event as
   NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
   registers it alongside 0x79. Live proof (r683, sema_dbg=1):

       GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
           hParent=0xcaf00001 hObject=0xcaf00003
       event registered hEvent=0xcaf00003 hSrc=0x0
           notifyIndex=1006632972 (off 8, slot 13)
           params=1100e0c1000000007e0000000c00003c0000000000000000
       EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0

   (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
   NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
   (`guest_read == our_write` in the rmstate dump), but the guest's
   event dispatch does NOT write the event's notification word — the
   residual delivery gap.

2. **The completion word found and driven (the actual unblock)**. The
   main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
   a u32 readiness word: `should-wait = (word != 1)`; the word lives at
   `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
   notification buffer; holder chain sub -> obj2 -> holder, resolved
   live by t79.py). The host can write it directly in the guest RAM
   file:

       PA = pagemap PFN << 12  (+0x80);  file offset = PA - 0x80000000
       (above-4G RAM; below-4G would be PA itself per the cycle's
       guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)

   Writing 1 flipped the wait and **vkCreateDevice returned success**
   (`VKDEVICE ok family=0`) — reproduced three times:
   * r683: manual write after a t79 probe;
   * r684: nudger (all 16 KB mapping words) + slotfill restart;
   * r684d: probe -> single-word write (the reliable recipe).

## The reliable recipe (per run)

1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
   slots re-arm, so re-run slotfill if the client is stuck in the
   sched_yield spin before the word-wait).
3. Wait for the main thread in `do_poll` (the word-wait).
4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
   host write `01 00 00 00` at `PA-0x80000000`.
5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
   image/memory/pipeline setup; its next wait is NOT the same word
   (the word stays 1) — next session: probe the follow-on state
   (attach fails while the client spins in the post-device phase;
   use the /proc-only checks + a longer observation window).

## Cautions learned

* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
  -4 give-up path (an RM error flag word gets written): restrict nudges
  to the single probed word.
* Attaching gdb to the client while it is in the tight post-device spin
  can hang the attach (leave a `timeout`-wrapped gdb, then kill that
  gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
  dead tracer otherwise wedges it).
* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
  python, not zsh.

## Tools

* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
  and prints the client's nvidiactl rw-s mapping sizes.
* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
  nvidiactl mapping (pagemap; run in the guest as root).
* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
  zero word (multi-page variant; prefer the single-word recipe).

## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6)

* The post-device spin is the **GL module bring-up**: the main thread's
  A8-layer retry loop (eglcore 0xa85d24) works on a work item whose
  embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6
  submit layer (0xd624e0) ORs **0x200000** into its pending word (the
  same engine-pending bit SESSION 75/76 watched on E+0x150), then the
  A8 loop yields via the B284B0 spin helper (sched_yield) waiting for
  the GL ucode init to complete on the GR engine.
* **The push windows ARE now readable** (first time): the resolver6
  recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} ->
  same-page 0x20000 memdesc PA -> `sema_map_add`) injected
  `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d
  68000000` = the 2-method semaphore-like push) and `0x4ca0000 ->
  0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5,
  `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d
  70/68`). The service now decodes the streams (`sema-push ... tail=0dc0
  012000000070...`) where SESSION 73-74 saw "unread".
  CAUTION: the resolver injects every candidate; a zero-page candidate
  injected last replaces the good mapping for that VA (re-inject the
  good page after the scan).
* **Remaining gate**: the service still reports `have_rel=0` for every
  GR push (the CE-shaped release matcher) and — decisively — the
  GR-forward machinery is NOT engaging: `drain held=4 starved=149
  forwarded=0 ... gr_nocap=4`. The GL ucode init work is never
  forwarded/executed, so the completion (and the 0x200000 bit clear)
  never arrives. Next: the GR forward/capture path for these channels
  (operand capture + release decode for the GR shape), then the client
  proceeds to VKFRAME.

## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe

* The GR/GL init releases decode only with the low-aperture acceptance
  (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms,
  commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group
  `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0
  pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ...
  02000001`); `have_rel=1`, `applied` rising, un-translated releases
  queue in svc_pending (417 observed).
* Corrected pair-specific push injections for r686:
  `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc),
  `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus
  `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window).
  CAUTION: resolver7 injects each page's descs for BOTH pairs (union
  loop) - fix it to keep per-pair pages or inject manually from the
  per-pair scan. The window-record layout is {tag, VA, SIZE, 0,
  "tail", ...} stride 0x40 (search the VA u64 with size at +8 and
  "tail" at +0x10).
* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no
  window records in the known record pages; the service cannot resolve
  them (targeted rm_mem miss) so the releases stay pending and the
  NvOgl wait spins. Find their PAs (client mapping pages / RM
  VA-tracker / live poll-watch), inject, and the pending releases
  should apply.

## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying

* **The record scan fix**: the RM window record is 0x40 bytes
  {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is
  stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT
  "tail" (the hex display `7461696c` reads as "tail" only as a number).
  Searching for b'tail' finds nothing; `wall2_recscan.py` searches for
  the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at
  -0x20/-0x18/-0x10. It found **all 75 records** in one pass.
* **The GR init semaphore windows** (r686):
  `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content:
  +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 /
  rel=0x4b17e30 pay=1 payloads) and
  `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the
  release applied there: +0xb10 became 2). Injections:
  `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and
  `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`.
* **State after the injections**: the low-VA releases decode
  (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at
  ~418 (the client re-kicks continuously); the NvOgl wait still spins
  (the A8 loop compares per-entry counter tables r14+0x5e8 vs the
  wanted set - the completion is not just the semaphore value). Next:
  the A8 counter tables' source (what fills r14+0x5e8 / the "wanted"
  table - likely the GR channel's own progress counters) or the GR
  forward/ucode execution (`forwarded=0 gr_nocap=7` still).

## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass

* `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a
  translation for the rank slot VA `0x20400fec0 + (19-r)*0x10`
  (r = gr_index = rank by ascending handle among the guest's GR
  channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live
  from the guest pin stream) or `svc_counter_derive` (the
  vaspaces/discovered/targeted walk chain, `sema_counter_derive=1`
  default). Both miss for the counter page -> no "counter auto" log,
  `pub_calls=0` -> the A8 loop's per-entry counter tables never fill and
  the NvOgl wait spins.
* The client does NOT map the counter pool: `/proc/<pid>/pagemap` for
  0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not
  present) and no VMA covers it - it is a UVM kernel-side object, so the
  pin stream cannot resolve it through the client's page tables. The
  stream's log shows `sem=0` (zero semaphore resolutions) even though
  the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44
  `20400f` entries; the stream's "channel-anchored expansion" for the
  UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the
  machinery that must find them (it is written for the CUDA second-client
  stall - "the only live uvmt event is the spinning sem").
* Bypass available now: the `sema_counter` parameter registers a channel
  slot manually - `echo <h_object>,<pa> > .../parameters/sema_counter`.
  Once the counter page's PA is known (via the guest kcore/UVM walk),
  this wires the publish path without a kernel rebuild.
* The rank slots for the GL client's channels (handles 0xbeec0100,
  0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 /
  0x20400ffb0.

## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported

* The client's own pagemap does NOT contain the counter pool: the
  semfix_uvmt tool's resolver (`guest_pmap_win.py <pid> 0x20400f000
  0x204010000 64`) returns NOTHING, as does the UVM pool range
  (`0x121000000 0x121100000`). The tool's `pin_counter_page` has
  therefore never succeeded for these clients (the STALL WATCHDOG's
  "re-pin counter" attempts across sessions produced no "counter:
  pinned" line).
* VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the
  emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its
  window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow
  there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`);
  poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and
  then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but
  did NOT move the client's wait. The vidmem route is not (or not only)
  the completion path; the A8 loop's tables have another source.
* gdb-attach hazard (again): attaches to the client in its post-device
  spin hang with no output; two gdb chains had to be killed by PID
  (never a pattern) and the client SIGCONT'd. Probe only while the
  client is in do_poll.

## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed)

* The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's
  method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns
  `(hi<<32)|count`. The count comes from **b1bf00**:
  - fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`;
  - slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)**
    which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client
    READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc));
    the loop takes the MIN of those reads.
* Live (r687, object handle 0xbeec0100): the window = VA
  0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0
  (slow path active), obj+0x1078=7 (fast-path field unused).
* **The host CAN write it**: resolving the window's PA via the client's
  pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via
  the RAM file) made the client read it - **obj+0x1070 = 0x100** (the
  method stores the returned count there) and the A8 "actual" table
  entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's
  completion counter can be driven from the host.
* The wait still spins: the A8 loop's first scan (compare
  a85e1d: actual vs want, want observed 0) passes with our value, but
  the loop continues into a second scan (compare a85ec1, driven by
  b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those
  (the second scan's actual source and the rsp+0x38 flag), and whether
  the other objects (entries 0-2 of the A8 table) need the same window
  writes.
* The other objects' windows were not observed (only 0xbeec0100 called
  a82f30 in the sampled window); the nvidia0-side window (obj+0x1478)
  is NOT present in the pagemap (VM_IO; host cannot write it via the
  RAM file).

## ADDENDUM (r687, session 84) — the a86038 want-scan mapped: ALL WANTS PASS with the driven window value; the residual gate is the A8 loop's internal callback/retry machinery

* Live dump of the a86038 scan (BP at 0xa8608e, per-index):
    i=0 want=0x0 actual=0x0 pend=no
    i=1 want=0x0 actual=0x0 pend=no
    i=2 want=0x0 actual=0x0 pend=no
    i=3 want=0x6 actual=0xffffffff00000100 pend=no
  (want from a82fd0(job_obj, i) = the job object's +0x98 array maxed
  against global+0x600; actual from r14+0x5e8 = our driven counter).
  NO pending entries -> the scan completes.
* Writing 0x100 to the ENTIRE readback window (offsets 0x0..0x1ff0 of VA
  0x7827e6c12000) did not clear the wait either.
* The loop still cycles: the bt stays at the a85f08-a85f1a yield call
  (b284b0(*(rsp+0x10), 0xb) - a bounded pause/yield spin of ~0x14
  units, hence the slow cadence), and the exit-attempt path a85f30
  decrements *(rbp) (w0=1) while firing the global callbacks
  (0x206f790/0x206f7a0 vtables: *(%rax), *0x10, *0x18) with args from
  rbp+0x30/0x38/0x40. The remaining gate is this callback/retry
  machinery (or another wait object), not the readback values.
* NEXT: resolve the global callback targets (0x206f790/0x206f7a0
  vtables at runtime) and trace the a85f30 exit path (what the
  callbacks do and what makes *(rbp) reach 0); or find how the real GPU
  writes the readback window (the op-4 methods 0x800000bb/0xbc the
  service skips) and emulate that write so the client's own machinery
  completes.

## ADDENDUM (r687, session 85) — **THE NvOgl WAIT CLEARED** (the session-count gate) and the client advanced to vkAllocateMemory (new wall: -2 OUT_OF_DEVICE_MEMORY)

* The A8 loop's exit requires the session count `*(u32*)(r14+0x5bc)`
  (= lib_base + 0x2070b7c, the global at +0x20705c0+0x5bc) to reach 0;
  the loop head increments `*(rbp)` and the entry scans pass with the
  driven window value, but with count=4 the exit checks
  (a85b48/a85e78: `if (r12d == 0)`) are never taken - the loop waits for
  the engine sessions to complete (count -> 0).
* **DIRECT TEST (succeeded)**: `set {unsigned int}0x<base+0x2070b7c> = 0`
  via gdb -> the loop exited, the NvOgl init completed, and the client
  ADVANCED past the post-device wait for the first time:

      VKDEVICE ok family=0
      VKFAIL vkAllocateMemory(dev, &ma, NULL, &img_mem) -> -2 (line 204)

* **NEW WALL**: `vkAllocateMemory` for the image (DEVICE_LOCAL) returns
  -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY) at vk_frame.c:204 - the guest
  driver's heap/type selection fails before any alloc RPC (no
  GSP_RM_ALLOC in dmesg). Next: the memory-heap reporting
  (FB_GET_INFO_V2 0x20801303 is handled; check the heap-size/type
  queries the driver uses for the DEVICE_LOCAL type, and the emulated
  VRAM size the driver sees).
* Recipe for the next run: slotfill + word nudge -> VKDEVICE ok; then
  write 0 to lib_base+0x2070b7c (gdb; the harness) once the NvOgl wait
  starts (the A8 loop's count gate); then the memory-alloc wall.

## ADDENDUM (r689, session 86) — **THE DEVICE_LOCAL -2 WALL CRACKED**: the 512 MiB emulated VRAM was the gate; the "nvidia-nvidia-vgpu-turing-2g" mdev type (2 GiB) lets the allocation proceed

* Evidence chain: the ICD hooks (t91.py) captured the client's allocation
  `ALLOC sType=5 size=0x4000 typeIndex=1` (a valid 16 KB DEVICE_LOCAL
  request); the standalone memory-properties dump (vk_memprops.py)
  showed device-local heap[0] = 512 MiB with a separate 2936 MiB
  non-local heap. With 512 MiB the driver failed the allocation
  (VK_ERROR_OUT_OF_DEVICE_MEMORY, no RM RPC).
* **FIX TESTED**: the cycle with `VGPU_MDEV_TYPE=nvidia-nvidia-vgpu-turing-2g`
  (start_vm_keep.sh honours the env var) -> `heap[0] size=0x80000000
  (2048 MiB)` (vk_memprops.py) -> the SAME client with the SAME recipe
  (slotfill + word nudge + count auto-clear) reaches `VKDEVICE ok` and
  the allocation NO LONGER fails -2: the client proceeds into
  `os_schedule` (a deeper wait).
* REMAINING in the new state: the client's later pushes (va=0x44852d4 /
  0x4ca52d4, then +0x586c) extend beyond the first window page and read
  `pa=0x0`/zeros (the sema injections covered page 1 only; the window's
  page list is scattered - the memdesc carries only the FIRST page).
  NEXT: extend the window injections to the full scattered page list
  (from the memdesc's page array) or accept 1-page pushes; then the
  allocation completes and the client proceeds to the buffer/pipeline.
* Tools: vk_memprops.py (standalone memory-properties dump; safe
  alongside other clients), vk_alloc_test.py (standalone allocation
  reproducer - NOTE: it hits vkCreateDevice -4 without the harness
  (word nudge + count clear), so it cannot replace the vk_frame run),
  t91.py (auto-clear + ICD memprops/alloc hooks).

## ADDENDUM (r689, session 87) — the window page-list hunt: the pages are scattered; content matches found but the VA<->page relation is unresolved

* The 2 GiB cycle's window records (recscan89): `0x4480000` descs =
  {0x107c47000 (live init stream), 0x103c06000, 0x12ecc2000};
  `0x4ca0000` descs = {0x11dbfa000, 0x12f398000 (live init stream)}.
  The window's later offsets (va+0x52d4, +0x586c, +0x9000) at
  first_PA+offset are ALL ZEROS -> the windows are NOT contiguous
  (scattered pages).
* The memdesc record (page 0x106128000) is {size=0x20000, flags
  0x03008000, PA=0x107c47000, kernel pointers 0xffff8d2e5e21ec50 /
  0x4fe7ec50 / 0x461287e8 ... , "head"/"tail"} — the page list hangs off
  the kernel pointers. A guest kcore walk of those pointers returned
  "unreadable" (stale addresses by the time of the walk; the walk must
  happen on FRESH pointers in the same pass as the dump).
* Content search for the distinctive init-stream prefix (`00 80 01 20
  b5 c5 00 00 04 00 04 20`) found the window pages:
  0x107c47000+0x0 (the injected init page), 0x103c06000+0x858,
  0x11dbfa000+0x858, 0x126d69000+0x59c — the later pages hold the same
  engine-init streams at arbitrary offsets; the VA<->page relation is
  NOT linear (the +0x59c push appears in pages whose +0x0 is zero), so
  the mapping cannot be reconstructed by offset alone.
* NEXT: (a) dump the memdesc + kcore-walk its page-list pointers in ONE
  guest pass (fresh addresses); or (b) inject the content-matched pages
  at trial VAs and watch the service's reads; or (c) the service-side
  decode of the window page lists. The client (2632) blocks in
  os_schedule at the allocation — the unread later pushes are the
  suspected gate.

## Session 88 addendum - the push-page fix, the gdb-kill wedge, the ring-mapping gap

**1. PATH-5 RULE (the push pages).** A GR window's pushes do NOT live at
the window base page: the r689-family windows carry their method streams in
WINDOW PAGE 5 (va offsets 0x52d4/0x559c/0x586c/0x596c all live in
0x5000-0x5fff). Standing injections at the base page resolve pa=0x0 for the
pushes (the service's `sema-pushall ... pa=0x0` lines) and the channel
holds/skips entries as "GR no-capture". The fix: page PA =
match_filoff - (va & 0xfff) + 0x80000000 for a content match of the push's
first8 bytes constrained to the same in-page offset; then
`echo 0x<va&~0xfff>,0x<pa>,0x1000,1 > .../sema_map_add`. Proven on r689:
channel beec0104's pushes at va=0x4d00000 (page -> PA 0x12f398000) went
from first8=00000000 to first8=20012000 0000c5c0, and `fast sweep`
applied 3281 -> 3809 (+30/sweep sustained).

**2. host_resolve.py** automates 1: it polls dmesg for sema-pushall lines
with pa=0x0 and non-zero first8, searches /dev/shm/vgpu-guest-ram
(offset-constrained), and injects the page. The service's 6 s no-capture
hold is the race window - run it BEFORE the client and keep it polling.

**3. NEVER SIGKILL AN ATTACHED GDB.** On r689, killing the stale t91.py
gdb (kill -9 on PID 5334) killed its tracees (client threads
2883/2884/2885/3048); one held the guest RM API rwsem write lock; the lock
leaked and every later RM call blocked (`[vkps] Update blocked for more
than 1228 seconds` in rmapiLockAcquire -> RmGssLegacyRpcCmd ->
_nv04ControlWithSecInfo). The t91 harness design is correct: `timeout 500`
SIGTERMs the gdb and it detaches cleanly. If a gdb must go, use SIGTERM
and wait; SIGKILL only when the guest is about to be cycled anyway. A
leaked kernel rwsem cannot be repaired live.

**4. The r690 gap: the GR channel ring mapping.** Fresh cycle r690 (mdev
f5a796b0): every RM RPC of the client init is answered (GR ctx commit,
channel allocs class 0x9067 hObject 0xcef00000/0xcef00080, EVPOSTs), but
the service reports `gpfifo done=0 nomap=938 pushes=0` - it cannot read
any GPFIFO ring, so no entry decodes, no pushall line appears, and the
guest's completion poll loops on RM control 0x20800a70 until the ~10 s RM
timeout -> vkCreateDevice returns -4 (VK_ERROR_DEVICE_LOST). r689's
service had done=19108 (rings mapped) - so the ring mapping is the
missing ingredient for a fresh client in r690; candidate paths: the FIFO
alloc params (12 bytes, GPFIFO VA inside) are not logged; the scanner saw
the CE window (ipins 4592) but no GR ring this boot. Next: log the 0x9067
alloc params (or scan the guest for the ring VA) and inject the ring page.

**5. gr_walk.py / gr_walk2.py** are the guest-side kcore walkers (fresh-pass
memdesc discovery; 12 records found, kptrs readable in-pass) kept as the
alternate grounding path.