# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion ## The two-part unblock 1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the nvkms/Vulkan path allocates its completion event as NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now registers it alongside 0x79. Live proof (r683, sema_dbg=1): GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011 hParent=0xcaf00001 hObject=0xcaf00003 event registered hEvent=0xcaf00003 hSrc=0x0 notifyIndex=1006632972 (off 8, slot 13) params=1100e0c1000000007e0000000c00003c0000000000000000 EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0 (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc = NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records (`guest_read == our_write` in the rmstate dump), but the guest's event dispatch does NOT write the event's notification word — the residual delivery gap. 2. **The completion word found and driven (the actual unblock)**. The main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads a u32 readiness word: `should-wait = (word != 1)`; the word lives at `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event notification buffer; holder chain sub -> obj2 -> holder, resolved live by t79.py). The host can write it directly in the guest RAM file: PA = pagemap PFN << 12 (+0x80); file offset = PA - 0x80000000 (above-4G RAM; below-4G would be PA itself per the cycle's guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000) Writing 1 flipped the wait and **vkCreateDevice returned success** (`VKDEVICE ok family=0`) — reproduced three times: * r683: manual write after a t79 probe; * r684: nudger (all 16 KB mapping words) + slotfill restart; * r684d: probe -> single-word write (the reliable recipe). ## The reliable recipe (per run) 1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional. 2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the slots re-arm, so re-run slotfill if the client is stuck in the sched_yield spin before the word-wait). 3. Wait for the main thread in `do_poll` (the word-wait). 4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA; host write `01 00 00 00` at `PA-0x80000000`. 5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to image/memory/pipeline setup; its next wait is NOT the same word (the word stays 1) — next session: probe the follow-on state (attach fails while the client spins in the post-device phase; use the /proc-only checks + a longer observation window). ## Cautions learned * Writing ALL 16 KB mapping words (the multi-page nudger) can trip the -4 give-up path (an RM error flag word gets written): restrict nudges to the single probed word. * Attaching gdb to the client while it is in the tight post-device spin can hang the attach (leave a `timeout`-wrapped gdb, then kill that gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a dead tracer otherwise wedges it). * zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in python, not zsh. ## Tools * `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA) and prints the client's nvidiactl rw-s mapping sizes. * `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB nvidiactl mapping (pagemap; run in the guest as root). * `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any zero word (multi-page variant; prefer the single-word recipe). ## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6) * The post-device spin is the **GL module bring-up**: the main thread's A8-layer retry loop (eglcore 0xa85d24) works on a work item whose embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6 submit layer (0xd624e0) ORs **0x200000** into its pending word (the same engine-pending bit SESSION 75/76 watched on E+0x150), then the A8 loop yields via the B284B0 spin helper (sched_yield) waiting for the GL ucode init to complete on the GR engine. * **The push windows ARE now readable** (first time): the resolver6 recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} -> same-page 0x20000 memdesc PA -> `sema_map_add`) injected `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d 68000000` = the 2-method semaphore-like push) and `0x4ca0000 -> 0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5, `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d 70/68`). The service now decodes the streams (`sema-push ... tail=0dc0 012000000070...`) where SESSION 73-74 saw "unread". CAUTION: the resolver injects every candidate; a zero-page candidate injected last replaces the good mapping for that VA (re-inject the good page after the scan). * **Remaining gate**: the service still reports `have_rel=0` for every GR push (the CE-shaped release matcher) and — decisively — the GR-forward machinery is NOT engaging: `drain held=4 starved=149 forwarded=0 ... gr_nocap=4`. The GL ucode init work is never forwarded/executed, so the completion (and the 0x200000 bit clear) never arrives. Next: the GR forward/capture path for these channels (operand capture + release decode for the GR shape), then the client proceeds to VKFRAME. ## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe * The GR/GL init releases decode only with the low-aperture acceptance (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms, commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0 pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ... 02000001`); `have_rel=1`, `applied` rising, un-translated releases queue in svc_pending (417 observed). * Corrected pair-specific push injections for r686: `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc), `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window). CAUTION: resolver7 injects each page's descs for BOTH pairs (union loop) - fix it to keep per-pair pages or inject manually from the per-pair scan. The window-record layout is {tag, VA, SIZE, 0, "tail", ...} stride 0x40 (search the VA u64 with size at +8 and "tail" at +0x10). * Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no window records in the known record pages; the service cannot resolve them (targeted rm_mem miss) so the releases stay pending and the NvOgl wait spins. Find their PAs (client mapping pages / RM VA-tracker / live poll-watch), inject, and the pending releases should apply. ## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying * **The record scan fix**: the RM window record is 0x40 bytes {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT "tail" (the hex display `7461696c` reads as "tail" only as a number). Searching for b'tail' finds nothing; `wall2_recscan.py` searches for the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at -0x20/-0x18/-0x10. It found **all 75 records** in one pass. * **The GR init semaphore windows** (r686): `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content: +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 / rel=0x4b17e30 pay=1 payloads) and `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the release applied there: +0xb10 became 2). Injections: `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`. * **State after the injections**: the low-VA releases decode (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at ~418 (the client re-kicks continuously); the NvOgl wait still spins (the A8 loop compares per-entry counter tables r14+0x5e8 vs the wanted set - the completion is not just the semaphore value). Next: the A8 counter tables' source (what fills r14+0x5e8 / the "wanted" table - likely the GR channel's own progress counters) or the GR forward/ucode execution (`forwarded=0 gr_nocap=7` still). ## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass * `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a translation for the rank slot VA `0x20400fec0 + (19-r)*0x10` (r = gr_index = rank by ascending handle among the guest's GR channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live from the guest pin stream) or `svc_counter_derive` (the vaspaces/discovered/targeted walk chain, `sema_counter_derive=1` default). Both miss for the counter page -> no "counter auto" log, `pub_calls=0` -> the A8 loop's per-entry counter tables never fill and the NvOgl wait spins. * The client does NOT map the counter pool: `/proc//pagemap` for 0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not present) and no VMA covers it - it is a UVM kernel-side object, so the pin stream cannot resolve it through the client's page tables. The stream's log shows `sem=0` (zero semaphore resolutions) even though the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44 `20400f` entries; the stream's "channel-anchored expansion" for the UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the machinery that must find them (it is written for the CUDA second-client stall - "the only live uvmt event is the spinning sem"). * Bypass available now: the `sema_counter` parameter registers a channel slot manually - `echo , > .../parameters/sema_counter`. Once the counter page's PA is known (via the guest kcore/UVM walk), this wires the publish path without a kernel rebuild. * The rank slots for the GL client's channels (handles 0xbeec0100, 0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 / 0x20400ffb0. ## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported * The client's own pagemap does NOT contain the counter pool: the semfix_uvmt tool's resolver (`guest_pmap_win.py 0x20400f000 0x204010000 64`) returns NOTHING, as does the UVM pool range (`0x121000000 0x121100000`). The tool's `pin_counter_page` has therefore never succeeded for these clients (the STALL WATCHDOG's "re-pin counter" attempts across sessions produced no "counter: pinned" line). * VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`); poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but did NOT move the client's wait. The vidmem route is not (or not only) the completion path; the A8 loop's tables have another source. * gdb-attach hazard (again): attaches to the client in its post-device spin hang with no output; two gdb chains had to be killed by PID (never a pattern) and the client SIGCONT'd. Probe only while the client is in do_poll. ## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed) * The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns `(hi<<32)|count`. The count comes from **b1bf00**: - fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`; - slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)** which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc)); the loop takes the MIN of those reads. * Live (r687, object handle 0xbeec0100): the window = VA 0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0 (slow path active), obj+0x1078=7 (fast-path field unused). * **The host CAN write it**: resolving the window's PA via the client's pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via the RAM file) made the client read it - **obj+0x1070 = 0x100** (the method stores the returned count there) and the A8 "actual" table entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's completion counter can be driven from the host. * The wait still spins: the A8 loop's first scan (compare a85e1d: actual vs want, want observed 0) passes with our value, but the loop continues into a second scan (compare a85ec1, driven by b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those (the second scan's actual source and the rsp+0x38 flag), and whether the other objects (entries 0-2 of the A8 table) need the same window writes. * The other objects' windows were not observed (only 0xbeec0100 called a82f30 in the sampled window); the nvidia0-side window (obj+0x1478) is NOT present in the pagemap (VM_IO; host cannot write it via the RAM file). ## ADDENDUM (r687, session 84) — the a86038 want-scan mapped: ALL WANTS PASS with the driven window value; the residual gate is the A8 loop's internal callback/retry machinery * Live dump of the a86038 scan (BP at 0xa8608e, per-index): i=0 want=0x0 actual=0x0 pend=no i=1 want=0x0 actual=0x0 pend=no i=2 want=0x0 actual=0x0 pend=no i=3 want=0x6 actual=0xffffffff00000100 pend=no (want from a82fd0(job_obj, i) = the job object's +0x98 array maxed against global+0x600; actual from r14+0x5e8 = our driven counter). NO pending entries -> the scan completes. * Writing 0x100 to the ENTIRE readback window (offsets 0x0..0x1ff0 of VA 0x7827e6c12000) did not clear the wait either. * The loop still cycles: the bt stays at the a85f08-a85f1a yield call (b284b0(*(rsp+0x10), 0xb) - a bounded pause/yield spin of ~0x14 units, hence the slow cadence), and the exit-attempt path a85f30 decrements *(rbp) (w0=1) while firing the global callbacks (0x206f790/0x206f7a0 vtables: *(%rax), *0x10, *0x18) with args from rbp+0x30/0x38/0x40. The remaining gate is this callback/retry machinery (or another wait object), not the readback values. * NEXT: resolve the global callback targets (0x206f790/0x206f7a0 vtables at runtime) and trace the a85f30 exit path (what the callbacks do and what makes *(rbp) reach 0); or find how the real GPU writes the readback window (the op-4 methods 0x800000bb/0xbc the service skips) and emulate that write so the client's own machinery completes. ## ADDENDUM (r687, session 85) — **THE NvOgl WAIT CLEARED** (the session-count gate) and the client advanced to vkAllocateMemory (new wall: -2 OUT_OF_DEVICE_MEMORY) * The A8 loop's exit requires the session count `*(u32*)(r14+0x5bc)` (= lib_base + 0x2070b7c, the global at +0x20705c0+0x5bc) to reach 0; the loop head increments `*(rbp)` and the entry scans pass with the driven window value, but with count=4 the exit checks (a85b48/a85e78: `if (r12d == 0)`) are never taken - the loop waits for the engine sessions to complete (count -> 0). * **DIRECT TEST (succeeded)**: `set {unsigned int}0x = 0` via gdb -> the loop exited, the NvOgl init completed, and the client ADVANCED past the post-device wait for the first time: VKDEVICE ok family=0 VKFAIL vkAllocateMemory(dev, &ma, NULL, &img_mem) -> -2 (line 204) * **NEW WALL**: `vkAllocateMemory` for the image (DEVICE_LOCAL) returns -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY) at vk_frame.c:204 - the guest driver's heap/type selection fails before any alloc RPC (no GSP_RM_ALLOC in dmesg). Next: the memory-heap reporting (FB_GET_INFO_V2 0x20801303 is handled; check the heap-size/type queries the driver uses for the DEVICE_LOCAL type, and the emulated VRAM size the driver sees). * Recipe for the next run: slotfill + word nudge -> VKDEVICE ok; then write 0 to lib_base+0x2070b7c (gdb; the harness) once the NvOgl wait starts (the A8 loop's count gate); then the memory-alloc wall. ## ADDENDUM (r689, session 86) — **THE DEVICE_LOCAL -2 WALL CRACKED**: the 512 MiB emulated VRAM was the gate; the "nvidia-nvidia-vgpu-turing-2g" mdev type (2 GiB) lets the allocation proceed * Evidence chain: the ICD hooks (t91.py) captured the client's allocation `ALLOC sType=5 size=0x4000 typeIndex=1` (a valid 16 KB DEVICE_LOCAL request); the standalone memory-properties dump (vk_memprops.py) showed device-local heap[0] = 512 MiB with a separate 2936 MiB non-local heap. With 512 MiB the driver failed the allocation (VK_ERROR_OUT_OF_DEVICE_MEMORY, no RM RPC). * **FIX TESTED**: the cycle with `VGPU_MDEV_TYPE=nvidia-nvidia-vgpu-turing-2g` (start_vm_keep.sh honours the env var) -> `heap[0] size=0x80000000 (2048 MiB)` (vk_memprops.py) -> the SAME client with the SAME recipe (slotfill + word nudge + count auto-clear) reaches `VKDEVICE ok` and the allocation NO LONGER fails -2: the client proceeds into `os_schedule` (a deeper wait). * REMAINING in the new state: the client's later pushes (va=0x44852d4 / 0x4ca52d4, then +0x586c) extend beyond the first window page and read `pa=0x0`/zeros (the sema injections covered page 1 only; the window's page list is scattered - the memdesc carries only the FIRST page). NEXT: extend the window injections to the full scattered page list (from the memdesc's page array) or accept 1-page pushes; then the allocation completes and the client proceeds to the buffer/pipeline. * Tools: vk_memprops.py (standalone memory-properties dump; safe alongside other clients), vk_alloc_test.py (standalone allocation reproducer - NOTE: it hits vkCreateDevice -4 without the harness (word nudge + count clear), so it cannot replace the vk_frame run), t91.py (auto-clear + ICD memprops/alloc hooks). ## ADDENDUM (r689, session 87) — the window page-list hunt: the pages are scattered; content matches found but the VA<->page relation is unresolved * The 2 GiB cycle's window records (recscan89): `0x4480000` descs = {0x107c47000 (live init stream), 0x103c06000, 0x12ecc2000}; `0x4ca0000` descs = {0x11dbfa000, 0x12f398000 (live init stream)}. The window's later offsets (va+0x52d4, +0x586c, +0x9000) at first_PA+offset are ALL ZEROS -> the windows are NOT contiguous (scattered pages). * The memdesc record (page 0x106128000) is {size=0x20000, flags 0x03008000, PA=0x107c47000, kernel pointers 0xffff8d2e5e21ec50 / 0x4fe7ec50 / 0x461287e8 ... , "head"/"tail"} — the page list hangs off the kernel pointers. A guest kcore walk of those pointers returned "unreadable" (stale addresses by the time of the walk; the walk must happen on FRESH pointers in the same pass as the dump). * Content search for the distinctive init-stream prefix (`00 80 01 20 b5 c5 00 00 04 00 04 20`) found the window pages: 0x107c47000+0x0 (the injected init page), 0x103c06000+0x858, 0x11dbfa000+0x858, 0x126d69000+0x59c — the later pages hold the same engine-init streams at arbitrary offsets; the VA<->page relation is NOT linear (the +0x59c push appears in pages whose +0x0 is zero), so the mapping cannot be reconstructed by offset alone. * NEXT: (a) dump the memdesc + kcore-walk its page-list pointers in ONE guest pass (fresh addresses); or (b) inject the content-matched pages at trial VAs and watch the service's reads; or (c) the service-side decode of the window page lists. The client (2632) blocks in os_schedule at the allocation — the unread later pushes are the suspected gate. ## Session 88 addendum - the push-page fix, the gdb-kill wedge, the ring-mapping gap **1. PATH-5 RULE (the push pages).** A GR window's pushes do NOT live at the window base page: the r689-family windows carry their method streams in WINDOW PAGE 5 (va offsets 0x52d4/0x559c/0x586c/0x596c all live in 0x5000-0x5fff). Standing injections at the base page resolve pa=0x0 for the pushes (the service's `sema-pushall ... pa=0x0` lines) and the channel holds/skips entries as "GR no-capture". The fix: page PA = match_filoff - (va & 0xfff) + 0x80000000 for a content match of the push's first8 bytes constrained to the same in-page offset; then `echo 0x,0x,0x1000,1 > .../sema_map_add`. Proven on r689: channel beec0104's pushes at va=0x4d00000 (page -> PA 0x12f398000) went from first8=00000000 to first8=20012000 0000c5c0, and `fast sweep` applied 3281 -> 3809 (+30/sweep sustained). **2. host_resolve.py** automates 1: it polls dmesg for sema-pushall lines with pa=0x0 and non-zero first8, searches /dev/shm/vgpu-guest-ram (offset-constrained), and injects the page. The service's 6 s no-capture hold is the race window - run it BEFORE the client and keep it polling. **3. NEVER SIGKILL AN ATTACHED GDB.** On r689, killing the stale t91.py gdb (kill -9 on PID 5334) killed its tracees (client threads 2883/2884/2885/3048); one held the guest RM API rwsem write lock; the lock leaked and every later RM call blocked (`[vkps] Update blocked for more than 1228 seconds` in rmapiLockAcquire -> RmGssLegacyRpcCmd -> _nv04ControlWithSecInfo). The t91 harness design is correct: `timeout 500` SIGTERMs the gdb and it detaches cleanly. If a gdb must go, use SIGTERM and wait; SIGKILL only when the guest is about to be cycled anyway. A leaked kernel rwsem cannot be repaired live. **4. The r690 gap: the GR channel ring mapping.** Fresh cycle r690 (mdev f5a796b0): every RM RPC of the client init is answered (GR ctx commit, channel allocs class 0x9067 hObject 0xcef00000/0xcef00080, EVPOSTs), but the service reports `gpfifo done=0 nomap=938 pushes=0` - it cannot read any GPFIFO ring, so no entry decodes, no pushall line appears, and the guest's completion poll loops on RM control 0x20800a70 until the ~10 s RM timeout -> vkCreateDevice returns -4 (VK_ERROR_DEVICE_LOST). r689's service had done=19108 (rings mapped) - so the ring mapping is the missing ingredient for a fresh client in r690; candidate paths: the FIFO alloc params (12 bytes, GPFIFO VA inside) are not logged; the scanner saw the CE window (ipins 4592) but no GR ring this boot. Next: log the 0x9067 alloc params (or scan the guest for the ring VA) and inject the ring page. **5. gr_walk.py / gr_walk2.py** are the guest-side kcore walkers (fresh-pass memdesc discovery; 12 records found, kptrs readable in-pass) kept as the alternate grounding path.