# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion

## The two-part unblock

1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the
   nvkms/Vulkan path allocates its completion event as
   NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now
   registers it alongside 0x79. Live proof (r683, sema_dbg=1):

       GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011
           hParent=0xcaf00001 hObject=0xcaf00003
       event registered hEvent=0xcaf00003 hSrc=0x0
           notifyIndex=1006632972 (off 8, slot 13)
           params=1100e0c1000000007e0000000c00003c0000000000000000
       EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0

   (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc =
   NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records
   (`guest_read == our_write` in the rmstate dump), but the guest's
   event dispatch does NOT write the event's notification word — the
   residual delivery gap.

2. **The completion word found and driven (the actual unblock)**. The
   main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads
   a u32 readiness word: `should-wait = (word != 1)`; the word lives at
   `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event
   notification buffer; holder chain sub -> obj2 -> holder, resolved
   live by t79.py). The host can write it directly in the guest RAM
   file:

       PA = pagemap PFN << 12  (+0x80);  file offset = PA - 0x80000000
       (above-4G RAM; below-4G would be PA itself per the cycle's
       guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000)

   Writing 1 flipped the wait and **vkCreateDevice returned success**
   (`VKDEVICE ok family=0`) — reproduced three times:
   * r683: manual write after a t79 probe;
   * r684: nudger (all 16 KB mapping words) + slotfill restart;
   * r684d: probe -> single-word write (the reliable recipe).

## The reliable recipe (per run)

1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional.
2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the
   slots re-arm, so re-run slotfill if the client is stuck in the
   sched_yield spin before the word-wait).
3. Wait for the main thread in `do_poll` (the word-wait).
4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA;
   host write `01 00 00 00` at `PA-0x80000000`.
5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to
   image/memory/pipeline setup; its next wait is NOT the same word
   (the word stays 1) — next session: probe the follow-on state
   (attach fails while the client spins in the post-device phase;
   use the /proc-only checks + a longer observation window).

## Cautions learned

* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the
  -4 give-up path (an RM error flag word gets written): restrict nudges
  to the single probed word.
* Attaching gdb to the client while it is in the tight post-device spin
  can hang the attach (leave a `timeout`-wrapped gdb, then kill that
  gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a
  dead tracer otherwise wedges it).
* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in
  python, not zsh.

## Tools

* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA)
  and prints the client's nvidiactl rw-s mapping sizes.
* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB
  nvidiactl mapping (pagemap; run in the guest as root).
* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any
  zero word (multi-page variant; prefer the single-word recipe).

## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6)

* The post-device spin is the **GL module bring-up**: the main thread's
  A8-layer retry loop (eglcore 0xa85d24) works on a work item whose
  embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6
  submit layer (0xd624e0) ORs **0x200000** into its pending word (the
  same engine-pending bit SESSION 75/76 watched on E+0x150), then the
  A8 loop yields via the B284B0 spin helper (sched_yield) waiting for
  the GL ucode init to complete on the GR engine.
* **The push windows ARE now readable** (first time): the resolver6
  recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} ->
  same-page 0x20000 memdesc PA -> `sema_map_add`) injected
  `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d
  68000000` = the 2-method semaphore-like push) and `0x4ca0000 ->
  0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5,
  `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d
  70/68`). The service now decodes the streams (`sema-push ... tail=0dc0
  012000000070...`) where SESSION 73-74 saw "unread".
  CAUTION: the resolver injects every candidate; a zero-page candidate
  injected last replaces the good mapping for that VA (re-inject the
  good page after the scan).
* **Remaining gate**: the service still reports `have_rel=0` for every
  GR push (the CE-shaped release matcher) and — decisively — the
  GR-forward machinery is NOT engaging: `drain held=4 starved=149
  forwarded=0 ... gr_nocap=4`. The GL ucode init work is never
  forwarded/executed, so the completion (and the 0x200000 bit clear)
  never arrives. Next: the GR forward/capture path for these channels
  (operand capture + release decode for the GR shape), then the client
  proceeds to VKFRAME.

## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe

* The GR/GL init releases decode only with the low-aperture acceptance
  (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms,
  commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group
  `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0
  pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ...
  02000001`); `have_rel=1`, `applied` rising, un-translated releases
  queue in svc_pending (417 observed).
* Corrected pair-specific push injections for r686:
  `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc),
  `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus
  `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window).
  CAUTION: resolver7 injects each page's descs for BOTH pairs (union
  loop) - fix it to keep per-pair pages or inject manually from the
  per-pair scan. The window-record layout is {tag, VA, SIZE, 0,
  "tail", ...} stride 0x40 (search the VA u64 with size at +8 and
  "tail" at +0x10).
* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no
  window records in the known record pages; the service cannot resolve
  them (targeted rm_mem miss) so the releases stay pending and the
  NvOgl wait spins. Find their PAs (client mapping pages / RM
  VA-tracker / live poll-watch), inject, and the pending releases
  should apply.

## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying

* **The record scan fix**: the RM window record is 0x40 bytes
  {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is
  stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT
  "tail" (the hex display `7461696c` reads as "tail" only as a number).
  Searching for b'tail' finds nothing; `wall2_recscan.py` searches for
  the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at
  -0x20/-0x18/-0x10. It found **all 75 records** in one pass.
* **The GR init semaphore windows** (r686):
  `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content:
  +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 /
  rel=0x4b17e30 pay=1 payloads) and
  `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the
  release applied there: +0xb10 became 2). Injections:
  `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and
  `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`.
* **State after the injections**: the low-VA releases decode
  (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at
  ~418 (the client re-kicks continuously); the NvOgl wait still spins
  (the A8 loop compares per-entry counter tables r14+0x5e8 vs the
  wanted set - the completion is not just the semaphore value). Next:
  the A8 counter tables' source (what fills r14+0x5e8 / the "wanted"
  table - likely the GR channel's own progress counters) or the GR
  forward/ucode execution (`forwarded=0 gr_nocap=7` still).