diff options
Diffstat (limited to 'docs/04-implementation/04-real-render-walls.md')
| -rw-r--r-- | docs/04-implementation/04-real-render-walls.md | 461 |
1 files changed, 461 insertions, 0 deletions
diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md index f6ea39d..6a29f6c 100644 --- a/docs/04-implementation/04-real-render-walls.md +++ b/docs/04-implementation/04-real-render-walls.md @@ -185,3 +185,464 @@ gdb leaves the VM stopped (recover with a detach-only connect); `stepi` loops over the stub freeze the guest's userspace for minutes - prefer breakpoints; every probe run with the caps-fill breakpoint changes the driver's path, so keep the fill in each experiment. + +## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD) + +**The fence wall is closed.** The client's per-frame fence wait polls a +16 KB BO word that a submit-push INC-5 semaphore group targets +(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a +GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring +already carries the entries it pre-built for the next submits, idx12..46), +so the GP_PUT-bounded pump walk never decoded it and a host-side assist had +to write the word. The service now sweeps that pre-built continuation +itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the +ring's validated translation, accepts only pushes that pass the strict +method-stream gate, extracts the addr-0x5c RELEASE groups and applies them +forward-only through the standard pending ledger. Live evidence: the sweep +finds the groups and the ledger applies the fence word once per frame; +`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still +black - the shader work itself is unexecuted. + +**The compute-dispatch shape (why nothing executes).** The workload is a +single compute dispatch (640x480 storage image). Its push is a DIRECT +method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0` +on subchannel 1, then the context windows (`0x7b4` local-memory window B, +`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and +the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data +port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO +inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture +that feeds the physical-engine forward path never fires - and the CE-shaped +ops path is not even consulted for GR-class channels by construction. Real +pixels therefore require a translator that recognizes the C5C0 section, +builds the vendor launch the physical channel expects (the same prelude/QMD +shape the CUDA path already executes) from the section's fields and data +runs, and forwards it with the existing operand/relocation machinery. The +method vocabulary overlaps the physical prelude's existing knowledge +(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. + +## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale + +**The live ring is in guest RAM and the sweep's decode was right.** +`walk_probe 0x4020000` (the client's own page tables, vas[0] +`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA +to **guest RAM** (PA `0x11623d000` on r759; the page moves per run: +`0x104b55000` on r761) and its content is the 14-entry ring the sweep +decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 -> +the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot +`0x4250110`, ... - all pushes inside THIS client's own allocation window +(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own +VIDHEAP log). The decoded ring is live content, not a stale page. + +**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759 +the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`; +the userd-derived candidate loop then tried `0xb0000 - 0x10000 = +0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push +pattern) passed the plausibility gate and whose USERD probe hit the real +0xb0000 block, so the pin landed. The pinned entry's push reads all-zero +through every source; the pump retried 39 times and logged `skipping +stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`, +while `sema_probe 0x4020000` then answered the wrong page +(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep +included). Section 10's "stale session page" conclusion was this pin, not +the sweep. + +**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` + +`svc_ring_walk_live`: a candidate whose entry0 push provably reads +all-zero through the client's page tables is rejected when the VAS walk +resolves the ring VA to a page with a live entry (applied at the +userd-derived, FB-convention and existing-pin paths; the walk cannot vote +for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates +and accepts the channel's known USERD cross-aperture (the driver places +GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3) +the util sweep arbitrates a dead channel-path ring entry against the +direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and +their dedup keys are per-client now (reset in `svc_session_flush`); (5) +`sema-copyent` logs once per copy OP, not per row - one 480-row frame +copy had exhausted the 128/session budget and hidden every later copy. +Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0 +pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100 +va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the +fence fold releases fire, and the client completes (`VKFRAME_DONE +frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`). + +**The two utility-channel LID uploads are the driver's zero buffers by +design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256 +lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 -> +`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method +buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`; +`0x4250110`) read zero for the whole client run - the driver never +writes them. The GR channel's own LID entry DOES carry a real payload +(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param= +0x4b86000 size=0x1900`), which proves the arm works with real bytes on +this boot too. + +**Where the code page stands.** `prog=0xdfd8feb900` (the captured +launch): `sema_probe` answered vidmem `0x62b900` early in the run while +`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer +mapping ROTATES (a code pool), and the shadow's learned lat entry for the +VA can go stale. Both pages read zero in the shadow: the driver's code +upload has not been observed on any decoded path (no LID, no CE copy - +the only `sema-copyent` op of the whole run is the 480-row frame +copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0 +bytes=0x12c000` and correctly copies the still-empty image). The +remaining question is the SASS delivery shape for the C5C0 compute path +(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale +lat preference for the rotating code VA. + +**Next steps (ranked).** +1. Resolve the code VA through the VAS walk at launch time (the walk is + the live truth; the lat entry goes stale as the pool rotates) and + watch the code page the QMD actually names. +2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs + with a bounded methods witness while the client is live (the params + arrive by LID; the code must arrive by a sibling path). +3. Then the `n_data=0` descriptor/operand wiring (section 8). + +## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running + +**The delivery shape (decoded from the CE channel).** A CE-class channel +(eng=11) carries, as entry[4] of its ring, a push whose method stream is a +LAUNCH_DMA copy with the SASS bytes INLINE: + +``` +sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500 +sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8 +sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1 +sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0 +``` + +The source is the push itself (`in=0x120a00004`, right after the method +header) and the destination is a vidmem VA (`out=0x205000`, then +`0x206000`, ... 4096 B per run) - the engine-side "download program +range" path. The Vulkan client's CE channels carry the SAME shaped +entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`, +19 entries swept) - so the SASS delivery is a CE inline copy, not the +utility-channel LID entries (those are the driver's zero buffers, +section 12). + +**Why the copies never ran (three gates).** (1) The CE channel's ring and +USERD live in VIDMEM and resolve only by boot-lottery: the FB convention +matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> +fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the +frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan +instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768), +so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs +(0x120000000+) need the client's VAS registered: the parked +COPY_SERVER_PDES candidate only registered at +15 s (age-out, +`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`, +8000 ms from the cycle) expired first - r763 logged `skipping stale +entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and +`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy +pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s +(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms +parked`) but the ring gate still blocks. (3) The shadow ring scan +fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited +retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no +candidate on r768: the ring/USERD content is volatile at scan instants +(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a +`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan +misses the window. + +**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its +own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already +does this) instead of scanning for ring entries: for every USERD-shaped +block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0 ++ the probe. 2. Or latch the ring at the first CE submission (the ring +content appears exactly when the driver writes entry0). 3. With the ring +latched and the VAS at +5 s, verify the inline copies execute +(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code +page gets content; then the `n_data` descriptor wiring (section 8). + +## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring + +**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in +the notifier-completion path for CE channels *before* the clear write - the +one instant the ring content and the USERD are both live - reads entry0 +through the channel translation and the trusted walk, then scans the shadow +(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB +aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs +the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating +candidate is pinned/latched exactly like the regular CE autopin. `928c0730` +fixed the entry decode: the push VA is 40 bits, the scan had declared it +`u32`, so `scan_hits` was zero by construction. + +**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.** +The scan started at offset 0 and accepted the first ring-shaped page for +*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d` +(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's +ring/page, its first release decoded to the wrong semaphore word (+0 +instead of +4) and was dropped as stale (`cur=1`, already written by +rank 0), so the client's rank-1 tracker wait never completed. victory_mail +then spun at 100% CPU with `pub_calls=0` on three consecutive boots +(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era +module (`25523f84`) on a fresh boot passed the same client (VADD +`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the +SESSION 133/134 code. The fix tries the rank's convention offset +(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and +never accepts an offset another CE rank already latched this session; an +unlatched rank stays for the regular per-pass latch. Verified on r781: +ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/ +0x120000, USERD +0x10000 each), rank 1's own semaphore word applied +(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed +(`pub_calls=185132+`, no spin). + +**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0): +`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0 +va=0x120000000 len=40)` - a live entry - but the client's pushes decoded +`unread=1 injected=0`: the guest pin scanner covered CUDA clients +(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk` +engaged the tools (demand pins active) but r782/r783's captures reported +`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first +64 MB, and no classic autopin either. + +**Where the Vulkan rings actually are.** The client's own vidheap ioctl +log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations: +`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos +(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's +`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and +of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image +for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000` +sibling) finds nothing - the ring content reaches neither the shadow at its +allocation offset nor guest RAM. + +**The frame chain (r783, live).** The Vulkan client completes three frames +(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black. +The live GR channel (`h=0xbeec0100`) shows the dependency explicitly: +`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together +with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000 +dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk +maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0 +pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the +framebuffer. The shadow at `0x570000` reads zero: the data pages are the +ones the CE inline copies are supposed to fill, and the GR launch is held +on their absence (the designed `dstpend` dependency). So the whole Vulkan +chain reduces once more to the un-latched CE ring: no ring -> no inline +i2m copies -> no data/code pages -> held launches -> black frame. + +**Next steps (ranked).** 1. Cover the client's real allocation in the ring +capture: raise the capture scan bound above the recorded channel +allocations (e.g. `0x6000000`) and/or seed candidates from the client's +vidheap offsets, then re-check whether the entry arrives at the +submission instant (a live `sema_vidmem_dump` at `0x4100000` while the +client is streaming would settle it; the post-run dumps read zero). 2. If +the shadow still reads zero at the allocation offset during live +submissions, the client-side write watch is the remaining forensic: find +the CPU pointer the driver writes GPFIFO entries through, its PA via the +guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a +latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd` +with the code dsts), the code/data pages fill, then the `n_data` +descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA +battery on the fixed module (the r781 boot was a slow-crawl boot; the +spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel +forward timeout. + +## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target + +**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).** + +*The write path works; the driver does not use it for the ring.* A direct +BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through +`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated +vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest +and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct +into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A +high-frequency watcher (12 candidate offsets, ~2.5 Hz, including +`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000 +USERDs) recorded ZERO nonzero dumps across the whole live streaming window +of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`, +`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR +shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the +ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of +the full 4 GB guest-RAM image for both patterns (little-endian, corrected +from an earlier reversed search) finds nothing either. The walk and the +learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page +(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap +ioctl log (`PREV149-OUT`) names the channel allocations +`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is +written neither to the shadow at its allocation offsets nor anywhere in +RAM reachable through the guest's own page tables. + +*A stale-block trap explains every historical "latch".* On CUDA-first +boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan +channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012` +block surviving in VRAM after its session. The decoded entry +(`va=0x120000000 len=40`) is the old session's (both clients' first CE +pushes are shape-identical), the expectations seed from the stale pool +words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan +client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on +every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000" +rows are the same artefact: both were CUDA-first boots. All genuine +first-submission latches on record read `put=1 get=0` at the capture +instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`) +in the first-submission capture - a just-submitted entry cannot be +consumed at the notifier-clear instant. + +*The frontier is now precise.* The Vulkan client's CE submissions complete +only through the fabricated completion machinery (the driver never reads +its ring back), its CE ring content is invisible to every host source, and +the GR launch for each frame is held on the unexecuted CE i2m copies +(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem +`0x570000`, zero). Next: the client-side write watch - find the CPU +pointer the guest driver writes GPFIFO entries through, its PA via the +guest pagemap, and why neither the GPU page table target nor any shadow +holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on +the driver's ring-write helper). + +## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page + +**The write-watch toolchain (all three instrument classes).** (1) kprobes: +`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE +and SEC2 submission paths per the open driver sources) is marked +`notrace` - `trace_kprobe: Could not probe notrace function +channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach +(this is why the named helpers "never fired" in earlier sessions: they +could never register). (2) QEMU-gdbstub hardware breakpoints work: a +verification `hbreak *__x64_sys_ioctl` fired on guest ioctls +(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store +(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx = +pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and +the source) stayed silent across a complete Vulkan run - even though that +same run produced the CE channel's notifier clear +(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the +first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ... +scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring +VA resolves through the guest page tables to a guest-RAM page that is +mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA +`0x101021000` -> client VA `0x7fffee69b000`; a later run PA +`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's +own read agree the page is ZERO. An in-client gdb (the t122 recipe plus +a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000` +from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET` +logged before `GLBASE`) caught ZERO writes across the whole run +(3 frames, `VKFRAME_DONE`, IDLE). + +**Consequences.** On vk-first boots the client creates its CE channel but +never submits CE work: no GPFIFO entry is written by the kernel +(`channelFillGpFifo` never executes), nor by the client (hardware watch, +zero hits), nor is any content found in any BAR shadow or in RAM. The +historical "Vulkan CE ring latches" were the dead CUDA client's stale +blocks (section 15). The image/upload path for the Vulkan client rides +its GR channels (`launch captured ... inline=8`; the frame copy held as +`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and +the non-black gate must be attacked on that path - the data/code pages +those launches depend on are never filled because no CE i2m copies are +ever queued, and the launch is held on their absence (dstpend/n_data). +Next: instrument the GR launch's dependency resolution (which pages the +dstpend gate waits for, and whether the inline records the capture holds +are sufficient to fill them) instead of hunting a CE ring that nothing +writes. |