diff options
31 files changed, 7287 insertions, 153 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 2eb02d61..91cb1c8 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,37 +1,59 @@ # Open-IOV Current State -> **Snapshot: 2026-10-24 ~20:40 (the host-side operand-resolution gap -> is closed and the shadow anchor is content-gated — r635 tracks REAL -> written content per shadow page (`r->content` on every landed write) -> and uses it as the operand anchor: a client's own H2D bytes that took -> the shadow lane anchor their span and arm a launch-level content -> anchor, so an answerless output candidate (the buffer the client never -> wrote) is served from a bounded synthetic page set (1024/candidate, -> 4096/launch, three-quarter-table guard) instead of failing the launch -> closed; trusted pin/walk/ledger answers keep the r633 budgets and no -> longer arm the cross mint, and a synthetic page can never re-anchor. -> The first draft of the change self-fed on its own minted pages and -> filled the shared shadow table on a green boot (6144/8192 measured, -> later clients crawled behind it); the content gate restored zero -> minted pages on a green boot and the historical green shape. Fresh -> r636 boot battery: 9 OK / 2 clean FAIL / 1 HANG — the best pass count -> repeated with strictly better failure modes: streams_events converted -> its historical hang into a clean client-visible `CUDA_ERROR` via the -> auto-fault, and streams2's red chain is now measured end-to-end (its -> H2D destinations are replay-denied, the GR forward defers on the CE -> stream-order watermark, the launch is refused as degenerate and the -> refusal faults cleanly; `STREAMS n=32768 bad=65534`). Remaining: the -> streams2 CE-order chain (a copy-pipeline class, not the fill), the -> first-wait crawl class (smiley/memset2d spin in libcuda before -> submission), used-boot allocation exhaustion (roundtrip/inplace_scale -> `CUDA_ERROR out of memory` after thirteen clients), then 15/15 + soak. -> GL runs on the guest GPU (next wall = the FBO class); the Looking -> Glass frame path is proven end-to-end (guest CUDA render -> KVMFR +> **Snapshot: 2026-10-26 (SESSION 143, post-reboot).** The r800 walk guard +> holds and the `gsp_phys_gr_min_data=0` bypass reaches the real engine: the +> guest Vulkan compute launch is now *submitted* (no oops, no degenerate-set +> refusal), the engine fetches the host-deposited QMD and the firmware reports +> a real `MMU Fault ... @ 0xdf_d8feb000, FAULT_PDE` at the guest program VA - +> the channel/context are alive, the program page is empty and unmapped. The +> util-sweep channel-visibility work (per-channel begin/skip/read=none lines) +> proves the Vulkan copy-class channels (0xbeef0101/0xbeef0102, engines 11/12) +> are registered and swept but never pushed, and that no channel carries +> program bytes or a `LAUNCH_DMA`: the code page (0x62b000 / rotating 0xc8b000) +> never fills. The frontier is therefore (1) the driver's staging-ring -> +> code-heap upload copy, which is not among the decoded pushes, and (2) the +> engine page-table mapping for the program VA. Separately, the refusal path no +> longer reaches the earlier three-black-frame baseline (`SYNTH-FAULT` vs the +> software completion/fence ordering), which must be restored before further +> observation. Docs: `docs/04-implementation/04-real-render-walls.md` sections +> 20-21. Commits: `34e611f4`, `c5510744`, `28def33e`, `0ca8ff9f`. +> +> **Snapshot: 2026-11-06 ~01:00 (the guest's REAL Vulkan pipeline now +> renders on the emulated vGPU: the first `VKFRAME seq=1..3` / +> `VKFRAME_DONE frames=3 640x480 saved=3` / `VKFRAME_IDLE ok` runs +> end-to-end — instance, device, allocation, image/view/buffer/shader/ +> descriptor/pipeline/command-buffer/fence, queue submit, fence release +> and frame readback all complete with a zero-touch service pipeline. +> The chain of fixes behind it: the push-stream parser now accepts the +> driver's real shapes (the `0x0001fff0` SET_SUBDEVICE_MASK bracket and +> zero NOP gaps — the class that silently skipped every large push), the +> fence fold (`sema_gr_ahead`) sweeps the pre-built GPFIFO continuation +> beyond GP_PUT and applies the submit's INC-5 RELEASE groups +> forward-only, the host-triggered VAS page-directory anchoring verifies +> the parked `COPY_SERVER_PDES` candidate against a content-located ring +> pair so the walk resolves every push VA before its first read, the +> COMPR+2MB VidHeap class is downgraded at the ioctl so +> `vkAllocateMemory` succeeds, the SEC2 scrubber's finish payload is +> authored by the service, the `0x9096` VidHeap enumeration and the +> `FIFO_GET_ENGINE_CONTEXT_PROPERTIES` ZCULL geometry are answered, and +> the C5C0-section -> vendor-launch translator found the real dispatch +> (a `SET_INLINE_QMD_ADDRESS_B` + 64-word run building QMD `0x4b85f00`, +> grid 40x30, plus the C5B5 copy-out). The frames are still BLACK: the +> driver delivers SASS/descriptors through its MME program + inline-to- +> memory stream (decoded in full; the payload rides the next GPFIFO +> entry as a source-view call), and the utility/copy-channel +> `LOAD_INLINE_DATA` arm that must materialize those writes is the +> current frontier (arbitration + `sema-utildec` witness landed). CUDA +> meanwhile: the best fresh-boot battery is 8 pass / 3 fail / 1 hang +> (the Vulkan work does not regress CUDA; always battery-first on a +> fresh boot); GL runs on the guest GPU (next wall = the FBO class); the +> Looking Glass frame path is proven end-to-end (guest render -> KVMFR > ivshmem -> host PNG frame); two guests with 512M + 1GiB emulated VRAM > coexist at boot level; the 0.9.9 release-preview demo video v1 is -> recorded and synced to files.vgpu.io. Frontier = 15/15 + multi-round -> soak, events A/B, GZDOOM (GL/Vulkan), two-guest concurrent CUDA, -> longevity/video v2, Windows guests).** +> synced to files.vgpu.io (the updated video with GZDOOM working is the +> next publish). Frontier = non-black frames, then Xorg -> GZDOOM +> (GL/Vulkan) -> KVMFR/LG frames -> CUDA 15/15 + soak -> the perfect +> 1.0.0 video -> Windows guests).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md index f6ea39d..f2ba0ab 100644 --- a/docs/04-implementation/04-real-render-walls.md +++ b/docs/04-implementation/04-real-render-walls.md @@ -185,3 +185,725 @@ gdb leaves the VM stopped (recover with a detach-only connect); `stepi` loops over the stub freeze the guest's userspace for minutes - prefer breakpoints; every probe run with the caps-fill breakpoint changes the driver's path, so keep the fill in each experiment. + +## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD) + +**The fence wall is closed.** The client's per-frame fence wait polls a +16 KB BO word that a submit-push INC-5 semaphore group targets +(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a +GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring +already carries the entries it pre-built for the next submits, idx12..46), +so the GP_PUT-bounded pump walk never decoded it and a host-side assist had +to write the word. The service now sweeps that pre-built continuation +itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the +ring's validated translation, accepts only pushes that pass the strict +method-stream gate, extracts the addr-0x5c RELEASE groups and applies them +forward-only through the standard pending ledger. Live evidence: the sweep +finds the groups and the ledger applies the fence word once per frame; +`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still +black - the shader work itself is unexecuted. + +**The compute-dispatch shape (why nothing executes).** The workload is a +single compute dispatch (640x480 storage image). Its push is a DIRECT +method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0` +on subchannel 1, then the context windows (`0x7b4` local-memory window B, +`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and +the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data +port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO +inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture +that feeds the physical-engine forward path never fires - and the CE-shaped +ops path is not even consulted for GR-class channels by construction. Real +pixels therefore require a translator that recognizes the C5C0 section, +builds the vendor launch the physical channel expects (the same prelude/QMD +shape the CUDA path already executes) from the section's fields and data +runs, and forwards it with the existing operand/relocation machinery. The +method vocabulary overlaps the physical prelude's existing knowledge +(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. + +## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale + +**The live ring is in guest RAM and the sweep's decode was right.** +`walk_probe 0x4020000` (the client's own page tables, vas[0] +`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA +to **guest RAM** (PA `0x11623d000` on r759; the page moves per run: +`0x104b55000` on r761) and its content is the 14-entry ring the sweep +decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 -> +the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot +`0x4250110`, ... - all pushes inside THIS client's own allocation window +(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own +VIDHEAP log). The decoded ring is live content, not a stale page. + +**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759 +the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`; +the userd-derived candidate loop then tried `0xb0000 - 0x10000 = +0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push +pattern) passed the plausibility gate and whose USERD probe hit the real +0xb0000 block, so the pin landed. The pinned entry's push reads all-zero +through every source; the pump retried 39 times and logged `skipping +stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`, +while `sema_probe 0x4020000` then answered the wrong page +(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep +included). Section 10's "stale session page" conclusion was this pin, not +the sweep. + +**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` + +`svc_ring_walk_live`: a candidate whose entry0 push provably reads +all-zero through the client's page tables is rejected when the VAS walk +resolves the ring VA to a page with a live entry (applied at the +userd-derived, FB-convention and existing-pin paths; the walk cannot vote +for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates +and accepts the channel's known USERD cross-aperture (the driver places +GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3) +the util sweep arbitrates a dead channel-path ring entry against the +direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and +their dedup keys are per-client now (reset in `svc_session_flush`); (5) +`sema-copyent` logs once per copy OP, not per row - one 480-row frame +copy had exhausted the 128/session budget and hidden every later copy. +Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0 +pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100 +va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the +fence fold releases fire, and the client completes (`VKFRAME_DONE +frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`). + +**The two utility-channel LID uploads are the driver's zero buffers by +design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256 +lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 -> +`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method +buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`; +`0x4250110`) read zero for the whole client run - the driver never +writes them. The GR channel's own LID entry DOES carry a real payload +(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param= +0x4b86000 size=0x1900`), which proves the arm works with real bytes on +this boot too. + +**Where the code page stands.** `prog=0xdfd8feb900` (the captured +launch): `sema_probe` answered vidmem `0x62b900` early in the run while +`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer +mapping ROTATES (a code pool), and the shadow's learned lat entry for the +VA can go stale. Both pages read zero in the shadow: the driver's code +upload has not been observed on any decoded path (no LID, no CE copy - +the only `sema-copyent` op of the whole run is the 480-row frame +copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0 +bytes=0x12c000` and correctly copies the still-empty image). The +remaining question is the SASS delivery shape for the C5C0 compute path +(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale +lat preference for the rotating code VA. + +**Next steps (ranked).** +1. Resolve the code VA through the VAS walk at launch time (the walk is + the live truth; the lat entry goes stale as the pool rotates) and + watch the code page the QMD actually names. +2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs + with a bounded methods witness while the client is live (the params + arrive by LID; the code must arrive by a sibling path). +3. Then the `n_data=0` descriptor/operand wiring (section 8). + +## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running + +**The delivery shape (decoded from the CE channel).** A CE-class channel +(eng=11) carries, as entry[4] of its ring, a push whose method stream is a +LAUNCH_DMA copy with the SASS bytes INLINE: + +``` +sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500 +sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8 +sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1 +sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0 +``` + +The source is the push itself (`in=0x120a00004`, right after the method +header) and the destination is a vidmem VA (`out=0x205000`, then +`0x206000`, ... 4096 B per run) - the engine-side "download program +range" path. The Vulkan client's CE channels carry the SAME shaped +entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`, +19 entries swept) - so the SASS delivery is a CE inline copy, not the +utility-channel LID entries (those are the driver's zero buffers, +section 12). + +**Why the copies never ran (three gates).** (1) The CE channel's ring and +USERD live in VIDMEM and resolve only by boot-lottery: the FB convention +matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> +fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the +frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan +instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768), +so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs +(0x120000000+) need the client's VAS registered: the parked +COPY_SERVER_PDES candidate only registered at +15 s (age-out, +`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`, +8000 ms from the cycle) expired first - r763 logged `skipping stale +entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and +`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy +pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s +(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms +parked`) but the ring gate still blocks. (3) The shadow ring scan +fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited +retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no +candidate on r768: the ring/USERD content is volatile at scan instants +(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a +`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan +misses the window. + +**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its +own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already +does this) instead of scanning for ring entries: for every USERD-shaped +block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0 ++ the probe. 2. Or latch the ring at the first CE submission (the ring +content appears exactly when the driver writes entry0). 3. With the ring +latched and the VAS at +5 s, verify the inline copies execute +(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code +page gets content; then the `n_data` descriptor wiring (section 8). + +## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring + +**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in +the notifier-completion path for CE channels *before* the clear write - the +one instant the ring content and the USERD are both live - reads entry0 +through the channel translation and the trusted walk, then scans the shadow +(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB +aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs +the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating +candidate is pinned/latched exactly like the regular CE autopin. `928c0730` +fixed the entry decode: the push VA is 40 bits, the scan had declared it +`u32`, so `scan_hits` was zero by construction. + +**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.** +The scan started at offset 0 and accepted the first ring-shaped page for +*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d` +(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's +ring/page, its first release decoded to the wrong semaphore word (+0 +instead of +4) and was dropped as stale (`cur=1`, already written by +rank 0), so the client's rank-1 tracker wait never completed. victory_mail +then spun at 100% CPU with `pub_calls=0` on three consecutive boots +(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era +module (`25523f84`) on a fresh boot passed the same client (VADD +`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the +SESSION 133/134 code. The fix tries the rank's convention offset +(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and +never accepts an offset another CE rank already latched this session; an +unlatched rank stays for the regular per-pass latch. Verified on r781: +ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/ +0x120000, USERD +0x10000 each), rank 1's own semaphore word applied +(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed +(`pub_calls=185132+`, no spin). + +**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0): +`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0 +va=0x120000000 len=40)` - a live entry - but the client's pushes decoded +`unread=1 injected=0`: the guest pin scanner covered CUDA clients +(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk` +engaged the tools (demand pins active) but r782/r783's captures reported +`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first +64 MB, and no classic autopin either. + +**Where the Vulkan rings actually are.** The client's own vidheap ioctl +log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations: +`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos +(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's +`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and +of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image +for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000` +sibling) finds nothing - the ring content reaches neither the shadow at its +allocation offset nor guest RAM. + +**The frame chain (r783, live).** The Vulkan client completes three frames +(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black. +The live GR channel (`h=0xbeec0100`) shows the dependency explicitly: +`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together +with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000 +dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk +maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0 +pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the +framebuffer. The shadow at `0x570000` reads zero: the data pages are the +ones the CE inline copies are supposed to fill, and the GR launch is held +on their absence (the designed `dstpend` dependency). So the whole Vulkan +chain reduces once more to the un-latched CE ring: no ring -> no inline +i2m copies -> no data/code pages -> held launches -> black frame. + +**Next steps (ranked).** 1. Cover the client's real allocation in the ring +capture: raise the capture scan bound above the recorded channel +allocations (e.g. `0x6000000`) and/or seed candidates from the client's +vidheap offsets, then re-check whether the entry arrives at the +submission instant (a live `sema_vidmem_dump` at `0x4100000` while the +client is streaming would settle it; the post-run dumps read zero). 2. If +the shadow still reads zero at the allocation offset during live +submissions, the client-side write watch is the remaining forensic: find +the CPU pointer the driver writes GPFIFO entries through, its PA via the +guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a +latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd` +with the code dsts), the code/data pages fill, then the `n_data` +descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA +battery on the fixed module (the r781 boot was a slow-crawl boot; the +spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel +forward timeout. + +## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target + +**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).** + +*The write path works; the driver does not use it for the ring.* A direct +BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through +`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated +vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest +and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct +into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A +high-frequency watcher (12 candidate offsets, ~2.5 Hz, including +`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000 +USERDs) recorded ZERO nonzero dumps across the whole live streaming window +of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`, +`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR +shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the +ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of +the full 4 GB guest-RAM image for both patterns (little-endian, corrected +from an earlier reversed search) finds nothing either. The walk and the +learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page +(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap +ioctl log (`PREV149-OUT`) names the channel allocations +`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is +written neither to the shadow at its allocation offsets nor anywhere in +RAM reachable through the guest's own page tables. + +*A stale-block trap explains every historical "latch".* On CUDA-first +boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan +channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012` +block surviving in VRAM after its session. The decoded entry +(`va=0x120000000 len=40`) is the old session's (both clients' first CE +pushes are shape-identical), the expectations seed from the stale pool +words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan +client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on +every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000" +rows are the same artefact: both were CUDA-first boots. All genuine +first-submission latches on record read `put=1 get=0` at the capture +instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`) +in the first-submission capture - a just-submitted entry cannot be +consumed at the notifier-clear instant. + +*The frontier is now precise.* The Vulkan client's CE submissions complete +only through the fabricated completion machinery (the driver never reads +its ring back), its CE ring content is invisible to every host source, and +the GR launch for each frame is held on the unexecuted CE i2m copies +(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem +`0x570000`, zero). Next: the client-side write watch - find the CPU +pointer the guest driver writes GPFIFO entries through, its PA via the +guest pagemap, and why neither the GPU page table target nor any shadow +holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on +the driver's ring-write helper). + +## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page + +**The write-watch toolchain (all three instrument classes).** (1) kprobes: +`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE +and SEC2 submission paths per the open driver sources) is marked +`notrace` - `trace_kprobe: Could not probe notrace function +channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach +(this is why the named helpers "never fired" in earlier sessions: they +could never register). (2) QEMU-gdbstub hardware breakpoints work: a +verification `hbreak *__x64_sys_ioctl` fired on guest ioctls +(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store +(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx = +pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and +the source) stayed silent across a complete Vulkan run - even though that +same run produced the CE channel's notifier clear +(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the +first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ... +scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring +VA resolves through the guest page tables to a guest-RAM page that is +mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA +`0x101021000` -> client VA `0x7fffee69b000`; a later run PA +`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's +own read agree the page is ZERO. An in-client gdb (the t122 recipe plus +a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000` +from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET` +logged before `GLBASE`) caught ZERO writes across the whole run +(3 frames, `VKFRAME_DONE`, IDLE). + +**Consequences.** On vk-first boots the client creates its CE channel but +never submits CE work: no GPFIFO entry is written by the kernel +(`channelFillGpFifo` never executes), nor by the client (hardware watch, +zero hits), nor is any content found in any BAR shadow or in RAM. The +historical "Vulkan CE ring latches" were the dead CUDA client's stale +blocks (section 15). The image/upload path for the Vulkan client rides +its GR channels (`launch captured ... inline=8`; the frame copy held as +`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and +the non-black gate must be attacked on that path - the data/code pages +those launches depend on are never filled because no CE i2m copies are +ever queued, and the launch is held on their absence (dstpend/n_data). +Next: instrument the GR launch's dependency resolution (which pages the +dstpend gate waits for, and whether the inline records the capture holds +are sufficient to fill them) instead of hunting a CE ring that nothing +writes. + +## 17. UPDATE 2026-10-26 (SESSION 138): THE GR LAUNCH'S WIRING IS PRINTED - one QMD-window page carries prog+pointer data (deduped and never scanned), its only fill candidate is an unresolvable inline word, and the backend refuses `n_data=0 (min 16)` + +**The witness (commit `10b09217`, sema_dbg-gated).** Per launch, bounded: +the candidate handle, `param_va`, `prog_va`, `n_pointer` + QMD pointer +pairs, inline count, and the resolved `gr_pages` list; then the fill's +built candidate set with the arg flag. + +**First capture (r788, Vulkan `h=0xbeec0100`, push=0x4f4509c).** +``` +sema-gr: cand h=0xbeec0100 param=0x4b86000 prog=0xdff0feb900 n_ptr=1 inline=8 n_pages=2 push=0x4f4509c +sema-gr: ptr[0] va=0xdff0febc00 word=34 +sema-gr: page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 +sema-gr: page[1] va=0xdff0feb000 pa=0x56b000 ap=2 qmd_word=48 +sema-gr: fill cand n=1 pages=2 push=0x4f4509c +sema-gr: win[0] va=0x43f00000 arg=1 +``` +Reading: the launch's page list is the param page and the +prog/QMD-window page (both resolving as `ap=2`, i.e. vidmem). +`ptr[0]` (QMD word 34, `0xdff0febc00`) shares its 4K page with the program +page, so `svc_gr_page_note` dedupes it into the `qmd_word=48` entry - and +the fill's scan passes cover only `qmd_word==32` and `32..47 even`, so +that page's content is never scanned for data pointers. The fill's single +candidate comes from the inline bytes instead (`0x43f00000`, arg=1) and +resolves in **no** vaspace (`svc-gr arg-miss base=0x43f00000 probe=none +... vas[0..3] rc=-2`), so `n_data` stays 0 and the backend refuses: +`vgsp-gr: GR submit refused: degenerate data set n_data=0 (min 16) +launch_va=0x4f4509c`. + +**Content state.** The shadow at the param page (`0x386000`), the arg +region (`0x386150`), the program area (`0x56b900`) and the pointer/data +area (`0x56bc00/0x56bd00`) reads all zero on this boot - the launch's +data and code bytes exist in no host-visible source (consistent with +section 16: nothing queues the CE i2m copies for the Vulkan client). + +**Next.** (1) Scan the deduped QMD-window page (or scan a page once under +every qmd_word that names it) so the pointer-region data words enter the +candidate set; (2) resolve the inline/dst candidates through the vidmem +serving lane (the pages ARE resolved to vidmem PAs - the missing piece is +the candidate list, not the resolver); (3) drive `n_data` toward the +launch's `min 16` (the window's expected page count) - and where the +window content must come from is the open question the byte-state above +poses. + +## 18. UPDATE 2026-10-26 (SESSION 139): THE FILL'S CANDIDATE SOURCE IS DISPROVEN - `0x43f00000` IS THE FLOAT 480.0f; after the float-shape filter the candidate set is EMPTY and the wall is content, not wiring + +**The false candidate (commit `add22329`).** A full dump of the launch's +param page (vidmem `0x386000`, r789) holds exactly two nonzero spots: +`+0x30: 08 00 00 00` and `+0x630: cd cc 4c 3e | 00 00 20 44 | 00 00 f0 +43 | 00 00 00 00` - i.e. the f32 scalars **0.2f, 640.0f, 480.0f** (the +640x480 viewport!). `0x43f00000` is 480.0f: the fill's only candidate +("arg base=0x43f00000 resolved 0 pages") has been a **float scalar +misread as a device VA** by the scan's shape filter (`>= 0x10000000`, +4-aligned). The GRCE inline write that does fire (`dst=0x4b86020 -> +pa=0x386020 bytes=1568 data8=0000000000000000`) writes the driver's +zero stub (the push carries only an 8-byte nonzero inline fragment at +`+0xb0`; the rest reads zero - the SESSION 130 zero-buffer finding). + +**The fix and verification.** All four scan sites (the three page scans +plus `svc_gr_scan_cand`) now reject float-shaped words (f32 exponent byte +in ~`[0x6c..0x8f]` = 2^-19..2^16; device VAs as f32 land in subnormal +exponents), and the bank-page scan covers word 48 (the deduped program +page). r790: **`sema-gr: fill cand n=0`** - no false candidates, no false +demand-loop traffic; the launch still refuses (`GR submit refused: +degenerate data set n_data=0 (min 16)`). + +**The wall, precisely.** The launch's real data pointers are not in any +scanned source: the param page holds scalars; the QMD pointer word +(`0xdff0febc00`) is one window buffer whose content is never written; the +inline payload is the zero stub. The content could only arrive via the +driver's i2m/LOAD_INLINE_DATA copies (which the Vulkan client never +queues - SESSION 137) or from the real GPU (physical forward, which the +n_data gate refuses first). This is now a pure content-source problem: +either find/queue the i2m that fills the QMD window, or break the +chicken-and-egg by forwarding a launch with the window zero (letting the +real engine read it) behind an explicit parameter. + +## 19. UPDATE 2026-10-26 (SESSION 140): THE n_data WIRING - the QMD's own pointer values ARE the launch's data set; the fill now reaches them (candidates appear, the page is correctly launch-internal), and the degenerate-set guard's bypass reaches the backend only to oops in the empty-set walk + +**Three wiring corrections (commit `5c0fcc21`).** (1) The QMD page itself +is noted into the request page list (word 34) - the pointer values live +inside the QMD (r790: word 34 = `0xdff0febc00`) and no scan ever covered +it. (2) The fill seeds candidates directly from the launch's own QMD +pointer pairs (`gr_qmd` words 34..47, decoded from the push's inline-QMD +run): r793 `sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)` - the values +need no page scan (the page shadows read empty; the +invisible-vidmem-write class). (3) The scans read pin-aware +(`svc_read_va` first) - the staging copy the decode reads is a pinned +page; a raw mapping of the vidmem pa sees zeros. + +**Result.** `fill cand n=1` with the QMD window page seeded; +`sema_probe 0xdff0feb000 -> pa=0x56b000 aperture=vidmem` resolves - but +the R3 hygiene filter classifies the page launch-internal, so `n_data` +stays 0: **this launch's QMD declares exactly one pointer pair and it +names its own QMD window - a launch with no external data, which the +"degenerate set" refusal class wrongly covers.** + +**The egg-break test + the oops.** With the r564 guard bypassed +(`gsp_phys_gr_min_data=0` - and the value clamp removed for the test -) +the refusal disappears and the forward ENTERS the backend: +`Oops: 0000 [#2] PREEMPT SMP NOPTI`, `Workqueue: gsp-phys-work +svc_fwd_work_fn [vgsp]`, `RIP: gsp_phys_backend_guest_gr_submit+0x4394` +(NULL-base table walk: CR2=0x1e93c, R14=0, RAX=0x1e000, RSI=0xdff0feb900 +- the program VA). The empty-data walk needs that guard before +`min_data=0` can be used; the clamp revert was withdrawn (crash +enabler). The copy pipeline is unaffected: `sema-gr: copy done +h=0xbeec0100 push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0` +runs cleanly in the same runs. + +**Next.** Guard the empty-set walk at `gsp_phys_backend_guest_gr_submit` +(the struct walk reading 0x04/0x10/0x38/0x3c off a NULL base with the +program VA in RSI), then re-test `min_data=0` for the no-external-data +launch class: the forward should reach the real engine, and the copy-out +(and the frame copy that already executes) can carry rendered pixels +back. + +## 20. UPDATE 2026-10-26 (SESSION 143): THE r800 GUARD HOLDS AND THE `min_data=0` BYPASS REACHES THE REAL ENGINE - IT FETCHES THE PROGRAM AND MMU-FAULTS WITH `FAULT_PDE` AT THE PROGRAM VA; THE WALL IS NOW THE PROGRAM PAGE'S MAPPING AND CONTENT, NOT THE DATA-SET GATE + +**The test (r801b).** `gsp_phys_gr_min_data=0` is now an explicit bypass +(commit `34e611f4`; 0 previously clamped to 1). On a clean cycle +(module from `34e611f4`, full params, `sema_gr_xlate=1`, +`sema_gr_util_scan=0`, `sema_vas_cand_ms=5000`, `sema_ce_grace_ms=20000`, +`sema_ce_hold_ms=0`, `sema_ops_log=8`, `sema_dbg=Y`, `min_data=0`) the +Vulkan client ran once: + +* **No oops.** `grep -c Oops` = 0 for the whole run; the r794 + `gsp_phys_backend_guest_gr_submit+0x4394` crash class is gone. +* **The refusal is gone.** Zero `GR submit refused: degenerate data set` + lines; the launch is submitted: + `GR LAUNCH submitted h=0xbeec0100 ... pages=3 shadow=1 + qmd_va=0x4b85f00 slot=3 words=239 ... qmdwin=0x108000`. +* **The engine ran and fetched.** After ~2.5 s: + `GR LAUNCH FAILED: timeout after 2500 ms (qmd_rel=0x00000000 + want=0x600df00d ...)`, and the real firmware reported: + `vgsp-fault: func=0x1006 ... errString="Ch 00000001, intr 00000000. + MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_GCC faulted @ 0xdf_d8feb000. + Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ"`. +* **The copy-out still works:** `sema-gr: copy done h=0xbeec0100 + push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0`. +* **The client now fails earlier than the black-frame runs:** it reports + `VKDEVICE ok family=0` then `VKFAIL vkWaitForFences ... -> 2` and no + frame is saved (the service synthesizes a client-visible fault for the + failed forward, where the refusal path used to software-complete and + let three black frames through). + +**The launch wiring measured on this boot.** +``` +page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 (param) +page[1] va=0xdfd8feb000 pa=0x62b000 ap=2 qmd_word=48 (program) +page[2] va=0x4b85000 pa=0x385000 ap=2 qmd_word=34 (QMD window bank) +``` +The program page resolves in the emulated vidmem at `0x62b000`, whose +backing host page (`0x5e0504000`) reads ZERO; `CODEHASH ... h0=0 ... +SUSPECT (first 16B)` and `FAULT-CAP ... prog_w48=0xd8feb900 +prog_pa=0x5e0504000 code_src=guest-hpa off=0x900 code64=000...` confirm +the source is empty. The QMD deposit keeps the guest program VA +(`w48/49=d8feb900 000000df`); in the retry only the `qmd_word 34` page is +copied to the window-0 shadow (`qmd_word 34 page copied to window-0 +shadow VA=0x1e8000`), so the program pointer is left at the guest VA and +the engine's page tables have no PDE for it. The post-mortem program +target reads `slot=30 ... w0..15 = 0`. + +**Reading.** The data-set gate and the r800 guard are settled: the +no-external-data launch is legitimately forwarded and the engine's +channel/context are alive (it fetched the QMD from the host deposit and +issued a real MMU fault). The remaining wall is exactly two coupled +facts: +1. **Mapping** - the engine's page tables do not map the guest program VA + (`FAULT_PDE`), and the host keeps that VA because the code source is + an empty page (the shadow rewrite is only used for sources it can + trust). +2. **Content** - the program page is empty in every host-visible source. + The driver's program-upload path stages the ucode and copies it to the + code heap (the observed CE `LAUNCH_DMA`/inline shape); that copy has + not been observed executing for the Vulkan client. + +**Next.** (1) Find and execute the Vulkan client's program-upload copy +(the staging->code-heap CE copy) so the program page becomes non-zero - +the utility/GR sweeps are the instruments, and the code page becoming +non-zero is the milestone. (2) Then map/relocate the program VA for the +engine (or install the PDE) so the fetch succeeds. (3) Then non-black +frames -> Xorg/GZDOOM -> KVMFR/LG -> CUDA 15/15 + soak -> the 1.0.0 +video -> Windows. Until (1)/(2) land, keep `min_data` at its default 16: +the bypass forwards a launch the engine cannot complete, which is worse +for the client than the software-completion fallback. + +## 21. UPDATE 2026-10-26 (SESSION 143b): THE COPY-CLASS CHANNELS ARE REGISTERED, SWEPT AND NEVER PUSHED; NO CHANNEL CARRIES PROGRAM BYTES OR A LAUNCH_DMA; THE CODE PAGE NEVER FILLS - AND THE REFUSAL PATH NO LONGER PRODUCES THE BLACK-FRAME BASELINE + +**The instrument (commit `c5510744`).** The util sweep now logs one-shot +`sema-utilsweep: begin|skip|read=none` for every channel plus a 128-row +per-pass cap, so "absent from the log" can no longer be confused with +"absent from the channel table". + +**Channel visibility (r803, full params, `util_scan=64`, `min_data=16`).** +Ten channels in the table, all with gpfifo/entries, **all USERD +unresolved** at first sight: +``` +begin h=0xbeef0101 eng=11 gpfifo=0x4100000 entries=32768 userd=0 +begin h=0xbeef0100 eng=0 gpfifo=0x4020000 entries=32768 userd=0 +begin h=0xbeef0102 eng=12 gpfifo=0x4160000 entries=32768 userd=0 +begin h=0xbeee0101 eng=11 gpfifo=0x4500000 ... +begin h=0xbeee0102 eng=12 gpfifo=0x4560000 ... +begin h=0xbeed0101 eng=12 gpfifo=0x49c0000 ... +begin h=0xbeee0100 eng=0 gpfifo=0x44a0000 ... +begin h=0xbeec0104 eng=0 gpfifo=0x4d20000 ... +begin h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ... +``` +The Vulkan client's copy-class channels **0xbeef0101 (eng 11) and +0xbeef0102 (eng 12) are in the table and swept** - and their rings carry +nothing all boot: `read=none` on first sight, `swept=0` on every pass, no +`utildec`, no LID, no copy. The client posted one event on 0xbeef0101 at +registration (`post_event hSrc=0xbeef0101`) and never pushed. + +**What did decode.** All boot: 3 LID writes only - the driver's two zero +buffers (256 B -> vidmem 0x423f000, 12288 B -> 0x4448000, both +`zero=1`) and the launch's own param bank (1568 B -> 0x4b86020, +`zero=0`). No `sema-lda`/`sema-opadd` rows at all, no `copy-in` rows, no +`utildec` on the two CE channels. Global swept=1652, writes=3. + +**The code page never fills.** `0x62b000` (the launch's program page for +`prog=0xdfd8feb900`) and the rotating-pool page `0xc8b000` read ALL-ZERO +before the client, after it, and at a final re-dump; the whole 256 MB +aperture holds only 12 non-zero dwords boot-wide (the viewport scalars +`0x44200000`/`0x43f00000` at 0x386634/0x386638, the releases, the +baseline). Nothing anywhere writes program bytes. + +**Client outcome.** `VKDEVICE ok family=0` then +`VKFAIL vkWaitForFences ... -> 2`; no frames, no PPMs. The service +answered `GR-COMPLETE-SW` 10 347 times for the same refused push while +the client kept waiting. This is the same outcome as the r801b forward +test, and it differs from the earlier black-frame baseline +(`VKFRAME seq=1..3`, three black PPMs): on a refused/failed forward the +service authors a client-visible `SYNTH-FAULT` (commit `c6c152cc`, +`sema_refusal_fault=1` since r618), and the current completion/release +path does not get the client past its fence. `util_scan` is excluded as +the cause (r801b failed with it off). + +**Reading.** The Vulkan client *believes* it uploaded its compute shader +(it dispatches with a valid program VA), but no CPU write, no CE LID, no +CE copy and no GR i2m in any observed channel ever delivers the program +bytes to the emulated vidmem. The driver's upload path is a +`staging ring -> code heap` copy (CPU memcpy into a staging BO, then a +GPU copy to the code heap); the copy is not among the decoded pushes and +the staging BO is not visible in the emulated vidmem. Two separate +questions now stand: +1. **Delivery** - where the driver queues that copy (a CE channel it + never pushes, a GR subchannel/method shape the parser does not + decode, or a sysmem staging BO that a different read path would + expose). +2. **Completion regression** - restore the refusal-path software + completion (or gate the synthetic fault) so the client reaches the + fence-released black-frame baseline again; this is needed for every + downstream observation. + +**Next.** (a) Dump and decode the client's push method stream around the +upload (the `staging ring -> code heap` copy) rather than more ring +sweeps; the staging BO's CPU write is the most likely host-visible copy +of the SASS. (b) Instrument the completion path for one refused Vulkan +launch (release ordering vs the synth-fault) to restore the black-frame +baseline. (c) Only then resume the mapping work (the r801b `FAULT_PDE` at +the program VA). diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 6fae2f2..2f3202e 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -2634,6 +2634,18 @@ EXPORT_SYMBOL_GPL(gsp_phys_boot_rm_flags); static void *gsp_phys_chunk_page_va(const struct gsp_phys_boot_chunk *c, u32 page) { + /* + * SESSION 141: an unallocated chunk (no per-page array AND no base + * va) must answer NULL. It previously returned + * (u8 *)NULL + page*PAGE_SIZE - a small non-NULL garbage pointer - + * so the caller's NULL guard passed and the empty-set post-mortem + * print dereferenced it: Oops #2 RIP + * gsp_phys_backend_guest_gr_submit+0x4394, CR2=0x1e93c with slot 30 + * (0x1e000 = 30*0x1000 + 0x900 off). min_data=0's no-external-data + * forward hit this on its first pass. + */ + if (!c->page_va && !c->va) + return NULL; if (c->page_va) return c->page_va[page]; return (u8 *)c->va + (size_t)page * GSP_PHYS_PAGE_SIZE; @@ -17678,8 +17690,40 @@ static u32 gsp_phys_gr_data_bi(u64 reloc_va) * launches resolve 968-969 records (the client's whole arena); a launch * with only a handful after the fill's drops has an unresolved argument * set and must not be handed to the engine (see the submit guard). + * + * SESSION 127: the Vulkan inline-QMD launch (the C5C0 translator) is a + * compute dispatch whose operand set is one descriptor page and one + * parameter bank - a handful of records is healthy for it. Tunable module + * parameter (default 16 preserves the CUDA-era threshold); the translator + * cycles pass gsp_phys_gr_min_data=2. */ #define GSP_PHYS_GR_MIN_DATA_RECS 16u +static unsigned int gsp_phys_gr_min_data_param = GSP_PHYS_GR_MIN_DATA_RECS; +module_param_named(gsp_phys_gr_min_data, gsp_phys_gr_min_data_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_min_data, + "GR submit minimum data records (degenerate-set refusal threshold, 0..4096; 0 = explicit bypass for the no-external-data launch class; default 16)"); + +/* + * r801 (SESSION 143): 0 is an explicit BYPASS, not a clamp to 1. The + * Vulkan launch's single QMD pointer pair names its own QMD window and its + * params are scalars, so the launch has no external data by construction; + * the bare n_data>=min gate wrongly classifies it degenerate. The bypass is + * safe now that gsp_phys_chunk_page_va answers NULL for an unallocated + * chunk (r800): before that guard the empty-set post-mortem print + * dereferenced a garbage pointer and oopsed at + * gsp_phys_backend_guest_gr_submit+0x4394. The proper predicate (all + * candidates launch-internal-filtered, none unresolved => not degenerate) + * replaces the bare threshold next; until then 0 stays an explicit, + * opt-in debug/test value and the default remains 16. + */ +static unsigned int gsp_phys_gr_min_data_value(void) +{ + unsigned int v = READ_ONCE(gsp_phys_gr_min_data_param); + + if (v > 4096) + v = 4096; + return v; +} #define GSP_PHYS_GR_GAP_MAX 4096u /* total late pages per launch */ /* * relfix PART 3 (r217 span containment): the late path authors ONLY the @@ -19289,10 +19333,11 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * into the same software-completion fallback without touching the * engine, and cannot wedge it for the next client. */ - if (req->n_data < GSP_PHYS_GR_MIN_DATA_RECS) { + if (req->n_data < gsp_phys_gr_min_data_value()) { gsp_phys_gr_refused++; pr_err_ratelimited("vgsp-gr: GR submit refused: degenerate data set n_data=%u (min %u) launch_va=0x%llx\n", - req->n_data, GSP_PHYS_GR_MIN_DATA_RECS, + req->n_data, + gsp_phys_gr_min_data_value(), req->push_va); return -ENODATA; } diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c index b8a3e37..7b7e230 100644 --- a/kernel/gsp_rm_emulation.c +++ b/kernel/gsp_rm_emulation.c @@ -2665,6 +2665,20 @@ static void gsp_rm_handle_kgr_get_context_buffers_info( * L2/LTC topology from here. Echo the requested indices and fill in the * emulated TU104 values. */ +/* + * m113: FB_GET_INFO_V2 COMPRESSION_SIZE (index 0x01, bytes). The header: + * "A return value of 0 indicates the GPU does not support compression" - + * but the guest driver still attempts a COMPR-enabled, 64 KB-aligned + * comptag allocation (vendor VidHeap alloc, attr COMPR=2) and fails it + * NV_ERR_NO_MEMORY (0x51) when the comptag pool has no space, which + * libcuda maps to VK_ERROR_OUT_OF_DEVICE_MEMORY (the vkAllocateMemory -2 + * gate). Report a comptag pool sized like the real Turing parts (~fb/16). + * 0 = the historic "no compression" reply. + */ +static uint gsp_rm_fb_compr_mb = 128; +module_param_named(wall2_fb_compr_mb, gsp_rm_fb_compr_mb, uint, 0644); +MODULE_PARM_DESC(wall2_fb_compr_mb, "FB_GET_INFO_V2 COMPRESSION_SIZE in MiB (default 128, 0 = none)"); + static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) { u64 fb = nv_vgpu_fb_emul_size(ctx->vgpu); @@ -2673,7 +2687,8 @@ static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) case TU104_FB_INFO_IDX_TILE_REGION_COUNT: return 0; case TU104_FB_INFO_IDX_COMPRESSION_SIZE: - return 0; + /* m113: non-zero comptag pool (bytes) - see gsp_rm_fb_compr_mb. */ + return gsp_rm_fb_compr_mb << 20; case TU104_FB_INFO_IDX_PARTITION_COUNT: return 1; case TU104_FB_INFO_IDX_BAR1_SIZE: @@ -3037,8 +3052,21 @@ static bool gsp_rm_is_channel_class(u32 class_id) case 0x906f: /* MAXWELL_CHANNEL_GPFIFO_A */ case 0xa06f: /* KEPLER_CHANNEL_GPFIFO_A */ case 0xb06f: /* PASCAL_CHANNEL_GPFIFO_A */ - case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A */ + case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A (PRE-TURING NAME) */ + case 0xc36f: /* VOLTA_CHANNEL_GPFIFO_A */ + /* + * r828: the RC watchdog's channel allocates 0xc36f on this + * device (the driver's gpfifoMapping loop picks the first + * class the gpu reports supported) - without it in this set + * the channel never reaches the progress service, its class2d + * notifier cycle (SET_NOTIFY_A/B + NOTIFY) never executes and + * the guest's krcWatchdog declares the GPU locked after its + * 7 s timeout as soon as robust channels are in play. + */ case 0xc46f: /* TURING_CHANNEL_GPFIFO_A */ + case 0xc56f: /* AMPERE_CHANNEL_GPFIFO_A */ + case 0xc86f: /* HOPPER_CHANNEL_GPFIFO_A */ + case 0xc96f: /* BLACKWELL_CHANNEL_GPFIFO_A */ return true; default: return false; @@ -3060,8 +3088,18 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, { struct sema_channel_desc desc; - if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) + if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) { + /* + * r828 diag: a channel-class alloc whose params are shorter + * than the notifier field is not registered, silently. The + * RC watchdog channel (hObject 0x31415900) never appeared in + * the sema table; name every such skip with its size so the + * shape question is answered by a log, not inference. + */ + pr_info("vgsp-rm: channel alloc hObject=0x%x params_size=%u too short (< %u); NOT registered\n", + h_object, params_size, GSPRM_CHAN_NOTIFIER_MEM + 8); return; + } memset(&desc, 0, sizeof(desc)); desc.vgpu = ctx->vgpu; @@ -3082,6 +3120,20 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, */ if (desc.engine_type >= 1 && desc.engine_type <= 8) desc.engine_type = 0; + /* + * r835: the channel's own address space (NV_CHANNEL_ALLOC_PARAMS: + * hContextShare @24, hVASpace @28 - confirmed by this struct's own + * known offsets: engineType @128 = 32 + 8 handles + 8 subdevice + * handles + 8 u64 userd offsets). Two clients map different objects + * at overlapping VAs (everyone's RM window sits at 0x1200xxxxx), so + * the global VA-keyed translation maps conflate them: the X server's + * ring VA resolved through the GL client's page tables (walk-probe: + * vas h=0xcaf00005 answered 0x120090000 with a zero page) and its + * push stream decoded as empty, wedging X exactly like the RC + * watchdog's ring. The service resolves a channel's VAs through this + * VAS first. + */ + memcpy(&desc.h_vaspace, params + 28, 4); memcpy(&desc.cid, params + GSPRM_CHAN_CID, 4); memcpy(&desc.instance_pa, params + GSPRM_CHAN_INSTANCE_MEM, 8); memcpy(&desc.userd_pa, params + GSPRM_CHAN_USERD_MEM, 8); @@ -3505,6 +3557,41 @@ MODULE_PARM_DESC(wall2_a097_rec6, "0x2080a097 reply word at params+0x1c (0xfffff module_param_named(wall2_a097_rec7, gsp_rm_a097_rec7, uint, 0644); MODULE_PARM_DESC(wall2_a097_rec7, "0x2080a097 reply word at params+0x20 (0xffffffff = echo)"); +/* + * m112b: the NV04_VID_HEAP_CONTROL enumeration pair 0x90960106/0x90960107 + * (legacy NVOS32 VidHeap API, class 0x9096). libnvidia-eglcore (file offset + * 0x586d80) walks three heaps: per heap it queries 0x90960106 (12 bytes: + * {u32 firstIdx, u32 lastIdx}; the client then enumerates the heap's memory + * blocks by index via 0x90960107 (56 bytes) while firstIdx+1 <= idx <= + * lastIdx). The 56-byte block entry is read at +0x00/+0x08 (two u64s), +0x28 + * (a u32 type compared against a table at eglcore 0x101b1c0) and +0x30 (a + * flag BYTE: when 0 the heap/block is skipped). The generic zero reply makes + * every heap look empty, the client concludes out-of-device-memory and + * vkAllocateMemory fails -2 WITHOUT any alloc reaching the emulation + * (dmesg-corroborated: only 0x9096* controls + a class 0xc5c0 alloc). + * Runtime-tunable so one boot can sweep reply variants (same pattern as the + * wall2_a097_* words). + */ +static uint gsp_rm_9096_lo; /* 0x90960106 firstIdx */ +static uint gsp_rm_9096_hi = 1; /* 0x90960106 lastIdx */ +module_param_named(wall2_9096_lo, gsp_rm_9096_lo, uint, 0644); +MODULE_PARM_DESC(wall2_9096_lo, "0x90960106 reply firstIdx (default 0)"); +module_param_named(wall2_9096_hi, gsp_rm_9096_hi, uint, 0644); +MODULE_PARM_DESC(wall2_9096_hi, "0x90960106 reply lastIdx (default 1)"); +static uint gsp_rm_9096_ent_addr; /* 0x90960107 entry +0x00 (u64) */ +module_param_named(wall2_9096_ent_addr, gsp_rm_9096_ent_addr, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_addr, "0x90960107 entry word at +0x00 (default 0)"); +static uint gsp_rm_9096_ent_size = 0x80000000u; /* entry +0x08 (u64) */ +module_param_named(wall2_9096_ent_size, gsp_rm_9096_ent_size, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_size, "0x90960107 entry word at +0x08 (default 0x80000000)"); +static uint gsp_rm_9096_ent_type = 3; /* entry +0x28 (u32) */ +module_param_named(wall2_9096_ent_type, gsp_rm_9096_ent_type, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_type, "0x90960107 entry word at +0x28 (default 3)"); +static uint gsp_rm_9096_ent_flag = 1; /* entry +0x30 (u8) */ + +module_param_named(wall2_9096_ent_flag, gsp_rm_9096_ent_flag, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_flag, "0x90960107 entry flag byte at +0x30 (default 1)"); + /* * The live session's oracle client/object for the 0x20810107 experiment, @@ -4217,6 +4304,18 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, "vgsp-rm: promote[%u] bufferId=%u bInitialize=%u bNonmapped=%u physAttr=0x%x pa=0x%llx va=0x%llx size=0x%llx", i, bid, e[30], e[31], attr, pa, eva, esz); + /* + * r820: register the driver's own statement + * of this kernel buffer's backing (r817 + * Xorg GL init: the page-table walk answered + * a stale zero page for the pushbuffer VA + * while this entry named the real one; a + * userspace pin scanner can never cover + * kernel-owned buffers). See + * gsp_sema_service_promote_add(). + */ + gsp_sema_service_promote_add(eva, pa, esz, + attr, bid); } } out_status = 0; @@ -4348,6 +4447,47 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + case 0x90960106: { /* NV04_VID_HEAP_CONTROL block-index range (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 12) { + memcpy(rp, pp, req.paramsSize); + *(u32 *)(rp + 0) = gsp_rm_9096_lo; + *(u32 *)(rp + 4) = gsp_rm_9096_hi; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960106 -> first=%u last=%u", + gsp_rm_9096_lo, gsp_rm_9096_hi); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + + case 0x90960107: { /* NV04_VID_HEAP_CONTROL block entry by index (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 56) { + memcpy(rp, pp, req.paramsSize); + *(u64 *)(rp + 0x00) = (u64)gsp_rm_9096_ent_addr; + *(u64 *)(rp + 0x08) = (u64)gsp_rm_9096_ent_size; + *(u32 *)(rp + 0x28) = gsp_rm_9096_ent_type; + *(u8 *)(rp + 0x30) = (u8)gsp_rm_9096_ent_flag; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960107 -> addr=0x%x size=0x%x type=%u flag=%u", + gsp_rm_9096_ent_addr, + gsp_rm_9096_ent_size, + gsp_rm_9096_ent_type, + gsp_rm_9096_ent_flag); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + default: /* Unknown control command - return success with zeroed params. * Do NOT echo the request: pure queries pass uninitialized diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index 993e507..f6ad3b2 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -205,6 +205,18 @@ struct svc_channel { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (doorbell token vector) */ + u32 h_vaspace; /* r835: the client VAS this channel was + * allocated in (NV_CHANNEL_ALLOC_PARAMS. + * hVASpace). Channel-scoped VA resolution + * consults this VAS's page tables FIRST: + * every client's RM window sits at the same + * VAs (0x1200xxxxx) and the global VA-keyed + * maps conflate them - the X server's ring + * VA resolved through the GL client's page + * tables (walk-probe vas h=0xcaf00005 + * answered 0x120090000) and its push stream + * decoded as zero pages, wedging X the same + * way the RC watchdog's ring was starved. */ u8 ce_index; /* rank among the live CE channels (creation order) */ u8 gr_index; /* rank among the live GR channels (creation order) */ u64 gpfifo_va; @@ -506,6 +518,20 @@ struct svc_channel { */ u8 userd_scans; /* + * r828: the RC watchdog's notifier VA, captured from the class2d + * SET_NOTIFY_A/B methods on subchannel 3 and zeroed at the NOTIFY + * (see svc_notify2d_write). Survives the pass snapshot via the + * write-back list. + */ + u64 notify2d_va; + /* r829: bounded count of zero-entry (bad-translation) holds; see the + * zero-entry hold in svc_pump_channels. Persists via the write-back. */ + u32 zero_held; + /* r831: the registered userd value is a real guest PA (the driver's + * own declaration; e.g. the RC watchdog channel). The ring auto-pin's + * ring+0x2000 convention must not overwrite it. */ + bool userd_reg; + /* * r471: the completion-counter publish must never advance the * guest's word past a push whose content was still unwritten when * the entry was consumed (the streams_events staging-reuse @@ -786,6 +812,12 @@ static u32 svc_pb_idle; /* consecutive passes with no new base */ static u32 svc_pb_last_n; /* bases before the active pass */ static unsigned long svc_pb_done; /* CE ranks whose push buffer is pinned */ static unsigned long svc_gr_ring_done; /* GR ranks whose ring is pinned */ +/* SESSION 133: ranks that already consumed their one shadow ring scan. */ +static unsigned long svc_gr_ring_scan_done; +static unsigned long svc_gr_ring_scan_last; /* rate limit for the scan */ +/* SESSION 138: bounded witness for the GR request builder's page list and + * the fill's candidate set (one launch's wiring printed per budget slice). */ +static u32 svc_gr_pages_log; static bool svc_pb_first_push(const u8 *buf) { @@ -1136,6 +1168,17 @@ struct sema_svc { u64 fwd_gr_ok; u64 fwd_gr_fallback; u64 fwd_gr_refused; + u32 fwd_gr_nocand_log; /* SESSION 127: first-N + * request-without-candidate + * witnesses */ + u64 fwd_gr_nordy; /* SESSION 127: backend-not- + * ready request refusals */ + u64 fwd_gr_holdbusy;/* SESSION 127: hold-active + * request busy verdicts */ + u64 fwd_gr_inflight;/* SESSION 127: inflight-bit + * request busy verdicts */ + u32 fwd_gr_enoe_log;/* SESSION 127: first-N fill + * ENOENT page witnesses */ u64 fwd_gr_defer; /* m150: held for data-arg retry */ u64 fwd_gr_dstpend; /* r541: held for an unexecuted * device write on a data page */ @@ -1541,10 +1584,21 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) ch->h_object = desc->h_object; ch->engine_type = desc->engine_type; ch->cid = desc->cid; + ch->h_vaspace = desc->h_vaspace; ch->gpfifo_va = desc->gpfifo_va; ch->gpfifo_entries = desc->gpfifo_entries; ch->userd_pa = desc->userd_pa; ch->userd_as = desc->userd_as ? desc->userd_as : SVC_APERTURE_SYS; + /* + * r831: a registered userd value that looks like a real guest + * PA (the RC watchdog's channel carries its own userd block) + * is the driver's own declaration and outranks the ring+0x2000 + * convention guess the ring auto-pin applies (that guess is + * right for the CUDA GR layout, where the registered value is + * pool-relative - observed 0x10000). Tiny registered values + * stay guesses. + */ + ch->userd_reg = desc->userd_pa >= 0x10000000ULL; ch->instance_pa = desc->instance_pa; ch->notifier_pa = desc->notifier_pa; ch->notifier_as = desc->notifier_as ? desc->notifier_as @@ -1586,12 +1640,26 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * channel; the stashed geometry survives, so the scrub path * does not depend on the channel-table identity at all. */ - if (desc->gpfifo_entries >= 4096 && desc->gpfifo_va && + /* + * m112b: the CeUtils scrubber is the ONLY exactly-4096-entry + * channel (NUM_COPY_BLOCKS). The old `>= 4096` test also + * matched every client channel (this driver version: 32768 + * entries, not the 1024 the comment below assumed), so the + * single scrub slot bound to whichever client channel + * registered last (observed r701: the slot held h=0xbeee0102 + * userd=0x8f400/entries=32768 and the pass authored put=0 for + * the wrong channel while the real scrubber's payload stayed + * zero - the vkAllocateMemory -2 gate). + */ + if (desc->gpfifo_entries == 4096 && desc->gpfifo_va && desc->userd_pa) { struct svc_scrub *s = svc_scrub_get(desc->vgpu, true); if (!s) goto scrub_done; /* table full: max guests */ + pr_info("sema-svc: scrub stash bind vgpu=%px slot=%d h=0x%x entries=%u\n", + (void *)desc->vgpu, (int)(s - svc_scrubs), + desc->h_object, desc->gpfifo_entries); /* Re-seed the counting on a fresh scrubber object or a * new client session (m176: the counter/submitted state * is per-session; a recycled scrubber handle after a @@ -1616,6 +1684,16 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, s->userd_as); + } else if (desc->gpfifo_entries >= 4096 && + desc->gpfifo_entries != 32768) { + /* m112b: a non-32768 channel with >=4096 entries that + * fails the stash condition (missing VA or USERD PA) + * is a scrubber candidate the service cannot author + * for - log it so a silent no-authoring boot is + * diagnosable. */ + pr_info_ratelimited("sema-svc: scrubber-stash MISS h=0x%x entries=%u va=0x%llx userd_pa=0x%llx\n", + desc->h_object, desc->gpfifo_entries, + desc->gpfifo_va, desc->userd_pa); } scrub_done: pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx\n", @@ -1662,6 +1740,8 @@ static u32 svc_opdef_clr_log; /* C3: opdef-clear caller witness */ static u32 svc_busy_defer_log; /* C8: GR busy-defer timeout witness */ static u32 svc_hold_fb_log; /* C7: hold failure/timeout fallback witness */ static u32 svc_cand_ovw_log; /* C9: candidate overwrite witness */ +static u32 svc_gr_push_dump; /* SESSION 127: bounded capture tail dump + * (sema_dbg only, 4 per boot) */ static u32 svc_inline_recs_log; /* C10: inline-recs digest witness */ static u32 svc_gr_await_log; /* r351: launch-held-for-trust witness */ static u32 svc_gr_retry_log; /* r351: retry-forward/proceed witness */ @@ -1814,6 +1894,64 @@ void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, * the lesser evil; it is logged loudly for the record. */ #define SVC_VAS_CAND_AGE_MS 15000u +/* + * r705h: the candidate age-out deadline is runtime-tunable. The parked + * COPY_SERVER_PDES candidate for the GL/Vulkan client ages out (UNVERIFIED) + * at exactly the moment the client's channels first register, so the + * content-verified walk can never anchor it; raising the deadline keeps the + * candidate parked until a host tool supplies a verified ring pair via + * sema_pdb_discover. 0 disables the age-out entirely. + */ +static u32 svc_vas_cand_age_ms = SVC_VAS_CAND_AGE_MS; +module_param_named(sema_vas_cand_ms, svc_vas_cand_age_ms, uint, 0644); + +/* + * SESSION 127 C5C0 translator gate. When 1, the sweep additionally + * translates continuation entries into launch work (tail-aware method + * parse, inline-QMD capture, LOAD_INLINE_DATA materialization, C5B5 + * copy-out capture, candidate-owned release withholding). + * + * SESSION 130 re-test of the r727 "translator regressions" (translator on, + * utility sweep OFF, r746): victory_saxpy and victory_math now PASS with + * checksums identical to the gate-off baseline (saxpy + * csum=981440890.500000), so the saxpy/math half of the r727 regression + * is FIXED (the SESSION 129/130 sweep changes); victory_mail PASSES. The + * remaining xlate-on regression is victory_multikernel: its launch is + * captured and submitted to the physical channel but the forward times + * out repeatedly (`GR LAUNCH FAILED: timeout after 2000 ms + * (qmd_rel=0xc0ffee01 want=0x600df00d ...) h=0x5c000019 + * push=0x20043c394`, `hold active`), so the client hangs - the default + * stays 0 until that forward path is fixed. The UTILITY SWEEP + * (`sema_gr_util_scan`) is the OTHER regression source: at 24 it broke + * saxpy (`bad=1047527`) and hung math on r745, so it stays opt-in + * (default 0) until its channel reads prefer the direct walk over pins. + */ +static bool gsp_sema_gr_xlate; +module_param_named(sema_gr_xlate, gsp_sema_gr_xlate, bool, 0644); +MODULE_PARM_DESC(sema_gr_xlate, + "C5C0 translator: sweep continuation entries into launch work (default 0 = releases-only sweep)"); +/* + * SESSION 128 utility-channel inline-download materialization + * (sema_gr_util_scan, default 0 = off): the driver's non-GR upload + * channels (the MEM2MEM/A140 + 2D utilities, registered but with an + * unresolved USERD so the pump's put-bounded entry loop never decodes + * them) carry the engine-side code/data downloads: the + * GRCE/INLINE_TO_MEMORY operand group on the mem2mem subchannel (0x180.. + * 0x18c), LAUNCH_DMA (0x1b0) and a LOAD_INLINE_DATA (0x1b4) run whose + * payload the engine fetches from the next GPFIFO entry (the + * source-view call). Without this pass those destinations stay zero in + * the vidmem shadow - observed: the Vulkan client's code-heap/SASS + * upload (12 KB) and command-stream staging (256 B) never land, so the + * forwarded launch has no program content. Bounded: scan up to N + * entries per channel per pass, materialize once-only keyed by + * (push_va, content hash, dst, bytes) via the same + * svc_grce_inline_write the GR sweep uses. Requires sema_gr_xlate. + */ +static u32 gsp_sema_gr_util_scan; +module_param_named(sema_gr_util_scan, gsp_sema_gr_util_scan, uint, 0644); +MODULE_PARM_DESC(sema_gr_util_scan, + "Utility-channel i2m sweep: entries scanned per pass (default 0 = off; requires sema_gr_xlate)"); +MODULE_PARM_DESC(sema_vas_cand_ms, "Parked-VAS-candidate age-out (ms, default 15000; 0 = never)"); static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) { @@ -1824,8 +1962,9 @@ static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) if (!svc_vas_cand[i].valid) continue; - if (time_before(jiffies, svc_vas_cand[i].stamp + - msecs_to_jiffies(SVC_VAS_CAND_AGE_MS))) + if (!svc_vas_cand_age_ms || + time_before(jiffies, svc_vas_cand[i].stamp + + msecs_to_jiffies(svc_vas_cand_age_ms))) continue; memset(&d, 0, sizeof(d)); d.h_vaspace = svc_vas_cand[i].h_vaspace; @@ -1833,7 +1972,7 @@ static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) d.num_entries = svc_vas_cand[i].entries; d.b_vidmem = svc_vas_cand[i].vidmem; pr_warn("sema-svc: VAS 0x%x page directory registered UNVERIFIED after %u ms parked (pdb=0x%llx entries=%u; the content-verified ring walk never anchored this boot)\n", - d.h_vaspace, SVC_VAS_CAND_AGE_MS, d.pdb, + d.h_vaspace, svc_vas_cand_age_ms, d.pdb, d.num_entries); gsp_sema_service_vaspace_add(&d); svc_vas_cand[i].valid = false; @@ -2651,6 +2790,139 @@ static void svc_lat_store(u64 va, u64 addr, int ap) } /* + * r820: driver-declared kernel-buffer translations (PROMOTE_CTX). + * + * The guest RM announces the physical backing of a channel's context + * buffers through NV2080_CTRL_CMD_GPU_PROMOTE_CTX: every entry carries + * (gpuVirtAddr, gpuPhysAddr, size, physAttr, bufferId). These are + * KERNEL-owned buffers - the nvidia-push pushbuffer, the progress-tracker + * segment, the FECS/patch/access-map blocks - so the userspace pin + * scanner can never cover them (/proc/<pid>/pagemap has no kernel VAs) + * and the page-table walk can answer with a stale/wrong page. Observed + * r817 (Xorg GL init): the walk resolved the 64 KB pushbuffer VA + * 0x120010000 to pa=0x103fed000, a page reading all zeros, while the + * driver's own promote entry said pa=0x11e120000. The NOP sanity push of + * __nvPushTestPushBuffer then never decoded, GP_GET never advanced and + * the X server died with "Failed to initialize DMA". + * + * The registry is a small table keyed by the entry VA, newest wins: a + * client that re-promotes the same buffer - observed between two Xorg + * starts, the same channel object announced a NEW pa for the same VA - + * replaces the old record, so a stale backing is never preferred over + * the live statement. VA-only bookings (bInitialize=0 re-promotes of + * already-initialized buffers, zero PA/size) and bNonmapped entries + * (va=0) carry no backing and must not erase the last statement. + * + * Lookups run before the learned cache in svc_lookup_trusted: the wrong + * r817 walk answer had been cached in the learned table (svc_lat_store + * after every walk success), so a promote hit must both pre-empt and + * overwrite that entry; the hit re-stores the corrected page into the + * learned table so the direct svc_lat_lookup callers heal too. + */ +#define SVC_PROMO_MAX 256 +struct svc_promo_rec { + u64 va; + u64 pa; + u64 size; + unsigned long seq; + u32 phys_attr; + u16 buffer_id; + bool valid; +}; +static struct svc_promo_rec svc_promo[SVC_PROMO_MAX]; +static unsigned long svc_promo_seq; +static DEFINE_SPINLOCK(svc_promo_lock); +static u32 svc_promo_hit_log; /* bounded "answered from PROMOTE_CTX" witness */ + +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id) +{ + struct svc_promo_rec *rec = NULL, *free_rec = NULL, *oldest = NULL; + unsigned long flags; + int i; + + if (!va || !pa || !size) + return; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + if (svc_promo[i].valid && svc_promo[i].va == va) { + rec = &svc_promo[i]; + break; + } + if (!svc_promo[i].valid && !free_rec) + free_rec = &svc_promo[i]; + if (svc_promo[i].valid && + (!oldest || svc_promo[i].seq < oldest->seq)) + oldest = &svc_promo[i]; + } + if (!rec) + rec = free_rec ? free_rec : oldest; + if (!rec) { + spin_unlock_irqrestore(&svc_promo_lock, flags); + return; + } + rec->va = va; + rec->pa = pa; + rec->size = size; + rec->phys_attr = phys_attr; + rec->buffer_id = (u16)buffer_id; + rec->seq = ++svc_promo_seq; + rec->valid = true; + spin_unlock_irqrestore(&svc_promo_lock, flags); +} + +static int svc_promo_lookup(u64 va, u64 *addr, int *ap) +{ + unsigned long flags; + u64 best_seq = 0; + int rc = -ENOENT; + int i; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + const struct svc_promo_rec *rec = &svc_promo[i]; + + /* + * r823: answer ONLY the base page. The promoted (va, size) + * pair is a VA-space booking, not a contiguous mapping: the + * ctx MAIN buffer's span [0x120020000, 0x120153000) contains + * OTHER buffers' VAs (the channel GPFIFO at 0x120090000), and + * the range arithmetic answered such an interior VA with + * MAIN.pa + offset - a zero page (observed r823: the GPFIFO + * VA resolved to 0x11ce70000, the ring-entry validation read + * va=0x0 len=0 and the latch never fired). Only the page at + * the declared VA has a proven claim to the declared PA; the + * pushbuffer's NOP sits exactly there (nvPushHeader writes at + * the segment base). + */ + if (!rec->valid || va < rec->va || + va - rec->va >= SEMA_PAGE_SIZE) + continue; + if (best_seq && rec->seq <= best_seq) + continue; + best_seq = rec->seq; + *addr = rec->pa + (va - rec->va); + /* + * physAttr 0x4 was observed only on vidmem entries (pa in + * the emulated FB aperture, e.g. 0x1f4000000), 0x5/0x6 on + * system memory. Unknown values fall to SYS, whose read + * guard (pa < SEMA_PA_MAX) still rejects nonsense. + */ + *ap = (rec->phys_attr == 0x4) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + rc = 0; + } + spin_unlock_irqrestore(&svc_promo_lock, flags); + if (!rc && svc_promo_hit_log < 64) { + svc_promo_hit_log++; + pr_info("sema-promo: va=0x%llx -> pa=0x%llx ap=%d (driver PROMOTE_CTX statement)\n", + va, *addr, *ap); + } + return rc; +} + +/* * VA shadow: page-granular backing for device VAs whose VA->PA translation * cannot be derived from the guest's page tables. * @@ -2984,6 +3256,52 @@ struct svc_ipin_alt { static struct svc_ipin_alt svc_ipin_alt[SVC_IPIN_ALT_SLOTS]; static u64 svc_pushalt_hits; /* push reads served by a displaced pair */ static u32 svc_pushalt_log; +/* r707x: per-client push-read attempt witness (pump read site). */ +static u32 svc_rdw_log; +/* r820: bounded per-session USERD-scan decision witness (svc_chan_userd). + * GR channels only and capped: a hot CE channel re-running this every + * ~16 ms burned the whole budget before the channel under diagnosis ever + * appeared. Each scan reports its decision, a scan that found nothing + * reportable logs a sweep summary. */ +static u32 svc_userdp_log; +/* r829: bounded witness for the zero-entry holds (bad ring translations). */ +static u32 svc_zero_entry_log; +static u8 svc_read_last_src; +/* r715: PA the winning source of the last push read resolved to. */ +static u64 svc_read_last_pa; +/* + * r820: set when the last svc_read_va_push_ex chunk was answered by a + * driver-declared PROMOTE_CTX kernel buffer. The push decode uses it to + * accept the NOP sanity push: nvidia-push's __nvPushTestPushBuffer writes + * NVA16F_DMA_NOP (0x00000000) into a promoted kernel pushbuffer, so the + * whole small push is zeros - unparseable as a method stream, but exactly + * right from a translation the driver itself declared. + */ +static bool svc_read_last_promote; +/* r715: per-client push-decode branch witness (pump read site). */ +static u32 svc_pushdec_log; +/* r707d: bounded push-read failure witness (see svc_read_va_push_ex). */ +static u32 svc_pushfail_log; +/* + * SESSION 132: the util-sweep witnesses are per-CLIENT budgets (reset in + * svc_session_flush), not per-boot: on a multi-client boot the driver-init + * reads of an earlier client exhausted the boot budget and hid the later + * clients' rows (the same fix the r707x/r715 pump witnesses got). + */ +static u32 svc_util_dec_log; +static u32 svc_gr_util_log; +static struct { u64 va; u64 h; } svc_util_dec_keys[32]; +/* + * SESSION 143 (r803): per-channel sweep VISIBILITY. The r802 evidence could + * not say whether the Vulkan client's copy-class channels (0xbeef0101/2, + * eng 11/12) were swept at all: the per-pass log capped at 24 rows and the + * early return (no gpfifo/entries) prints nothing. These one-shot (per + * client, reset in svc_session_flush) keys name the first sweep of EVERY + * channel - begin or skip + reason - so "channel absent from the log" can + * never again be confused with "channel not in the table". + */ +static u32 svc_util_seen_keys[32]; +static u32 svc_util_seen_n; static int svc_ipin_alt_lookup(u64 va, u64 *pa, int *ap) { @@ -3265,11 +3583,56 @@ static bool svc_resolve_in_progress(u64 va) * VA above the window base, including sysmem-backed VAs, so callers that must * not act on a guess (push execution) use this variant. */ +/* + * r835: the VAS handle of the channel currently being serviced (0 = none). + * Channel-scoped VA resolution consults this VAS's page tables FIRST: the + * global pin/lat/map layers are VA-keyed and every client's RM window sits + * at the same VAs (0x1200xxxxx), so a lookup outside the owning client's + * page tables can answer ANOTHER client's page - the X server's ring + * resolved to the GL client's (zero) page and its pushes decoded empty, + * wedging X's readback wait exactly like the RC watchdog's ring starvation. + */ +static u32 svc_cur_vas; + +static int svc_chanvas_lookup(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, + int *ap) +{ + int i; + + if (!svc_cur_vas) + return -ENOENT; + for (i = 0; i < SVC_MAX_VASPACES; i++) { + u64 pa = 0; + int a = SVC_APERTURE_SYS; + + if (!svc.vas[i].valid || + svc.vas[i].h_vaspace != svc_cur_vas) + continue; + if (svc_gmmu_walk(vgpu, svc.vas[i].pdb, svc.vas[i].entries, + va & ~(u64)(SEMA_PAGE_SIZE - 1), + svc.vas[i].ap, &pa, &a, false)) + continue; + *addr = pa + (va & (SEMA_PAGE_SIZE - 1)); + *ap = a; + return 0; + } + return -ENOENT; +} + static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { u64 pa = 0; + /* + * r835: the decoded channel's own page tables answer first. When a + * channel is being serviced (svc_cur_vas set) its VAS is the only + * authoritative source for its VAs; the VA-keyed layers below may + * hold another client's pages for the same numeric VA. A failed + * walk falls through to the old chain unchanged. + */ + if (!svc_chanvas_lookup(vgpu, va, addr, ap)) + return 0; if (!svc_pin_lookup_any(va, addr, ap)) { return 0; } @@ -3313,6 +3676,23 @@ static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, svc_lat_store(va, pa, SVC_APERTURE_SYS); return 0; } + /* + * r826: driver-declared PROMOTE_CTX buffers are the LAST resort, not + * the first. The promoted (va, pa) pair names a BOOKING, and even its + * base page can belong to a different buffer than the one mapped at + * that VA (r823: the ctx-MAIN booking's span swallowed the GPFIFO VA; + * r826: the PATCH booking's base VA 0x120001000 hosts nvidia-push's + * separately mapped 4 KB progress semaphore - writing the release to + * the booking's pa left the driver's own polled page at zero and the + * X server failed at its 10 s wait while the release value sat in the + * wrong page). The page-table walks describe the real mappings and + * must answer first; the promote record is only a fallback for VAs + * the walks cannot resolve at all. + */ + if (!svc_promo_lookup(va, addr, ap)) { + svc_lat_store(va, *addr, *ap); + return 0; + } return -ENOENT; } @@ -6330,10 +6710,11 @@ static void svc_gr_cand_capture(struct svc_channel *ch, svc_gr_cand.param_size, svc_gr_cand.rel_va, svc_gr_cand.rel_payload, svc_gr_cand.rel_flags, svc_gr_cand.n_pointer, svc_gr_cand.n_inline); - if (push && push_len >= 16) { + if (gsp_sema_dbg && svc_gr_push_dump < 4 && push && push_len >= 16) { u32 n = push_len / 4; u32 i, j; + svc_gr_push_dump++; if (n > 256) n = 256; for (i = 0; i < n; i += 16) { @@ -6401,6 +6782,31 @@ static bool svc_fwd_defer_arm(void) gsp_phys_work_defer_release_enabled(); } +/* + * r824: the release deferral must be owned by a candidate of THIS push. + * svc_fwd_defer_arm() answers globally (any unoffered candidate of the + * boot's ring); an unrelated lingering candidate then parked the release + * of a push that has no forwarding at all. Observed: the X/GL DMA test's + * progress-tracker release (the kernel's progress semaphore) was recorded + * into svc_fwd_rel and dropped at the next pass's reset, every pass, so + * the semaphore never advanced and Xorg failed at its 10 s wait - while + * every other stage of the chain (latch, NOP acceptance, decode) had + * just started working. A release whose push owns no candidate cannot + * wait for a forwarded copy that will never exist; it applies inline. + */ +static bool svc_fwd_cand_owns_push(u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + /* * P6b wave-1.5 deferred guest releases: every release group the current @@ -6954,6 +7360,20 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u64 *done_out, u32 rows, u32 dpitch, u32 spitch, u32 owner_ch, u32 owner_idx, u64 owner_hash) { + /* + * fail4: name every copy the executor is asked to run (pre- + * resolution), so a copy that fails or is held before any witness + * still has an entry record. SESSION 132: one record per COPY OP, + * not per row - a 480-row frame copy logged 480 rows and exhausted + * the per-session budget (128) inside the first op, hiding every + * later copy (including any SASS upload). + */ + if (gsp_sema_dbg && svc.copyent_log < 128) { + svc.copyent_log++; + pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu rows=%u dpitch=0x%x spitch=0x%x dphys=%u dap=%u sphys=%u sap=%u\n", + svc.cur_ch, dst, src, total, rows, dpitch, spitch, + dst_phys, dst_aperture, src_phys, src_aperture); + } if (rows > 1 && dpitch) { u64 done = 0; int rc = 0; @@ -7003,17 +7423,6 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot *done_out = 0; /* 0 until the first byte lands */ if (!total || total > (256ULL << 20)) return -EINVAL; - /* - * fail4: name every copy the executor is asked to run (pre- - * resolution), so a copy that fails or is held before any witness - * still has an entry record. - */ - if (gsp_sema_dbg && svc.copyent_log < 128) { - svc.copyent_log++; - pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu dphys=%u dap=%u sphys=%u sap=%u\n", - svc.cur_ch, dst, src, total, dst_phys, dst_aperture, - src_phys, src_aperture); - } snap = svc_copy_snap_lookup(dst_orig, src_orig, total, owner_ch, owner_idx, owner_hash); @@ -8848,18 +9257,20 @@ static bool svc_stream_parse(const u8 *buf, u32 len) memcpy(&hdr, buf + off, 4); if (hdr == 0) { /* - * UVM's large page-table pushes begin with a - * zero-filled header region (observed 0x1004 - * bytes) before the first method, so zeros before - * the first method are leading padding; zeros - * after one are trailing padding and end the - * stream. + * NVC06F_DMA_NOP is 0x00000000: a zero word is + * NOP padding. UVM's large page-table pushes begin + * with a zero-filled header region (observed 0x1004 + * bytes) and the GR submit pushes carry zero gaps + * between method groups (observed r715: the 1432- + * and 17316-byte pushes pad between the context + * setup and the dispatch groups), so zeros are + * skipped anywhere in the stream, not only before + * the first method or at the tail. The exact-end + * requirement and the valid-header walk below are + * the false-positive arbiters. */ - if (!methods) { - off += 4; - continue; - } - break; /* trailing zero padding */ + off += 4; + continue; } op = (hdr >> 29) & 0x7; count = (hdr >> 16) & 0xfff; @@ -8879,16 +9290,27 @@ static bool svc_stream_parse(const u8 *buf, u32 len) return false; off += 4 + count * 4; break; + case 0: /* GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK */ + case 2: /* GRP2_USE_TERT */ case 4: /* IMMD_DATA_METHOD: header only */ - off += 4; - break; case 7: /* END_PB_SEGMENT: header only */ + /* + * r715: the class header defines op 0 as + * GRP0_USE_TERT (0) and op 2 as GRP2_USE_TERT - + * 4-byte header-only formats. The GR submit + * pushes emit 0x0001fff0 (SET_SUBDEVICE_MASK, + * value 0xfff) around their semaphore groups; the + * old parser rejected the first one on every big + * push, so the whole push read implausible and + * the entries were skipped without decode + * (session 124's consume-without-decode). + */ off += 4; break; default: return false; } - if (++methods > 1024) + if (++methods > 8192) return false; } while (off < len && buf[off] == 0) @@ -8947,6 +9369,127 @@ static unsigned long svc_put_log_at[32]; */ static u32 svc_userd_dbg; static u32 svc_userd_dbg_ch[32]; +/* SESSION 134: per-CE-channel budget of the first-submission ring captures + * (see svc_ce_firstsub_capture) and the start of an unlatched CE channel's + * completion hold (see svc_ce_hold_ms). Session state; reset by + * svc_session_flush. */ +static u32 svc_ce_firstsub_n[32]; +static unsigned long svc_ce_hold_since[32]; +/* + * SESSION 134c: the FB offset each CE rank's first-submission capture latched + * (0 = none). The shadow scan is rank-blind on its own - it takes the first + * ring-shaped page in the shadow - and on r779 that latched rank 0's block + * (fb 0x80000, userd 0x90000) for rank 1 as well (both channels reported + * scan_hits=1 latch=1 off=0x80000). The rank-1 channel's first release was + * then decoded from rank 0's ring/page and stale-dropped (cur=1 already from + * rank 0's applied release), so the client's rank-1 tracker wait never + * completed and the client spun forever (victory_mail stall, three boots). + * The capture now prefers the rank's convention offset and refuses any + * offset another CE rank already latched. Session state; reset by + * svc_session_flush. + */ +static u64 svc_ce_firstsub_off[32]; +/* + * SESSION 144: FB ring-block ownership ACROSS engine classes. The GR + * autopin fallback scan (SESSION 133c) and the CE first-submission + * capture (SESSION 134c) each solved the same problem only within their + * own class; a live-but-foreign ring block passes every shape gate, so a + * channel whose own ring is not written yet latches another channel's + * block permanently and never reads its own entries. Observed on + * r805-r808: every CUDA GR rank's userd-shape scan latch resolved the CE + * rank-0 block (fb 0x80000, userd 0x90000; e.g. + * "auto-pinned GR ring h=0x5c000019 rank=0 ... -> fb 0x80000 (userd-shape + * scan ... entry0 va=0x120000000 len=40 put=35 get=35)" while the CE + * handle decoded the identical entry[35] raw words from pa=0x10350641c), + * so no GR launch was ever captured and victory_mail hung after DEVICE. + * A channel that cannot see its own block must stay unlatched (and retry + * next pass) rather than steal a foreign one. Keyed by the FB ring + * offset; claims are session state, reset by svc_session_flush. + */ +struct svc_fb_claim { + u32 h; /* owning channel handle */ + u64 ring_off; /* FB ring offset */ + u64 userd_off; /* FB USERD offset (0 = unknown) */ +}; +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL +static struct svc_fb_claim svc_fb_claim_tab[40]; +static u32 svc_fb_claim_n; + +static void svc_fb_claim_add(u32 h, u64 ring_off, u64 userd_off) +{ + u32 i; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == h) { + svc_fb_claim_tab[i].ring_off = ring_off; + svc_fb_claim_tab[i].userd_off = userd_off; + return; + } + } + if (svc_fb_claim_n < ARRAY_SIZE(svc_fb_claim_tab)) { + svc_fb_claim_tab[svc_fb_claim_n].h = h; + svc_fb_claim_tab[svc_fb_claim_n].ring_off = ring_off; + svc_fb_claim_tab[svc_fb_claim_n].userd_off = userd_off; + svc_fb_claim_n++; + } +} + +/* + * true = the candidate FB block (ring at ring_off, USERD at userd_off if + * known) is already claimed by a DIFFERENT channel, or is a live CE + * rank's convention block even before that rank latched, i.e. it must not + * be latched or stolen. Same-channel re-latches pass. + */ +static bool svc_fb_claim_foreign(struct nv_vgpu_device *vgpu, u32 self_h, + u64 ring_off, u64 userd_off) +{ + u32 i; + unsigned long flags; + int k; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == self_h) + continue; + if (svc_fb_claim_tab[i].ring_off == ring_off) + return true; + if (userd_off && svc_fb_claim_tab[i].userd_off == userd_off) + return true; + } + /* + * SESSION 144b: reserve every live CE rank's convention block even + * before that rank has latched. r809: CE rank 1's block at fb + * 0xa0000 was still unclaimed when GR rank 0's userd-shape scan ran + * (rank 1 latched through the plain autopin path, not the + * first-submission capture) and the GR channel stole it again. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (k = 0; k < SVC_MAX_CHANNELS; k++) { + u64 conv; + + if (!svc.chan[k].valid || svc.chan[k].vgpu != vgpu || + !svc.chan[k].engine_type || + svc.chan[k].h_object == self_h || + svc.chan[k].ce_index >= 32) + continue; + conv = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * svc.chan[k].ce_index; + if (ring_off == conv || + (userd_off && userd_off == conv + 0x10000)) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + return true; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +static void svc_fb_claim_reset(void) +{ + svc_fb_claim_n = 0; + memset(svc_fb_claim_tab, 0, sizeof(svc_fb_claim_tab)); +} + /* * ISSUE-0002 (r47): deferred FB sweep for a CE channel whose layout USERD * probe fails at discovery. Session/reset-scoped, scheduled on the first @@ -10014,8 +10557,19 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } if (st->qmd_n > svc_gr_qmd_seen) svc_gr_qmd_seen = st->qmd_n; - if (st->qmd_have == 3) - st->qmd_dev_va = (((u64)st->qmd_hi << 32) | st->qmd_lo) << 8; + /* + * SESSION 127 C5C0 translator: the Vulkan driver writes + * SET_INLINE_QMD_ADDRESS_B only (the upper half stays absent for a + * <4G QMD; observed beec0100 idx23: `0x31c = 0x4b85f` then a 64-word + * 0x320 run). Accept the B-only form with an implicit zero upper + * half - the old A+B requirement left qmd_dev_va at 0 and the + * candidate logged with qmd=0x0 (the field is identity/logging + * only, but the 0x0 read confused every prior session's forensics). + */ + if (st->qmd_have & 2) + st->qmd_dev_va = + (((u64)((st->qmd_have & 1) ? st->qmd_hi : 0) << 32) | + st->qmd_lo) << 8; if (st->qmd_n == SVC_QMD_WORDS) { /* * Static: a 16-entry record array is ~1.2 KB and this is @@ -10037,15 +10591,85 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, svc_grce_dst_ring_next : SVC_GRCE_DST_MAX; if (have > GSP_BRIDGE_INLINE_MAX) have = GSP_BRIDGE_INLINE_MAX; - for (i = 0; i < have; i++) { - u32 idx = (svc_grce_dst_ring_next - 1 - i) % - SVC_GRCE_DST_MAX; - const struct svc_grce_rec *rec = &svc_grce_dst_ring[idx]; - - recs[n].dst_va = rec->dst_va; - recs[n].len = rec->len; - memcpy(recs[n].data, rec->data, rec->len); - n++; + /* + * SESSION 127 C5C0 translator: the rec ring is session-wide + * and mixes every client's LOAD_INLINE_DATA destinations. A + * stale rec whose page this QMD does not name made the fill + * refuse the whole launch (GR fill ENOENT + * va=0x80412000 qmd_word=NONE covered=1 - the Vulkan + * client's cancel page inherited from the driver-init + * phase). Keep only recs whose page is one of the pages the + * captured QMD names (the QMD buffer, the parameter bank, + * the program, the pointer pairs) - exactly the pages the + * backend's inline-served fill can use; keep everything when + * the QMD names none (fail-open). + */ + { + u64 keep[12]; + u32 nkeep = 0, k; + + if (st->qmd_dev_va) + keep[nkeep++] = st->qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[32] | + ((u64)(st->qmd[33] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[48] | + ((u64)(st->qmd[49] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + for (k = 0; k < SVC_GR_POINTER_MAX - 1 && nkeep < 12; k++) { + u32 w = 34 + 2 * k; + u64 pv = (u64)st->qmd[w] | + ((u64)(st->qmd[w + 1] & 0x1ffff) + << 32); + + if (pv) + keep[nkeep++] = pv & + ~(u64)(SEMA_PAGE_SIZE - 1); + } + for (i = 0; i < have; i++) { + u32 idx = (svc_grce_dst_ring_next - 1 - i) % + SVC_GRCE_DST_MAX; + const struct svc_grce_rec *rec = + &svc_grce_dst_ring[idx]; + u64 rpage = rec->dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + bool keepit = !nkeep; + u32 j; + + for (j = 0; j < nkeep; j++) + if (keep[j] == rpage) { + keepit = true; + break; + } + /* + * SESSION 127 (gated): a stale session-wide + * rec whose page resolves nowhere made the + * fill refuse the whole launch; keep it when + * it resolves (the CUDA inline-served + * pages), drop only unresolvable unnamed + * ones. With the translator off, the + * historical keep-all behaviour stands. + */ + if (!keepit && gsp_sema_gr_xlate) { + u64 rpa = 0; + u32 rap = SVC_APERTURE_SYS; + + if (svc_resolve_va(vgpu, rpage, &rpa, + &rap) && + svc_walk_vaspaces(vgpu, rpage, + &rpa, &rap)) + continue; + } else if (!keepit) { + continue; + } + recs[n].dst_va = rec->dst_va; + recs[n].len = rec->len; + memcpy(recs[n].data, rec->data, rec->len); + n++; + } } /* * C10 (GAP-2 A5): duplicate destinations in the newest-first @@ -10202,6 +10826,60 @@ static void svc_push_raw_dump(const u8 *buf, u32 len, u64 push_va, h_object, push_va, off, len); } +/* + * r828: the guest kernel's RC watchdog (krcWatchdog_IMPL, 7 s notify + * timeout) posts a notifier cycle on its own channel (object 0x31415900) + * through the class2d subchannel (NVA06F_SUBCHANNEL_2D = 3): SET_NOTIFY_A/B + * name the notifier VA, NOTIFY (WRITE_ONLY) is the GPU's "write a zeroed + * NvNotification" - the driver set the status word to 0xFFFF and declares + * "GPU is probably locked" when it never comes back as 0. Observed: robust + * channels enabled by the GL client (GZDOOM), then NVRM krcWatchdog every + * 8 s and Xorg dead on its screen readback. The emulation must author the + * notification write, exactly like svc_notify_complete does for the + * work-submit token. + */ +#define SVC_2D_SUBCH 3 /* NVA06F_SUBCHANNEL_2D */ +#define SVC_2D_SET_NOTIFY_A 0x0104 /* payload: addr bits [56:32] */ +#define SVC_2D_SET_NOTIFY_B 0x0108 /* payload: addr bits [31:0] */ +#define SVC_2D_NOTIFY 0x010c /* payload: 0 = WRITE_ONLY */ + +static u64 svc_notify2d_done; +static u32 svc_notify2d_log; + +static void svc_notify2d_write(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + /* 16 bytes = the NvNotification (status word at +14). */ + u8 zero[16] = { 0 }; + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (!ch->notify2d_va) + return; + if (svc_lookup(vgpu, ch->notify2d_va, &pa, &ap)) { + if (gsp_sema_dbg && svc_notify2d_log < 16) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx unresolved (RC watchdog notifier; guest re-posts every second)\n", + ch->h_object, ch->notify2d_va); + } + return; + } + if (svc_write_ap(vgpu, pa, svc_ap_of(ap), zero, sizeof(zero))) + return; + svc_notify2d_done++; + /* + * r834: pin the page that accepted the ack; the same stale-map + * defense as the push-side pin in the decode. + */ + svc_pin_store(ch->notify2d_va & ~(u64)(SEMA_PAGE_SIZE - 1), + pa & ~(u64)(SEMA_PAGE_SIZE - 1), svc_ap_of(ap)); + if (gsp_sema_dbg && svc_notify2d_log < 16) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx -> pa=0x%llx zeroed (RC watchdog acked)\n", + ch->h_object, ch->notify2d_va, pa); + } +} + static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 len, u64 page_pa, bool allow_ops, struct svc_channel *ch, u32 entry_idx, @@ -10219,6 +10897,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * and the matmul r322 starved/hung note). */ svc.cur_ch = ch->h_object; + /* r835: decode-scoped VAS for channel-first VA resolution. */ + svc_cur_vas = ch->h_vaspace; /* * r34x: the acquire-wait capture is PER RING ENTRY. It used to be * reset only at SET_OBJECT, so a decode pass over a later entry (or @@ -10752,6 +11432,48 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, continue; } + if (subch == SVC_2D_SUBCH && count >= 1 && addr >= 0x0100 && + addr <= 0x010c) { + /* + * r828: the RC watchdog's notifier cycle (class2d + * methods on subchannel 3). See svc_notify2d_write: + * without the zeroed notification the guest's + * krcWatchdog declares the GPU locked after 7 s and + * the X/GL stack wedges on screen readback. + */ + switch (addr) { + case SVC_2D_SET_NOTIFY_A: + ch->notify2d_va = (u64)(v0 & 0x1ffffff) << 32; + /* + * r834: this method stream reached us through + * a REAL read (svc_read_last_pa); pin its page + * so the watchdog's push translation cannot go + * stale later - the content-scanned map answers + * a zero page once the injected pin ages out, + * staging exactly the starvation the ack must + * not depend on. + */ + if (svc_read_last_pa) + svc_pin_store(push_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + svc_read_last_pa & + ~(u64)(SEMA_PAGE_SIZE - 1), + SVC_APERTURE_SYS); + break; + case SVC_2D_SET_NOTIFY_B: + ch->notify2d_va |= (u64)v0; + break; + case SVC_2D_NOTIFY: + if (count == 1 && v0 == 0) /* WRITE_ONLY */ + svc_notify2d_write(vgpu, ch); + break; + default: + break; + } + off += 4 + count * 4; + continue; + } + if (subch == 0 && addr == 0) { /* SET_OBJECT on the channel subchannel: a push starts */ n_ops = 0; @@ -10844,7 +11566,16 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.va = va; or.payload = payload; or.entry_idx = entry_idx; + /* + * r824: the deferral needs a + * candidate that OWNS this push + * (see svc_fwd_cand_owns_push) - + * a release from a push with no + * forwarding has nothing to wait + * for and applies inline. + */ if (svc_fwd_defer_arm() && + svc_fwd_cand_owns_push(push_va) && !svc.op_tr_pending) { svc_opdef_drop(vgpu, ch, entry_idx); @@ -12655,6 +13386,17 @@ static void svc_gr_defer_reset(void); static void svc_session_flush(void) { + /* r707x: per-client witness budgets. */ + svc_pushfail_log = 0; + svc_rdw_log = 0; + svc_pushdec_log = 0; + /* SESSION 132: util-sweep witnesses + their dedup keys are per-client. */ + svc_util_dec_log = 0; + svc_gr_util_log = 0; + memset(svc_util_dec_keys, 0, sizeof(svc_util_dec_keys)); + /* SESSION 143: the per-channel sweep-visibility keys are per-client. */ + svc_util_seen_n = 0; + memset(svc_util_seen_keys, 0, sizeof(svc_util_seen_keys)); unsigned long flags; u32 i; @@ -12760,6 +13502,7 @@ static void svc_session_flush(void) svc_pb_last = 0; svc_pb_last_n = 0; svc_gr_ring_done = 0; + svc_gr_ring_scan_done = 0; svc_ring_n = 0; /* * The learned tables are session state too: the primary VA->PA pin @@ -12785,6 +13528,16 @@ static void svc_session_flush(void) * name the previous client's pages (ISSUE-0002 fan-out). */ memset(svc_ipin_alt, 0, sizeof(svc_ipin_alt)); + /* + * r820: PROMOTE_CTX records name one client's kernel buffers, and a + * new client's channels re-announce their own (the same VAs can back + * with different PAs across sessions): session state. A flush skips + * entirely while any channel is live, so a running client's records + * cannot be wiped under it. + */ + memset(svc_promo, 0, sizeof(svc_promo)); + svc_promo_seq = 0; + svc_promo_hit_log = 0; memset(svc_cnt_der, 0, sizeof(svc_cnt_der)); svc_cnt_der_n = 0; /* r551: delivered-release marks are per-session (same handles reused). */ @@ -12869,6 +13622,10 @@ static void svc_session_flush(void) memset(svc_put_log_at, 0, sizeof(svc_put_log_at)); svc_userd_dbg = 0; memset(svc_userd_dbg_ch, 0, sizeof(svc_userd_dbg_ch)); + memset(svc_ce_firstsub_n, 0, sizeof(svc_ce_firstsub_n)); + memset(svc_ce_hold_since, 0, sizeof(svc_ce_hold_since)); + memset(svc_ce_firstsub_off, 0, sizeof(svc_ce_firstsub_off)); + svc_fb_claim_reset(); svc_userd_sweep_log = 0; memset(svc_userd_sweep_at, 0, sizeof(svc_userd_sweep_at)); memset(svc_userd_sweep_n, 0, sizeof(svc_userd_sweep_n)); @@ -13835,7 +14592,14 @@ static bool svc_push_prefix_ok(const u8 *buf, size_t len) if (off + 4 > len) return true; /* all zeros: not refutable here */ op = (hdr >> 29) & 0x7; - if (op == 4 || op == 7) /* header-only methods */ + /* + * r715: op 0 (GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK) and op 2 + * (GRP2_USE_TERT) are 4-byte header-only formats per the class + * header; the GR submit pushes start with 0x0001fff0 + * (SET_SUBDEVICE_MASK), so a correct injected pin for such a page + * was rejected here and the read fell through to the walk. + */ + if (op == 0 || op == 2 || op == 4 || op == 7) /* header-only */ return true; if (op != 1 && op != 3 && op != 5) return false; @@ -13884,12 +14648,30 @@ static u32 svc_pushpg_split_log; * validated candidate). Reads stay defensively copied through * svc_read_ap()/vfio_dma_rw(), which enforce the aperture bounds. */ +/* r707d: bounded push-read failure witness (see below). */ +static u32 svc_pushfail_log; +/* + * r707e: last content-backed ring pair resolved by any channel. The VAS + * candidate verify is retried with it on EVERY pump pass: the parked + * COPY_SERVER_PDES candidate for the client's user VAS parks at + * device-create and must register before the submit's first push read + * (a failed GR entry read is never retried with a fresh VAS), but the ring + * resolution and the park race within one pass - the every-pass retry + * closes that race in a single boot. + */ +static u64 svc_lastgood_ring_va; +static u64 svc_lastgood_ring_pa; + + static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u64 va, void *buf, size_t len, bool skip_ipin) { u8 *p = buf; + /* r820: per-read; the decode reads it right after the call. */ + svc_read_last_promote = false; + while (len) { u64 page_va = va & ~(u64)(SEMA_PAGE_SIZE - 1); size_t chunk = min_t(size_t, len, @@ -13899,6 +14681,7 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, bool have_win, done = false; bool first = (p == buf); int last_rc = -ENOENT; + int trusted_rc = 0; u32 src; have_win = !svc_pb_win_lookup(ch, page_va, &win_pa); @@ -13952,7 +14735,9 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, default: /* learned cache / discovered map / page * directories / targeted rm_mem resolve */ - if (svc_lookup_trusted(vgpu, va, &pa, &ap)) + trusted_rc = svc_lookup_trusted(vgpu, va, + &pa, &ap); + if (trusted_rc) continue; break; } @@ -14013,10 +14798,47 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, */ if (src == 3 && first && !memchr_inv(p, 0, chunk)) continue; + svc_read_last_src = (u8)src; + svc_read_last_pa = pa; + /* + * r820: name whether the trusted answer came from a + * driver-declared PROMOTE_CTX buffer; the decode uses + * it for the zero-NOP acceptance (see the flag). + */ + if (src == 4) { + u64 promo_pa = 0; + int promo_ap = SVC_APERTURE_SYS; + + svc_read_last_promote = + !svc_promo_lookup(page_va, &promo_pa, + &promo_ap); + } done = true; } - if (!done) + if (!done) { + svc_read_last_src = 0xff; + if (svc_pushfail_log < 96) { + u64 ipa = 0, apa = 0; + int iap = 0, aap = 0; + bool has_ipin = + (svc_ipin_lookup(va, &ipa, &iap) == 0); + bool has_alt = + (svc_ipin_alt_lookup(va, &apa, &aap) == 0); + + svc_pushfail_log++; + pr_info("sema-pushfail: va=0x%llx chunk=0x%llx page=0x%llx last_rc=%d trusted_rc=%d ipin=%d pa=0x%llx alt=%d pa=0x%llx win=%d pa=0x%llx\n", + (unsigned long long)va, + (unsigned long long)chunk, + (unsigned long long)page_va, + last_rc, trusted_rc, has_ipin, + (unsigned long long)ipa, + has_alt, + (unsigned long long)apa, + have_win, + (unsigned long long)win_pa); + } return last_rc; + } p += chunk; va += chunk; len -= chunk; @@ -14081,6 +14903,22 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, : SVC_APERTURE_FB; /* + * r820 diag: the Xorg DMA-test window is 10 s and the latch hinges on + * this function's bounded scan re-runs (userd_scans < 8); a pass + * cadence stretched by a rebuild can spend every scan before the VAS + * candidate registers (r820: the ring resolved at +1 s but no latch + * line followed). One bounded row per call names the state the scan + * decides on. + */ + if (gsp_sema_dbg && !ch->engine_type && svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp h=0x%x eng=%u scans=%u rrc=%d ring_pa=0x%llx ring_ap=%d userd_pa=0x%llx as=%d ok=%u\n", + ch->h_object, ch->engine_type, ch->userd_scans, + rrc, ring_pa, ring_ap, ch->userd_pa, ch->userd_as, + ch->userd_ok ? 1 : 0); + } + + /* * ISSUE-0002 (r40): prefer the layout-derived CE USERD and accept it * while it is still idle. In a second session the registered userdMem * offset (pool-relative, see below) aliases a DIFFERENT rank's real @@ -14276,12 +15114,27 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * USERD cannot pass that test. One scan per channel (the result * persists through the pump's write-back), logged on latch. */ - if (!ch->engine_type && ch->userd_scans < 8 && ring_pa) { + /* + * r820: the latch scan is capped per channel so a channel that + * never anchors cannot sweep forever. But the resolution the scan + * needs can arrive LATE: the Xorg GL channel's ring resolves only + * when the client's VAS page directory registers (COPY_SERVER_PDES + * age-out), and r820 showed every one of the 8 pre-resolution scans + * spent before that moment (the pass cadence stretched by a + * rebuild), so the 10 s DMA-test window closed with the ring + * resolved and no retry left. Once ring_pa is non-zero the scan is + * cheap (the sweep validates against a resolved ring), so it may + * keep retrying up to a wider cap; the client's own timeout still + * bounds the channel's life. + */ + if (!ch->engine_type && ring_pa && + ch->userd_scans < (ring_pa ? 64u : 8u)) { u64 off, lim; u8 *base = vgpu->bars[1].vaddr; u64 best = 0, best_dist = ~0ULL; u32 best_put = 0, best_get = 0; bool best_pending = false; + u32 gate_n = 0, vfail_n = 0; ch->userd_scans++; lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); @@ -14292,23 +15145,33 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u32 k; u64 dist; bool pending; + int rck = -1; + u64 rkva = 0; + u32 rklen = 0; memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); if (!hp || hp > ch->gpfifo_entries || hg > hp) continue; + gate_n++; /* Validate: every claimed entry decodes to a * pushbuffer VA in the ring's window. */ for (k = 0; k < hp && k < 64; k++) { u8 e[8]; u64 eva; u32 elen, a0, a1; + int rdrc; - if (svc_read_va(vgpu, + rdrc = svc_read_va(vgpu, ch->gpfifo_va + (u64)k * 8, - e, sizeof(e))) { + e, sizeof(e)); + if (rdrc) { ok = false; + rck = (int)k; + /* -1: read failure rc; else the + * window check below. */ + rkva = (u64)rdrc; break; } memcpy(&a0, e, 4); @@ -14329,11 +15192,29 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, SVC_GR_RAM_WIN / 2 || elen < 4 || elen > 0x100000) { ok = false; + rck = (int)k; + rkva = eva; + rklen = elen; break; } } - if (!ok) + if (!ok) { + /* + * r820 diag: name why a scanned + * block was rejected (read rc or + * the entry VA/len window). + */ + vfail_n++; + if (gsp_sema_dbg && !ch->engine_type && + svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp scan off=0x%llx put=%u rejected k=%d (read_rc=%lld va=0x%llx len=%u)\n", + off, hp, rck, + (long long)rkva, + rkva, rklen); + } continue; + } /* * Several blocks can validate: a previous * session leaves its old USERD holding the same @@ -14373,6 +15254,17 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, best_pending ? "" : ", idle"); return 0; } + /* + * r820 diag: the sweep found nothing adoptable. gates + * counts put/get-coherent blocks seen, vfails the ones + * rejected by the ring-entry validation. + */ + if (gsp_sema_dbg && svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp sweep h=0x%x scans=%u gates=%u vfails=%u best=0x%llx\n", + ch->h_object, ch->userd_scans, + gate_n, vfail_n, best); + } } } return -ENOENT; @@ -14401,6 +15293,293 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * pump's completion clear before the normal path resumes. */ #define SVC_NOTIFIER_ERR_HOLD (2 * HZ) +/* + * SESSION 134: CE first-submission capture. + * + * The CE (copy-engine) rings are placed by the guest's allocator at a + * location that varies per boot and is NOT reliably described by the + * channel's registered gpfifo VA through the discovered page tables (r763 + * matched the fixed FB convention and the sweep decoded the inline-SASS + * entry; r765-r772 read zeros everywhere, the CE channels never latched, + * the driver's inline LAUNCH_DMA code downloads never executed and the + * frames stayed black). The logged instance where ring content and USERD + * were both live is the channel's first notifier clear - the guest's + * doorbell for its first submission, processed by the pump in the same + * pass that writes the ring. At that instant this hook: + * 1. reads entry0 through the channel's own translation (svc_lookup), + * 2. reads entry0 through the trusted walk (the VAS may have just + * landed since the registration read), + * 3. scans the shadow (0x100 step, first 64 MB, the GR-scan bound) for + * a ring-shaped page whose entry0 names a 1MB-aligned push VA in the + * CE window [0x120000000,0x121000000), and for every hit runs the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate + * whose USERD reads coherent (or idle 0/0) is PINNED and LATCHED + * exactly like svc_ring_autopin's CE path. + * Bounded: at most 4 captures per CE channel, 16 witness rows per boot; + * resets with the session (svc_session_flush). + */ +static u64 svc_ce_firstsub_log; + +/* + * SESSION 134: bounded completion hold for unlatched CE channels. + * + * The clear write in svc_notify_complete is what makes the guest treat the + * submission as consumed; the driver then recycles the ring and its content + * is gone before the client's VAS lands (sema_vas_cand_ms), leaving the CE + * ring latch with nothing readable (r769-r774: rrc=-2 at the first + * submission, the walk answering a zero page later). Holding the notifier + * busy for a bounded window keeps the driver's submission in flight - and + * with it the ring content that the regular per-pass latch path + * (svc_chan_userd through svc_lookup) needs - until the VAS registers and + * the ring reads coherently. 0 = off (the pre-SESSION-134 behaviour). + */ +static u64 svc_ce_hold_ms; +module_param_named(sema_ce_hold_ms, svc_ce_hold_ms, ullong, 0644); +MODULE_PARM_DESC(sema_ce_hold_ms, "Hold an unlatched CE channel's completion open for up to N ms so the ring stays live for the latch (0 = off)"); + +/* + * SESSION 134c rank-aware first-submission capture (see the capture below). + * The CE block convention: ring at 0x80000 + 0x20000 * rank, USERD at + * ring + 0x10000 (observed for every CE channel across boots; the same + * convention svc_chan_userd applies in its regular per-pass latch). + */ +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL + +/* + * One candidate offset for the CE first-submission capture: validate the + * ring shape (`scan_hits` counts shape hits) and, on a shape hit, run the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate whose + * USERD reads coherent (or idle 0/0) is PINNED and LATCHED exactly like + * svc_ring_autopin's CE path. Returns true when latched. + */ +static bool svc_ce_firstsub_try_off(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u8 *sbase, + u64 lim, u64 off, u32 *scan_hits, + u64 *latch_off, u64 *latch_upa, + u32 *latch_put, u32 *latch_get, + bool own_block) +{ + static const u64 udeltas[] = { 0x10000, 0x2000, 0x1000, 0x0 }; + u32 h0, h1, hlen; + u64 hva; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if ((hva & 0xfffff) || hva < 0x120000000ULL || + hva >= 0x121000000ULL || hlen < 4 || hlen > 0x10000) + return false; + (*scan_hits)++; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + int prc; + + if (upa + 4 > lim) + continue; + prc = svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3); + /* + * SESSION 144d: the rank's OWN convention block (own_block, + * the +0x10000 candidate at the rank's convention offset) is + * latched even when its stale content probes incoherent + * (g > p) or idle - the guest rewrites put/get on its next + * submission and the pump's per-channel last_get processes + * entry0 exactly once. r813: client #2's convention block at + * fb 0x90000 read put=1/get=146 (g > p -> -EINVAL) at the + * capture instant, the coherence gate rejected it, and the + * never-used 0x82000 zero block was latched instead, so the + * client's first push never drained and it hung. The gates + * below stay for every heuristic candidate (other deltas and + * the fallback scan), preserving the SESSION 135 protection. + */ + if (!own_block || ui != 0) { + if (prc && prc != -EAGAIN) + continue; + } + /* + * SESSION 135: at the FIRST-SUBMISSION instant the channel's + * submission is by definition unconsumed - the notifier clear + * this capture runs before is what would let the driver treat + * it as consumed. A candidate whose USERD reads a consumed + * state (get >= put && put > 0) cannot be this channel's live + * just-submitted work; it is a PREVIOUS session's leftover + * block surviving in VRAM (r785: the capture latched the dead + * CUDA client's caf00012 block at fb 0x80000, put=1 get=1, + * for the Vulkan channel beef0101 - the decoded entry was the + * old session's, the client stalled on its expectations). + * Idle 0/0 stays acceptable (svc_userd_probe's own rule). + */ + /* + * SESSION 144c: this consumed-state rejection is for FOREIGN + * blocks found by the heuristic deltas. The rank's OWN + * convention block (+0x10000) legitimately carries the + * previous session's consumed state until the guest + * re-initialises the reused allocation - rejecting it made + * the r812 second client fall through to a never-used + * 0x82000 block and stall forever (cefirst row: off=0x80000 + * upa=0x82000 put=0 get=0 while the client waited for the + * first push's completion). The pump's per-channel last_get + * still processes entry0 exactly once, so latching the + * convention block with stale put/get is correct. + */ + if (!own_block && p3 && p3 == g3) + continue; + /* latch exactly like the CE autopin */ + svc_pin_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + set_bit(ch->ce_index, &svc_ring_done); + svc_ring_n++; + *latch_off = off; + *latch_upa = upa; + *latch_put = p3; + *latch_get = g3; + return true; + } + return false; +} + +static void svc_ce_firstsub_capture(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 ring_pa = 0, wpa = 0; + u32 e0 = 0, e1 = 0, walk_e0 = 0, walk_e1 = 0; + u32 scan_hits = 0, latched = 0; + u64 latch_off = 0, latch_upa = 0; + u32 latch_put = 0, latch_get = 0; + int ring_ap = 0, wap = SVC_APERTURE_SYS; + int rrc = -1, wrc = -1; + + if (ch->engine_type == 0 || ch->ce_index >= 32) + return; + if (svc_ce_firstsub_n[ch->ce_index] >= 4) + return; + svc_ce_firstsub_n[ch->ce_index]++; + + if (sbase && lim >= 0x1000) { + /* 1. the channel's own translation at this instant. */ + rrc = svc_lookup(vgpu, ch->gpfifo_va, &ring_pa, &ring_ap); + if (!rrc && ring_pa) { + u8 e[8]; + + if (!svc_read_ap(vgpu, ring_pa, ring_ap, e, + sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + } + } + /* 2. the trusted walk at this instant. */ + if (!svc_lookup_trusted(vgpu, ch->gpfifo_va, &wpa, &wap)) { + u8 e[8]; + + wrc = 0; + if (!svc_read_ap(vgpu, wpa, svc_ap_of(wap), e, + sizeof(e))) { + memcpy(&walk_e0, e, 4); + memcpy(&walk_e1, e + 4, 4); + } + } + /* 3. shadow scan for a CE-window ring page. */ + { + /* + * SESSION 134c rank-aware order: try the rank's + * convention offset first, then the bounded scan, + * and never accept an offset another CE rank already + * latched. The rank-blind first-hit latched rank 0's + * block for rank 1 on r779 (both rows read + * off=0x80000 upa=0x90000), so the rank-1 channel's + * first release decoded from rank 0's ring/page and + * was stale-dropped (cur=1 already) - the client's + * rank-1 tracker wait never completed and it spun + * forever. A rank whose block is not in the shadow + * at this instant now simply stays unlatched for the + * regular per-pass CE latch instead of stealing a + * foreign block. + */ + u64 exp = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * ch->ce_index; + u64 off; + + latched = svc_ce_firstsub_try_off(vgpu, ch, sbase, + lim, exp, &scan_hits, + &latch_off, &latch_upa, + &latch_put, &latch_get, + true); + for (off = 0; !latched && off + 0x90 <= lim; + off += 0x100) { + unsigned int ri; + bool foreign = false; + + if (off == exp) + continue; + for (ri = 0; ri < 32; ri++) { + if (ri == ch->ce_index || + !svc_ce_firstsub_off[ri]) + continue; + if (svc_ce_firstsub_off[ri] == off) { + foreign = true; + break; + } + } + if (foreign) + continue; + /* + * SESSION 144: symmetric protection - a CE + * fallback scan must not steal a block another + * channel (CE rank or GR) already claims. + */ + if (svc_fb_claim_foreign(vgpu, ch->h_object, off, + off + 0x10000)) + continue; + latched = svc_ce_firstsub_try_off(vgpu, ch, + sbase, lim, + off, &scan_hits, + &latch_off, + &latch_upa, + &latch_put, + &latch_get, + false); + } + if (latched) { + svc_ce_firstsub_off[ch->ce_index] = latch_off; + /* + * SESSION 144: publish the claim so a GR rank + * whose own ring is not written yet cannot + * latch this CE block (r805-r808: every CUDA + * GR rank stole the CE rank-0 block and the + * launch was never captured). + */ + svc_fb_claim_add(ch->h_object, latch_off, + latch_upa); + } + } + } + + if (gsp_sema_dbg && svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u n=%u rrc=%d rpa=0x%llx rap=%d e0=0x%x e1=0x%x wrc=%d wpa=0x%llx wap=%d we0=0x%x we1=0x%x scan_hits=%u latch=%u off=0x%llx upa=0x%llx put=%u get=%u\n", + ch->h_object, ch->ce_index, + svc_ce_firstsub_n[ch->ce_index], rrc, ring_pa, + ring_ap, e0, e1, wrc, wpa, wap, walk_e0, walk_e1, + scan_hits, latched, latch_off, latch_upa, + latch_put, latch_get); + } + if (latched) + pr_info("sema-cefirst: LATCHED h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd 0x%llx put=%u get=%u; first-submission capture)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + latch_off, latch_upa, latch_put, latch_get); +} + static void svc_notify_complete(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -14453,6 +15632,41 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, return; /* + * SESSION 134: a visible doorbell means a submission just landed - + * for a CE-class channel this is the one instant its ring content + * and USERD are both live. Capture/snapshot it BEFORE the clear + * write below (the clear is what makes the guest treat the + * submission as complete and recycle the ring), see + * svc_ce_firstsub_capture. + */ + if (ch->engine_type) + svc_ce_firstsub_capture(vgpu, ch); + + /* + * SESSION 134: keep an unlatched CE submission open so the ring + * content survives until the latch can read it (see + * svc_ce_hold_ms). The regular per-pass latch path retries while + * the hold runs; the hold expires un-latched into the old + * behaviour, so a channel the emulation can never resolve cannot + * strand the guest. + */ + if (ch->engine_type && !ch->userd_ok && svc_ce_hold_ms && + ch->ce_index < 32) { + if (!svc_ce_hold_since[ch->ce_index]) + svc_ce_hold_since[ch->ce_index] = jiffies; + if (time_before(jiffies, + svc_ce_hold_since[ch->ce_index] + + msecs_to_jiffies(svc_ce_hold_ms))) + return; + if (svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u completion hold expired (%llu ms) unlatched; clearing\n", + ch->h_object, ch->ce_index, svc_ce_hold_ms); + } + svc_ce_hold_since[ch->ce_index] = jiffies; + } + + /* * Clear both slots: they are adjacent, and whichever one the client is * polling must read as completed for a healthy channel. */ @@ -14495,8 +15709,6 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, * against the ring's first GPFIFO entry, so a layout change costs a failed * pin, not a wrong one. */ -#define SVC_CE_RING_FB_BASE 0x80000ULL -#define SVC_CE_RING_FB_STRIDE 0x20000ULL static unsigned long svc_ring_done; /* CE ranks whose ring is pinned */ static bool svc_fbs_scan_done; /* one-shot FB entry scan diagnostic */ @@ -14532,6 +15744,62 @@ static int svc_chan_entry0(struct nv_vgpu_device *vgpu, struct svc_channel *ch, bool gsp_sema_gr_rings = true; /* + * SESSION 132 dead-entry guard for ring candidates. + * + * The r759 false positive: a vidmem candidate (0xa0000) whose first GPFIFO + * entry (push 0x120100000/40) passed the plausibility gate - the USERD + * probe hit the channel's REAL USERD at 0xb0000, which the VRAM shape scan + * had just found 0x10000 after the candidate by coincidence - was pinned + * as the GR ring while the client's own page tables resolve the ring VA to + * a guest-RAM page (0x11623d000) whose entry0 is live (push 0x4000000/ + * 0x8bc, decoded by the util sweep). The pump then read the pinned page, + * could not fetch the entry's push (all-zero through every source), and + * skipped the entry after 39 failed rescues - the live ring stayed + * undrained and every upload behind it invisible. + * + * A candidate whose first entry cannot deliver a method stream is not this + * channel's ring. The push head is read through the client's own page + * tables (guest truth); a push that cannot be resolved at all does NOT + * mark the entry dead (the VAS may not be registered yet on an early + * pass, and vidmem-backed pushes are not in the client's tables at all). + */ +static bool svc_ring_entry_dead(struct nv_vgpu_device *vgpu, u32 e0, u32 e1) +{ + u64 va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + u32 len = ((e1 >> 10) & 0x1fffff) << 2; + u8 head[16]; + + if (!va || len < 8 || len > 0x10000) + return false; /* unusable entry; the gate rejects it */ + if (svc_read_va_trusted(vgpu, va, head, sizeof(head))) + return false; /* cannot tell: do not reject */ + return !memchr_inv(head, 0, sizeof(head)); +} + +/* + * SESSION 132: does the client's own page table (VAS walk only - never the + * pin/lat/map sources, so a bad pin cannot vote for itself) resolve the + * ring VA to a page whose first entry is LIVE? Used to reject a dead + * candidate only when a live alternative provably exists. + */ +static bool svc_ring_walk_live(struct nv_vgpu_device *vgpu, u64 gpfifo_va) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 we[8]; + u32 w0, w1; + + if (svc_walk_vaspaces(vgpu, gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &wpa, &wap)) + return false; /* walk cannot resolve: no verdict */ + if (svc_read_ap(vgpu, wpa, wap, we, sizeof(we))) + return false; + memcpy(&w0, we, 4); + memcpy(&w1, we + 4, 4); + return !svc_ring_entry_dead(vgpu, w0, w1); +} + +/* * Validate a candidate translation for a GR ring: the first GPFIFO entry * must decode to a 1 MB aligned push in the client VA window with a sane * length, and the USERD at +SVC_GR_USERD_OFF must be readable through the @@ -14558,6 +15826,14 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, *len < 4 || *len > 0x10000) return false; /* + * SESSION 132: a candidate whose entry0 push provably reads all-zero + * through the client's own page tables cannot deliver a method + * stream (the pump's rescue path eventually skips exactly these) - + * reject it so the autopin falls through to a live candidate. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) + return false; + /* * Gate: the USERD must be readable. rc==0 is an active get/put * pair; rc==-EAGAIN is a readable idle (0/0) block, accepted so a * freshly created channel is not rejected. -EINVAL (implausible @@ -14567,6 +15843,20 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, *get = 0; prc = svc_userd_probe(vgpu, cand_pa + SVC_GR_USERD_OFF, cand_ap, ch->gpfifo_entries, put, get); + /* + * SESSION 132: the driver places the GPFIFO and the USERD + * independently (r759: ring in guest RAM 0x11623d000, USERD in + * vidmem 0xb0000), so a walk-resolved ring cannot pass a + * same-aperture USERD probe. Accept the channel's own USERD when + * one is already known - the put/get liveness gate is unchanged. + */ + if (prc != 0 && prc != -EAGAIN && ch->userd_ok && ch->userd_pa && + ch->userd_pa != cand_pa + SVC_GR_USERD_OFF) { + *put = 0; + *get = 0; + prc = svc_userd_probe(vgpu, ch->userd_pa, ch->userd_as, + ch->gpfifo_entries, put, get); + } return prc == 0 || prc == -EAGAIN; } @@ -15597,6 +16887,58 @@ static bool svc_gr_ram_ring_scan(struct nv_vgpu_device *vgpu, } /* + * r705h: host-triggered pdb discovery. The automatic latch + * (svc_vas_candidates_verify + svc_xvas_pdb_discover) only runs from the RAM + * ring scan, which sweeps [0,6GB) before the client's pushes exist and then + * gives up for the boot; and per-client VAS state is flushed at session end. + * This entry point lets a host tool that CONTENT-LOCATED a verified + * {ring VA -> ring PA} pair (the ring page's GPFIFO entries are its + * fingerprint) hand the pair to the service while the client is alive, so: + * 1. every parked COPY_SERVER_PDES candidate is verified against the pair + * (fast: a walk each), and + * 2. if none verifies, the vidmem walk-root sweep anchors on the pair. + * Registered VASes make the client's pushbuffer VAs walkable and the pump + * can then decode the submit's pushes (the fence release among them). + */ +static u64 svc_pdb_disc_va; +static u64 svc_pdb_disc_pa; +static bool svc_pdb_disc_pending; + +/* + * r707r: bounded channel rewind (debug). After the VAS pdb anchor lands, + * entries skipped before it never re-decode on their own (their unread + * window expires into a skip), so the release they carry would be lost for + * that put epoch. A host tool asks for a rewind (sema_chan_rewind: a hex + * h_object, or "all"); the pump re-processes the channel from slot 0 with + * the consumed-slot bitmap and the per-entry rescue state cleared. + */ +static u32 svc_rewind_h; +static bool svc_rewind_all; +static bool svc_rewind_pending; + +int gsp_sema_service_chan_rewind(u32 h_object) +{ + if (!svc.task) + return -ENODEV; + svc_rewind_h = h_object; + svc_rewind_all = (h_object == 0xffffffffu); + svc_rewind_pending = true; + wake_up_process(svc.task); + return 0; +} + +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa) +{ + if (!svc.task) + return -ENODEV; + svc_pdb_disc_va = ring_va & ~0xfffULL; + svc_pdb_disc_pa = ring_pa & ~0xfffULL; + svc_pdb_disc_pending = true; + wake_up_process(svc.task); + return 0; +} + +/* * r496f: discover a kernel-RM client VAS's page directory in the emulated * VRAM by verification. The CPU RM builds internal-VAS page tables in * vidmem and never publishes them (SET_PAGE_DIRECTORY is an externally @@ -15731,11 +17073,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, ch->gpfifo_entries, &put, &get)) continue; + /* + * SESSION 132: reject a dead candidate when + * the client's own page tables provably + * resolve the ring VA to a live page (the + * r759 vidmem false positive). + */ + if (svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va)) + continue; svc_pin_store(ch->gpfifo_va, cand, cap); svc_lat_store(ch->gpfifo_va, cand, cap); ch->userd_as = cap; ch->userd_ok = true; svc_gr_ring_base = cand; + if (cap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, cand, + ch->userd_pa); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (userd-derived d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -15766,13 +17120,17 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, len >= 4 && len <= 0x10000 && !svc_userd_probe(vgpu, pa + SVC_GR_USERD_OFF, SVC_APERTURE_FB, ch->gpfifo_entries, - &put, &get)) { + &put, &get) && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); ch->userd_pa = pa + SVC_GR_USERD_OFF; ch->userd_as = SVC_APERTURE_FB; ch->userd_ok = true; svc_gr_ring_base = base; + svc_fb_claim_add(ch->h_object, pa, + pa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -15804,7 +17162,9 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, (u64)(e0 & 0xfffffffc); len = ((e1 >> 10) & 0x1fffff) << 2; if (!(va & 0xfffff) && va >= SEMA_VA_LO && - va < SEMA_VA_HI && len >= 4 && len <= 0x10000) { + va < SEMA_VA_HI && len >= 4 && len <= 0x10000 && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { /* * Best effort only: a wrapped GP_PUT * legitimately exceeds the entry count, so a @@ -15819,10 +17179,20 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get); svc_pin_store(ch->gpfifo_va, pin_pa, pin_ap); svc_lat_store(ch->gpfifo_va, pin_pa, pin_ap); - ch->userd_pa = pin_pa + SVC_GR_USERD_OFF; - ch->userd_as = svc_ap_of(pin_ap); - ch->userd_ok = true; + /* r831: keep a registered real-PA userd. */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else { + ch->userd_pa = + pin_pa + SVC_GR_USERD_OFF; + ch->userd_as = svc_ap_of(pin_ap); + ch->userd_ok = true; + } svc_gr_ring_base = base; + if (svc_ap_of(pin_ap) == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, pin_pa, + pin_pa + + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -15854,10 +17224,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get)) { svc_pin_store(ch->gpfifo_va, wpa, wap); svc_lat_store(ch->gpfifo_va, wpa, wap); - ch->userd_pa = wpa + SVC_GR_USERD_OFF; - ch->userd_as = wap; - ch->userd_ok = true; + /* + * r831: the ring+0x2000 userd convention is a guess + * for the CUDA GR layout; a channel whose REGISTERED + * userd is a real PA (the RC watchdog) keeps its own + * (the candidate gate above probes it as fallback). + */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else { + ch->userd_pa = wpa + SVC_GR_USERD_OFF; + ch->userd_as = wap; + ch->userd_ok = true; + } svc_gr_ring_base = wpa; + if (wap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, wpa, + wpa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (trusted walk; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -15867,6 +17250,185 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, } } + /* + * SESSION 133: bounded shadow ring scan fallback. + * + * The CE-class rings live in VIDMEM and the FB convention only + * matches when the guest's VRAM allocator happens to place them at + * the conventional offsets (r763: matched, fb 0x80000/0xa0000, the + * channel decoded; r765: read zeros, the ring VA resolved to a + * stale RAM page whose entry0 was dead, the CE channel never + * latched and its inline SASS copies never decoded - black + * frames). When every deterministic candidate has failed, scan the + * shadow for a ring-shaped page: entry0 must name a 1 MB aligned + * push VA in the client window with a sane length, must not be + * provably dead (svc_ring_entry_dead - the r759 false-positive + * guard), and the USERD at the observed deltas (+0x10000 for the + * CE family, +0x2000 for GR) must read a coherent put/get. One + * bounded scan per rank per boot; a false candidate still costs + * only a failed probe. + */ + if (!test_bit(ch->gr_index, &svc_gr_ring_scan_done)) { + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 off; + + /* + * SESSION 133b: the rings are NOT page-aligned (r767: the + * USERD shape scan found the channel blocks at 0xfee00/ + * 0xff000/0xff200, so the rings sit at unaligned offsets like + * 0xefe00); step by 0x100 and RETRY (rate-limited) - a first + * pass can run before the client has written the ring or the + * USERD, and a consumed once-bit would never rescan. + */ + if (time_before(jiffies, (unsigned long)svc_gr_ring_scan_last + + msecs_to_jiffies(1500))) + return; + svc_gr_ring_scan_last = jiffies; + if (sbase && lim > 0x1000) { + static const u64 udeltas[] = { 0x10000, SVC_GR_USERD_OFF, + 0x1000 }; + + /* + * SESSION 133c pass 1: find the USERD by its OWN + * shape (put@+0x8c, get@+0x88 - the find_userd + * layout) and derive the ring from it. The USERD + * words appear at channel creation and stay; the + * ring entry may be written later, so this pass is + * the stabler trigger. Each hit is validated by the + * derived ring's entry0 (shape + not-dead), so a + * coincidental put/get pair costs a failed check. + */ + for (off = 0; off + 0x90 <= lim; off += 0x100) { + u32 up, ug; + unsigned int ui; + + memcpy(&up, sbase + off + 0x8c, 4); + if (up < 1 || up > 65535) + continue; + memcpy(&ug, sbase + off + 0x88, 4); + if (ug > up) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 roff; + u32 h0, h1, p3 = 0, g3 = 0; + u64 hva; + u32 hlen; + + if (off < udeltas[ui]) + continue; + roff = off - udeltas[ui]; + memcpy(&h0, sbase + roff, 4); + memcpy(&h1, sbase + roff + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if (!hva || (hva & 0xfffff) || + hva < SEMA_VA_LO || + hva >= SEMA_VA_HI || hlen < 4 || + hlen > 0x10000) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + /* + * SESSION 144: never latch a foreign + * block. Without this every CUDA GR + * rank stole the CE rank-0 block while + * its own ring was unwritten + * (r805-r808); staying unlatched here + * lets the FB-convention path latch the + * real ring once the client writes it. + */ + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + roff, off)) + continue; + if (svc_userd_probe(vgpu, off, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + ch->userd_pa = off; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = roff; + svc_fb_claim_add(ch->h_object, roff, off); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd-shape scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, roff, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + /* pass 2: ring-entry-shaped pages (below) */ + for (off = 0; off + 8 <= lim; off += 0x100) { + u32 h0, h1; + u64 hva; + u32 hlen; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if (!hva || (hva & 0xfffff) || + hva < SEMA_VA_LO || hva >= SEMA_VA_HI || + hlen < 4 || hlen > 0x10000) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + + if (upa + 4 > lim) + continue; + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + off, upa)) + continue; + if (svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = off; + svc_fb_claim_add(ch->h_object, off, + upa); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (shadow scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, off, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + } + } + if (!test_and_set_bit(ch->gr_index, &svc_gr_ring_tried)) { u8 d1[8] = { 0 }, d2[8] = { 0 }; int cap = ch->userd_as ? ch->userd_as : SVC_APERTURE_SYS; @@ -16075,6 +17637,13 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); + /* + * SESSION 144b: record the block ownership here too - CE ranks that + * latch through this plain autopin path (not the first-submission + * capture) were invisible to the GR fallback scan, which then stole + * their block (r809: CE rank 1 at fb 0xa0000). + */ + svc_fb_claim_add(ch->h_object, pa, pa + 0x10000); set_bit(ch->ce_index, &svc_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u)\n", @@ -18534,6 +20103,156 @@ static void svc_fwd_hold_direct_note(void) } /* + * SESSION 127 C5C0 translator: pending GRCE copy-out list. + * + * The Vulkan compute client submits its frame work in the pre-built + * continuation beyond GP_PUT: beec0100 idx23 carries the inline-QMD launch + * (0x31c = 0x4b85f, 64-word 0x320 run) and idx24 carries the TURING_DMA_COPY_A + * (0xC5B5, subch 4) image->buffer copies that un-tile the storage image into + * the client's sysmem readback buffer. Both pushes are only visible to the + * sema_gr_ahead sweep (see svc_gr_ahead_push), not to the put-bounded main + * walk. + * + * Copy ordering: the copy must read the image AFTER the forwarded launch + * completed (the engine's output is copied back into the VA shadow on the + * completion edge). The sweep only records the copy; it is executed inside + * svc_fwd_hold_gr_release, i.e. after the backend published done and after + * the copy-out, immediately before the launch's guest release word is + * authored - so the client's fence never observes a pre-render image. + * A bounded fallback run (sweep, when no forward is owed) keeps the list + * from stalling when the launch was refused. + */ +#define SVC_GR_COPY_PEND 8 +struct svc_gr_copy_pend { + bool valid; + u32 h_object; + u64 push_va; + u64 hash; + u64 src; + u64 dst; + u32 bytes; /* per-row byte count */ + u32 rows; + u32 dpitch; + u32 spitch; + u32 dma; /* LAUNCH_DMA word (remap/transfer bits, logging) */ + u32 remap; /* SET_REMAP_COMPONENTS (logging) */ +}; +static struct svc_gr_copy_pend svc_gr_copy_pends[SVC_GR_COPY_PEND]; +static u32 svc_gr_copy_pend_next; +static u64 svc_gr_copy_run_total; +static u64 svc_gr_copy_run_log; + +static void svc_gr_copy_pend_add(u32 h_object, u64 push_va, u64 hash, + u64 src, u64 dst, u32 bytes, u32 rows, + u32 dpitch, u32 spitch, u32 dma, u32 remap) +{ + u32 i; + + if (!bytes || !rows || !src || !dst) + return; + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + + if (p->valid && p->push_va == push_va && p->src == src && + p->dst == dst && p->bytes == bytes && p->rows == rows) + return; /* already recorded (idempotent re-sweep) */ + } + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + if (p->valid) + continue; /* keep live ones; take a free slot */ + p->valid = true; + p->h_object = h_object; + p->push_va = push_va; + p->hash = hash; + p->src = src; + p->dst = dst; + p->bytes = bytes; + p->rows = rows; + p->dpitch = dpitch ? dpitch : bytes; + p->spitch = spitch ? spitch : bytes; + p->dma = dma; + p->remap = remap; + pr_info("sema-gr: copy pend h=0x%x push=0x%llx src=0x%llx dst=0x%llx row=0x%x rows=%u dpitch=0x%x spitch=0x%x dma=0x%x remap=0x%x\n", + h_object, push_va, src, dst, bytes, rows, p->dpitch, + p->spitch, dma, remap); + return; + } + /* All slots live: overwrite the oldest position. */ + { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + pr_warn_ratelimited("sema-gr: copy pend overflow (dropping push=0x%llx)\n", + p->push_va); + memset(p, 0, sizeof(*p)); + } +} + +/* + * Execute and retire every pending GRCE copy. Called from the completion + * edge (svc_fwd_hold_gr_release) and, as a bounded fallback, from the sweep + * when no forward is owed. The executor is the same svc_copy() the CE/GCE + * op path uses (handles vidmem shadow sources and sysmem destinations). + */ +static void svc_gr_copy_pend_run(struct nv_vgpu_device *vgpu) +{ + u32 i; + + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + u64 done = 0; + int rc; + + if (!p->valid) + continue; + rc = svc_copy(vgpu, p->dst, p->src, p->bytes, 0, + SVC_APERTURE_SYS, 0, SVC_APERTURE_SYS, &done, + p->rows, p->dpitch, p->spitch, p->h_object, 0, + p->hash); + svc_gr_copy_run_total++; + if (rc == 0 || rc == SVC_COPY_FORWARDED) { + p->valid = false; + if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy done h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rows=%u rc=%d\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, p->rows, rc); + } + } else if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy retry h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rc=%d done=0x%llx\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, rc, done); + } + } +} + +/* + * Is this release VA owned by a live, not-yet-authored launch candidate + * captured from THIS push? (Only the launch's own push's release is + * withheld; a continuation fence entry elsewhere in the ring keeps the + * historical immediate-forward behaviour.) + */ +static bool svc_gr_ahead_owned_rel_va(u64 va, u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].rel_done && + svc_gr_cand_ring[ri].rel_va == va && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + +/* * Author a GR launch completion release (the QMD-level semaphore the * physical engine writes when the launch completes) in software, * forward-only. Used when a hold resolves without a successful hardware @@ -18581,6 +20300,14 @@ static void svc_fwd_hold_gr_release(struct nv_vgpu_device *vgpu, { if (!svc_fwd_req.gr) return; + /* + * SESSION 127 C5C0 translator: the launch is complete and its output + * pages were copied back into the VA shadow on this edge - run any + * pending GRCE copy-outs (image -> linear readback buffer) BEFORE + * authoring the client's release word, so the fence can never clear + * on a pre-render image. + */ + svc_gr_copy_pend_run(vgpu); svc_gr_sw_release(vgpu, ch ? ch->h_object : 0, svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload); @@ -18718,6 +20445,13 @@ static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) svc_fwd_req.gr_push_va, waited); return; } + /* + * SESSION 127 C5C0 translator: this edge authors the guest release + * without a hold - run any pending GRCE copy-outs first, exactly like + * the hold's success edge, so the fence cannot clear on a pre-render + * image. + */ + svc_gr_copy_pend_run(vgpu); svc_gr_sw_release(vgpu, svc_fwd_req.gr_h_object, svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload); @@ -20062,8 +21796,34 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) goto note_done; } } - if (svc_resolve_va(vgpu, page, &pa, &ap)) - return; + if (svc_resolve_va(vgpu, page, &pa, &ap)) { + /* + * SESSION 127 C5C0 translator (gated): a client-VAS page. The + * Vulkan + * QMD's program, parameter and descriptor pages live in the + * client's own page tables (vidmem aperture) and the trusted + * sources - pins, learned cache, discovered map - do not + * carry them, which made the fill refuse the launch with + * -ENOENT. Resolve through the registered-VAS walk instead: + * the exact path walk_probe proves (r707w). + */ + u64 wpa = 0; + u32 wap = SVC_APERTURE_SYS; + + if (!gsp_sema_gr_xlate || + svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + return; + pa = wpa; + ap = svc_ap_of(wap); + svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "vas"; + goto note_done; + } svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); /* @@ -20551,6 +22311,14 @@ static void svc_gr_scan_cand(u64 *wins, u32 *n_cand, const u8 *data, u32 bytes) if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* SESSION 139: float-shaped words are scalars, not device VAs + * (see the scan-site note; r789's 480.0f = 0x43f00000). */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < *n_cand; w++) if (wins[w] == wv) @@ -21056,18 +22824,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; if (pp->qmd_word != 32u || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } for (k = 0x150u / 8u; k < 256; k++) { u64 v = get_unaligned_le64(page + k * 8); u64 wv; @@ -21075,6 +22858,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -21109,18 +22909,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; if (pp->qmd_word != 32u || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { u64 v, wv; @@ -21130,6 +22945,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -21142,19 +22974,38 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; - if (pp->qmd_word < 32 || pp->qmd_word > 47 || + /* SESSION 139: include word 48 (see the r788 finding: the + * word-34 pointer dedupes into the program's 4K page whose + * entry carries qmd_word=48; the old 32..47-even filter left + * that page unscanned). */ + if (pp->qmd_word < 32 || pp->qmd_word > 48 || (pp->qmd_word & 1) || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } /* * relfix PART 3: whole page, matching the rewrite's scan * range (it walks 0x0..0xfff qword-pairs); the old 0..0x7ff @@ -21167,6 +23018,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -21179,6 +23047,58 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, svc_gr_scan_inline(wins, &n_cand, pp->va); } /* + * SESSION 140 (the n_data wiring): seed the candidate set with the + * launch's OWN QMD pointer pairs (words 34..47, already decoded into + * gr_qmd from the push's inline-QMD run). These ARE the launch's + * data-window pointers - the r792 evidence: the capture holds + * ptr[0]=0xdff0febc00 but the fill's page scans read only empty + * shadow pages (the QMD page's vidmem shadow is zero; its content + * never arrives - the invisible-vidmem-write class), so the pointer + * values could never become candidates and n_data stayed 0 (refused + * degenerate). The values need no page scan - use them directly. + */ + { + u32 w2; + + for (w2 = 0; w2 + 1 < 16; w2 += 2) { + u64 pv = (u32)svc_fwd_req.gr_qmd[34 + w2] | + ((u64)(svc_fwd_req.gr_qmd[35 + w2] & + 0x1ffffu) << 32); + u64 pw; + u32 fexp; + + if (pv < 0x10000000ULL || pv >= (1ULL << 47) || + (pv & 3)) + continue; + fexp = (u32)(pv >> 23) & 0xffu; + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + pw = pv & ~(u64)(SEMA_PAGE_SIZE - 1); + for (w = 0; w < n_cand; w++) + if (wins[w] == pw) + break; + if (w == n_cand && n_cand < ARRAY_SIZE(wins)) { + wins[n_cand++] = pw; + is_arg[n_cand - 1] = true; + if (svc_gr_pages_log < 64) { + svc_gr_pages_log++; + pr_info("sema-gr: qmd-ptr seed va=0x%llx (word %u) h=0x%x\n", + pw, 34 + w2, + svc_fwd_req.gr_h_object); + } + } + } + } + if (gsp_sema_dbg && svc_gr_pages_log < 48) { + svc_gr_pages_log++; + pr_info("sema-gr: fill cand n=%u pages=%u push=0x%llx\n", + n_cand, svc_fwd_req.gr_n_pages, + svc_fwd_req.gr_push_va); + for (w = 0; w < n_cand && w < 12; w++) + pr_info("sema-gr: win[%u] va=0x%llx arg=%u\n", + w, wins[w], is_arg[w]); + } + /* * r635: the launch-level CONTENT anchor for the host-side fallback, * computed BEFORE the span loop so candidate order cannot decide. * An argument candidate whose span already holds the client's own @@ -21817,7 +23737,12 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, u64 qpa = 0; u32 qap = SVC_APERTURE_SYS; - if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap)) { + if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap) || + (gsp_sema_gr_xlate && + !svc_walk_vaspaces(vgpu, + svc_gr_cand.qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + &qpa, &qap))) { svc_fwd_req.gr_qmd_pa = qpa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_qmd_ap = (u8)svc_ap_of(qap); @@ -21877,12 +23802,42 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_gr_page_note(vgpu, svc_gr_cand.param_va, 32); if (svc_gr_cand.prog_va) svc_gr_page_note(vgpu, svc_gr_cand.prog_va, 48); + /* + * SESSION 140: the QMD page itself carries the launch's pointer + * VALUES (QMD words 32..49 live inside it - the param, the + * pointer-region pairs and the program). Without this note the fill + * never scans it, so the pointer values (r790: word 34 = + * 0xdff0febc00) can never become candidates and n_data stays 0 + * (refused: degenerate data set). Noted under word 34 so the + * 32..48-even bank-page scan covers it. + */ + if (svc_gr_cand.qmd_dev_va) + svc_gr_page_note(vgpu, svc_gr_cand.qmd_dev_va, 34); for (i = 0; i < svc_gr_cand.n_pointer; i++) svc_gr_page_note(vgpu, svc_gr_cand.pointer_va[i], svc_gr_cand.pointer_word[i]); for (i = 0; i < svc_gr_cand.n_inline; i++) svc_gr_page_note(vgpu, svc_gr_cand.inline_recs[i].dst_va, SVC_GR_QMD_WORD_NONE); + if (gsp_sema_dbg && svc_gr_pages_log < 24) { + svc_gr_pages_log++; + pr_info("sema-gr: cand h=0x%x param=0x%llx prog=0x%llx n_ptr=%u inline=%u n_pages=%u push=0x%llx\n", + svc_gr_cand.h_object, svc_gr_cand.param_va, + svc_gr_cand.prog_va, svc_gr_cand.n_pointer, + svc_gr_cand.n_inline, svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + for (i = 0; i < svc_gr_cand.n_pointer && i < 8; i++) + pr_info("sema-gr: ptr[%u] va=0x%llx word=%u\n", i, + svc_gr_cand.pointer_va[i], + svc_gr_cand.pointer_word[i]); + for (i = 0; i < svc_fwd_req.gr_n_pages && i < 12; i++) { + const struct svc_gr_page *pg = + &svc_fwd_req.gr_pages[i]; + + pr_info("sema-gr: page[%u] va=0x%llx pa=0x%llx ap=%u qmd_word=%u\n", + i, pg->va, pg->pa, pg->ap, pg->qmd_word); + } + } if (svc_fwd_req.gr_page_overflow || !svc_fwd_req.gr_n_pages) return -E2BIG; for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { @@ -21890,10 +23845,19 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, if (pg->pa) continue; - if (pg->qmd_word == SVC_GR_QMD_WORD_NONE) - return -ENOENT; - if (!svc_gr_page_inline_covers(pg->va)) + if (pg->qmd_word == SVC_GR_QMD_WORD_NONE || + !svc_gr_page_inline_covers(pg->va)) { + if (svc.fwd_gr_enoe_log < 8) { + svc.fwd_gr_enoe_log++; + pr_info("vgsp-work: GR fill ENOENT va=0x%llx qmd_word=%u inline=%u covered=%d n_pages=%u push=0x%llx\n", + pg->va, pg->qmd_word, + svc_gr_cand.n_inline, + svc_gr_page_inline_covers(pg->va), + svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + } return -ENOENT; + } } /* * GRFIX FIX1c: the fill hosts the self-enforcing decision point and @@ -22480,11 +24444,25 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, } } if (!gsp_phys_gr_forward_enabled() || !svc_gr_cand.valid || - svc_gr_cand.h_object != ch->h_object) + svc_gr_cand.h_object != ch->h_object) { + if (svc.fwd_gr_nocand_log < 8) { + svc.fwd_gr_nocand_log++; + pr_info("vgsp-work: GR request without candidate h=0x%x (forward=%d cand_valid=%d cand_h=0x%x offered=%d)\n", + ch->h_object, gsp_phys_gr_forward_enabled(), + svc_gr_cand.valid, svc_gr_cand.h_object, + svc_gr_cand.offered); + } return false; + } svc_gr_data_stalled = false; /* m150: only the fill sets this */ svc.fwd_gr_req++; if (!gsp_phys_backend_gr_ready() || !gsp_phys_guest_mem_enabled()) { + svc.fwd_gr_nordy++; + if ((svc.fwd_gr_nordy & 0x3F) == 1) + pr_info("vgsp-work: GR request refused: backend not ready h=0x%x ready=%d gmem=%d n=%llu\n", + ch->h_object, gsp_phys_backend_gr_ready(), + gsp_phys_guest_mem_enabled(), + svc.fwd_gr_nordy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; svc_gr_bridge_fallback(vgpu); @@ -22495,6 +24473,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * request's done/hw_rc are still being read. The capture falls back. */ if (svc_fwd_hold.active) { + svc.fwd_gr_holdbusy++; + if ((svc.fwd_gr_holdbusy & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: hold active h=0x%x (hold ch=0x%x n=%llu)\n", + ch->h_object, svc_fwd_hold.ch_handle, + svc.fwd_gr_holdbusy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; /* @@ -22515,6 +24498,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, if (svc_fwd_paused || test_and_set_bit_lock(SVC_FWD_INFLIGHT, &svc_fwd_state)) { mutex_unlock(&svc_fwd_lock); + svc.fwd_gr_inflight++; + if ((svc.fwd_gr_inflight & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: inflight h=0x%x (state=0x%lx n=%llu)\n", + ch->h_object, svc_fwd_state, + svc.fwd_gr_inflight); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; if (!gsp_cuda_bridge_enabled()) { @@ -22541,7 +24529,8 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; - if ((svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) + if (svc.fwd_gr_refused < 8 || + (svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) pr_info("vgsp-work: GR forward request refused rc=%d h=0x%x push=0x%llx; bridge fallback\n", rc, ch->h_object, svc_gr_cand.push_va); svc_gr_bridge_fallback(vgpu); @@ -23508,8 +25497,1148 @@ static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, put, get, ch->submissions, waited, reason); } +/* + * --------------------------------------------------------------------- + * SESSION 126 - FENCE FOLD: channel-anchored sweep of a GR-class + * channel's PRE-BUILT ring continuation. + * + * The Vulkan client's fence waits never clear unaided because the + * driver's kick boundary (GP_PUT) stops at the first batch while the ring + * already carries the continuation entries it pre-built for the next + * submits - and the fence's own INC-5 RELEASE group (observed + * [2005c017][0x45ba090][0][payload 1][0][0x02100001]) lives in one of + * those entries, so the pump's GP_PUT-bounded walk never decodes it + * (r715/r716: put=11, entries 12..47 carried content, no dmesg row ever + * mentioned 0x45ba). The host-side fenceword_assist wrote the wait word + * directly; this fold makes the service author it from the driver's own + * push bytes. + * + * Doctrine: unlike the disabled page-content sweep (svc_apply_page_data, + * off since the false-positive corruption incidents), this sweep is + * CHANNEL-ANCHORED: it only walks entries reachable through the ring's + * validated GPFIFO translation, only accepts entries whose bytes pass the + * strict method-stream gate, and only applies the semaphore release + * groups it finds - forward-only, through the standard pending ledger + * (svc_pending_add). No consumption bookkeeping is touched (last_get, + * put_prev, slot_done_epoch and the replay state stay as the pump left + * them), so a later real kick of the same entries decodes them normally; + * repeated sweeps are idempotent (the pending ledger dedups (va,payload) + * and the apply path is forward-only). + * + * Gated by the same live-epoch arm as the counter publish + * (ch->gr_pub_armed) so a residual ring from a previous session cannot + * author releases before the session switch is proven. + * --------------------------------------------------------------------- + */ +unsigned gsp_sema_gr_ahead_max = 64; +module_param_named(sema_gr_ahead, gsp_sema_gr_ahead_max, uint, 0644); +MODULE_PARM_DESC(sema_gr_ahead, + "GR-class channels: max GPFIFO entries beyond GP_PUT whose channel-anchored pushes are swept for semaphore releases (0=off)"); + + +static u32 svc_gr_ahead_log; +static u64 svc_gr_ahead_entries; +static u64 svc_gr_ahead_rels; +static u64 svc_gr_ahead_skips; +/* SESSION 127: releases withheld because a live candidate owns the word. */ +static u64 svc_gr_ahead_owned; +/* SESSION 127: bounded witness for the sweep-caller's launch request. */ +static u32 svc_gr_ahead_req_log; +/* SESSION 128: inline (LOAD_INLINE_DATA) writes authored by the sweep. */ +static u64 svc_gr_ahead_inline_writes; + +/* + * SESSION 129 util-channel witness: the utility channel's subchannel-2 + * LOAD_INLINE_DATA arm never fired (writes=0 on r737) while the same arm + * fired for in-push payloads on the GR channel - i.e. the failure sits in + * the split (tail-served) shape. This struct carries the per-entry story + * out of the translator for one bounded, deduplicated witness line: + * which read source answered (attempt A = the channel path, 1 = the direct + * registered-VAS walk), whether the strict parse accepted the entry, and + * what the LID arm did (saw/fired/refused a still-zero payload). + */ +struct svc_util_wit { + u32 p_src; /* attempt that won: 0 = channel, 1 = VAS walk, 0xff none */ + u32 pa_ok; /* attempt-A parse (method count) */ + u32 pb_ok; /* attempt-B parse (method count) */ + u32 t_src; /* tail read source: 0 = channel, 1 = walk, 0xff none */ + u32 t_len; /* tail length served */ + u32 t0, t1; /* SESSION 130: first two words of the served tail */ + bool saw_lid; /* a LOAD_INLINE_DATA arm was reached */ + bool fired; /* ... and authored its write */ + bool zero; /* ... but the tail payload was still all-zero (pending) */ + u32 dst; + u32 line_len; + u32 line_lc; + u32 count; +}; +static struct svc_util_wit svc_util_wit; +static u64 svc_util_pending_zero; + +/* + * SESSION 127 C5C0 translator, bounded tail materialization buffer: a + * LOAD_INLINE_DATA run whose payload overruns the entry end is served from + * the next ring entry (the driver splits large inline runs into a methods + * entry plus a dedicated data slot entry). 4 KB covers the observed + * param-bank write (1568 B) in full and the first page of the big uploads. + */ +#define SVC_AHEAD_TAIL_INLINE_MAX 32768 +static u8 svc_ahead_inline_buf[SVC_AHEAD_TAIL_INLINE_MAX]; +static u64 svc_ahead_inline_trunc; + +/* + * SESSION 128: one-copy materialization bound for the utility/CE sweep. + * The engine-side copy-in the sweep replays (a TURING_DMA_COPY_A pitch + * copy from a client-owned staging buffer into the vidmem shadow) moves + * the driver's code/constant uploads; the observed SASS/code-heap class is + * tens of KB, so 4 MB is a generous cap that still refuses a bogus + * geometry before it can walk the shadow. + */ +#define SVC_AHEAD_COPY_IN_MAX (4ULL << 20) + +/* + * SESSION 128: materialize one TURING_DMA_COPY_A pitch copy-in that an + * unprocessed copy-class channel carried - src bytes live in the client's + * staging buffer (guest RAM, resolved through the trusted walk like any + * operand page), dst is the vidmem page the driver staged the upload for + * (code heap / constant store). This is the emulated engine write for the + * copy the real CE would perform; it is once-only keyed by (push_va, + * content hash, dst, bytes) so re-scans of a stable ring are free. + */ +static u64 svc_ahead_copy_in_skip; +static void svc_ahead_copy_in(struct nv_vgpu_device *vgpu, u32 ch_h, + u64 push_va, u64 hash, u64 src_va, u64 dst_va, + u32 row, u32 rows, u32 dma) +{ + static u8 cb[65536]; + u64 bytes = (u64)row * (rows ? rows : 1); + u64 done = 0; + u64 spa = 0, dpa = 0; + int sap = SVC_APERTURE_SYS, dap = SVC_APERTURE_SYS; + int rc = 0; + bool src_ok, dst_ok; + + if (!bytes || src_va < SVC_LOW_REL_VA_LO || src_va >= SEMA_VA_HI || + dst_va < SVC_LOW_REL_VA_LO || dst_va >= SEMA_VA_HI) + return; + /* + * Only the upload direction is materialized: the source must be a + * resolved client page in sysmem/guest RAM (the staging buffer the + * client CPU-wrote), and the destination must be vidmem (resolved to + * the FB aperture) or a not-yet-resolved VA whose bytes the VA + * shadow will hold. A resolved SYS destination is a copy-OUT (the + * engine reading vidmem into a client buffer): replaying it here + * would overwrite the client's buffer with shadow garbage before the + * producing launch ran, so it is skipped - the GR channels' copy-out + * machinery owns that direction. + */ + src_ok = !svc_lookup(vgpu, src_va, &spa, &sap) && sap != SVC_APERTURE_FB; + dst_ok = svc_lookup(vgpu, dst_va, &dpa, &dap) || + dap == SVC_APERTURE_FB; + if (!src_ok || !dst_ok) { + svc_ahead_copy_in_skip++; + if (svc_ahead_copy_in_skip <= 8) + pr_info("sema-utilsweep: copy-in skipped (src_ok=%d dst_ok=%d) push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx\n", + src_ok, dst_ok, push_va, src_va, dst_va, + bytes); + return; + } + if (bytes > SVC_AHEAD_COPY_IN_MAX) { + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-utilsweep: copy-in too large push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx cap=0x%llx\n", + push_va, src_va, dst_va, bytes, + SVC_AHEAD_COPY_IN_MAX); + return; + } + if (svc_inline_once_seen(push_va, hash, dst_va, (u32)bytes)) + return; + svc_inline_once_store(push_va, hash, dst_va, (u32)bytes); + while (done < bytes) { + u32 chunk = bytes - done > sizeof(cb) ? + (u32)sizeof(cb) : (u32)(bytes - done); + + rc = svc_read_va(vgpu, src_va + done, cb, chunk); + if (rc) { + rc = -EIO; + break; + } + rc = svc_grce_inline_write(vgpu, dst_va + done, cb, chunk, + ch_h); + if (rc) + break; + done += chunk; + } + svc_gr_ahead_inline_writes++; + pr_info("sema-utilsweep: copy-in src=0x%llx -> dst=0x%llx bytes=0x%llx dma=0x%x done=0x%llx rc=%d h=0x%x\n", + src_va, dst_va, bytes, dma, done, rc, ch_h); +} + +/* One 4-byte word of a method stream that may continue into the tail. */ +static u32 svc_ahead_w32(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + u32 off) +{ + u32 v = 0; + + if (off + 4 <= len) + memcpy(&v, buf + off, 4); + else if (tail && off >= len && (u64)(off - len) + 4 <= taillen) + memcpy(&v, tail + (off - len), 4); + return v; +} + +/* + * Strict method-stream parse with a one-entry tail (see the sweep comment). + * A run whose payload overruns this entry's end may be completed from the + * next ring entry; anything else invalid means the page is not a method + * stream (the same false-positive arbiter as svc_stream_full_ok). Returns + * the method count, or 0; *used_tail reports that the tail served a + * trailing data run (nothing after it is parsed). + */ +static u32 svc_ahead_parse(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + bool *used_tail) +{ + u32 off = 0, methods = 0; + + *used_tail = false; + for (;;) { + u32 hdr, op, count; + u64 end; + + if (off + 4 > len) + break; + memcpy(&hdr, buf + off, 4); + if (hdr == 0) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + count = (hdr >> 16) & 0xfff; + switch (op) { + case 1: /* INC_METHOD */ + case 3: /* NON_INC_METHOD */ + case 5: /* ONE_INC */ + if (!count || count > 0xfff) + return 0; + end = (u64)off + 4 + (u64)count * 4; + if (end > len) { + if (!tail || (u64)(end - len) > taillen) + return 0; + *used_tail = true; + off = len; + goto done; + } + off = (u32)end; + break; + case 0: /* GRP0_USE_TERT */ + case 2: /* GRP2_USE_TERT */ + case 4: /* IMMD_DATA_METHOD */ + case 7: /* END_PB_SEGMENT */ + off += 4; + break; + default: + return 0; + } + if (++methods > 8192) + return 0; + } +done: + while (off < len && buf[off] == 0) + off++; + return (methods > 0 && (off == len || *used_tail)) ? methods : 0; +} + +/* + * SESSION 127 C5C0 translator sweep pass over one coherent push. Besides the + * original release extraction (svc_gr_ahead_push, below), it recognizes, for + * pre-built continuation entries: + * + * - the C5C0 inline-QMD window (SET_INLINE_QMD_ADDRESS_A/B + the 64-word + * LOAD_INLINE_QMD_DATA run) and feeds it to the standard svc_qmd_accum + * capture, so the physical GR forward (the CUDA QMD path) fires for a + * launch that never reaches the GP_PUT-bounded main walk - observed + * beec0100 idx23: `0x31c = 0x4b85f` + `0x320 x 64`; + * - GRCE / KEPLER_INLINE_TO_MEMORY_B LOAD_INLINE_DATA runs (op 3 at + * 0x1b4, offsets from the 0x180/0x184/0x188/0x18c group), materialized + * into the VA shadow exactly like the main decode does - observed + * beef0100 idx00 (256 B -> 0x423f000, payload in idx01) and idx08 + * (12288 B -> 0x4448000, payload in idx09), and beec0100 idx23 + * (1568 B -> 0x4b86020, payload in-push); + * - TURING_DMA_COPY_A (0xC5B5, subch >= 2) copy operands + * (0x400/0x404 in, 0x408/0x40c out, 0x410/0x414 pitch, 0x418/0x41c + * line length/count, 0x300 LAUNCH_DMA, 0x700..0x708 remap) recorded as + * pending copy-outs for the completion edge (svc_gr_copy_pend_run) - + * observed beec0100 idx24: image 0xdfd8d60000 -> readback 0x050b0000, + * 640x480 rows. + * + * Returns the number of release groups queued (as before). The buffer must + * pass the strict tail-aware parse first; a page that is not a method + * stream is not processed at all. + */ +static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + const u8 *buf, u32 len, const u8 *tail, + u32 taillen, u64 push_va, bool inline_only) +{ + static struct svc_push_state stq; /* service thread only */ + static u32 dw[66]; /* QMD window words */ + u32 off = 0, nrel = 0, methods, i; + u64 off_in = 0, off_out = 0; + u32 have_in = 0, have_out = 0; + u32 pitch_in = 0, pitch_out = 0; + u32 line_len = 0, line_count = 0; + u32 last_dma = 0, last_remap = 0; + u32 cp_subch = ~0u; + bool used_tail = false; + bool forward = gsp_phys_gr_forward_enabled(); + bool xlate = gsp_sema_gr_xlate; + + if (!xlate) { + /* Historical sweep: releases from exactly-covered entries. */ + tail = NULL; + taillen = 0; + } + methods = svc_ahead_parse(buf, len, tail, taillen, &used_tail); + if (!methods) + return 0; + + memset(&stq, 0, sizeof(stq)); + stq.push_va = push_va; + + while (off + 4 <= len) { + u32 hdr, op, addr, count, subch; + u32 w0, w1, w2, w3, w4; + u64 end; + + memcpy(&hdr, buf + off, 4); + if (!hdr) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + addr = (hdr & 0x1fff) << 2; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + if (op == 0 || op == 2 || op == 4 || op == 7) { + off += 4; + continue; + } + if (op != 1 && op != 3 && op != 5) + break; + if (!count || count > 0xfff) + break; + end = (u64)off + 4 + (u64)count * 4; + /* + * Tail-served trailing run: process from the head word, then + * stop (the tail is a data slot, nothing parses after it). + */ + w0 = svc_ahead_w32(buf, len, tail, taillen, off + 4); + w1 = svc_ahead_w32(buf, len, tail, taillen, off + 8); + w2 = svc_ahead_w32(buf, len, tail, taillen, off + 12); + w3 = svc_ahead_w32(buf, len, tail, taillen, off + 16); + w4 = svc_ahead_w32(buf, len, tail, taillen, off + 20); + + /* + * C5C0 inline-QMD window (mirrors the main decode's intercept): + * only accumulate once the run declares itself a QMD window. + */ + if (xlate && forward && !inline_only && subch >= 1 && + addr >= SVC_QMD_ADDR_A && + addr <= SVC_QMD_DATA + (SVC_QMD_WORDS - 1) * 4 && + (stq.qmd_have || + (addr <= SVC_QMD_ADDR_B && + (u64)addr + (u64)count * 4 > SVC_QMD_ADDR_A))) { + u32 n = count > 66 ? 66 : count; + + for (i = 0; i < n; i++) + dw[i] = svc_ahead_w32(buf, len, tail, taillen, + off + 4 + 4 * i); + svc_qmd_accum(vgpu, ch, &stq, addr, n, + (const u8 *)dw, buf, len, true, false); + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY_B LOAD_INLINE_DATA: materialize the + * DMA write (the engine would do this; the emulation authors + * the bytes into the VA shadow so the forwarded launch's + * operand pages - parameter banks - are real). + */ + if (xlate && op == 3 && subch >= 1 && + addr == GRCE_M_LOAD_INLINE_DATA && + have_out && line_len) { + u64 nbytes = (u64)line_len * + (line_count ? line_count : 1); + u32 cap = count * 4; + u32 bytes, k, words; + u64 h = svc_push_hash(buf, len); + bool tail_served = end > len; + bool refuse_zero = false; + + if (nbytes > cap) + nbytes = cap; + bytes = (u32)nbytes; + if (bytes > SVC_AHEAD_TAIL_INLINE_MAX) { + bytes = SVC_AHEAD_TAIL_INLINE_MAX; + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-ahead: inline write truncated push=0x%llx dst=0x%llx want=%llu cap=%u\n", + push_va, off_out, nbytes, + SVC_AHEAD_TAIL_INLINE_MAX); + } + words = (bytes + 3) / 4; + for (k = 0; k < words; k++) + memcpy(svc_ahead_inline_buf + 4 * k, + &(u32){ svc_ahead_w32(buf, len, tail, + taillen, + off + 4 + 4 * k) }, + 4); + /* + * SESSION 129: a tail-served payload whose head is still + * all zeros means the tail source answered with an empty + * page (a stale pin shadow, or the driver has not staged + * the bytes yet) - authoring zeros over the destination + * would corrupt the code heap. Leave the write pending + * (no once-store, the sweep retries every pass) until a + * nonzero payload arrives. In-push payloads are never + * gated: the bytes are part of the flushed submit. + */ + if (inline_only && tail_served && bytes && + !memchr_inv(svc_ahead_inline_buf, 0, + min_t(u32, bytes, 16))) + refuse_zero = true; + if (inline_only) { + svc_util_wit.saw_lid = true; + svc_util_wit.dst = (u32)off_out; + svc_util_wit.line_len = line_len; + svc_util_wit.line_lc = line_count; + svc_util_wit.count = count; + if (refuse_zero) + svc_util_wit.zero = true; + } + if (refuse_zero) { + /* + * SESSION 130: a zero-headed tail-served + * payload is applied under a VARIANT once-key + * (h | 1) instead of being refused forever: + * the destination is currently empty, so + * authoring zeros is idempotent and correct + * for the driver's internal zero buffer (the + * observed 256 B utility i2m = exactly + * GF100_ZERO_BUFFER_SIZE), while a later + * non-zero payload for the same entry still + * matches its own (h) key and lands - the + * refusal can no longer stall the arm + * (r737/r739: `writes=0` with the payload + * page stably zero). + */ + svc_util_pending_zero++; + if (bytes && !svc_inline_once_seen(push_va, + h | 1, + off_out, + bytes)) { + svc_inline_once_store(push_va, h | 1, + off_out, bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, + off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + } else if (bytes && !svc_inline_once_seen(push_va, h, + off_out, + bytes)) { + svc_inline_once_store(push_va, h, off_out, + bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + /* + * Record the destination and the fresh payload bytes + * for the launch fill's inline-served pages, exactly + * like the main decode's LOAD_INLINE_DATA arm: the + * QMD capture later in this entry collects the ring + * and the fill can serve the parameter bank from it + * even while its shadow page is still empty. + */ + { + struct svc_grce_rec *rec = + &svc_grce_dst_ring[svc_grce_dst_ring_next % + SVC_GRCE_DST_MAX]; + + /* + * A refused (still-zero) tail payload is not + * recorded either: the next pass re-reads the + * tail and the real bytes land then. + */ + if (!refuse_zero) { + rec->dst_va = off_out; + rec->len = min_t(u32, bytes, + GSP_BRIDGE_INLINE_BYTES); + memcpy(rec->data, svc_ahead_inline_buf, + rec->len); + svc_grce_dst_ring_next++; + } + } + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY operand group (0x180..0x18c), for + * subchannel 1 AND 2. The mem2mem subchannel (2, the + * A140/KEPLER_INLINE_TO_MEMORY_B upload lane the utility + * channels drive) carries the same operand methods as the + * compute subchannel (1). This group MUST be parsed BEFORE + * the TURING-copy branch below: that branch's `subch >= 2` + * guard also matches subchannel 2 and would swallow the + * 0x180..0x18c methods unread, leaving have_out/line_len + * unset - the exact reason the utility channel's subch-2 + * LOAD_INLINE_DATA arm never fired (r737/r739 witness: + * `lidn=1 subch=2 ... saw=0 dst=0 ll=0`). + */ + if (xlate && (subch == 1 || subch == 2) && + addr >= 0x180 && addr <= 0x18c && count) { + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x180: + line_len = vv[k]; + break; + case 0x184: + line_count = vv[k]; + break; + case 0x188: + off_out = (off_out & 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x18c: + off_out = (off_out & + ~0xffffffffULL) | vv[k]; + have_out = 1; + break; + default: + break; + } + } + } else if (xlate && subch >= 2) { + /* TURING_DMA_COPY_A operand runs (subch >= 2). */ + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + if (addr >= 0x400 && addr <= 0x41c && n) { + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x400: + off_in = (off_in & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_in = 1; + break; + case 0x404: + off_in = (off_in & + ~0xffffffffULL) | + vv[k]; + have_in = 1; + break; + case 0x408: + off_out = (off_out & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x40c: + off_out = (off_out & + ~0xffffffffULL) | + vv[k]; + have_out = 1; + break; + case 0x410: + pitch_in = vv[k]; + break; + case 0x414: + pitch_out = vv[k]; + break; + case 0x418: + line_len = vv[k]; + break; + case 0x41c: + line_count = vv[k]; + break; + default: + break; + } + } + } else if (addr >= 0x700 && addr <= 0x708 && n) { + for (k = 0; k < n; k++) { + if (addr + 4 * k == 0x700) + last_remap = (last_remap & + ~0xffffu) | + (vv[k] & 0xffffu); + else if (addr + 4 * k == 0x708) + last_remap = (last_remap & + 0xffffu) | + ((vv[k] & 0xffffu) + << 16); + } + } else if (addr == 0x300 && count) { + u32 dma = w0; + + /* + * DATA_TRANSFER_TYPE bits 1:0; 0 = NONE (a + * release-only / semaphore op - no data + * movement, nothing to record). + */ + last_dma = dma; + if ((dma & 0x3) && have_in && have_out && + line_len && line_count && + off_in && off_out) { + if (inline_only) + svc_ahead_copy_in(vgpu, + ch->h_object, push_va, + svc_push_hash(buf, + len), + off_in, off_out, + line_len * 4, + line_count, dma); + else if (forward) { + svc_gr_copy_pend_add(ch->h_object, + push_va, svc_push_hash(buf, + len), + off_in, off_out, + line_len * 4, + line_count, + pitch_out ? pitch_out : + line_len * 4, + pitch_in, dma, + last_remap); + cp_subch = subch; + } + } + } + } + + if (!inline_only && addr == HOST_M_SEM_ADDR_LO && count >= 5) { + u64 va; + + va = ((u64)(w1 & 0xff) << 32) | + (u64)(w0 & 0xfffffffc); + op = w4 & 0x7; + if ((op == HOST_SEM_OP_RELEASE || + op == HOST_SEM_OP_REDUCTION) && w2 && + va >= SVC_LOW_REL_VA_LO && va < SEMA_VA_HI) { + /* + * A release targeting a live launch candidate's + * own QMD release word is owned by the + * completion edge (post-copy-out); sweeping it + * here would clear the client's fence before + * the engine output landed. + */ + if (xlate && forward && + svc_gr_ahead_owned_rel_va(va, push_va)) { + svc_gr_ahead_owned++; + } else { + svc_pending_add(va, w2, + op == HOST_SEM_OP_REDUCTION ? + 1 : 0); + nrel++; + } + } + } + if (end > len) + break; + off += 4 + count * 4; + } + (void)cp_subch; + return nrel; +} + +static void svc_gr_ahead_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put) +{ + static u8 abuf[2048]; + static u8 nbuf[2048]; + u32 i, maxent, nrel = 0, swept = 0, nskip = 0; + + if (!put || !ch->gpfifo_entries) + return; + maxent = min_t(u32, gsp_sema_gr_ahead_max, 512); + for (i = 0; i < maxent; i++) { + u32 idx = (put + i) % ch->gpfifo_entries; + u8 ent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen, tail_len = 0; + u64 va; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)idx * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + break; /* unwritten slot: end of the continuation */ + if (!va || !len || len > SVC_PUSH_MAX) + break; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + break; + rlen = min_t(u32, len, sizeof(abuf)); + if (svc_read_va_push(vgpu, ch, va, abuf, rlen)) { + /* + * Unreadable here (pre-anchor translation) is a + * reason to stop this pass: the continuation is not + * proven coherent, so nothing is authored. + */ + nskip++; + break; + } + /* + * SESSION 127 C5C0 translator: read the next ring entry as + * the tail of a stream whose final inline run overruns this + * entry (the driver splits large LOAD_INLINE_DATA payloads + * into a dedicated data slot entry; observed beef0100 + * idx00->idx01 and idx08->idx09). + */ + { + u32 nidx = (put + i + 1) % ch->gpfifo_entries; + u8 nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 ne0, ne1, nlen; + u64 nva; + + if (!svc_read_va(vgpu, + ch->gpfifo_va + + (u64)nidx * SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent))) { + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + if (nva && nlen < SVC_PUSH_MAX) { + tail_len = min_t(u32, nlen, + sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, + nbuf, tail_len)) + tail_len = 0; + } + } + } + swept++; + nrel += svc_gr_ahead_push(vgpu, ch, abuf, rlen, nbuf, + tail_len, va, false); + } + svc_gr_ahead_entries += swept; + svc_gr_ahead_rels += nrel; + svc_gr_ahead_skips += nskip; + if (swept && (nrel || svc_gr_ahead_log < 64)) { + svc_gr_ahead_log++; + pr_info("sema-ahead: h=0x%x put=%u swept=%u rels=%u owned=%llu skipped=%u (total rels=%llu)\n", + ch->h_object, put, swept, nrel, svc_gr_ahead_owned, + nskip, svc_gr_ahead_rels); + } +} + +/* + * SESSION 129: read a VA straight through the registered-VAS walks only + * (newest registration first), one page at a time - the exact path + * walk_probe and sema_probe prove. The util sweep's channel read can be + * shadowed by a stale injected pin / window guess for the utility VAs + * (observed r737: the subchannel-2 LID arm never fired although the push + * page content is real and the VAS walk resolves both the methods entry + * and its payload slot); this helper is the arbitration fallback that + * bypasses those sources entirely. + */ +static int svc_util_walk_read(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len) +{ + u8 *p = buf; + + while (len) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + size_t off = va & (SEMA_PAGE_SIZE - 1); + size_t chunk = min_t(size_t, len, SEMA_PAGE_SIZE - off); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (svc_walk_vaspaces(vgpu, page, &pa, &ap)) + return -ENOENT; + if (svc_read_ap(vgpu, pa + off, svc_ap_of(ap), p, chunk)) + return -EIO; + p += chunk; + va += chunk; + len -= chunk; + } + return 0; +} + +/* + * SESSION 129: does this entry window carry a LOAD_INLINE_DATA header? Used + * by the bounded util-sweep witness to select only the entries that can + * exercise the arm (one per distinct content, so a stable ring logs once). + */ +static u32 svc_util_lid_scan(const u8 *buf, u32 len, u32 *first_subch) +{ + u32 k, n = 0; + + *first_subch = 0xff; + for (k = 0; k + 4 <= len; k += 4) { + u32 hdr; + + memcpy(&hdr, buf + k, 4); + if (((hdr >> 29) & 0x7) == 3 && + ((hdr & 0x1fff) << 2) == GRCE_M_LOAD_INLINE_DATA) { + if (*first_subch == 0xff) + *first_subch = (hdr >> 13) & 0x7; + n++; + } + } + return n; +} + +/* + * SESSION 128: utility-channel inline-download materialization. + * + * The driver's upload/utility channels (the MEM2MEM/A140 + C5B5 + 2D + * composite: observed h=0xbeef0100 with gpfifo=0x4020000) are registered + * with an unresolved USERD (userd_ok=0), so the put-bounded entry loop in + * svc_pump_channels never decodes their pushes and the LOAD_INLINE_DATA + * downloads they carry (the client's code-heap/SASS upload, command-stream + * staging) never land anywhere the emulation can see: the destinations + * stay zero in the vidmem shadow. + * + * This pass scans a bounded window of the channel's GPFIFO directly (the + * gpfifo VA resolves through the trusted walk even when the USERD does + * not) and feeds each entry + its successor (the source-view call carrying + * the payload) through the proven sweep decoder in inline-only mode: the + * GRCE/INLINE_TO_MEMORY operand group (0x180..0x18c on subch 1/2), + * LAUNCH_DMA and the LOAD_INLINE_DATA run are materialized by + * svc_grce_inline_write; releases, QMD windows and copy-pend records are + * skipped so nothing about the client's fence/completion state changes. + * Once-only keyed by (push_va, content hash, dst, bytes), so a stable ring + * costs only the reads. Requires sema_gr_xlate; bounded by + * sema_gr_util_scan entries per channel per pass (0 = off). + * + * SESSION 129 read arbitration: attempt A is the channel path (unchanged); + * when A fails to read, or reads content the strict parse rejects, attempt + * B re-reads the push and the tail through svc_util_walk_read and the + * parse arbitrates - a source that yields a method stream wins. The r737 + * evidence (arm never fired while the content is real) pointed at exactly + * this: a shadowed channel read consumed the entry without a parseable + * buffer while the walk resolves the true pages. + */ +static void svc_gr_util_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ +/* + * The observed utility-channel pushes are the ~17 KB setup block and the + * two split uploads (256 B command stream + 12 KB code/heap payload); a + * 64 KB window covers every one of them with room to spare while keeping + * the two static buffers modest. + */ +#define SVC_UTIL_BUF (64 * 1024) + static u8 ubuf[SVC_UTIL_BUF]; + static u8 nbuf[SVC_UTIL_BUF]; + static u8 bbuf[SVC_UTIL_BUF]; + static u8 tbuf[SVC_UTIL_BUF]; + static u64 svc_gr_util_entries; + static u64 svc_gr_util_writes; + static u64 svc_util_tail_alt; /* SESSION 130: tails re-read through + * the direct walk because the + * channel read was zero-headed */ + static u64 svc_util_tail_try; /* ... and the walk attempts (alt is + * the subset that yielded non-zero) */ + u32 i, maxent = min_t(u32, gsp_sema_gr_util_scan, 64); + u32 swept = 0, nmade = 0; + bool first_sight = false; + + if (!gsp_sema_gr_xlate || !gsp_sema_gr_util_scan) + return; + if (!ch->gpfifo_va || !ch->gpfifo_entries) { + /* SESSION 143: name every channel the sweep cannot cover. */ + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + pr_info("sema-utilsweep: skip h=0x%x eng=%u reason=no-gpfifo gpfifo=0x%llx entries=%u userd=%d\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0); + } + return; + } + { + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + first_sight = true; + pr_info("sema-utilsweep: begin h=0x%x eng=%u gpfifo=0x%llx entries=%u userd=%d cid=%u\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0, + ch->cid); + } + } + for (i = 0; i < maxent && i < ch->gpfifo_entries; i++) { + u8 ent[SVC_GPFIFO_ENTRY_SIZE], nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen; + u64 va, before; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + continue; /* unwritten slot */ + if (!va || !len || len > SVC_PUSH_MAX) + continue; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + continue; + /* + * SESSION 132 stale-translation arbitration: when the + * channel-path entry is dead (its push provably reads + * all-zero through the client's page tables) while the + * direct VAS walk answers a live entry on the same ring + * slot, the channel path is reading a stale page (the r759 + * pin false positive) - the walk's entry is the channel's + * ring. A dead entry with no live walk alternative is left + * to the existing push arbitration below. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) { + u8 went[SVC_GPFIFO_ENTRY_SIZE]; + + if (!svc_util_walk_read(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + went, sizeof(went))) { + u32 we0, we1; + u64 wva; + u32 wlen; + + memcpy(&we0, went, 4); + memcpy(&we1, went + 4, 4); + wva = ((u64)(we1 & 0xff) << 32) | + (u64)(we0 & 0xfffffffc); + wlen = ((we1 >> 10) & 0x1fffff) << 2; + if (!svc_ring_entry_dead(vgpu, we0, we1) && + wva && wlen && wlen <= SVC_PUSH_MAX && + wva >= SVC_LOW_REL_VA_LO && + wva < SEMA_VA_HI) { + e0 = we0; + e1 = we1; + va = wva; + len = wlen; + } + } + } + rlen = min_t(u32, len, sizeof(ubuf)); + /* + * SESSION 129 read arbitration + bounded witness (see the + * sweep's header comment). Attempt A: the channel path, + * unchanged. Attempt B: the direct registered-VAS walk, used + * when A cannot read or cannot parse; the strict parse then + * arbitrates which content is the method stream. + */ + { + u32 push_rc, parse_a = 0, parse_b = 0; + u32 tl_a = 0, tl_b = 0; + u32 lidn, lidsc = 0xff, slot; + u64 nva = 0; + u32 nlen = 0; + u64 hkey; + u8 *pbuf = ubuf; + bool ut = false, seen = false; + + push_rc = svc_read_va_push(vgpu, ch, va, ubuf, + rlen); + /* + * The payload of a trailing LOAD_INLINE_DATA run rides + * the next GPFIFO entry (the driver's source-view + * GPFIFO call, observed beef0100 idx00->idx01 and + * idx08->idx09): read it as the tail exactly like + * svc_gr_ahead_sweep does. + */ + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)((i + 1) % ch->gpfifo_entries) * + SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent)) == 0) { + u32 ne0, ne1; + + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + } + if (!push_rc && nva && nlen && nlen < SVC_PUSH_MAX) { + tl_a = min_t(u32, nlen, sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, nbuf, + tl_a)) + tl_a = 0; + } + if (!push_rc) + parse_a = svc_ahead_parse(ubuf, rlen, nbuf, + tl_a, &ut); + if (push_rc || !parse_a) { + bool bpush = false; + + if (nva && nlen && nlen < SVC_PUSH_MAX) { + u32 want = min_t(u32, nlen, + sizeof(nbuf)); + + if (!svc_util_walk_read(vgpu, nva, + nbuf, + want)) + tl_b = want; + } + if (!svc_util_walk_read(vgpu, va, bbuf, + rlen)) { + bpush = true; + parse_b = svc_ahead_parse(bbuf, rlen, + nbuf, tl_b, + &ut); + /* + * A's buffer is authoritative only + * when A actually read it; otherwise + * (or when only B parses) B's walk + * content is the stream. + */ + if (push_rc || + (parse_b && !parse_a)) { + pbuf = bbuf; + tl_a = tl_b; + } + } + if (push_rc && !bpush) + continue; /* unreadable both ways */ + } + /* + * SESSION 130 tail-content arbitration: a tail-served + * payload whose channel head is still all-zero is not + * the staged bytes (the page the channel translation + * points at is empty, or the read is shadowed by a + * stale pin). Try the direct registered-VAS walk - + * the same path attempt B proves - before handing + * the tail to the LID arm, whose zero-headed refusal + * would otherwise stall the upload every pass. + */ + if (tl_a && nva && + !memchr_inv(nbuf, 0, min_t(u32, tl_a, 16))) { + u32 want = tl_a; + + svc_util_tail_try++; + if (!svc_util_walk_read(vgpu, nva, tbuf, + want) && + memchr_inv(tbuf, 0, + min_t(u32, want, 16))) { + memcpy(nbuf, tbuf, want); + svc_util_tail_alt++; + } + } + memset(&svc_util_wit, 0, sizeof(svc_util_wit)); + if (tl_a >= 4) + memcpy(&svc_util_wit.t0, nbuf, 4); + if (tl_a >= 8) + memcpy(&svc_util_wit.t1, nbuf + 4, 4); + before = svc_gr_ahead_inline_writes; + (void)svc_gr_ahead_push(vgpu, ch, pbuf, rlen, nbuf, + tl_a, va, true); + if (svc_gr_ahead_inline_writes != before) + nmade++; + swept++; + /* + * Bounded witness, one line per distinct LID-carrying + * (or unparseable) entry content: which attempt was + * used, the parses, the tail resolution and what the + * LID arm did. + */ + lidn = svc_util_lid_scan(pbuf, rlen, &lidsc); + if (!lidn && parse_a) + continue; + hkey = svc_push_hash(pbuf, rlen); + for (slot = 0; slot < 32; slot++) + if (svc_util_dec_keys[slot].h == hkey && + svc_util_dec_keys[slot].va == va) { + seen = true; + break; + } + if (seen || svc_util_dec_log >= 96) + continue; + slot = svc_util_dec_log & 31; + svc_util_dec_keys[slot].va = va; + svc_util_dec_keys[slot].h = hkey; + svc_util_dec_log++; + pr_info("sema-utildec: h=0x%x i=%u va=0x%llx len=0x%x p_src=%u pa=%u pb=%u tail_va=0x%llx tail=0x%x t8=%08x %08x alt=%llu/%llu lidn=%u subch=%u saw=%d fired=%d zero=%d dst=0x%x ll=%u lc=%u f8=%08x %08x\n", + ch->h_object, i, va, rlen, + pbuf == bbuf ? 1u : 0u, parse_a, parse_b, + nva, tl_a, svc_util_wit.t0, svc_util_wit.t1, + svc_util_tail_alt, svc_util_tail_try, + lidn, lidsc, + svc_util_wit.saw_lid ? 1 : 0, + svc_util_wit.fired ? 1 : 0, + svc_util_wit.zero ? 1 : 0, + svc_util_wit.dst, svc_util_wit.line_len, + svc_util_wit.line_lc, + rlen >= 4 ? *(const u32 *)pbuf : 0, + rlen >= 8 ? *(const u32 *)(pbuf + 4) : 0); + } + } + svc_gr_util_entries += swept; + svc_gr_util_writes += nmade; + if (first_sight && !swept) + pr_info("sema-utilsweep: read=none h=0x%x gpfifo=0x%llx entries=%u\n", + ch->h_object, ch->gpfifo_va, ch->gpfifo_entries); + if (swept && (nmade || svc_gr_util_log < 128)) { + svc_gr_util_log++; + pr_info("sema-utilsweep: h=0x%x swept=%u writes=%u (total %llu/%llu)\n", + ch->h_object, swept, nmade, svc_gr_util_entries, + svc_gr_util_writes); + } +} + static void svc_pump_channels(struct nv_vgpu_device *vgpu) { + if (svc_pdb_disc_pending) { + svc_pdb_disc_pending = false; + pr_info("sema-pdb-disc: requested va=0x%llx pa=0x%llx\n", + svc_pdb_disc_va, svc_pdb_disc_pa); + svc_vas_candidates_verify(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + svc_xvas_pdb_discover(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + } /* Only ever called from the service thread. */ static struct svc_channel local[SVC_MAX_CHANNELS]; /* @@ -23523,6 +26652,36 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) bool stalled = false; spin_lock_irqsave(&svc.chan_lock, flags); + if (svc_rewind_pending) { + svc_rewind_pending = false; + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu) + continue; + if (!svc_rewind_all && c->h_object != svc_rewind_h) + continue; + pr_info("sema-chan: rewind h=0x%x last_get=%u put=%u (slot bitmap + rescue state cleared)\n", + c->h_object, + (unsigned int)c->last_get, + (unsigned int)c->put_prev); + bitmap_zero(c->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + c->last_get = 0; + c->rescue_fail = 0; + c->unread_va = 0; + c->unread_until = 0; + c->defer_since = 0; + } + } + /* + * r707e: every-pass candidate verify with the last-good ring pair. + * Cheap (<=16 parked candidates, one walk each) and idempotent; the + * first pass after the user-VAS candidate parks registers it, so the + * client's very first push reads resolve. + */ + if (svc_lastgood_ring_pa) + svc_vas_candidates_verify(vgpu, svc_lastgood_ring_va, + svc_lastgood_ring_pa); /* * r507o stream-order: walk the CE channels BEFORE the GR channels. * The client's stream order places its H2D copies before the kernel @@ -23740,6 +26899,25 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!rrc) svc_pb_pair(vgpu, ch); + if (!rrc && ring_pa) { + svc_lastgood_ring_va = ch->gpfifo_va; + svc_lastgood_ring_pa = ring_pa; + } + /* + * r707f (FOLD): the moment ANY channel's ring resolves through + * the channel object, verify the parked COPY_SERVER_PDES VAS + * candidates against that content-backed pair. The candidates + * are parked at device-create time, before the client's compute + * channels first submit, so registering here lands the VAS page + * directory BEFORE the submit's entries get their first push + * read - the window the host-triggered path could not win (the + * entry's first read and the ring's first resolution happen in + * the same pass; a pair that arrives later cannot help, because + * a failed GR entry read is never retried with the fresh VAS). + */ + if (!rrc) + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + ring_pa); /* * ISSUE-0002 (r47 diag): run the deferred CE USERD sweep for a @@ -23760,6 +26938,24 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) jiffies + 27 * HZ; } + /* + * SESSION 128: utility/upload channels carry the engine's + * inline code/data downloads (LOAD_INLINE_DATA on the + * mem2mem subchannel) and the copy-class channels (eng=11/12) + * carry the driver's pitch copy-ins (staging -> code heap / + * constant store). With an unresolved USERD the put-bounded + * entry loop below never decodes them (the !userd_ok branch + * continues before it); with a resolved USERD the main + * decode's LOAD_INLINE_DATA/operand arms still only accept + * subchannel 1, so the subchannel-2 downloads are missed + * either way. Scan every channel's GPFIFO directly here and + * materialize the downloads (see svc_gr_util_sweep); the + * inline-only mode plus the once-only key make the overlap + * with the main loop and the GR sweep a no-op. + */ + if (gsp_sema_gr_util_scan && gsp_sema_gr_xlate) + svc_gr_util_sweep(vgpu, ch); + if (!ch->userd_ok) { if (svc_chan_userd(vgpu, ch, rrc ? 0 : ring_pa, rrc ? -1 : ring_ap, &put, &get, @@ -23784,6 +26980,23 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * r832: the RC watchdog's notifier ack must not depend on the + * ring reads. Its VA is known from the channel's first decode + * (ch->notify2d_va, captured from SET_NOTIFY_A/B) and the ack + * is the same zeroed NvNotification the decode path writes: + * the guest re-arms status=0xFFFF every cycle and checks it + * back to 0 within 7 s. Under heavy GL traffic the watchdog + * ring's translation can go stale for stretches (its entries + * then read as zeros and are consumed as empty), which starved + * exactly this write and wedged the guest with "GPU is + * probably locked" while every other channel kept running. + * Writing the ack on every pass that sees work pending is + * idempotent. + */ + if (!ch->engine_type && ch->notify2d_va && put != get) + svc_notify2d_write(vgpu, ch); + + /* * ISSUE-0002 diag: put snapshots for CE channels, bounded per * session. The throttle is indexed by rank (the channel struct * iterated here is a local copy, so a per-channel stamp never @@ -23977,6 +27190,50 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->put_prev = put; /* + * SESSION 126 fence fold: sweep the ring continuation beyond + * GP_PUT for semaphore releases (see svc_gr_ahead_sweep). + * GR-class channels only, live-armed epochs only, and only + * with the ring translation proven through the channel + * object (rrc == 0). + */ + if (!ch->engine_type && ch->gr_pub_armed && + gsp_sema_gr_ahead_max && !rrc && ring_pa) { + svc_gr_ahead_sweep(vgpu, ch, put); + /* + * SESSION 127 C5C0 translator: offer a launch the + * sweep captured from the continuation (the Vulkan + * client's inline-QMD push lives beyond GP_PUT, so + * the main walk's per-entry request never sees it). + * svc_gr_request is a no-op without a matching + * unoffered candidate and keeps it on a busy + * verdict, so a bounded per-pass retry here is the + * sweep-captured launch's entire dispatch path. + */ + if (gsp_sema_gr_xlate && + gsp_phys_gr_forward_enabled()) { + if (svc_gr_cand.valid && + svc_gr_cand.h_object == ch->h_object && + svc_gr_ahead_req_log < 8) { + svc_gr_ahead_req_log++; + pr_info("sema-ahead: request h=0x%x push=0x%llx offered=%d\n", + ch->h_object, + svc_gr_cand.push_va, + svc_gr_cand.offered); + } + (void)svc_gr_request(vgpu, ch); + } + /* + * Fallback copy-out retirement: when no forward is + * owed (the launch was already consumed or the + * backend is off), run any pending GRCE copies so + * they cannot sit forever (the completion edge is + * the primary path). + */ + if (gsp_sema_gr_xlate && !svc_gr_forward_pending()) + svc_gr_copy_pend_run(vgpu); + } + + /* * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. * Runs before the entry loop so a wedged session epoch is * recovered (and logged) instead of spinning: see @@ -24018,6 +27275,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) get_before = ch->last_get; applied_before = svc.releases_applied; + /* r835: this channel's VAS answers its VAs first (see + * svc_chanvas_lookup). Refreshed per channel turn; a channel + * without a known VAS clears it back to the old chain. */ + svc_cur_vas = ch->h_vaspace; /* * C4 (GAP-1 H1): the GR walk witness - does put advance, and @@ -24182,6 +27443,40 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * r829: a zero-length / zero-VA entry is a bad read + * (a stale or wrong translation), never work. The old + * path consumed it as an empty push and advanced + * GPGet; observed r830 on the RC watchdog channel: + * two bogus zero entries consumed through a + * mis-resolved ring page pushed the userd's GPGet past + * the driver's own GPPut, every later userd probe then + * read get>put and the channel's REAL userd was + * rejected - the channel stopped being serviced and + * the guest declared the GPU locked. Hold the entry + * instead (bounded; the per-channel counter persists + * through the pass snapshot); a corrected translation + * (the ring auto-pin lands within seconds) re-reads + * it. Past the cap the old consume-as-empty behavior + * resumes so an unresolvable ring cannot livelock the + * channel. + */ + if (va == 0 || len == 0) { + if (ch->zero_held < 512 && !svc_hold_bypass()) { + ch->zero_held++; + svc.gpfifo_held++; + if (gsp_sema_dbg && + svc_zero_entry_log < 16) { + svc_zero_entry_log++; + pr_info("sema-chan: zero-entry hold h=0x%x idx=%llu raw=%08x%08x (bad ring translation; not consumed)\n", + ch->h_object, idx, + e1, e0); + } + stalled = true; + continue; + } + } + + /* * Cheap wait for an entry already inside its * expensive-wait window: read + rescue + hold were * already paid for this VA, and a pin-batch rewind @@ -24229,9 +27524,25 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (pb) { int rescue_rc; - bool unreadable = - svc_read_va_push(vgpu, ch, va, - pb, len) != 0; + int rd_rc; + bool unreadable; + bool zero_nop = false; + bool displaced = false; + bool have_ipin = false; + + svc_read_last_src = 0xff; + svc_read_last_pa = 0; + rd_rc = svc_read_va_push(vgpu, ch, va, + pb, len); + unreadable = rd_rc != 0; + if (gsp_sema_dbg && svc_rdw_log < 6000 && + !ch->engine_type) { + svc_rdw_log++; + pr_info("sema-rdw: h=0x%x va=0x%llx len=%u rc=%d src=%u\n", + ch->h_object, va, len, + rd_rc, + (unsigned)svc_read_last_src); + } bool injected = false; bool allow_ops = (ch->engine_type != 0); @@ -24256,8 +27567,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u64 ipa; int iap; - injected = (svc_ipin_lookup(va, &ipa, - &iap) == 0); + have_ipin = + (svc_ipin_lookup(va, &ipa, + &iap) == 0); + /* + * r715: the entry's treatment + * follows the source that + * ACTUALLY answered the read, + * not a fresh page-granularity + * ipin lookup. The read falls + * through to the trusted walk + * (src 4) whenever an ipin's + * page fails its first-chunk + * guard, and taking the + * injected branch then skipped + * the only source that could + * answer: the entry was + * consumed without ever + * consulting the registered + * VAS walk (session 124). + */ + injected = have_ipin && + (svc_read_last_src == 0); /* * r496e: an all-zero push on a * content-scanned-ring channel is the @@ -24265,8 +27596,23 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * dword - NVA06F_DMA_NOP), not a bad * translation: accept it and let the * decode find no ops. + * r820/r826: the same push arrives on + * KERNEL channels (nvidia-push's + * __nvPushTestPushBuffer writes the + * NOP into the kernel pushbuffer); its + * promote entry is a booking, so the + * trusted walk answers - a SMALL + * all-zero GR push resolved by the + * trusted chain (src=4) is the NOP. + * Real launches are large (a QMD push + * is hundreds of bytes), so the size + * bound keeps an unread legitimate + * push from being consumed as empty. */ - bool zero_nop = ch->gr_ram_ring && + zero_nop = (ch->gr_ram_ring || + (!ch->engine_type && + svc_read_last_src == 4 && + len <= 16)) && len >= 4 && !memchr_inv(pb, 0, len); @@ -24299,7 +27645,82 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(fb); } } else { - unreadable = true; + /* + * r707v: a primary-source + * hit (injected pin, window + * guess, lat cache) whose + * content is implausible + * must not shadow the + * registered VAS walk - a + * stale pin otherwise makes + * every push read "resolve" + * to the wrong page, fail + * plausibility and turn the + * entry unreadable forever. + * Retry once with the pins + * bypassed so the trusted + * walk (svc_walk_vaspaces) + * answers; accept only a + * plausible stream. + */ + u8 *vb = kmalloc(len, + GFP_KERNEL); + + if (vb) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + bool ok = false; + + /* + * r707w: consult ONLY the + * registered VAS walk (the + * exact path walk_probe + * proves) - not + * svc_lookup_trusted, whose + * pin/lat hits would answer + * with the same wrong page + * again and keep the entry + * unreadable. + */ + int wr = svc_walk_vaspaces(vgpu, + va & ~0xfffULL, + &wpa, &wap); + int rr = wr ? wr : svc_read_ap(vgpu, + wpa + (va & 0xfff), + wap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : + SVC_APERTURE_SYS, + vb, len); + bool pl = !rr ? svc_method_stream_plausible(vb, len) + : false; + + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk rc=%d pa=0x%llx plaus=%d\n", + ch->h_object, va, + wr ? wr : rr, + wpa + (va & 0xfff), + pl ? 1 : 0); + } + if (!wr && !rr && pl) { + memcpy(pb, vb, len); + svc.pushpg_fallback++; + ok = true; + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk ok pa=0x%llx\n", + ch->h_object, va, + wpa + (va & 0xfff)); + } + } + kfree(vb); + if (!ok) + unreadable = true; + } else { + unreadable = true; + } } /* * Still implausible (or @@ -24315,18 +27736,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u8 *db = kmalloc(len, GFP_KERNEL); - if (db && - svc_push_read_displaced(vgpu, - ch->h_object, va, - db, len)) { - memcpy(pb, db, len); - injected = false; - unreadable = false; - } - kfree(db); - } - } - } + if (db && + svc_push_read_displaced(vgpu, + ch->h_object, va, + db, len)) { + memcpy(pb, db, len); + injected = false; + unreadable = false; + displaced = true; + } + kfree(db); + } + } + } /* * The entry's page was unreadable through @@ -24345,11 +27767,40 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(pb, db, len); injected = false; unreadable = false; + displaced = true; } kfree(db); } /* + * r715 pushdec witness: name the branch + * the entry takes straight after the + * read, before the rescue/hold paths. + * Session 124 had an entry consumed + * between the read (rc=0 src=4) and the + * decode with no row naming why; this + * line makes the branch and the content + * (first 8 bytes) visible on every read. + * Budget: 512 rows per client (reset in + * svc_session_flush). + */ + if (gsp_sema_dbg && svc_pushdec_log < 2048) { + svc_pushdec_log++; + pr_info("sema-pushdec: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx plausible=%d injected=%d ipin=%d zero_nop=%d displaced=%d unreadable=%d first8=%08x %08x\n", + ch->h_object, va, len, + (unsigned)svc_read_last_src, + svc_read_last_pa, + svc_method_stream_plausible(pb, len) ? 1 : 0, + injected ? 1 : 0, + have_ipin ? 1 : 0, + zero_nop ? 1 : 0, + displaced ? 1 : 0, + unreadable ? 1 : 0, + len >= 4 ? *(const u32 *)pb : 0, + len >= 8 ? *(const u32 *)(pb + 4) : 0); + } + + /* * Content rescue before any fake-release * fallback (CE channels): the push bytes * are in guest RAM even when the VA->PA @@ -25195,7 +28646,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!ch->engine_type && va && len && gsp_phys_gr_forward_enabled() && (svc_gr_qmd_seen > 0 || - all_zero) && + (all_zero && !zero_nop)) && (!entry_replay || svc_gr_cand_push_owed(va)) && !(svc_gr_cand.valid && @@ -26072,6 +29523,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].gr_ram_logged = ch->gr_ram_logged; svc.chan[k].gr_ram_ring = ch->gr_ram_ring; svc.chan[k].userd_scans = ch->userd_scans; + svc.chan[k].notify2d_va = ch->notify2d_va; + svc.chan[k].zero_held = ch->zero_held; + svc.chan[k].userd_reg = ch->userd_reg; svc.chan[k].logged = ch->logged; svc.chan[k].logged_expected = ch->logged_expected; svc.chan[k].notified = ch->notified; @@ -26485,12 +29939,13 @@ static void svc_handle_chan_dump(struct nv_vgpu_device *vgpu) svc_read_ap(vgpu, ch->notifier_pa + 16 + 8, ch->notifier_as, &info1, 4); } - pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", + pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx armed=%d notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", ch->h_object, ch->engine_type, ch->gpfifo_va, ch->gpfifo_entries, ch->userd_pa, ch->userd_as, ch->userd_ok, ch->last_get, ch->submissions, ch->expected_sem_payload, ch->last_rel_va, ch->counter_pa, + ch->gr_pub_armed, ch->notifier_pa, ch->notifier_as, info0, st0, info1, st1, ch->notified); @@ -26992,19 +30447,43 @@ static void svc_scrub_pass(struct nv_vgpu_device *vgpu) u32 put = 0; u64 delta; - if (!s || !s->userd_ok || !s->pa_ok || !s->entries) + if (!s) return; + if (!s->userd_ok || !s->pa_ok || !s->entries) { + /* m112b: silent-return diagnostic (bounded per boot). */ + if (s->log_budget < 8) { + s->log_budget++; + pr_info("sema-svc: scrub pass idle vgpu=%px slot=%d userd_ok=%d pa_ok=%d entries=%u gen=%u\n", + (void *)vgpu, (int)(s - svc_scrubs), + s->userd_ok, s->pa_ok, s->entries, + s->session_gen); + } + return; + } if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as, &put, sizeof(put))) { /* Retry through the alternate aperture (as registration did). */ if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as == SVC_APERTURE_FB ? SVC_APERTURE_SYS : SVC_APERTURE_FB, - &put, sizeof(put))) + &put, sizeof(put))) { + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass read-fail userd=0x%llx as=%u\n", + s->userd_pa, s->userd_as); + } return; + } } - if (put >= s->entries) + if (put >= s->entries) { + /* m112b: junk-translation diagnostic (bounded). */ + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass junk put=%u entries=%u userd=0x%llx\n", + put, s->entries, s->userd_pa); + } return; /* junk translation; retry next pass */ + } if (!s->seeded) { s->seeded = true; diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h index fbc7e4d..63c7f82 100644 --- a/kernel/gsp_sema_service.h +++ b/kernel/gsp_sema_service.h @@ -31,6 +31,7 @@ struct sema_channel_desc { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (the doorbell token vector) */ + u32 h_vaspace; /* r835: VAS handle the channel was allocated in */ u64 gpfifo_va; u32 gpfifo_entries; u64 userd_pa; /* NvA06FControl: GPGet @0x88, GPPut @0x8c */ @@ -89,6 +90,23 @@ struct sema_vaspace_desc { void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc); void gsp_sema_service_vaspace_del(u32 h_vaspace); + +/* + * r820: a driver-declared kernel-buffer translation from an + * NV2080_CTRL_CMD_GPU_PROMOTE_CTX entry (gpuVirtAddr/gpuPhysAddr/size/ + * physAttr of one context buffer). These buffers - the nvidia-push + * pushbuffer, the progress-tracker segment, the FECS/patch/access-map + * blocks - are kernel-owned, so the userspace pin scanner cannot cover + * them and the page-table walk can answer with a stale/wrong page + * (r817 Xorg GL init: the walk resolved the 64 KB pushbuffer VA to a + * zero page while the promote entry named the real one). The sema + * service consults these records before the learned cache and the walks + * in svc_lookup_trusted; newest statement per VA wins. VA-only bookings + * (zero pa/size) and bNonmapped entries (va=0) are ignored. Records are + * session state, cleared by svc_session_flush. + */ +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id); /* * r496j: park a page-directory candidate (from COPY_SERVER_PDES) until a * content-validated walk proves it belongs to a live client; only then is @@ -110,6 +128,8 @@ int gsp_sema_service_dump_channels(void); int gsp_sema_service_add_map(u64 gpu_va, u64 pa, u64 len, u32 ap); int gsp_sema_service_dump_vidmem(u64 addr, u64 len); int gsp_sema_service_find_vidmem(u64 pattern, u64 max_hits); +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa); +int gsp_sema_service_chan_rewind(u32 h_object); int gsp_sema_service_counter_add(u32 h_object, u64 pa); void gsp_sema_service_doorbell(u32 token); diff --git a/kernel/main.c b/kernel/main.c index dd11d12..7016647 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -2960,6 +2960,61 @@ module_param_cb(sema_map_add, &nv_vgpu_sema_map_add_ops, NULL, 0200); MODULE_PARM_DESC(sema_map_add, "Inject a GPU VA -> guest PA mapping (hex va,hex pa)"); /* + * Debug: retrigger the client-VAS page-directory discovery with a host + * content-verified {ring VA -> ring PA} pair (the ring page's GPFIFO + * entries are the fingerprint): + * echo 0x<va>,0x<pa> > .../parameters/sema_pdb_discover + */ +static int nv_vgpu_sema_pdb_discover_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 va = 0, pa = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &va) || kstrtoull(comma + 1, 0, &pa)) + return -EINVAL; + + return gsp_sema_service_pdb_discover(va, pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_pdb_discover_ops = { + .set = nv_vgpu_sema_pdb_discover_set, +}; + +module_param_cb(sema_pdb_discover, &nv_vgpu_sema_pdb_discover_ops, NULL, 0200); +MODULE_PARM_DESC(sema_pdb_discover, "Retrigger the VAS pdb discovery with a verified ring pair (hex va,hex pa)"); + +/* + * Debug: bounded channel rewind. Clears the consumed-slot bitmap and the + * per-entry rescue state for one channel (hex h_object) or every channel + * ("all"), so the pump re-processes its ring from slot 0 - used after the + * VAS pdb anchor lands so entries skipped before it re-decode (the release + * they carry would otherwise be lost for that put epoch). + */ +static int nv_vgpu_sema_chan_rewind_set(const char *val, + const struct kernel_param *kp) +{ + u32 h = 0xffffffffu; + + if (strncmp(val, "all", 3) && kstrtouint(val, 0, &h)) + return -EINVAL; + return gsp_sema_service_chan_rewind(h); +} + +static const struct kernel_param_ops nv_vgpu_sema_chan_rewind_ops = { + .set = nv_vgpu_sema_chan_rewind_set, +}; + +module_param_cb(sema_chan_rewind, &nv_vgpu_sema_chan_rewind_ops, NULL, 0200); +MODULE_PARM_DESC(sema_chan_rewind, "Rewind a channel's ring (hex h_object or 'all')"); + +/* * Debug: register a channel's GPU completion counter (guest PA of the u32 * tracking counter the GPU would write when a push completes): * echo <h_object>,<pa> > .../parameters/sema_counter @@ -3427,6 +3482,47 @@ static int nv_vgpu_dump_vidmem_set(const char *val, const struct kernel_param *k mutex_unlock(&nv_vgpu_driver.lock); return 0; } + if (!strncmp(buf, "map", 3)) { + /* + * Coarse non-zero map: one line per 1 MB block that holds + * any non-zero dword (SESSION 131: locate the live client's + * uploaded content - the image BO, the code heap, the rings - + * inside the 256 MB aperture without 65536 range dumps). + */ + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = vgpu->bars[1].size; + size_t blk, nblk = (size + (1 << 20) - 1) >> 20; + size_t o; + + if (!base || !size) + continue; + for (blk = 0; blk < nblk; blk++) { + size_t start = blk << 20; + size_t end = start + (1 << 20); + u32 nz = 0, first = 0; + + if (end > size) + end = size; + for (o = start; o + 4 <= end; o += 4) { + u32 v; + + memcpy(&v, base + o, 4); + if (v) { + if (!nz) + first = (u32)(o & 0xFFFFF); + nz++; + } + } + if (nz) + pr_info("vgsp: vidmem map blk 0x%zx nz=%u first=0x%x\n", + start, nz, first); + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return 0; + } comma = strchr(buf, ','); if (comma) diff --git a/tools/guest-diag/fbcopy_pub.c b/tools/guest-diag/fbcopy_pub.c new file mode 100644 index 0000000..a465617 --- /dev/null +++ b/tools/guest-diag/fbcopy_pub.c @@ -0,0 +1,345 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * fbcopy_pub.c - capture the vGPU framebuffer WITHOUT X and publish frames + * into the kvmfr ivshmem BAR. + * + * The X server's screen has 1280x720 pixels in the emulated VRAM, which the + * guest maps as the vGPU's BAR1 (resource1). The X SHM readback path can + * stall behind the X server's own push wait; this tool reads the framebuffer + * pixels straight out of the BAR1 mapping instead (host RAM backing, no GPU + * or X round-trip), swizzles BGRA -> RGBA and writes the kvmfr frame + * protocol the host reader already decodes. + * + * Usage (guest, as root): + * ./fbcopy_pub test one solid red frame (pipeline check) + * ./fbcopy_pub find motion-scan BAR1, print hot offsets + * ./fbcopy_pub pub <off> [n] [fps] [w] [h] [pitch] + * copy w*h from BAR1@off (pitch) + * + * Find/build on the host (same glibc as the guest): gcc -O2 -o fbcopy_pub + * fbcopy_pub.c + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <dirent.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <time.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" */ +#define DEF_W 1280 +#define DEF_H 720 +#define DEF_PITCH 5120 /* xorg-vgpu.conf: 1280*4 rounded */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +static const char *find_vgpu_resource1(void) +{ + static char path[320]; + DIR *d = opendir("/sys/bus/pci/devices"); + struct dirent *de; + + if (!d) + return NULL; + while ((de = readdir(d))) { + char p[340], buf[32]; + FILE *f; + int n; + + if (de->d_name[0] == '.') + continue; + snprintf(p, sizeof(p), "/sys/bus/pci/devices/%s/vendor", + de->d_name); + f = fopen(p, "r"); + if (!f) + continue; + n = fread(buf, 1, sizeof(buf) - 1, f); + fclose(f); + if (n <= 0) + continue; + buf[n] = 0; + if (strtoul(buf, NULL, 16) != 0x10de) + continue; + snprintf(path, sizeof(path), + "/sys/bus/pci/devices/%s/resource1", de->d_name); + closedir(d); + return path; + } + closedir(d); + return NULL; +} + +/* + * The vfio mdev region mmap is limited (256 MB EINVAL, 64 MB OK): open the + * resource once and map windows no larger than this via map_window(). + */ +#define CHUNK_MAX (64UL << 20) + +static unsigned char *map_window(int fd, size_t off, size_t len) +{ + if (len > CHUNK_MAX) + return MAP_FAILED; + return mmap(NULL, len, PROT_READ, MAP_SHARED, fd, (off_t)off); +} + +static double t_ms(const struct timespec *a, const struct timespec *b) +{ + return (double)(b->tv_sec - a->tv_sec) * 1e3 + + (double)(b->tv_nsec - a->tv_nsec) / 1e6; +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "find"; + const char *bar_path = "/sys/bus/pci/devices/0000:00:03.0/resource2"; + const char *res1 = find_vgpu_resource1(); + struct stat st; + size_t map_size, i; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, vram_fd; + + setvbuf(stdout, NULL, _IONBF, 0); + if (!res1) { + printf("NO_VGPU\n"); + return 1; + } + printf("VGPU_RES1 %s\n", res1); + if (stat(res1, &st) || !st.st_size) { + printf("STAT_FAIL\n"); + return 1; + } + map_size = (size_t)st.st_size; + printf("VRAM_SIZE 0x%zx (%zu MB)\n", map_size, map_size >> 20); + vram_fd = open(res1, O_RDONLY | O_SYNC); + if (vram_fd < 0) { + perror(res1); + return 1; + } + + /* kvmfr BAR next: every mode needs it except find. */ + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + perror(bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + close(fd); + if (bar == MAP_FAILED) { + perror("mmap kvmfr"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + if (!strcmp(mode, "test")) { + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + size_t n = (size_t)DEF_W * DEF_H * 4; + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = DEF_W; + hdr->height = DEF_H; + hdr->stride = DEF_W * 4; + hdr->format = 0; + for (i = 0; i < n; i += 4) { + dst[i + 0] = (unsigned char)(argc > 2 ? atoi(argv[2]) + : 255); + dst[i + 1] = (unsigned char)(argc > 3 ? atoi(argv[3]) + : 0); + dst[i + 2] = (unsigned char)(argc > 4 ? atoi(argv[4]) + : 0); + dst[i + 3] = 0xff; + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = 1; + printf("TEST_FRAME_PUBLISHED\n"); + return 0; + } + + if (!strcmp(mode, "find")) { + /* + * Sparse motion scan: sample 256 B of every 4 KB page twice, + * 300 ms apart; score pages by differing bytes and report the + * hottest pages plus per-16MB moving counts. The framebuffer + * is the contiguous ~4 MB block with the most motion. The + * region is mapped one <=64 MB window at a time (the vfio + * mdev mmap refuses the full 256 MB). + */ + size_t pages = map_size >> 12; + size_t cpages = CHUNK_MAX >> 12; + unsigned char *s0, *s1; + uint32_t *score; + size_t p, chunk; + int top; + + s0 = malloc(cpages * 256); + s1 = malloc(cpages * 256); + score = calloc(pages, sizeof(uint32_t)); + if (!s0 || !s1 || !score) + return 1; + for (chunk = 0; chunk * CHUNK_MAX < map_size; chunk++) { + size_t off = chunk * CHUNK_MAX; + size_t len = map_size - off; + size_t np = 0; + struct timespec a, b; + unsigned char *w; + + if (len > CHUNK_MAX) + len = CHUNK_MAX; + w = map_window(vram_fd, off, len); + if (w == MAP_FAILED) { + printf("CHUNK_MAP_FAIL off=0x%zx\n", off); + return 1; + } + np = len >> 12; + for (p = 0; p < np; p++) + memcpy(s0 + p * 256, w + (p << 12), 256); + clock_gettime(CLOCK_MONOTONIC, &a); + usleep(300000); + for (p = 0; p < np; p++) + memcpy(s1 + p * 256, w + (p << 12), 256); + clock_gettime(CLOCK_MONOTONIC, &b); + munmap(w, len); + printf("SCAN chunk=0x%zx pages=%zu dt=%.0fms\n", + off, np, t_ms(&a, &b)); + for (p = 0; p < np; p++) { + unsigned char *x = s0 + p * 256; + unsigned char *y = s1 + p * 256; + uint32_t d = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j] != y[j]) + d++; + score[(off >> 12) + p] = d; + } + } + /* per-16MB summary */ + for (i = 0; i + 4096 <= pages; i += 4096) { + uint64_t s = 0; + size_t j; + + for (j = i; j < i + 4096; j++) + s += score[j]; + if (s) + printf("BLOCK off=0x%zx moving_bytes=%llu\n", + i << 12, (unsigned long long)s); + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (score[p] > bs) { + bs = score[p]; + best = p; + } + if (!bs) + break; + printf("HOT off=0x%zx score=%u\n", best << 12, bs); + score[best] = 0; + } + printf("FIND_DONE\n"); + return 0; + } + + if (!strcmp(mode, "pub")) { + unsigned long long off = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + int frames = argc > 3 ? atoi(argv[3]) : 30; + int fps = argc > 4 ? atoi(argv[4]) : 5; + int w = argc > 5 ? atoi(argv[5]) : DEF_W; + int h = argc > 6 ? atoi(argv[6]) : DEF_H; + int pitch = argc > 7 ? atoi(argv[7]) : DEF_PITCH; + unsigned long long need = off + + (unsigned long long)(h - 1) * pitch + + (unsigned long long)w * 4; + size_t aligned = (size_t)off & ~0xfffUL; + size_t win = (size_t)(need - aligned); + unsigned char *wmap; + int f, y, x; + + if (need > map_size) { + printf("OFF_RANGE_FAIL need=0x%llx size=0x%zx\n", + need, map_size); + return 1; + } + if (win > CHUNK_MAX) { + printf("WIN_TOO_BIG 0x%zx\n", win); + return 1; + } + wmap = map_window(vram_fd, aligned, win); + if (wmap == MAP_FAILED) { + printf("WIN_MAP_FAIL off=0x%zx len=0x%zx\n", + aligned, win); + return 1; + } + printf("PUB off=0x%llx frames=%d fps=%d %dx%d pitch=%d\n", + off, frames, fps, w, h, pitch); + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + for (f = 0; f < frames; f++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + unsigned char *src = wmap + ((size_t)off - aligned); + + for (y = 0; y < h; y++) { + unsigned char *s = src + (size_t)y * pitch; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + d[4 * x + 0] = s[4 * x + 2]; + d[4 * x + 1] = s[4 * x + 1]; + d[4 * x + 2] = s[4 * x + 0]; + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(f + 1); + __sync_synchronize(); + printf("FRAME seq=%d ms=%.1f\n", f + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d\n", f); + return 0; + } + + printf("USAGE: fbcopy_pub test|find|pub <off> [n] [fps] [w] [h] [pitch]\n"); + return 1; +} diff --git a/tools/guest-diag/fenceword_assist.py b/tools/guest-diag/fenceword_assist.py new file mode 100644 index 0000000..9e207e2 --- /dev/null +++ b/tools/guest-diag/fenceword_assist.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# fenceword_assist.py - keep the Vulkan fence wait word of a live vk_frame120 +# client released (session 125). +# +# SUPERSEDED (SESSION 126, commit 95e7f57e): the service now authors these +# releases itself. The `sema_gr_ahead` fence fold sweeps the channel's +# pre-built ring continuation beyond GP_PUT through the ring's validated +# GPFIFO translation, decodes the INC-5 RELEASE groups it finds (addr 0x5c +# count>=5) and queues them in the standard pending ledger; r717 evidence: +# `sema-ahead: h=0xbeec0100 put=11 swept=10 rels=4` followed 30 ms later by +# `sema-svc: deferred release va=0x45ba090 payload=1 (was cur=0)`, with +# VKFRAME seq/DONE and NO assist running (see +# backups/evidence-20261002-victory/r717-fencefold/). Keep this tool only as +# the A/B fallback for boots that predate the fold (or with +# sema_gr_ahead=0). +# +# The client's fence wait polls a word in a 16 KB /dev/nvidiactl BO mapped +# read-write into the client: the init wait word lives at +0x80 (the t122 +# "BW" chaser writes it once) and the per-submit fence word at +0x90 of the +# SAME BO (GPU VA <base>+0x90, delivered by the submit push's INC-5 semaphore +# group). While the emulation does not apply that group, this host assist +# writes 1 to the +0x90 slot whenever it reads 0 - the direct predecessor of +# the service-side fold (see Handoff SESSION 125/126). +# +# Usage (host, while a t122-driven client runs): +# tools/guest-diag/fenceword_assist.py [seconds] [guest-pid] +import os +import subprocess +import sys +import time + +VMEXEC = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), + "..", "..", "scripts", "vgpu_vm.py")) +PY = r''' +import struct, sys, time +pids = [p for p in sys.argv[1:] if p] +maps_of = {} +def vma(pid, addr): + for line in open("/proc/%d/maps" % pid): + rng, perms = line.split(None, 2)[:2] + a, b = [int(x, 16) for x in rng.split("-")] + if a <= addr < b: + return line.strip() + return None +found = {} +for pid in pids: + try: + maps = [l for l in open("/proc/%d/maps" % pid) if "nvidia" in l and "rw-s" in l] + except OSError: + continue + for l in maps: + rng, perms, off = l.split(None, 3)[:3] + a, b = [int(x, 16) for x in rng.split("-")] + if b - a != 0x4000: + continue + try: + mem = open("/proc/%d/mem" % pid, "r+b", 0) + mem.seek(a + 0x80); init = mem.read(4) + mem.seek(a + 0x90); fw = mem.read(4) + except Exception: + continue + if init == b"\x01\x00\x00\x00" and fw == b"\x00\x00\x00\x00": + found[pid] = (a + 0x90, mem) + print("found pid=%s word=%#x" % (pid, a + 0x90), flush=True) +end = time.time() + float(sys.argv[1]) +while time.time() < end: + for pid, (w, mem) in list(found.items()): + try: + mem.seek(w) + if mem.read(4) == b"\x00\x00\x00\x00": + mem.seek(w); mem.write(struct.pack("<I", 1)); mem.flush() + print("released pid=%s word=%#x" % (pid, w), flush=True) + except Exception: + del found[pid] + time.sleep(0.5) +print("assist done", flush=True) +''' + + +def main(): + dur = sys.argv[1] if len(sys.argv) > 1 else "300" + pid = sys.argv[2] if len(sys.argv) > 2 else "" + b64 = __import__("base64").b64encode(PY.encode()).decode() + subprocess.run([VMEXEC, "exec", "--timeout", str(int(dur) + 60), + "echo %s | base64 -d > /tmp/fwa.py; sudo python3 /tmp/fwa.py %s %s" + % (b64, dur, pid)]) + + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/guest_pin_stream.py b/tools/guest-diag/guest_pin_stream.py index 175ed23..8768f2f 100644 --- a/tools/guest-diag/guest_pin_stream.py +++ b/tools/guest-diag/guest_pin_stream.py @@ -1432,7 +1432,14 @@ def cdt_pid(): # Processes that hold a /dev/nvidia* fd but are not CUDA clients; the # scanner must never commit to them (a transient nvidia-smi at boot was # picked once and its exited pagemap read back EOF forever). - non_clients = ("nvidia-smi", "nvidia-persistenced", "Xorg", "X") + # + # SESSION 146: Xorg/X used to be excluded here too, but the GL/GZDOOM + # path needs the X server's pages pinned - Xorg screen init fails with + # "Failed to initialize DMA / Failed to allocate push buffer" when the + # nvidia-push buffers cannot be resolved (observed r816). Xorg is a + # long-lived process (unlike the nvidia-smi transient the exclusion + # guarded against), so it is a valid, stable pin target. + non_clients = ("nvidia-smi", "nvidia-persistenced") cands = [] for d in os.listdir('/proc'): if not d.isdigit(): diff --git a/tools/guest-diag/wall2/README-session77.md b/tools/guest-diag/wall2/README-session77.md index f45dd70..d830bef 100644 --- a/tools/guest-diag/wall2/README-session77.md +++ b/tools/guest-diag/wall2/README-session77.md @@ -573,3 +573,623 @@ into the "durable harness bundle": marker-clean + word-chase + count-clear + (once VKFRAME) the release/alloc phases. (3) The bNotifyList=1 (gsp_rm_ev_nlist) path stays armed for the event-signal experiment; post/EVPOST logging re-budgeted (gsp_rm_ev_log=2000 set). + +**13. Session 93: THE ZERO-PTRACE PIPELINE (r693) - VKDEVICE with no gdb-wedge risk.** +Split the harness by risk: (a) ONE gdb (t101.py) for the INSTANCE-phase +marker clears only (base+0xB33D30 handler, clears u16@+0xe/+0xc and u32@+0 +of each entry; self-disables after 24 clears; NO other BPs so it cannot +fire constantly); (b) the WAIT WORDS handled by /tmp/unwait.py - a guest +python poller with ZERO ptrace that scans the newest vk_frame's rw-s +nvidia mappings of exactly 0x4000 bytes and writes 1 to any zero u32 at +mapping+0x80 via /proc/PID/mem (the session-77 geometry, confirmed by t79 +chains: word = the +0x80 of a 16 KB rw-s nvidiactl mapping). Result on +fresh r693 (mdev 261980af): t101 CLEAR#1..#5 then idle; unwait wrote FIVE +wait words (0x7876ead57080, 0x7876f1816080, 0x7876ea47f080, +0x7876ea81a080, 0x7876ead51080); client reached `VKINSTANCE ok / VKDEVS 2 +/ VKPICK / VKDEVICE ok family=0` with NO wedge exposure. live_walk (PT-walk +injector) ran alongside. The client then sat in the post-device wait that +the +0x80 poller does not cover (t102.py = the one-shot safe diagnostic: +attach -> print PC/base/A8-count(base+0x2070b7c)/bt -> detach, NO continue +and NO breakpoints - use it when the slot is free). + +**14. THE RM-LOCK WEDGE ROOT CAUSE + RULE.** A gdb killed (SIGTERM) while +the tracee sits AT a breakpoint leaves the tracee STOPPED; if that stop is +inside an RM api critical section (the rwlock held across an RM call!), the +RM write lock stays held and EVERY other client blocks in +os_acquire_rwlock_write (state D) - observed directly: old client 1725 +stopped by its wedged t99 gdb -> new client 7967 hung `state=D wchan= +os_acquire_rwlock_write` on ioctl 0xc020462b; killing the gdb + `kill -CONT` +resumed 1725 out of the ptrace stop but it resumed INSIDE the same RM wait, +so the lock stayed held until the VM was cycled. RULES: never let a gdb +timeout fire while a tracee is at a frequent-firing BP; prefer BPs that +fire once or self-disable; prefer no-ptrace pollers for anything with a +repeating pattern; the safe diagnostic is attach -> read -> detach without +continue. + +**15. Session 94: THE INSTANCE CLEARS ARE REQUIRED (zero-gdb disproved); t104 is the safe form.** +Pure zero-gdb run (fresh r694, client + unwait.py ONLY, no gdb): unwait +wrote three +0x80 words and the client still ended `VKFAIL vkCreateDevice +... -4` - WITHOUT the instance-phase marker clears the device gate fails +even with the words. With t104.py (single BP at +0xB33D30; clears +u16@+0xe/+0xc/u32@+0; self-exits on cleared>=8 OR chits>=40 OR 25 s since +the last clear -> returns True so the script tail runs and batch gdb exits +cleanly, no outer-timeout dependence) + unwait + live_walk: CLEAR#1..#5, +two words written, client reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / +VKDEVICE ok family=0` (r694 client 3725). The client then sat in the +post-device wait again; the t102 diagnostic could not attach because the +t104 gdb still held the slot (its 25 s-idle exit never fired - no +post-instance B33D30 hits - and its 150 s timeout likely fired at a stop += the known wedge risk). NEXT FORM: make t104 detach ITSELF from inside +the handler once the clears are done (`gdb.execute("detach")` from the +stop handler; the tracee resumes; the script ends and gdb exits) so NO +gdb is ever held while the client runs - then t102 always has the slot +and the wedge class disappears end-to-end. + +**16. The allocation/post-device phase is the remaining gate.** With +VKDEVICE green the client waits in a site that is neither the +0x80 word +pattern nor (likely) the A8 count path; the next session should (a) get +the detach-first t104 live, (b) run t102 immediately after VKDEVICE to +name the wait (read PC/bt/A8 count = base+0x2070b7c), and (c) inspect the +service for this client (pushes decoded / releases applied / counter +pub_calls) to see whether the allocation needs the live injections only +or a service-side gap. + +**17. Session 95: THE GDB THREAD-STOP INTERFERENCE + the SIGKILL nuance.** +The t105 attach (t=1.5 s, before the client's worker threads spawned) left +four worker threads in `ptrace_stop.part.0` (futex) - the client's main +thread span waiting on them, so the client froze at the post-device wait +with the gdb holding the trace. TERM/INT/HUP on the wedged gdb were all +ignored (batch gdb stuck in continue). SIGKILL on the gdb (1584) was tried +with the reasoning that all stopped threads sat at futex (NO RM lock held): +result - the tracer died, the tracees DETACHED+RESUMED (client 1518 +SURVIVED with its workers back in futex_wait_queue; contrast session 88 +where SIGKILLed-tracer tracees were KILLED mid-syscall). So the SIGKILL +outcome depends on the tracee's stop context; the mission's never-SIGKILL +rule stands, but this datapoint matters when judging recoverability. +After the gdb died, 1518 still held the RM write lock from its own +post-device wait (its wait word was no longer polled because unwait.py +tracks the NEWEST vk_frame and a fresh test client 6555 was launched and +blocked D-state on os_acquire_rwlock_write) - the unwait-newest coupling +must be replaced by ALL-clients polling in the next form. + +**18. HARNESS DESIGN RULES (accumulated).** (1) A gdb attached across +thread-spawn windows stops worker threads and freezes the client - attach +only AFTER all threads exist, or ensure the gdb explicitly resumes new +threads, or do the work with NO gdb. (2) Any gdb that must run through +thread-spawn windows MUST self-detach quickly (the instance phase spans +~10-20 s and the clears arrive in ~5 bursts) and NEVER rely on outer +timeouts. (3) unwait.py should poll EVERY live vk_frame (not just the +newest). (4) The durable answer for all of this is the service-side +folding: the instance markers and the +0x80 words are guest-RAM writes a +kernel-side service can perform on the same triggers (the marker semantics +and the word semantics are understood) - that removes gdb and /proc +helpers from the loop entirely. + +**19. Session 96: THE DETACH-FIRST CLEARER PROVEN (t106) + the orphan-tracer plague.** +t106.py (single +0xB33D30 BP; clears markers; detaches FROM THE HANDLER as +soon as cleared>=5 or chits>=14: `gdb.execute("detach")` + return True so +the batch gdb exits) on fresh r696 (mdev 241af1c1): CLEAR#1..#5, +`BC-DETACHED cleaned=5 chits=5`, `T106-DONE chits=5 cleared=5`, +`TracerPid: 0` - THE SLOT IS LEFT FREE. unwait.py wrote five words and the +client (1549) reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok +family=0`. THE CLEAN FORM OF THE CLEARER EXISTS NOW. + +**20. THE ORPHAN-TRACER PLAGUE (root cause of every t102 failure).** Every +INTERRUPTED gdb attempt leaves an attached orphan: (a) piping gdb to +`head` closes the pipe early and the gdb stays attached (hit twice); (b) an +ssh command dying/timing out mid-gdb-add leaves the gdb running (hit +again). Symptoms: later attaches fail with `Could not attach ... already +traced by process N` / `ptrace: Inappropriate ioctl for device` and the +t102 log stays empty. RECOVERY (session-95 evidence): TERM is usually +ignored (wedged in continue); SIGKILL the orphan + `kill -CONT <client>`; +the tracee SURVIVES when it was running/at-futex and detaches cleanly +(TracerPid 0). STRICT DISCIPLINE: never pipe gdb output (redirect to a +file, read it after); never let an ssh command end while a gdb may be +attached (generous timeouts); after every gdb run verify TracerPid 0. + +**21. THE NEXT FORM (t107, one gdb, one window).** Combine the proven +clearer and the diagnostic in ONE batch gdb: attach -> clears -> detach -> +`python time.sleep(...)` idle -> RE-ATTACH (gdb can attach again after +detach) -> read PC / bt / A8 count (t102 content) -> detach -> print - +one ssh session, one window, everything self-contained, no orphan risk +(every path ends in an explicit detach; wrap each phase in try/except with +a final detach). Expected: names the post-device wait (allocation/UVM vs +A8 count base+0x2070b7c) so the allocation can be passed and the first +VKFRAME reached. + +**22. Session 97: THE DEEP SUBMISSION STATE + THE COUNTER-SLOT WEB.** +The r696 client (1549) reached the deepest state yet: the service reported +`gpfifo done=11576 pushes=10932 counter pub_calls=1356` and climbing +(gpfifo done was 0 in the pre-pipeline boots) - the client submits the full +work stream and the service decodes/processes it (pairs=14395 map=14318, +ipins=9240 from live_walk). The client then sits in the KNOWN userspace +post-device spin (main thread `running` in all samples; one tid in +os_acquire_rwlock_read). NEW: a counter-slot collision spam +(`counter auto ... pa=0x5fedfff0 rejected (slot owned by another channel; +not installed)`, 232+ callbacks suppressed) - the 0x20400f000 injection +resolved the WHOLE counter VA family to ONE page, so both live channels +collided; the manual bypass `echo <h>,<pa> > .../parameters/sema_counter` +(h_object,pa syntax, main.c:2965) was applied for both handles +(0xbeee0100,0xbeec0100 -> 0x5fedfff0) and pub_calls jumped 1356 -> 3739. +The client gate (VKDEVICE -> frames) persists: its remaining wait is the +userspace spin word NOT covered by the +0x80/16K poller (candidate: the +tracker/counter the service now advances - watch whether pub traffic +alone resolves the spin, else t107-GDB the spin site). ENVIRONMENT: one +orphan gdb (10092 on 1549) + stale channels (beec0100) from prior +attempts; the orphan-kill discipline applies (SIGKILL by PID + CONT + +verify TracerPid). + +**23. THE IOCTL-ARG SAMPLING AXIS (no ptrace).** /proc/<tid>/syscall shows +the running ioctl's ARGS (fd, cmd, arg) when caught - sampling the main +thread repeatedly during an ioctl-loop phase names the exact stuck RM +control without any gdb. Use it before ptrace-based diagnostics when the +thread does syscalls at a catchable rate. + +**24. Session 98: THE POST-DEVICE SPIN NAMED - THE SEC2 SCRUBBER AUTH-TAG WAIT.** +perf (no ptrace!) on the spun client: top symbol `channelReadChannelMemdesc` +24.33% (then scheduler internals) - the guest's nvidia driver hot-loops +reading the CHANNEL MEMDESC. Source decode (nvidia-570.144-reference, +sec2_utils.c): the only reference caller is +`_sec2utilsUpdateGetPtr` -> `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` inside `_sec2utilsGetNextAuthTagSlot`, whose loop +waits for the scrub auth-tag slots to free; `authTagBufSemaOffset = +finishPayloadOffset + CHANNEL_ENGINE_SEMAPHORE_SIZE` (4). So the spin = +the SEC2 MEMORY-SCRUBBER channel waiting for its auth-tag semaphore (a u32 +inside the channel memdesc buffer) to advance - the GPU never delivers it +in our emulation, so the client waits forever (each poll = one full memdesc +transfer = the 24% cost). THE FIX DIRECTION: the service must complete the +scrubber's semaphore updates - the scrubber channel registers in the +service (`scrubber channel 0x... detected (gpfifo=... userd=...)`) and its +submits carry semaphore releases targeting the auth-tag VA +(pbGpuVA + authTagBufSemaOffset) in the channel buffer; on decode, write +the release value to that VA's page (the channel buffer is a guest-RAM +allocation the service can resolve/pin; the pending queue holding 17 +entries is the suspect pool for the unresolvable ones - inspect +svc_pending[]). Service-side counter to watch after the fix: the scrub +loop exits, the client proceeds to the allocation/pipeline. Tool: perf +record/report on the guest is the no-ptrace spin-pointer (`perf record -p +<pid> -- sleep 3; perf report --stdio --sort=symbol,dso`). + +**25. Session 99: THE SCRUBBER GATE CONNECTED TO THE CE CHANNELS' ok=0.** +chan_dump on r696 (client 1549 spun): six channels - +`h=0xbeee0100 eng=0 gpfifo=0x44a0000 userd=0xfee00 ok=1 get=6 sub=6354 rel=0x4213020 crel=0x5fedfff0 notif=0x11cd51060` +`h=0xbeec0100 eng=0 gpfifo=0x4cc0000 userd=0xfee00 ok=1 rel=0x4923020 crel=0x5fedfff0` +`h=0xbeec0104 eng=0 gpfifo=0x4d20000 userd=0xfee00 ok=1 rel=0x4b17e30 crel=0x5fedffe0` +and the CE-class channels `beee0101 (eng=11, gpfifo=0x4500000 userd=0x8f600 as=2 ok=0)`, +`beee0102 (eng=12, gpfifo=0x4560000 userd=0x8f400 ok=0)`, `beed0101 (eng=12, +gpfifo=0x49c0000 userd=0x8f200 ok=0)` - ALL ok=0: the CE userd discovery +(failed probe: EAGAIN 0/0 or EINVAL) means no CE pump, no CE sema applies, +so the SEC2 scrubber channel (a CE-class channel) never sees its auth-tag +semaphore advance and the client spins in _sec2utilsGetNextAuthTagSlot. +The service's candidates (registered PA, ring+0x10000 in both apertures, +8-slot list) all miss for the CE channels. `find_userd` (FB scan for a +live put@0x8c/get@0x88 pair) found ZERO candidates - the CE userds are not +in FB with live puts (either the client's PUT writes go to a page the +service's candidates miss, or the CE userds genuinely idle). A raw host-RAM +shape scan (put in 1..65535, get<=put) is too noisy (24 hits in the first +28KB of BIOS-era low RAM) - needs tighter validation. NEXT APPROACH: find +the REAL userd page by DIFF-WATCH - sample the candidate pages (and the +CE ring pages) twice around a scrub submit and diff for the page whose ++0x88/+0x8c pair changes; the guest kernel writes GP_PUT there, so the +changing page is the true userd. Then fix the discovery (or add a +sema_userd-style injection param) so the CE pump engages and the scrubber +completes. NOTE: the GR channels' userd=0xfee00 (as=2, FB) and ok=1 with +get=6/sub=6354 - the GR machinery runs fine; only the CE/scrubber path is +dark. + +**26. Session 100: THE CE USERD HUNT EXHAUSTED THE STANDARD LAYOUTS; THE CIRCULAR WAIT.** +Candidates examined for the CE userds this session: (a) the registered FB +offsets (0x8f600/0x8f400/0x8f200 as=2) - ALL ZERO in the FB shadow +(sema_vidmem_dump 0x8f100,0x600 twice); (b) the ring-PA + 0x10000 linear +guess - random/zero/table garbage (the allocations are NON-CONTIGUOUS); +(c) the PDB walks of gpfifo+0x10000 (0x4501000->pa 0x125e91000, +0x4561000->pa 0x132032000, 0x49c1000->pa 0x11bddd000 via pdb 0x1152f4000) +- ALL THREE PAGES ALL ZERO. The CE userds were never written - and the +deeper reading: the SEC2 scrub flow is a CIRCULAR WAIT: the scrubber waits +for its previous scrub pushes to be CONSUMED (the auth-tag sema advances +only when the GPU runs the scrub) -> the consumption needs the CE pump -> +the CE pump needs a live userd -> the userd is written BY SUBMITS -> the +submits are held by the wait. In a working boot the guest's first scrub +push happens BEFORE the wait loop (the loop only waits when the slots are +full) - so in our boots the FIRST scrub push must have gone into the CE +ring, the service failed to consume it (ok=0/userd dark), and the client +now waits for a completion that will never come. + +**27. THE DIRECT BYPASS (next session, ranked).** (1) Write the auth-tag +semaphore in the CHANNEL BUFFER (memdesc) page directly: the scrub loop's +`_sec2utilsUpdateGetPtr` reads `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` = a u32 at memdesc+finishPayloadOffset+4; advancing +that value frees the slots and unblocks the client WITHOUT the CE pump. +The memdesc = the channel buffer allocation (a VA in the client's space); +resolve it via the PDB (the guest's channel struct gives the buffer VA; +the service's channel registration/alloc params carry it) and write the +incremented value from the service (or the host RAM file). (2) OR decode +the scrub pushes the service already sees (check sema-push for the SCRUBBER +channel - its submits carry the auth-tag VA; the release decode should +target it; the service's release applies would write the sema if the VA +resolves). (3) The registered FB userd offsets being zero suggests the +client's kernel maps the userd through a VA whose PA the discovery misses; +the diff-watch (two samples around activity) remains the fallback proof. + +**28. Session 101: THE LOOP SEMANTICS + perf trace (NO-PTRACE SYSCALL TRACING).** +The wait loop decoded exactly (sec2_utils.c:420-441): +`while (nextPut == pSec2Utils->authTagGetIndex) { gpuCheckTimeout(); +authTagGetIndex = _sec2utilsUpdateGetPtr(); osSpinLoop(); }` - it exits +when the memdesc sema (u32 at pChannelBufferMemdesc + +authTagBufSemaOffset; authTagBufSemaOffset = finishPayloadOffset + 4) +advances past nextPut; it also has a GPU timeout that would error out. +perf trace on the spun client (no ptrace!) revealed the live pattern: +tid 1793 = a 10ms-timed futex loop (WAKE val=1 then +WAIT_BITSET|CLOCK_REALTIME 10ms - the paced retry); tid 1788 = repeated +`openat -> read(fd, 299 bytes) -> close -> futex-wait` - a transient-fd +POLLING READER of a small blob (the fd is opened and closed each cycle, so +fd 28 no longer exists; the client's persistent fds: /dev/nvidia-modeset +(10), /dev/nvidia0 (11/12/14/15), GLCache .toc/.bin (16/17), memfd +\"allocation fd\" (18)). The 299-byte read is the likely userland face of +the memdesc read path (the 24% channelReadChannelMemdesc hot loop). +NEXT: (a) catch the transient fd path by sampling `ls -l /proc/<pid>/fd` +every ~0.2 s in a loop around the burst (or perf trace -s during a burst); +(b) if it is a driver read() of the memdesc, resolve its buffer and write +the auth-tag sema value (advance past nextPut) via the RAM file; (c) the +loop timeout (gpuCheckTimeout) may eventually error the client - the write +must land first. + +**29. Session 102: fd-sampler miss, no nvidia debugfs; the memdesc hunt narrows.** +The 0.15 s /proc/1549/fd sampler over 20 s caught NO transient path (the +open/read(299)/close reader is bursty or dormant between bursts; the static +fds are /dev/nvidiactl, /dev/nvidia0, /dev/nvidia-modeset, a socket, the +GLCache .toc/.bin pair, and a memfd "allocation fd"). No nvidia debugfs +tree exists in the guest. Remaining ranked paths to the auth-tag sema +(memdesc + finishPayloadOffset + 4): (1) content-locate the scrubber's +channel buffer by its method stream (the C5B5/C56F shapes: SET_OBJECT +0xc5c0/0xc5b5, headers 0x200121ed/0x200120a9 observed in the session-78 +window dumps) and read/write the sema inside; (2) the scrubber channel's +own sema-push decodes (its pushes carry the auth-tag VA = pbGpuVA + +authTagBufSemaOffset; grep the boot's sema-pushall for the scrubber +channel id and read the rel= VA - then PDB-walk it); (3) sample the fd +burst narrower (0.02 s) right after a client restart when the burst is +hot. The service-side ideal: on detecting the scrubber channel, resolve +its channel buffer and pre-advance the auth-tag sema (a durable fold). + +**30. Session 103: THE SCRUB STREAM FOUND; ONE RELEASE DELIVERED; THE SEMA PAGE STILL AMBIGUOUS.** +Content-located the CE/scrub init method stream (headers 0x200121ed/ +0x200120a9 searched): full stream at `pa=0x114e98000` = SET_OBJECT +0xC5C0/0xC5B5 (0x20012000/0x20012080 headers), engine-init methods +0x200121ed(3)/0x200120a9(1)/0x2001209c(0x0c000001), bulk methods +(0x8000xxxx), then TWO REPORT_SEM releases in one buffer: +`0x200426c0 00000000 04b19b10 ...` (va=0x4b19b10) and +`0x200426c0 00000000 04b17e30 ...` (va=0x4b17e30), tail 0x2001c00d +70/68. Twins at pa=0x1026ad000+0x87c and pa=0x12afba000+0x87c (identical +content at identical offsets - multi-mapped copies). Release resolution: +`0x4b17e30 -> pa=0x114efbe30 (SYS) cur=1` - THAT release WAS delivered (the +service applied it, cur=1). `0x4b19b10` resolves via the walker's big-half +2MB branch to FB 0x660b10 holding garbage (0xD9F26F94) - poke_vidmem +tests ([0x660b10]=1 then =0x40000000) did NOT flip the client, and the +linear-block guesses (buffer VA 0x4b00000 <-> stream pa 0x114e98000, sema +candidate 0x114eb1b10) read all zero. NEXT: the auth-tag sema page must be +picked by VALIDATION (the existing value must be a small plausible +GetPtr): walk 0x4b19000 returning ALL chains (small/64K/2MB both +apertures), pick the page whose u32 at +0x1b10 (or the offset the memdesc +actually uses) is a small number; or take the buffer aperture from the +channel's alloc params (the service's registration) and walk with that +aperture exclusively. The fix remains: advance that u32 (write via the +host RAM file if SYS, poke_vidmem if FB). + +**31. Session 104: THE SEMA VA 0x4b19b10 IS NOT THE LOOP'S PAGE; THE KPROBE IS THE ANSWER.** +The validated multi-chain walk (small-4KB and big chains BOTH) resolves +VA 0x4b19b10 -> FB 0x660b10 (both chains converge; the guest's own PTs say +the object lives in vidmem). poke_vidmem writes at 0x660b10 (0x1, +0x40000000) and 0x660b14 (0x1, 0x40000000) LANDED (verified via +sema_vidmem_dump: [0x660b10]=0x11112222) yet the client does not flip and +perf still shows channelReadChannelMemdesc at 24.52%. CONCLUSION: the +REPORT_SEM VA 0x4b19b10 (and 0x4b17e30) are NOT the auth-tag sema the loop +reads - they are the GL-init release objects; the real auth-tag sema is at +pChannelBufferMemdesc + finishPayloadOffset + 4 (a VA and page we do not +yet know; finishPayloadOffset is NOT 0x19b10). THE DEFINITIVE INSTRUMENT: +a guest kprobe on channelReadChannelMemdesc (the offset argument = %si = +the authTagBufSemaOffset VALUE; and the pChannel pointer lets us read the +channel struct -> pChannelBufferMemdesc): `echo 'p:crcd +channelReadChannelMemdesc offset=%si' >> /sys/kernel/tracing/kprobe_events` ++ enable the event + read tracing pipe WHILE the loop runs. This names the +offset and the memdesc in one shot, then the sema page = walk(memdesc+off) +and the advance (RAM file if SYS; FB + poke if the converged mapping). + +**32. Session 105: KPROBE BLOCKED (notrace); THE BAR1-PATH ANALYSIS; THE CHANNEL-FIELD OFFSETS.** +(1) The kprobe on channelReadChannelMemdesc is IMPOSSIBLE: the guest's +dmesg says `trace_kprobe: Could not probe notrace function +channelReadChannelMemdesc [nvidia]` - the function is annotated notrace +(ftrace/perf-probe/BPF-kprobe all blocked). The tracefs filesystem had to +be mounted first (mount -t tracefs tracefs /sys/kernel/tracing) and it +already carries UVM kprobes from earlier sessions (uvmt/uvch/uvbp/uvtw/ +uvsv...). The static sec2utils functions (the callers) are absent from +kallsyms (inlined) - no probe target on that path. +(2) BAR1-path analysis: the guest driver's memdesc read uses +TRANSFER_FLAGS_USE_BAR1|SHADOW_ALLOC|SHADOW_INIT_MEM - a CPU read of the +GUEST'S BAR1 mapping, which the mdev mmap maps onto the HOST's vidmem +shadow (the same buffer poke_vidmem writes). The pokes at FB 0x660b10/ +0x660b14 LANDED (verified) yet the loop did not advance, so the memdesc +page is NOT the FB 0x660b10 the PT walk converged on (VA 0x4b19b10 belongs +to the GL-init release object, not the channel-buffer memdesc). +(3) The channel registration parse exposes the guest-visible channel +fields and their alloc-param offsets (gsp_rm_emulation.c +gsp_rm_register_channel: GSPRM_CHAN_GPFIFO_OFFSET/ENTRIES, ENGINE_TYPE, +CID, INSTANCE_MEM, USERD_MEM(+APERTURE), NOTIFIER_MEM(+APERTURE)) - the +channel-buffer/memdesc field (pChannelBufferMemdesc) is not among them; +the next session should dump a CE channel's FULL alloc params (the +service's rate-limited alloc dumps or a fresh boot) and look for the +channel-buffer VA/PA field, then sema = buffer + finishPayloadOffset + 4 +(per-channel offset, e.g. the 0x19b10-style for the GL channel). +(4) Fallback if the field is not in the params: per-VA content hunt for +the auth-tag sema VALUE (the retprobe would have given it; without +kprobes: the sema value = a small slot counter - scan the CE channel's +allocation candidates for a plausible counter that CHANGES across two +samples). + +**33. Session 106: THE CHANNEL BUFFER IS NOT A CLIENT PARAM; THE TAIL SWEEP AND HASH SCAN NEGATIVE.** +NV_CHANNEL_ALLOC_PARAMS (alloc_channel.h:300-341) layout mapped to the +service offsets: gpFifoOffset@8/entries@16, engineType@128, cid@132, +instanceMem@144, userdMem@168, ramfcMem@192, mthdbufMem@216, +hPhysChannelGroup/internalFlags, errorNotifierMem@248... - THE CHANNEL +BUFFER (pChannelBufferMemdesc) IS NOT IN THE PARAMS: it is a +driver-internal memdesc created by channelAlloc, so the alloc-params path +cannot name it (the kprobe/debuginfo path was the right one; it is +notrace-blocked). The finish-payload tail of the stream page +(0x114e98000 +0xd4..+0x118) was swept with 0x40000000 (18 u32s; the +pre-sweep values: +0xd8=0x68000000 finish marker, +0xe0=0x901025a6, ++0xe4..+0xec=0x8000xxxx methods) - no flip. A 16-page high-entropy scan of +0x114e98000..0x114ea8000 for the auth-tag SHA-256 array: ZERO high-entropy +32-byte blocks (the hash array is elsewhere / another allocation). NEXT: +(1) the memmgrMemDescBeginTransfer SHADOW insight - the read happens from +a FRESH kmalloc'd shadow copied from the memdesc each call; the shadow's +source = the memdesc's backing page; the memdesc page must be found by +TRANSFER-TIME properties (not the VA walk); (2) or a guest perf hardware +watchpoint on the driver's BAR1 ioremap VA + the (unknown) memdesc offset; +(3) or the service-side broad assist: increment a RANGE of plausible FB +sema cells periodically (blind but bounded), watching for the flip. + +**34. Session 107: FB ASSIST SWEEPS NEGATIVE - THE READ IS NOT FB-SOURCED; THE TWINS ARE THE NEXT TARGET.** +Two bounded FB "sema assist" sweeps (distinct values per cell, +0x40000000|idx): (a) the walked page 0x660b00..0x661000 (320 cells) - no +flip; (b) the CE-channel allocation cluster 0x8f000..0x90000 (1024 cells, +where the registered userds live) - no flip; perf re-confirmed +channelReadChannelMemdesc 23.13% (the loop still runs). CONCLUSION: the +memdesc read does NOT come from the FB shadow the CPU can poke - the +transfer's SHADOW_INIT must read the memdesc through a path that bypasses +these bytes, or the memdesc backing is SYSTEM memory via an unknown VA. +NEXT (one-minute test): the STREAM TWINS - pa 0x1026ad000 and +0x12afba000 hold BYTE-IDENTICAL copies of the CE method stream (verified at ++0x7c0/+0x880: same 0x68000000 finish marker area, same 0x200120a9/9c +methods) - one of them is very likely the memdesc BACKING in sysmem (the +driver copies the same stream per channel/boot). Sweep BOTH twins' +stream-area pages (reuse the assist code; 1 page each) and watch for the +flip; if it flips, bisect the winning page to the sema cell and make it +durable (host RAM file writes = zero-cost, no poke_vidmem needed for +sysmem). Failing that: content-diff the twins across a driver transition +(the memdesc copy updates during submits; the copy that CHANGES is the +backing). + +**35. Session 108: THE CONTENT-DIFF - 9 ACTIVE PAGES + THE CONSTANT-READ DEDUCTION.** +Page-hash diff over three +/-256KB windows (twins 0x1026ad000/0x12afba000, +stream 0x114e98000) across 15 s: NINE pages change: +0x102690000, 0x102698000 (440 word diffs - a buffer written continuously = +the push copy!), 0x1026a7000 (+0xe8: 934 -> 940 - a slow clean counter), +0x1026b3000 (+0x48: 32768 -> 1 transition), 0x1026b5000 (+0x244/+0x24c: +24987 -> 32765, two u16s at ~650/s), 0x1026c2000 (+0xe8: 232 -> 3, a +wrapping counter mod ~233), 0x1026e6000 (+0x18: 2764 -> 1930, +0x88/+0x90 +decreasing - countdowns), 0x114e6b000, 0x114e6e000. THE DEDUCTION: the +stuck loop's read value must be CONSTANT (its page shows no writes; the +GPU never advances the sema), so the auth-tag sema is a static cell - +likely zero - and the fix is to make the read NONZERO (advance it) at the +right page. None of the 9 changing pages is it (their cells move on their +own). NEXT: (1) restrict the search to CONSTANT pages among the CE +channel's candidate backing (the twins region pages that DID NOT change +but sit adjacent to the active set); (2) target the sema cell: the cell +that equals nextPut (small constant, e.g. 0) at the memdesc+off; sweep the +CONSTANT pages' zero u32s in a bounded ring around the twins/active set +with 1 (the sysmem write is free); (3) the durable service fold once +found. + +**36. Session 109: THE BURST vs HEARTBEAT DISCOVERY + THE 200ms READER.** +perf trace of the whole process over 10 s with -e ioctl: ZERO ioctls from +ANY thread; meanwhile the process shows a strict 200 ms pattern on thread +1788: `openat(filename ptr 0x5eaec740) -> read(fd 28, count 299) -> close` +(the transient fd the fd-sampler missed!) and thread 1793's 10 ms +futex re-arms. The perf-reported 24% channelReadChannelMemdesc (a kernel +function only reachable via RM calls) CONTRADICTS the zero-ioctl window: +the hot kernel work is BURSTY (periodic rounds between the 200 ms +heartbeats), not continuous. The 299-byte blob read every 200 ms = the +client's health poll (unnamed path; candidate: /proc/driver/nvidia/... +informational node - find it by reading /proc/PID/io or by narrowing the +sampler to the 200 ms phase). THE MODEL NOW: the client idles (futex +waits) + wakes every 200 ms to poll a node + periodically runs RM bursts +(the burst = a full retry round hitting channelReadChannelMemdesc). NEXT: +(1) name the 299-byte node (it may BE the wait's condition - a status the +service must change); (2) capture the BURST with perf (a long record + +per-thread breakdown) to see which thread runs the memdesc reads (the RM +burst thread) and correlate with the host-side RPC traffic in the same +window (the host dmesg RPC lines during a burst = the exact RM call!). + +**37. Session 110: THE LOOP IS CONTINUOUS INSIDE ONE LONG IOCTL (THE LOCK MODEL CLOSED).** +(1) The 299-byte/200ms reader: the filename pointer 0x5eaec740 is NEVER in +the client's maps (30 Hz sampler over 15 s, zero hits) yet openat returns +28 every cycle - a decode artifact/transience, and at 5 reads/s it is +irrelevant to the 24% blocker. Treat it as a RED HERRING. (2) A 20-second +perf record (80K samples): channelReadChannelMemdesc 24.65% + a scheduler +family (pick_next_task_fair/update_curr/pick_eevdf/__schedule/ +schedule_timeout/os_schedule ~30% total) = the SEC2 wait loop running +CONTINUOUSLY. The earlier zero-ioctl windows are explained: THE MAIN +THREAD SITS INSIDE ONE LONG-LIVED ioctl (entered hours ago; the loop runs +in the kernel inside it) - and that ioctl HOLDS THE RM WRITE LOCK, which +closes the lock model: tid 1791 waits on the RM read lock, new clients +block D-state (os_acquire_rwlock_write), all consistent. (3) PEBS/ +mem-loads events are unavailable in the guest perf (no memory-address +sampling) - the read address cannot be sampled. Instruments for the sema +address are now exhausted short of the notrace blocker or the blind +service-side assist. +NEXT: (1) the blind assist, refined: the loop's READ is CONTINUOUS and +the sema is CONSTANT - sweep ONLY constant small-value cells in the CE +candidate pages (see session 108's active set for the region), one value +at a time, 2 s per batch, watching the flip; (2) or a graft: since the +loop runs inside the client's blocked ioctl, unloading the CLIENT (kill) +releases the RM lock and the fresh client restarts the same phase - use +fresh-client runs for any A/B of service-side fixes; (3) the durable +service-side fold remains the endgame once the cell is known. + +**38. Session 111: THE FB IS FULLY DISPROVED (WHOLE-FB FILL, NO FLIP); THE SYS-FILL WEDGED THE GUEST (CYCLE NEXT).** +(1) scan_vidmem for the stream signatures (0x200121ed, 0x200120a9) and even +0x68: ZERO hits across 256MB - THE FB SHADOW WAS ALL ZEROS (before any +pokes): the GPU-side objects are never written in our emulation (the +driver->GPU DMA never runs), so the driver's memdesc SHADOW_INIT copies +zeros and the sec2 loop reads 0 forever (nextPut==authTagGetIndex==0 -> +infinite wait; the complete model). (2) The refined assist: swept FB +0x80000..0x90000 (16384 cells, distinct values; 60s/32KB via poke_vidmem) +- no flip. (3) THE DEFINITIVE TEST: `fill_vidmem 0x40000000` filled the +WHOLE 256MB FB shadow in-kernel (`vgsp: fill_vidmem value=0x40000000 +size=0x10000000 done`) - STILL NO FLIP: THE LOOP'S READ DOES NOT COME FROM +THE FB SHADOW AT ALL; the channel-buffer memdesc is SYS-BACKED (or another +backing) in our emulation, not the FB. (4) The follow-up SYS fill (128 +candidate pages near the active set overwritten with 0x40000000 via the +RAM file) CLOBBERED LIVE DRIVER DATA and WEDGED THE GUEST (SSH banner +timeout; QEMU alive) - the next session MUST CYCLE (fresh boot); lesson: +never bulk-overwrite pages adjacent to the identified active set (0x102698000 +was a hot staging buffer). +NEXT: (1) fresh cycle; (2) the sema is a SYS page - apply the full-page +marker fill ONLY to genuinely-constant, non-active pages (re-run the +content-diff first, then fill the constant set in small batches with a +client check after each, NOT the active neighborhoods); (3) or the +service-side durable route: have the SERVICE write the marker into the +channel-buffer page once it can resolve it (the memdesc VA = the CE +channel's buffer; the service's channel registration fields + the PT walk +give the page candidates). + +**39. Sessions 112b-114: THE SCRUB GATE FELL, THE COMPR WALL WAS DECODED AND FLIPPED, AND THE FIRST VULKAN SUBMISSION LANDED (VKFENCES IS THE NEW WALL).** +Context first (sessions 112b/112c, commits `97489187`/`cb7d8ce7`/`752e2b30`/ +`d6ccdaf3`/`8700f65a`): the CeUtils scrubber's finish payload is authored +in-service (bound to the EXACT 4096-entry channel), the client then leaves +`scrubCheckAndWaitForSize` permanently and runs +vkCreateImage -> vkGetImageMemoryRequirements -> vkAllocateMemory; the +remaining `-2` was decomposed to the guest-internal VidHeap `#149`-shape +call (attr COMPR_ANY + 2MB page + owner 0x474C0000) returning 0x51 +NV_ERR_NO_MEMORY; session 113 read that as a 64KB comptag/CBM alloc. This +session REFINED it: the COMPR+2MB class is not always 64KB - the r705 +runs show the *image data* alloc itself (size 0x140000 = 1.25 MB for +640x480x4 + pad, attr 0x1002000, flags 0xc10d) in exactly the same shape. +The "separate 64KB CBM" reading was too narrow; every alloc of the class +{attr&0xF000 == 0x2000, 2MB page, owner 0x474C0000} fails 0x51. + +**A. THE ZERO-TOUCH CREATE RECIPE (NEW, SCRIPTED).** Two live assists are +required per client, both proven again this session: (1) the +instance-phase marker clears at eglcore+0xB33D30 (t106/t99 `BC`, 5 +clears); (2) THE DEVICE-PHASE WAIT-WORD CHASER at eglcore+0xDCEBD9 +(t99 `BW`: word = *(holder+0x58)+*(u32)(obj2+0x60) with +obj2 = *(*(rbx+0x70)+0x58); write 1 when 0). Without (2) the client +reaches `VKDEVICE`'s poll(`/dev/nvidia0`, 10 ms) retry loop and sits +there INDEFINITELY (session-76 state; the a097 engine-monitor worker +keeps polling 0x2080a097 every 1.25 s regardless - its reply words are +still a red herring for the main-thread gate). With (1)+(2) the run is +`VKINSTANCE ok / VKDEVS 2 / VKDEVICE ok family=0` then straight to the +image path. `t99.py`-style attach works too: attaching `BW` to an +already-stuck client writes the word and the client immediately +proceeds (proven on a stuck pid 15797: WORDWROTE x2 -> VKDEVICE ok -> +VKFAIL -2 at the alloc). +NEW TOOLS: `t120.py` (BC+BW+single-shot COMPR->non-COMPR flip), +`t121.py` (BC+BW+full VidHeap trace from #120), `t122.py` (BC+BW+flip +for ANY COMPR+2MB internal alloc, size >= 0x10000), `a8clear.py` (the +no-ptrace count-gate clearer for base+0x2070b7c, reconstructed). + +**B. THE FLIP (THE DECISIVE EXPERIMENT).** With `t122.py`: at the +vidheap ioctl whose params match (attr&0xF000)==0x2000, owner +0x474C0000, size>=0x10000, the script clears attr bits 13:12 +(COMPR_ANY -> NONE) in the client's own params before the kernel sees +them. Result (r705c, reproducible twice on the same boot): +`HIT149 size=0x140000 attr=0x1002000 -> 0x1000000` (image data alloc +SUCCEEDS), a second `HIT149 size=0x140000 attr=0x1022004 -> 0x1020004` +(buffer-class COMPR alloc) also succeeds, and the client runs the WHOLE +pipeline for the first time in project history: +vkCreateImageView -> buffer -> shader module -> descriptor set +layout/pool -> pipeline layout -> compute pipeline -> command pool -> +command buffer -> fence -> vkResetFences -> **vkQueueSubmit OK** -> +`VKFAIL vkWaitForFences(dev, 1, &fence, VK_TRUE, 30000000000ull) -> 2 +(line 459)` - THE FIRST VULKAN SUBMISSION, then the FIRST FENCE TIMEOUT +(30 s, the client's own bound; the run exits rc=1 cleanly). + +**C. THE NEW WALL = THE FIRST FENCE RELEASE.** Host-side evidence in the +submit window: the client's GR channel registers +(`sema-chan: h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ent=32768 +userd=0xfee00 put=11 get=2 ring rc=0 pa=0x1217e3000 ap=1`) - the guest +driver published 11 GPFIFO entries, the emulation's reader consumed 2; +the service still reports the channel-init pushes unread +(`sema-entry: unread h=0xbeec0100 idx=2 va=0x4ca059c len=976 want=0 +exp=0 unread=1`) and `gr_wedge` grew 3 -> 9 over the runs. So the fence's +release is never authored after the submit - this is the session-76/93 +"wall #2" class (push-page readability/completion), now the ONLY gate +between the flipped client and the first `VKFRAME seq/DONE` + PPM. + +**D. DURABLE-FIX LEADS (for the COMPR class, no guest patching).** +(1) memmgrAllocHal_GM107 (570.144) downgrades COMPR_ANY -> NONE when +`pMemorySystemConfig->bDisableCompbitBacking` is set - the guest's +MEMORY_SYSTEM_STATIC_CONFIG comes from the GSP (the MEMSYS_GET_STATIC_ +CONFIG control 0x20800a1c / the GSP static info path); answering it (or +the same knob wherever the guest's config actually comes from) is the +host-side durable equivalent of the flip. 0x20800a1c was NEVER observed +in the RM_CONTROL log, so first find where the guest's config is fed +(control cache? GSP init args?) before coding the handler. +(2) Alternatively locate glcore's COMPR decision (attr construction at +eglcore+0x2c83fb/0x2c8465 and +0x50e038/0x50e894/0x50e8bd per session +113) and answer the capability that drives it. (3) The session-85 +device-local -2 story (2g mdev type) remains a prerequisite context. + +NEXT (ranked): (1) author the fence release for the first submit (the +push-page-5 content-match / pinning machinery; or have the service +complete the fence when it consumes the submit's GPFIFO entries); (2) +then the first VKFRAME seq/DONE + PPM; (3) fold the COMPR flip into a +host-side answer (see D); (4) the mission series (Xorg/GZDOOM, KVMFR, +CUDA 15/15 + soak, test series, video, Windows). + +**40. Session 115: THE FENCE WALL DECOMPOSED - THE CLIENT VAS PAGE DIRECTORY IS UNDISCOVERABLE AND THE PUSHES ARE NONCONTIGUOUS; RING PAGES ARE CONTENT-LOCATABLE AND INJECTABLE, BUT THE RING PAIR ALONE DOES NOT UNLOCK THE PUSH DECODE.** +(1) The fence timeout is upstream of everything else: the sema service cannot +decode the submit's GR pushes (`sema-entry: unread ... va=0x4ca0000 +len=1432` -> `skipping stale entry ... after 185-194 failed rescues`; +`work_gr submitted=0` - no GR forward ever starts). (2) The pump's push +reads resolve ONLY through the client VAS page directory (or pins). The +service has NO pdb for this client: `walk_probe` of the ring and push VAs +prints `discovered pdb=0x0 found=0` and no registered VASes; the r496f +vidmem pdb discovery never ran (it is only called from the RAM ring-scan +latch), and the r496j COPY_SERVER_PDES candidate (0x109b9c000) is garbage +(its content is not a page table). (3) THE RAM RING SCAN MISSES BY +CONSTRUCTION for this client: it sweeps [0, 6 GB) in 12 slices and gives +up (`found no candidate in 12 slice(s)`), but the sweep runs BEFORE the +client's pushes exist and never re-runs - while the ring page content is +provably in RAM (content-located below). (4) CONTENT LOCATION WORKS: the +GR ring page (ring VA 0x4cc0000, first entries {va=0x4ca0000 len=1432, +va=0x4ca0598 len=4}) was found in the guest RAM file by its first two GPFIFO +entries (at GPA 0x12C600000 then 0x12898c000 on later runs - per-run +allocations), and `sema_map_add` injection of the TRUE ring pair is +accepted (`INJECT va=0x4d20000 pa=0x13619d000 : ok`). BUT the pushes do not +become readable from the ring pair alone: the pushbuffer is NONCONTIGUOUS +(the window guess `ring_pa - 0x20000` reads zeros for both the GR and the +CE channels), the pushes are not CPU-mapped in the client process (the +ring GPA is absent from /proc/PID/pagemap of every rw-s nvidia mapping), +and a bottom-up search for the level-3 table (a 4 KB page whose index-2 +big/small entry maps into the ring's 2 MB window) found ZERO candidates in +RAM - the page tables are almost certainly in VIDMEM (r496f's note) or use +a shape the walker does not model. (5) The service's OWN successful read of +one CE init push (`sema-push: h=0xbeec0104 va=0x4d00000 ... rel=0x4b19b10 +pay=1 have_rel=1`, `ops_runs=0`) did NOT apply its release either - +whatever resolved that one read is not reproducible for the GR pushes. +NEW TOOLS: `fence_assist.py` (host: watches a `dmesg -w` capture for the +pump's entry decodes, content-locates the ring pages by their first two +entries in the RAM file, injects the pairs via sema_map_add - proven to +find and inject, but the ring pair alone is insufficient) and +`find_push.py` (guest: tests whether the client CPU-maps the ring GPA). +NEXT (ranked, concrete): (1) find the client VAS page directory: extract the +pdb from the GUEST kernel's RM objects (kcore walk of the VAS/GPU object +for this channel - the GPFIFO ring page content anchors the search), then +register it via the service's VAS table (a new sysfs/debugfs entry point +or the existing candidate path) - this is the "PDB injection" the mission +names; (2) or scan VIDMEM for the pdb with the ring pair as the verifier +(extend `svc_xvas_pdb_discover`'s trigger: call it from the ring-injection +path, not only from the RAM scan latch); (3) make the RAM ring scan re-arm +per channel with the pushes present (its 12-slice give-up is the reason +the auto-discovery never latches on this client); (4) then the submit +decodes, the release lands and `vkWaitForFences` returns -> VKFRAME. diff --git a/tools/guest-diag/wall2/a8clear.py b/tools/guest-diag/wall2/a8clear.py new file mode 100644 index 0000000..261312a --- /dev/null +++ b/tools/guest-diag/wall2/a8clear.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +# a8clear.py - no-ptrace A8/NvOgl count clearer: finds each live vk_frame's +# libnvidia-eglcore base (the off==0 mapping) and writes 0 to the u32 at +# base+0x2070b7c whenever it is nonzero (the count gate t91 cleared via gdb). +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +log = set() +end = time.time() + DUR +while time.time() < end: + for pid in pids(): + base = None + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "libnvidia-eglcore" not in p[-1]: + continue + try: + off = int(p[2], 16) + except Exception: + continue + if off == 0: + base = int(p[0].split("-")[0], 16) + break + except Exception: + continue + if not base: + continue + w = base + 0x2070b7c + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, w, 0) + v = os.read(fd, 4) + if len(v) == 4: + n = int.from_bytes(v, "little") + if n: + os.lseek(fd, w, 0) + os.write(fd, b"\x00\x00\x00\x00") + k = (pid, n) + if len(log) < 40: + log.add(k) + print("A8CLEAR pid=%d count=%u -> 0" % (pid, n), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.5) +print("A8CLEAR-DONE", flush=True) diff --git a/tools/guest-diag/wall2/fasttick.py b/tools/guest-diag/wall2/fasttick.py new file mode 100644 index 0000000..413d3ba --- /dev/null +++ b/tools/guest-diag/wall2/fasttick.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +# fasttick.py - watch the service log for the GR0 channel's ring resolution +# (sema-chan: h=0xbeec0100 ... ring rc=0 pa=0x...) and immediately hand the +# verified {ring VA -> ring PA} pair to the service (sema_pdb_discover), so +# the parked VAS pdb candidate can be verified before/at age-out. +import re, subprocess, sys, time + +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +RING_VA = 0x4cc0000 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = time.time() + dur + fired = set() + seen_pdbs = set() + print("# fasttick start (%.0fs)" % dur, flush=True) + while time.time() < end: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + tail = out.splitlines()[-120:] + for ln in tail: + m = re.search(r"sema-chan: h=0xbeec0100 eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if m: + ring_va = int(m.group(1), 16) + ring_pa = int(m.group(2), 16) + key = (ring_va, ring_pa) + if key not in fired: + fired.add(key) + cmd = "echo 0x%x,0x%x > %s" % (ring_va, ring_pa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("[%s] TRIGGER va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), ring_va, ring_pa, r.returncode), flush=True) + m2 = re.search(r"page directory (?:registered UNVERIFIED after \d+ ms parked \()?pdb=0x([0-9a-f]+) entries=(\d+)", ln) + if not m2: + m2 = re.search(r"VA space 0x[0-9a-f]+ page directory 0x([0-9a-f]+) entries=(\d+)", ln) + if m2: + key = int(m2.group(1), 16) + if key not in seen_pdbs: + seen_pdbs.add(key) + print("[%s] PDB candidate 0x%x entries=%s" % ( + time.strftime("%H:%M:%S"), key, m2.group(2)), flush=True) + time.sleep(0.3) + print("# fasttick end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/fence_assist.py b/tools/guest-diag/wall2/fence_assist.py new file mode 100644 index 0000000..4ea0b0c --- /dev/null +++ b/tools/guest-diag/wall2/fence_assist.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# fence_assist.py - content-locate the client's ring pages in guest RAM from the +# service's own GPFIFO entry decodes (dmesg) and inject the TRUE VA->PA pairs +# via sema_map_add, so the pump's push reads (and its pushbuffer pairing) have +# a correct anchor. +import mmap, re, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" + +def gpa_of(fo): + return fo if fo < 0x80000000 else fo - 0x80000000 + 0x100000000 + +def inject(va, pa, ln=0x1000, flag=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va & ~0xfff, pa & ~0xfff, ln, flag, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("INJECT va=0x%x pa=0x%x : %s" % (va & ~0xfff, pa & ~0xfff, + "ok" if r.returncode == 0 else "FAIL"), flush=True) + +def search(mm, pats, results): + CH = 8 << 20 + size = len(mm) + off = 0 + while off < size: + d = mm[off:off+CH] + for (name, pat) in pats.items(): + if name in results: + continue + i = d.find(pat) + if i >= 0: + results[name] = gpa_of(off + i) + off += CH - 32 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + end = time.time() + dur + pats = {} # name -> pattern bytes + ring_va = {} # name -> ring VA + seen = set() # raw entry lines processed + done = set() # injected names + print("# fence_assist start (%.0fs)" % dur, flush=True) + while time.time() < end: + try: + with open("/tmp/opencode/dmsg_w.txt", "r", errors="ignore") as df: + tail = df.read().splitlines()[-6000:] + out = "\n".join(tail) + except OSError: + out = "" + tail = [] + tail = [ln for ln in tail if "sema-chan:" in ln or "sema-push" in ln] + cur = {} + for ln in tail: + m = re.search(r"sema-chan: entry\[(\d+)\] h=0x([0-9a-f]+) eng=\d+ raw=([0-9a-f]+) va=0x([0-9a-f]+)", ln) + if m: + idx, h, raw, va = int(m.group(1)), m.group(2), int(m.group(3), 16), int(m.group(4), 16) + cur.setdefault(h, {})[idx] = raw + if idx == 0: + ring_va[h] = va # entry[0] va is the push VA (ring_va - 0x20000) + for h, ents in cur.items(): + key = (h, tuple(sorted(ents.items()))) + if key in seen or len(ents) < 2: + continue + seen.add(key) + name = h + pat = b"".join(struct.pack("<Q", ents[i]) for i in sorted(ents)[:2]) + pats[name] = pat + print("# channel h=0x%s entries=%s pattern ready" % (h, sorted(ents)), flush=True) + if pats: + results = {} + threads = [] + # split the map among 4 threads + LO = 0x100000000; HI = 0x180000000 + size = len(mm) + step = size // 4 + class Sl(shim := object): + def __init__(self, a, b): + self.a, self.b = a, b + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + if off < 0x80000000 and b <= 0x80000000: pass + d = mm[off:off+CH] + for (nm, p) in list(pats.items()): + if nm in results: continue + i = d.find(p) + if i >= 0: + results[nm] = gpa_of(off + i) + off += CH - 32 + for k in range(4): + a = 0x80000000 + k * (0x80000000 // 4) + b = 0x80000000 + (k + 1) * (0x80000000 // 4) + t = threading.Thread(target=worker, args=(a, b)) + t.start(); threads.append(t) + for t in threads: t.join() + for nm, gpa in results.items(): + if nm in done: continue + done.add(nm) + # ring page content starts with entry[0]; the ring VA is the + # channel's gpfifo VA = we reconstruct: entry0 va is the push, + # ring = first "gpfifo=0x..." dmesg line for this h + m = re.search(r"sema-chan: h=0x%s eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+)" % nm, out) + rva = int(m.group(1), 16) if m else None + if rva: + print("RING FOUND h=0x%s gpfifo_va=0x%x -> gpa=0x%x" % (nm, rva, gpa), flush=True) + inject(rva, gpa, 0x1000) + time.sleep(0.6) + print("# fence_assist done, injected %d" % len(done), flush=True) + +main() diff --git a/tools/guest-diag/wall2/find_push.py b/tools/guest-diag/wall2/find_push.py new file mode 100644 index 0000000..4bf8ccc --- /dev/null +++ b/tools/guest-diag/wall2/find_push.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +# find_push.py - locate the client's CPU mapping that owns the GR ring page +# (GPA 0x12C600000 as content-located on the host) and print the CPU VA<-> +# GPU VA correspondence for the low GPU VAs (0x4ca0000 etc). +import os, struct, sys, subprocess, mmap, time + +RING_GPA = 0x12C600000 +PUSH_GVA = 0x4ca0000 +RING_GVA = 0x4cc0000 + +def pids(): + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + +def read_pagemap(pid, va): + try: + with open("/proc/%d/pagemap" % pid, "rb") as f: + f.seek((va // 4096) * 8) + d = f.read(8) + if len(d) < 8: + return None + e = int.from_bytes(d, "little") + if not (e >> 63) & 1: + return None + return (e & ((1 << 55) - 1)) << 12 + except OSError: + return None + +def main(): + for pid in pids(): + print("pid=%d" % pid) + try: + maps = open("/proc/%d/maps" % pid).read().splitlines() + except OSError: + continue + for line in maps: + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x1000000: + continue + # scan pages; look for RING_GPA + for va in range(lo, hi, 0x10000): + for off in range(0, 0x10000, 0x1000): + pa = read_pagemap(pid, va + off) + if pa == RING_GPA: + print("FOUND ring page: cpu_va=0x%x (map 0x%x-0x%x %s)" % (va+off, lo, hi, p[-1])) + delta_cpu = (va + off) - RING_GVA + push_cpu = PUSH_GVA + delta_cpu + print(" delta_cpu = 0x%x" % delta_cpu) + for gva in (PUSH_GVA, PUSH_GVA+0x1000, 0x4ca059c, 0x4ca0980, 0x4ca5870, 0x4903000): + cva = gva + delta_cpu + pa2 = read_pagemap(pid, cva & ~0xfff) + print(" gva=0x%x -> cpu=0x%x -> pa=%s" % (gva, cva, hex(pa2) if pa2 else None)) + return + print(" ring GPA 0x%x not found in this pid's mappings" % RING_GPA) + +main() diff --git a/tools/guest-diag/wall2/kc_dump_fn.py b/tools/guest-diag/wall2/kc_dump_fn.py new file mode 100644 index 0000000..e481d24 --- /dev/null +++ b/tools/guest-diag/wall2/kc_dump_fn.py @@ -0,0 +1,21 @@ +import struct, sys +VA = int(sys.argv[1], 16) +N = int(sys.argv[2], 16) if len(sys.argv) > 2 else 0x800 +f = open("/proc/kcore","rb") +hdr = f.read(64) +e_phoff = struct.unpack_from("<Q", hdr, 0x20)[0] +e_phentsize = struct.unpack_from("<H", hdr, 0x36)[0] +e_phnum = struct.unpack_from("<H", hdr, 0x38)[0] +segs = [] +for i in range(e_phnum): + f.seek(e_phoff + i*e_phentsize) + ph = f.read(e_phentsize) + p_type, p_flags, p_offset, p_vaddr, p_paddr, p_filesz, p_memsz = struct.unpack_from("<IIQQQQQ", ph, 0) + if p_type == 1: + segs.append((p_vaddr, p_memsz, p_offset)) +for v, sz, off in segs: + if v <= VA < v+sz: + f.seek(off + (VA - v)) + d = f.read(min(N, v+sz-VA)) + sys.stdout.write(d.hex()) + break diff --git a/tools/guest-diag/wall2/pdbtick.py b/tools/guest-diag/wall2/pdbtick.py new file mode 100644 index 0000000..d65b79d --- /dev/null +++ b/tools/guest-diag/wall2/pdbtick.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# pdbtick.py - content-locate the GR0 ring page (deterministic first three +# GPFIFO entries) in the guest RAM file, then hand the verified +# {ring VA -> ring PA} pair to the service (sema_pdb_discover), which +# verifies the parked COPY_SERVER_PDES candidates and falls back to the +# vidmem walk-root sweep. Runs while the client is alive. +import mmap, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +PAT = struct.pack("<QQQ", 0x0005980004ca0000, 0x8000040004ca0598, + 0x0003d00004ca059c) +RING_VA = 0x4cc0000 + +def gpa_of(x): + return x if x < 0x80000000 else x - 0x80000000 + 0x100000000 + +def main(): + fobj = open(F, "rb") + mm = mmap.mmap(fobj.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + nthr = 8 + step = size // nthr + res = [] + lock = threading.Lock() + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + d = mm[off:off + CH] + i = d.find(PAT) + while i >= 0: + with lock: + res.append(off + i) + i = d.find(PAT, i + 1) + off += CH - 64 + ts = [] + for k in range(nthr): + a = k * step + b = (k + 1) * step if k < nthr - 1 else size + t = threading.Thread(target=worker, args=(a, b)); t.start(); ts.append(t) + for t in ts: t.join() + if not res: + print("RING NOT FOUND") + return None + hits = [gpa_of(x) for x in res] + print("ring candidates:", [hex(h) for h in hits], flush=True) + for gpa in hits: + cmd = "echo 0x%x,0x%x > %s" % (RING_VA, gpa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("sema_pdb_discover 0x%x,0x%x rc=%d" % (RING_VA, gpa, r.returncode), flush=True) + return hits + + +def loop(): + import time as _t + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = _t.time() + dur + fired = set() + while _t.time() < end: + hits = main() + if hits: + new = [h for h in hits if h not in fired] + if new: + fired.update(new) + _t.sleep(1.0) + print("# pdbtick end", flush=True) + +loop() diff --git a/tools/guest-diag/wall2/runbook.py b/tools/guest-diag/wall2/runbook.py new file mode 100644 index 0000000..256d255 --- /dev/null +++ b/tools/guest-diag/wall2/runbook.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# runbook.py - watch for the GR0 channel ring registration, hand the pair to +# sema_pdb_discover, then (after the pdb anchor) issue bounded +# sema_chan_rewind calls so entries skipped before the anchor re-decode. +import re, subprocess, sys, time + +PDB = "/sys/module/vgsp/parameters/sema_pdb_discover" +REW = "/sys/module/vgsp/parameters/sema_chan_rewind" + +def dmesg_tail(n=150): + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + return out.splitlines()[-n:] + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 180 + end = time.time() + dur + fired = set() + rewinds = {} + print("# runbook start (%.0fs)" % dur, flush=True) + while time.time() < end: + for ln in dmesg_tail(): + m = re.search(r"sema-chan: h=0x([0-9a-f]+) eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if not m: + continue + h = int(m.group(1), 16) + gv = int(m.group(2), 16) + rp = int(m.group(3), 16) + key = (h, gv, rp) + if key in fired: + # after the anchor has had time to land, rewind this channel + if h == 0xbeec0100: + t0 = rewinds.setdefault(key, time.time()) + n = int((time.time() - t0) // 8) + done = rewinds.get(("done", key), 0) + if n > done: + subprocess.run(["sudo", "sh", "-c", + "echo all > %s" % REW], capture_output=True) + rewinds[("done", key)] = n + print("[%s] REWIND all (h=0x%x, n=%d)" % ( + time.strftime("%H:%M:%S"), h, n), flush=True) + continue + fired.add(key) + r = subprocess.run(["sudo", "sh", "-c", + "echo 0x%x,0x%x > %s" % (gv, rp, PDB)], + capture_output=True) + print("[%s] TRIGGER h=0x%x va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), h, gv, rp, r.returncode), flush=True) + # report decodes/releases + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + for ln in out.splitlines()[-8:]: + if ("sema-push: h=0xbeec0100" in ln and "have_rel=1" in ln) or \ + "registered after verified" in ln or "rewind" in ln or \ + ("sema-push: h=0xbeec0100" in ln and "len=1432" in ln): + print("LOG %s" % ln.strip()[-170:], flush=True) + time.sleep(0.4) + print("# runbook end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/t101.py b/tools/guest-diag/wall2/t101.py new file mode 100644 index 0000000..436c609 --- /dev/null +++ b/tools/guest-diag/wall2/t101.py @@ -0,0 +1,55 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception as e: + if state["chits"] <= 4: + print("chitfail %s" % e, flush=True) + if state["cleared"] >= 24: + self.enabled = False + print("BC-DISABLED", flush=True) + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T101-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t102.py b/tools/guest-diag/wall2/t102.py new file mode 100644 index 0000000..759d974 --- /dev/null +++ b/tools/guest-diag/wall2/t102.py @@ -0,0 +1,31 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +try: + pc = int(gdb.parse_and_eval("$pc")) + print("PC 0x%x off=0x%x" % (pc, pc - base if base else 0)) +except Exception as e: + print("pcfail %s" % e) +try: + C = base + 0x2070B7C + print("A8COUNT %u" % int(gdb.parse_and_eval("*(unsigned int*)0x%x" % C))) +except Exception as e: + print("countfail %s" % e) +try: + print(gr("bt 6", to_string=True)) +except Exception as e: + print("btfail %s" % e) +gr("detach") +gr("quit") +print("T102-DONE") diff --git a/tools/guest-diag/wall2/t104.py b/tools/guest-diag/wall2/t104.py new file mode 100644 index 0000000..5a4c059 --- /dev/null +++ b/tools/guest-diag/wall2/t104.py @@ -0,0 +1,56 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "t0":time.time(), "last":time.time()} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + state["last"] = time.time() + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 8 or state["chits"] >= 40 or (time.time() - state["last"]) > 25.0: + self.enabled = False + print("BC-EXIT cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T104-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t106.py b/tools/guest-diag/wall2/t106.py new file mode 100644 index 0000000..ceaac44 --- /dev/null +++ b/tools/guest-diag/wall2/t106.py @@ -0,0 +1,61 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "done":False} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + if state["done"]: + return False + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 5 or state["chits"] >= 14: + state["done"] = True + try: + gr("detach") + print("BC-DETACHED cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + except Exception as e: + print("detachfail %s" % e, flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("contend %s" % e, flush=True) +print("T106-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t112c-gdbstub.py b/tools/guest-diag/wall2/t112c-gdbstub.py new file mode 100644 index 0000000..27d0b9d --- /dev/null +++ b/tools/guest-diag/wall2/t112c-gdbstub.py @@ -0,0 +1,42 @@ +import gdb + +gr = gdb.execute + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +state = {"hits": {}} + +class HB(gdb.Breakpoint): + def __init__(self, pa, tag, quiet_after=8): + super().__init__("*0x%x" % pa, type=gdb.BP_BREAKPOINT, internal=False) + self.tag = tag + self.quiet_after = quiet_after + self.silent = True + def stop(self): + n = state["hits"].get(self.tag, 0) + 1 + state["hits"][self.tag] = n + try: + rdi = int(gdb.parse_and_eval("$rdi")) + rsi = int(gdb.parse_and_eval("$rsi")) + rdx = int(gdb.parse_and_eval("$rdx")) + rcx = int(gdb.parse_and_eval("$rcx")) + r8 = int(gdb.parse_and_eval("$r8")) + r9 = int(gdb.parse_and_eval("$r9")) + if n <= self.quiet_after or self.tag in ("PRED", "COMPR"): + print("HIT %s #%d rdi=0x%x rsi=0x%x rdx=0x%x rcx=0x%x r8=0x%x r9=0x%x" % + (self.tag, n, rdi, rsi, rdx, rcx, r8, r9), flush=True) + except Exception as e: + print("err", e, flush=True) + return False + +HB(0xffffffffc0c403b0, "VIDH") +HB(0xffffffffc0d667a0, "PMA", 4) +HB(0xffffffffc0d6ae30, "PRED") +HB(0xffffffffc0d57d60, "COMPR", 16) +HB(0xffffffffc0d607f0, "SCRUB", 4) +print("HB-SET", flush=True) +gr("continue") +print("T112B-DONE %s" % state["hits"], flush=True) diff --git a/tools/guest-diag/wall2/t120.py b/tools/guest-diag/wall2/t120.py new file mode 100644 index 0000000..68a1b8b --- /dev/null +++ b/tools/guest-diag/wall2/t120.py @@ -0,0 +1,148 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2 and size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + state["n"] += 1 + print("HIT149 n=%d attr=0x%x flags=0x%x owner=0x%x align=0x%x" % ( + state["n"], attr, flags, owner, u64(arg + 0x60)), flush=True) + if state["n"] == 1: + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t121.py b/tools/guest-diag/wall2/t121.py new file mode 100644 index 0000000..1ae25ed --- /dev/null +++ b/tools/guest-diag/wall2/t121.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 attr=0x%x flags=0x%x" % (attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t122.py b/tools/guest-diag/wall2/t122.py new file mode 100644 index 0000000..8c9c6fd --- /dev/null +++ b/tools/guest-diag/wall2/t122.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size >= 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 size=0x%x attr=0x%x flags=0x%x" % (size, attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/unwait.py b/tools/guest-diag/wall2/unwait.py new file mode 100644 index 0000000..94e677c --- /dev/null +++ b/tools/guest-diag/wall2/unwait.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +# unwait.py - guest-side, zero-ptrace wait-word poller: writes 1 to the +0x80 +# u32 of every zero 16 KB rw-s nvidiactl mapping of the newest vk_frame. +import os, re, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def newest_pid(): + try: + out = subprocess.run(["pgrep", "-n", "-x", "vk_frame"], capture_output=True).stdout + return int(out.decode().strip()) + except Exception: + return None +def maps(pid): + try: + return open("/proc/%d/maps" % pid).read() + except Exception: + return "" +logged = set() +end = time.time() + DUR +while time.time() < end: + pid = newest_pid() + if pid: + for line in maps(pid).splitlines(): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + try: + lo, hi = [int(x, 16) for x in p[0].split("-")] + except Exception: + continue + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, wa, 0) + v = os.read(fd, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + os.lseek(fd, wa, 0) + os.write(fd, b"\x01\x00\x00\x00") + key = (pid, wa) + if key not in logged: + logged.add(key) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.4) +print("UNWAIT-DONE", flush=True) diff --git a/tools/guest-diag/wall2/unwait2.py b/tools/guest-diag/wall2/unwait2.py new file mode 100644 index 0000000..cd4169b --- /dev/null +++ b/tools/guest-diag/wall2/unwait2.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +# unwait2.py - no-ptrace guest-side harness: +# (a) writes 1 to any zero u32 at +0x80 of every 0x4000-byte rw-s nvidia +# mapping of EVERY live vk_frame; +# (b) scans each live vk_frame's anonymous/heap rw-p mappings (16-byte +# aligned) for marker slots {u32@0==0, u16@+0xe==0xffff} and clears +# u16@+0xe to 0 (the instance-phase pending markers the t105 gdb used +# to clear). No ptrace anywhere. +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +def rd(fd, off, n): + try: + os.lseek(fd, off, 0) + return os.read(fd, n) + except Exception: + return b"" +def wr(fd, off, data): + try: + os.lseek(fd, off, 0) + os.write(fd, data) + return True + except Exception: + return False +wlog = set(); mlog = set() +end = time.time() + DUR +last_scan = 0.0 +while time.time() < end: + for pid in pids(): + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + except Exception: + continue + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + v = rd(fd, wa, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + if wr(fd, wa, b"\x01\x00\x00\x00"): + k = (pid, wa) + if k not in wlog: + wlog.add(k) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + except Exception: + pass + if time.time() - last_scan > 1.5: + budget = 64 + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or p[1] != "rw-p": + continue + path = p[-1] + if path.startswith("/") and "nvidia" not in path and path != "[heap]": + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x10000000: + hi = lo + 0x10000000 + off = lo + while off < hi: + chunk = rd(fd, off, 0x100000) + if len(chunk) < 32: + break + j = 0 + while j + 16 <= len(chunk): + if chunk[j:j+4] == b"\x00\x00\x00\x00" and chunk[j+0xe:j+0x10] == b"\xff\xff": + wa = off + j + 0xe + if (pid, wa) not in mlog: + if wr(fd, wa, b"\x00\x00"): + mlog.add((pid, wa)) + print("MARKCLR pid=%d va=0x%x" % (pid, wa), flush=True) + j += 16 + off += 0x100000 + budget -= 1 + if budget <= 0: + break + if budget <= 0: + break + except Exception: + pass + last_scan = time.time() + os.close(fd) + time.sleep(0.4) +print("UNWAIT2-DONE", flush=True) diff --git a/tools/lg/build_video.sh b/tools/lg/build_video.sh new file mode 100755 index 0000000..4f80be8 --- /dev/null +++ b/tools/lg/build_video.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# build_video.sh - build the final open-iov 1.0.0 demo video: a drawtext +# title card, the split-screen main segment (two GZDOOM VMs, one per pane, +# captured through the KVMFR ivshmem path), and an optional end card, +# concatenated into one 1280x720@30 libx264 yuv420p mp4. +# +# Usage: +# build_video.sh <title-text> <main.mp4> [end-text] +# build_video.sh <title-text> <dirA> <dirB> [end-text] +# title-text text for the title card (required, non-empty) +# main.mp4 pre-built split-screen video (e.g. from split_screen.sh) +# dirA dirB two frame-%04d.ppm dirs; split_screen.sh (next to this +# script) is invoked first at FPS to build the main segment +# end-text optional text for the end card (omitted = no end card) +# +# Output: $OUT if set, else ./open-iov-1.0.0-demo.mp4 in the current dir. +# Env knobs: FPS (default 30), CARD_SECS (default 5), FONTSIZE (default 48). +# +# Approach: title/end cards are generated with ffmpeg drawtext (white on +# dark 0x101010, 1280x720); the main segment is normalized to 1280x720@30 +# (aspect-preserving fit + letterbox pad, audio dropped - the demo video is +# video-only); the three segments are concatenated with the concat demuxer +# followed by ONE re-encode. Stream-copy concat is brittle across +# separately-generated mp4s (timestamp/param mismatches); the re-encode is +# cheap at these lengths and guarantees a clean single-stream output. +# The title font is auto-detected (DejaVuSans/NotoSans); the script fails +# with a clear message if no TTF font exists. +# +# Requires: ffmpeg with libx264 + drawtext (libfreetype); split_screen.sh +# next to this script for the two-dirs form. +set -euo pipefail + +usage() { + sed -n '3,31p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "build_video.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" +ffmpeg_has drawtext -filters || die "ffmpeg has no drawtext filter (needs libfreetype)" + +# --- locate a usable TTF font ------------------------------------------- +FONT="" +for f in \ + /usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf \ + /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf \ + /usr/share/fonts/truetype/noto/NotoSans-Regular.ttf \ + /usr/share/fonts/dejavu/DejaVuSans.ttf; do + if [ -f "$f" ]; then FONT=$f; break; fi +done +if [ -z "$FONT" ] && command -v fc-match >/dev/null 2>&1; then + f=$(fc-match -f '%{file}' sans 2>/dev/null || true) + [ -n "$f" ] && [ -f "$f" ] && FONT=$f +fi +[ -n "$FONT" ] || die "no usable TTF font found (looked for DejaVuSans, NotoSans; \ +install fonts-dejavu-core or set up fontconfig)" +echo "build_video.sh: font=$FONT" >&2 + +# --- args ---------------------------------------------------------------- +[ "$#" -ge 1 ] || { usage; exit 2; } +TITLE=$1; shift +[ -n "$TITLE" ] || die "title text must be non-empty" +MAIN="" DIR_A="" DIR_B="" END="" +if [ "$#" -ge 1 ] && [ -d "$1" ]; then + DIR_A=$1 + [ "$#" -ge 2 ] && [ -d "$2" ] || die "two-dirs form needs two directories: <dirA> <dirB> [end-text]" + DIR_B=$2 + END=${3:-} +elif [ "$#" -ge 1 ]; then + [ -f "$1" ] || die "main input not found: $1" + MAIN=$1 + END=${2:-} +else + die "missing main input (mp4 file or two frame dirs)" +fi + +FPS=${FPS:-30} +CARD_SECS=${CARD_SECS:-5} +FONTSIZE=${FONTSIZE:-48} +OUT=${OUT:-$PWD/open-iov-1.0.0-demo.mp4} +HERE=$(cd "$(dirname "$0")" && pwd) + +TMP=$(mktemp -d "${TMPDIR:-/tmp}/build_video.XXXXXX") +trap 'rm -rf "$TMP"' EXIT + +# --- cards (drawtext; text via textfile so ':' ',' '%' etc. are literal) -- +make_card() { + local text=$1 out=$2 tf="$TMP/card-$$.txt" + printf '%s' "$text" > "$tf" + ffmpeg -hide_banner -loglevel warning -y \ + -f lavfi -i "color=c=0x101010:s=1280x720:r=$FPS:d=$CARD_SECS" \ + -vf "drawtext=fontfile=$FONT:textfile=$tf:fontcolor=white:fontsize=$FONTSIZE:expansion=none:x=(w-text_w)/2:y=(h-text_h)/2" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$out" +} + +echo "build_video.sh: title card (${CARD_SECS}s)..." >&2 +make_card "$TITLE" "$TMP/title.mp4" + +# --- main segment -------------------------------------------------------- +if [ -n "$DIR_A" ]; then + [ -x "$HERE/split_screen.sh" ] || die "split_screen.sh not found/executable next to build_video.sh" + echo "build_video.sh: building split-screen main from $DIR_A + $DIR_B..." >&2 + "$HERE/split_screen.sh" "$DIR_A" "$DIR_B" "$TMP/split.mp4" "$FPS" + MAIN=$TMP/split.mp4 +fi +echo "build_video.sh: normalizing main segment to 1280x720@$FPS..." >&2 +ffmpeg -hide_banner -loglevel warning -y -i "$MAIN" \ + -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=$FPS" \ + -an -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$TMP/main.mp4" + +# --- concat [title, main, end] ------------------------------------------ +printf "file '%s'\n" "$TMP/title.mp4" "$TMP/main.mp4" > "$TMP/list.txt" +if [ -n "$END" ]; then + echo "build_video.sh: end card (${CARD_SECS}s)..." >&2 + make_card "$END" "$TMP/end.mp4" + printf "file '%s'\n" "$TMP/end.mp4" >> "$TMP/list.txt" +fi + +echo "build_video.sh: concatenating..." >&2 +ffmpeg -hide_banner -loglevel warning -y -f concat -safe 0 -i "$TMP/list.txt" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -select_streams v:0 \ + -show_entries stream=width,height,r_frame_rate,duration \ + -show_entries format=duration,size \ + -of default=nw=1 "$OUT" | sed 's/^/build_video.sh: /' >&2 +fi +echo "VIDEO_DONE $OUT" >&2 diff --git a/tools/lg/split_screen.sh b/tools/lg/split_screen.sh new file mode 100755 index 0000000..55e11ee --- /dev/null +++ b/tools/lg/split_screen.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# split_screen.sh - compose a side-by-side (split-screen) mp4 from two PPM +# frame sequences captured through the KVMFR ivshmem path (guest publishers: +# tools/cuda-ab/frame_producer.cu or tools/guest-diag/xshm_pub.c; host reader: +# tools/lg/frame_reader.py, which writes frame-%04d.ppm into a directory). +# +# Usage: split_screen.sh <dirA> <dirB> <out.mp4> [fps] [left_w] [left_h] +# dirA left-side frame-%04d.ppm directory (any starting index) +# dirB right-side frame-%04d.ppm directory +# out.mp4 output file (libx264, yuv420p) +# fps output frame rate (default 30; pass the measured capture rate, +# e.g. 10, for real-time playback of ~10 fps captures) +# left_w optional left-pane width; the left pane is fitted into +# left_w x H with letterbox padding (aspect preserved) +# left_h optional pane height H; BOTH panes are scaled to height H +# (default: native height of dirA's first frame). Widths follow +# each input's aspect ratio (forced even for yuv420p). +# +# Unequal sequence lengths: the shorter side is padded by repeating its last +# frame. hstack is a framesync filter whose repeatlast=1 default holds the +# shorter input's last frame until the longer input ends; shortest=0 is +# passed explicitly to make that the contract (verified on ffmpeg 6.1.1 for +# both input orders). No symlink/tpad workaround is needed. +# +# Fails loudly (exit 1) on: missing ffmpeg/libx264, missing input dirs, or an +# input dir with no frame-*.ppm files. Requires: ffmpeg (libx264); ffprobe is +# used for the final summary when present. +set -euo pipefail + +usage() { + sed -n '3,28p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "split_screen.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +# Read width/height from the PPM P6 header (line 2: "<w> <h>"). +ppm_dims() { + local line w h + line=$(sed -n '2p' "$1") + read -r w h _ <<< "$line" + [ -n "${w:-}" ] && [ -n "${h:-}" ] || return 1 + echo "$w $h" +} + +# Scan a frame dir: sets PD_COUNT PD_FIRST PD_LAST PD_W PD_H PD_FILE +probe_dir() { + local dir="$1" label="$2" files f dim + [ -d "$dir" ] || die "$label directory not found: $dir" + mapfile -t files < <(find "$dir" -maxdepth 1 -name 'frame-*.ppm' -type f \ + -printf '%f\n' | sort) + PD_COUNT=${#files[@]} + [ "$PD_COUNT" -gt 0 ] || die "$label directory has no frame-*.ppm files: $dir" + f=${files[0]} + PD_FIRST=${f#frame-}; PD_FIRST=${PD_FIRST%.ppm}; PD_FIRST=$((10#$PD_FIRST)) + f=${files[$((PD_COUNT - 1))]} + PD_LAST=${f#frame-}; PD_LAST=${PD_LAST%.ppm}; PD_LAST=$((10#$PD_LAST)) + if [ $((PD_LAST - PD_FIRST + 1)) -ne "$PD_COUNT" ]; then + echo "split_screen.sh: WARN: $label sequence has gaps ($PD_COUNT files," \ + "indices $PD_FIRST..$PD_LAST); ffmpeg stops at the first gap" >&2 + fi + PD_FILE=$(printf '%s/frame-%04d.ppm' "$dir" "$PD_FIRST") + dim=$(ppm_dims "$PD_FILE") || die "$label: cannot read PPM header of $PD_FILE" + PD_W=${dim% *}; PD_H=${dim#* } +} + +[ "$#" -ge 3 ] || { usage; exit 2; } +DIR_A=$1 DIR_B=$2 OUT=$3 FPS=${4:-30} LW=${5:-} LH=${6:-} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" + +probe_dir "$DIR_A" left; A_COUNT=$PD_COUNT A_FIRST=$PD_FIRST A_H=$PD_H +probe_dir "$DIR_B" right; B_COUNT=$PD_COUNT B_FIRST=$PD_FIRST + +# Pane height H: left_h if given, else dirA's native height. Keep even. +H=${LH:-$A_H} +if [ $((H % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: pane height $H rounded down to even for yuv420p" >&2 + H=$((H / 2 * 2)) +fi +[ "$H" -ge 2 ] || die "pane height must be >= 2" + +# Left pane: fit into left_w x H (letterbox) when left_w given, else scale +# to height H with aspect preserved. Right pane: always scale to height H. +if [ -n "$LW" ]; then + if [ $((LW % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: left_w $LW rounded down to even for yuv420p" >&2 + LW=$((LW / 2 * 2)) + fi + [ "$LW" -ge 2 ] || die "left_w must be >= 2" + LEFT="[0:v]scale=${LW}:${H}:force_original_aspect_ratio=decrease,"\ +"pad=${LW}:${H}:(ow-iw)/2:(oh-ih)/2,setsar=1[l]" +else + LEFT="[0:v]scale=-2:${H},setsar=1[l]" +fi +RIGHT="[1:v]scale=-2:${H},setsar=1[r]" +# shortest=0 + framesync repeatlast=1 (default): shorter side's last frame is +# repeated until the longer side ends (see header comment). +FILTER="${LEFT};${RIGHT};[l][r]hstack=shortest=0" + +echo "split_screen.sh: left=$DIR_A ($A_COUNT frames from $A_FIRST)" \ + "right=$DIR_B ($B_COUNT frames from $B_FIRST) pane_h=$H fps=$FPS" >&2 + +ffmpeg -hide_banner -loglevel warning -y \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$A_FIRST" -i "$DIR_A/frame-%04d.ppm" \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$B_FIRST" -i "$DIR_B/frame-%04d.ppm" \ + -filter_complex "$FILTER" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -count_frames -select_streams v:0 \ + -show_entries stream=width,height,nb_read_frames \ + -of default=nw=1 "$OUT" | sed 's/^/split_screen.sh: /' >&2 +fi +echo "SPLIT_DONE $OUT" >&2 diff --git a/tools/vk/vk_frame.c b/tools/vk/vk_frame.c index 8b864ff..62548be 100644 --- a/tools/vk/vk_frame.c +++ b/tools/vk/vk_frame.c @@ -27,6 +27,8 @@ #define FRAME_OFFSET 0x100000UL #define HDR_MAGIC 0x4b564d46u +static VkImageTiling g_tiling = VK_IMAGE_TILING_OPTIMAL; + struct frame_hdr { uint32_t magic; uint32_t version; @@ -59,6 +61,11 @@ static uint64_t now_ns(void) int main(int argc, char **argv) { + if (argc > 7 && atoi(argv[7]) == 1) { + g_tiling = VK_IMAGE_TILING_LINEAR; + printf("TILING linear\n"); + } + int frames = argc > 1 ? atoi(argv[1]) : 60; uint32_t W = argc > 2 ? (uint32_t)atoi(argv[2]) : 640; uint32_t H = argc > 3 ? (uint32_t)atoi(argv[3]) : 480; @@ -179,7 +186,7 @@ int main(int argc, char **argv) ci.mipLevels = 1; ci.arrayLayers = 1; ci.samples = VK_SAMPLE_COUNT_1_BIT; - ci.tiling = VK_IMAGE_TILING_OPTIMAL; + ci.tiling = g_tiling; ci.usage = VK_IMAGE_USAGE_STORAGE_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT; ci.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; @@ -244,7 +251,25 @@ int main(int argc, char **argv) break; } } - VKC(vkAllocateMemory(dev, &ma, NULL, &buf_mem)); + { + int rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + if (rc_ != VK_SUCCESS) { + VkPhysicalDeviceMemoryProperties mp2; + uint32_t k2; + vkGetPhysicalDeviceMemoryProperties(phys, &mp2); + for (k2 = 0; k2 < mp2.memoryTypeCount && rc_ != VK_SUCCESS; k2++) { + if (!(mr.memoryTypeBits & (1u << k2))) + continue; + ma.memoryTypeIndex = k2; + rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + printf("BUF retry type=%u rc=%d\n", k2, rc_); + } + } + if (rc_ != VK_SUCCESS) { + printf("VKFAIL vkAllocateMemory(buf) -> %d (line %d)\n", rc_, __LINE__); + return 1; + } + } VKC(vkBindBufferMemory(dev, buf, buf_mem, 0)); } /* shader + pipeline */ |