diff options
208 files changed, 65599 insertions, 1101 deletions
@@ -159,3 +159,14 @@ userspace/sdks/ *.bak-* *.orig *.rej + +# Demo/release videos (task: "0.9.9 release preview" lives in the repo root, +# never tracked - large binary; the file is produced locally from the host +# KVMFR capture). +/*.mp4 +/open-iov-*.mp4 +kernel/golden-mme-tu104.h + +# generated per-cycle launcher + run scratch (r1780 cycle tooling) +tools/guest-diag/launch_qemu_r1780.run.sh +tools/guest-diag/run/ diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 27295df..5762b4f 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,12 +1,667 @@ # Open-IOV Current State -> **Snapshot: 2026-10-20 ~20:00 (two more fixes landed: the SWAP_RX cursor -> and the pin-wins ledger divergence that cured a guest-kernel-corruption -> panic; guest CUDA kernels green on fresh boots - `victory_mail`, -> `victory_matmul` (first-run flake), transformer, `vector_add` (1M, -> `bad=0`), `vecadd_loop` (100 launches); two guests with two emulated VRAM -> sizes (512M + 1GiB) coexist live; frontier = the multi-pass content class -> and the streams2 launch-order wait).** +> **Snapshot: 2026-11-11 ~10:45 (SESSION 156 continued; HEAD `4acac093` == +> `vgpu/master`; working tree carries in-flight kernel work; supersedes the +> 2026-11-10 ~16:45 entry below).** +> The **first non-black GPU-rendered GL output and the first captured video** +> landed on the emulated vGPU. r1764 (`131d44e0`) proved the pbuffer path: the +> probe returns `PROBE rc=0 err=0x0 pixel=ff0000ff` and the off-screen frame +> producer published 40/40 animated 640x480 frames rendered on the physical +> GPU (pbuffer -> sysmem -> kvmfr ivshmem -> host `frame_reader.py`). r1769 +> validated the V6 ring-ledger scrub release (`f6184ac3`): X green 4/4 cycles, +> zero panics/wedges, `gl_frames 500/500` non-black, 454 PPMs, and a 640x480 +> h264 10 fps 45.4 s video +> (`backups/evidence-20261002-victory/r1769/vgpu-gl-animation.mp4`). The +> scrubber release value is the guest's `lastSubmittedWorkId`; the ring ledger +> counts GPFIFO pushes and advances fin forward-only (never all-ones), so the +> fin=12 underflow class is closed. +> +> Commit `4acac093` (V9/V10) resolves the **direct-GL GR ring `0xbeef0100`** +> (beef-family always-probe, per-channel promote backoff, rank-collision +> fallback, parked-candidate eviction). Receipts: `RING-VAS` mapped, RAM scan +> `va=0x4020000 -> pa=0x1ee8000`, 132 `sema-push`, 101 `sema-sw` releases +> advancing `0x1 -> 0x27` (39 frames); glxgears 504-516 FPS with no +> regression; a ~120 s GZDoom run had zero DDX channel-idle timeouts. GZDoom +> now **passes the VBO wall whenever the shared nvClip `modeNum` is even**: +> on a fresh X it logs window -> `GL_VERSION 4.6.0` -> 640x480 -> `I_Init` -> +> `V_Init: allocate screen` with 0 SIGSEGV, and its direct-GL channels +> register with the GR lane live. The SIGSEGV at `0x20` was root-caused +> (coredump + LD_PRELOAD shim + host gdb) to `FFlatVertexBuffer::Copy` -> +> `memcpy(NULL, ..., 640)` because `glMapBufferRange`/`glBufferData` return +> NULL + `0x505` when the client's `__glNVStateInit` fails the modeswitch +> test with an odd `modeNum` (live read: wedged X = 3, fresh X = 2). The +1 +> is the guest kernel's nvKms deferred close (the host completes +> UPDATE -> FINISHED but no control arrives at the +1) - guest-side, not +> module-induced. +> +> **New top wall:** the direct-GL **CE ring `0xbeef0101` (gpfifo=0x4100000) +> never resolves** - `VAS-PROMOTE cand pdb=0x108eb3000 walk rc=-2` plus +> `NOT mapped by any registered VAS` every ~7 s - so GZDoom's CE work never +> executes and the guest wedges ~90 s after `V_Init`. Next: walk all parked +> candidates regardless of owner VAS and/or keep >=2 fresh same-VAS parks. +> **Window/screen pixels remain unproven:** glxgears' window is still black +> (the GLX probe/frames path is a per-window lottery), and X window creation +> wedges on 3/6 boots in an unkillable Xorg spin (`_nv052600rm+0x7b`); one +> cycle also saw 2x "Wait for channel idle timed out". +> +> Walls ranked: (1) direct-GL CE ring `0xbeef0101` resolution (blocks the +> GZDoom game loop); (2) window/screen pixels and the GLX per-window lottery; +> (3) X startup/stability and the VASpace/ScreenInit allocation lottery +> (`_nv052600rm+0x7b`, `0x51`, vmalloc); (4) the DDX 4 GB sysmem +> allocation/OOM (12 GB guest); (5) sustained non-black window frames. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): HEAD `4acac093` == `vgpu/master`; working tree +> `M kernel/gsp_sema_service.c` (in-flight campaign work); evidence under +> `backups/evidence-20261002-victory/r1769/`, `r1774/`, `r1775/`; pstore +> empty; no reboot during the check. +> +> **Snapshot: 2026-11-10 ~16:45 (SESSION 156 continued; HEAD `df6e71bb`, +> tree has in-flight kernel/tool work; supersedes the 06:10 entry below).** +> The **first real GPU readback** landed: a guest +> `glClear(red)+glFinish+glReadPixels` pbuffer probe returned +> `rc=0 err=0x0 pixel=ff0000ff` (r1734), with the host logging +> `section captured` plus `GR LAUNCH submitted` on the DDX channel +> `0x1000014`. That proves the emulated path can render and read back through +> the unmodified guest driver. The RM write-lock holder was identified as +> Xorg in the DDX vidheap scrub wait (NVOS32 `ALLOC_SIZE` 0xa0000, spinning +> on `vmap+0x6c004`); `df6e71bb` authors the all-ones release pair for a +> scanner-registered scrub buffer with no USERD GP_PUT, which released the +> lock and made the readback work. +> +> Still open: the direct-GL client channel `0xbfef0002` opens a C597 section +> but emits only 32/64-byte setup/notify kicks (`ops_runs=0`, `section +> captured`/`GR LAUNCH` 0) - the GL draws never reach the host. A synchronous +> window draw wedges X (`xwd -root` writes 0 bytes) while the pbuffer probe +> works, so screen capture must use the ivshmem frame path. X startup remains +> a lottery (`Failed to initialize DMA` / push-buffer `0x51` / occasional +> vmalloc 96 GB) and X can die after forward activity. No non-black screen +> pixels yet. Milestone position: between M-GL1 and M-GL2; the readback pixel +> is execution proof, not a screen frame. +> +> Walls ranked: (1) direct-GL draw submission/capture on `0xbfef0002`; +> (2) window/screen present and X stability; (3) X startup/RM degradation +> (`0x51`); (4) the DDX 4 GB sysmem allocation/OOM (12 GB guest); (5) +> non-black screen frames and sustained rate. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): host up ~42 h (fresh boot 2026-11-08 22:29); +> `vgsp` loaded + live cycle (QEMU 128357, mdev +> `c949938a-0391-4aa0-8d49-1679878fe5cd`); `tree` has the campaign's in-flight +> kernel/tool work (HEAD `df6e71bb`); pstore empty; no reboot during the +> check. +> +> **Snapshot: 2026-11-10 ~06:10 (SESSION 156 continued; HEAD `bb802909`, +> tree has in-flight kernel work from the campaign; supersedes the 05:30 +> entry below).** +> The X and GR reliability work landed. `ebf3130f` fixes the X DMA-init path +> (the pristine-FB USERD scan now runs when the ring is present but unusable, +> the foreign-FB claim is offset-correct, the parked VAS candidate is +> content-validated at the pump turn, and the owner-VAS fallback covers +> `hVASpace==0`) and flips the GR pair-close/pair-safe replay fixes to default +> ON. Result: **X direct rendering green on attempt 1 in three consecutive +> cycles (r1687/1688/1689)**, `backend_timeout=0` (was 6), `backend_ok=5-7`, +> and draws retire in 3-2400 ms. `bb802909` (WIP beef-ring promote + +> draw-channel CB scan) is committed; the campaign has uncommitted kernel work +> in flight. +> +> The remaining wall is still the **replayed draw writes 0 pages** (render +> target unchanged; `bounce copy-out` reports 0 changed). It is now precise: +> (1) the beef GR ring resolution race - `0xbeef0100` registers with `vas=0` +> before its CE siblings publish `0xbeef0300`; the VAS is then registered +> UNVERIFIED by the 1 ms age-out, so the ring at `0x4020000` stays +> `ring_pa=0` and the fixed-CB scan (gated to the `0xbeef0000` family) never +> runs. The fix is to gate the candidate promote on *verified* rather than +> *known* and retry every pump turn. (2) Even when the scan runs (`r1687`), it +> decodes 1492 `0x3800` offsets plus 1497 `0x39c8` data runs (1092 words) but +> the image's `0x80` MVP block is zero; identified parser bugs (subchannel +> bits 15:13, `0x238c` ONE_INC inline data, selector B/C swap) are not yet +> applied, and the MVP may live at an offset >= `0x1000` or a selector B/C +> binding. (3) The render-target bind (`svc_surf_prelude_capture`, +> last-trusted-bind-wins) must be confirmed to match the page the copy-out +> compares. +> +> The 2D path is green end-to-end, the guest's real GL draws forward and +> execute on the physical GPU, and X is reliable; no rendered pixel through +> the emulated vGPU yet. Milestone position: between M-GL1 and M-GL2. +> Estimate (subjective): first non-black frame hours-to-days if the beef-ring +> and CB decode behave like their predecessors; the two-VM GPU-accelerated +> video after that. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): r1691 VM up (QEMU 80456, mdev +> `eba5a4f0-ef85-4ea5-9fb4-4ef1a6b04a3b`), X green on attempt 1, glxgears +> 300x300 black, `work_gr backend_ok=5 timeout=0`, `bounce copy-out 0 +> changed`, beef ring unresolved; `vgsp` loaded (HEAD `bb802909` plus the +> campaign's in-flight kernel work), `guest_ram active=1`; pstore empty; no +> reboot during the check. +> +> **Snapshot: 2026-11-10 ~05:30 (SESSION 156 continued; HEAD `cb8e822b`, +> tree clean, `vgpu/master` in sync; supersedes the 19:15 entry below).** +> The execution chain came alive overnight. The guest's **real GL draws now +> forward and execute on the physical GPU** (`work_gr backend_ok=9` on r1683, +> `backend_ok=5` on r1685), the 2D path is green end-to-end (accelerated X +> fills read back correctly via `xwd`), and the earlier glxgears poll hang is +> gone: the GSP event-IRQ fix (`a478f0b4`) now sets the RISC-V SWGEN0 +> IRQSTAT/IRQMASK/IRQDEST and latches the CPU-tree leaf so the guest services +> the event. The working recipe is a 12 GB file-backed guest +> (`-object memory-backend-file ... /dev/shm/vgpu-guest-ram`), the +> `gsp-phys-guest-ram` debugfs translation, the RM-call ladder stages 5/3/4, +> the full r1624 module-argument set, and `sema_vas_cand_ms=1` (which made X +> green on attempt 1 in r1685). The beef GR channel `0xbeef0100` now captures +> real GL draw sections (`len=4808`, `win=1201 words`, `prog=0x4390900`). +> +> The new frontier is the **constant buffer**: the draw executes but the +> engine writes 0 pages, and `bounce copy-out` reports 0 changed. The GL +> draw's constant-buffer source is forced to the seed page because +> `sema-gl: cb-copy src=0x0cd70000 ... first8=00000000 ... src=synthetic-id` +> reads all zeros; a guest-RAM scan found no MVP matrix, and the +> `0x3f800000` hits in vidmem are shader microcode immediates, not matrices. +> With a zero MVP every vertex collapses to the origin, so there are no +> fragments regardless of viewport/render-target. The next step is to decode +> the CB-bind method on the beef channel, resolve that VA through the channel +> VAS, and copy the real page into the seed page. Secondary walls: the MME +> front-end busy/deferred class, the GR stale-wedge recovery, the X DDX +> push-buffer get-word/notifier recycle (X can still die after a few forward +> frames into the RM-degraded state), and the DDX's 4 GB sysmem allocation +> that forces the 12 GB guest. +> +> Honest state: still no rendered pixel through the emulated vGPU and no +> video. The execution chain (submit -> forward -> physical GR -> stores) is +> now live for the guest's own GL work; the first pixel is gated on the MVP +> constant-buffer content and the present/copy-out. Milestone position: +> between M-GL1 and M-GL2. Estimate (subjective): a first non-black frame +> hours-to-days if the constant-buffer decode behaves like its predecessors; +> the two-VM GPU-accelerated video after that. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): r1685 VM up (QEMU 72960, mdev +> `0dcda61f-a8ef-4111-8993-35942d2d5392`), X green, glxgears 300x300 +> rendering black, `work_gr backend_ok=5`; `vgsp` loaded (HEAD `cb8e822b` +> plus the runtime parameters above), `guest_ram active=1`, ladder stage 4; +> pstore empty; no reboot during the check. +> +> **Snapshot: 2026-11-09 ~19:15 (SESSION 156 continued; HEAD `21e0d611`, +> tree clean; supersedes the 19:05 entry below).** +> The forced-GLX path moved the renderer wall: with +> `SDL_VIDEO_X11_FORCE_EGL=0` + OpenGL, GZDoom and glxgears now use +> `libnvidia-glcore` (the EGL/Mesa fallback is gone). The minimal windowed +> reproducer (`glxgears`) then hangs in the glcore kick wait: the client polls +> the per-slot `/dev/nvidia-modeset` fd (`poll(POLLIN, 1 s)`); the host's +> sweep completes the beef notifier memory (`kick-slot completed +> h=0xbeef0101 ...`, `client-slot cleared h=0xbeef0100`), but the guest kernel +> never signals the event the poll waits on. The remaining edge is +> kernel-visible completion: complete the exact notifier page the guest +> kernel watches and signal the guest event (the kick-notify executor), not +> just write the memory. GZDoom's window creation fails/hangs at the same step +> and can still allocate the DDX's 4 GB sysmem block, which occasionally OOMs +> the 12 GB guest. +> +> Current walls, ranked: (1) the kernel-visible GL present/kick completion +> above; (2) the X DDX push-buffer get-word/notifier recycle (a diagnostic +> get-word write released the DDX and let GZDoom reach `MAP01` on llvmpipe +> earlier, but the kernel lever is not implemented); (3) the DDX's 4 GB +> sysmem allocation (12 GB guest workaround; suspected emulation gap); (4) +> the direct-GL draws still report `forwarded=0` - decoded and host-completed, +> not dispatched to the physical GR engine. No rendered frame through the +> emulated vGPU yet; the earlier `MAP01` was a llvmpipe datapoint, not the +> target. Milestone position: between M-GL1 and M-GL2. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): host up 20 h (fresh boot 2026-11-08 22:29); +> `vgsp` loaded (refcount 0) + mdev +> `d22c826b-871d-4540-9fc6-63cf3e87f823` + QEMU r1679 live (12 GB guest, +> glxgears spinning); `nvidia-smi` reports no supported GPU (the vgsp claim +> is held); pstore empty; no reboot during the check. +> +> **Snapshot: 2026-11-09 ~19:05 (SESSION 156 continued; HEAD `e342bd0f`, +> tree clean, `vgpu/master` in sync; supersedes the 12:45 entry below).** +> Since the 12:45 entry the campaign closed the X-side execution machinery +> and moved the pixel gate to concrete blockers. The X server's deaths were +> separated from driver bugs: the DDX waits are fixed by `408d67d3` +> (class-aware display arm - the 0xc57e window class has no NOTIFY bit, so a +> bound notifier is the arm; bfef `0x8000 -> 0` NvNotification completion; +> the decoded GL release is applied via `sema_gl_hold_apply`, default off; +> the 0x8000 sweep is extended to the X DDX's own RM channels), on top of the +> PMA-scrub deadlock fixes (`3701f985`, `ad93b227`) and the channel-VAS push +> read (`c3887218`). The residual death is an OOM: the X DDX allocates 4 GB +> of system pages at startup (kprobe and gdbstub: `nv_alloc_pages` count +> 1048576 from Xorg), which an 8 GB guest cannot absorb once GZDoom adds its +> own footprint; the current run uses a 12 GB guest. +> +> With the DDX unblocked, GZDoom reached `MAP01` for the first time, but on +> **Mesa llvmpipe**: the NVIDIA Vulkan ICD fails `vkCreateDevice` and the +> default OpenGL/EGL path falls back to Mesa (`libEGL ... failed to create +> dri2 screen`). The emulated-vGPU render path is therefore not the one that +> rendered; the campaign is forcing GLX (`SDL_VIDEO_X11_FORCE_EGL=0`) on the +> 12 GB run and has fully decoded the DDX push-buffer wait: the 2D reserve +> exits only when the GPU-written engine get word advances (`get <= put`) and +> the kick requires the notifier to be recycled to `0xffff` after completion. +> A diagnostic host write of the get word released the loop and let GZDoom +> reach `MAP01`; the kernel levers (publish/advance the get word; recycle the +> notifier) are the next implementation. +> +> Honest state: still no rendered frame through the emulated vGPU and no +> video; GZDoom's `MAP01` is a llvmpipe datapoint, not the target. Milestone +> position: between M-GL1 and M-GL2. Current walls, ranked: (1) renderer +> selection - the NVIDIA Vulkan ICD must create a device or the forced-GLX +> path must reach glcore; (2) the DDX push-buffer get-word/notifier +> completion; (3) the 4 GB X DDX sysmem allocation (suspected emulation gap) +> that forces the 12 GB guest; (4) the direct-GL draws still report +> `forwarded=0` - decoded and host-completed, not dispatched to the physical +> GR engine. Estimate (subjective): a first NVIDIA-path frame hours-to-days +> if these behave like their predecessors; the two-VM GPU-accelerated video +> after that. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. +> +> Live at check (read-only): host up 20 h (fresh boot 2026-11-08 22:29); +> `vgsp` loaded (refcount 0) + mdev +> `d3a9e554-f948-45df-b81b-b5e987b0bee8` + QEMU r1678 live (12 GB guest); +> `nvidia-smi` reports no supported GPU (the vgsp claim is held); pstore +> empty; no reboot during the check. +> +> **Snapshot: 2026-11-09 ~12:45 (SESSION 156; HEAD `4acc81e8`, tree clean, +> `vgpu/master` in sync; supersedes the 2026-11-06 ~19:00 entry below).** +> The graphics front moved from the DS SPH validation gate to a working +> display/GL stack with one named X-side completion wall. Since 11-06 the +> 38-commit arc ending at `4acc81e8` landed: the host-side EVO display engine +> (`035d033c`: core/window/imm channels 0xc57d/0xc57e/0xc57b, NV-DMA push +> parser, SET_NOTIFIER/UPDATE-FINISHED, raster line) makes the guest display +> chain real only with `disp_caps_seed=1` (the caps window reads all-zero +> otherwise -> "Unsupported base display class"); the display instance table +> is resolved in the guest BAR2 aperture as Linux enumerates it (BAR3) and +> notifier writes are routed by target (FB shadow vs guest PA); +> `SET_NOTIFIER_DMA` is `valid=1` and `UPDATE -> FINISHED` is 12/12; lost +> completion events are force-retried. With that, nvidia-modeset/NvKms +> register (`/dev/dri/card0` + `renderD128` + `fb0`, `nvidia-smi -L` green), +> X settles, `glxinfo` reports OpenGL 4.6.0 NVIDIA 570.144 with direct +> rendering, and GZDoom obtains a real GL context. The approved reboot +> (2026-11-08 22:29) cleared the stale host GSP-RM state that had wedged +> every boot in `NV_ESC_RM_VID_HEAP_CONTROL`; post-reboot runs reproduce +> none of it. +> +> The windowed path then exposed and closed three completion walls: +> (a) the UVM channel-pool tracker - a channel rank N owns pool slot N (the +> earlier rank+1 model pre-satisfied the next rank and lost the write), the +> release is fanned through every published VAS translation (the pool page +> moves between RM-memdesc and VAS-walk pages; `80cbbd04`, `a3060a54`) and +> the init slot escalates 1..4 (`f10714c9`), so six UVM channels now pass +> their init waits; (b) the direct-GL GR release - the deferral is scoped to +> the push's own candidate (a global cross-pass arm let stale candidates +> defer every later release; `1ad9216a`) and `sema-sw` writes now land for +> `beef0100`; (c) the direct-GL CE channels `beef0101/0102` - a VRAM-shape +> USERD scan for the beef CE family (`f2476897`) finds their rings and their +> submissions retire; the X DDX kick channel `bfef0002` decodes its ring head +> without a USERD and authors the class-2d notification (`4acc81e8`). +> Fullscreen GZDoom's mode set still wedges X; the campaign path is a +> settled X + windowed GZDoom. +> +> Honest state: GATE1 still not met - no rendered GZDoom frame and no +> emulated-vGPU video. Current wall (r1664): Xorg spins ~30-41% CPU in +> `_nv018351rm` while GZDoom blocks on the X GLX reply +> (`_xcb_wait_for_reply`) then returns to its glcore spin. The remaining +> objects are the X RM ioctl behind `_nv018351rm`, the X GLX +> context-creation completion (direct-GL channel completion vs a display +> flip notifier), and the GZDoom kick-slot word (not the RM notifier page). +> Milestone position: between M-GL1 and M-GL2 (the client and display stacks +> execute; pixels are the gate). Estimate (subjective): a first windowed +> GZDoom frame hours-to-days if the wall behaves like its predecessors; the +> two-VM GPU-accelerated video follows after a real frame, with 60 fps/no +> forcing weeks after that. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the SESSION +> 143 Vulkan fence regression is not re-measured. Watch next: the +> `_nv018351rm` ioctl identification, the X GLX completion object, and the +> client kick-slot word - then a rendered windowed GZDoom before any +> capture. +> +> Live at check (read-only): host up 14 h (fresh boot 2026-11-08 22:29); +> `vgsp` loaded (refcount 0) + mdev +> `dc0cfb79-fd4e-490a-8f08-e5228caeaed0` + QEMU r1664 live; `nvidia-smi` +> reports no supported GPU (the vgsp claim is held); pstore empty; no +> reboot during the check. +> +> **Snapshot: 2026-11-06 ~19:00 (SESSION 155 continued; r1562 committed +> post-reboot; HEAD `2f5b838c`, 1 unpushed; supersedes the ~12:55 entry +> below).** The DS SPH validation wall narrowed to a single bit this +> afternoon: r1562 (per-stage synth SPH disable mask, `2f5b838c`) reached +> `ds 80000000->81040000` = **SPH18 only** (GS check2, the geometry-stage +> attribute map) with the client's own stage VAs +> (0x01990900/0x01990300/0x01990200/0x01990000 across j0/j1/j2/j5). Two +> fixes got there: the pair-safe synth bind (`fe03bfe8`) emits the class's +> own END ahead of a carried-open BEGIN/END pair and orders TYPE/ +> REGISTER_COUNT before the SPH address pair - the r1558 EC 0x0d +> ILLEGAL_METHOD_IN_BEGIN_END class is gone (r1560: 13 beef launches, +> 3 DS-FAST, 0 Class Error); and the client's own stage-disable records +> (`0x2000<-0`, `0x2080<-0x20`, `0x20c0<-0x30`, `0x2100<-0x40`) cleared +> the VSB 5/7 tess-mode pair (r1561), proving the DS validates every +> stage - even a disabled one. The DS bitmap is decoded: `SPH<n>`, +> n = stage*4 + check over VSA/VSB/TCS/TES/GS/PS, with four checks +> (type/version, mpipe/stage binding, attribute map, stream-out). +> +> The r1562 run then poisoned (`badf1201` priv-error, TSG preempt + +> resched) and entered the known GR timeout/wedge cascade (MME-REGS / +> FETCH-WIT / MME-FAST spin, 18:47:21); the VM was closed and the host was +> rebooted cleanly (logind `systemd-reboot` 18:47:20-24; pstore empty, no +> crash). The reboot is not recorded in the acting handoff (last write +> 17:59) - reason inferable from dmesg; no post-reboot plan written +> there. Post-reboot: fresh boot 18:47:56; apt nvidia 570.211.01 green +> (RTX 2080 Super Max-Q); `vgsp` was down at the check moment; the acting +> session resumed the stack at 18:56 (vfio prereqs) and r1563 is in +> preparation (18:57). +> +> Afternoon wave since the 12:55 entry: `5fad0adb` (slot-selective +> `sema_sph_bind_mask`), `c822fa34` (engine-visible MME selector + RT0 +> third-dimension seeds), `3bb62602` (bounded setup-push hold; hold>0 +> stalls the ring behind the unreadable 60216 B setup entry - 0 is the +> reproducible base), `fe03bfe8` (pair-safe + per-stage bind disable), +> `91a4b48e` (docs: real render walls), `2f5b838c` (r1562). Earlier in +> the wave (r1541-r1558): the PS-stage-only bind, the `0x2390` +> CB-selector poison fix, the EC 0xd2 colour-target third-dimension +> decode and the completed RT0 bind - the draw validates and stage5 +> populates. +> +> Honest state: GATE1 still not met - r1561's two captures 60 s apart are +> identical and nonblack=0 (black 800x600 window and black root); the +> best copy-out is 1-2 engine-changed pages that do not land on the +> visible surface. Milestone position: between M-GL1 and M-GL2. Estimate +> (subjective): GS 18 is a single named wall (GS-specific attribute/ +> stream state - glxgears emits no GEOMETRY SPH), first non-black frames +> hours-to-days if it behaves like its predecessors; 60 fps/no-forcing +> and a video on the emulated vGPU weeks after that; full CUDA breadth, +> multi-VM and Windows months. +> +> CUDA front unchanged (backgrounded while graphics holds the device; its +> 10-30 status stands): `41343c63` green (`victory_mail` `VADD bad=0` + +> `STATUS OK`; `vecadd_loop` 100 launches PASS); the fresh-boot battery's +> red class is D2H-stage with release-discipline fixes staged; the +> SESSION 143 Vulkan fence regression is not re-measured. Watch next: +> r1563's DS bitmap (does GS 18 clear?), then a non-poison DS transition +> with >2 changed pages before any pixel capture. +> +> **Snapshot: 2026-11-06 ~12:55 (SESSION 155 continued; r1538 live; +> HEAD `81bde9b7`; supersedes the ~01:00 entry below).** The graphics +> campaign moved from "the DRAW macro executes" to "**the first calls-kept +> draw windows execute with engine-authored stores, and the DS SPH +> validation is the last gate before pixels**". The fixed-CB upload is +> decoded end-to-end (MME-wrapper form, the assembler image delivered to +> the engine-visible seed page, real-CB receipts gating it). The +> class-error wall that had kept the first calls-kept window from +> executing was a BEGIN/END pair carried across the window boundary; +> `81bde9b7` (`sema_gl_pair_close_entry`) emits the driver's own END form +> (`0x1614` + event word) before the first window run when the tracker +> carries an open pair. r1529 (P0 config + close): 44 beef launches, +> **0 Class Error**, `mac=0x22` x16, and 1-2 engine-changed copy-out +> pages per launch - the slot-34 DRAW_IAB macro (IRAM entry 0x265, 75 +> words) executes (r1526, before the fix: 11 launches, first calls-kept +> window poisoned at SET_FRONT_FACE `0x191c`, error 0x0d +> ILLEGAL_METHOD_IN_BEGIN_END). +> +> The close is not default yet: r1532 (close + SPH window 0x10004) +> misfired after 14 failed launches - the injected END itself faulted +> (`0x1614`, error 0x0d) because the tracker believed a pair was open +> that the front end had already lost - so it stays default-off until the +> pair model is committed only on retire. The SPH version window is not +> the DS gate: r1535 (P0 + close + `sema_sph_version_val=0x10004`) ran 49 +> launches with 0 Class Error and `mac=0x22` x18, but the DS still +> escalated `80000000->84000420->86040624->86040666` and only 1-2 pages +> changed (seed bi=768, clear bi=772/773). The DS signature +> (`stage=00000000,01990900,01990200,00000000,00000000,01990000`, +> `region=0/0`) is SPH2|SPH5|SPH9|SPH10|SPH18 with M_PIPE_STAGE=VSB|TI; +> the three stage VAs are legal V4 headers, so the failing entries are +> channel-level (cache/other binds), not those headers. Ranked next: +> (1) skip binds whose page does not start with SPH version 4 +> (`sema_sph_bind_filter`, receipt `SPH-BIND-FILTER`); (2) emit the +> program region as one 2-word INC run and fix the MME shadow read index +> to `(mthd >> 2) & 0xfff` before trusting the `scratch[0x78]` evidence; +> (3) make the pair model transactional (commit pair state only on +> retire, then re-enable pair-close by default); (4) sweep every bound +> SPH VA's first word and the per-stage bind list. +> +> Live at check (read-only): host up 5 h 13 m (fresh boot this morning); +> `vgsp` loaded (refcount 0) + mdev +> `c71c1617-b502-4ff0-9659-64df695aaa27` + QEMU 484365; r1538 (SPH +> bind/region experiment) is live and left running; its early receipts +> show `SPH-BIND emitted` and `SPH-REGION emitted` (bases +> 0x1500000/0x15c0000/0x1990000/0x1ab0000), mixed 1-2/0 changed-page +> copy-outs, and a 0-byte screen-capture fetch (no non-black frame yet); +> `pstore` empty; no reboot during the check (in-session cycles only); +> HEAD `81bde9b7` == `vgpu/master`, tree clean; `nvidia-smi` reports no +> GPU (the vgsp claim is held). +> +> CUDA front (backgrounded while the graphics campaign holds the device; +> its 10-30 status stands): the physical-engine path is green +> (`41343c63`; `victory_mail` `VADD bad=0` + `STATUS OK`; `vecadd_loop` +> 100 launches PASS); the fresh-boot battery's red class is D2H-stage +> (release-before-work, ledger-vs-writer divergence, memset +> readback/writeback race - all proven; release-discipline fixes staged, +> not yet validated); the SESSION 143 Vulkan fence/completion regression +> is still not re-measured, and the guest Vulkan pipeline renders +> end-to-end but black (the inline-to-memory upload arm is its named +> gap). Two guests with two mdevs coexist at boot level (guest-B CUDA +> awaits per-guest ownership). The Vulkan black-frame path and the GL +> pixel gate are separate; the GL draw is the one currently executing. +> +> Honest state: GATE1 is not met - the X screen is black/static and the +> best capture remains the earlier 2.08% non-black static twin frames +> (not pixels); the app's ~450 FPS is fence-level completion. The +> milestone position is between M-GL1 and M-GL2 (the gate = a swap-hook +> capture with real motion). Estimate (subjective): first non-black +> frames hours-to-days if the DS gate behaves like its predecessors; +> 60 fps / no-forcing and a video on the emulated vGPU weeks after that; +> full CUDA breadth, multi-VM and Windows months. Watch next: r1538's DS +> result, the SPH bind-filter / region-form fixes, and a first non-black +> xwd on a fresh cycle. +> +> **Snapshot: 2026-11-04 ~12:45 (graphics-execution campaign, SESSION +> 153/154 continued, r1442-r1451; HEAD `bbda751e`, tree clean, +> `vgpu/master` in sync).** Since the 10-30 snapshot the active front is +> **graphics execution**: the emulated vGPU went from "the method stream +> retires but writes nothing" to "the direct GL client's DRAW macro +> executes to completion and the engine writes pages - rasterised pixels +> are the remaining wall". Engine-verified milestones since 10-30: the +> first beef launches retiring (`fdfb4673`) and the byte-exact +> present-blit proof (`eef52abe`); the first full window retire +> (`e3ecd6a3`) and the DRAW-class window retire with stubbed macros +> (`8528df77`); full MME macro-program capture with the offline proof that +> the packed image equals the live program (`66d95894`, `ffaa39f3`); the +> first engine store retirements (`output retirement 2/901 pages changed`, +> CONT-104) with `0x2390` root-caused to LOAD_CONSTANT_BUFFER and the +> constant-buffer-selector slot fix; the MME IRAM readback gate whose live +> debug-port reads match the packed image at macro entry, and slot-57 +> macro execution to its last instruction with stores growing 9 -> 238 -> +> 305 of ~903 (CONT-107/CONT-109, `8df33f1f`, `0eb64118`); the first +> engine-changed render-target pages in the completion copy-out (r1385, +> CONT-111); SESSION 154's client-formation protocol (one X-client +> teardown + re-create makes formation reliable), the golden macro overlay +> (`94c432d9`) and the first slot-34 (DRAW_IAB) execution via the +> pipeline-stage bind lever (`b0171765`); and the 11-03/11-04 +> release-sequencing wave (settle hold, strict release, front-end-busy +> deferral, window hold) plus the pipeline-binding emission fix +> (`79545530`) together with carrying the program-address pair - the two +> walls that blocked the called-macro path (`0x2090` binding rejection, +> the VA-0 constant fetch) are down: on the best boot both called macros +> reach their ends (slot 57 `mpc=930`, slot 2 `mpc=22`), 9 launches clean +> (7 consecutive), store retirements and engine-changed copy-out pages +> observed. Current graphics walls (ranked): (a) residual per-launch +> poison classes under the call/window-content lottery (`0x2390` LCB, +> `0x1614`, the `0x100 deaddead` cascade), with levers built or in flight - +> the latest (`bbda751e`) drops the attr-prelude SET_PIPELINE run the front +> end refuses cold; (b) the newly named **raster blocker** - draws retire +> without storing to the X framebuffer, so render-target/surface binding +> is the next gate; (c) the present/capture path once (b) lands. Honest +> state: GATE1 is not met - the X screen is static/black and the guest +> glxgears window reads 2.08% non-black static twin frames (not pixels); +> the app loop's ~450 FPS is fence-level completion, not rendering; the +> milestone position is between M-GL1 and M-GL2 (the gate = a swap-hook +> capture with real motion). **CUDA front (backgrounded while the +> graphics campaign holds the device; its 10-30 status stands):** the +> physical-engine path is green (`41343c63` cross-class ring-claim fix; +> `victory_mail` `VADD bad=0` + `STATUS OK`; `vecadd_loop` 100 launches +> PASS); the fresh-boot battery's red class is D2H-stage +> (release-before-work, a ledger-vs-writer divergence, a memset +> readback/writeback race - all proven; release-discipline/opdef-deadline +> fixes staged, not yet validated); the SESSION 143 Vulkan fence/ +> completion regression is still not re-measured, and the guest Vulkan +> pipeline renders end-to-end but black (the inline-to-memory upload arm +> is its named gap). Two guests with two mdevs coexist at boot level +> (guest-B CUDA awaits per-guest ownership). Live at check (read-only): +> host up 1 d 20 h; `vgsp` loaded (refcount 0) + mdev +> `6daed337-d05d-4185-a6fa-5a978d7d1cc7` + QEMU `e726ba0b`; campaign r1451 +> in flight (the run at the check instant showed the known +> WEDGE-REARM/`-110` failure class); pstore empty; `nvidia-smi` reports no +> GPU (the claim is held). Commits this arc (about 90, +> `fdfb4673`..`bbda751e`): the retire ladder, `2bc41213`..`ffaa39f3` +> (first stores), `8df33f1f`/`0eb64118`/`e2560a54` (macro completion, +> engine-changed pages), `94c432d9`, `b0171765`, and the SESSION 154 +> ladder `63541b75`..`bbda751e`. +> +> **Snapshot: 2026-10-30 ~12:20 (SESSION 153, r1013 live; HEAD `a64169db`).** +> The arc since the SESSION 143 snapshot split in two. Guest CUDA on the +> physical engine: the post-reboot stall is root-caused and fixed - the GR +> ring-autopin fallback and the CE first-submission capture each guarded +> block ownership only within their own engine class, so a channel could +> latch a live foreign FB block (`41343c63`: cross-class block claims, +> pre-latch CE reservations, own-block latch semantics) - and the fixed +> build is green repeatedly (`victory_mail` `VADD bad=0` + `STATUS OK`; +> `vecadd_loop` 100 launches PASS). The fresh-boot battery's remaining red +> class is D2H-stage (release-before-work, a ledger-vs-writer page +> divergence and a memset readback/writeback race are all proven; +> release-discipline fixes are staged but uncommitted; the opdef-deadline +> slot-complete path is the next fix). X/GL on the emulated vGPU: the wall +> that stood since the early sessions is closed - `fffae7b1` gets Xorg `:0` +> through the nvidia-push DMA sanity push and ScreenInit (`glxinfo` = +> OpenGL 4.6.0 NVIDIA 570.144, direct rendering, GZDoom launches), with the +> RC watchdog channel registered/serviced (`8898bd00`+`42c9626b`) and +> per-client VAS resolution (`38ba1e2d`: a channel's VAs resolve through +> its own VAS first). The app-side 10 s/frame waits are the client's own +> give-up manager; the `sema_xw_mirror` pair completes presents (glxgears +> latches 128-146 FPS, 200+ in the later shim runs) but produces no +> rendered output - the GL draw stream is marshalled, not executed. The +> engine now executes the real GL method stream in two forms: the C597 +> section capture + synthesized compute launch retires its QMD on real +> sm_75 SASS (`0ee574a3`..`595e023a`), and the verbatim 3D method-stream +> replay lane retires real 3D methods with no QMD (r932); the hard limit +> is that rasterised GL pixels still need the real 3D pipeline. SESSION +> 153's headline: the direct GL client's device-init gate is root-caused to +> the driver-shared memfd slot word (bit 0 set = glcore skips its +> class-0xc46f channel creation); cleared, the client allocates its +> channels (0xbeef0100/0101/0102 plus 0x902d/0xc5b5/0xc597/0x9072/0xa140/ +> 0x90f1) and `glReadPixels` returns with GL error 0 - fixed field-side by +> the kernel keeper `sema_shared_hold_pa` (`a64169db`, holds bit0 clear +> deterministically) plus host hold/toggle automation (the MakeCurrent +> handshake needs the 2<->3 transition). Current walls (ranked): (a) the +> fresh-boot Xorg DMA-init failure in r1013 (a 10 s `Failed to initialize +> DMA` -> no X server, so no client; a different wait from the member-spin +> class); (b) the client PUT-visibility wall - the direct client's channels +> never submit (its process writes no GP_PUT anywhere, the BAR0/BAR1-shadow +> premise was falsified by direct measurement, and the submitting DDX-side +> channel's USERD is guest RAM already tracked by the RAM scan); (c) the +> pixel wall - no render content reaches any ring (X-side pushes are 32-64 +> B housekeeping BEGINs; every capture/readback is black). Landmine: +> killing/exiting a direct client can trip a 4 GB `nv_alloc_system_pages` +> guest-kernel oops - never kill clients; re-cycle instead. Delivered on +> the emulated vGPU: the R18 CUDA-animation video (`70134652`; 60/60 +> frames zero faults, frames exact vs CPU math, single-page kernels only - +> the window-0 code shadow carries one 4 KB page, so kernels >4 KB stay +> hostile until it grows) and two two-VM GZDoom videos (`375d627a` tools; +> the GPU-accelerated one runs on virtio-gpu/virgl on the host iGPU, not +> the emulated vGPU, which remains the core frontier). Host clean, pstore +> empty; live: r1013 mdev `95446a6c-d5da-4e74-920b-5ac0d7ccc303`, QEMU +> 2335256, guest no Xorg/no GL client, pin/sem helpers up. Also open: the +> SESSION 143 Vulkan-client completion regression (fence timeout with the +> push re-completed) - not re-measured since r816, priority moved to the +> GL path. Commits this arc: `41343c63`, `fffae7b1`, `8898bd00`, +> `42c9626b`, `38ba1e2d`, `6f2e6193`, `0ee574a3`..`595e023a`, `70134652`, +> `375d627a`, `1fd8d50a`, `f500779b`..`a64169db` (SESSION 153 ladder). +> +> **Snapshot: 2026-10-26 (SESSION 143, post-reboot).** The r800 walk guard +> holds and the `gsp_phys_gr_min_data=0` bypass reaches the real engine: the +> guest Vulkan compute launch is now *submitted* (no oops, no degenerate-set +> refusal), the engine fetches the host-deposited QMD and the firmware reports +> a real `MMU Fault ... @ 0xdf_d8feb000, FAULT_PDE` at the guest program VA - +> the channel/context are alive, the program page is empty and unmapped. The +> util-sweep channel-visibility work (per-channel begin/skip/read=none lines) +> proves the Vulkan copy-class channels (0xbeef0101/0xbeef0102, engines 11/12) +> are registered and swept but never pushed, and that no channel carries +> program bytes or a `LAUNCH_DMA`: the code page (0x62b000 / rotating 0xc8b000) +> never fills. The frontier is therefore (1) the driver's staging-ring -> +> code-heap upload copy, which is not among the decoded pushes, and (2) the +> engine page-table mapping for the program VA. Separately, the refusal path no +> longer reaches the earlier three-black-frame baseline (`SYNTH-FAULT` vs the +> software completion/fence ordering), which must be restored before further +> observation. Docs: `docs/04-implementation/04-real-render-walls.md` sections +> 20-21. Commits: `34e611f4`, `c5510744`, `28def33e`, `0ca8ff9f`. +> +> **Snapshot: 2026-11-06 ~01:00 (the guest's REAL Vulkan pipeline now +> renders on the emulated vGPU: the first `VKFRAME seq=1..3` / +> `VKFRAME_DONE frames=3 640x480 saved=3` / `VKFRAME_IDLE ok` runs +> end-to-end — instance, device, allocation, image/view/buffer/shader/ +> descriptor/pipeline/command-buffer/fence, queue submit, fence release +> and frame readback all complete with a zero-touch service pipeline. +> The chain of fixes behind it: the push-stream parser now accepts the +> driver's real shapes (the `0x0001fff0` SET_SUBDEVICE_MASK bracket and +> zero NOP gaps — the class that silently skipped every large push), the +> fence fold (`sema_gr_ahead`) sweeps the pre-built GPFIFO continuation +> beyond GP_PUT and applies the submit's INC-5 RELEASE groups +> forward-only, the host-triggered VAS page-directory anchoring verifies +> the parked `COPY_SERVER_PDES` candidate against a content-located ring +> pair so the walk resolves every push VA before its first read, the +> COMPR+2MB VidHeap class is downgraded at the ioctl so +> `vkAllocateMemory` succeeds, the SEC2 scrubber's finish payload is +> authored by the service, the `0x9096` VidHeap enumeration and the +> `FIFO_GET_ENGINE_CONTEXT_PROPERTIES` ZCULL geometry are answered, and +> the C5C0-section -> vendor-launch translator found the real dispatch +> (a `SET_INLINE_QMD_ADDRESS_B` + 64-word run building QMD `0x4b85f00`, +> grid 40x30, plus the C5B5 copy-out). The frames are still BLACK: the +> driver delivers SASS/descriptors through its MME program + inline-to- +> memory stream (decoded in full; the payload rides the next GPFIFO +> entry as a source-view call), and the utility/copy-channel +> `LOAD_INLINE_DATA` arm that must materialize those writes is the +> current frontier (arbitration + `sema-utildec` witness landed). CUDA +> meanwhile: the best fresh-boot battery is 8 pass / 3 fail / 1 hang +> (the Vulkan work does not regress CUDA; always battery-first on a +> fresh boot); GL runs on the guest GPU (next wall = the FBO class); the +> Looking Glass frame path is proven end-to-end (guest render -> KVMFR +> ivshmem -> host PNG frame); two guests with 512M + 1GiB emulated VRAM +> coexist at boot level; the 0.9.9 release-preview demo video v1 is +> synced to files.vgpu.io (the updated video with GZDOOM working is the +> next publish). Frontier = non-black frames, then Xorg -> GZDOOM +> (GL/Vulkan) -> KVMFR/LG frames -> CUDA 15/15 + soak -> the perfect +> 1.0.0 video -> Windows guests).** > This is a dated public > status page, not live state; it is refreshed on every progress check. Live > working notes are maintained outside this repository; the public @@ -23,6 +678,20 @@ program against the physical TU104 backend (a `vectorAdd`-class kernel whose results verify), then the remaining open gates (multi-VM, 3D, Windows, isolation, performance, portability). +**Graphics objective (companion).** Land the first rasterised frame from an +unmodified guest graphics client on the emulated vGPU - the graphics ladder +**M-GL1** (client context-init executes) -> **M-GL2** (`glxgears` non-black +with motion) -> **M-GL3** (GZDoom GL through the mdev) -> **M-GL4** (60 fps, +no forcing) -> **M-GL5/M-GL6** (two VMs; video) - with gate definitions in +[`docs/04-implementation/05-graphics-execution-plan.md`](docs/04-implementation/05-graphics-execution-plan.md). +The current position is between M-GL1 and M-GL2: the display and GL stacks +now execute in the guest (NvKms registers, `/dev/dri`, X settles, `glxinfo` +4.6.0 direct, GZDoom obtains a real GL context) and windowed GZDoom reaches +renderer startup; the remaining gate is the X RM ioctl / GLX completion path +before a rendered frame (see the snapshot above). Both objectives are needed +for the 1.0 mission; the CUDA front is backgrounded while the graphics +campaign holds the device. + **Status.** - **Emulation front: GREEN.** The driver-API smoke completes end-to-end @@ -515,11 +1184,24 @@ isolation, performance, portability). can drive the host-owned device to an ECC Xid; shadowed BAR2 prevents it - committed and holding, no Xid since). Recurring and separate: guest-kernel boot panics (a NULL-in-interrupt/swap class) can kill a no-claim run at startup - - guest-side only, host stays healthy. A new host panic class was logged - 2026-10-05 11:18 with the (saved, reverted) pool-write-refusal experiment: a - sleeping VFIO mutex taken in atomic context in the sema GMMU walk plus a - vmalloc fatal exception in the bridge marshal path; reboot recovered the - host, no Xid. + guest-side only, host stays healthy. A new host panic class was logged + 2026-10-05 11:18 with the (saved, reverted) pool-write-refusal experiment: a + sleeping VFIO mutex taken in atomic context in the sema GMMU walk plus a + vmalloc fatal exception in the bridge marshal path; reboot recovered the + host, no Xid. +5. **Graphics execution (the display/GL campaign; current top front).** The + production display and GL stacks now run end-to-end in the guest - the host + EVO display engine serves the push-buffer channels, NvKms registers, X + settles, `glxinfo` is green and GZDoom obtains a real GL context - and the + direct-GL GR/CE channels submit. The remaining gates are: (a) the X RM + ioctl behind `_nv018351rm` and the GLX context-creation completion + (direct-GL channel completion vs a display flip notifier); (b) the GZDoom + kick-slot word (the client polls a kick-slot GPU VA, not the RM notifier + page); (c) the raster/present path once (a)-(b) complete. Fullscreen + GZDoom's mode set still wedges X; the campaign path is a settled X + + windowed GZDoom. GATE1 (a non-black frame with motion) is unmet. Full + context: the 2026-11-09 and 2026-11-06 snapshots above and + [`docs/04-implementation/05-graphics-execution-plan.md`](docs/04-implementation/05-graphics-execution-plan.md). ## Options to accelerate debugging toward end-to-end CUDA (candidate techniques; not directives) @@ -671,9 +1353,11 @@ meant to execute on the physical GPU. | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | -| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime initializes end-to-end with **real releases** and the launch now executes through the host bridge (see next row); a full `nvidia-smi` (per-process/utilization) remains unverified | -| CUDA kernel execution (`vectorAdd`) | ✅ **First green end-to-end guest CUDA (2026-10-04 ~16:20, no-claim host bridge):** `MAIL` + `VADD n=1048576 bad=0 checksum=392344090.500000` + `STATUS OK`, three consecutive deterministic supervised cycles, committed `cf6ff542`. The vadd runs through the host CUDA bridge daemon; forwarding guest engine work to the real GPU is the remaining backend path. | -| Graphics (3D), video | ❌ | +| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime initializes end-to-end with **real releases**; a full `nvidia-smi` (per-process/utilization) remains unverified | +| CUDA kernel execution (`vectorAdd`) | ✅ **Guest CUDA executes on the physical GR engine (host bridge disabled, since 2026-10-14):** `victory_mail` `VADD bad=0` + `STATUS OK` and `vecadd_loop` 100 launches PASS; matmul/transformer green after the 2026-10-20 boot-lottery fix (`c72f8948`). Remaining red class: the D2H-stage data classes (release-discipline fixes staged); launch-latency robustness; then battery/soak/multi-tenant breadth. | +| Graphics execution - OpenGL/X on the emulated vGPU | 🟡 the production display/GL stack now runs: nvidia-modeset/NvKms register (`/dev/dri/card0`, `renderD128`, `fb0`; `nvidia-smi -L`), X settles, `glxinfo` reports 4.6.0 (NVIDIA, direct rendering), GZDoom obtains a real GL context, and the direct-GL GR/CE channels submit (`beef0100/0101/0102`); the UVM channel-pool, GR release-deferral and CE USERD walls are closed. **No rendered frame yet:** the remaining gate is the X RM ioctl/GLX completion (`_nv018351rm`) plus the client kick-slot word; fullscreen GZDoom's mode set still wedges X, so the campaign uses a settled X + windowed GZDoom. Ladder position: between M-GL1 and M-GL2. | +| Vulkan on the emulated vGPU | 🟡 the guest pipeline renders end-to-end but black; the inline-to-memory upload arm is the named gap; the SESSION 143 fence/completion regression is still open. | +| Video | 🟡 R18 CUDA-animation video on the emulated vGPU (60/60 frames, exact vs CPU math); two two-VM GZDoom videos (the GPU-accelerated one is the labelled virtio-gpu/virgl path, not the emulated vGPU); the two-VM GPU-accelerated GZDoom video on the emulated vGPU is not yet produced. | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | Test status is tracked with the CUDA driver-API smoke test diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index ef4b258..242f470 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,20 +131,26 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the full CUDA - driver-API path through context creation, allocation and verified copies - (the driver-API smoke passes). -* **Needs improvement**: engine execution first — the backend is live and - CE-class copies and their releases are forwarded to the physical copy engine - (default off) with a fault relay, but for CUDA kernel execution the guest's - launch now reaches the front-end and the inline launch-descriptor install has - not been observed (context validity at dispatch is ruled out; see - [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) - — then the remaining reply fidelity (some consumed - fields are still generic fallbacks), the memory/page-table model, Windows - guest support, non-KVM hypervisors, and performance instrumentation. Each of - these is tracked with explicit "known-unknowns" in + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, the full CUDA + driver-API path, and **CUDA kernel execution on the physical GPU** — + verified guest clients (vector-add, matrix-multiply, transformer) execute on + the engine with engine-authored completions relayed to the guest. On the + graphics side, the X server and GLX run on the emulated vGPU, the engine + executes GL method streams (verbatim 3D replay and synthesized launches), + and the guest Vulkan pipeline runs end to end. +* **Needs improvement**: the staged-transfer class (some large multi-chunk + CUDA clients still mis-verify; failure modes proven), the every-boot + acceptance campaign, per-guest execution isolation for simultaneous guests, + the rendered-pixel frontier for OpenGL/Vulkan (program delivery plus + program-VA mapping), the remaining reply fidelity (some consumed fields are + still generic fallbacks), Windows guest support, non-KVM hypervisors, and + performance instrumentation. Each of these is tracked with explicit + "known-unknowns" in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); + execution detail: + [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) + and + [`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md); for the CUDA analysis area, start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index caaad01..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md new file mode 100644 index 0000000..cc16b04 --- /dev/null +++ b/docs/04-implementation/04-real-render-walls.md @@ -0,0 +1,1216 @@ +# Real-render walls: Vulkan and Xorg on the emulated vGPU + +Status: OPEN (2026-10-29, SESSION 64). This page records the measured state +of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg +server - on the emulated vGPU, with the exact evidence and the next +subsystem to implement. It complements +[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md) +(which covers the CUDA -> ivshmem -> host frame transport that is already +proven). + +## 1. What works today + +* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**: + `vkCreateInstance` succeeds and two physical devices are reported - + `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the + software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes: + `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`. +* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device + platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older + `egl_frame`). +* CUDA is unaffected (its clients run through the same emulated GSP; see + the battery evidence in the handoff). + +## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE + +`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An +`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the +ICD's first nvKms ioctl failing: + +``` +NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE +``` + +i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a +non-success status; the userspace frees everything and reports the +generic initialization error. Reading the driver's own modeset source +(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed: + +1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS` + (`0x730102`): a zero reply logs `No heads found on board!` and fails the + alloc. The emulated GSP RSP answered all NV0073 controls through the + generic zero-reply path. +2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED` + (`0x730107`) must report at least one display id, and the connector + path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA` + `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector. + +The host GSP-RM emulation now implements a minimal headless display: +one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`), +one static HDMI connector on display id 0 (SOR + single-link TMDS, which +avoids the DP-AUX library), plus DDC support. With this shim the alloc +gets past heads and `NoValidDisplays` and proceeds through the connector +queries and the display-object allocations (classes `0xc570`, `0xc57d`, +`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing +later in the display-core/modeset-channel path. That remaining subsystem +is the wall: the modeset core channel and the per-display dpy objects +must be emulated (a coupled object + channel + memory set whose exact +wire forms should be captured from a real-GPU session). + +A related experiment ruled out one shortcut: booting the guest with +`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does +not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and +fails, so the display path cannot be sidestepped by disabling KMS. + +## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test + +Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then: + +``` +(EE) NVIDIA(GPU-0): Failed to initialize DMA. +(EE) NVIDIA(0): Failed to allocate push buffer +``` + +`nvidia-push` logs `Failed to initialize DMA.` from +`__nvPushTestPushBuffer()`: right after allocating a push buffer the +driver writes a NOP and waits (10 s) for the channel to consume it. Host +evidence shows why it never does: + +* the X driver's channel registers (`class 0xc46f` GPFIFO channel, + `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with + **`submissions=0`** - its published GP_PUT never advances in the host's + view, because the host needs the channel's VA->PA translations to read + its USERD/GPFIFO and that resolution comes from the guest-side scanner + that currently follows *CUDA clients only* (`targeted resolve + va=0x20400f000: full RAM scan found nothing`); +* on a busy boot (Xorg started while a CUDA battery ran) the channel's + completion-counter slot is rejected: `counter auto h=0x1000013 rank=0 + pa=0x11c827ff0 rejected (slot owned by another channel)` - two + concurrent clients both derive rank 0's slot from the per-device + structural formula. + +Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution +path for the scanner or the host, and (2) counter-slot ownership keyed by +client identity (or a per-client slot domain), not just rank. + +## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged) + +`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init +entry returns a "busy" status once, sets a state bit, and the eglcore +caller does not retry. Ten in-process retries (`egl_retry`) all fail, so +the state machine fails on a real input - the return-value trace of its +callees remains the way in (see +[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)). + +## 5. Why this matters for the frame path + +Everything downstream of a working GL/Vulkan context is already in place: +a guest publisher writes frames into the kvmfr ivshmem BAR +(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout +as the proven CUDA `frame_producer`), the host reader decodes them +(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing +piece is a *working* graphics API on the guest GPU - the two subsystems +above. +## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected + +Three new instruments settled the remaining questions: + +* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`) + decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 = + `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**. +* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope** + (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`): + breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped + from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped + the driver's class list (42 classes; all `NVC57x` display classes are + present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no + HAL class list, so the "base channel" fallback path can never succeed). + Breaking at the capability parse showed the **capability notifier buffer + is all zeros** (nothing emulates the display engine that would fill it). +* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS` + and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the + parse, the driver flips from the dead base path to the **window path** + (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0` + unblocks the flow is confirmed. + +The window path then fails one step later: `nvRmApiAlloc(display, +class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED` +(0x56) locally in the guest kernel** (it never reaches the emulated +GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU +architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the +display/modeset channel classes are rejected (the reference source's +`_rpcAllocObjectPrologue` documents "Display classes are added for DCE, +vGPU Guest does not support those") - a vGPU guest's display is meant to +be emulated by the **host-side vGPU display service** (the vGPU RPC +display path), which the emulation does not implement. + +**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated +vGPU needs (a) the display capability fill on the host side (mechanics +proven by the gdb experiment) and (b) a vGPU display service: the +display-engine MMIO/RPC backend the guest expects from its host (the +equivalent of NVIDIA's vGPU display plugin), or the specific vGPU +configuration that makes the kernel accept the display channel class. +Until (b) exists the guest's NVIDIA display path is closed by design, and +GZDOOM-class workloads cannot render on the guest GPU. +## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path + +A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is +**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`), +`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any +`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes +addressable for live gdb breakpoints on the QEMU stub. With it: + +* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for + 0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register + mapping `hClass = r8d` was verified live). +* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters** + with class 0xc57e, calls `serverAllocResourceUnderLock`, and that + function returns **0x56** - *before* the GSP-RM alloc RPC + (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never + sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573 + return 0. +* So the last blocker for the window path is a **local RS-layer validation + inside `serverAllocResourceUnderLock`** (candidates: the inlined + privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup, + or `clientAllocResource`'s construct dispatch). The 0x56-producing + instruction is not yet pinned - a `stepi` loop over the QEMU stub is far + too slow and wedged the guest (recovered by a fresh cycle); the next + session should use conditional breakpoints on the function's call-return + sites (the approach in `gdbdiag19` works; the earlier run had truncated + addresses and needs the `0xffffffff00000000` base restored) or a + watchpoint on the status word. + +**gdb-stub field notes**: the stub is single-client; killing an attached +gdb leaves the VM stopped (recover with a detach-only connect); +`stepi` loops over the stub freeze the guest's userspace for minutes - +prefer breakpoints; every probe run with the caps-fill breakpoint changes +the driver's path, so keep the fill in each experiment. + +## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD) + +**The fence wall is closed.** The client's per-frame fence wait polls a +16 KB BO word that a submit-push INC-5 semaphore group targets +(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a +GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring +already carries the entries it pre-built for the next submits, idx12..46), +so the GP_PUT-bounded pump walk never decoded it and a host-side assist had +to write the word. The service now sweeps that pre-built continuation +itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the +ring's validated translation, accepts only pushes that pass the strict +method-stream gate, extracts the addr-0x5c RELEASE groups and applies them +forward-only through the standard pending ledger. Live evidence: the sweep +finds the groups and the ledger applies the fence word once per frame; +`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still +black - the shader work itself is unexecuted. + +**The compute-dispatch shape (why nothing executes).** The workload is a +single compute dispatch (640x480 storage image). Its push is a DIRECT +method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0` +on subchannel 1, then the context windows (`0x7b4` local-memory window B, +`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and +the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data +port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO +inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture +that feeds the physical-engine forward path never fires - and the CE-shaped +ops path is not even consulted for GR-class channels by construction. Real +pixels therefore require a translator that recognizes the C5C0 section, +builds the vendor launch the physical channel expects (the same prelude/QMD +shape the CUDA path already executes) from the section's fields and data +runs, and forwards it with the existing operand/relocation machinery. The +method vocabulary overlaps the physical prelude's existing knowledge +(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. + +## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale + +**The live ring is in guest RAM and the sweep's decode was right.** +`walk_probe 0x4020000` (the client's own page tables, vas[0] +`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA +to **guest RAM** (PA `0x11623d000` on r759; the page moves per run: +`0x104b55000` on r761) and its content is the 14-entry ring the sweep +decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 -> +the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot +`0x4250110`, ... - all pushes inside THIS client's own allocation window +(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own +VIDHEAP log). The decoded ring is live content, not a stale page. + +**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759 +the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`; +the userd-derived candidate loop then tried `0xb0000 - 0x10000 = +0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push +pattern) passed the plausibility gate and whose USERD probe hit the real +0xb0000 block, so the pin landed. The pinned entry's push reads all-zero +through every source; the pump retried 39 times and logged `skipping +stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`, +while `sema_probe 0x4020000` then answered the wrong page +(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep +included). Section 10's "stale session page" conclusion was this pin, not +the sweep. + +**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` + +`svc_ring_walk_live`: a candidate whose entry0 push provably reads +all-zero through the client's page tables is rejected when the VAS walk +resolves the ring VA to a page with a live entry (applied at the +userd-derived, FB-convention and existing-pin paths; the walk cannot vote +for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates +and accepts the channel's known USERD cross-aperture (the driver places +GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3) +the util sweep arbitrates a dead channel-path ring entry against the +direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and +their dedup keys are per-client now (reset in `svc_session_flush`); (5) +`sema-copyent` logs once per copy OP, not per row - one 480-row frame +copy had exhausted the 128/session budget and hidden every later copy. +Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0 +pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100 +va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the +fence fold releases fire, and the client completes (`VKFRAME_DONE +frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`). + +**The two utility-channel LID uploads are the driver's zero buffers by +design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256 +lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 -> +`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method +buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`; +`0x4250110`) read zero for the whole client run - the driver never +writes them. The GR channel's own LID entry DOES carry a real payload +(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param= +0x4b86000 size=0x1900`), which proves the arm works with real bytes on +this boot too. + +**Where the code page stands.** `prog=0xdfd8feb900` (the captured +launch): `sema_probe` answered vidmem `0x62b900` early in the run while +`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer +mapping ROTATES (a code pool), and the shadow's learned lat entry for the +VA can go stale. Both pages read zero in the shadow: the driver's code +upload has not been observed on any decoded path (no LID, no CE copy - +the only `sema-copyent` op of the whole run is the 480-row frame +copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0 +bytes=0x12c000` and correctly copies the still-empty image). The +remaining question is the SASS delivery shape for the C5C0 compute path +(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale +lat preference for the rotating code VA. + +**Next steps (ranked).** +1. Resolve the code VA through the VAS walk at launch time (the walk is + the live truth; the lat entry goes stale as the pool rotates) and + watch the code page the QMD actually names. +2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs + with a bounded methods witness while the client is live (the params + arrive by LID; the code must arrive by a sibling path). +3. Then the `n_data=0` descriptor/operand wiring (section 8). + +## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running + +**The delivery shape (decoded from the CE channel).** A CE-class channel +(eng=11) carries, as entry[4] of its ring, a push whose method stream is a +LAUNCH_DMA copy with the SASS bytes INLINE: + +``` +sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500 +sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8 +sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1 +sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0 +``` + +The source is the push itself (`in=0x120a00004`, right after the method +header) and the destination is a vidmem VA (`out=0x205000`, then +`0x206000`, ... 4096 B per run) - the engine-side "download program +range" path. The Vulkan client's CE channels carry the SAME shaped +entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`, +19 entries swept) - so the SASS delivery is a CE inline copy, not the +utility-channel LID entries (those are the driver's zero buffers, +section 12). + +**Why the copies never ran (three gates).** (1) The CE channel's ring and +USERD live in VIDMEM and resolve only by boot-lottery: the FB convention +matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> +fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the +frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan +instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768), +so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs +(0x120000000+) need the client's VAS registered: the parked +COPY_SERVER_PDES candidate only registered at +15 s (age-out, +`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`, +8000 ms from the cycle) expired first - r763 logged `skipping stale +entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and +`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy +pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s +(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms +parked`) but the ring gate still blocks. (3) The shadow ring scan +fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited +retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no +candidate on r768: the ring/USERD content is volatile at scan instants +(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a +`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan +misses the window. + +**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its +own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already +does this) instead of scanning for ring entries: for every USERD-shaped +block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0 ++ the probe. 2. Or latch the ring at the first CE submission (the ring +content appears exactly when the driver writes entry0). 3. With the ring +latched and the VAS at +5 s, verify the inline copies execute +(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code +page gets content; then the `n_data` descriptor wiring (section 8). + +## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring + +**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in +the notifier-completion path for CE channels *before* the clear write - the +one instant the ring content and the USERD are both live - reads entry0 +through the channel translation and the trusted walk, then scans the shadow +(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB +aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs +the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating +candidate is pinned/latched exactly like the regular CE autopin. `928c0730` +fixed the entry decode: the push VA is 40 bits, the scan had declared it +`u32`, so `scan_hits` was zero by construction. + +**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.** +The scan started at offset 0 and accepted the first ring-shaped page for +*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d` +(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's +ring/page, its first release decoded to the wrong semaphore word (+0 +instead of +4) and was dropped as stale (`cur=1`, already written by +rank 0), so the client's rank-1 tracker wait never completed. victory_mail +then spun at 100% CPU with `pub_calls=0` on three consecutive boots +(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era +module (`25523f84`) on a fresh boot passed the same client (VADD +`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the +SESSION 133/134 code. The fix tries the rank's convention offset +(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and +never accepts an offset another CE rank already latched this session; an +unlatched rank stays for the regular per-pass latch. Verified on r781: +ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/ +0x120000, USERD +0x10000 each), rank 1's own semaphore word applied +(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed +(`pub_calls=185132+`, no spin). + +**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0): +`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0 +va=0x120000000 len=40)` - a live entry - but the client's pushes decoded +`unread=1 injected=0`: the guest pin scanner covered CUDA clients +(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk` +engaged the tools (demand pins active) but r782/r783's captures reported +`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first +64 MB, and no classic autopin either. + +**Where the Vulkan rings actually are.** The client's own vidheap ioctl +log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations: +`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos +(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's +`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and +of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image +for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000` +sibling) finds nothing - the ring content reaches neither the shadow at its +allocation offset nor guest RAM. + +**The frame chain (r783, live).** The Vulkan client completes three frames +(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black. +The live GR channel (`h=0xbeec0100`) shows the dependency explicitly: +`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together +with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000 +dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk +maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0 +pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the +framebuffer. The shadow at `0x570000` reads zero: the data pages are the +ones the CE inline copies are supposed to fill, and the GR launch is held +on their absence (the designed `dstpend` dependency). So the whole Vulkan +chain reduces once more to the un-latched CE ring: no ring -> no inline +i2m copies -> no data/code pages -> held launches -> black frame. + +**Next steps (ranked).** 1. Cover the client's real allocation in the ring +capture: raise the capture scan bound above the recorded channel +allocations (e.g. `0x6000000`) and/or seed candidates from the client's +vidheap offsets, then re-check whether the entry arrives at the +submission instant (a live `sema_vidmem_dump` at `0x4100000` while the +client is streaming would settle it; the post-run dumps read zero). 2. If +the shadow still reads zero at the allocation offset during live +submissions, the client-side write watch is the remaining forensic: find +the CPU pointer the driver writes GPFIFO entries through, its PA via the +guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a +latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd` +with the code dsts), the code/data pages fill, then the `n_data` +descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA +battery on the fixed module (the r781 boot was a slow-crawl boot; the +spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel +forward timeout. + +## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target + +**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).** + +*The write path works; the driver does not use it for the ring.* A direct +BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through +`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated +vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest +and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct +into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A +high-frequency watcher (12 candidate offsets, ~2.5 Hz, including +`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000 +USERDs) recorded ZERO nonzero dumps across the whole live streaming window +of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`, +`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR +shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the +ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of +the full 4 GB guest-RAM image for both patterns (little-endian, corrected +from an earlier reversed search) finds nothing either. The walk and the +learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page +(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap +ioctl log (`PREV149-OUT`) names the channel allocations +`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is +written neither to the shadow at its allocation offsets nor anywhere in +RAM reachable through the guest's own page tables. + +*A stale-block trap explains every historical "latch".* On CUDA-first +boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan +channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012` +block surviving in VRAM after its session. The decoded entry +(`va=0x120000000 len=40`) is the old session's (both clients' first CE +pushes are shape-identical), the expectations seed from the stale pool +words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan +client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on +every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000" +rows are the same artefact: both were CUDA-first boots. All genuine +first-submission latches on record read `put=1 get=0` at the capture +instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`) +in the first-submission capture - a just-submitted entry cannot be +consumed at the notifier-clear instant. + +*The frontier is now precise.* The Vulkan client's CE submissions complete +only through the fabricated completion machinery (the driver never reads +its ring back), its CE ring content is invisible to every host source, and +the GR launch for each frame is held on the unexecuted CE i2m copies +(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem +`0x570000`, zero). Next: the client-side write watch - find the CPU +pointer the guest driver writes GPFIFO entries through, its PA via the +guest pagemap, and why neither the GPU page table target nor any shadow +holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on +the driver's ring-write helper). + +## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page + +**The write-watch toolchain (all three instrument classes).** (1) kprobes: +`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE +and SEC2 submission paths per the open driver sources) is marked +`notrace` - `trace_kprobe: Could not probe notrace function +channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach +(this is why the named helpers "never fired" in earlier sessions: they +could never register). (2) QEMU-gdbstub hardware breakpoints work: a +verification `hbreak *__x64_sys_ioctl` fired on guest ioctls +(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store +(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx = +pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and +the source) stayed silent across a complete Vulkan run - even though that +same run produced the CE channel's notifier clear +(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the +first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ... +scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring +VA resolves through the guest page tables to a guest-RAM page that is +mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA +`0x101021000` -> client VA `0x7fffee69b000`; a later run PA +`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's +own read agree the page is ZERO. An in-client gdb (the t122 recipe plus +a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000` +from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET` +logged before `GLBASE`) caught ZERO writes across the whole run +(3 frames, `VKFRAME_DONE`, IDLE). + +**Consequences.** On vk-first boots the client creates its CE channel but +never submits CE work: no GPFIFO entry is written by the kernel +(`channelFillGpFifo` never executes), nor by the client (hardware watch, +zero hits), nor is any content found in any BAR shadow or in RAM. The +historical "Vulkan CE ring latches" were the dead CUDA client's stale +blocks (section 15). The image/upload path for the Vulkan client rides +its GR channels (`launch captured ... inline=8`; the frame copy held as +`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and +the non-black gate must be attacked on that path - the data/code pages +those launches depend on are never filled because no CE i2m copies are +ever queued, and the launch is held on their absence (dstpend/n_data). +Next: instrument the GR launch's dependency resolution (which pages the +dstpend gate waits for, and whether the inline records the capture holds +are sufficient to fill them) instead of hunting a CE ring that nothing +writes. + +## 17. UPDATE 2026-10-26 (SESSION 138): THE GR LAUNCH'S WIRING IS PRINTED - one QMD-window page carries prog+pointer data (deduped and never scanned), its only fill candidate is an unresolvable inline word, and the backend refuses `n_data=0 (min 16)` + +**The witness (commit `10b09217`, sema_dbg-gated).** Per launch, bounded: +the candidate handle, `param_va`, `prog_va`, `n_pointer` + QMD pointer +pairs, inline count, and the resolved `gr_pages` list; then the fill's +built candidate set with the arg flag. + +**First capture (r788, Vulkan `h=0xbeec0100`, push=0x4f4509c).** +``` +sema-gr: cand h=0xbeec0100 param=0x4b86000 prog=0xdff0feb900 n_ptr=1 inline=8 n_pages=2 push=0x4f4509c +sema-gr: ptr[0] va=0xdff0febc00 word=34 +sema-gr: page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 +sema-gr: page[1] va=0xdff0feb000 pa=0x56b000 ap=2 qmd_word=48 +sema-gr: fill cand n=1 pages=2 push=0x4f4509c +sema-gr: win[0] va=0x43f00000 arg=1 +``` +Reading: the launch's page list is the param page and the +prog/QMD-window page (both resolving as `ap=2`, i.e. vidmem). +`ptr[0]` (QMD word 34, `0xdff0febc00`) shares its 4K page with the program +page, so `svc_gr_page_note` dedupes it into the `qmd_word=48` entry - and +the fill's scan passes cover only `qmd_word==32` and `32..47 even`, so +that page's content is never scanned for data pointers. The fill's single +candidate comes from the inline bytes instead (`0x43f00000`, arg=1) and +resolves in **no** vaspace (`svc-gr arg-miss base=0x43f00000 probe=none +... vas[0..3] rc=-2`), so `n_data` stays 0 and the backend refuses: +`vgsp-gr: GR submit refused: degenerate data set n_data=0 (min 16) +launch_va=0x4f4509c`. + +**Content state.** The shadow at the param page (`0x386000`), the arg +region (`0x386150`), the program area (`0x56b900`) and the pointer/data +area (`0x56bc00/0x56bd00`) reads all zero on this boot - the launch's +data and code bytes exist in no host-visible source (consistent with +section 16: nothing queues the CE i2m copies for the Vulkan client). + +**Next.** (1) Scan the deduped QMD-window page (or scan a page once under +every qmd_word that names it) so the pointer-region data words enter the +candidate set; (2) resolve the inline/dst candidates through the vidmem +serving lane (the pages ARE resolved to vidmem PAs - the missing piece is +the candidate list, not the resolver); (3) drive `n_data` toward the +launch's `min 16` (the window's expected page count) - and where the +window content must come from is the open question the byte-state above +poses. + +## 18. UPDATE 2026-10-26 (SESSION 139): THE FILL'S CANDIDATE SOURCE IS DISPROVEN - `0x43f00000` IS THE FLOAT 480.0f; after the float-shape filter the candidate set is EMPTY and the wall is content, not wiring + +**The false candidate (commit `add22329`).** A full dump of the launch's +param page (vidmem `0x386000`, r789) holds exactly two nonzero spots: +`+0x30: 08 00 00 00` and `+0x630: cd cc 4c 3e | 00 00 20 44 | 00 00 f0 +43 | 00 00 00 00` - i.e. the f32 scalars **0.2f, 640.0f, 480.0f** (the +640x480 viewport!). `0x43f00000` is 480.0f: the fill's only candidate +("arg base=0x43f00000 resolved 0 pages") has been a **float scalar +misread as a device VA** by the scan's shape filter (`>= 0x10000000`, +4-aligned). The GRCE inline write that does fire (`dst=0x4b86020 -> +pa=0x386020 bytes=1568 data8=0000000000000000`) writes the driver's +zero stub (the push carries only an 8-byte nonzero inline fragment at +`+0xb0`; the rest reads zero - the SESSION 130 zero-buffer finding). + +**The fix and verification.** All four scan sites (the three page scans +plus `svc_gr_scan_cand`) now reject float-shaped words (f32 exponent byte +in ~`[0x6c..0x8f]` = 2^-19..2^16; device VAs as f32 land in subnormal +exponents), and the bank-page scan covers word 48 (the deduped program +page). r790: **`sema-gr: fill cand n=0`** - no false candidates, no false +demand-loop traffic; the launch still refuses (`GR submit refused: +degenerate data set n_data=0 (min 16)`). + +**The wall, precisely.** The launch's real data pointers are not in any +scanned source: the param page holds scalars; the QMD pointer word +(`0xdff0febc00`) is one window buffer whose content is never written; the +inline payload is the zero stub. The content could only arrive via the +driver's i2m/LOAD_INLINE_DATA copies (which the Vulkan client never +queues - SESSION 137) or from the real GPU (physical forward, which the +n_data gate refuses first). This is now a pure content-source problem: +either find/queue the i2m that fills the QMD window, or break the +chicken-and-egg by forwarding a launch with the window zero (letting the +real engine read it) behind an explicit parameter. + +## 19. UPDATE 2026-10-26 (SESSION 140): THE n_data WIRING - the QMD's own pointer values ARE the launch's data set; the fill now reaches them (candidates appear, the page is correctly launch-internal), and the degenerate-set guard's bypass reaches the backend only to oops in the empty-set walk + +**Three wiring corrections (commit `5c0fcc21`).** (1) The QMD page itself +is noted into the request page list (word 34) - the pointer values live +inside the QMD (r790: word 34 = `0xdff0febc00`) and no scan ever covered +it. (2) The fill seeds candidates directly from the launch's own QMD +pointer pairs (`gr_qmd` words 34..47, decoded from the push's inline-QMD +run): r793 `sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)` - the values +need no page scan (the page shadows read empty; the +invisible-vidmem-write class). (3) The scans read pin-aware +(`svc_read_va` first) - the staging copy the decode reads is a pinned +page; a raw mapping of the vidmem pa sees zeros. + +**Result.** `fill cand n=1` with the QMD window page seeded; +`sema_probe 0xdff0feb000 -> pa=0x56b000 aperture=vidmem` resolves - but +the R3 hygiene filter classifies the page launch-internal, so `n_data` +stays 0: **this launch's QMD declares exactly one pointer pair and it +names its own QMD window - a launch with no external data, which the +"degenerate set" refusal class wrongly covers.** + +**The egg-break test + the oops.** With the r564 guard bypassed +(`gsp_phys_gr_min_data=0` - and the value clamp removed for the test -) +the refusal disappears and the forward ENTERS the backend: +`Oops: 0000 [#2] PREEMPT SMP NOPTI`, `Workqueue: gsp-phys-work +svc_fwd_work_fn [vgsp]`, `RIP: gsp_phys_backend_guest_gr_submit+0x4394` +(NULL-base table walk: CR2=0x1e93c, R14=0, RAX=0x1e000, RSI=0xdff0feb900 +- the program VA). The empty-data walk needs that guard before +`min_data=0` can be used; the clamp revert was withdrawn (crash +enabler). The copy pipeline is unaffected: `sema-gr: copy done +h=0xbeec0100 push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0` +runs cleanly in the same runs. + +**Next.** Guard the empty-set walk at `gsp_phys_backend_guest_gr_submit` +(the struct walk reading 0x04/0x10/0x38/0x3c off a NULL base with the +program VA in RSI), then re-test `min_data=0` for the no-external-data +launch class: the forward should reach the real engine, and the copy-out +(and the frame copy that already executes) can carry rendered pixels +back. + +## 20. UPDATE 2026-10-26 (SESSION 143): THE r800 GUARD HOLDS AND THE `min_data=0` BYPASS REACHES THE REAL ENGINE - IT FETCHES THE PROGRAM AND MMU-FAULTS WITH `FAULT_PDE` AT THE PROGRAM VA; THE WALL IS NOW THE PROGRAM PAGE'S MAPPING AND CONTENT, NOT THE DATA-SET GATE + +**The test (r801b).** `gsp_phys_gr_min_data=0` is now an explicit bypass +(commit `34e611f4`; 0 previously clamped to 1). On a clean cycle +(module from `34e611f4`, full params, `sema_gr_xlate=1`, +`sema_gr_util_scan=0`, `sema_vas_cand_ms=5000`, `sema_ce_grace_ms=20000`, +`sema_ce_hold_ms=0`, `sema_ops_log=8`, `sema_dbg=Y`, `min_data=0`) the +Vulkan client ran once: + +* **No oops.** `grep -c Oops` = 0 for the whole run; the r794 + `gsp_phys_backend_guest_gr_submit+0x4394` crash class is gone. +* **The refusal is gone.** Zero `GR submit refused: degenerate data set` + lines; the launch is submitted: + `GR LAUNCH submitted h=0xbeec0100 ... pages=3 shadow=1 + qmd_va=0x4b85f00 slot=3 words=239 ... qmdwin=0x108000`. +* **The engine ran and fetched.** After ~2.5 s: + `GR LAUNCH FAILED: timeout after 2500 ms (qmd_rel=0x00000000 + want=0x600df00d ...)`, and the real firmware reported: + `vgsp-fault: func=0x1006 ... errString="Ch 00000001, intr 00000000. + MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_GCC faulted @ 0xdf_d8feb000. + Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ"`. +* **The copy-out still works:** `sema-gr: copy done h=0xbeec0100 + push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0`. +* **The client now fails earlier than the black-frame runs:** it reports + `VKDEVICE ok family=0` then `VKFAIL vkWaitForFences ... -> 2` and no + frame is saved (the service synthesizes a client-visible fault for the + failed forward, where the refusal path used to software-complete and + let three black frames through). + +**The launch wiring measured on this boot.** +``` +page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 (param) +page[1] va=0xdfd8feb000 pa=0x62b000 ap=2 qmd_word=48 (program) +page[2] va=0x4b85000 pa=0x385000 ap=2 qmd_word=34 (QMD window bank) +``` +The program page resolves in the emulated vidmem at `0x62b000`, whose +backing host page (`0x5e0504000`) reads ZERO; `CODEHASH ... h0=0 ... +SUSPECT (first 16B)` and `FAULT-CAP ... prog_w48=0xd8feb900 +prog_pa=0x5e0504000 code_src=guest-hpa off=0x900 code64=000...` confirm +the source is empty. The QMD deposit keeps the guest program VA +(`w48/49=d8feb900 000000df`); in the retry only the `qmd_word 34` page is +copied to the window-0 shadow (`qmd_word 34 page copied to window-0 +shadow VA=0x1e8000`), so the program pointer is left at the guest VA and +the engine's page tables have no PDE for it. The post-mortem program +target reads `slot=30 ... w0..15 = 0`. + +**Reading.** The data-set gate and the r800 guard are settled: the +no-external-data launch is legitimately forwarded and the engine's +channel/context are alive (it fetched the QMD from the host deposit and +issued a real MMU fault). The remaining wall is exactly two coupled +facts: +1. **Mapping** - the engine's page tables do not map the guest program VA + (`FAULT_PDE`), and the host keeps that VA because the code source is + an empty page (the shadow rewrite is only used for sources it can + trust). +2. **Content** - the program page is empty in every host-visible source. + The driver's program-upload path stages the ucode and copies it to the + code heap (the observed CE `LAUNCH_DMA`/inline shape); that copy has + not been observed executing for the Vulkan client. + +**Next.** (1) Find and execute the Vulkan client's program-upload copy +(the staging->code-heap CE copy) so the program page becomes non-zero - +the utility/GR sweeps are the instruments, and the code page becoming +non-zero is the milestone. (2) Then map/relocate the program VA for the +engine (or install the PDE) so the fetch succeeds. (3) Then non-black +frames -> Xorg/GZDOOM -> KVMFR/LG -> CUDA 15/15 + soak -> the 1.0.0 +video -> Windows. Until (1)/(2) land, keep `min_data` at its default 16: +the bypass forwards a launch the engine cannot complete, which is worse +for the client than the software-completion fallback. + +## 21. UPDATE 2026-10-26 (SESSION 143b): THE COPY-CLASS CHANNELS ARE REGISTERED, SWEPT AND NEVER PUSHED; NO CHANNEL CARRIES PROGRAM BYTES OR A LAUNCH_DMA; THE CODE PAGE NEVER FILLS - AND THE REFUSAL PATH NO LONGER PRODUCES THE BLACK-FRAME BASELINE + +**The instrument (commit `c5510744`).** The util sweep now logs one-shot +`sema-utilsweep: begin|skip|read=none` for every channel plus a 128-row +per-pass cap, so "absent from the log" can no longer be confused with +"absent from the channel table". + +**Channel visibility (r803, full params, `util_scan=64`, `min_data=16`).** +Ten channels in the table, all with gpfifo/entries, **all USERD +unresolved** at first sight: +``` +begin h=0xbeef0101 eng=11 gpfifo=0x4100000 entries=32768 userd=0 +begin h=0xbeef0100 eng=0 gpfifo=0x4020000 entries=32768 userd=0 +begin h=0xbeef0102 eng=12 gpfifo=0x4160000 entries=32768 userd=0 +begin h=0xbeee0101 eng=11 gpfifo=0x4500000 ... +begin h=0xbeee0102 eng=12 gpfifo=0x4560000 ... +begin h=0xbeed0101 eng=12 gpfifo=0x49c0000 ... +begin h=0xbeee0100 eng=0 gpfifo=0x44a0000 ... +begin h=0xbeec0104 eng=0 gpfifo=0x4d20000 ... +begin h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ... +``` +The Vulkan client's copy-class channels **0xbeef0101 (eng 11) and +0xbeef0102 (eng 12) are in the table and swept** - and their rings carry +nothing all boot: `read=none` on first sight, `swept=0` on every pass, no +`utildec`, no LID, no copy. The client posted one event on 0xbeef0101 at +registration (`post_event hSrc=0xbeef0101`) and never pushed. + +**What did decode.** All boot: 3 LID writes only - the driver's two zero +buffers (256 B -> vidmem 0x423f000, 12288 B -> 0x4448000, both +`zero=1`) and the launch's own param bank (1568 B -> 0x4b86020, +`zero=0`). No `sema-lda`/`sema-opadd` rows at all, no `copy-in` rows, no +`utildec` on the two CE channels. Global swept=1652, writes=3. + +**The code page never fills.** `0x62b000` (the launch's program page for +`prog=0xdfd8feb900`) and the rotating-pool page `0xc8b000` read ALL-ZERO +before the client, after it, and at a final re-dump; the whole 256 MB +aperture holds only 12 non-zero dwords boot-wide (the viewport scalars +`0x44200000`/`0x43f00000` at 0x386634/0x386638, the releases, the +baseline). Nothing anywhere writes program bytes. + +**Client outcome.** `VKDEVICE ok family=0` then +`VKFAIL vkWaitForFences ... -> 2`; no frames, no PPMs. The service +answered `GR-COMPLETE-SW` 10 347 times for the same refused push while +the client kept waiting. This is the same outcome as the r801b forward +test, and it differs from the earlier black-frame baseline +(`VKFRAME seq=1..3`, three black PPMs): on a refused/failed forward the +service authors a client-visible `SYNTH-FAULT` (commit `c6c152cc`, +`sema_refusal_fault=1` since r618), and the current completion/release +path does not get the client past its fence. `util_scan` is excluded as +the cause (r801b failed with it off). + +**Reading.** The Vulkan client *believes* it uploaded its compute shader +(it dispatches with a valid program VA), but no CPU write, no CE LID, no +CE copy and no GR i2m in any observed channel ever delivers the program +bytes to the emulated vidmem. The driver's upload path is a +`staging ring -> code heap` copy (CPU memcpy into a staging BO, then a +GPU copy to the code heap); the copy is not among the decoded pushes and +the staging BO is not visible in the emulated vidmem. Two separate +questions now stand: +1. **Delivery** - where the driver queues that copy (a CE channel it + never pushes, a GR subchannel/method shape the parser does not + decode, or a sysmem staging BO that a different read path would + expose). +2. **Completion regression** - restore the refusal-path software + completion (or gate the synthetic fault) so the client reaches the + fence-released black-frame baseline again; this is needed for every + downstream observation. + +**Next.** (a) Dump and decode the client's push method stream around the +upload (the `staging ring -> code heap` copy) rather than more ring +sweeps; the staging BO's CPU write is the most likely host-visible copy +of the SASS. (b) Instrument the completion path for one refused Vulkan +launch (release ordering vs the synth-fault) to restore the black-frame +baseline. (c) Only then resume the mapping work (the r801b `FAULT_PDE` at +the program VA). + +## Session 154 (2026-11-02): formation protocol, execution regression, first engine-final stores + +Three durable findings from the r1414-r1427 experiment series (host module +`vgsp.ko` unchanged; insmod-config + protocol only): + +1. **Client formation needs one X-client teardown + re-create ("F1").** The + boot's first DDX registration leaves a stale X-client VAS whose page + directory never resolves the completion-counter page (`counter page + 0x20400f000 unreadable via vaspaces/discovered/targeted; slot stays + pending`); the GL client then never allocates its channel group + (`class=0xa06c` / `chan alloc hObject=0xbeef0100`). Terminating Xorg once + and restarting it re-offers a fresh `VAS 0x10005` page directory; the + counter derives ~36-64 s later and formation completes on the next client + draws. Seven consecutive boots without this step never formed; five boots + with it formed every time. +2. **The IRAM-gate readback default regressed macro execution.** With both + IRAM-gate sites enabled (`sema_gl_mme_iram_gate_break=1` + `sema_gl_mme_iram_gate_build=1`), the first window-bearing launch after the + pack wedges the front end: grfifo frozen FULL, launch `events=0`, + `macro-status=0`, `GP_GET-frozen`, no macro ever executes. A fresh boot + with both sites off (`=0`) executes macros again (`macro-status` with + run=1 progressions, `events>0` ends, SURF/ATTR preludes emitted) - the + pre-CONT-107 behaviour. Runtime flips do not recover an already-wedged + boot (the lane state is sticky), so the sites must be off from insmod. + (`sema_gl_mme_keep_clamp=0` must NOT be used: it breaks the capture path, + `sa=0` / IMMATURE-SKIP flood.) +3. **Golden overlay + gates off produces the first engine-final stores.** + With `sema_gl_mme_golden=1` and both gate sites off from insmod, the + called-but-uncaptured slots (DRAW_IAB slot 34 `mac=0x22`, slot 57 + `mac=0x39`) run with zero call drops, and the engine writes guest pages: + `output retirement wait: 3/900 pages changed after 217 ms (settled)`, + `GUEST-WIT retire ... changed (direct shadow/alias stores)` - first + non-zero store retirements on this lane. The screen surface itself stays + unchanged (GATE-1 still black); the lane still poisons + (`badf1201` -> WEDGE-RECOVER) after ~60 s, and the stores land on + relocated-operand pages rather than the render target. + +## Session 154b (2026-11-03): the SPH wall - draw executes, DS rejects at SPH validation + +Follow-up state after the first store retirements, three further levers: + +1. **The macro "run bit" is a completion latch, not an arming gate.** The + engine's macro-status sampler reads back `0x4044b4`; a completed + slot-34 body latches `mac=0x22 mpc=637` with `run=0` (637 = the body's + last instruction). The earlier belief that the DRAW never ran was a + sampler artefact - the slot-34 body is a short straight-line program the + transition-gated logger never caught mid-flight. +2. **`sema_gl_sph_bind` (pipeline-stage bind emission) was 0 on every boot + to r1433 and is the bind lever.** Enabling it emits the per-window + pipeline-stage binds (`SPH-BIND emitted ... win-binds=2 j1-type=0x11 + initblk=17`), zeroes the previous SPH11 signature, and produces the first + observed slot-34 execution (`mac=0x22 run=1`) with live draw-surface + retirements (`retirement wait: 3/2/1 of 901`, OUT-PAGE-WIT on + 0x120700000-0x120705000 = draw-surface rows 614-618, operand/uniform + values). +3. **The residual wall is the DS stage validation.** DS-ESR decode: + bits 23:0 = SPH<n>_ERR, bits 29:24 = pipeline stage. `a2040800` = + SPH11|SPH18; with binds on, `a2040a00` = SPH9 (the vertex SPH fetched + from the window's own j=1 bind alias is rejected; the DS `region=` + readback is 0/0). Variants: `sema_sph_version_val=0x00040003` flips the + SPH9 attribution but not the wall; `sema_sph_region_legacy=1` emits the + legacy 0x1608/0x160c pair (base 0x121201000) but is refused at the method + level (`Cl 0000c597 Off 00001608`); `sema_sph_region_base=0x01500000` + propagates but the launch is refused by an MME class error + (`Cl 0000902d Off 0x2000`) before the DS stage is reached. + +Current best lineage: `golden=1 golden_force=1 hold_calls=1` + +IRAM-gate break/build 0 + `sa_rearm=0` + `sph_bind=1` (binds + draw +execution + draw-surface retirements; still no rasterized pixels - the +region record for the synthesized stage slots remains the named gap). + +## Session 154c (2026-11-04): whole-shadow page-hash bracket closes the observation gap + +The copy-out samples the launch's tracked output pages, so "0/900 pages +changed" could not distinguish "the engine rasterizes but into an untracked +page" from "the engine rasterizes nothing". A whole-shadow bracket now hashes +every 4 KB page of the emulated vidmem before the forwarded GR publish and +diffs after the output-retirement wait (`sema_gl_vidmem_bracket`, plus the +manual `vidmem_hash_snap`/`vidmem_hash_diff` parameters). On the r1457 boot, +across 11 forwarded launches (10 `NOWRITE-CONFIRMED`, `n_data`/`tracked` +900-902, `wrote=0`), only USERD/ring/PBDMA-progress pages changed (0x80, +0x100, 0x18f, 0x1b0, 0xbc0); no page in the render-target range +0x200000-0x500000 and none at the bounce VAs 0xa000000/0xc000000. The +render-target surface and its bounce candidates are byte-stable, so the +no-pixel state is an execution gap, not an observation gap. + +The live window was also decoded against the class method tables. Present: +the colour-target record (0x840 = SET_COLOR_TARGET, 1280x720, X8R8G8B8, +blocklinear), raster enable (0x37c), shader pipeline binds (vertex 0x11, +tessellation-init 0x21, pixel 0x51), vertex attributes and stream formats, +the index buffer, the slot-2 vertex-stream macro, the slot-0x22 draw call +(0x3910 = CALL_MME_MACRO(34)), and Begin/End. Absent: the viewport family +(0x0a00-0x0a14 scale/offset, 0x0c08/0x0c0c clip, 0x1924 pixel centre, +0x192c/0x193c clip control), scissor, blend and depth/stencil. In a fresh +replayed context a missing viewport clips all geometry, which matches the +zero-fragment observations. The next experiment synthesizes a gated viewport +prelude at the window preamble and re-runs the bracket; a pass is any changed +page inside the render-target range. + +The synthesized viewport was then closed with the driver's own semantics: the +live viewport macro is MME slot 33 (CALL method 0x3908), the clip rectangle is +0x0c00/0x0c04, and the clip control must read 0x00021800 (Z_CLIP_RANGE +ZERO_ONE | GEOMETRY_CLIP FRUSTUM_XYZ) - the earlier value 1 left the geometry +clip in its WZERO mode. The colour-target closure was also found: the client's +pixel shader has MRT_ENABLE=0 and writes target 0, while the window binds only +target 1. Both a corrected viewport/clip prelude and a target-0 bind with a +two-target SET_CT_SELECT are now gated behind module parameters, and the +pair-state orphan END fix stopped the EC0x0d poison after held windows. With +all of that in place and the DRAW macro executing (DRAW_VERTEX_ARRAY, 60/46 +vertices, QUAD_STRIP), the whole-shadow bracket still reports no render-target +page change. + +The remaining input is the shader content. The program pages carried valid SPH +headers but their code was a 144-byte SPH+EXIT test image supplied through the +`vs_hex`/`ps_hex` debug parameters, which overwrote the client's real SASS; +that image writes no position, so every vertex is degenerate. Removing the +test images lets the real SASS survive at the bound program addresses, but the +engine still stores nothing. The open checks are: whether the real vertex +shader writes position and reads the attribute stream; why the client's own +upload accounting still reports zero applied bytes; whether the vertex fetch +bases hold the geometry the guest writes; and a positive control that draws a +constant-position full-screen triangle with a colour pixel shader. + +The shader-content wall was then broken open: the driver's real program +images were recovered from the init-push payload (the guest had already freed +the page, so the bytes were read from the host RAM backing store). Disassembly +of the 128-byte SPH headers plus SASS shows a genuine pixel shader (IPA/MUFU/ +FMUL/EXIT) and two genuine vertex shaders (ALD attribute loads, FFMA/FMUL, +MUFU.RSQ transform path; 1408 and 1536 bytes). Feeding the pixel image through +the existing per-stage program path (`ps_hex`) and switching the synthesized +viewport to raw scale/offset words produced the first engine render-target +writes on record: three of the 900 render-target pages changed, three pages +were copied back from the bounce buffer, the whole-vidmem page hash saw a +launch-page change, and the captured root window contains a handful of +non-black pixels. The pipeline now runs vertex -> raster -> pixel -> render +target end to end; what remains is coverage - the viewport/clip transform +collapses the geometry to a few isolated pixels - plus a delivery detail: a +window whose vertex-shader upload was never captured has an empty VTG page, +and a vertex image applied only to the host shadow is lost when the window-0 +shadow is re-copied from the guest page. The next work is to deliver the +captured vertex image into the guest page for every window and to calibrate +the viewport against the client's own CALL(33) arguments or a synthetic +full-screen control draw. + +## Session 154d (r1494-r1496): fixed-CB stability and contiguous VTG streams + +Two delivery defects on the path from the client's constants and vertex buffers +to the engine were isolated and fixed; neither alone produces fragments yet. + +1. **The fixed constant buffer must be re-delivered every draw.** The MME + constant-buffer base is forced to a page inside the client's screen + allocation and the engine reads it through the launch's bounce alias. That + page is ordinary guest memory: the client (and the guest console) can + overwrite it between launches. The previous fallback wrote an identity + matrix whenever the source VA failed to resolve, which wiped an already + validated real CB (`cb-copy ... src=synthetic-id`). The copy now remembers + the last page that resolved through a SYS (engine-visible) source and passed + the matrix gate, and reuses it for the rest of the boot when a later + resolution misses. The synthetic identity is used only when no real CB was + ever resolved. The copy is not skipped on an unchanged hash, because the + seed page can have been rewritten by the guest since the previous launch + (receipts: `src=vaswalk` -> `src=cached-real` with the client's MVP in + `m[0x80..0xbc]`, and a guest-witness readback of the real matrix words). + +2. **A VTG stream must be one contiguous engine-visible range.** The vertex + stream macro payload is `[stream, end_hi, end_lo, base_hi, base_lo]`, where + `end - base` is the stream length. Relocating `end` and `base` as two + independent single-page operands put a page-crossing stream's ends in + unrelated shadow windows: the computed length went negative and the fetch + past the base page read an invalid PTE (the `miss=1` poison seen once the + real vertex shader started executing). The payload pair is now mapped into + one fresh contiguous fetch line (up to 16 x 4 KB) and both words are + rewritten from that line; unresolvable pages map the module zero page so a + read is a degenerate vertex, never a fault. The receipt + (`MME-VSTREAM-LINE ... pages=N dummy=... line=... first8=...`) reads the + client's real vertex floats back through the mapped page. + +**Remaining wall (measured, r1496).** The active indexed draw's vertex and +index buffers are vidmem (BAR1-aperture) allocations: the launch page set +resolves them to FB page `0xb9f000`, but the emulated vidmem shadow for that +page is all zero (`dump_vidmem` proof), while the same vertex/index data is +present in guest system memory as the client's staging/backing copy. The +engine therefore fetches zeros for the active draw even though the stream +mapping is correct, and the copy-out still reports `0/900` changed pages. The +fixed-CB source VA also moves between boots (resolved in r1492/r1493/r1495, +unresolved in r1496), so a single hard-coded source VA is not sufficient. The +DS continues to latch SPH stage errors on some windows. The next work is to +source the active vertex/index page from the sysmem backing (or to make the +vidmem upload deterministic) and to make the CB source discoverable per boot. + +## Session 154e (r1497-r1499, 2026-11-05): real-FB fetch delivery, per-boot CB, DS post-clear + +Three changes closed the "engine fetches zeros for the active draw" wall and +moved the measured state to "the engine clears the target but draws no +geometry". + +1. **Empty-shadow FB fetch pages are delivered from the real frame buffer.** + The active indexed draw's vertex/index buffers are mapped by the client's + own page tables with the vidmem aperture, but no emulated path ever + populated the BAR1 shadow copy of those pages (the upload did not pass + through an inline write, a decoded CE copy or a CPU aperture write), so + the engine fetched zeros. Two delivery paths now cover the page: + * when every byte of the shadow page is zero, a launch fetch page (a QMD + pointer, not the data-fill surface) is mapped with a VID-aperture leaf + to the real frame-buffer offset the guest page table names + (`FETCH-VID-MAP va=... fb=... qmd=...`); + * a vertex-stream page uses the same rule inside the contiguous fetch + line (`MME-VSTREAM-LINE ... vid=N fb0=...`); + * if the same VA also has a sysmem mapping with nonzero bytes, those + bytes are copied into the shadow page (`FETCH-SYSBACK`), which covers + boots where the client keeps a CPU-visible staging copy. + A page whose shadow already has content is never touched, so the working + sysmem streams, the fixed-CB seed and every engine-written page keep + their identity. Render-target/surface pages (QMD word NONE) are + deliberately excluded: their stores must land in the emulated page so the + copy-out can ship them back to the guest. + +2. **The fixed CB is discovered per boot.** When the client's + `SET_MME_SHADOW_SCRATCH(0x18)` write reaches a decoded push it is used as + the fixed-CB base; when that capture is absent the historically observed + base is tried as a candidate and the matrix gate decides. In r1499 the + candidate resolved through the VAS walk to the client's real page + (`first8=3ff35a31 be800000 be9e377b 3f737870 ...`, the client's MVP), and + 42 of 96 copies used the real page with the rest served from the + remembered validated page. + +3. **The DS HWW ESR latch is cleared after each replayed window as well as + before it.** The latch is W1C; the pre-clear made each window's validation + set readable, the post-clear keeps a failed window's pending report from + poisoning the next window's entry state. The poison count stayed at 0-1 + per run with the mask knob enabled. + +**Measured state (r1499).** The active draw's streams now read the client's +real vertex floats (`MME-VSTREAM-LINE base=0x44fea00 first8=3f800000 00000000 +bf000000 bf800000 ...` and `base=0x44ff000 first8=4069999a ...`), and the +engine executes without a fetch fault. It writes an opaque-black clear +(`0xff000000`) into the render-target shadow page at FB offset `0x200000` +and into the forced RT seed page at `0x504000` (`dump_vidmem` readback), the +bounce copy-out ships the changed seed pages, and the guest's window capture +is uniformly black (`xwd -id`, 800x600, non-black pixels 0, bbox 0,0,0,0). +The clear reaching the target proves the RT0 bind and the color-store path; +no gear fragments are produced. The next wall is geometry coverage: the +index-buffer bind, the vertex-shader position output and the viewport/clip +transform must be checked against the client's own draw arguments (the +active index page currently reads as vertex floats at its bound VA, so the +index/vertex role or the index count is also a candidate). + +## Session 154 (r1515-r1517): the fixed-CB upload is decoded but its bulk is not in the method stream + +The client publishes the fixed-CB base once per context +(`SET_MME_SHADOW_SCRATCH(0x18)`, method `0x3460`, value = VA >> 8) and then +writes the constant store through the C597 methods +`SET_CONSTANT_BUFFER_SELECTOR_A/B/C` (`0x2380/0x2384/0x2388`), +`LOAD_CONSTANT_BUFFER_OFFSET` (`0x238c`) and `LOAD_CONSTANT_BUFFER(i)` +(`0x2390`). The emulator now scans every decoded push of the client's GL +channel for those runs, assembles the first 4 KB of the constant buffer, +validates it with the strict projection gate, and delivers a passing image +to the forced MME seed page; a receipt reports the push, base, selector, +run/word counts, the `c[0x0][0x80..0xbc]` block and whether the projection +shape passed (`CB-UPLOAD`, `CB-UPLOAD-SCAN`, `cb-copy src=const-upload`). + +Measured on a validated boot (r1517, glxgears formed, draws submitted): the +60216-byte setup push is read and the scratch-18 base is captured +(VA `0x4460000`), but its `0x2390` content is only 10 state-patch words +(selector/offset pairs, no matrix rows). The base page resolves to the +emulated vidmem offset `0xb10000` and reads zero; the neighbouring +`0x43e0000` page is client vertex data, and the guest-RAM projection-shape +scanner finds only transient copies. The draw therefore still runs with the +synthetic identity (r1517 `GL-COVER` clip space inside 50/64 sampled +vertices) and the window capture is black (`xwd -id`, non-black 0). The +remaining work is to capture the constant-buffer *bulk* transfer itself +(the staging buffer to the vidmem CB page), not the method-stream patches. + + +## Session 155 cont 8 (2026-11-06): DS SPH bitmap semantics and the BEGIN/END placement wall + +Two results worth keeping: + +1. **The DS SPH bitmap is `stage*4+check` over the six GF100 pipeline stages** + (VSA, VSB, TCS, TES, GS, PS). The six per-stage SPH start registers + (`DS_HWW_ESR_SPH_INFO_STAGE0..5`, 0x405c00..0x405c14) and the live + `DS-FAST` stage VAs plus the fetched SPH header types prove the stage + index: stage1 0x…900 is a V4T1 `SHADER_TYPE=1` VTG image, stage2 0x…200 a + V4T1 `TYPE=2` tess-init image, stage5 0x…0 a V4T2 `TYPE=5` pixel image. + The four checks per stage are the four SPH-V4 validation groups: header + type/version, mpipe/stage binding, attribute map (IMAP/OMAP + register + count + stream formats), and stream-out/topology. A clean boot with only + VSB/PS bound reported `a0040a06` = SPH1|2 (VSA checks 1/2), SPH9|11 (TCS + checks 1/3), SPH18 (GS check2) - i.e. the unbound stages are the failures. + Enabling the TCS bind added SPH10 inside the TCS group (r1554c), which is + the live proof of the `stage*4+check` split. + +2. **Synthesizing the missing VSA/GS stage records clears those bits but + their method run lands inside BEGIN/END.** Binding VSA/GS to the client's + own VTG image (`mask=0x35`) moved the bitmap from `a0040a06` to + `900400a0` (SPH1|2|9|11 gone; SPH5|7|18 remain), then the next window + raised `Class Error Offset 0x2018 Data 019a0900 EC 0x0d` - + `ILLEGAL_METHOD_IN_BEGIN_END`. The value is accepted; the placement is + the wall (the synth block is prepended to a window whose previous END was + dropped). Next: make the stage-record emission BEGIN/END-safe (emit after + an explicit END when the FE begin state is still open) and keep the + client's own TYPE/REGISTER_COUNT-before-address order. + +`SET_SPH_VERSION`/`CHECK_SPH_VERSION` (0x16a4/0x16a8) is not a lever for +this bitmap: blob-derived, `0x10004` and `0x40004` all produce the same DS +value on the same boot. diff --git a/docs/04-implementation/05-graphics-execution-plan.md b/docs/04-implementation/05-graphics-execution-plan.md new file mode 100644 index 0000000..6089548 --- /dev/null +++ b/docs/04-implementation/05-graphics-execution-plan.md @@ -0,0 +1,721 @@ +# Graphics execution plan: real GL pixels through the emulated vGPU + +> **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the +> **graphics execution layer**: the component that must execute the guest +> NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that +> glxgears and GZDoom's OpenGL renderer produce real pixels through the +> emulated mdev. Nothing here is implemented yet beyond the pieces cited; +> every statement carries a code location or a measured session finding. +> +> Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA +> compute chain, proven), `docs/04-implementation/04-real-render-walls.md` +> (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log +> (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3 +> SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16 +> (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public +> class headers are the interface authority; no guest driver change is +> allowed. +> +> Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl, +> VirtualGL, llvmpipe, any second device are banned for this work), **no +> fake/labelled software rendering** presented as GL, **host-only emulation** +> (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard +> rule; a reboot needs the handoff update plus explicit human approval). + +--- + +## 0. What "real pixels" means here + +The acceptance target is: a stock guest GL application's draw commands are +executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch, +primitive assembly, rasterization, fragment work, ROP), its render target is +readable by the guest, and that image reaches the capture path through the +emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a +software rasterizer, or a host display round-trip. + +Two consequences drive every milestone below: + +1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work + proved the engine will execute a *synthesized* C5C0 compute launch built + from the GL stream and will retire it + (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need + the 3D pipeline and the client's own per-draw state; rasterization is + fixed-function GR (`session-149-BRIEF.md:288-291`). +2. **The GL client's own channel is the source of truth.** The C597 section + the emulation can capture today comes from a readable channel (the X + server side); the GL client's GPFIFO ring (`0xbeef0100` family, + `0xbfef0002` observed in SESSION 153) is where its context-init uploads + and per-frame draws live. Until that ring is readable, no approach can + derive a correct draw (`session-149-BRIEF.md:438-451`). + +--- + +## A. Gap list, ordered by dependency + +Each gap lists what exists (with the code location that implements it), what +is missing, and the measurable gate that closes it. A gap is closed only when +its gate has been met on a fresh cycle with evidence kept. + +### G-A Readability: the GL client's ring and pushes must read + +**Exists.** The pump resolves rings through an autopin ladder (notifier- +derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with +dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602` +(`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`), +`:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The +failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan +found no candidate in N slice(s)`). Push reads report through `sema-rdw` +(`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client +selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`, +`tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already +defaults to the widened GL prefix set +(`tools/guest-diag/pin_supervisor.sh:70-83`: +`cdt,va,victory,Xorg,X,glxgears,gzdoom`). + +**Missing.** For the GL client channel the ring page is not found by content +at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12 +slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are +just above the old 64 MB capture bound in the Vulkan case +(`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's +ring candidates are shape-based; a client ring written late (after the DMA +test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines, +`guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not +covering the GL client until SESSION 153; the supervisor default now does. + +**Gate G-A.** On a run with glxgears alive for 60 s and the standard X +recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for +the GL client's handle; (b) every client push logs `sema-pushdec ... src>0 +unreadable=0`; (c) at least one entry is consumed from the client ring +(pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane +`va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153 +item 1). + +### G-B Capture: the client's C597 sections and per-frame draws must decode + +**Exists.** A gated recognizer (`sema_gl_xlate`, default 0, +`kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT` +class bind of `0xC597` (TURING_A) on the channel subchannel and collects +observed VA operands: program (`0x2014`/`0x2018` runs), +tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`), +upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA +payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word +QMD and captures it through the unchanged GR candidate ring +(`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by +a standalone subchannel-0 pair tracker (`:12045-12084`) and the section +walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`). +Completion of the capture is gated by content trust: a plausible stream +admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified +rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard +requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines: +`sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...` +(`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...` +(`:7261`). + +**Missing.** +1. The recognizer's method vocabulary covers init-shaped operands only. The + per-draw 3D state and draw-issuing methods (vertex streams/index, render + target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are + not decoded (`svc_gl_field` handles six addresses, + `:6945-7049`; the C597 stream carries far more). +2. The GL program/constant delivery is a context-init upload on the client + channel and is lost today: R17 measured that no readable push ever writes + `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized + launch therefore only runs a 12-28 B fallback blob or an operator hex + program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not + pixels. +3. Capture must survive the "one-shot setup push rewritten in place" + class: re-captures come from hash-verified replays now + (`:29069-29080`), but the per-frame draws are separate entries and need + the same treatment per entry. + +**Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl: +section captured` on the **client's** handle with nonzero `prog/param/rel`, +one captured section per frame (bounded replays suppressed), and a decoded +draw record (new `sema-gl-draw` witness) whose vertex/primitive count is +consistent with the app (e.g. glxgears gear geometry). Program source +priority shows `trusted-prog` (real bytes) for the context-init upload. + +### G-C Execution semantics: the GL work must run on the real 3D engine + +This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3. + +### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest + +**Exists.** The physical backend builds engine-side page tables and shadow +windows for launch data: Tier-1 identity install or Tier-2 shadow copy +(`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank +window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a +24-page shadow chunk (`:1317`), QMD-window relocation +(`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that +relocates the inline-QMD address words and data words +(`:16878-16921`). Data pages are copied in before the doorbell and copied +back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`). +`svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or +the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`). + +**Missing.** +1. Address-word relocation is QMD-only (words 32-49 and the indirect-page + pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at + class-specific offsets; each must be resolved to a shadowed page and the + word rewritten (or the page mapped at its guest VA). +2. The engine-visible page budget is small: 16 pages per request + (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per + win0 bank (`booter:1174`). A GL frame names program + constants + texture + pools + vertex stream + render target + ZS; the shadow pool and the + per-request page list must grow or pages must be grouped per buffer. +3. Render-target writeback: the copy-out tracks only the launch's own data + pages; a 3D draw writes the RT, which must be mapped into guest-visible + memory (BAR1 shadow or pinned guest RAM) and copied back on the + completion edge, exactly like the current bounce/copy-out but for the RT + and depth surfaces. +4. Program/constant bytes: R17's measurement stands — the program page is + empty in every host-visible source until the CE/LID delivery executes + (`docs/04-implementation/04-real-render-walls.md` §20-§21). + +**Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with +`FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`, +`booter:18997-18999`), and a measurable post-launch change in the emulated +vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class, +`docs/04-implementation/04-real-render-walls.md` §11). + +### G-E Completion and sync: the guest must be released by real completions + +**Exists.** Engine completion is polled (`qmd_rel/ch_rel`, +`booter:20814-20825`), the launch's release0 can be redirected to the module +semaphore and relayed (`booter:19499-19520`), the hold edge and the +backend-outcome edge author the guest word (`kernel/gsp_sema_service.c: +21329-21345`, `:21298-21322`), and failures synthesize a client-visible +channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`). +The pre-built ring continuation beyond GP_PUT is swept for releases +(`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`). + +**Missing.** The guest GL client's waits are still satisfied by the give-up +ladder or by host pokes: the X server's present wait spins on the X channel +member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror` +`kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares +`obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the +emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`: +"one u32 write 3 → 128.8 FPS", but only via `/proc/<pid>/mem`). The correct +long-term fix is the one named in addendum 16b §C.4: **the emulation +completes the real syncs so the equality is satisfied, with no lying pokes**. + +**Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no +`sema_xw_mirror` target written, and the guest's waits exiting on a natural +completion edge. + +### G-F Present/frame path: the rendered image must land where the guest and the capture can read it + +**Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap +hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`), +`tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher), +`tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder +`tools/lg/frame_reader.py` and the resample/encode scripts. + +**Missing.** On the emulated-vGPU stack the X screen is black for GL windows +(DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels +exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of +the GL push, R13) is decoded for releases but not executed as a display +path, and the capture must come from the GL client's swap, not from X. + +**Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom). + +### G-G Concurrency: two VMs, two mdevs + +**Exists (designed in).** The service thread iterates every open device each +pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that +"with two, both guests get a pump turn per iteration (hardware forwarding +stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest +sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges +are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`). + +**Missing / to verify.** Single-inflight forwarding and holds +(`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs +do not co-reside: the write/read ledger is keyed by VA page only and its own +comment says collisions "only occur ... across clients whose pages do not +co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`), +`svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); +the win0 banks are module-global per submit epoch (`booter:19406-19410`); the +server-managed VAS relocation windows are a finite global set +(`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to +avoid a second fragile RM init", so a second RM bootstrap on the same module +is unproven. + +**Gate G-G.** M-GL5 (below). + +--- + +## B. Execution-semantics approaches + +### (i) 3D method-stream replay on the real engine + +Forward the client's captured C597 method stream (class bind + 3D state + +upload/draw methods + in-stream releases) to the module's real GR channel +and let the fixed-function pipeline execute it. + +What must be mirrored/provided: + +| Requirement | Where it stands | +|---|---| +| Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. | +| 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. | +| 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. | +| Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. | +| Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. | +| Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). | + +Pros: authentic rasterization is done by the hardware pipeline; no state is +invented; the engine has already proven it fetches and executes forwarded +work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is +unchanged. + +Cons/risks: the promoted context may still be compute-shaped (class object +construction on a wired golden channel deadlocked in earlier experiments, +`booter:816-822`; V7 is the tested path, so construct there); the 3D method +vocabulary is large (state validation can silently fault); address-word +relocation must cover every class encoding; ~6-12 operand pages per frame +exceed today's 8-page win0 bank, requiring grouping/expansion. + +Effort: L. Depends on G-A, G-B. + +### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream + +On Turing, a QMD is a *compute* launch descriptor; there is no QMD that +rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and +author the state + a DRAW method ourselves — program (`0x2014/0x2018`, +already decoded), constant buffers (bind + LID uploads, partly decoded), +vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer, +`gsp_sema_service.c:6979-6983`; the real stream/index methods are not), +render targets/ZS (not decoded; the only observed target is the release +target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded). +The vertex count for the draw would be inferred, not read. + +Pros: fewer pages to mirror than a full replay if only draw-relevant state is +bound; still uses the hardware rasterizer. + +Cons: we author state the driver owns — any wrong field silently +mis-renders; the per-draw state lives in the client ring, so G-A/G-B are +still prerequisites; the program bytes are still lost without G-B; it is a +subset of (i)'s work plus extra invention. + +Effort: L. Risk: high. Not first. + +### (iii) Bounded compute translation for the subset GZDoom uses + +Replace each GL draw with a compute kernel (or the existing synthesized +C5C0 launch carrying the client's real SASS) that rasterizes triangles into +the RT. + +What breaks correctness, concretely: +- **Fixed-function raster and ROP**: depth test, stencil, alpha blending, + culling, clipping, perspective-correct interpolation, MSAA — none exist in + a compute launch; reimplementing them is writing a GPU. +- **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes + and sRGB conversion are texture-unit behaviour. +- **Shader semantics**: the client's GLSL is already compiled to SASS by the + guest driver; executing that SASS as a whole-SM compute program does not + give it vertex inputs, varying interpolation, or per-fragment coverage — + the very things the 3D pipeline supplies. +- **The claim itself**: even GZDoom's first frame uses FBOs + (scene → post-process), depth testing, alpha blending and mipmapped + textures; output would be a lookalike, not the renderer's pixels, and + could not be verified against a reference. + +Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern +kernel proving the lane), never as the milestone path, and it must be +labelled as such. Not first. + +### Recommendation + +**Implement (i) first.** It is the only approach that yields authentic GL +pixels; its difficult prerequisites (G-A ring readability, G-B stream +capture) are shared by every alternative; and the backend already provides +the three pieces that make it tractable: a real execution lane with a +validated context (mode 35), a generic 0xC597 class-object allocator, and a +verbatim method-window replay with address/QMD rewriting plus page shadowing +and completion relay. (ii) is (i)'s draw-issuing step taken prematurely; +(iii) trades correctness for nothing. + +Order of implementation inside (i): **G-A** → **G-B** → replay the captured +*context-init* section (M-GL1, proves class bind + uploads + completion on +the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3). + +--- + +## C. Milestone ladder + +All gates are measured with the artifacts listed in section D; a milestone is +met only when its gate reproduces on **two consecutive fresh cycles** (three +for M-GL1) with the module hash recorded. + +### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop + +Gate (all in one run): +1. GL client ring resolved: no `sema-chan: GR ring h=<client> ... found no + candidate` row after the first minute; an autopin row exists + (`sema-chan: auto-pinned GR ring ...`). +2. Client pushes read: `sema-pushdec` rows for the client handle with + `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them. +3. Section captured from the client: `sema-gl: section captured h=<client>` + with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay + admission), at least once per client lifetime. +4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h=<client> + ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +... + qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`. +5. Client behaviour: glxgears no longer loops on context creation (no + repeated init-failure retry; the X server stays alive; `xdpyinfo` OK, + `DMAFAIL=0`), and the app reaches its first swap. +6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client + (≤ a handful in the window), no guest oops. + +Expected failure signatures (test one at a time): +`GR ring ... found no candidate` (G-A); +`unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class); +`GR request without candidate` / `cand_valid=0` +(`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the +request); +`GR submit refused: degenerate data set n_data=N (min M)` +(`booter:19352`; operand list, G-D); +`FAULT-PDE` at the program VA + `CODEHASH h0=0` +(`booter:18806-18883`, `:18997`; G-B/G-D); +`GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not +retire — class/sched fields or completion wiring). + +### M-GL2 — glxgears renders non-black frames + +Gate: +1. A GZF2 record stream is captured from the **GL client's swap** (see D for + the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use + SDL). +2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct + dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels + (motion, not a static noise tile). +3. The same time window contains the per-frame GR launch + completion in + dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa). +4. The first frames are reached without any sync poke. If a diagnostic poke + is used to unblock the run, that run is labelled diagnostics and does not + satisfy the gate. + +Failure signatures: swap hook fires but frames all zero → no rasterization +(G-C); `xwd`/X path black → expected, do not use; frames show only clear +colour → draws executed but state/vertex data wrong (G-D); +`FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost +(G-B). + +### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev + +Gate: +1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0` + backed by the emulated vGPU (no Xvfb, no Mesa overrides). +2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including + gameplay (not only the menu); the in-frame FPS counter ≥30. +3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT + loop; the X server stays alive for the whole run. +4. Capture fetched via QGA and decoded offline; a screenshot + (`+screenshot`) matches the captured frame within a small pixel delta. + +Failure signatures: window opens, swap fires, all frames black → draws not +executed; `GR submit refused ... n_data` → operand pages; app crashes with +`Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not +SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame → +forward failed or refused. + +### M-GL4 — 60 fps sustained in ONE VM, no forcing + +Gate: +1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s + measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement + itself), and GZDoom's counter ≈60. +2. **No** poke/force/mirror daemon is running and no runtime module param is + used to fake progress: `sema_xw_mirror` is `0,0` (stopped, + `kernel/main.c:3103-3129`), no `/proc/<pid>/mem` write helper, no + strike/deadline forcing. +3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up). +4. The X present waits exit with member advancing naturally (Xorg log WAIT + lines absent or resolving without the kickfix daemon), and the glcore + equality word no longer requires the bump. +5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the + same boot after the GL run (battery-first ordering). + +Failure signature: the two ~11 s glcore waits return (the app's own strike +ladder; S144 addendum 16b) → G-E; X WAIT with static member → present +completion not authored; `GR-STALL-backend` per frame → completion is +software-only and the app's ladder is what "ran". + +### M-GL5 — Two VMs, two mdevs concurrently + +Verification checklist before the gate: +1. Second mdev creates and its guest RM bootstraps (the S151 open question: + "a second fragile RM init"); both devices appear in the service loop + (`gsp_sema_service.c:32119-32236`). +2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key + (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), + `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do + not collide in practice or key them by `(vgpu, va)`. +3. Guest RAM: both VMs' memory-backend ranges registered and resolving + (the cycle registers one file per guest; verify the per-device map). +4. Server VAS relocation: no two guests' data windows alias in the managed + region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`). +5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized + submissions; single-inflight holds (`:19878`) do not starve one guest. + +Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present +non-black frames; each ≥30 fps (60 target); `victory_mail` green in both +afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` / +`LEDGER-RESET` absent), no host oops, no guest wedge. + +Failure signatures: one VM starves the other's forward slot (`GR request +busy: hold active`, `:25509`); second guest's launches resolve the first +guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second +RM init fails in the boot ladder. + +### M-GL6 — Flawless 60 fps split-screen video + +Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real +GZDoom GL gameplay through the two mdevs, encoded with the existing +resample/compose pipeline, uploaded with the files-upload skill and +pull-back-md5 verified. Encode with `libx264` (the R18 path did: +`/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns +the physical GPU (the host NVIDIA driver is displaced). + +--- + +## D. Experiment recipes + +### D.0 Standard cycle (all milestones) + +Host, one controlled attempt per cycle (script leaves the VM running): + +``` +nohup /tmp/opencode/start_vm_keep.sh rNNN "<EXTRA_ARGS>" \ + > /tmp/opencode/rNNN-cycle.log 2>&1 & +# poll until: DONE (no teardown; VM left running) (~70-90 s) +``` + +`EXTRA_ARGS` = the SESSION 149 standard profile +(`session-149-BRIEF.md:65-67`), verbatim: + +``` +gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 +gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 +gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F +gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 +gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1 +gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1 +gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0 +gsp_phys_fault_relay=1 sema_bigcopy_log=512 +``` + +GL overlay (M-GL1..M-GL3 runs): append + +``` +sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0 +sema_gr_util_scan=0 sema_dbg=Y +``` + +(`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`; +`min_data=0` is the explicit no-external-data bypass validated in S143 and +used by R15-R17; `util_scan>0` is a measured regression source, keep 0; +`sema_dbg=Y` raises the bounded witnesses). The base script's ladder already +passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by +the boot ladder — verify the boot log names mode 35 +(`booter:8612-8613`). + +Post-load, before Xorg: + +``` +sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms' +sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 & +``` + +Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle): + +``` +echo ubuntu | sudo -S setsid bash -c \ + 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 </dev/null &' +# require: DISPLAY=:0 xdpyinfo -> 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0 +``` + +After Xorg is up: + +``` +sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush' +``` + +Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the +echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`, +`/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the +acceptance-grep output. + +### D.1 M-GL1 recipe + +Guest: start glxgears and watch the client channel: + +``` +DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \ + > /tmp/glx1.log 2>&1 </dev/null & +``` + +Host acceptance greps on the dmesg capture (client handle filled from the +first `sema-chan` row): + +``` +grep -E "auto-pinned GR ring h=0x<client>" # G-A pass +grep -E "sema-pushdec: h=0x<client>.*unreadable=0" # G-A pass +grep -E "sema-gl: section captured h=0x<client>" # G-B pass +grep -E "GR LAUNCH submitted h=0x<client>" # capture->forward +grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW" # engine executed/completed +``` + +If G-A fails: confirm the guest scanner is running the widened prefixes +(`/tmp/guest_pin_stream.log`; supervisor env at +`tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via +`sema_trace_h=<client handle>` (S148 addendum 10) to see which source fails, +and use `echo <va> >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B +fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed +replay: wait one fresh push or use the hash-admitted path +(`gsp_sema_service.c:29069-29080`) — do not add a bypass. + +Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3 +cycles pass. + +### D.2 M-GL2 recipe (glxgears pixels) + +The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`, +`SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls +neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` → +`glReadPixels`, same GZF2 record layout), build: + +``` +gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl +``` + +Guest: + +``` +DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \ + LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \ + glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 </dev/null & +``` + +Fetch + decode on the host: + +``` +python3 tools/guest-diag/qga_fetch.py /dev/shm/glxframes.bin /tmp/opencode/glxframes.bin +# parse GZF2 records (tools/lg/frame_reader.py / gz_final_gpu.py's reader), +# then measure: % pixels > 8/255, colour count, motion between frames. +``` + +Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep +the shim's swap hook ordered as "capture then forward" (same as sdl_cap). + +### D.3 M-GL3 recipe (GZDoom GL) + +Guest (xorg.conf untouched, the emulated GPU as the only GPU): + +``` +VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \ +GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \ +DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \ + +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \ + +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 </dev/null & +# window management with tools/guest-diag/{xresize,xfocus,xtest_move}.c as in +# SESSION 151; XTEST alone is ignored by SDL here - use the XSendEvent path. +``` + +Host: `qga_fetch.py /dev/shm/gzframes.bin /tmp/opencode/gz.bin`, decode, then +`/tmp/opencode/gz_final_gpu.py` (resample to natural-speed 60 fps) for the +video. **Do not** set `LIBGL_ALWAYS_SOFTWARE` or the Mesa EGL vendor list: +that is the S151 software path and would invalidate the milestone. + +### D.4 M-GL4 recipe (60 fps, no forcing) + +Same as D.1/D.3, plus: + +- ensure `sema_xw_mirror` is stopped (`echo 0,0 > .../sema_xw_mirror`) and no + forcing/kickfix daemon is running (`pgrep -af` exact names first); +- measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2 + timestamps; +- keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B + proved they are not the pacer (`session-149-BRIEF.md:96-99`); +- battery-first CUDA regression in the same boot. + +### D.5 M-GL5 recipe (two VMs) + +Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both +with the vGPU device this time. Verify in order: second mdev appears; second +guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device +handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps +in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets). + +### D.6 M-GL6 recipe (video) + +Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py` +(timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18 +pattern) → files-upload skill → pull-back md5 verification. 1280x440 +split (640x440 per guest) matches the two delivered videos. + +--- + +## E. What must NOT be done + +1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no + llvmpipe/Mesa software GL, no second physical GPU, no host display + round-trip. The SESSION 152 video is explicitly *not* evidence for this + plan. +2. **No fake or mislabelled rendering.** No software rasterizer or compute + lookalike presented as GZDoom's OpenGL renderer. A compute-translation + probe (approach iii) may exist only as a labelled diagnostic and is never + an acceptance artifact. +3. **No lying sync pokes in accepted runs.** The `/proc/<pid>/mem` counter + bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any + milestone evidence must state they were absent. +4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit + human approval, never automatic. Guest VM cycles and Xorg restarts on the + test VM are the allowed recovery paths. +5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no + injected code, patched firmware or modified driver in the guest; capture + shims are LD_PRELOAD observation only and must not alter GL semantics. +6. **No time/timing claims from unsupported sources.** The physical backend + is the execution authority; simulator (L0/L1) findings never substitute + for a live engine result. +7. **No unguarded experiments on the physical GPU.** Every cycle is + supervised, single-attempt, evidence-captured (the cycle script's hard + abort signatures are not bypassed). + +--- + +## F. Open questions and note-vs-code discrepancies + +1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief + names it for glxgears, but glxgears is GLX and never calls + `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX + variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom + (M-GL3). +2. **The GL-client pin prefix fix already looks landed.** + `tools/guest-diag/pin_supervisor.sh:73-83` defaults + `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle + starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still + lists the "permanent fix" as pending. Verify the running guest scanner + actually received the widened env (`/tmp/guest_pin_stream.log`) before + treating G-A as closed. +3. **The logged failure strings differ slightly from the briefs.** The code + prints `vgsp-work: GR request without candidate` + (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the + ring failure is `sema-chan: GR ring h=... found no candidate` + (`:17633`). Grep acceptance scripts for the exact prefixes. +4. **`n_data` threshold history.** The default is 16 + (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit + bypass for the no-external-data launch class, enabled by the r800 guard. + The brief's "min 1"/"min 16" variants refer to different experiments; the + GL profile below uses 0 and must say so in every evidence record. +5. **Stale project status row.** `docs/README.md:114` still lists CUDA + execution as "no grid has dispatched yet", which predates the proven + physical forward and the R18 CUDA video. This plan relies on the newer + code and `docs/04-implementation/03-execution-frontier.md`, not that row. +6. **The 3D context on the execution lane is the main unknown.** Mode 35's + capture "carries no 0xc597 on the user lane" + (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the + execution channel is V7. The first implementation experiment must prove a + 0xC597 class object can be constructed on V7 after the accepted promote + (same helper, `:8936`) without the wired-golden deadlock class + (`:816-822`), before any stream replay work.
\ No newline at end of file diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 2f10b35..1391426 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,26 +17,32 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes, -`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes -(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, -allocation, copies, free and destroy complete, and the copies are verified -byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical -copy engine (default off) with engine-authored releases and a fault relay; for -GR kernel execution a forwarded guest launch now reaches the front-end but the -inline launch-descriptor install has not been observed (no grid dispatched; -context validity at dispatch is ruled out — -[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). -Before Rung 3 -is claimed, the -**reply-fidelity gate** applies: a control whose reply fields the driver -consumes is not implemented until the reply is coherent (status-only → -coherent zeroed params → coherent values for consumed fields → exact values) — -see -[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). -New to CUDA analysis here: start at +**Current rung (2026-10-30):** Rungs 1–3 are green for the verified CUDA +clients: the stock driver probes, `nvidia-smi` enumerates, the full CUDA +driver-API smoke passes (`ALL OK` 64/64), and guest kernels **execute on the +physical engine** — vector-add, matrix-multiply and transformer clients verify +on fresh boots with engine-authored completions relayed to the guest. What +remains before Rung 3 is claimed outright: the staged-transfer class (some +large multi-chunk clients still mis-verify; failure modes proven) and the +every-boot acceptance campaign — plus the **reply-fidelity gate** below. Rung 4 +(sustained) has partial evidence (a 100-launch loop, long soak pending); rung 5 +(multi-tenant) is partial: two mediated devices and two guests coexist, and +per-guest execution isolation plus fairness measurement are in progress. On the +graphics side, the X server/GLX run on the emulated vGPU, the engine executes +GL method streams, and the guest Vulkan pipeline runs end to end; rendered +(non-black) frames are the frontier +([`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md)). +**No performance numbers are published yet**; the A/B programme in §3 is the +next measurement step, and the first scheduled public measurement is the +graphics 60 fps target (no forcing). New to CUDA analysis here: start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). +The **reply-fidelity gate** applies to every emulated control: a control whose +reply fields the driver consumes is not implemented until the reply is +coherent (status-only → coherent zeroed params → coherent values for consumed +fields → exact values) — see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). + ## 2. Correctness ### 2.1 Test corpus diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index cb3ea28..972e9b6 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -422,6 +422,38 @@ anything that is not lowmem; the module rule is recorded in and the shadow mapping is described in [`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). +### 5.4 Case study: vGPU guest bring-up — three size/state traps + +Three independent values can make a stock guest driver fail long before the +GPU work starts. They are worth checking in this order. + +1. **FB info size units.** The framebuffer-info ABI documents the RAM, + total/usable/heap, mappable-heap, BAR1 and BAR1-avail size indices in + **kilobytes**; compression and L2-cache sizes are in bytes. Returning + bytes for the kilobyte indices over-reports the emulated framebuffer by + 1024x, which the X driver turns into a huge surface allocation. The + fix is a per-index unit conversion, not a clamp. +2. **GPU GART size.** A multi-gigabyte GART reply makes the guest RM + allocate that much system memory for the GART at GL init. On a 12 GiB + guest a 4 GiB GART is a fatal allocation: it appears as a + `nv_alloc_pages` burst with a page count in the hundreds of thousands. + The emulated GART should be small (hundreds of MiB at most), matched to + the guest RAM budget. +3. **Shared nvClip mode-number parity.** The first X server start after a + guest boot can leave the shared `screens[0].modeNum` odd. The GL client + reads bit 0 as "mode switch in progress" and clears its init bits, so + every buffer allocation/map fails (typically `GL_OUT_OF_MEMORY`) and a + client that trusts a non-NULL map pointer crashes later. A second X + start normally leaves the value even and the same client proceeds. + Treat this as a guest X-driver state issue; do not chase it as a host + VAS/page-table problem. + +Diagnosis pattern: separate *allocation size* problems (values in the +hundreds of MiB/GiB, no host channel activity) from *buffer mapping* +problems (GL errors with no ioctl traffic, i.e. a userspace init gate) from +*host translation* problems (ring/USERD walks failing with `rc=-2`). The +three have different fixes and should not be conflated. + ## 6. Release engineering * Versions are tagged; each tag carries the rung/matrix table and the diff --git a/docs/README.md b/docs/README.md index 27e04e9..95ef85c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,7 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- ## 3. Where is the project today? -**Honest status (2026-10-02):** +**Honest status (2026-10-30):** A dated snapshot of this status, refreshed on every progress check, is kept at [`../STATE-CURRENT.md`](../STATE-CURRENT.md). @@ -111,28 +111,30 @@ A dated snapshot of this status, refreshed on every progress check, is kept at | GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | | Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🟡 A stock guest program now reaches its kernel launch and submits it through the forwarding path; the front-end fetches the ring entry and executes methods, but the inline launch-descriptor (QMD) install has not been observed and no grid has dispatched yet. Context validity at dispatch is ruled out; the frontier is the launch method-stream form. Copy-engine-class work, its releases and its fault relay remain forwarded (parameters default off). See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | -| OpenGL / Vulkan | 🔴 Not yet | -| Windows guest | 🔴 Not yet | -| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | - -We also know exactly where the CUDA bring-up stands and what the next step is: -the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes -the request parameters instead of returning zeros), the engine inventory is -complete, and the driver-API smoke test now completes end to end — `cuCtxCreate` -returns and the 64-byte payload round-trips exactly (64/64, re-verified -2026-10-02 on the current build). The full -mechanism, with offsets and the decompiled call path, is in +| CUDA kernel execution | ✅ Working — verified guest clients execute kernels on the physical GPU with engine-authored completions relayed to the guest: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK` on fresh boots, and `vecadd_loop` passes 100 launches. Remaining: the staged-transfer class (some large multi-chunk clients still mis-verify; failure modes proven), the every-boot acceptance campaign, and per-guest execution isolation. See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | +| OpenGL (3D graphics) | 🟡 In progress — the X server and GLX run on the emulated vGPU with the stock driver; the engine executes real GL method streams (verbatim 3D replay and synthesized launches); a direct GL client creates its channels and its readback path returns clean. Rendered (non-black) frames are the frontier. See [`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md) | +| Vulkan | 🟡 In progress — the guest Vulkan pipeline runs end to end on the emulated vGPU (device, allocation, pipeline, command buffer, fence, queue submit, readback); frames are still black — program delivery and program-VA mapping are the frontier | +| Windows guest | 🔴 Not yet (target) | +| Multi-tenant isolation (mdev per VM) | 🟡 Two mediated devices with different emulated VRAM sizes and two guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | + +Guest CUDA now executes on the physical GPU: the driver-API path completes end +to end, forwarded launches reach the engine, the grid dispatches, the engine +authors the completion and the host relays it, and the guest's synchronisation +waits resolve (verified with vector-add, matrix-multiply and transformer +clients on fresh boots). The mechanism, with offsets and the decompiled call +path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). -The remaining frontier is guest-facing execution. Copy-engine-class copies and -their releases are forwarded to real hardware with release relay and fault -relay in place (parameters default off). For GR (graphics/compute), the guest -now reaches and submits a real kernel launch: the ring entry is fetched and -front-end methods execute, but the inline launch-descriptor install has not -been observed and no grid has dispatched yet. The wall is the form of the -forwarded launch method stream — not context validity, which is ruled out at -dispatch. The chain, the evidence and the live investigation lines are in -[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md). +The remaining CUDA work is breadth and hardening: the staged-transfer class +(some large multi-chunk clients still mis-verify; the failure modes are +proven), the every-boot acceptance campaign, and per-guest execution isolation +for simultaneous guests. Graphics shares the same submission machinery and is +the other active frontier: the X server and GLX run on the emulated vGPU, the +engine executes GL method streams, and the guest Vulkan pipeline runs end to +end — rendered (non-black) pixels are the wall. The chain, the evidence and the +live investigation lines are in +[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) +and +[`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md). The backend, a module-owned GPU running its real GSP firmware as the resource-manager server, is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md); @@ -184,6 +186,7 @@ Read in order if you are new. Jump directly if you know what you need. | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | | 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | +| 25b | [Real-render walls: Vulkan and Xorg on the emulated vGPU](04-implementation/04-real-render-walls.md) | developers, debuggers | 10 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | diff --git a/kernel/Makefile b/kernel/Makefile index da8f240..3bae9d7 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -8,6 +8,7 @@ vgsp-y := \ gsp_emulation.o \ gsp_rm_emulation.o \ gsp_sema_service.o \ + gsp_display_engine.o \ gsp_cuda_bridge.o \ gsp_rpc_client.o \ gsp_rpc_integration.o \ diff --git a/kernel/bar_emulation.c b/kernel/bar_emulation.c index d1702c8..0590112 100644 --- a/kernel/bar_emulation.c +++ b/kernel/bar_emulation.c @@ -177,6 +177,11 @@ void nv_vgpu_msix_clear_pending(struct nv_vgpu_device *vgpu, int vector) spin_unlock(&ctx->msix_lock); } +static bool nv_vgpu_disp_caps_seed; +module_param_named(disp_caps_seed, nv_vgpu_disp_caps_seed, bool, 0644); +MODULE_PARM_DESC(disp_caps_seed, + "Seed the NVC573 display caps window (BAR0+0x640000: head0|SOR0, windows 0-7) so the guest display subsystem selects the window path (default 0; A/B)"); + int nv_vgpu_bar_emul_init(struct nv_vgpu_device *vgpu) { struct nv_vgpu_bar_emul_ctx *ctx; @@ -228,6 +233,25 @@ int nv_vgpu_bar_emul_init(struct nv_vgpu_device *vgpu) *(u32 *)((u8 *)ctx->bar0_shadow + 0x000) = 0x164000a1; *(u32 *)((u8 *)ctx->bar0_shadow + 0x004) = 0x00000000; *(u32 *)((u8 *)ctx->bar0_shadow + 0xa00) = 0x164a1000; + /* + * SESSION 155 CONT 32: display caps window seed (A/B, default 0). + * The guest NVIDIA display driver's NV_PGRAPH/NV_PDISP SYS_CAP / + * SYS_CAPB reads at BAR0+0x640000/0x640004 return 0 on the + * emulated vGPU, so the display subsystem selection sees + * UsableWindowCount=0, takes the base-channel path and fails the + * class-list check (0x927c) -> "Failed to select a display + * subsystem". Seed the NVC573 caps window (head0|SOR0, + * windows 0-7) so the driver selects the window path. + */ + if (READ_ONCE(nv_vgpu_disp_caps_seed)) { + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640000) = 0x00000101; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640004) = 0x000000ff; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x6405e8) = 0x00000085; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640144) = 0x00000100; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640608) = 0x003c0000; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640680) = 0x0000021f; + *(u32 *)((u8 *)ctx->bar0_shadow + 0x640780) = 0x0000021f; + } /* * Pre-set the GSP secure-boot (GFW) completion state. diff --git a/kernel/bar_init.c b/kernel/bar_init.c index 6515613..5fa889a 100644 --- a/kernel/bar_init.c +++ b/kernel/bar_init.c @@ -8,6 +8,11 @@ #include "gsp_phys_gpu.h" #include "irq.h" +static bool nv_vgpu_bar1_full; +module_param_named(bar1_full, nv_vgpu_bar1_full, bool, 0644); +MODULE_PARM_DESC(bar1_full, + "Expose the full emulated FB size as the BAR1 aperture (default 0 = physical 256 MiB; A/B for the display instance-memory mapping)"); + int nv_vgpu_bar_init(struct nv_vgpu_device *vgpu) { struct pci_dev *pdev = vgpu->pdev; @@ -72,7 +77,19 @@ int nv_vgpu_bar_init(struct nv_vgpu_device *vgpu) * its BAR1 from the VFIO region info and vidmem bounds/mmap * math must match the shadow.) */ - vgpu->bars[1].size = bar_size; + /* + * A/B knob for the display instance-memory mapping: the RM + * addresses the 64 KB display instance block at the top of + * the emulated FB (0x1fff0000 with a 512 MiB FB). With the + * physical 256 MiB BAR1 aperture that address is not + * directly mappable, and the guest's instance-memory writes + * never reach the emulated vidmem shadow (r1633: notifier + * context-DMA handles unresolvable). bar1_full=1 exposes + * the whole emulated FB as BAR1 for this test; default 0 + * keeps the historical 256 MiB aperture (CUDA-safe). + */ + vgpu->bars[1].size = nv_vgpu_bar1_full ? + nv_vgpu_fb_emul_size(vgpu) : bar_size; /* Emulated vidmem: a dedicated host-RAM buffer backing the guest's * VRAM aperture. Do NOT map the host GPU's real BAR1 - guest VRAM * accesses must hit our own clean buffer, not the (broken) host diff --git a/kernel/gsp_cuda_bridge.h b/kernel/gsp_cuda_bridge.h index 2fb63d7..472fafb 100644 --- a/kernel/gsp_cuda_bridge.h +++ b/kernel/gsp_cuda_bridge.h @@ -15,7 +15,15 @@ struct nv_vgpu_device; * targeted resolve). */ #define GSP_BRIDGE_INLINE_MAX 16 -#define GSP_BRIDGE_INLINE_BYTES 64 +/* + * SESSION 153-CONTINUED-88: 256 -> 2048. The GL program delivery needs the + * driver's REAL captured upload payloads (observed: a 1536 B SPH+SASS image + * at 0x4350900 and a 1408 B image at 0x4350f00, w0=0x020a0481) to reach the + * engine-visible page; the 256 B cap truncated them. Every user of this + * record type grows with it (rings, bridge queues - all static storage; the + * only stack user, svc_gl_flush's prs[], was moved to static storage). + */ +#define GSP_BRIDGE_INLINE_BYTES 2048 struct gsp_bridge_inline { u64 dst_va; diff --git a/kernel/gsp_display_engine.c b/kernel/gsp_display_engine.c new file mode 100644 index 0000000..98f85fe --- /dev/null +++ b/kernel/gsp_display_engine.c @@ -0,0 +1,1597 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * gsp_display_engine.c - minimal host-side display (EVO) engine emulation. + * + * The guest's nvidia-modeset drives the display through per-channel push + * buffers: it fills a 4 KB sysmem ring with NVC57D/NVC57E methods and kicks + * the channel by writing the PUT register of the channel's BAR0 UDISP + * control window. The hardware display engine consumes the methods, advances + * GET, and completes an update by writing the NV_DISP_NOTIFIER status + * (FINISHED) to the notifier surface the driver bound to the channel. The + * driver's nvEvoWaitForCompNotifier() polls that word; without it every mode + * set waits 3 s for the lost-notification WAR (and flip events never fire). + * + * This file implements the engine contract from the host side: + * + * 1. channel registration: the kernel RM tells GSP-RM about each display + * channel's push buffer through the internal control + * NV2080_CTRL_CMD_INTERNAL_DISPLAY_CHANNEL_PUSHBUFFER (0x20800a58): + * {addressSpace, physicalAddr, limit, cacheSnoop, hclass, + * channelInstance, valid, pbTargetAperture, channelPBSize, + * subDeviceId}. The emulator records it here. + * + * 2. PUT polling + GET emulation: PUT (offset 0x0) and GET (0x4) live in + * the channel's control page in the BAR0 shadow (UDISP windows: core + * 0x680000, window n 0x690000+n*0x1000, cursor n 0x6D8000+n*0x1000 - + * kdispGetDisplayChannelUserBaseAndSize_v03_00 + + * kdispGetBaseOffset on TU104). The guest's mmap of BAR0 is the + * emulated shadow, so its PUT write is visible to the service thread + * and our GET write is visible to the guest (nvEvoMakeRoom / + * EvoCoreReadGet). + * + * 3. the push-buffer parser: NV DMA opcodes METHOD(0)/JUMP(1)/ + * NONINC(2)/SET_SUBDEVICE_MASK(3)/ONE_INC(5). The driver wraps the + * ring by writing a JUMP-to-zero (0x20000000) at offset_max and then + * kicking PUT=0; the parser follows that. + * + * 4. completion: on UPDATE (0x200) with the notifier armed + * (SET_CONTEXT_DMA_NOTIFIER 0x208/0x21c and SET_NOTIFIER_CONTROL + * 0x20c/0x220 written earlier in the stream), write the + * NV_DISP_NOTIFIER status word 0x80000000 (STATUS=FINISHED at 31:30) + * to the notifier surface and post the completion event on the + * channel's RM object so the driver's flip/wait callback runs. + * + * NOTIFIER ADDRESS RESOLUTION (exact, not a heuristic): the guest's display + * instance memory lives in FB (ADDR_FBMEM, memmgrGetRsvdMemoryBase + 0) and + * is CPU-written by the kernel RM. Its layout is 8 KB hash table followed + * by 32-byte context-DMA entries (NV_UDISP_HASH_BASE/OBJ_MEM_BASE, NV_DMA_SIZE + * 20, DISPLAY_CONTEXT_DMA_INST_SIZE 2*16). A hash entry is + * {ht_ObjectHandle = ctx DMA handle, ht_Context = client[13:0] | + * instance_chunk[24:14] | dispChannelNum[31:25]}. The push buffer's + * SET_CONTEXT_DMA_NOTIFIER carries that ctx DMA handle, so the engine locates + * the hash entry with that handle, reads the instance entry (base/limit at + * words 1..4, address in 256-byte units) and gets the exact notifier PA. + * No guess can corrupt an unrelated guest page; when the lookup fails the + * engine does not write (the driver's 3 s lost-notification WAR still + * completes the wait). + * + * Session r1621; see Handoff-vGPU-IO.md and + * backups/evidence-20261002-victory/r1621/. + */ + +#define pr_fmt(fmt) "vgsp-disp: " fmt + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/spinlock.h> +#include <linux/jiffies.h> +#include <linux/ktime.h> +#include <linux/vfio.h> +#include <linux/delay.h> + +#include "nvidia-vgpu-vfio.h" +#include "bar_emulation.h" +#include "gsp_display_engine.h" +#include "gsp_rm_emulation.h" +#include "gsp_sema_service.h" + +#define GSP_DISP_MAX_CHANNELS 24 +#define GSP_DISP_MAX_NOTES 24 + +/* BAR0 UDISP channel-control windows (TU104 / DISP v04.00 layout). */ +#define GSP_DISP_CORE_CTL_BASE 0x680000 +#define GSP_DISP_WIN_CTL_BASE 0x690000 +#define GSP_DISP_CURS_CTL_BASE 0x6D8000 + +/* NVC57D/NVC57E control-page register offsets. */ +#define GSP_DISP_REG_PUT 0x000 +#define GSP_DISP_REG_GET 0x004 +#define GSP_DISP_REG_GET_LINE 0x208 + +/* Display method offsets (NVC57D core / NVC57E window). */ +#define GSP_DISP_M_UPDATE 0x200 +#define GSP_DISP_M_CORE_SET_NOTIFIER_DMA 0x208 +#define GSP_DISP_M_CORE_SET_NOTIFIER_CTRL 0x20c +#define GSP_DISP_M_WIN_SET_NOTIFIER_DMA 0x21c +#define GSP_DISP_M_WIN_SET_NOTIFIER_CTRL 0x220 + +/* NV_DISP_NOTIFIER__0_STATUS FINISHED lives at 31:30. */ +#define GSP_DISP_NOTIFIER_FINISHED 0x80000000u +#define GSP_DISP_NOTIFIER_CTRL_NOTIFY BIT(12) +/* The CONTROL offset field is in 16-byte units; expose it in bytes. */ +#define GSP_DISP_NOTIFIER_CTRL_OFFSET(c) ((((c) >> 4) & 0xff) * 16) +/* Coalesce a refused completion wake-up, then re-post it. Must be at least + * the RM event hint dedup window (HZ / 4 = 250 ms) so the retry is not + * coalesced into the same post that starved it; at most four attempts/s per + * waiting channel. */ +#define GSP_DISP_EVENT_RETRY_MS 250 + +/* Display instance memory (v03_00 layout). */ +#define GSP_DISP_INST_HASH_OFF 0x0000 +#define GSP_DISP_INST_HASH_END 0x2000 +#define GSP_DISP_INST_OBJ_OFF 0x2000 +#define GSP_DISP_INST_OBJ_END 0x10000 +#define GSP_DISP_INST_ENTRY_SIZE 32 + +/* Raster timing used for the NVC57E_GET_LINE emulation (1920x1080@60). */ +#define GSP_DISP_VBLANK_LINES 1125 +#define GSP_DISP_ACTIVE_LINES 1080 +#define GSP_DISP_FRAME_NS 16666667ULL + +struct gsp_disp_chan { + struct nv_vgpu_device *vgpu; + u32 hclass; + u32 instance; + u32 h_object; /* RM channel object (event source) */ + u64 pb_pa; + u32 limit; /* bytes-1 as reported by 0xa58 */ + u32 address_space; + u32 ctl_off; /* BAR0 UDISP control window */ + u32 consumed; /* next push-buffer offset to execute */ + u32 last_put; /* for change detection / logs */ + bool valid; + + /* Completion-notifier programming captured from the push buffer. */ + u32 notifier_handle; + u32 notifier_ctrl; + u64 notifier_pa; + bool notifier_pa_valid; + u8 notifier_target; /* instance-table target: 1 FB, 2/3 sysmem */ + u32 notifier_writes; + u32 notifier_posts; + bool event_pending; /* completion event send failed; retry in pump */ + unsigned long event_retry_at; /* next forced re-post attempt */ + u32 event_drop_logs; + u32 update_logs; + u32 ctrl_logs; + + /* Stats. */ + u64 updates; + u64 methods; + u64 wraps; + bool logged_update; + unsigned long last_dbg; +}; + +struct gsp_disp_note { + struct nv_vgpu_device *vgpu; + u32 hclass; + u32 instance; + u32 h_object; +}; + +static struct { + spinlock_t lock; + struct gsp_disp_chan ch[GSP_DISP_MAX_CHANNELS]; + struct gsp_disp_note note[GSP_DISP_MAX_NOTES]; + u8 *pbuf; /* 4 KB push-buffer bounce (service thread only) */ + u8 *ibuf; /* 64 KB display instance-memory bounce */ + unsigned long boot_jiffies; + u32 log_budget; + u32 push_dumps; /* bounded raw push dumps (disp_dbg) */ + u32 im_logs; /* bounded instance-memory diagnostics */ + /* Cached display-instance-memory base per guest (<= 2 guests). */ + struct { + struct nv_vgpu_device *vgpu; + u64 base; + u64 pa; /* instMemPhysAddr from 0x20800a49 */ + u64 size; + u32 addr_space; + int found; /* 1 found, 0 not yet tried, -1 given up */ + /* Recent failed handle scans (do not rescan the world). */ + struct { + u32 handle; + unsigned long until; + } nf[4]; + } im[2]; +} disp; + +/* + * A/B and discovery knobs. disp_engine=0 removes every effect (registration + * is a no-op, the pump does nothing), so the previous behaviour is one module + * load away. + */ +static unsigned gsp_disp_engine = 1; +module_param_named(disp_engine, gsp_disp_engine, uint, 0644); +MODULE_PARM_DESC(disp_engine, "Emulate the display engine (PUT/GET, push-buffer parser, notifier, events; default 1)"); + +static unsigned gsp_disp_dbg; +module_param_named(disp_dbg, gsp_disp_dbg, uint, 0644); +MODULE_PARM_DESC(disp_dbg, "Verbose display-engine diagnostics (default 0)"); + +static unsigned gsp_disp_finish = 1; +module_param_named(disp_finish, gsp_disp_finish, uint, 0644); +MODULE_PARM_DESC(disp_finish, "Write FINISHED to the notifier on UPDATE (default 1)"); + +static unsigned gsp_disp_vblank = 1; +module_param_named(disp_vblank, gsp_disp_vblank, uint, 0644); +MODULE_PARM_DESC(disp_vblank, "Emulate the NVC57E_GET_LINE raster counter (default 1)"); + +static unsigned gsp_disp_events = 1; +module_param_named(disp_events, gsp_disp_events, uint, 0644); +MODULE_PARM_DESC(disp_events, "Post the channel completion event on an armed UPDATE (default 1)"); + +/* + * r1765: strict display-event delivery. Default 0 keeps the legacy + * behaviour bit-for-bit (full A/B revert); when set, an armed UPDATE posts + * only to the events registered on that channel (no all-events fallback), + * and unrelated completions/the 0.5 s hint skip display-channel events + * (see gsp_disp_engine_owns_object()). Live A/B: echo 1 > + * /sys/module/vgsp/parameters/disp_events_strict. + */ +static unsigned gsp_disp_events_strict = 0; +module_param_named(disp_events_strict, gsp_disp_events_strict, uint, 0644); +MODULE_PARM_DESC(disp_events_strict, + "Deliver a display UPDATE completion only to events registered on that channel, no all-events fallback (default 0 = legacy; 1 = r1765 strict)"); + +static unsigned long long gsp_disp_notifier_pa; +module_param_named(disp_notifier_pa, gsp_disp_notifier_pa, ullong, 0644); +MODULE_PARM_DESC(disp_notifier_pa, "Explicit notifier surface guest PA override (0 = resolve from the display instance table)"); + +static unsigned gsp_disp_dump; +module_param_named(disp_dump, gsp_disp_dump, uint, 0644); +MODULE_PARM_DESC(disp_dump, "Write 1 to dump the display-engine channel table to dmesg (self-clearing)"); + +static int gsp_disp_read(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, + size_t len) +{ + return vfio_dma_rw(&vgpu->vfio_dev, gpa, buf, len, false); +} + +static int gsp_disp_write(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, + size_t len) +{ + return vfio_dma_rw(&vgpu->vfio_dev, gpa, buf, len, true); +} + +/* + * A notifier whose context-DMA object targets ADDR_FBMEM (1) lives in the + * emulated FB shadow; targets 2/3 are guest sysmem. The instance table + * stores vidmem offsets, but accept a physical FB address by rebasing it + * on vgpu->fb_base when the direct interpretation is out of range. + */ +static bool gsp_disp_fb_off(struct nv_vgpu_device *vgpu, u64 pa, size_t len, + u64 *off) +{ + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + + if (!vgpu->bars[1].vaddr) + return false; + if (pa + len <= fb_size) { + *off = pa; + return true; + } + if (vgpu->fb_base && pa >= vgpu->fb_base && + pa - vgpu->fb_base + len <= fb_size) { + *off = pa - vgpu->fb_base; + return true; + } + return false; +} + +static int gsp_disp_notifier_read(struct nv_vgpu_device *vgpu, u64 pa, + u32 target, void *buf, size_t len) +{ + u64 off; + + if (target == 1) { + if (!gsp_disp_fb_off(vgpu, pa, len, &off)) + return -ENOENT; + memcpy(buf, (u8 *)vgpu->bars[1].vaddr + off, len); + return 0; + } + return gsp_disp_read(vgpu, pa, buf, len); +} + +static int gsp_disp_notifier_write(struct nv_vgpu_device *vgpu, u64 pa, + u32 target, void *buf, size_t len) +{ + u64 off; + + if (target == 1) { + if (!gsp_disp_fb_off(vgpu, pa, len, &off)) + return -ENOENT; + memcpy((u8 *)vgpu->bars[1].vaddr + off, buf, len); + return 0; + } + return gsp_disp_write(vgpu, pa, buf, len); +} + +/* BAR0 shadow access. The guest mmap writes it without our lock, so the + * register words are read with READ_ONCE; writes are single u32 stores. */ +static u32 gsp_disp_bar0_read(struct nv_vgpu_device *vgpu, u32 off) +{ + struct nv_vgpu_bar_emul_ctx *bctx = vgpu->bar_emul_ctx; + u32 v = 0; + + if (!bctx || !bctx->bar0_shadow || off + 4 > bctx->bar0_shadow_size) + return 0; + v = READ_ONCE(*(u32 *)((u8 *)bctx->bar0_shadow + off)); + return v; +} + +static void gsp_disp_bar0_write(struct nv_vgpu_device *vgpu, u32 off, u32 v) +{ + struct nv_vgpu_bar_emul_ctx *bctx = vgpu->bar_emul_ctx; + unsigned long flags; + + if (!bctx || !bctx->bar0_shadow || off + 4 > bctx->bar0_shadow_size) + return; + spin_lock_irqsave(&bctx->shadow_lock, flags); + *(u32 *)((u8 *)bctx->bar0_shadow + off) = v; + spin_unlock_irqrestore(&bctx->shadow_lock, flags); +} + +static u32 gsp_disp_ctl_offset(u32 hclass, u32 instance) +{ + switch (hclass) { + case GSP_DISP_CLASS_CORE: + return GSP_DISP_CORE_CTL_BASE; + case GSP_DISP_CLASS_WINDOW: + return GSP_DISP_WIN_CTL_BASE + instance * 0x1000; + case GSP_DISP_CLASS_WINDOW_IMM: + /* + * Immediate-mode window channels sit 32 windows above the + * ordinary window block (0x690000 + (instance+32)*0x1000 + * = 0x6d0000 + instance*0x1000). The old mapping put them + * in the cursor-PIO block at 0x6d8000, so their PUT was + * never seen and their flips never completed (r1638). + */ + return GSP_DISP_WIN_CTL_BASE + (instance + 32) * 0x1000; + case GSP_DISP_CLASS_CURSOR_PIO: + return GSP_DISP_CURS_CTL_BASE + instance * 0x1000; + default: + return 0; + } +} + +static bool gsp_disp_pa_is_push_buffer(u64 pa) +{ + int i; + + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + if (disp.ch[i].valid && disp.ch[i].pb_pa == pa) + return true; + } + return false; +} + +/* ------------------------------------------------------------------ */ +/* Display instance memory (FB): hash table + context-DMA entries */ +/* ------------------------------------------------------------------ */ + +/* + * Locate the display instance-memory block by finding a hash entry that + * names ctx DMA handle @handle. The instance memory lives in FB + * (ADDR_FBMEM, base NV_UDISP_HASH_BASE=0, 64 KB), so walk the emulated FB + * shadow in 64 KB blocks and inspect each block's 8 KB hash table. On + * success the block base is cached; a handle lookup then needs only 1024 + * 8-byte reads. + */ +static int gsp_disp_instmem_find(struct nv_vgpu_device *vgpu, u32 handle) +{ + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + u8 *fb = vgpu->bars[1].vaddr; + u64 blk; + int slot = -1; + int i; + + if (!fb || !handle) + return -ENODEV; + + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + if (disp.im[i].vgpu == vgpu) { + slot = i; + break; + } + if (!disp.im[i].vgpu && slot < 0) + slot = i; + } + if (slot < 0) + return -ENOSPC; + disp.im[slot].vgpu = vgpu; + + for (blk = 0; blk + GSP_DISP_INST_OBJ_END <= fb_size; blk += 0x10000) { + u64 e; + + for (e = GSP_DISP_INST_HASH_OFF; + e < GSP_DISP_INST_HASH_END; e += 8) { + u32 obj = *(u32 *)(fb + blk + e); + u32 ctx = *(u32 *)(fb + blk + e + 4); + u32 inst = (ctx >> 14) & 0x7ff; + u32 chn = (ctx >> 25) & 0x7f; + u64 ent; + + if (obj != handle) + continue; + if (chn > 80) + continue; + if (inst < (GSP_DISP_INST_OBJ_OFF / 0x20) || + inst >= (GSP_DISP_INST_OBJ_END / 0x20)) + continue; + ent = blk + (u64)inst * GSP_DISP_INST_ENTRY_SIZE; + { + u32 w0 = *(u32 *)(fb + ent + 0); + u32 w1 = *(u32 *)(fb + ent + 4); + u32 w3 = *(u32 *)(fb + ent + 12); + u64 base = ((u64)w1) << 8; + + if ((w0 & 0x3) >= 1 && (w0 & 0x3) <= 3 && + (w0 & BIT(2)) && base > 0x1000 && + (w3 != 0)) { + disp.im[slot].base = blk; + disp.im[slot].found = 1; + if (gsp_disp_dbg) + pr_info("instance memory at FB+0x%llx (hash entry 0x%llx, handle 0x%x, chn %u)\n", + blk, e, handle, chn); + return 0; + } + } + } + } + disp.im[slot].found = -1; + return -ENOENT; +} + +/* + * Read from the display instance memory: ADDR_FBMEM (2) lives in the + * emulated BAR1 shadow; ADDR_SYSMEM (1) is guest physical memory. + */ +static int gsp_disp_im_read(struct nv_vgpu_device *vgpu, u64 pa, + u32 addr_space, u64 off, void *buf, size_t len) +{ + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + u8 *fb = vgpu->bars[1].vaddr; + + if (addr_space == 2) { /* ADDR_FBMEM */ + if (!fb || pa + off + len > fb_size) + return -ENOENT; + memcpy(buf, fb + pa + off, len); + return 0; + } + return vfio_dma_rw(&vgpu->vfio_dev, pa + off, buf, len, false); +} + +/* + * NV2080_CTRL_CMD_INTERNAL_DISPLAY_WRITE_INST_MEM (0x20800a49) announced the + * display instance memory location. The notifier context-DMA handles resolve + * through this block's 8 KB hash table, so record it per guest. + */ +void gsp_disp_engine_inst_mem(struct nv_vgpu_device *vgpu, u64 pa, u64 size, + u32 addr_space) +{ + unsigned long flags; + int i, slot = -1; + + if (!gsp_disp_engine || !vgpu || !pa || !size) + return; + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + if (disp.im[i].vgpu == vgpu) { + slot = i; + break; + } + if (!disp.im[i].vgpu && slot < 0) + slot = i; + } + if (slot >= 0) { + disp.im[slot].vgpu = vgpu; + disp.im[slot].pa = pa; + disp.im[slot].size = size; + disp.im[slot].addr_space = addr_space; + disp.im[slot].base = 0; + disp.im[slot].found = 0; + } + spin_unlock_irqrestore(&disp.lock, flags); + pr_info("inst mem: pa=0x%llx size=0x%llx addrSpace=%u\n", pa, size, + addr_space); +} + +/* + * Parse one 64 KB display instance block: 8 KB context-DMA hash table + * (word0 = ht_ObjectHandle, word1 = {chn[31:25], instance[24:14], + * client[13:0]}) + 32-byte object entries at instance*32 holding + * {target, base_lo, base_hi, limit_lo, limit_hi} in 256-byte units. + * *limit is the byte END address, matching the table encoding. + */ +static int gsp_disp_parse_hash(u32 handle, const u8 *buf, u64 *pa, + u64 *limit, u32 *target) +{ + u64 e; + + for (e = 0; e < GSP_DISP_INST_HASH_END; e += 8) { + u32 obj, ctx, inst, chn; + u64 ent; + u32 w0, w1, w2, w3, w4; + + memcpy(&obj, buf + e, 4); + if (obj != handle) + continue; + memcpy(&ctx, buf + e + 4, 4); + inst = (ctx >> 14) & 0x7ff; + chn = (ctx >> 25) & 0x7f; + if (chn > 80) + continue; + if (inst < (GSP_DISP_INST_OBJ_OFF / 0x20) || + inst >= (GSP_DISP_INST_OBJ_END / 0x20)) + continue; + ent = (u64)inst * GSP_DISP_INST_ENTRY_SIZE; + if (ent + GSP_DISP_INST_ENTRY_SIZE > GSP_DISP_INST_OBJ_END) + continue; + memcpy(&w0, buf + ent + 0, 4); + memcpy(&w1, buf + ent + 4, 4); + memcpy(&w2, buf + ent + 8, 4); + memcpy(&w3, buf + ent + 12, 4); + memcpy(&w4, buf + ent + 16, 4); + if ((w0 & 0x3) < 1 || (w0 & 0x3) > 3 || !(w0 & BIT(2))) + continue; + *pa = ((((u64)(w2 & 0x7f)) << 32) | w1) << 8; + *limit = ((((u64)(w4 & 0x7f)) << 32) | w3) << 8; + *target = w0 & 0x3; + return 0; + } + return -ENOENT; +} + +/* + * The BAR1 aperture is smaller than the FB (256 MB window over 512 MB on + * this vGPU), so the RM writes vidmem through a shifted BAR1 block and the + * announced instance PA does not index the shadow directly. Find the block + * by searching the shadow for the ctxDMA handle itself. + */ +static int gsp_disp_find_block_fb(struct nv_vgpu_device *vgpu, u32 handle, + u64 *base, u64 *pa, u64 *limit, + u32 *target) +{ + u8 *fb = vgpu->bars[1].vaddr; + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + u64 off; + + if (!fb) + return -ENODEV; + for (off = 0; off + GSP_DISP_INST_OBJ_END <= fb_size; off += 8) { + u32 a; + + memcpy(&a, fb + off, 4); + if (a != handle) + continue; + if (!gsp_disp_parse_hash(handle, fb + (off & ~0xffffULL), + pa, limit, target)) { + *base = off & ~0xffffULL; + return 0; + } + } + return -ENOENT; +} + +/* Same search through guest RAM (the coherent-mapping fallback). */ +static int gsp_disp_find_block_ram(struct nv_vgpu_device *vgpu, u32 handle, + u64 *base, u64 *pa, u64 *limit, + u32 *target) +{ + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + const size_t chunk = 1 << 20; + u8 *buf = kmalloc(chunk, GFP_KERNEL); + u64 off; + int ret = -ENOENT; + + if (!buf) + return -ENOMEM; + for (off = 0; off + chunk <= fb_size; off += chunk) { + size_t i; + + if (vfio_dma_rw(&vgpu->vfio_dev, off, buf, chunk, false)) + continue; + for (i = 0; i + 8 <= chunk; i += 8) { + u32 a; + + memcpy(&a, buf + i, 4); + if (a != handle) + continue; + if (!gsp_disp_parse_hash(handle, + buf + (i & ~0xffffULL), + pa, limit, target)) { + *base = off + (i & ~0xffffULL); + ret = 0; + goto out; + } + } + } +out: + kfree(buf); + return ret; +} + +/* + * The guest RM writes the display instance block through its BAR2 aperture + * (memdescMapInternal -> BAR2), which vgsp exposes as a flat host-RAM + * shadow. BAR2 offsets are only 4 KiB-aligned, so locate the hash entry by + * handle and derive the block base from its position: the table is the + * first 8 KiB of the block, hence the entry lies in the 4 KiB page at + * block+0 or block+0x1000. + */ +static int gsp_disp_find_block_bar2(struct nv_vgpu_device *vgpu, u32 handle, + u64 *base, u64 *pa, u64 *limit, + u32 *target) +{ + int bar; + + /* + * The 32 MiB RM BAR2 aperture is the 64-bit PCI BAR2/3 pair; on + * this platform it is enumerated as resource BAR3 (BAR2 is empty + * or a low dword), and r1635 proved the instance block lives in + * the bars[3] shadow (offset 0x3000). Scan both shadows. + */ + for (bar = 2; bar <= 3; bar++) { + u8 *bar2 = vgpu->bars[bar].vaddr; + u64 size = vgpu->bars[bar].size; + u64 s; + + if (!bar2 || !size) + continue; + for (s = 0; s + 8 <= size; s += 8) { + u32 obj, ctx, inst; + u64 cand[2]; + int c; + + memcpy(&obj, bar2 + s, 4); + if (obj != handle) + continue; + memcpy(&ctx, bar2 + s + 4, 4); + inst = (ctx >> 14) & 0x7ff; + if (inst < (GSP_DISP_INST_OBJ_OFF / 0x20) || + inst >= (GSP_DISP_INST_OBJ_END / 0x20)) + continue; + + cand[0] = s & ~0xfffULL; + cand[1] = cand[0] - 0x1000; + for (c = 0; c < 2; c++) { + u64 b = cand[c]; + u64 ent, base64, lim64; + u32 w0, w1, w2, w3, w4, tgt; + + if (s < b || s - b >= 0x2000) + continue; + ent = b + (u64)inst * GSP_DISP_INST_ENTRY_SIZE; + if (ent + 5 * 4 > size) + continue; + memcpy(&w0, bar2 + ent + 0, 4); + memcpy(&w1, bar2 + ent + 4, 4); + memcpy(&w2, bar2 + ent + 8, 4); + memcpy(&w3, bar2 + ent + 12, 4); + memcpy(&w4, bar2 + ent + 16, 4); + tgt = w0 & 0x3; + if (tgt < 1 || tgt > 3) + continue; + base64 = ((((u64)(w2 & 0x7f)) << 32) | w1) << 8; + lim64 = ((((u64)(w4 & 0x7f)) << 32) | w3) << 8; + if (lim64 <= base64 || lim64 - base64 > (1 << 20)) + continue; + *base = b; + *pa = base64; + *limit = lim64; + *target = tgt; + if (disp.im_logs < 8) { + disp.im_logs++; + pr_info("inst mem: handle 0x%x via bar%d block 0x%llx -> base=0x%llx limit=0x%llx target=%u\n", + handle, bar, (unsigned long long)b, + (unsigned long long)base64, + (unsigned long long)lim64, tgt); + } + return 0; + } + } + } /* bar */ + return -ENOENT; +} + +/* + * Resolve a context-DMA handle to its surface guest PA. Prefers the block + * announced by 0x20800a49; if the handle is not there (BAR1 aperture shift + * / coherent mapping / BAR2 write path), searches the shadow, then guest + * RAM, then the BAR2 shadow for the block that actually contains it. Falls + * back to the legacy whole-FB scan when the announce control was never seen. + * Returns 0 on success and fills *pa, *limit (byte end address) and + * *target (1 = FB, 2/3 = guest sysmem). + */ +static int gsp_disp_ctxdma_pa(struct nv_vgpu_device *vgpu, u32 handle, + u64 *pa, u64 *limit, u32 *target) +{ + u64 fb_size = nv_vgpu_fb_emul_size(vgpu); + u8 *fb = vgpu->bars[1].vaddr; + unsigned long flags; + u64 im_pa = 0; + u32 im_space = 0; + bool have_im = false; + u64 base = 0; + bool found = false; + int slot = -1; + int i; + + if (!handle) + return -ENODEV; + if (gsp_disp_notifier_pa) { + *pa = gsp_disp_notifier_pa; + *limit = *pa + 0x1000; + *target = 2; + return 0; + } + + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + if (disp.im[i].vgpu == vgpu && disp.im[i].pa && + disp.im[i].size) { + im_pa = disp.im[i].pa; + im_space = disp.im[i].addr_space; + have_im = true; + break; + } + } + spin_unlock_irqrestore(&disp.lock, flags); + + if (have_im) { + u64 base2 = 0; + const char *src = "announce"; + + if (!disp.ibuf) + return -ENOMEM; + if (gsp_disp_im_read(vgpu, im_pa, im_space, 0, disp.ibuf, + GSP_DISP_INST_OBJ_END)) + return -ENOENT; + + if (!gsp_disp_parse_hash(handle, disp.ibuf, pa, limit, target)) { + if (gsp_disp_dbg && disp.im_logs < 8) { + disp.im_logs++; + pr_info("inst mem: handle 0x%x at announced pa=0x%llx space=%u\n", + handle, im_pa, im_space); + } + return 0; + } + + /* Skip the expensive scans if this handle failed recently. */ + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + int k; + + if (disp.im[i].vgpu != vgpu) + continue; + for (k = 0; k < ARRAY_SIZE(disp.im[i].nf); k++) { + if (disp.im[i].nf[k].handle == handle && + time_before(jiffies, + disp.im[i].nf[k].until)) { + spin_unlock_irqrestore(&disp.lock, + flags); + return -ENOENT; + } + } + break; + } + spin_unlock_irqrestore(&disp.lock, flags); + + if (!gsp_disp_find_block_fb(vgpu, handle, &base2, pa, limit, + target)) { + src = "shadow"; + } else if (!gsp_disp_find_block_ram(vgpu, handle, &base2, pa, + limit, target)) { + src = "ram"; + } else if (!gsp_disp_find_block_bar2(vgpu, handle, &base2, pa, + limit, target)) { + /* The BAR2 scan emits its own single log line. */ + return 0; + } else { + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + int k; + + if (disp.im[i].vgpu != vgpu) + continue; + for (k = 0; k < ARRAY_SIZE(disp.im[i].nf); k++) { + unsigned long until = + disp.im[i].nf[k].until; + + if (disp.im[i].nf[k].handle == handle || + time_after_eq(jiffies, until)) { + disp.im[i].nf[k].handle = handle; + disp.im[i].nf[k].until = + jiffies + 5 * HZ; + break; + } + } + break; + } + spin_unlock_irqrestore(&disp.lock, flags); + if (gsp_disp_dbg && disp.im_logs < 8) { + disp.im_logs++; + pr_info("inst mem: handle 0x%x not in any block (announced pa=0x%llx space=%u)\n", + handle, im_pa, im_space); + } + return -ENOENT; + } + if (gsp_disp_dbg && disp.im_logs < 8) { + disp.im_logs++; + pr_info("inst mem: handle 0x%x via %s block 0x%llx -> pa=0x%llx limit=0x%llx target=%u\n", + handle, src, base2, *pa, *limit, *target); + } + return 0; + } + + /* Legacy: locate the block by scanning the FB shadow. */ + if (!fb) + return -ENODEV; + for (i = 0; i < ARRAY_SIZE(disp.im); i++) { + if (disp.im[i].vgpu == vgpu) { + slot = i; + break; + } + } + if (slot < 0 || !disp.im[slot].found) { + if (gsp_disp_instmem_find(vgpu, handle)) { + /* + * The guest may write the instance block through its + * BAR2 aperture instead of the FB shadow. + */ + if (!gsp_disp_find_block_bar2(vgpu, handle, &base, pa, + limit, target)) + return 0; + return -ENOENT; + } + slot = -1; + for (i = 0; i < ARRAY_SIZE(disp.im); i++) + if (disp.im[i].vgpu == vgpu) { + slot = i; + break; + } + if (slot < 0) + return -ENOENT; + } + base = disp.im[slot].base; + if (!base || base + GSP_DISP_INST_OBJ_END > fb_size) + return -ENOENT; + + for (i = 0; i < (GSP_DISP_INST_HASH_END - GSP_DISP_INST_HASH_OFF); + i += 8) { + u64 e = base + i; + u32 obj = *(u32 *)(fb + e); + u32 ctx = *(u32 *)(fb + e + 4); + u32 inst = (ctx >> 14) & 0x7ff; + u64 ent; + u32 w0, w1, w2, w3, w4; + + if (obj != handle) + continue; + if (inst < (GSP_DISP_INST_OBJ_OFF / 0x20) || + inst >= (GSP_DISP_INST_OBJ_END / 0x20)) + continue; + ent = base + (u64)inst * GSP_DISP_INST_ENTRY_SIZE; + w0 = *(u32 *)(fb + ent + 0); + w1 = *(u32 *)(fb + ent + 4); + w2 = *(u32 *)(fb + ent + 8); + w3 = *(u32 *)(fb + ent + 12); + w4 = *(u32 *)(fb + ent + 16); + if ((w0 & 0x3) < 1 || (w0 & 0x3) > 3 || !(w0 & BIT(2))) + continue; + *pa = ((((u64)(w2 & 0x7f)) << 32) | w1) << 8; + *limit = ((((u64)(w4 & 0x7f)) << 32) | w3) << 8; + *target = w0 & 0x3; + found = true; + break; + } + + if (!found) + return -ENOENT; + return 0; +} + +void gsp_disp_engine_note_object(struct nv_vgpu_device *vgpu, u32 hclass, + u32 instance, u32 h_object) +{ + unsigned long flags; + int i, slot = -1; + + if (!gsp_disp_engine || !vgpu || !h_object) + return; + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_NOTES; i++) { + if (disp.note[i].vgpu == vgpu && disp.note[i].hclass == hclass && + disp.note[i].instance == instance) { + slot = i; + break; + } + } + if (slot < 0) + for (i = 0; i < GSP_DISP_MAX_NOTES; i++) + if (!disp.note[i].vgpu) { + slot = i; + break; + } + if (slot >= 0) { + disp.note[slot].vgpu = vgpu; + disp.note[slot].hclass = hclass; + disp.note[slot].instance = instance; + disp.note[slot].h_object = h_object; + + /* + * The push-buffer control arrives BEFORE the alloc (observed + * ordering: control, registration, alloc), so a channel + * registered from the control has h_object=0 and the + * completion-event post in gsp_disp_method() would stay + * inert. Backfill it when the alloc lands. + */ + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + struct gsp_disp_chan *c = &disp.ch[i]; + + if (c->valid && c->vgpu == vgpu && + c->hclass == hclass && c->instance == instance) { + c->h_object = h_object; + break; + } + } + } + spin_unlock_irqrestore(&disp.lock, flags); +} + +/* + * Caller must hold disp.lock. (The unlocked form exists because + * gsp_disp_engine_channel_add() already holds the lock; calling a + * self-locking helper there deadlocked the RPC thread on the first + * display push-buffer control - r1621c.) + */ +static u32 gsp_disp_find_object_locked(struct nv_vgpu_device *vgpu, u32 hclass, + u32 instance) +{ + u32 h_object = 0; + int i; + + for (i = 0; i < GSP_DISP_MAX_NOTES; i++) { + if (disp.note[i].vgpu == vgpu && + disp.note[i].hclass == hclass && + disp.note[i].instance == instance) { + h_object = disp.note[i].h_object; + break; + } + } + return h_object; +} + +void gsp_disp_engine_channel_add(struct nv_vgpu_device *vgpu, u32 hclass, + u32 instance, u64 pb_pa, u64 limit, + u32 address_space, bool valid) +{ + struct gsp_disp_chan *ch = NULL; + unsigned long flags; + int i; + + if (!gsp_disp_engine || !vgpu) + return; + + if (!valid || !pb_pa || !gsp_disp_ctl_offset(hclass, instance)) { + /* + * Always log a rejected registration: this is the one line + * that says whether the guest programmed a channel the + * engine could not take (PIO channel, zero PA, unknown + * class). It is boot-only volume. + */ + pr_info("channel skipped: class=0x%x inst=%u pa=0x%llx limit=0x%llx valid=%d\n", + hclass, instance, pb_pa, limit, valid); + return; + } + + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + struct gsp_disp_chan *c = &disp.ch[i]; + + if (c->valid && c->vgpu == vgpu && c->hclass == hclass && + c->instance == instance) { + ch = c; + break; + } + } + if (!ch) { + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + if (!disp.ch[i].valid) { + ch = &disp.ch[i]; + break; + } + } + } + if (!ch) { + spin_unlock_irqrestore(&disp.lock, flags); + pr_warn("channel table full; class=0x%x inst=%u not emulated\n", + hclass, instance); + return; + } + + memset(ch, 0, sizeof(*ch)); + ch->vgpu = vgpu; + ch->hclass = hclass; + ch->instance = instance; + ch->h_object = gsp_disp_find_object_locked(vgpu, hclass, instance); + ch->pb_pa = pb_pa; + ch->limit = (u32)min_t(u64, limit, 0xfff); + ch->address_space = address_space; + ch->ctl_off = gsp_disp_ctl_offset(hclass, instance); + ch->consumed = 0; + ch->last_put = 0; + ch->valid = true; + if (hclass == GSP_DISP_CLASS_CORE && instance == 0) + disp.boot_jiffies = jiffies; + spin_unlock_irqrestore(&disp.lock, flags); + + pr_info("channel class=0x%x inst=%u hObject=0x%x pb=0x%llx limit=0x%x ctl=0x%x\n", + hclass, instance, ch->h_object, pb_pa, ch->limit, ch->ctl_off); +} + +/* + * r1765 flip-event shield predicate: is @h_object a registered display + * channel object of @vgpu whose events must stay off the all-events + * shotgun? Called by gsp_rm_emulation.c before nudging an event in the + * fallback paths. Only meaningful while disp_events_strict is enabled - + * with the legacy default (0) this always returns false, so every shield + * site is inert and the shotgun path is unchanged. + */ +bool gsp_disp_engine_owns_object(struct nv_vgpu_device *vgpu, u32 h_object) +{ + unsigned long flags; + bool owned = false; + int i; + + if (!gsp_disp_engine || !gsp_disp_events_strict || !vgpu || !h_object) + return false; + + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + if (disp.ch[i].valid && disp.ch[i].vgpu == vgpu && + disp.ch[i].h_object == h_object) { + owned = true; + break; + } + } + spin_unlock_irqrestore(&disp.lock, flags); + return owned; +} +EXPORT_SYMBOL_GPL(gsp_disp_engine_owns_object); + +/* + * Apply one parsed method. Only the completion-notifier programming and the + * UPDATE completion point carry engine semantics for the boot; everything + * else is state the engine would apply to the display pipeline (head/window/ + * cursor registers) which we deliberately do not render. + */ +/* + * Is this UPDATE's notifier armed? + * + * NVC57D (core) SET_NOTIFIER_CONTROL carries a NOTIFY bit (12:12) and the + * class disables the notifier with ctxDMA 0 + control 0. + * + * NVC57E (window) SET_NOTIFIER_CONTROL has only MODE (0:0) and OFFSET + * (11:4); there is no NOTIFY bit. The bound notifier ctxDMA *is* the + * enable, and the class disables with ctxDMA 0 + control 0. Treating + * bit 12 as the arm bit made every window flip compute armed=0, so the + * emulation never wrote FINISHED and never posted the channel event for + * the X server's window channel (0xc57e). The DDX's composite/flip + * waits then sat out their timeout (Xorg log "Wait for channel idle + * timed out") and the display pipeline stalled. + */ +static bool gsp_disp_update_armed(const struct gsp_disp_chan *ch) +{ + if (ch->hclass == GSP_DISP_CLASS_CORE) + return (ch->notifier_ctrl & + GSP_DISP_NOTIFIER_CTRL_NOTIFY) != 0; + return ch->notifier_handle != 0 && ch->notifier_pa_valid; +} + +static void gsp_disp_method(struct nv_vgpu_device *vgpu, + struct gsp_disp_chan *ch, u32 addr, u32 value, + u32 push_off) +{ + bool is_core = ch->hclass == GSP_DISP_CLASS_CORE; + u32 notifier_dma = is_core ? GSP_DISP_M_CORE_SET_NOTIFIER_DMA : + GSP_DISP_M_WIN_SET_NOTIFIER_DMA; + u32 notifier_ctrl = is_core ? GSP_DISP_M_CORE_SET_NOTIFIER_CTRL : + GSP_DISP_M_WIN_SET_NOTIFIER_CTRL; + + ch->methods++; + + if (addr == notifier_dma) { + ch->notifier_handle = value; + /* Resolve the surface now: the handle is the instance-table + * key and the surface is bound before any notifier wait. */ + if (value && gsp_disp_finish) { + u64 pa = 0, lim = 0; + u32 target = 0; + + if (gsp_disp_ctxdma_pa(vgpu, value, &pa, &lim, + &target) == 0) { + ch->notifier_pa = pa; + ch->notifier_target = target; + ch->notifier_pa_valid = true; + } + } + pr_info("h=0x%x push=0x%x SET_NOTIFIER_DMA handle=0x%x -> pa=0x%llx target=%u valid=%d\n", + ch->hclass, push_off, value, ch->notifier_pa, + ch->notifier_target, ch->notifier_pa_valid); + return; + } + if (addr == notifier_ctrl) { + ch->notifier_ctrl = value; + if (gsp_disp_dbg || ch->ctrl_logs < 8) { + ch->ctrl_logs++; + pr_info("h=0x%x push=0x%x SET_NOTIFIER_CTRL=0x%x (notify=%u off=%u armed=%d handle=0x%x pa_valid=%d)\n", + ch->hclass, push_off, value, + !!(value & GSP_DISP_NOTIFIER_CTRL_NOTIFY), + GSP_DISP_NOTIFIER_CTRL_OFFSET(value), + gsp_disp_update_armed(ch), + ch->notifier_handle, ch->notifier_pa_valid); + } + return; + } + if (addr == GSP_DISP_M_UPDATE) { + u32 off_bytes = GSP_DISP_NOTIFIER_CTRL_OFFSET(ch->notifier_ctrl); + u64 pa = ch->notifier_pa + off_bytes; + u32 target = ch->notifier_target; + u32 finished = GSP_DISP_NOTIFIER_FINISHED; + bool armed = gsp_disp_update_armed(ch); + bool wrote = false; + + ch->updates++; + if (gsp_disp_finish && armed && ch->notifier_handle && + ch->notifier_pa_valid) { + u32 before = 0, after = 0; + + /* + * Safety: the target must look like a notifier status + * word (0 = NOT_BEGUN, as initialized immediately + * before an armed update) and must not be a display + * push buffer. Nothing else is ever written. + */ + if (gsp_disp_pa_is_push_buffer(pa & ~0xfffULL) || + gsp_disp_notifier_read(vgpu, pa, target, &before, + sizeof(before)) || + (before != 0 && before != GSP_DISP_NOTIFIER_FINISHED)) { + if (ch->update_logs < 8) { + ch->update_logs++; + pr_warn("h=0x%x UPDATE notifier pa=0x%llx target=%u rejected (val=0x%x)\n", + ch->hclass, pa, target, before); + } + } else if (gsp_disp_notifier_write(vgpu, pa, target, + &finished, + sizeof(finished)) == 0) { + ch->notifier_writes++; + wrote = true; + gsp_disp_notifier_read(vgpu, pa, target, &after, + sizeof(after)); + if (after != finished && ch->update_logs < 8) { + ch->update_logs++; + pr_warn("h=0x%x notifier readback 0x%x at pa=0x%llx target=%u\n", + ch->hclass, after, pa, target); + } + } + } + /* + * Post the channel completion event: the driver registered an + * NV01_EVENT on the channel object and the real engine (via + * RM) signals it when the update's notifier is written. This + * is what completes nvidia-drm flips and X GLX syncs. + * + * r1666: only a REAL FINISHED transition may raise the event. + * Posting on every UPDATE (armed=0 / no notifier write) made + * the DDX's NVKMS FLIP_OCCURRED handler (nvidia_drv.so + * 0xaee70) reap stale notifier slots and hit its fixed-point + * layer retry (Xorg spun there at 100 % in r1665). + */ + if (gsp_disp_events && ch->h_object && wrote) { + /* + * r1765: strict mode delivers only to the events + * registered on this channel; a miss returns -ENOENT + * and the pump retries instead of shotgunning every + * registered event. + */ + int rc = (gsp_disp_events_strict ? + gsp_rm_post_channel_event_strict : + gsp_rm_post_channel_event)(vgpu, + ch->h_object); + + if (!rc) { + ch->notifier_posts++; + ch->event_pending = false; + } else { + /* + * The event burst cap (or a transiently full + * status queue) silently dropped the + * wake-up. A completed UPDATE whose event is + * lost wedges the client's flip wait forever + * (r1636: X glxinfo stuck in + * os_acquire_rwlock_write), so retry from the + * pump instead of giving up. + */ + ch->event_pending = true; + ch->event_retry_at = jiffies + + msecs_to_jiffies(GSP_DISP_EVENT_RETRY_MS); + if (ch->event_drop_logs < 8) { + ch->event_drop_logs++; + pr_info("h=0x%x UPDATE completion event deferred rc=%d (hObj=0x%x)\n", + ch->hclass, rc, ch->h_object); + } + } + } + if (ch->update_logs < 8 || gsp_disp_dbg) { + ch->update_logs++; + pr_info("h=0x%x push=0x%x UPDATE armed=%d hObj=0x%x handle=0x%x notifier=0x%llx/%u target=%u -> %s (posts=%u)\n", + ch->hclass, push_off, armed, ch->h_object, + ch->notifier_handle, ch->notifier_pa, off_bytes, + target, wrote ? "FINISHED" : "none", + ch->notifier_posts); + } + } +} + +/* + * Parse the NV DMA segment [ch->consumed, put) of one channel. The ring is + * at most 4 KB, so a single guest read covers it; the driver's wrap JUMP is + * followed. + */ +static void gsp_disp_run(struct nv_vgpu_device *vgpu, struct gsp_disp_chan *ch, + u32 put) +{ + u32 off = ch->consumed; + u32 start = off; + u32 steps = 0; + + if (!disp.pbuf || put > ch->limit + 4) + return; + if (gsp_disp_read(vgpu, ch->pb_pa, disp.pbuf, 4096)) + return; + + /* + * Bounded raw dump of the pushed segment (disp_dbg only): the head + * shows the first methods, the tail shows what completes the update + * (UPDATE 0x200 / SET_NOTIFIER 0x208/0x20c). Bounded to 4 pushes so + * a boot cannot flood the log. + */ + if (gsp_disp_dbg && disp.push_dumps < 4 && put > start) { + u32 dwords = put - start; + u32 d, shown = min_t(u32, dwords, 16); + + disp.push_dumps++; + pr_info("h=0x%x push@0x%x len=%u dwords head:", + ch->hclass, start, dwords); + for (d = 0; d < shown; d++) + pr_cont(" %08x", + *(u32 *)(disp.pbuf + start + 4 * d)); + if (dwords > 48) { + pr_cont(" ... tail:"); + for (d = dwords - 32; d < dwords; d++) + pr_cont(" %08x", + *(u32 *)(disp.pbuf + start + 4 * d)); + } + pr_cont("\n"); + } + + while (off != put && steps++ < 8192) { + u32 hdr, op, count, addr; + + if (off & 3 || off + 4 > 4096) + break; + hdr = *(u32 *)(disp.pbuf + off); + op = hdr >> 29; + count = (hdr >> 18) & 0x3ff; + /* + * NV_UDISP_DMA_METHOD_OFFSET is 15:2 and is a DWORD index; + * gsp_disp_method() works in BYTE method addresses (0x200 + * UPDATE, 0x208/0x20c notifier), so shift back to bytes. + * Before this fix the executor compared 0x82 against 0x200 + * and the FINISHED write could never match (r1624/r1625). + */ + addr = ((hdr >> 2) & 0x3fff) << 2; + + switch (op) { + case 0: /* METHOD (incrementing) */ + case 5: /* ONE_INC */ + if (count == 0 || count > 0x400 || + off + 4 + count * 4 > 4096) { + off += 4; + break; + } + { + u32 k; + + for (k = 0; k < count; k++) + gsp_disp_method(vgpu, ch, addr + 4 * k, + *(u32 *)(disp.pbuf + off + 4 + + 4 * k), off); + } + off += 4 + count * 4; + break; + case 2: /* NONINC_METHOD */ + if (count == 0 || count > 0x400 || + off + 4 + count * 4 > 4096) { + off += 4; + break; + } + { + u32 k; + + for (k = 0; k < count; k++) + gsp_disp_method(vgpu, ch, addr, + *(u32 *)(disp.pbuf + off + 4 + + 4 * k), off); + } + off += 4 + count * 4; + break; + case 1: /* JUMP: bits 11:2 are the byte target */ + { + u32 target = hdr & 0xffc; + + ch->wraps++; + if (target > 4092) { + off = put; + break; + } + if (gsp_disp_dbg) + pr_info("h=0x%x JUMP at 0x%x -> 0x%x (put=0x%x)\n", + ch->hclass, off, target, put); + off = target; + } + break; + case 3: /* SET_SUBDEVICE_MASK */ + case 4: /* IMMD (data in the header) */ + default:/* END / reserved */ + off += 4; + break; + } + } + + /* + * Only publish a read pointer when the segment parsed through to the + * driver's PUT. A segment that runs off the 4 KB ring (unwritten or + * not-yet-visible content) or hits misaligned data must NOT consume + * anything: publishing GET past PUT wedged nvidia-modeset on the + * window channel (r1630: GET=0x1000, "Error while waiting for GPU + * progress"). Leaving consumed unchanged also retries next pass, + * which covers a ring the guest has kicked but not yet flushed. + */ + if (off != put) { + if (gsp_disp_dbg && disp.push_dumps < 8) { + u32 d, shown = 16; + + disp.push_dumps++; + pr_info("h=0x%x push-noparse 0x%x..0x%x (consumed stays 0x%x) head:", + ch->hclass, start, put, ch->consumed); + if (put > start && put - start < shown) + shown = put - start; + for (d = 0; d < shown; d++) + pr_cont(" %08x", + *(u32 *)(disp.pbuf + start + 4 * d)); + pr_cont("\n"); + } + return; + } + + ch->consumed = off; + ch->last_put = put; + if (gsp_disp_dbg && off != start) + pr_info("h=0x%x executed push 0x%x..0x%x (%llu methods)\n", + ch->hclass, start, put, ch->methods); +} + +void gsp_disp_engine_pump(struct nv_vgpu_device *vgpu) +{ + struct gsp_disp_chan snap[GSP_DISP_MAX_CHANNELS]; + unsigned long flags; + unsigned long now = jiffies; + int n = 0, i, j; + + if (!gsp_disp_engine || !vgpu) + return; + + if (READ_ONCE(gsp_disp_dump)) { + WRITE_ONCE(gsp_disp_dump, 0); + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + struct gsp_disp_chan *c = &disp.ch[i]; + + if (!c->valid || c->vgpu != vgpu) + continue; + pr_info("dump class=0x%x inst=%u hObj=0x%x pb=0x%llx gen=%u/%u notif_h=0x%x notif_pa=0x%llx ctrl=0x%x writes=%u posts=%u updates=%llu\n", + c->hclass, c->instance, c->h_object, c->pb_pa, + c->consumed, c->last_put, c->notifier_handle, + c->notifier_pa, c->notifier_ctrl, + c->notifier_writes, c->notifier_posts, + c->updates); + } + spin_unlock_irqrestore(&disp.lock, flags); + } + + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + if (!disp.ch[i].valid || disp.ch[i].vgpu != vgpu) + continue; + snap[n++] = disp.ch[i]; + } + spin_unlock_irqrestore(&disp.lock, flags); + + for (i = 0; i < n; i++) { + struct gsp_disp_chan *s = &snap[i]; + u32 put, get; + + put = gsp_disp_bar0_read(vgpu, s->ctl_off + GSP_DISP_REG_PUT) & + 0xffc; + + /* + * Raster line for the window channels: NVC57E_GET_LINE at + * 0x208. Bit 15 set = blanking, 14:0 = line. A free-running + * 1080p60 counter is enough for the driver's scanline/vblank + * reads to make progress. + */ + if (gsp_disp_vblank && s->hclass == GSP_DISP_CLASS_WINDOW) { + u64 ns = (u64)(now - disp.boot_jiffies) * + (NSEC_PER_SEC / HZ); + u32 frame_off = (u32)(ns % GSP_DISP_FRAME_NS); + u32 line = (u32)div_u64((u64)frame_off * + GSP_DISP_VBLANK_LINES, + GSP_DISP_FRAME_NS); + u32 val = line & 0x7fff; + + if (line >= GSP_DISP_ACTIVE_LINES) + val |= BIT(15); + gsp_disp_bar0_write(vgpu, + s->ctl_off + GSP_DISP_REG_GET_LINE, + val); + } + + if (gsp_disp_dbg && time_after(jiffies, s->last_dbg + HZ)) { + s->last_dbg = jiffies; + pr_info("poll h=0x%x inst=%u put=0x%x consumed=0x%x ctl=0x%x\n", + s->hclass, s->instance, put, s->consumed, + s->ctl_off); + } + if (put != s->consumed) { + gsp_disp_run(vgpu, s, put); + + /* + * Publish our read pointer; the driver's nvEvoMakeRoom + * and EvoCoreReadGet rely on it to reuse the ring. + */ + gsp_disp_bar0_write(vgpu, s->ctl_off + GSP_DISP_REG_GET, + s->consumed); + get = gsp_disp_bar0_read(vgpu, + s->ctl_off + GSP_DISP_REG_GET); + + if (disp.log_budget < 4096) { + u64 age = (u64)(now - disp.boot_jiffies) * + (NSEC_PER_SEC / HZ); + + disp.log_budget++; + pr_info("pump h=0x%x inst=%u put=0x%x get=0x%x consumed=0x%x at %llu ms\n", + s->hclass, s->instance, put, get, + s->consumed, div_u64(age, 1000000)); + } + } + + /* + * Retry a completion event whose send was refused (burst cap / + * transient queue state, or the hint dedup coalescing it into a + * post the waiter never saw). Runs outside disp.lock (the post + * takes the RM status-queue lock) and on idle passes too: the + * fatal case is an UPDATE that is the channel's last push, so a + * retry gated on new work never runs (r1649). The forced post + * ignores the per-event hint dedup; the retry window bounds it + * to one attempt per 250 ms per waiting channel while the burst + * cap still guards the status ring. + */ + if (s->event_pending && s->h_object && gsp_disp_events && + time_after_eq(jiffies, s->event_retry_at)) { + /* r1765: retry with the same strict/legacy entry + * point the UPDATE post used. */ + if (!(gsp_disp_events_strict ? + gsp_rm_post_channel_event_force_strict : + gsp_rm_post_channel_event_force)(vgpu, + s->h_object)) { + s->event_pending = false; + s->notifier_posts++; + pr_info("h=0x%x UPDATE completion event re-posted after deferral (posts=%u)\n", + s->hclass, s->notifier_posts); + } else { + s->event_retry_at = jiffies + + msecs_to_jiffies(GSP_DISP_EVENT_RETRY_MS); + } + } + + /* Write the bookkeeping back if the slot still names this + * channel (registration may have replaced it during a boot + * round in another context). */ + spin_lock_irqsave(&disp.lock, flags); + for (j = 0; j < GSP_DISP_MAX_CHANNELS; j++) { + struct gsp_disp_chan *c = &disp.ch[j]; + + if (c->valid && c->vgpu == vgpu && + c->hclass == s->hclass && + c->instance == s->instance && + c->pb_pa == s->pb_pa) { + c->consumed = s->consumed; + c->last_put = s->last_put; + c->methods = s->methods; + c->updates = s->updates; + c->wraps = s->wraps; + c->notifier_handle = s->notifier_handle; + c->notifier_ctrl = s->notifier_ctrl; + c->notifier_pa = s->notifier_pa; + c->notifier_target = s->notifier_target; + c->notifier_pa_valid = s->notifier_pa_valid; + c->notifier_writes = s->notifier_writes; + c->notifier_posts = s->notifier_posts; + c->event_pending = s->event_pending; + c->event_retry_at = s->event_retry_at; + c->event_drop_logs = s->event_drop_logs; + c->update_logs = s->update_logs; + c->logged_update = s->logged_update; + c->last_dbg = s->last_dbg; + break; + } + } + spin_unlock_irqrestore(&disp.lock, flags); + } +} + +void gsp_disp_engine_vgpu_cleanup(struct nv_vgpu_device *vgpu) +{ + unsigned long flags; + int i; + + /* Fresh diagnostic budgets for the next mdev/boot. */ + disp.push_dumps = 0; + disp.im_logs = 0; + + spin_lock_irqsave(&disp.lock, flags); + for (i = 0; i < GSP_DISP_MAX_CHANNELS; i++) { + if (disp.ch[i].valid && disp.ch[i].vgpu == vgpu) { + pr_info("channel class=0x%x inst=%u dropped (mdev cleanup; updates=%llu)\n", + disp.ch[i].hclass, disp.ch[i].instance, + disp.ch[i].updates); + memset(&disp.ch[i], 0, sizeof(disp.ch[i])); + } + } + for (i = 0; i < GSP_DISP_MAX_NOTES; i++) + if (disp.note[i].vgpu == vgpu) + memset(&disp.note[i], 0, sizeof(disp.note[i])); + for (i = 0; i < ARRAY_SIZE(disp.im); i++) + if (disp.im[i].vgpu == vgpu) + memset(&disp.im[i], 0, sizeof(disp.im[i])); + spin_unlock_irqrestore(&disp.lock, flags); +} + +int gsp_disp_engine_init(void) +{ + spin_lock_init(&disp.lock); + memset(disp.ch, 0, sizeof(disp.ch)); + memset(disp.note, 0, sizeof(disp.note)); + memset(disp.im, 0, sizeof(disp.im)); + disp.pbuf = kmalloc(4096, GFP_KERNEL); + if (!disp.pbuf) + return -ENOMEM; + disp.ibuf = kmalloc(GSP_DISP_INST_OBJ_END, GFP_KERNEL); + if (!disp.ibuf) { + kfree(disp.pbuf); + disp.pbuf = NULL; + return -ENOMEM; + } + disp.boot_jiffies = jiffies; + disp.log_budget = 0; + disp.push_dumps = 0; + disp.im_logs = 0; + pr_info("display engine emulation %s\n", + gsp_disp_engine ? "enabled" : "disabled"); + return 0; +} + +void gsp_disp_engine_fini(void) +{ + kfree(disp.pbuf); + disp.pbuf = NULL; + kfree(disp.ibuf); + disp.ibuf = NULL; + memset(disp.ch, 0, sizeof(disp.ch)); + memset(disp.note, 0, sizeof(disp.note)); + memset(disp.im, 0, sizeof(disp.im)); +} diff --git a/kernel/gsp_emulation.c b/kernel/gsp_emulation.c index 47c8151..b82bab1 100644 --- a/kernel/gsp_emulation.c +++ b/kernel/gsp_emulation.c @@ -644,10 +644,99 @@ void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id eventfd_signal(trigger); } +/* + * Raise the GSP engine event interrupt. + * + * Mirrors what a real GSP-RM does when it queues an event for the host: + * - program the RISC-V interrupt mask/destination so SWGEN0 is routed to + * the CPU (read-only to the guest; the GSP firmware owns them), + * - set SWGEN0 in the falcon IRQSTAT, + * - latch the GSP stall vector (155) in the host CPU interrupt tree: + * LEAF[4] bit 27 and TOP[0] bit 2 (UVM_SHARED subtree 2), + * - signal MSI-X. + * + * Without the leaf bit the guest's primary ISR returns IRQ_NONE and the + * threaded handler (kgspService_TU102 -> kgspRpcRecvEvents -> nv_post_event) + * never runs, so a client blocked in poll() on /dev/nvidia0 (a GLX swap) + * waits forever even though the event is sitting in the status queue. + */ +void gsp_emu_raise_gsp_event_irq(struct gsp_emu_context *ctx) +{ + u32 reg, val; + u32 leaf_idx = GSP_EMU_GSP_EVENT_VECTOR / 32; + u32 leaf_bit = GSP_EMU_GSP_EVENT_VECTOR % 32; + u32 subtree = GSP_EMU_GSP_EVENT_VECTOR / 64; + u32 top_idx = subtree / 32; + u32 top_bit = subtree % 32; + + if (!ctx || !ctx->bar0_vaddr) + return; + + /* IRQMASK / IRQDEST: route SWGEN0 to the CPU (idempotent). */ + reg = GSP_EMU_REG_RISCV_IRQMASK; + val = gsp_emu_shadow_read32(ctx, reg); + if (!(val & GSP_EMU_FALCON_IRQSTAT_SWGEN0)) + gsp_emu_shadow_write32(ctx, reg, + val | GSP_EMU_FALCON_IRQSTAT_SWGEN0); + reg = GSP_EMU_REG_RISCV_IRQDEST; + val = gsp_emu_shadow_read32(ctx, reg); + if (!(val & GSP_EMU_FALCON_IRQSTAT_SWGEN0)) + gsp_emu_shadow_write32(ctx, reg, + val | GSP_EMU_FALCON_IRQSTAT_SWGEN0); + + /* Pending source: SWGEN0. */ + reg = GSP_EMU_REG_FALCON_IRQSTAT; + val = gsp_emu_shadow_read32(ctx, reg); + gsp_emu_shadow_write32(ctx, reg, + val | GSP_EMU_FALCON_IRQSTAT_SWGEN0); + + /* CPU interrupt tree: LEAF[leaf_idx] |= bit, TOP[top_idx] |= bit. */ + reg = GSP_EMU_REG_CPU_INTR_LEAF_BASE + leaf_idx * 4; + val = gsp_emu_shadow_read32(ctx, reg); + gsp_emu_shadow_write32(ctx, reg, val | (1U << leaf_bit)); + reg = GSP_EMU_REG_CPU_INTR_TOP_BASE + top_idx * 4; + val = gsp_emu_shadow_read32(ctx, reg); + gsp_emu_shadow_write32(ctx, reg, val | (1U << top_bit)); + + gsp_emu_inject_completion_interrupt(ctx, 0); +} + /* ============================================================ * Emulation Thread * ============================================================ */ +/* Advance the emulated GPU PTIMER (NV_PTIMER_TIME_0/1 at BAR0 0x9400/0x9410). + * + * The guest's nvidia.ko RM uses this register as its time source for timed + * waits (e.g. _nv015102rm -> _nv018147rm -> _nv018351rm). BAR0 is a plain + * host shadow mapped to the guest, so nothing advances these registers by + * default: every RM deadline becomes unreachable and the DDX/GL client + * spins forever inside the driver (observed as an Xorg hang at GDI/window + * creation). Seed from the existing value to keep time continuous, then + * advance by the host monotonic clock. + */ +static void gsp_emu_tick_ptimer(struct gsp_emu_context *ctx) +{ + u64 now, val; + + if (!ctx->bar0_vaddr) + return; + + now = ktime_get_ns(); + if (!ctx->ptimer_seeded) { + u32 lo = gsp_emu_shadow_read32(ctx, 0x9400); + u32 hi = gsp_emu_shadow_read32(ctx, 0x9410); + + ctx->ptimer_base = ((u64)hi << 32) | lo; + ctx->ptimer_t0 = now; + ctx->ptimer_seeded = true; + } + + val = ctx->ptimer_base + (now - ctx->ptimer_t0); + gsp_emu_shadow_write32(ctx, 0x9400, (u32)val); + gsp_emu_shadow_write32(ctx, 0x9410, (u32)(val >> 32)); +} + static int gsp_emu_thread_fn(void *data) { struct gsp_emu_context *ctx = data; @@ -659,6 +748,9 @@ static int gsp_emu_thread_fn(void *data) nv_vgpu_info(ctx->vgpu, "GSP emulation thread started"); while (!kthread_should_stop()) { + /* Keep the guest-visible GPU timer advancing (RM timed waits) */ + gsp_emu_tick_ptimer(ctx); + /* Read doorbell */ doorbell_val = gsp_emu_read_doorbell(ctx); diff --git a/kernel/gsp_emulation.h b/kernel/gsp_emulation.h index 9a8f241..16f4db4 100644 --- a/kernel/gsp_emulation.h +++ b/kernel/gsp_emulation.h @@ -116,6 +116,29 @@ #define GSP_EMU_REG_CPU_INTR_LEAF_BASE (GSP_EMU_REG_FULL_PHYS_OFFSET + 0x1000) #define GSP_EMU_REG_CPU_INTR_TOP_BASE (GSP_EMU_REG_FULL_PHYS_OFFSET + 0x1600) +/* GSP event interrupt (Turing, physical-GPU path). + * + * A real GSP-RM posts an event into the status queue and raises the GSP + * engine interrupt: it sets SWGEN0 in the falcon IRQSTAT and the host CPU's + * interrupt tree latches the GSP stall vector (155 = LEAF[4] bit 27, UVM_SHARED + * subtree 2 -> TOP[0] bit 2). The guest's primary ISR (nvidia_isr) then + * returns IRQ_WAKE_THREAD, kgspService_TU102 reads + * IRQSTAT & RISCV_IRQMASK & RISCV_IRQDEST, sees SWGEN0, calls + * kgspRpcRecvEvents -> nv_post_event -> wake_up_interruptible, which is what + * releases a client blocked in poll() on /dev/nvidia0 (e.g. a GLX swap). + * + * We were only signalling the MSI-X eventfd, so the primary ISR returned + * IRQ_NONE (no pending leaf bit) and the event thread never ran: IRQ 33 + * counted thousands of interrupts while the guest's nvidia worker threads sat + * idle and the client polled forever. IRQMASK/IRQDEST are read-only to the + * guest (the GSP firmware owns them), so the emulated GSP must program them. */ +#define GSP_EMU_REG_FALCON_IRQSCLR 0x110004 /* -W- : write 1 to clear */ +#define GSP_EMU_REG_FALCON_IRQSTAT 0x110008 /* R-- : pending sources */ +#define GSP_EMU_REG_RISCV_IRQMASK 0x1112b4 /* R-I : enabled sources */ +#define GSP_EMU_REG_RISCV_IRQDEST 0x1112b8 /* RW : routed to CPU */ +#define GSP_EMU_FALCON_IRQSTAT_SWGEN0 (1U << 6) +#define GSP_EMU_GSP_EVENT_VECTOR 155 /* MC_ENGINE_IDX_GSP stall */ + /* Windowed aliases of the VF L2 invalidate registers (GPU_VREG_* path). */ #define GSP_EMU_REG_VF_L2_SYSMEM_INV_WIN (GSP_EMU_REG_FULL_PHYS_OFFSET + 0x000f00) #define GSP_EMU_REG_VF_L2_PEERMEM_INV_WIN (GSP_EMU_REG_FULL_PHYS_OFFSET + 0x000f04) @@ -169,6 +192,14 @@ struct gsp_emu_context { u32 bar0_window_base; /* last-seen NV_PBUS_BAR0_WINDOW BASE (page) */ u32 bar0_window_target; /* last-seen TARGET (0=vidmem, 2/3=sysmem) */ bool bar0_window_valid; /* base/target captured at least once */ + + /* Emulated PTIMER (BAR0 0x9400/0x9410): the guest RM's time source. + * Seeded from the shadow's current 64-bit value on the first tick so + * the guest-observed time stays continuous, then advanced by the host + * monotonic clock. Without this the RM's timed waits never expire. */ + u64 ptimer_base; + u64 ptimer_t0; + bool ptimer_seeded; }; /* Shadow accessors shared with gsp_rm_emulation.c */ @@ -195,6 +226,9 @@ void gsp_emu_cleanup(struct nv_vgpu_device *vgpu); void gsp_emu_clear_boot_state(struct gsp_emu_context *ctx, const char *reason); int gsp_emu_process_rpc(struct gsp_emu_context *ctx); void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id); +/* Raise the GSP event interrupt (falcon SWGEN0 + CPU_INTR leaf/top + MSI-X) + * so the guest's ISR drains the status queue and wakes poll() waiters. */ +void gsp_emu_raise_gsp_event_irq(struct gsp_emu_context *ctx); /* ============================================================ * Debug/Statistics diff --git a/kernel/gsp_phys_booter.c b/kernel/gsp_phys_booter.c index 967dc25..189d8b7 100644 --- a/kernel/gsp_phys_booter.c +++ b/kernel/gsp_phys_booter.c @@ -46,6 +46,7 @@ #include <linux/vgaarb.h> #include <linux/acpi.h> #include <linux/mm.h> +#include <linux/vmalloc.h> #include <asm/barrier.h> #include <asm/unaligned.h> @@ -57,6 +58,15 @@ #include "gsp_phys_booter_load_img.h" #include "gsp_phys_gsp_rm_boot_img.h" #include "gsp_phys_registry_img.h" +/* + * SESSION 153-CONTINUED-114: golden MME slot library (untracked build + * input, .gitignore'd - see the file header for provenance). Used only + * when sema_gl_mme_golden=1: called-but-uncaptured slots are served + * from the driver's own generated macro code instead of being stubbed, + * so the DRAW_IAB call (slot 34) runs deterministically on every boot + * (the full-capture profile becomes a per-boot lottery otherwise). + */ +#include "golden-mme-tu104.h" #include "gsp_phys_registry_img_srvctx.h" static bool gsp_phys_boot_param; @@ -129,6 +139,16 @@ MODULE_PARM_DESC(gsp_phys_gr_reloc_resolve, * per-launch RELOC-CENSUS line. */ #define GSP_PHYS_GR_COPYWIT_MAX 12u +/* + * CONT-112: the changed-page witness gets its OWN budget. A store-rich + * launch walks the tracked pages in bi order and the unchanged pages come + * first, so the shared 12-line budget was always consumed by the OUT-UNCH + * flood before the loop reached the handful of engine-changed pages - the + * r1385 evidence has "2/900 pages changed" + OUT-ALIAS-WIT for bi=8960/8965 + * but no OUT-PAGE-WIT va->hpa pair for either. With the split counter both + * classes are witnessed on every launch (<=12 unchanged + <=16 changed). + */ +#define GSP_PHYS_GR_COPYWIT_CHG_MAX 16u static bool gsp_phys_gr_copyout_dump_param = true; module_param_named(gsp_phys_gr_copyout_dump, gsp_phys_gr_copyout_dump_param, bool, 0644); @@ -140,6 +160,7 @@ static inline bool gsp_phys_gr_copywit_enabled(void) return READ_ONCE(gsp_phys_gr_copyout_dump_param); } + /* * r405 late-write probes (the r405 "partial write" class: the scale launch * of streams2 wrote 1 of its 32 expected in-place pages and every other @@ -151,9 +172,19 @@ static inline bool gsp_phys_gr_copywit_enabled(void) * "ENGINE STILL WRITING" proves the copy-out shipped before the stores * landed (the fix is a longer settle window); "stable" proves the engine * genuinely stopped, which needs a different treatment. - * - gsp_phys_gr_settle_ms (default 300): the content-stable window the + * - gsp_phys_gr_settle_ms (default 100): the content-stable window the * partial-changed rule requires before it declares the stores settled. * Raise it to give a still-writing engine its time (A/B in one cycle). + * + * r552 (2026-10-23): measured A/B on real boots. Every launch with a + * partial changed-page count (read-mostly kernels: 1..64 of 1000+ pages + * written, so the changed>=n_all shortcut can never fire) paid the full + * window plus scan overhead: 683-901 ms per launch at 300 ms. At 100 ms + * the same boots measured 231-247 ms of retire wait and the launch + * latency p50 dropped 1119 ms -> 651 ms (n=22 launches; baseline 684 ms) + * with the battery unchanged (12/15, 0 hangs) and a green 2-round soak + * on the next boot. The content-stability hash still gates the verdict; + * only the minimum stable duration shrank. */ static bool gsp_phys_gr_late_probe_param; module_param_named(gsp_phys_gr_late_probe, gsp_phys_gr_late_probe_param, @@ -161,11 +192,11 @@ module_param_named(gsp_phys_gr_late_probe, gsp_phys_gr_late_probe_param, MODULE_PARM_DESC(gsp_phys_gr_late_probe, "Diagnostic: after the GR retire wait, re-scan the tracked pages +500 ms later and log whether the engine is still writing (default 0)"); -static int gsp_phys_gr_settle_ms_param = 300; +static int gsp_phys_gr_settle_ms_param = 100; module_param_named(gsp_phys_gr_settle_ms, gsp_phys_gr_settle_ms_param, int, 0644); MODULE_PARM_DESC(gsp_phys_gr_settle_ms, - "GR retire wait: content-stable window (ms) required before a partial changed-page count counts as settled (default 300)"); + "GR retire wait: content-stable window (ms) required before a partial changed-page count counts as settled (default 100; 300 was the pre-r552 value)"); static inline u32 gsp_phys_gr_settle_iters(void) { @@ -1120,18 +1151,61 @@ bool gsp_phys_boot_requested(void) * from gpFifoOffset, the pushbuffer from the GPFIFO ring entry GET, the CE * operands from the method block): SET_SEMAPHORE_A/B are always GPU VAs with * no mode bit (clc5b5.h:39-44; uvm_pascal_ce.c:42-63) and the user channel's - * copy is virtual-mode on both sides (uvm_maxwell_ce.c:179-205). All five - * pages live in one 2 MB PD0 slot below the VAS vaSize limit: with + * copy is virtual-mode on both sides (uvm_maxwell_ce.c:179-205). All the + * ladder pages live in one 2 MB PD0 slot below the VAS vaSize limit: with * VA_BASE = 0x00100000 every parent index (48:47, 46:38, 37:29, 28:21) is 0 - * and the five 4 KB pages take small-PT slots 0x100..0x104 - * (kern_gmmu_fmt_gp10x.c:16-83; uvm_pascal_mmu.c:25-32,172-204). + * and the 4 KB pages take small-PT slots in the single 128 B PTE line + * 0x100..0x10f (kern_gmmu_fmt_gp10x.c:16-83; uvm_pascal_mmu.c:25-32,172-204). + * + * SESSION 153-CONTINUED-66: the module pushbuffer is TWO pages. The MME + * context-setup prelude matured to 901 words while the beef window runs 390 + * words and the SA-RAM stubs add 29..47, so the single-launch block needs + * ~1400 words = 5.6 KB: the 1-page pushbuffer made every prelude-bearing + * launch print "MME-PRELUDE deferred (window=390 prelude=901 sa=47 exceeds + * block; replaying window alone)" and go out prelude=0. The block now spans + * VA_PB..VA_PB+0x2000 with the second page at small-PT slot 0x102. The + * slot 0x102 page used to be CE SRC; SRC moved to 0x109 (it is only read by + * the CE copy path and the GR inline-QMD destination default). Slot 0x10a + * is the dedicated warm-semaphore probe page: the probe used to share the + * pushbuffer's upper half, which a >0x800-byte forwarded block now occupies. */ #define GSP_PHYS_RMCALL_VA_BASE 0x00100000ULL #define GSP_PHYS_RMCALL_VA_GPFIFO GSP_PHYS_RMCALL_VA_BASE #define GSP_PHYS_RMCALL_VA_PB (GSP_PHYS_RMCALL_VA_BASE + GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_SRC (GSP_PHYS_RMCALL_VA_BASE + 2 * GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_DST (GSP_PHYS_RMCALL_VA_BASE + 3 * GSP_PHYS_PAGE_SIZE) -#define GSP_PHYS_RMCALL_VA_SEM (GSP_PHYS_RMCALL_VA_BASE + 4 * GSP_PHYS_PAGE_SIZE) +/* SESSION 153-CONTINUED-66: page 2 of the module pushbuffer (same PT line). */ +#define GSP_PHYS_RMCALL_VA_PB2 (GSP_PHYS_RMCALL_VA_BASE + 2 * GSP_PHYS_PAGE_SIZE) +/* + * SESSION 153-CONTINUED-77: page 3 of the module pushbuffer (slot 0x103) + * so a matured MME prelude (cap 2048 words) plus the largest chunked + * window fit one fetch block. The block walk is linear in VA, so the + * three pushbuffer VAs must stay contiguous: VA_DST (previously at + * base+0x3000, slot 0x103) moves to base+0xB000 (slot 0x10b, free). + */ +#define GSP_PHYS_RMCALL_VA_PB3 (GSP_PHYS_RMCALL_VA_BASE + 3 * GSP_PHYS_PAGE_SIZE) +/* + * SESSION 153-CONTINUED-77: page 4 (slot 0x104) after the r1163 cycle + * showed the capture cap-limited at 2048 words (ir_end=0x265, 34 slots + * covered). VA_SEM moves 0x104 -> 0x10c (free) so the four pushbuffer + * pages stay contiguous for the linear fetch. + */ +#define GSP_PHYS_RMCALL_VA_PB4 (GSP_PHYS_RMCALL_VA_BASE + 4 * GSP_PHYS_PAGE_SIZE) +#define GSP_PHYS_RMCALL_VA_DST (GSP_PHYS_RMCALL_VA_BASE + 0xB000ULL) +#define GSP_PHYS_RMCALL_VA_SEM (GSP_PHYS_RMCALL_VA_BASE + 0xC000ULL) +/* SESSION 153-CONTINUED-66: CE SRC moved off the pushbuffer's second page. */ +#define GSP_PHYS_RMCALL_VA_SRC (GSP_PHYS_RMCALL_VA_BASE + 0x9000ULL) +/* SESSION 153-CONTINUED-66: warm-semaphore probe page (own PTE, no sharing). */ +#define GSP_PHYS_RMCALL_VA_GR_PROBE (GSP_PHYS_RMCALL_VA_BASE + 0xA000ULL) +/* + * SESSION 153-CONTINUED-96: secondary CE (COPY0) lane. A COPY0 channel on + * the same VAS translates through the same 2 MiB window PDB, but CE class + * methods are never legal on the GR0 ladder channel, so the ladder's own + * FBMEM readback (gsp_phys_ctx_readback_try) needs its own engine. The + * lane's GPFIFO ring and one-page pushbuffer get their own small-PT slots + * 0x10d/0x10e in the same 128 B PTE line the ladder's pages occupy, planted + * in pdb_build (stage 3, before any engine fetch; there is no invalidate). + */ +#define GSP_PHYS_RMCALL_VA_CE_GPFIFO (GSP_PHYS_RMCALL_VA_BASE + 0xD000ULL) +#define GSP_PHYS_RMCALL_VA_CE_PB (GSP_PHYS_RMCALL_VA_BASE + 0xE000ULL) /* * m69: window-0 shadow pool - 16 module pages holding the Tier-2 copies of * the QMD's program/param pages, so the GR engine never has to read through @@ -1417,6 +1491,24 @@ bool gsp_phys_boot_requested(void) #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_PM_FB 0xcaf000b8u #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM0 0xcaf000c0u #define GSP_PHYS_RMCALL_HANDLE_USER_LANE_FAULT_MEM1 0xcaf000c4u +/* + * SESSION 153-CONTINUED-96: the secondary COPY0 lane's TSG/channel handles + * (free in the ladder's client range: 0xcaf000c8/0xcc sit above the mode-35 + * fault-mem handles and below the registered-buffer base 0xcaf01000). + */ +#define GSP_PHYS_RMCALL_HANDLE_CE_TSG 0xcaf000c8u +#define GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL 0xcaf000ccu +/* + * Mode 35, graphics-execution plan §F first experiment: the TURING_A + * (0xc597, GR_OBJECT_TYPE_3D) class object constructed on the V7 + * user-lane channel AFTER its promote - the missing half the mode-35 + * capture never provided ("mode 35: ... the capture carries no 0xc597 on + * the user lane"). Best-effort like the TURING_COMPUTE_A: a rejection is + * logged and the lane keeps running; a construction on V7 must not hit + * the wired-golden deadlock class (m3a-m3g, that channel). 0xcaf00094 is + * free between 0xcaf00090 and 0xcaf000a0. + */ +#define GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D 0xcaf00094u /* * Mode-35 V7 channel flags/privilege and fault/PM geometry, decoded from the @@ -1642,6 +1734,19 @@ enum gsp_phys_rmcall_buf { GSP_PHYS_RMCALL_BUF_GOLDEN_PT0, GSP_PHYS_RMCALL_BUF_GOLDEN_PT1, GSP_PHYS_RMCALL_BUF_GOLDEN_PT2, + /* + * SESSION 153-CONTINUED-96: secondary COPY0 lane buffers (one page + * each). The instance/USERD/RAMFC/mthdbuf are the lane's own memdescs + * in the channel alloc (a second channel must never share the GR + * lane's hardware USERD); the GPFIFO ring and pushbuffer page are + * engine-reached through their planted PTE slots. + */ + GSP_PHYS_RMCALL_BUF_CE_GPFIFO, + GSP_PHYS_RMCALL_BUF_CE_USERD, + GSP_PHYS_RMCALL_BUF_CE_INSTANCE, + GSP_PHYS_RMCALL_BUF_CE_RAMFC, + GSP_PHYS_RMCALL_BUF_CE_MTHDBUF, + GSP_PHYS_RMCALL_BUF_CE_PB, GSP_PHYS_RMCALL_BUF_COUNT }; @@ -1657,6 +1762,20 @@ enum gsp_phys_rmcall_buf { #define GSP_PHYS_RMCALL_TIMEOUT_MS 2000u #define GSP_PHYS_RMCALL_POLL_MS 5u /* + * SESSION 148 perf (termbin review, verified in-tree): every RPC reply wait + * polled with msleep(5). With several quanta per round trip and ~8 + * serialized PDE_UPDATE forwards per forwarded GR launch, the 5 ms quantum + * dominated per-launch latency (~95% of the ~189 ms host service time per + * launch). Wait with sub-millisecond quanta instead: the reply is observed + * within one quantum, cutting each round trip to ~1 ms. POLL_MS remains for + * the poll accounting prints. + */ +#define GSP_PHYS_RMCALL_POLL_US 150u +static inline void gsp_phys_rmcall_wait_quantum(void) +{ + usleep_range(GSP_PHYS_RMCALL_POLL_US, GSP_PHYS_RMCALL_POLL_US + 200u); +} +/* * GR launch release poll window. A CE copy completes in single-digit ms, but * a GR launch can force the engine's first context switch for the channel; * the FECS ctxsw path is measured in fractions of a second, so the window is @@ -1708,6 +1827,19 @@ enum gsp_phys_rmcall_buf { #define GSP_PHYS_RMCALL_CE_SUBCHANNEL 4u /* + * SESSION 153-CONTINUED-68: SEC_OP_ONE_INC = 5 (class/cl906f.h:181 + * NV906F_DMA_SEC_OP_ONE_INC, NVC06F_DMA_ONEINCR_* field layout). The + * client emits its MME IR uploads (method 0x114) as op=5 runs whose first + * data word is the IR instruction address and whose remaining words feed + * the IR data port (the r1099/r1116 captures: hdr 0xa0100045 = op 5, + * cnt 16, method 0x114, data [pc, code...]). op=1 (INC_METHOD) walks the + * METHOD address per data word instead (0x114 -> 0x118 -> 0x11c -> + * 0x120), so an IR upload emitted with op=1 writes one stray IR word and + * then clobbers the SA pointer/data registers. + */ +#define GSP_PHYS_RMCALL_METHOD_ONEINC 5u + +/* * CE (TURING_DMA_COPY_A 0xc5b5) method offsets and field values, from * class/clc5b5.h: SET_OBJECT 0x0, SET_SEMAPHORE_A/B/PAYLOAD 0x240/0x244/0x248, * LAUNCH_DMA 0x300, OFFSET_IN_UPPER/LOWER 0x400/0x404, OFFSET_OUT_UPPER/ @@ -2355,6 +2487,16 @@ static struct gsp_phys_boot_stage { u32 rmcall_gr_promote_size; u32 rmcall_gr_hvirt; /* mode-6 NV01_MEMORY_VIRTUAL handle */ u32 rmcall_token; /* GET_WORK_SUBMIT_TOKEN result */ + /* + * SESSION 153-CONTINUED-96: secondary COPY0 lane (same VAS). Alloc'd + * and scheduled only on a GR0 ladder (CE class is illegal on the GR + * channel); rmcall_ce_lane_valid means the token was obtained and the + * lane can accept CE copy blocks. Zero on a COPY0 ladder. + */ + u32 rmcall_ce_tsg; /* KEPLER_CHANNEL_GROUP_A handle */ + u32 rmcall_ce_channel; /* TURING_CHANNEL_GPFIFO_A handle */ + u32 rmcall_ce_token; /* its GET_WORK_SUBMIT_TOKEN result */ + bool rmcall_ce_lane_valid; /* alloc + BIND/schedules + token done */ u32 rmcall_notifier_mem; /* FBMEM error-notifier object handle */ u64 rmcall_notifier_base; /* its returned frame-buffer address */ bool rmcall_notifier_valid; /* the FBMEM notifier alloc succeeded */ @@ -2447,6 +2589,18 @@ static struct gsp_phys_boot_stage { */ struct gsp_phys_boot_chunk rmcall_gr_qmdwin; u64 rmcall_gr_qmdwin_va; /* planted VA; 0 = guest destination */ + /* + * SESSION 153-CONTINUED-66: page 2 of the module pushbuffer plus the + * dedicated warm-semaphore probe page. Allocated lazily by + * gsp_phys_rmcall_pdb_build() and planted in the ladder's 128 B PTE + * line at slots 0x102 (VA_PB2) and 0x10a (VA_GR_PROBE), before the + * engine's first fetch. Page 0 continues the registered BUF_PB chunk + * so stage 2's registration and every existing pushbuffer user are + * unchanged; only the >4 KB GR launch block reaches into these pages. + */ + struct gsp_phys_boot_chunk rmcall_gr_pb2; + struct gsp_phys_boot_chunk rmcall_gr_probe; + bool rmcall_gr_pb2_planted; /* m104: module page holding the injected mail_kernel message. */ struct gsp_phys_boot_chunk rmcall_gr_mail; /* @@ -2623,6 +2777,18 @@ EXPORT_SYMBOL_GPL(gsp_phys_boot_rm_flags); static void *gsp_phys_chunk_page_va(const struct gsp_phys_boot_chunk *c, u32 page) { + /* + * SESSION 141: an unallocated chunk (no per-page array AND no base + * va) must answer NULL. It previously returned + * (u8 *)NULL + page*PAGE_SIZE - a small non-NULL garbage pointer - + * so the caller's NULL guard passed and the empty-set post-mortem + * print dereferenced it: Oops #2 RIP + * gsp_phys_backend_guest_gr_submit+0x4394, CR2=0x1e93c with slot 30 + * (0x1e000 = 30*0x1000 + 0x900 off). min_data=0's no-external-data + * forward hit this on its first pass. + */ + if (!c->page_va && !c->va) + return NULL; if (c->page_va) return c->page_va[page]; return (u8 *)c->va + (size_t)page * GSP_PHYS_PAGE_SIZE; @@ -3169,6 +3335,10 @@ static void gsp_phys_rmcall_reset(void) boot_stage.rmcall_gr_promote_size = 0; boot_stage.rmcall_gr_hvirt = 0; boot_stage.rmcall_token = 0; + boot_stage.rmcall_ce_tsg = 0; + boot_stage.rmcall_ce_channel = 0; + boot_stage.rmcall_ce_token = 0; + boot_stage.rmcall_ce_lane_valid = false; boot_stage.rmcall_notifier_mem = 0; boot_stage.rmcall_notifier_base = 0; boot_stage.rmcall_notifier_valid = false; @@ -3204,6 +3374,11 @@ static void gsp_phys_rmcall_reset(void) memset(&boot_stage.rmcall_gr_qmdwin, 0, sizeof(boot_stage.rmcall_gr_qmdwin)); boot_stage.rmcall_gr_qmdwin_va = 0; + memset(&boot_stage.rmcall_gr_pb2, 0, + sizeof(boot_stage.rmcall_gr_pb2)); + memset(&boot_stage.rmcall_gr_probe, 0, + sizeof(boot_stage.rmcall_gr_probe)); + boot_stage.rmcall_gr_pb2_planted = false; memset(&boot_stage.rmcall_gr_shadow_win0, 0, sizeof(boot_stage.rmcall_gr_shadow_win0)); boot_stage.rmcall_gr_shadow_win0_used = 0; @@ -4911,6 +5086,773 @@ static u32 gsp_phys_engine_type_from_errstr(const char *err) } /* + * SESSION 153-CONTINUED-12: last published GR launch identity, set by the + * single-inflight submit path so a PBDMA fault dump can name the entry the + * PBDMA is stuck on (block length in dwords, guest push VA, object handle). + */ +static u32 gsp_phys_gr_last_words; +static u32 gsp_phys_gr_last_h; +static u64 gsp_phys_gr_last_push_va; + +/* + * SESSION 153-CONTINUED-12: on a robust-channel PBDMA error (exceptType + * 0x20) dump every PBDMA's latched instruction state and, for the PBDMA + * whose PB_FETCH sits inside the module pushbuffer page, the raw block words + * around the fetch point plus the whole block with indices. The latched + * raw instruction (HDR_SHADOW) can be matched against the block dump to + * name the exact word the PBDMA rejected ("invalid method encoding"); the + * fetch position shows how far the walk got. Read-only MMIO + a racy read + * of the pb page (best effort, diagnostic only); bounded per boot. + */ +#define GSP_PHYS_PBDMA_DUMP_MAX 64u + +/* + * SESSION 153-CONTINUED-52: the published method block's DMA address, so the + * PBDMA fault dump can name the exact word the PBDMA stopped on + * (offset = PB_FETCH - last block DMA). Set beside gsp_phys_gr_last_words. + */ +static u64 gsp_phys_gr_last_pb_dma; + +static const char *gsp_phys_pbdma_type_name(u32 t) +{ + static const char *names[8] = { "SSDM", "INC", "STORE_SDM", "NON_INC", + "IMMD", "INC_ONCE", "USE_SDM", "END_SEG" }; + + return names[t & 7u]; +} + +/* + * SESSION 153-CONTINUED-52: MME fault forensics. The C597 replay's dominant + * failure on draw-bearing windows is a real MME exception + * (NV_PGRAPH_PRI_MME_HWW_ESR 0x404490 latched ILLEGAL_OPCODE / + * MISSING_MACRO_DATA / ... - "unknown instruction fetched from the MME + * Instruction RAM", see gr_error_gk104.c grErrorLogMmeException). INFO + * carries the faulting MME PC and the macro address; INFO2 the instruction + * register. Read them post-failure (the latch survives) so the exact macro + * fetch point is on record. Read-only; bounded by its call sites. + */ +static void gsp_phys_mme_regs_log(void __iomem *bar0, const char *tag) +{ + u32 esr, info, info2, info3, info4, trapped; + + if (!bar0) + return; + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + /* SESSION 153-CONTINUED-106: INFO2 is 0x40449c (0x404498 is CONFIG). */ + info2 = ioread32(bar0 + 0x0040449cu); + info3 = ioread32(bar0 + 0x004044a8u); + info4 = ioread32(bar0 + 0x004044acu); + trapped = ioread32(bar0 + 0x00400704u); + pr_warn("vgsp-gr: MME-REGS %s esr=%08x(pc_valid=%u illegal=%u miss=%u extra=%u) pc=%u macro_addr=0x%x ir=%08x,%08x,%08x trapped=%08x\n", + tag, esr, (info >> 28) & 1u, (esr >> 2) & 1u, + (esr >> 0) & 1u, (esr >> 1) & 1u, info & 0xfffu, + (info >> 12) & 0xffu, info2, info3, info4, trapped); +} + +/* + * SESSION 153-CONTINUED-107: MME IRAM debug port readback. + * + * CONT-106 found the debug port at 0x4044d0 host-side; the definitive field + * spec (tu104 dev_graphics_nobundle.h:2166 NV_PGRAPH_PRI_MME_DEBUG) is + * bit 0 STATUS (R; 0=RUNNING, 1=STOPPED) + * bit 1 ENABLE (RW; init DISABLED) + * bit 2 IRAM_RD (write 1 = TRIGGER, reads 0 = DONE) + * bits 26:15 IRAM_PC (RW, 12 bits = the IRAM word address) + * and the fetched group lands in NV_PGRAPH_PRI_MME_HWW_ESR_INFO4/3/2 + * (0x4044ac/a8/9c, field IR 31:0 each). CONT-106's r1364 read at PC 0 + * returned the reconstructed image's first three words + * (0x00003463,0x190d82f0,0x39804300), so the data path is the INFO + * triplet. The read is a two-step hardware handshake: set ENABLE, write + * the PC with IRAM_RD=1 (trigger), WAIT for IRAM_RD to read back 0 + * (DONE), then read the group. `a` is an IRAM WORD address (callers pass + * instruction * 3, matching gsp_phys_gr_mme_ir_pack[] indexing). + * Read-only diagnostic; bounded by its call sites. + */ +static void gsp_phys_mme_iram_read3(void __iomem *bar0, u32 a, u32 out[3]) +{ + u32 i; + + iowrite32(0x2u, bar0 + 0x004044d0u); /* ENABLE */ + iowrite32(((a & 0xfffu) << 15) | 6u, bar0 + 0x004044d0u); + for (i = 0; i < 1000u; i++) { + if (!(ioread32(bar0 + 0x004044d0u) & 0x4u)) + break; + cpu_relax(); + } + out[0] = ioread32(bar0 + 0x004044acu); + out[1] = ioread32(bar0 + 0x004044a8u); + out[2] = ioread32(bar0 + 0x0040449cu); +} + +/* + * Defined next to the packed-image array (gsp_phys_gr_mme_ir_pack); forward + * declaration so the post-doorbell forensics spin can call it. + */ +static void gsp_phys_mme_iram_probe(void __iomem *bar0, const char *tag, + u32 h_object); + +/* + * SESSION 153-CONTINUED-107: probe enable (see the sema_gl_mme_iram_probe + * knob beside sema_gl_mme_iram_gate). Declared here for the forensics spin + * above; the matching module_param sits with the gate knob. + */ +static bool gsp_phys_rmcall_gr_mme_iram_probe; + +/* + * SESSION 153-CONTINUED-69: pre-reset MME fault capture. Everything sampled + * so far was post-mortem: by the time the 5 ms launch poll ticks, the + * GSP-RM's FECS exception handler has already read NV_PGRAPH_PRI_MME_HWW_ESR + * (0x404490) - the relayed errString "ESR 0x404490=0x80000004" is the only + * surviving trace of the raw ILLEGAL_OPCODE value - and either cleared the + * latch back to 0x80000000 or left the whole PRI block in the 0xbadfxxxx + * priv-error poison. This bounded spin runs in the submit thread immediately + * after the publish/doorbell (execution starts within the first + * milliseconds) and records EVERY transition of the fault registers at + * MMIO read rate, so the faulting ESR (+INFO pc/macro when the valid bit is + * set, and the instruction register INFO2 that names the rejected + * instruction word) is on the record before any clear. Read-only; bounded + * per boot and per launch (the spin costs at most sema_mme_fast_ms of the + * launch thread's time and stops early once the poison is seen). + */ +static u32 gsp_phys_mme_fast_ms = 400u; +/* r1497: defined with the emission knob below; used by the post-launch + * DS ESR clear. */ +static bool gsp_phys_rmcall_gr_ds_mask_clear; +module_param_named(sema_mme_fast_ms, gsp_phys_mme_fast_ms, uint, 0644); +MODULE_PARM_DESC(sema_mme_fast_ms, + "Spin-sample the MME fault registers right after each GR launch doorbell for this many ms, logging every transition before the GSP-RM clears them (0 = off; default 400)"); + +/* + * SESSION 153-CONTINUED-106: post-timeout re-spin. The r1363 chronology + * shows the MME ILLEGAL_OPCODE relay arriving up to ~2.6 s AFTER the sem + * wait failed (launch failed 403088.34, ILLEGAL_OPCODE relayed 403090.96); + * every launch-window spin had already ended, so the latch was never seen + * with its INFO/IR. This bounded second spin runs on the first timeout + * exits of a boot, exactly where the r1363 trap sat. + */ +static u32 gsp_phys_mme_post_ms = 2000u; +module_param_named(sema_mme_post_ms, gsp_phys_mme_post_ms, uint, 0644); +MODULE_PARM_DESC(sema_mme_post_ms, + "Post-timeout MME fault-register re-spin length in ms on the first GR launch-timeout exits (0 = off; default 2000)"); + +/* + * SESSION 153-CONTINUED-83: store-side emission forensics. + * + * The store wall: draws retire but `output retirement wait: 0/N pages + * changed` and gen0=0 (no MME-generated method witnessed). The client's + * Turing draw is EMITTED BY AN MME MACRO (slot 34 = the draw macro: + * BEGIN(0x1618) -> SET_VERTEX_ARRAY_START(0xd74) -> END(0x1614), per the + * offline replay of sa_full_stream2.bin and the nvk reference builder), so + * "no stores" == "the draw macro never emitted". This probe answers three + * questions with one line, at the launch boundary: + * + * 1. DS_HWW_ESR (0x405840) + the per-stage SPH_INFO snapshot + * (0x405c00+si*4 START / 0x405c24+si*4 UPPER / 0x405c1c,c20 REGION): + * which pipeline stages failed SPH validation / what VA they fetched. + * 2. MME_HWW_ESR_INFO (0x404494) PC + MACRO_ADDRESS + PC_VALID, + * INFO_EX (0x4044a4) + IR words INFO2/3/4 (0x40449c/a8/ac): did the + * MME actually execute a macro, and where did it stop. + * 3. FE_DECODE_STATE_EX(i) (0x40a630+i*4): MME_PC 11:0, MACRO 21:12, + * GENERATED 31 - the generated-method witness, all four lanes, at + * the launch end (the retire-time read races the release writes). + * + * Bounded: 24 calls per boot, beef family only, one line per call. + */ +/* + * SESSION 153-CONTINUED-83: MME shadow-RAM read port (bar0 0x404488 index, + * 0x40448c data). The MME tracks class methods in its shadow RAM when the + * class's SET_MME_SHADOW_RAM_CONTROL mode selects tracking; reading the + * port tells us (a) whether the tracked window's methods ever reached the + * MME and (b) what values the MME holds for the draw-relevant methods. + * A read is: index = nvclass(15:0) | method_addr(27:16) | READ(30), then + * read data. Bounded by the caller; host-side diagnostic only. + */ +static u32 gsp_phys_mme_shadow_read(void __iomem *bar0, u32 nvclass, u32 mthd) +{ + iowrite32(nvclass | ((mthd & 0xfffu) << 16) | (1u << 30), + bar0 + 0x00404488u); + return ioread32(bar0 + 0x0040448cu); +} + +static void gsp_phys_store_side_log(void __iomem *bar0, const char *tag, + u32 h_object) +{ + static u32 ss_n; + u32 si, sph[6], sphu[6], rr_a, rr_b, ds, ds2, mminfo, mmex; + u32 mm2, mm3, mm4, fex[4], fed[4], fds[4]; + + if (!bar0 || ss_n >= 40u) + return; + ss_n++; + for (si = 0; si < 6u; si++) { + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + sphu[si] = ioread32(bar0 + 0x00405c24u + si * 4u); + } + for (si = 0; si < 4u; si++) { + fex[si] = ioread32(bar0 + 0x0040a630u + si * 4u); + fed[si] = ioread32(bar0 + 0x0040a620u + si * 4u); + fds[si] = ioread32(bar0 + 0x0040a640u + si * 4u); + } + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + ds = ioread32(bar0 + 0x00405840u); + ds2 = ioread32(bar0 + 0x00405848u); + mminfo = ioread32(bar0 + 0x00404494u); + mmex = ioread32(bar0 + 0x004044a4u); + mm2 = ioread32(bar0 + 0x0040449cu); + mm3 = ioread32(bar0 + 0x004044a8u); + mm4 = ioread32(bar0 + 0x004044acu); + pr_info("vgsp-store: %s h=0x%x ds=%08x/%08x sph=%08x,%08x,%08x,%08x,%08x,%08x sphU=%08x,%08x,%08x,%08x,%08x,%08x rr=%08x,%08x\n", + tag, h_object, ds, ds2, + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + sphu[0], sphu[1], sphu[2], sphu[3], sphu[4], sphu[5], + rr_a, rr_b); + pr_info("vgsp-store: %s h=0x%x mme_esr_info=%08x(pcv=%u pc=%u mac=0x%x) ex=%08x ir=%08x,%08x,%08x fed=%08x,%08x,%08x,%08x fex=%08x,%08x,%08x,%08x\n", + tag, h_object, mminfo, (mminfo >> 28) & 1u, mminfo & 0xfffu, + (mminfo >> 12) & 0xffu, mmex, mm2, mm3, mm4, + fed[0], fed[1], fed[2], fed[3], + fex[0], fex[1], fex[2], fex[3]); + /* + * MME shadow-RAM readback for the draw-relevant C597 methods (tracked + * methods show the last value the MME saw; zeros = never tracked). + * Also probe the MME setup pair and the draw trigger/vertex-start. + */ + pr_info("vgsp-store: %s h=0x%x shadow[c597] 0x114=%08x 0x11c=%08x 0x124=%08x 0x260=%08x 0x840=%08x 0xd74=%08x 0xd78=%08x 0x1618=%08x\n", + tag, h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x114u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x11cu), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x124u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x260u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd74u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd78u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x1618u)); + /* + * r1453 capture (LOG-ONLY, zero behaviour change): the engine's own + * macro-state shadow values for the slow-path STATE 0xD24/0xD25 + * inputs. d24 = SET_MME_SHADOW_SCRATCH(0x24) = method 0x3490, d25 = + * scratch 0x25 = method 0x3494 (the client/X-lane never writes + * either, so the MME shadow is the only non-blind source of these + * values). 0xd24/0xd25 are read as literal method indices through + * the same shadow port (same helper; masked to 12 bits) so the + * tracker interpretation is captured alongside the write methods. + * Rate-limited to 8 lines per boot, no new module parameter; the + * reads use the existing shadow-read path only (no emission change). + */ + { + static u32 d24_d25_cap_n; + + if (d24_d25_cap_n < 8u) { + d24_d25_cap_n++; + pr_info("vgsp-store: %s h=0x%x shadow[c597] d24d25 0x3490=%08x 0x3494=%08x 0xd24=%08x 0xd25=%08x\n", + tag, h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3490u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3494u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd24u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd25u)); + } + } + /* + * SESSION 153-CONTINUED-83 probe 2: the CALL trigger methods the + * window uses (are the calls even reaching the MME?), a repeat read + * (does a read clear?), and the dword-address interpretation of the + * draw/RT methods (0xd74>>2, 0x840>>2) to pin the index units. + */ + pr_info("vgsp-store: %s h=0x%x shadow2[c597] 0x3800=%08x 0x3810=%08x 0x3910=%08x 0x3938=%08x 0x3940=%08x 0x840r=%08x 0xd74dw=%08x 0x840dw=%08x 0x1618r=%08x\n", + tag, h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3800u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3810u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3910u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3938u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x3940u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0xd74u >> 2), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x840u >> 2), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x1618u >> 2)); +} + +static bool gsp_phys_mme_fast_sample(void __iomem *bar0, const char *tag, + u32 h_object, u32 ms_override) +{ + static u32 fast_boot_n; + u32 ms = ms_override ? ms_override : READ_ONCE(gsp_phys_mme_fast_ms); + u32 esr_p, ds_p, gf_p, tr_p, n = 0; + u32 fed0_p, fed1_p; + u32 ds_log_n = 0; + ktime_t t_end; + bool poisoned = false; + + /* + * SESSION 153-CONTINUED-106: the budget was 48, which a beef-lane boot + * with the wedged-engine relaunch train exhausts long before the + * first MME trap is relayed (r1363: 27 launch spins used by +300 s, + * trap traffic still arriving). Raised so every beef-family launch + * of a diagnostic boot keeps its spin. + */ + if (!bar0 || !ms || fast_boot_n >= 256u) + return false; + /* + * Beef-family channels get the full budget (the draw-class windows + * are the ones with the MME fault); other channels are sampled only + * a few times so a fast X-channel launch train cannot consume it. + */ + if ((h_object & 0xfffff000u) != 0xbeef0000u && fast_boot_n >= 2u) + return false; + fast_boot_n++; + esr_p = ioread32(bar0 + 0x00404490u); + ds_p = ioread32(bar0 + 0x00405840u); + gf_p = ioread32(bar0 + 0x00400504u); + tr_p = ioread32(bar0 + 0x00400704u); + /* + * SESSION 153-CONTINUED-109: the poison episode can already be + * latched when the sample's FIRST read happens. r1378's first + * window+live launch: `MME-FAST begin launch ... esr=badf1201 + * ds=badf1201 grfifo=badf1201 trap=badf1201` with the very next + * read clean (`esr badf1201->80000000 ... grfifo badf1201-> + * 00000001`). The historical detector only marks `poisoned` when a + * POST-entry read shows poison, so the episode was never handed to + * the WEDGE-RECOVER decision - no recovery ran, the prelude latch + * and live_gen stayed set, and every later calls-bearing window + * executed macro 57 against the wiped MME context: sked + * `mme_esr=80000004 (ILLEGAL_OPCODE) mme_mac=0x39`, then the FE + * parked at pc=513/mac=0xf1 and the stores stopped (0/905 after the + * 305-page burst). Treat an entry read of poison as the same + * witness; the FECS has already reset the context, so the recovery + * (latch clear + cursor rewind + live_gen invalidate) is exactly + * what is owed. + */ + if (((gf_p & 0xffff0000u) == 0xbadf0000u) || + ((esr_p & 0xffff0000u) == 0xbadf0000u) || + ((ds_p & 0xffff0000u) == 0xbadf0000u)) + poisoned = true; + /* + * SESSION 153-CONTINUED-84: FE_DECODE_STATE_EX(0)/(1) also drive a + * transition log - the MME's macro/generated witness can flip + * there without touching ESR/DS/grfifo, which is exactly the + * "did the MME start a macro after this doorbell" question the + * reloaded windows keep failing. + */ + fed0_p = ioread32(bar0 + 0x0040a630u); + fed1_p = ioread32(bar0 + 0x0040a634u); + pr_info("vgsp-gr: MME-FAST begin %s h=0x%x esr=%08x ds=%08x grfifo=%08x trap=%08x fe0=%08x fe1=%08x (spin %u ms)\n", + tag, h_object, esr_p, ds_p, gf_p, tr_p, fed0_p, fed1_p, ms); + /* + * SESSION 153-CONTINUED-107: IRAM chronology probe (see the + * gsp_phys_mme_iram_probe comment). Bounded to the first 8 fast + * samples of the boot so the launch train cannot flood; knob-gated + * because the debug-port writes disturb a live launch (see the + * sema_gl_mme_iram_probe comment). + */ + if (fast_boot_n <= 8u && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_probe)) + gsp_phys_mme_iram_probe(bar0, "begin", h_object); + t_end = ktime_get() + (s64)ms * 1000000ll; + for (;;) { + u32 esr = ioread32(bar0 + 0x00404490u); + u32 ds = ioread32(bar0 + 0x00405840u); + u32 gf = ioread32(bar0 + 0x00400504u); + u32 tr = ioread32(bar0 + 0x00400704u); + u32 fed0 = ioread32(bar0 + 0x0040a630u); + u32 fed1 = ioread32(bar0 + 0x0040a634u); + + if (esr != esr_p || ds != ds_p || gf != gf_p || tr != tr_p || + fed0 != fed0_p || fed1 != fed1_p) { + if (n < 12u) { + u32 info = ioread32(bar0 + 0x00404494u); + /* + * SESSION 153-CONTINUED-106: the IR word + * registers are 0x40449c (INFO2), 0x4044a8 + * (INFO3), 0x4044ac (INFO4). The old read at + * 0x404498 was NV_PGRAPH_PRI_MME_CONFIG + * (RW), which is why every "info2_ir" line so + * far showed the shadow-filter enable, not the + * rejected instruction. + */ + u32 info2 = ioread32(bar0 + 0x0040449cu); + u32 info3 = ioread32(bar0 + 0x004044a8u); + u32 info4 = ioread32(bar0 + 0x004044acu); + u32 mmeir = ioread32(bar0 + 0x0040a630u); + u32 mmeir1 = ioread32(bar0 + 0x0040a634u); + u32 mmedata = ioread32(bar0 + 0x00400708u); + /* + * SESSION 153-CONTINUED-106: the MME's own + * macro PC/state at the transition instant + * (0x4044b4: PC 11:0, MACRO 19:12, RUNNING + * 20) - the "is the MME mid-macro at the + * trap" witness. + */ + u32 mmemac = ioread32(bar0 + 0x004044b4u); + + /* + * r1151 (SESSION 153-CONTINUED-72): + * GR-STATE: the FE decode-state EX slot 0 + * records the last MME-GENERATED method + * write (bit31 generated, 21:12 macro + * address, 11:0 pc). gen0=1 with the + * kept slot's macro address is the + * "kept macro executes" witness during + * the launch; slot 1 is the second + * lane of the same record set. + */ + pr_warn("vgsp-gr: MME-FAST %s h=0x%x esr %08x->%08x info=%08x(pc_valid=%u pc=%u mac=0x%x) ir=%08x,%08x,%08x ds %08x->%08x grfifo %08x->%08x trap=%08x data=%08x fed=%08x,%08x (pc0=%u mac0=0x%x gen0=%u) fe1=%08x (pc1=%u mac1=0x%x gen1=%u)\n", + tag, h_object, esr_p, esr, + info, (info >> 28) & 1u, + info & 0xfffu, + (info >> 12) & 0xffu, + info2, info3, info4, + ds_p, ds, gf_p, gf, tr, + mmedata, mmeir, mmeir1, + mmeir & 0xfffu, + (mmeir >> 12) & 0x3ffu, + (mmeir >> 31) & 1u, + fed1, + fed1 & 0xfffu, + (fed1 >> 12) & 0x3ffu, + (fed1 >> 31) & 1u); + pr_warn("vgsp-gr: MME-FAST %s h=0x%x macro-status=%08x(mpc=%u mac=0x%x run=%u)\n", + tag, h_object, mmemac, + mmemac & 0xfffu, + (mmemac >> 12) & 0xffu, + (mmemac >> 20) & 1u); + /* + * SESSION 153-CONTINUED-107: sample the live + * IRAM at every fault-register transition, so + * the chronology (words loaded early vs wiped + * by the poison/DROP machinery) is on record + * for the same instant the ESR changes. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_iram_probe)) + gsp_phys_mme_iram_probe(bar0, "trans", + h_object); + } + /* + * SESSION 153-CONTINUED-106: when the DS latch CHANGES + * into a fault (the SPH validation miss at draw + * dispatch), read the per-stage SPH_INFO START/UPPER + * and PROGRAM_REGION pair IMMEDIATELY - the FECS + * clears the whole block within ~1 ms (r1364: + * 404004.338 poison seen, 404004.339 clean), so this + * is the only in-flight window that names the VA each + * failing stage fetched. + */ + if (ds != ds_p && (ds & 0x7fffffffu) && ds_log_n < 8u) { + ds_log_n++; + u32 s0 = ioread32(bar0 + 0x00405c00u); + u32 s1 = ioread32(bar0 + 0x00405c04u); + u32 s2 = ioread32(bar0 + 0x00405c08u); + u32 s3 = ioread32(bar0 + 0x00405c0cu); + u32 s4 = ioread32(bar0 + 0x00405c10u); + u32 s5 = ioread32(bar0 + 0x00405c14u); + u32 u0 = ioread32(bar0 + 0x00405c24u); + u32 u1 = ioread32(bar0 + 0x00405c28u); + u32 u2 = ioread32(bar0 + 0x00405c2cu); + u32 u3 = ioread32(bar0 + 0x00405c30u); + u32 u4 = ioread32(bar0 + 0x00405c34u); + u32 u5 = ioread32(bar0 + 0x00405c38u); + u32 rr_a = ioread32(bar0 + 0x00405c1cu); + u32 rr_b = ioread32(bar0 + 0x00405c20u); + + pr_warn("vgsp-gr: DS-FAST %s h=0x%x ds %08x->%08x stage=%08x,%08x,%08x,%08x,%08x,%08x upper=%02x,%02x,%02x,%02x,%02x,%02x region=%08x,%08x esr=%08x\n", + tag, h_object, ds_p, ds, + s0, s1, s2, s3, s4, s5, + u0 & 0xffu, u1 & 0xffu, u2 & 0xffu, + u3 & 0xffu, u4 & 0xffu, u5 & 0xffu, + rr_a, rr_b, + ioread32(bar0 + 0x00404490u)); + /* + * SESSION 153-CONTINUED-106: the pending DS + * report poisons the whole PRI block + * (every register reads 0xbadf1201 - the + * first-capture above shows exactly that). + * Clear the report masks and re-read: the + * CONT-91 rule says a DS miss with the masks + * cleared latches without the poison, so the + * per-stage SPH snapshot should become + * readable. + */ + { + u32 m0 = ioread32(bar0 + 0x00405844u); + u32 m2 = ioread32(bar0 + 0x0040584cu); + + iowrite32(0u, bar0 + 0x00405844u); + iowrite32(0u, bar0 + 0x0040584cu); + udelay(80); + pr_warn("vgsp-gr: DS-FAST2 %s h=0x%x masks=%08x/%08x ds_after=%08x stage=%08x,%08x,%08x,%08x,%08x,%08x upper=%02x,%02x,%02x,%02x,%02x,%02x region=%08x,%08x\n", + tag, h_object, m0, m2, + ioread32(bar0 + 0x00405840u), + ioread32(bar0 + 0x00405c00u), + ioread32(bar0 + 0x00405c04u), + ioread32(bar0 + 0x00405c08u), + ioread32(bar0 + 0x00405c0cu), + ioread32(bar0 + 0x00405c10u), + ioread32(bar0 + 0x00405c14u), + ioread32(bar0 + 0x00405c24u) & 0xffu, + ioread32(bar0 + 0x00405c28u) & 0xffu, + ioread32(bar0 + 0x00405c2cu) & 0xffu, + ioread32(bar0 + 0x00405c30u) & 0xffu, + ioread32(bar0 + 0x00405c34u) & 0xffu, + ioread32(bar0 + 0x00405c38u) & 0xffu, + ioread32(bar0 + 0x00405c1cu), + ioread32(bar0 + 0x00405c20u)); + } + } + n++; + esr_p = esr; + ds_p = ds; + gf_p = gf; + tr_p = tr; + fed0_p = fed0; + fed1_p = fed1; + /* + * The priv-error poison is the terminal state: the + * FECS has already logged and the rest of the + * registers are 0xbadfxxxx. Stop early so the launch + * thread resumes its normal poll. + */ + if ((gf & 0xffff0000u) == 0xbadf0000u) { + poisoned = true; + break; + } + } + if (ktime_after(ktime_get(), t_end)) + break; + } + pr_info("vgsp-gr: MME-FAST end %s h=0x%x events=%u%s\n", + tag, h_object, n, poisoned ? " (poison)" : ""); + /* + * r1497: clear the DS HWW ESR latch (0x405840, W1C) after the + * launch as well as before it. The pre-clear makes each window's + * validation set readable; a draw that latches a6040e24 otherwise + * leaves the pending report to poison the next window's entry state + * (r1496: ds a6040e24 read at the following begin while poison was + * otherwise low). The clear is W1C and host-only, like the mask + * write above, so the A/B knob covers both sites. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_ds_mask_clear) && bar0) { + static u32 ds_post_log; + u32 esr = ioread32(bar0 + 0x00405840u); + + if (esr) { + /* + * SESSION 155 CONT 14: RESET (bit30, RESET_TASK) + * must be set with the W1C clears so the HWW + * latch is actually reset (reference-RM idiom); + * without it the per-draw attribution is unsound. + */ + iowrite32(esr | 0x40000000u, + bar0 + 0x00405840u); + if (ds_post_log < 24u) { + ds_post_log++; + pr_info("vgsp-gr: DS-ESR post-clear=%08x h=0x%x tag=%s\n", + esr, h_object, tag); + } + } + } + /* + * SESSION 153-CONTINUED-107: post-spin IRAM sample - the state left + * when the poison (if any) has just cleared, before the poll/retire + * work touches anything. + */ + if (fast_boot_n <= 8u && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_probe)) + gsp_phys_mme_iram_probe(bar0, "end", h_object); + /* + * SESSION 153-CONTINUED-83 store-side emission forensics: right after + * the launch's retire spin, snapshot the DS shader-program-header + * latches (which pipeline stages failed SPH validation), the MME + * execution state (did the MME run a macro, and where is its PC), and + * the FE decode-state EX slots (did any method decode carry the MME + * GENERATED bit). Bounded per boot (beef-family only). + */ + if ((h_object & 0xfffff000u) == 0xbeef0000u) + gsp_phys_store_side_log(bar0, tag, h_object); + /* + * SESSION 153-CONTINUED-72: return the poison observation to the + * launch site. The r1149 race: the poison flooded during this spin + * and the FECS cleared it before the poll loop's WEDGE-RECOVER check + * read grfifo, so recovery missed, the prelude latch stayed set and + * the engine wedged for the boot. The caller now latches this flag + * into the recovery gate instead of relying on a live register read. + */ + return poisoned; +} + +static void gsp_phys_pbdma_fault_dump(u32 chid) +{ + static u32 dump_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u64 pb_va = boot_stage.rmcall_va_pb; + u32 i; + + if (!bar0 || !gsp_phys_gr_last_words || dump_n >= GSP_PHYS_PBDMA_DUMP_MAX) + return; + dump_n++; + pr_info("vgsp-gr: PB-FAULT-DUMP chid=%u last_block words=%u h=0x%x push=0x%llx pb_va=0x%llx\n", + chid, gsp_phys_gr_last_words, gsp_phys_gr_last_h, + gsp_phys_gr_last_push_va, pb_va); + for (i = 0; i < 12; i++) { + u32 b = 0x40000u + i * 0x2000u; + u32 intr0 = readl(bar0 + b + 0x108u); + u32 gpf = readl(bar0 + b + 0x050u); + u32 pbf = readl(bar0 + b + 0x054u); + u32 pbfhi = readl(bar0 + b + 0x058u); + u32 pbhd = readl(bar0 + b + 0x084u); + u32 pbcnt = readl(bar0 + b + 0x088u); + u32 pbd0 = readl(bar0 + b + 0x08cu); + u32 hdrsh = readl(bar0 + b + 0x118u); + u32 fetch64 = ((u64)(pbfhi & 0xffu) << 32) | (pbf & ~3u); + u32 trig; + const char *trig_src; + s64 off_words = -1; + s64 voff = -1; + + if (!intr0 && !pbf && !hdrsh && !pbhd) + continue; + /* + * SESSION 153-CONTINUED-52: the real driver's rule for the + * instruction that triggered the PBDMA interrupt: if the + * DECODED header (PB_HEADER) type is IMMD or PB_COUNT's value + * field is zero, the raw instruction is PB_DATA0; otherwise it + * is HDR_SHADOW (fifo_tu102.c: "If the TYPE field of + * NV_PPBDMA_PB_HEADER is IMMD or the VALUE field of + * NV_PPBDMA_PB_COUNT is zero, then the raw PB instruction + * stored in NV_PPBDMA_PB_DATA0 is the one that triggered the + * interrupt. Otherwise ... HDR_SHADOW"). + */ + if (((pbhd >> 29) & 7u) == 4u || (pbcnt & 0x1fffu) == 0u) { + trig = pbd0; + trig_src = "DATA0"; + } else { + trig = hdrsh; + trig_src = "HDRSHADOW"; + } + if (gsp_phys_gr_last_pb_dma && + fetch64 >= gsp_phys_gr_last_pb_dma && + fetch64 < gsp_phys_gr_last_pb_dma + + (u64)gsp_phys_gr_last_words * 4u) + off_words = (s64)(fetch64 - gsp_phys_gr_last_pb_dma) / 4; + /* + * SESSION 153-CONTINUED-59: the SAME offset computed against + * the GPU VA the engine fetches (the r1102 WINPROBE/PB-FAULT + * agreement: pbfetch 0x101ebc with pb_va 0x101000 and 943 + * words = exactly the block end). voff >= words is the + * past-the-block-end prefetch position. + */ + voff = (s64)-1; + if (pb_va && fetch64 >= pb_va && + fetch64 < pb_va + (u64)GSP_PHYS_PAGE_SIZE) + voff = (s64)(fetch64 - pb_va) / 4; + pr_info("vgsp-gr: PB-FAULT-DUMP P%u intr0=%08x gpf=%u pbfetch=0x%llx len=%u off=%lld voff=%lld/%u hdr=%08x(%s m=0x%x cnt=%u sub=%u) cnt=%08x data0=%08x shadow=%08x pbhd=%08x(type=%s) TRIG=%08x(%s)\n", + i, intr0, gpf, fetch64, pbfhi >> 11, off_words, voff, + gsp_phys_gr_last_words, + hdrsh, gsp_phys_pbdma_type_name(hdrsh >> 29), + (hdrsh & 0x1fffu) << 2, + (hdrsh >> 16) & 0xfffu, (hdrsh >> 13) & 7u, + pbcnt, pbd0, hdrsh, pbhd, + gsp_phys_pbdma_type_name(pbhd >> 29), + trig, trig_src); + } + { + struct gsp_phys_boot_chunk *pbch = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; + u32 *pw = pbch->pages ? gsp_phys_chunk_page_va(pbch, 0) : NULL; + + if (pw && pb_va) { + u32 n = gsp_phys_gr_last_words < 384u ? + gsp_phys_gr_last_words : 384u; + u32 w; + + for (w = 0; w < n; w += 16) { + u32 m; + + pr_info("vgsp-gr: PB-FAULT-DUMP pb[%3u]= %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + w, pw[w], w + 1 < n ? pw[w + 1] : 0, + w + 2 < n ? pw[w + 2] : 0, w + 3 < n ? pw[w + 3] : 0, + w + 4 < n ? pw[w + 4] : 0, w + 5 < n ? pw[w + 5] : 0, + w + 6 < n ? pw[w + 6] : 0, w + 7 < n ? pw[w + 7] : 0, + w + 8 < n ? pw[w + 8] : 0, w + 9 < n ? pw[w + 9] : 0, + w + 10 < n ? pw[w + 10] : 0, w + 11 < n ? pw[w + 11] : 0, + w + 12 < n ? pw[w + 12] : 0, w + 13 < n ? pw[w + 13] : 0, + w + 14 < n ? pw[w + 14] : 0, w + 15 < n ? pw[w + 15] : 0); + } + /* + * SESSION 153-CONTINUED-59: the block TAIL and the + * words PAST the block end (the r1102 fault fetch + * sits exactly at the end - voff == words). The past + * words are what a prefetching PBDMA decodes next; + * the tail shows whether the block itself ends at an + * entry boundary. + */ + { + u32 words = gsp_phys_gr_last_words; + u32 lo = words > 48u ? words - 48u : 0u; + u32 hi = words + 32u; + + if (hi > 1024u) + hi = 1024u; + for (w = lo; w < hi; w += 16) { + pr_info("vgsp-gr: PB-FAULT-DUMP %s[%3u]= %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + w < words ? "tail" : "past", w, + pw[w], w + 1 < 1024u ? pw[w + 1] : 0, + w + 2 < 1024u ? pw[w + 2] : 0, w + 3 < 1024u ? pw[w + 3] : 0, + w + 4 < 1024u ? pw[w + 4] : 0, w + 5 < 1024u ? pw[w + 5] : 0, + w + 6 < 1024u ? pw[w + 6] : 0, w + 7 < 1024u ? pw[w + 7] : 0, + w + 8 < 1024u ? pw[w + 8] : 0, w + 9 < 1024u ? pw[w + 9] : 0, + w + 10 < 1024u ? pw[w + 10] : 0, w + 11 < 1024u ? pw[w + 11] : 0, + w + 12 < 1024u ? pw[w + 12] : 0, w + 13 < 1024u ? pw[w + 13] : 0, + w + 14 < 1024u ? pw[w + 14] : 0, w + 15 < 1024u ? pw[w + 15] : 0); + } + } + } + } +} + +/* + * SESSION 153-CONTINUED-86: the fault-time register witness. The relayed + * errString names the fault; these reads add the machine state behind it at + * the earliest host observation point (the 0x1006 consume), before the + * FECS/GSP-RM clear the latches: MME_HWW_ESR (0x404490) + INFO (0x404494: + * PC_VALID/PC/MACRO_ADDRESS) + INFO2 (IR word), DS_HWW_ESR (0x405840) + + * ESR_2 (0x405848) + the SPH_INFO stage starts (0x405c00+si*4) that name the + * exact VA each failing pipeline stage fetched its shader program header + * from, the SPH program-region pair (0x405c1c/c20), and the trapped method + * address/data (0x400704/8). GR-class faults only (0xd graphics exception, + * 0x45 class error; PBDMA 0x20 has its own dump), bounded per boot so a fault + * storm cannot flood dmesg. + */ +static void gsp_phys_fault_regs_log(u32 except_type) +{ + static u32 frl_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u32 esr, info, info2, ds, ds2, trap_a, trap_d, rr_a, rr_b, sph[6], si; + + if (!bar0 || frl_n >= 48u) + return; + if (except_type != 0xdu && except_type != 0x45u) + return; + frl_n++; + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + /* SESSION 153-CONTINUED-106: INFO2 is 0x40449c (0x404498 is CONFIG). */ + info2 = ioread32(bar0 + 0x0040449cu); + ds = ioread32(bar0 + 0x00405840u); + ds2 = ioread32(bar0 + 0x00405848u); + for (si = 0; si < 6u; si++) + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + trap_a = ioread32(bar0 + 0x00400704u); + trap_d = ioread32(bar0 + 0x00400708u); + pr_info("vgsp-fault: regs except=0x%x mme_esr=%08x(pcv=%u ill=%u miss=%u extra=%u) mme_info=%08x(pc=%u mac=0x%x) mme_ir=%08x ds=%08x/%08x sph=%08x,%08x,%08x,%08x,%08x,%08x rr=%08x,%08x trap=%08x data=%08x\n", + except_type, esr, (info >> 28) & 1u, (esr >> 2) & 1u, + esr & 1u, (esr >> 1) & 1u, info, info & 0xfffu, + (info >> 12) & 0xffu, info2, ds, ds2, + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + rr_a, rr_b, trap_a, trap_d); +} + +/* * Note every consumed 0x1006 in the bounded last-4 ring, advance the fault * epoch (the relay submit snapshot compares against it) and log the designed * observation line when the fault relay is armed. Decoding and counting are @@ -4926,10 +5868,14 @@ static void gsp_phys_fault_observe(const struct gsp_phys_rx_record *rec, GSP_PHYS_FAULT_NOTE_MAX; gsp_phys_fault_seen++; gsp_phys_fault_epoch++; - if (gsp_phys_fault_relay_enabled()) + if (gsp_phys_fault_relay_enabled()) { pr_info_ratelimited("vgsp-fault: func=0x1006 seq=%u exceptType=0x%x runlistId=%u chid=%u prevXid=0x%x errString=\"%s\"\n", rec->seq_num, out->except_type, out->runlist_id, out->chid, out->prev_xid, out->err); + gsp_phys_fault_regs_log(out->except_type); + } + if (out->except_type == 0x20u) + gsp_phys_pbdma_fault_dump(out->chid); } /* Note every consumed 0x1020 for the C3 corroboration ring. */ @@ -6827,6 +7773,35 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); /* + * Mode-35 KGR static-data snapshots. The user-lane query set + * (gsp_phys_rmcall_user_lane_kgr_queries) is answered by the LIVE GSP with + * the device's true GR static data; the replies are kept here so the + * guest-side emulation can serve the identical bytes instead of fabricating + * GR values. The concrete consumer is the X driver's 3D-engine selection: + * nv3dAllocDevice maps GR_GET_INFO index SM_VERSION (0x0c) through a table + * and fails with "Failed to select a 3D engine" when it reads back as + * SM_VERSION_NONE (0) - the fabricated fill never set it, while the live + * GSP's KGR_GET_INFO reply carries the real value (0x75 for TU104) along + * with every other GR static field (warps, caps, topology, ctx buffers). + * + * One entry per captured cmd; the generation key ties the cache to the + * backend session exactly like gsp_phys_backend_static above, and the + * getter refuses entries once the session has moved on. + */ +#define GSP_PHYS_KGR_SNAP_ENTRIES 10 +#define GSP_PHYS_KGR_SNAP_MAX_SIZE 3712u /* KGR_GET_INFO: 8 engines x 58 x 8 */ + +struct gsp_phys_kgr_snap { + u32 cmd; + u32 size; + bool valid; + u32 generation; + u8 data[GSP_PHYS_KGR_SNAP_MAX_SIZE]; +}; + +static struct gsp_phys_kgr_snap gsp_phys_kgr_snaps[GSP_PHYS_KGR_SNAP_ENTRIES]; + +/* * Drop the cached static-config snapshot. Called when the backend session * leaves READY (or a new claim starts): the cache generation key is bumped * past the current backend generation so a concurrent getter cannot serve the @@ -6835,9 +7810,13 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_static_wait); */ void gsp_phys_backend_static_invalidate(void) { + unsigned int i; + mutex_lock(&gsp_phys_rm_lock); gsp_phys_backend_static.valid = false; gsp_phys_backend_static.generation = gsp_phys_backend_generation() + 1; + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) + gsp_phys_kgr_snaps[i].valid = false; mutex_unlock(&gsp_phys_rm_lock); } @@ -6891,6 +7870,71 @@ int gsp_phys_backend_static_get(struct gsp_phys_backend_static_config *out) } EXPORT_SYMBOL_GPL(gsp_phys_backend_static_get); +/* + * Copy the captured reply for cmd into dst. Returns 0 on success (with + * *out_len set), -ENODATA when nothing was captured for cmd (or the entry's + * generation no longer matches the backend session), -ENOSPC when the + * snapshot does not fit dst_len, -EAGAIN while the backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len) +{ + unsigned int i; + + if (!dst || !out_len) + return -EINVAL; + + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return -EAGAIN; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + struct gsp_phys_kgr_snap *s = &gsp_phys_kgr_snaps[i]; + + if (s->cmd != cmd) + continue; + if (!s->valid || + s->generation != gsp_phys_backend_generation()) + return -ENODATA; + if (s->size > dst_len) + return -ENOSPC; + memcpy(dst, s->data, s->size); + *out_len = s->size; + return 0; + } + return -ENODATA; +} +EXPORT_SYMBOL_GPL(gsp_phys_kgr_snap_get); + +/* + * Record one mode-35 KGR query reply. Called under the boot thread only + * (same thread as the query itself), so no lock is needed beyond the + * generation key. + */ +static void gsp_phys_kgr_snap_store(u32 cmd, const void *data, u32 size) +{ + unsigned int i; + struct gsp_phys_kgr_snap *slot = NULL; + + if (size > GSP_PHYS_KGR_SNAP_MAX_SIZE) + return; + + for (i = 0; i < ARRAY_SIZE(gsp_phys_kgr_snaps); i++) { + if (gsp_phys_kgr_snaps[i].cmd == cmd) { + slot = &gsp_phys_kgr_snaps[i]; + break; + } + if (!slot && !gsp_phys_kgr_snaps[i].valid) + slot = &gsp_phys_kgr_snaps[i]; + } + if (!slot) + return; + + slot->cmd = cmd; + slot->size = size; + slot->generation = gsp_phys_backend_generation(); + memcpy(slot->data, data, size); + slot->valid = true; +} + /* ============================================================ * Host RM client ladder (R1): gsp-phys-rm-call * ============================================================ */ @@ -9691,6 +10735,87 @@ static u64 gsp_phys_rmcall_pde(u64 pa) } /* + * SESSION 153-CONTINUED-92: leaf PTE for the engine-visible guest/scratch + * mappings with the configurable v2 KIND field (bits 63:56). + * + * m96 documented the symptom that motivates this: on Turing a kind=0 PTE is + * what the SM I-fetch refuses with MMU_NACK while the FE tolerates it, and + * `gsp_phys_rmcall_gr_pte_kind` was introduced to plant the win0/identity + * pool PTEs with a kind (UVM turing uses GENERIC_MEMORY 0x06). The knob was + * never applied to the scratch-PT pool mappings - every shadow VA the 3D + * window's program operands are relocated to (the VAs the DS fetches the + * SPH through) carried kind=0. The DS validates a program header, i.e. runs + * a program fetch; if that path is the SM-side one, kind=0 is a candidate + * cause for the SPH rejection that is independent of content (and explains + * why no content A/B ever moved the wall). With the knob 0 this is + * byte-identical to the previous PTE encoding; nonzero = experiment. + */ +static u64 gsp_phys_rmcall_gr_pte_leaf(u64 hpa) +{ + u64 pte = gsp_phys_rmcall_pte(hpa); + int kind = gsp_phys_rmcall_gr_pte_kind_value(); + + if (kind) + pte |= (u64)(u32)kind << 56; + return pte; +} + +/* + * r1497: a VID-aperture leaf pointing at the page's real frame-buffer + * offset. The guest's own page tables encode exactly this + * (VALID=1, APERTURE=VID(0), KIND=6 GENERIC_MEMORY, PA>>12 in [56:8]), + * so the engine reads the real FBMEM page the client uploaded instead of + * an emulated BAR1 shadow that no upload ever populated. + */ +static u64 gsp_phys_rmcall_gr_pte_leaf_vid(u64 fb_off) +{ + return 1ULL | (((fb_off >> 12) & 0x1ffffffULL) << 8) | + ((u64)6u << 56); +} + +/* + * r1497: is the engine-visible shadow page at `hpa` entirely empty? + * Only then may the real-FB VID leaf replace it; pages the module seeded + * (fixed CB), the engine wrote, or the client wrote through the emulated + * aperture keep their shadow identity. + */ +static bool gsp_phys_gr_shadow_page_empty(u64 hpa) +{ + const u8 *p; + u32 i; + + if (!hpa || !pfn_valid((unsigned long)(hpa >> PAGE_SHIFT))) + return false; + p = (const u8 *)__va(hpa); + for (i = 0; i < GSP_PHYS_PAGE_SIZE; i++) + if (p[i]) + return false; + return true; +} + +/* + * r1497: resolve a fetch VA to its real frame-buffer offset when the page + * is FB-aperture AND the emulated shadow copy of it is all-zero. This is + * the active indexed draw's vertex/index class (r1496): the client's page + * tables map the page to vidmem, the upload never reached an emulated + * path, and the engine fetched zeros. Returns true with *fb_off set. + */ +static bool gsp_phys_gr_fetch_vid_off_locked(u64 va, u64 *fb_off) +{ + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1), + &r)) + return false; + if (r.ap != 2 /* SVC_APERTURE_FB */ || !r.hpa) + return false; + if (!gsp_phys_gr_shadow_page_empty(r.hpa)) + return false; + *fb_off = r.pa; + return true; +} + +/* * GP10X leaf PTE for a frame-buffer (VIDEO_MEMORY aperture 0) address: * VALID=1 (bit 0), APERTURE=0 (bits 2:1), VOL=1 (bit 3), frame-buffer byte * offset >> 12 in the VIDEO address field (bits 32:8; dev_mmu.h:592-595, @@ -9820,6 +10945,26 @@ static void gsp_phys_rmcall_pdb_build(void) (u8 *)gsp_phys_chunk_page_va(pt, 0) + (size_t)sem_idx * 8); /* + * SESSION 153-CONTINUED-96: secondary CE lane GPFIFO ring (slot 0x10d) + * and its one-page pushbuffer (slot 0x10e), planted here in the same + * 128 B PTE line as the ladder's own pages - stage 3, before any + * engine fetch (no invalidate exists). Always planted (the lane's + * page/PTE cost is two pages; a disabled lane simply never rings its + * doorbell), so pdb_build stays deterministic. + */ + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_CE_GPFIFO], 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)((GSP_PHYS_RMCALL_VA_CE_GPFIFO >> 12) & 0x1ffu) * 8); + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_chunks[ + GSP_PHYS_RMCALL_BUF_CE_PB], 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)((GSP_PHYS_RMCALL_VA_CE_PB >> 12) & 0x1ffu) * 8); + /* * Rank-2 qmdwin: plant the dedicated module page for the forwarded * inline-QMD destination in this same small-PT line (slot 0x107, VA * 0x107000; slots 0x105/0x106 stay the P6a guest slots below). This @@ -9866,6 +11011,66 @@ static void gsp_phys_rmcall_pdb_build(void) } } /* + * SESSION 153-CONTINUED-66: plant page 2 of the module pushbuffer + * (VA_PB2, small-PT slot 0x102) and the dedicated warm-semaphore + * probe page (VA_GR_PROBE, slot 0x10a) in the same 128 B PTE line, + * at stage 3 before the engine's first fetch (no invalidate exists). + * Page 0 stays the registered BUF_PB chunk (stage 2 unchanged); this + * chunk is engine-reached purely through these PTEs, exactly like + * the qmdwin pair above. Without it the GR launch block keeps the + * single-page bound (<= 4 KB). + */ + if (!boot_stage.rmcall_gr_pb2.pages && boot_stage.pdev) { + if (!gsp_phys_chunk_alloc(boot_stage.pdev, + &boot_stage.rmcall_gr_pb2, 3u)) + gsp_phys_chunk_zero(&boot_stage.rmcall_gr_pb2); + else + pr_warn("vgsp-boot: rm-call: pushbuffer page2/3/4 allocation failed; >4 KB launches keep the single-page block\n"); + } + if (!boot_stage.rmcall_gr_probe.pages && boot_stage.pdev) { + if (!gsp_phys_chunk_alloc(boot_stage.pdev, + &boot_stage.rmcall_gr_probe, 1u)) + gsp_phys_chunk_zero(&boot_stage.rmcall_gr_probe); + else + pr_warn("vgsp-boot: rm-call: warm-sem probe page allocation failed; the probe keeps the pushbuffer upper half\n"); + } + if (boot_stage.rmcall_gr_pb2.pages && boot_stage.rmcall_gr_probe.pages) { + u32 pb2_idx = (u32)((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ffu); + u32 pb3_idx = (u32)((GSP_PHYS_RMCALL_VA_PB3 >> 12) & 0x1ffu); + u32 pb4_idx = (u32)((GSP_PHYS_RMCALL_VA_PB4 >> 12) & 0x1ffu); + u32 pr_idx = (u32)((GSP_PHYS_RMCALL_VA_GR_PROBE >> 12) & + 0x1ffu); + + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb2_idx * 8); + if (boot_stage.rmcall_gr_pb2.pages >= 2u) + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 1)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb3_idx * 8); + if (boot_stage.rmcall_gr_pb2.pages >= 3u) + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_pb2, 2)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pb4_idx * 8); + put_unaligned_le64(gsp_phys_rmcall_pte( + (u64)gsp_phys_chunk_page_dma( + &boot_stage.rmcall_gr_probe, 0)), + (u8 *)gsp_phys_chunk_page_va(pt, 0) + + (size_t)pr_idx * 8); + dma_wmb(); + boot_stage.rmcall_gr_pb2_planted = true; + pr_info("vgsp-boot: rm-call: GR launch pb page2/3/4 + warm-sem probe planted VA=0x%llx/0x%llx/0x%llx/0x%llx ptSlots=0x%x/0x%x/0x%x/0x%x (up to four-page block; before first fetch)\n", + GSP_PHYS_RMCALL_VA_PB2, GSP_PHYS_RMCALL_VA_PB3, + GSP_PHYS_RMCALL_VA_PB4, GSP_PHYS_RMCALL_VA_GR_PROBE, + pb2_idx, pb3_idx, pb4_idx, pr_idx); + } + /* * m69: plant the window-0 shadow pool PTEs (current win0 slot range) in the * same proven small-PT line, before the engine's first fetch. The * launch copies the QMD's program/param page content into these pages @@ -13605,6 +14810,12 @@ static int gsp_phys_rmcall_user_lane_userd(void __iomem *bar0) * (CONTEXT_BUFFERS_INFO in particular) may be prepared by the query. All are * zero-in best-effort getters at the capture's payload sizes; a failure * warns and the lane continues. + * + * Every successful reply is retained via gsp_phys_kgr_snap_store(): the + * guest-side emulation serves these exact bytes for the same commands, so + * the guest's GL/3D stack reads the live GSP's true GR static values + * (SM_VERSION in particular - see the snapshot comment near + * gsp_phys_backend_static_get). */ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) { @@ -13643,6 +14854,8 @@ static void gsp_phys_rmcall_user_lane_kgr_queries(void __iomem *bar0) q[i].name); pr_info("vgsp-boot: rm-call: mode 35 KGR query %s: size=%u status=0x%x rpc_result=0x%x ret=%d\n", q[i].name, q[i].size, status, rpc_result, ret); + if (ret == 0 && rpc_result == 0 && status == 0) + gsp_phys_kgr_snap_store(q[i].cmd, buf, q[i].size); kvfree(buf); } } @@ -13915,6 +15128,16 @@ static int gsp_phys_rmcall_user_lane_main_promote(void __iomem *bar0) status, rpc_result, GSP_PHYS_RMCALL_HANDLE_USER_LANE, boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_va); kvfree(p); + /* + * SESSION 153-CONTINUED-96: the F2 mapped-MAIN promote is the server's + * acceptance of the ladder's own context buffer (PA+VA, bInitialize=1), + * i.e. exactly the condition rmcall_ctx_valid records. Modes 0-34 set it + * at the end of their promote step (line ~13675); mode 35 promotes here, + * so without this the state file read valid=0 and the engine-ctx + * readback gate silently skipped. + */ + if (!ret) + boot_stage.rmcall_ctx_valid = true; return ret; } @@ -14635,6 +15858,35 @@ static int gsp_phys_rmcall_stage3(void __iomem *bar0) boot_stage.rmcall_gr_object = GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR_OBJ; } + /* + * Graphics-execution plan §F first 3D-lane experiment: + * construct the TURING_A (0xc597) class object on V7 + * after the accepted promote, through the same generic + * alloc helper the golden lanes use. The plan's open + * question is whether a 0xc597 construct on the + * execution lane is accepted without the wired-golden + * deadlock class; the result line below is the + * experiment's evidence either way, and the construct + * never gates the lane. + */ + { + u32 g3_rpc = 0, g3_status = 0; + int g3_ret = gsp_phys_rmcall_alloc_gr_object_class(bar0, + GSP_PHYS_RMCALL_CLASS_GR_3D, + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D, + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + "mode 35 TURING_A on V7 (plan-F first 3D-lane experiment, after promote)", + &g3_rpc, &g3_status); + + pr_info("vgsp-boot: rm-call: mode 35 plan-F experiment: TURING_A (0xc597) on V7 hChannel=0x%x -> ret=%d hObject=0x%x status=0x%x rpc_result=0x%x (%s)\n", + GSP_PHYS_RMCALL_HANDLE_USER_LANE, + g3_ret, + g3_ret ? 0 : + GSP_PHYS_RMCALL_HANDLE_USER_LANE_GR3D, + g3_status, g3_rpc, + g3_ret ? "rejected/absent; lane continues" : + "accepted; V7 carries a 3D class object for the method-stream replay"); + } pr_info("vgsp-boot: rm-call: mode 35: execution channel switched from V6 0x%x to V7 0x%x (captured flags=0x%x internalFlags=0x%x hContextShare=0x%x hVASpace=0, gpFifoVa=0x%llx entries=%u, sysmem USERD, sysmem instance/ramfc (m62 readback), TURING_COMPUTE_A 0x%x); V6 stays allocated and unscheduled, stage 4 + forwarding run on V7\n", boot_stage.rmcall_channel, GSP_PHYS_RMCALL_HANDLE_USER_LANE, @@ -14952,23 +16204,31 @@ static u32 gsp_phys_rmcall_userd_ctl_off(void) * launch path publish byte-identically; the CE caller's semaphore clear and * method-block build happen before this call. The caller must hold * gsp_phys_rm_lock. - */ -static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, - u32 words, - struct gsp_phys_rmcall_ce_submit *out) + * + * SESSION 153-CONTINUED-96: the lane carries the channel's GPFIFO ring, + * hardware USERD chunk, out-fill chunks and work-submit token, so the + * secondary COPY0 lane publishes through the identical body on its own + * ring/USERD/doorbell token (a second channel must never advance the GR + * lane's GP_PUT or ring the GR channel's doorbell). + */ +struct gsp_phys_rmcall_lane { + struct gsp_phys_boot_chunk *ring; + struct gsp_phys_boot_chunk *userd; + struct gsp_phys_boot_chunk *pb; + struct gsp_phys_boot_chunk *sem; + struct gsp_phys_boot_chunk *src; + struct gsp_phys_boot_chunk *dst; + u32 token; + const char *tag; +}; + +static int gsp_phys_rmcall_publish_lane_locked(void __iomem *bar0, + u64 pb_gpu_va, u32 words, + const struct gsp_phys_rmcall_lane *lane, + struct gsp_phys_rmcall_ce_submit *out) { - struct gsp_phys_boot_chunk *ring = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO]; - struct gsp_phys_boot_chunk *userd_c = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; - struct gsp_phys_boot_chunk *pb = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; - struct gsp_phys_boot_chunk *sem = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; - struct gsp_phys_boot_chunk *src = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC]; - struct gsp_phys_boot_chunk *dst = - &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST]; + struct gsp_phys_boot_chunk *ring = lane->ring; + struct gsp_phys_boot_chunk *userd_c = lane->userd; u32 *ring_va, *userd; u32 gpput, gpget, slot, e0, e1; u32 runlist_id, ch_id, doorbell; @@ -14981,11 +16241,39 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, gpput = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPPUT_OFF); gpget = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPGET_OFF); if (gpput >= GSP_PHYS_RMCALL_GPFIFO_ENTRIES) { - pr_err("vgsp-boot: rm-call: USERD GP_PUT=%u is outside the %u-entry ring; refusing to submit\n", - gpput, GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + pr_err("vgsp-boot: rm-call: %s USERD GP_PUT=%u is outside the %u-entry ring; refusing to submit\n", + lane->tag, gpput, GSP_PHYS_RMCALL_GPFIFO_ENTRIES); return -EIO; } - slot = gpput; + /* + * SESSION 153-CONTINUED-46: publish at the ENGINE's GP_GET, not at + * the raw GP_PUT. Observed on every stalled beef boot: the USERD + * drifts (GPGet=5 while GPPut cycles 1/2/4/7 - the engine's GET ran + * past the publisher's PUT, e.g. after a wrapped or otherwise + * deranged ring), and a ring is MODULO: PUT < GET numerically means + * (PUT-GET) mod ENTRIES pending entries - hundreds - so the PBDMA + * walks stale slots, executes garbage method blocks and the launch + * never retires (the trapped=0x8000022c / no-dispatch class). The + * forwarded channel is strictly single-inflight, so the engine's + * GET names exactly the next free slot: write our entry there and + * advance PUT to GET+1 so the PBDMA sees exactly one pending entry - + * ours. When GP_GET is out of range (uninitialised USERD), fall back + * to the old PUT-derived slot. + */ + slot = (gpget < GSP_PHYS_RMCALL_GPFIFO_ENTRIES) ? gpget : gpput; + { + static u32 slotfix_log; + + if (gpget != gpput && slotfix_log < 64) { + slotfix_log++; + pr_info("vgsp-boot: rm-call: %s publish at engine GP_GET=%u (raw GP_PUT=%u was %s; pending-would-be=%u)\n", + lane->tag, gpget, gpput, + gpput == gpget ? "equal" : + (gpput > gpget ? "ahead" : "behind"), + (gpput + GSP_PHYS_RMCALL_GPFIFO_ENTRIES - gpget) % + GSP_PHYS_RMCALL_GPFIFO_ENTRIES); + } + } /* * e0: bits 31:2 hold the GET field, which itself is (fetch address @@ -15002,8 +16290,8 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, */ e0 = (u32)pb_gpu_va & ~3u; if (unlikely(e0 != (u32)((pb_gpu_va >> 2) << 2))) - pr_err("vgsp-boot: rm-call: GPFIFO GET encode mismatch: e0=0x%08x decodes to fetch 0x%llx, expected pb VA 0x%llx\n", - e0, (u64)(e0 & ~3u), pb_gpu_va); + pr_err("vgsp-boot: rm-call: %s GPFIFO GET encode mismatch: e0=0x%08x decodes to fetch 0x%llx, expected pb VA 0x%llx\n", + lane->tag, e0, (u64)(e0 & ~3u), pb_gpu_va); e1 = (u32)((pb_gpu_va >> 32) & 0xffu) | (words << 10); put_unaligned_le32(e0, (u8 *)ring_va + slot * 8); put_unaligned_le32(e1, (u8 *)ring_va + slot * 8 + 4); @@ -15012,8 +16300,10 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * GP_PUT advances on the client sysmem USERD chunk, which is the * channel's hardware USERD because the params carry that same chunk; * the host write is therefore what the engine polls (cycle 21). + * CONT-46: PUT = slot+1 (slot was chosen from GP_GET above), so the + * single new entry is the entire pending set. */ - put_unaligned_le32((gpput + 1) % GSP_PHYS_RMCALL_GPFIFO_ENTRIES, + put_unaligned_le32((slot + 1) % GSP_PHYS_RMCALL_GPFIFO_ENTRIES, (u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPPUT_OFF); /* * Doorbell value: workSubmitToken = ((runlistId << 16) & 0x7f00000) | @@ -15022,12 +16312,12 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * write the formula's result (identical unless the token carries stray * high bits, which is logged). */ - runlist_id = (boot_stage.rmcall_token >> 16) & 0x7fu; - ch_id = boot_stage.rmcall_token & 0xfffu; + runlist_id = (lane->token >> 16) & 0x7fu; + ch_id = lane->token & 0xfffu; doorbell = ((runlist_id << 16) & 0x7f00000u) | (ch_id & 0xfffu); - if (doorbell != boot_stage.rmcall_token) - pr_warn("vgsp-boot: rm-call: token 0x%x carries bits outside runlistId[22:16]|chId[11:0]; writing the recomposed doorbell 0x%x\n", - boot_stage.rmcall_token, doorbell); + if (doorbell != lane->token) + pr_warn("vgsp-boot: rm-call: %s token 0x%x carries bits outside runlistId[22:16]|chId[11:0]; writing the recomposed doorbell 0x%x\n", + lane->tag, lane->token, doorbell); /* * C3: snapshot the fault epoch immediately before the doorbell, so a * 0x1006 consumed by this submission's timeout drain is accepted only @@ -15047,14 +16337,33 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, out->doorbell = doorbell; out->runlist_id = runlist_id; out->ch_id = ch_id; - out->pb_dma = (u64)gsp_phys_chunk_page_dma(pb, 0); - out->sem_dma = (u64)gsp_phys_chunk_page_dma(sem, 0); - out->src_dma = (u64)gsp_phys_chunk_page_dma(src, 0); - out->dst_dma = (u64)gsp_phys_chunk_page_dma(dst, 0); + out->pb_dma = (u64)gsp_phys_chunk_page_dma(lane->pb, 0); + out->sem_dma = (u64)gsp_phys_chunk_page_dma(lane->sem, 0); + out->src_dma = (u64)gsp_phys_chunk_page_dma(lane->src, 0); + out->dst_dma = (u64)gsp_phys_chunk_page_dma(lane->dst, 0); } return 0; } +static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, + u32 words, + struct gsp_phys_rmcall_ce_submit *out) +{ + const struct gsp_phys_rmcall_lane lane = { + .ring = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO], + .userd = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD], + .pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB], + .sem = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM], + .src = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC], + .dst = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST], + .token = boot_stage.rmcall_token, + .tag = "ladder", + }; + + return gsp_phys_rmcall_publish_lane_locked(bar0, pb_gpu_va, words, + &lane, out); +} + /* * One CE copy block on the ladder channel: clear the module semaphore, build * the methods for bytes into the module pushbuffer, publish the GPFIFO entry @@ -15076,10 +16385,30 @@ static int gsp_phys_rmcall_publish_locked(void __iomem *bar0, u64 pb_gpu_va, * site passes false, so the default COPY0 ladder emits the same bytes as * before. */ -static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, - u64 dst_va, u64 sem_va, u32 bytes, - u32 sem_payload, bool gr, - struct gsp_phys_rmcall_ce_submit *out) +/* + * SESSION 153-CONTINUED-12: pb_off selects where in the module pushbuffer + * page the method block is written and fetched from. The warm-semaphore + * probe (gsp_phys_rmcall_gr_sem_push_locked) used the upper half so it + * could never overwrite a forwarded launch's pending method block: + * measured with the PBDMA dump, the probe's 10-dword block landing on + * words 0..9 of a pending 139-word 3D replay made the PBDMA walk the + * probe's SET_OBJECT/REPORT_SEM tail and then decode the replay window's + * operand 0x00230200 as an instruction -> PBENTRY on chid 4 and no + * retire. Every forwarded GR/CE block stays at offset 0. + * + * SESSION 153-CONTINUED-66: pb_alt/pb_alt_va point the block at a + * dedicated one-page module buffer (the probe's VA_GR_PROBE page) instead + * of the registered pushbuffer; a forwarded GR block may now span both + * pushbuffer pages, so no fixed offset of that page is safe for the probe. + */ +static int gsp_phys_rmcall_ce_submit_off_locked(void __iomem *bar0, + u64 src_va, u64 dst_va, + u64 sem_va, u32 bytes, + u32 sem_payload, bool gr, + u32 pb_off, + struct gsp_phys_boot_chunk *pb_alt, + u64 pb_alt_va, + struct gsp_phys_rmcall_ce_submit *out) { struct gsp_phys_boot_chunk *pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; @@ -15108,11 +16437,28 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, if (!bar0 || !boot_stage.rmcall_token || !bytes || bytes > GSP_PHYS_RMCALL_COPY_SIZE) return -EINVAL; + if (pb_off & 3u) + return -EINVAL; pb_gpu_va = boot_stage.rmcall_va_pb; if (!pb_gpu_va || (pb_gpu_va & 3)) return -EIO; + /* + * SESSION 153-CONTINUED-66: pb_alt selects a dedicated one-page module + * buffer instead of the registered pushbuffer page (the warm-semaphore + * probe uses its own PTE-planted page now that a forwarded GR block may + * span BOTH pushbuffer pages). pb_off stays relative to that page. + */ + if (pb_alt && pb_alt->pages) { + pb = pb_alt; + pb_gpu_va = pb_alt_va; + if (!pb_gpu_va || (pb_gpu_va & 3)) + return -EIO; + } - pb_va = gsp_phys_chunk_page_va(pb, 0); + if (!gsp_phys_chunk_page_va(pb, 0) || pb_off > GSP_PHYS_PAGE_SIZE) + return -EIO; + pb_va = (u32 *)((u8 *)gsp_phys_chunk_page_va(pb, 0) + pb_off); + pb_gpu_va += pb_off; ring_va = gsp_phys_chunk_page_va(ring, 0); userd = gsp_phys_chunk_page_va(userd_c, 0); if (!pb_va || !ring_va || !userd || @@ -15133,9 +16479,82 @@ static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, dst_va, bytes, sem_payload); + if (pb_off + words * 4 > GSP_PHYS_PAGE_SIZE) + return -EIO; + return gsp_phys_rmcall_publish_locked(bar0, pb_gpu_va, words, out); } +static int gsp_phys_rmcall_ce_submit_locked(void __iomem *bar0, u64 src_va, + u64 dst_va, u64 sem_va, u32 bytes, + u32 sem_payload, bool gr, + struct gsp_phys_rmcall_ce_submit *out) +{ + return gsp_phys_rmcall_ce_submit_off_locked(bar0, src_va, dst_va, + sem_va, bytes, sem_payload, + gr, 0u, NULL, 0, out); +} + +/* + * SESSION 153-CONTINUED-96: one CE copy block on the secondary COPY0 lane. + * Byte-for-byte the CE submission body (same method block, same guards, + * same module-semaphore clear) but on the lane's own GPFIFO ring, USERD + * chunk, one-page pushbuffer and work-submit token. Only legal when the + * lane's bringup obtained a token (rmcall_ce_lane_valid). The caller must + * hold gsp_phys_rm_lock and polls the module semaphore separately, exactly + * like the readback's bounded page loop. + */ +static int gsp_phys_rmcall_ce_lane_submit_locked( + void __iomem *bar0, u64 src_va, u64 dst_va, u64 sem_va, + u32 bytes, u32 sem_payload, + struct gsp_phys_rmcall_ce_submit *out) +{ + const struct gsp_phys_rmcall_lane lane = { + .ring = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_GPFIFO], + .userd = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_USERD], + .pb = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_CE_PB], + .sem = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM], + .src = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SRC], + .dst = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_DST], + .token = boot_stage.rmcall_ce_token, + .tag = "CE lane", + }; + struct gsp_phys_boot_chunk *sem = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; + resource_size_t bar0_len; + u32 *pb_va; + u32 words; + + if (!boot_stage.rmcall_ce_lane_valid) + return -EOPNOTSUPP; + if (!boot_stage.pdev) + return -EINVAL; + bar0_len = pci_resource_len(boot_stage.pdev, 0); + if (bar0_len < GSP_PHYS_RMCALL_DOORBELL_OFF + 4) + return -EIO; + if (!bar0 || !boot_stage.rmcall_ce_token || !bytes || + bytes > GSP_PHYS_RMCALL_COPY_SIZE) + return -EINVAL; + pb_va = gsp_phys_chunk_page_va(lane.pb, 0); + if (!pb_va || !gsp_phys_chunk_page_va(lane.ring, 0) || + !gsp_phys_chunk_page_va(lane.userd, 0) || + !gsp_phys_chunk_page_va(sem, 0)) + return -EIO; + + /* A stale release from a previous submission must not be observed. */ + WRITE_ONCE(*(u32 *)gsp_phys_chunk_page_va(sem, 0), 0); + wmb(); + + words = gsp_phys_rmcall_build_methods(pb_va, sem_va, src_va, dst_va, + bytes, sem_payload); + if (words * 4 > GSP_PHYS_PAGE_SIZE) + return -EIO; + + return gsp_phys_rmcall_publish_lane_locked(bar0, + GSP_PHYS_RMCALL_VA_CE_PB, + words, &lane, out); +} + /* * H1 stage-4 GR verification: one GR semaphore release on the ladder channel, * shared by stage 4's GR branch and the debugfs trigger 9 re-run. The caller @@ -15193,11 +16612,33 @@ static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) unsigned long deadline; int ret; - ret = gsp_phys_rmcall_ce_submit_locked(bar0, GSP_PHYS_RMCALL_VA_SRC, + /* + * SESSION 153-CONTINUED-12: this probe runs while a forwarded launch + * may still be pending in the same pushbuffer page. It must not land + * on words 0.. of that page: measured PB-FAULT-DUMP shows the probe's + * 10-dword block overwriting a pending 139-word 3D replay's head, so + * the PBDMA walks the probe then decodes a replay operand as an + * instruction (PBENTRY, no retire). + * + * SESSION 153-CONTINUED-66: the forwarded GR block now spans BOTH + * pushbuffer pages (up to 2048 dwords = 0x2000 bytes), so the old + * upper-half placement (0x800) would clobber it. The probe has its + * own PTE-planted page (VA_GR_PROBE, small-PT slot 0x10a); when that + * plant is unavailable the probe falls back to 0x800 as before. + */ + ret = gsp_phys_rmcall_ce_submit_off_locked(bar0, + GSP_PHYS_RMCALL_VA_SRC, GSP_PHYS_RMCALL_VA_DST, GSP_PHYS_RMCALL_VA_SEM, GSP_PHYS_RMCALL_COPY_SIZE, GSP_PHYS_RMCALL_SEM_PAYLOAD, true, + boot_stage.rmcall_gr_probe.pages ? + 0u : 0x800u, + boot_stage.rmcall_gr_probe.pages ? + &boot_stage.rmcall_gr_probe : + NULL, + boot_stage.rmcall_gr_probe.pages ? + GSP_PHYS_RMCALL_VA_GR_PROBE : 0, &sub); if (ret) { pr_err("vgsp-boot: rm-call: GR submission refused before the doorbell (%d)\n", @@ -15242,7 +16683,7 @@ static int gsp_phys_rmcall_gr_sem_push_locked(void __iomem *bar0) gsp_phys_status_drain(NULL); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } gpget = get_unaligned_le32((u8 *)userd + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPGET_OFF); @@ -15342,7 +16783,7 @@ static void gsp_phys_rmcall_ctx_image_probe(void __iomem *bar0) offs[k], poll); break; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } if (poll != GSP_PHYS_RMCALL_SEM_PAYLOAD) break; @@ -15443,7 +16884,7 @@ static int gsp_phys_rmcall_ce_copy_va(void __iomem *bar0, u64 src_va, *sem_out = poll; return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } } @@ -15626,7 +17067,7 @@ int gsp_phys_backend_ce_copy_submit(u64 src_va, u64 dst_va, u64 sem_va, gsp_phys_work_timed_out); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } /* @@ -16145,7 +17586,7 @@ static int gsp_phys_guest_ce_copy_common(u64 src_hpa, u32 src_off, u64 dst_hpa, ret = -ETIMEDOUT; goto out; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } /* @@ -16381,15 +17822,33 @@ EXPORT_SYMBOL_GPL(gsp_phys_backend_guest_ce_copy); #define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1 0x00407040u #define GSP_PHYS_RMCALL_PRI_SKED_DEBUG_1_EN_CWD_SLOT_COUNT 0x800u /* Method block bound: init(152) + inline run(67) + PCAS(4) + REPORT_SEM(8) + - * the optional QMD-virt pair(2). */ -#define GSP_PHYS_RMCALL_GR_METHOD_WORDS 512u + * the optional QMD-virt pair(2). Raised 512 -> 1024 (SESSION 153- + * CONTINUED-53) so a 3D stream replay can carry the captured MME setup + * prelude (<=512 words) ahead of the largest chunked window (<=448) in one + * block. SESSION 153-CONTINUED-66: 1024 -> 2048 because the matured prelude + * reached 901 words and the deferral test (window=390 + prelude=901 + sa=47 + * vs the bound) dropped it on every launch; the block now spans the module + * pushbuffer's TWO 4 KB pages (VA_PB..VA_PB2) and is copied page-wise by + * gsp_phys_gr_pb_copy(). */ +#define GSP_PHYS_RMCALL_GR_METHOD_WORDS 4096u static u64 gsp_phys_gr_submitted; static u64 gsp_phys_gr_ok; static u64 gsp_phys_gr_refused; static u64 gsp_phys_gr_timed_out; +/* + * SESSION 153-CONTINUED-60: 3D-stream launches whose window was all-zero + * (the client's push page caught mid-rewrite). They are never handed to the + * engine: the zero method stream trips the PBDMA (trap 0x1168 -> priv-error + * poison, see the ZEROWIN-SKIP site). Counted so a boot's zero-window rate + * is visible without dmesg archaeology. + */ +static u64 gsp_phys_gr_zerowin_skips; static u64 gsp_phys_gr_retries; /* fresh-entry launch retries published */ static u64 gsp_phys_gr_pool_exhausted; +/* r1497: launch fetch pages mapped through a real-FB VID leaf (empty shadow). */ +static u64 gsp_phys_gr_vid_map_pages; +static u32 gsp_phys_gr_vid_log; /* m156 (E5): zero-write retries where the warm sem push proved the engine * wedged (the persistent-lottery class); further retries are skipped. */ static u64 gsp_phys_gr_sem_wedged; @@ -16472,7 +17931,7 @@ static bool gsp_phys_rmcall_gr_init_emitted; * stale-QMD execution) landed after its launch was shipped. */ static u32 gsp_phys_gr_prev_out_bi; -static u32 gsp_phys_gr_prev_out_csum; +static u64 gsp_phys_gr_prev_out_csum; static bool gsp_phys_gr_prev_out_valid; /* r425: SKED/FECS/CWD register witness (one line; the healthy-vs-wedged @@ -16627,16 +18086,2799 @@ static u32 *gsp_phys_rmcall_build_gr_init(u32 *p) } /* + * r597: extract the SEM_EXECUTE ACQUIRE groups from the captured guest + * prelude and copy them, verbatim, ahead of the rebuilt inline-QMD window. + * Two encodings are recognised: + * - C46F host-semaphore group: INC_METHOD addr 0x5c count>=5, words + * {LO, HI, PAYLOAD_LO, PAYLOAD_HI, EXECUTE}; EXECUTE & 7 != 1 is an + * acquire (the pump's own op decode). + * - C5C0 SET_REPORT_SEMAPHORE A..D: addr 0x1b00 count>=4, word D's low + * two bits select release (0) / acquire (non-zero). + * Only the wait methods are taken, so the proven window (init + inline QMD + * + PCAS + release) is untouched. Returns the number of words emitted. + */ +static u32 gsp_phys_rmcall_gr_emit_acquires(u32 *p, + const struct gsp_phys_gr_submit *req, + u32 budget) +{ + u32 off = 0, words = 0; + + if (!gsp_phys_rmcall_gr_acq_value() || !req || !req->raw) + return 0; + while (off < req->raw_qmd_hdr && words + 8 < budget) { + u32 hdr = req->raw[off]; + u32 op = (hdr >> 29) & 0x7; + u32 count = (hdr >> 16) & 0xfff; + u32 addr = (hdr & 0x1fff) << 2; + u32 n = 1 + count; + + if ((op != 1 && op != 3) || !count || + off + n > req->raw_qmd_hdr) { + off++; + continue; + } + if (addr == 0x5c && count >= 5) { + u32 v4 = req->raw[off + 5]; + u64 va = ((u64)(req->raw[off + 2] & 0xff) << 32) | + (u64)(req->raw[off + 1] & 0xfffffffc); + + if ((v4 & 0x7) != 1 && va) { + memcpy(p + words, &req->raw[off], n * 4); + words += n; + } + } else if (addr >= 0x1b00 && addr <= 0x1b0c && + count >= 4) { + u32 d = req->raw[off + 4] & 0x3; + + if (d != 0) { + memcpy(p + words, &req->raw[off], n * 4); + words += n; + } + } + off += n; + } + return words; +} + +/* + * r929 (GL 3D execution, Path A): replay a captured raw 3D method window. + * The sema service marks stream captures (the X server's C597 sections that + * carried their own operands) with a raw window (req->stream); when this is + * on, gsp_phys_rmcall_build_gr_launch emits SET_OBJECT TURING_A + the window + * verbatim + the channel release tail instead of the compute prelude/QMD, so + * the engine executes the REAL 3D methods (state binds, LINE_LENGTH_IN + + * LAUNCH_DMA uploads, program/tex binds) and, once draws are in the window, + * rasterizes. No effect on any submit without a stream window (all CUDA + * launches). Default on; A/B with gsp_phys_rmcall_gr_3d=0. + */ +static bool gsp_phys_rmcall_gr_3d = true; +module_param_named(gsp_phys_rmcall_gr_3d, gsp_phys_rmcall_gr_3d, bool, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_3d, + "Replay a captured raw C597 method window verbatim instead of the compute prelude (default 1)"); + +/* + * SESSION 153-CONTINUED-59: drop single-word SDM-type entries (SEC_OP 0 + * SSDM / 2 STORE_SDM / 6 USE_SDM, tu102 dev_esched_pbdma.h type map) from + * every replayed stream: the 3D window patch, the MME prelude and the attr + * prelude emission. The r1102 PBENTRY first-sighting forensics localized + * the fault decode point to the MME prelude head (WINPROBE/PB-FAULT-DUMP + * P0 HDR_SHADOW=04666d14 = prelude[2], PB_DATA0=04646d00 = prelude[3], + * pbfetch=0x101ebc = block end after the whole 943-word block was + * prefetched, intr0 bit 18 in the guest relay): the client's setup push + * carries a 21-word run of `04666d00 04666d10 04666d14 04646d00 ...` + * (SDM-family single-word encodings, method fields 0x3400/0x3440/0x3450) + * that svc_mme_prelude_capture stored verbatim; the physical PBDMA refuses + * them ("invalid method encoding used in method", NV_PPBDMA_INTR_0 + * PBENTRY), which also explains why the whole block was fetched but the + * REPORT_SEMAPHORE never retired. The guest-vGPU parts (subdevice masks) + * are meaningless on the single-subdevice ladder channel - the same call + * the sub!=0 multi-word drop makes. Zero words are kept (benign padding). + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sdm_drop = true; +module_param_named(sema_gl_sdm_drop, gsp_phys_rmcall_gr_sdm_drop, bool, + 0644); +MODULE_PARM_DESC(sema_gl_sdm_drop, + "Drop single-word SDM-type entries (SEC_OP 0/2/6) from 3D window replays, the MME prelude and the attr prelude (default 1)"); + +/* + * Whole-vidmem page-hash bracket for the forwarded GR launch path. When set, + * hash every 4 KB page of the first open device's BAR1 shadow immediately + * before the publish, then re-hash once the output retirement wait returns + * and log the pages whose content changed. This proves whether the engine + * wrote any vidmem page, not just the output pages the copy-out samples. + * Default 0: no allocation, no hashing and no change to the submit path. + */ +static bool gsp_phys_gr_vidmem_bracket; +module_param_named(sema_gl_vidmem_bracket, gsp_phys_gr_vidmem_bracket, bool, + 0644); +MODULE_PARM_DESC(sema_gl_vidmem_bracket, + "Hash every vidmem page before and after a forwarded GR launch to report writes outside the sampled output pages (default 0)"); + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude. The r1073-r1077 + * in-flight capture proved the draw-window replay raises a real MME + * ILLEGAL_OPCODE on CALL_MME_MACRO(2)=0x3810 (mme_pc=0, mme_ir=1, mme_mac + * =2) because the ladder context never received the client's macro + * program. The client does upload it (0x11c/0x114 pointer pairs, 0x118 + * code, 0x120 entry addresses, 0x124 shadow control; see + * svc_mme_prelude_capture); with this knob the launch builder preplays + * those captured runs verbatim - once per capture generation - right + * after the SET_OBJECT C597 bind and ahead of the window, so the macros + * the window calls are resident. Declared before build_gr_launch (its + * reader); the matching capture-side knob is sema_mme_prelude_cap. + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_mme_prelude = true; +module_param_named(sema_gl_mme_prelude, gsp_phys_rmcall_gr_mme_prelude, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_prelude, + "Preplay the captured client MME context-setup runs once per capture generation ahead of the first 3D window replay (default 1)"); + +/* + * SESSION 153-CONTINUED-63: MME START-ADDRESS RAM completion + stub knob. + * The captured SA table (svc_mme_sa_*) may be incomplete (the prelude + * buffer is capped at 512 words); every CALLED slot the table does not + * cover is loaded with a harmless end-noop stub at a fixed IR address so + * the call becomes a no-op instead of a jump to IR 0 with garbage state + * (the r1116 0x238c poison). Default 1; A/B with 0. + */ +#define GSP_PHYS_RMCALL_GR_MME_STUB_PC 0xC0u +/* + * SESSION 153-CONTINUED-69: the stub body, take 3. The CONT-68 body + * (20208c00 38c40046 038c4301) was copied from macro(0) instruction 3 - + * but that is a MID-MACRO instruction: the offline MME64 simulator shows + * it carries two output writes with mthd=NONE, so in isolation (method + * register still 0) it EMITS SET_OBJECT <- 0 twice - exactly the relayed + * trap "ChID 0001, Class 0000c597, Offset 00000000, Data 00000000" - and + * its end_next=1 requires the following instruction, so the macro also + * runs past the stub into whatever the IR holds there. + * + * The real minimal terminator is what Mesa's mme_tu104_builder_finish() + * emits: two default NOPs with end_next set on the FIRST (end_next - + * "end after the next instruction" - so the second NOP executes and then + * the macro returns). Both NOPs encode as w2=0x318c0300 (pred=ZERO, + * ALU0/1 dst=ZERO src=ZERO,ZERO, outs NONE) and emit NOTHING. The + * end_next bit is w2 bit 0. + */ +static const u32 gsp_phys_gr_mme_stub_code[6] = { + 0x00000000u, 0x318c0000u, 0x318c0301u, /* NOP, end_next=1 */ + 0x00000000u, 0x318c0000u, 0x318c0300u, /* NOP tail (executed) */ +}; +static bool gsp_phys_rmcall_gr_sa_stub = true; +module_param_named(sema_gl_mme_sa_stub, gsp_phys_rmcall_gr_sa_stub, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_stub, + "Load a harmless end-noop stub into every MME macro slot the window calls but the captured START-ADDRESS RAM table does not cover (default 1; the r1116 0x238c lane poison class)"); + +/* + * SESSION 153-CONTINUED-68: stub EVERY called slot, covered or not + * (attribution: with all calls no-op'd the window either still faults + * (the fault is in non-call content) or faults differently (the fault is + * macro execution). Default 0; A/B knob. + */ +static bool gsp_phys_rmcall_gr_sa_stub_all; +module_param_named(sema_gl_mme_sa_stub_all, gsp_phys_rmcall_gr_sa_stub_all, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_stub_all, + "Stub every macro slot the window calls, including slots the captured SA table covers (attribution knob; default 0)"); + +static inline bool gsp_sema_gr_sa_stub_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sa_stub); +} + +/* + * SESSION 153-CONTINUED-69: optional explicit stub IR address override. + * 0 = derive from the capture (ir_end, floor 0xC0). Nonzero pins the stub + * body/call target to this instruction address - the stub-landing A/B for + * the IR-boundary/ONEINC pointer-unit question. + */ +static u32 gsp_phys_rmcall_gr_mme_stub_pc; +module_param_named(sema_gl_mme_stub_pc, gsp_phys_rmcall_gr_mme_stub_pc, uint, + 0644); +MODULE_PARM_DESC(sema_gl_mme_stub_pc, + "Explicit MME stub IR instruction address (0 = derive from capture ir_end; try 0xf0/0x100 for the IR-boundary A/B)"); + +/* + * SESSION 153-CONTINUED-69: no-stall CALL drop. With the stub terminating + * macros as pure NOPs the macro consumes none of the CALL_MME_DATA + * parameter words the window pushes; the MME parameter port then + * backpressures on the window's larger calls - the r1142 stall parked the + * PBDMA mid-block with HDR_SHADOW = the 16-arg CALL_MME_MACRO(57) + * ONEINC run and the block never completed (while every <=4-arg window + * retired in ~100 us). Drop every CALL_MME_MACRO/CALL_MME_DATA run of a + * STUBBED slot from the emitted window instead: no call, no parameter + * words, no backpressure. Covered slots keep their calls (their real code + * consumes the parameters). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_call_drop = true; +module_param_named(sema_gl_mme_call_drop, gsp_phys_rmcall_gr_call_drop, bool, + 0644); +MODULE_PARM_DESC(sema_gl_mme_call_drop, + "Drop CALL_MME_MACRO/DATA runs of stubbed slots from emitted 3D windows (prevents the noop-stub parameter-port stall; default 1)"); + +/* + * SESSION 153-CONTINUED-70: per-slot macro re-enable mask, one slot at a + * time. The r1142-r1144 state drops EVERY stubbed-slot CALL run: the window + * retires on real HW (~94 us) but the macros that expand the driver's + * per-draw state (vertex stream binds, constant binds, ...) never execute, + * so no state the draw needs exists and every launch still stores 0 pages. + * A slot named in this mask is never treated as stubbed: its + * CALL_MME_MACRO/DATA words stay in the emitted window and the engine + * executes the macro against the replayed context (SA RAM + IR program as + * the client's own uploads left them; a slot the client never uploaded + * reads SA=0 exactly like silicon and lands in macro 0). A/B one bit per + * cycle; keep the proven-good set empty until the offline MME64 sim has + * shown the slot's macro emits sane methods for its caller args. + */ +static u64 gsp_phys_rmcall_gr_keep_mask; +module_param_named(sema_gl_mme_keep_mask, gsp_phys_rmcall_gr_keep_mask, + ullong, 0644); +MODULE_PARM_DESC(sema_gl_mme_keep_mask, + "Slots whose CALL_MME_MACRO/DATA runs are never dropped/stubbed (u64 bitmask; one slot at a time for macro re-enable)"); + +/* + * SESSION 153-CONTINUED-113: clamp the keep mask to capture-covered slots. + * The r1405 poison anatomy (traced live): the draw window calls + * CALL_MME_MACRO(34) and CALL_MME_MACRO(57); the capture holds 13 SA slots + * (0..12, IR <= 0xe9) while keep_mask=~0 exempted every slot from stubbing, + * so the calls ran, the MME fetched a byte the packed program does not hold + * (`info=badf1201 pc_valid=1 pc=513 mac=0x1f1`, `miss=1`, ir=badf1201) and + * the priv-error wedge wiped the context; the second recovery's hard re-arm + * left the engine unresponsive (probe rc=-110) and GP_GET frozen for the + * boot. A kept slot the capture cannot serve can only ever fetch + * non-resident code: the CONT-80 r1186 note already pairs "drop the runs + + * stub SA[slot]" as the safe combination for a nop_drop slot - apply the + * same combination to any kept slot outside the capture's coverage. The + * decision reads the live SA table per window, so a later capture + * generation that covers the slot releases it automatically. Default 1 + * (keep_clamp); 0 = the r1405 all-real behavior (A/B). + */ +static bool gsp_phys_rmcall_gr_keep_clamp = true; +module_param_named(sema_gl_mme_keep_clamp, gsp_phys_rmcall_gr_keep_clamp, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_keep_clamp, + "Clamp kept slots to capture-covered slots: a keep_mask slot with no captured SA entry (or uncovered IR) is stubbed + its CALL runs dropped instead of fetching non-resident code (default 1)"); + +static u32 gsp_phys_gr_keep_clamp_log; + +/* + * SESSION 153-CONTINUED-114: golden MME slot overlay (see the include + * comment). The capture is a per-boot lottery (r1407/r1385 got the full + * 55-slot program; r1405/6/10/11/12 only 13 slots), and a called slot + * outside the capture is stubbed with its CALL runs dropped - which is + * why the draw never executes on partial boots. The golden library is + * the same driver-emitted microcode word-for-word (validated against a + * live full capture), so a golden-filled slot is byte-shape-identical + * to a captured one downstream (pack image, 0x11c pairs, IRAM gate). + * + * gsp_phys_gr_golden_slot: returns 1 and the slot's instruction entry + + * word pointer when the library can serve slot j (and the knob is on). + * gsp_phys_gr_golden_active: the knob as a bool, for the cheap sites. + */ +static unsigned int gsp_phys_rmcall_gr_mme_golden; +module_param_named(sema_gl_mme_golden, gsp_phys_rmcall_gr_mme_golden, + uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_golden, + "Golden MME slot overlay: serve called-but-uncaptured SA slots from the driver's own generated macro code (untracked kernel/golden-mme-tu104.h) instead of stubbing them, so CALL_MME_MACRO(DRAW_IAB etc.) runs deterministically (default 0; 1 = fill uncovered slots; 2 = force/overwrite all)"); +static unsigned int gsp_phys_rmcall_gr_mme_golden_force; +module_param_named(sema_gl_mme_golden_force, gsp_phys_rmcall_gr_mme_golden_force, + uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_golden_force, + "Force the golden overlay over captured slots too (A/B for a stale capture; default 0 = fill uncovered only)"); + +/* + * r1448 STRATEGY 3.3 (golden-completion): with golden_force on, a called + * slot the golden library cannot serve is stubbed (SA pair -> harmless + * end-noop) and its CALL runs are dropped, and its captured body is + * zeroed in the packed IR - only golden bodies can execute. This closes + * the r1447 residual (slot 0, entry 0, was the one executed non-golden + * body). 0 = pre-r1448 behavior (capture bodies of non-golden slots may + * execute). + */ +static bool gsp_phys_rmcall_gr_mme_golden_only = true; +module_param_named(sema_gl_mme_golden_only, gsp_phys_rmcall_gr_mme_golden_only, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_golden_only, + "Golden-only completion: with golden_force on, blank the packed body and stub/drop CALL runs of every slot the golden library cannot serve (default 1; 0 = current behavior)"); + +static u32 gsp_phys_gr_golden_log; +static u32 gsp_phys_gr_golden0_log; + +static int gsp_phys_gr_golden_slot(u32 j, u32 *entry, const u32 **words, + u32 *nwords) +{ + const struct golden_mme_tu104_slot *gs; + + if (!READ_ONCE(gsp_phys_rmcall_gr_mme_golden)) + return 0; + if (j >= GOLDEN_MME_TU104_SLOTS || j >= 64u) + return 0; + gs = &golden_mme_tu104_sa[j]; + if (!(gs->flags & GMME_F_PRESENT) || !gs->words) + return 0; + if ((u64)gs->off + gs->words > GOLDEN_MME_TU104_WORDS) + return 0; + /* + * r1448 STRATEGY 3.3: entry 0 is a valid IR address - slot 0's + * SETUP_FIXED_CONST body begins at IR word 0, and the window calls + * slot 0 (CALL at win-off 56). The pre-r1448 `!gs->entry` reject + * left slot 0 as the only executed non-golden body (r1447 54/55 + * overlay). Accept it and log the acceptance once so the coverage + * change is visible in dmesg. + */ + if (!gs->entry && !READ_ONCE(gsp_phys_gr_golden0_log)) { + WRITE_ONCE(gsp_phys_gr_golden0_log, 1u); + pr_info("vgsp-gr: MME-GOLDEN slot 0 accepted (entry=0 valid; golden body %u words)\n", + gs->words); + } + *entry = gs->entry; + *words = &golden_mme_tu104_words[gs->off]; + *nwords = gs->words; + return 1; +} + +/* + * r1448 STRATEGY 3.3 golden-only completion: with golden_force on and + * the golden_only knob on, a slot the golden library cannot serve must + * never execute. Returns true when the slot is outside the golden corpus + * and has to be blanked/stubbed (fail-closed). + */ +static bool gsp_phys_gr_slot_golden_blank(u32 j) +{ + u32 gent, gn; + const u32 *gw; + + if (!READ_ONCE(gsp_phys_rmcall_gr_mme_golden_only) || + !READ_ONCE(gsp_phys_rmcall_gr_mme_golden_force)) + return false; + return !gsp_phys_gr_golden_slot(j, &gent, &gw, &gn); +} + +/* + * SESSION 153-CONTINUED-79: per-launch method-block word cap + prelude drain. + * The r1172-r1175 A/B: every beef launch whose emitted block exceeded ~2000 + * words (3083-word full-SA prelude + window, 3315-3484 words) NEVER STARTED + * on the physical engine (MME-FAST events=0, grfifo frozen, ch_rel never + * written) while every block <= ~1300 words (1002-word partial capture + + * window) executed (retires in 90-170 us when the executed macro stayed + * inside the loaded IR). The failure class is fetch-side (a stuck GPFIFO + * entry), not content: once parked, every later entry also failed until a + * poison-latch TSG re-arm reset the PBDMA. The full-SA capture (55 slots, + * 3083 words, macros that chain past the first 15 slots e.g. macro 0 -> + * IR 0x201 = slot-25 entry) therefore cannot be replayed in ONE block. + * Emit at most gsp_phys_rmcall_gr_block_cap words per launch and drain the + * (SDM-sanitized, run-aligned) prelude across consecutive launches with a + * generation-keyed cursor; the emission latch is set only when the final + * part has been emitted, so a completed program is re-requested after the + * WEDGE-RECOVER latch clear. Default 2048; 0 = legacy 4096 module buffer. + */ +static u32 gsp_phys_rmcall_gr_block_cap = 2048u; +module_param_named(sema_gl_mme_block_cap, gsp_phys_rmcall_gr_block_cap, uint, + 0644); +MODULE_PARM_DESC(sema_gl_mme_block_cap, + "Max words emitted per GR launch method block (0 = 4096 module buffer; the >2048-word blocks never start on the physical engine)"); + +/* Generation-keyed drain cursor for the split MME prelude emission. */ +static u32 gsp_phys_gr_mme_pre_emit_gen; +static u32 gsp_phys_gr_mme_pre_emit_h; +static u32 gsp_phys_gr_mme_pre_emit_off; +/* + * CONT-105: generation-keyed cursor for the SPLIT packed MME prelude + * (see the sema_gl_mme_pack_split comment). pack_off counts IR words + * (always a multiple of 3 = whole instructions); pack_sa records that the + * SA table has gone out for this pass. The pair is reset together with the + * drain cursor above whenever the emission restarts. + */ +static u32 gsp_phys_gr_mme_pack_off; +static bool gsp_phys_gr_mme_pack_sa; + +/* + * CONT-103: prelude residency witness + calls-kept block shrink. + * The CONT-79 split-emission re-carries the whole prelude prefix on EVERY + * launch while keep_mask is set (idempotent re-emission, CONT-72), so the + * first calls-kept launch is 1613 prelude words + the 410-word window = + * 2126-2142 words - over the ~2050-word fetch wall (CONT-79: oversized + * blocks never start, PBDMA parks). The deadlock: the calls can only run + * after the program is resident, and the launch that would run them is too + * big to start. The re-emission is only needed while the program is NOT + * known resident; the launch that carries the final part is the witness + * (its poll break with GP_GET advanced proves the engine consumed it). + * live_gen records the generation whose carrier executed; while it equals + * the capture generation the drain restart is skipped and the calls-kept + * launch ships window-only (~530 words, well under the wall). + * sema_gl_mme_prelude_live_skip=0 restores the unconditional re-emission + * for a one-flip A/B. + */ +static u32 gsp_phys_gr_mme_pending_gen; /* final part carried by the last + * launch; consumed at its poll */ +static u32 gsp_phys_gr_mme_live_gen; /* generation whose carrier executed */ +static bool gsp_phys_rmcall_gr_mme_live_skip = true; +module_param_named(sema_gl_mme_prelude_live_skip, + gsp_phys_rmcall_gr_mme_live_skip, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_prelude_live_skip, + "Skip the forced per-launch prelude re-emission once the generation's final part has been carried by a launch the engine executed (default 1; 0 = always re-emit, pre-CONT-103 behavior)"); + +/* + * CONT-103: elide the program-address pair runs (method space + * [0x2054, 0x2158]) from a calls-kept window. With the prelude-live skip the + * calls-kept launch starts, and the engine then traps 0x80002058 on the + * relocated (shadow-pool) pair operand and parks - the CONT-101 trap class. + * The binds persist on the channel from the earlier calls-held launches, so + * dropping the re-binds removes the trapping write; the calls and the MME + * program stay. Off by default: the A/B against the trapping state. + */ +static bool gsp_phys_rmcall_gr_mme_nop_pair; +module_param_named(sema_gl_mme_nop_pair, gsp_phys_rmcall_gr_mme_nop_pair, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_nop_pair, + "Elide program-address pair runs (0x2054/0x2094/0x2154 families) from a calls-kept 3D window (default 0; the engine traps 0x2058 on the relocated shadow operand)"); + +/* + * r1453 PAIR-CHAIN (sema_gl_pair_chain): the offline EC 0x0d decode + * (BEGIN_END_ACTIVE, nv50.c:173) and the r1452b window dumps show the + * guest's draws arriving as back-to-back normal/normal BEGIN (0x1618) + * ... END (0x1614) pairs with no WAIT_FOR_IDLE (0x110) / SEND_GO_IDLE + * (0x13c) drain between them, and the FE then refuses the closing END + * (trap 0x80001614, Data 0, EC 0x0d) - and every later state write + * (0x2058 PROGRAM_ADDRESS_B) while a pair stays active. + * + * The guest's BEGIN data word is carried VERBATIM end to end (the + * window is a raw-push slice: svc_gl_flush -> req->stream -> + * gsp_phys_rmcall_gr_3d_patch_window memcpy). Nothing in the kernel + * tree constructs or normalises a 0x586/0x585 word: the only 32-bit + * BEGIN data form is a count-carrying run at method 0x1618 whose first + * payload word is the full data (SPLIT_MODE is bits 30:29 of THAT + * word); the 13-bit IMMD header form (op 4, data 8/7) cannot express + * the split bits and is what the observed raw pairs carry. This pass + * therefore never rebuilds a run: it (a) receipts the full BEGIN data + * word it carries, (b) inserts ONE WAIT_FOR_IDLE<-0 (INC 0x110, 1 data + * word) ahead of a following normal-begin pair so each END closes a + * drained pair, and (c) leaves split-mode chains (BEGIN 1/2/3) alone. + * 0 = legacy bytes (pass not run; held-call elision unchanged). + */ +static bool gsp_phys_rmcall_gr_pair_chain = true; +module_param_named(sema_gl_pair_chain, gsp_phys_rmcall_gr_pair_chain, + bool, 0644); +MODULE_PARM_DESC(sema_gl_pair_chain, + "One logical BEGIN..END chain per launch: carry the full 32-bit BEGIN data (SPLIT_MODE 30:29), insert WAIT_FOR_IDLE(0x110) drains between back-to-back normal/normal pairs, and extend pair elision to held-call windows (default 1; 0 = legacy bytes)"); +/* + * r1465 PAIR-ORPHAN-DROP (sema_gl_pair_orphan_drop): the pair_open/split + * tracker is a projection of the FE's pair state, so it must only advance + * through windows the FE actually consumed. A dropped/held/LOAD-ONLY + * window is a pair-state no-op (restore the launch-entry snapshot), and an + * END observed while the model is closed is provably orphaned (its opener + * window was dropped) - the FE has no pair to close and faults EC 0x0d, so + * the END run is elided. Default 1; 0 = legacy bytes (leak/0x1614). + */ +static bool gsp_phys_rmcall_gr_pair_orphan_drop = true; +module_param_named(sema_gl_pair_orphan_drop, + gsp_phys_rmcall_gr_pair_orphan_drop, bool, 0644); +MODULE_PARM_DESC(sema_gl_pair_orphan_drop, + "Pair-state sync: roll pair_open/open_split back on dropped/held/LOAD-ONLY windows and elide orphan END runs when no pair is open (default 1; 0 = legacy bytes)"); +/* + * SESSION 155-2: r1526's first calls-kept draw window faulted with a GR + * class error at SET_FRONT_FACE (0x191c <- 0x901) EC 0x0d + * ILLEGAL_METHOD_IN_BEGIN_END: the window entered with a BEGIN..END pair + * still open from the previous emitted window, so the first client state + * method ran inside it. Close the carried pair with the driver's own END + * form (0x1614 + event word, event 1 for split chains 1/2) before the + * window, then let the normal walk rebase the model. Gated: default 0 + * (legacy bytes). + */ +static bool gsp_phys_rmcall_gr_pair_close_entry = true; /* PAIR-SAFE: default ON (CONT.399) */ +module_param_named(sema_gl_pair_close_entry, + gsp_phys_rmcall_gr_pair_close_entry, bool, 0644); +MODULE_PARM_DESC(sema_gl_pair_close_entry, + "Emit an END (0x1614) for a BEGIN..END pair carried open into the window before its first run (default 0; A/B)"); +/* + * SESSION 155 CONT 9: the synthesized SPH-BIND prelude (j0/j2/j4/j5) is + * PREPENDED ahead of the window, so when the previous window left a + * BEGIN..END pair carried open the prelude's method runs land inside it - + * r1558 (mask=0x35) took EC 0x0d ILLEGAL_METHOD_IN_BEGIN_END at 0x2018. + * sema_gl_pair_close_entry closes the pair but is applied to the WHOLE + * window (prepends included) at the end of the pass; this knob makes the + * prelude itself self-contained: when the tracker says a pair is open, the + * END (0x1614 + event word, event 1 for split chains 1/2) is emitted as the + * FIRST word of the prelude, and each synthesized stage's + * SET_PIPELINE_SHADER TYPE and REGISTER_COUNT are emitted BEFORE its SPH + * address pair (the client's own stage-1 order). Default 0 = legacy bytes. + */ +static bool gsp_phys_rmcall_gr_sph_bind_pair_safe = true; /* PAIR-SAFE: default ON (CONT.399) */ +module_param_named(sema_gl_sph_bind_pair_safe, + gsp_phys_rmcall_gr_sph_bind_pair_safe, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_bind_pair_safe, + "Make the synthesized SPH-BIND prelude BEGIN/END-safe: close a carried-open pair with END(0x1614) before the prelude and emit TYPE/REGISTER_COUNT before each stage address pair (default 0; legacy bytes)"); +/* + * SESSION 155 CONT 9 (TASK B): the window replayed for a non-tessellated, + * non-geometry draw carries only the VSB (j=1) record; the client's + * channel state leaves VSA/TCS/TES/GS explicitly DISABLED. The replay + * channel instead inherits the setup-push program records (TCS enabled, + * 0x2080 <- 0x20 is a disable the ATTR-PRELUDE drops), so the DS validates + * VSB in tessellation mode and GS against a stale program - the r1560 + * 0x900400a0 SPH5/7/18 bits. Emit the client's own disable records, each + * value quoted from the driver's unchanged-program path: + * 0x2000 <- 0x00 SET_PIPELINE_SHADER(0) ENABLE=FALSE|TYPE=VERTEX_CULL_BEFORE_FETCH + * 0x2080 <- 0x20 SET_PIPELINE_SHADER(2) ENABLE=FALSE|TYPE=TESSELLATION_INIT + * 0x20c0 <- 0x30 SET_PIPELINE_SHADER(3) ENABLE=FALSE|TYPE=TESSELLATION + * 0x2100 <- 0x40 SET_PIPELINE_SHADER(4) ENABLE=FALSE|TYPE=GEOMETRY + * and suppress the synthesized j0/j2/j4 binds (a disabled stage has no + * program to bind). Default 0 = legacy bytes. + * + * SESSION 155 CONT 10: the all-four bool became a per-stage mask, bit j = + * synth slot j (0=VSA, 1=VSB, 2=TCS, 3=TES, 4=GS, 5=PS). A set bit emits + * that stage's client disable record (for the stages whose record the + * driver's unchanged-program path writes: 0/2/3/4) and skips that slot's + * synthesized bind (j0/j2/j4/j5; j1 is the window's own VTG record). + */ +static u32 gsp_phys_rmcall_gr_sph_stage_disable; +module_param_named(sema_gl_sph_stage_disable, + gsp_phys_rmcall_gr_sph_stage_disable, uint, 0644); +MODULE_PARM_DESC(sema_gl_sph_stage_disable, + "Per-stage synth-bind disable mask (bit j = synth slot j: 0=VSA,1=VSB,2=TCS,3=TES,4=GS,5=PS): emit the client's SET_PIPELINE_SHADER disable record for that stage and skip its synthesized bind (default 0; legacy bytes)"); + +/* + * SESSION 155 CONT 11: the last DS latch bit in the r1562 replay is SPH18 + * (GS check2, attribute map). r1562 proved the DS validates the GS stage + * even when the driver's own disable record (0x2100 <- 0x40) leaves it + * unbound: SPH18 latched with stage4 START=0. The client has no geometry + * program, so no GS SPH exists anywhere in its push. This knob binds a + * synthesized header-only GEOMETRY SPH (128 B, pass-through maps) to + * stage 4 while keeping SET_PIPELINE_SHADER(4) ENABLE=0, so the DS has a + * valid operand at SPH_INFO_STAGE4_START but the engine never executes + * it. The image is authored by the sema service at family+0x780 (same 4K + * page as the bound VTG/TCS/PS images) and the address pair below is + * relocated by the same window pass. Default 0 = legacy bytes. + */ +static bool gsp_phys_rmcall_gr_sph_gs_img; +module_param_named(sema_gl_sph_gs_img, gsp_phys_rmcall_gr_sph_gs_img, + bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_gs_img, + "Bind a synthesized header-only GEOMETRY SPH to stage 4 with ENABLE=0 (family+0x780; satisfies the SPH18 GS attribute-map check; default 0)"); + +bool gsp_phys_rmcall_gr_sph_gs_img_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_gs_img); +} + +/* + * SESSION 155 CONT 12: the synthesized GS SPH's attribute-map variant. + * The r1564 run proved the DS fetches the image (stage4 START nonzero) + * yet SPH18 persists and the VTG-class check2 bits return, so the map + * CONTENT is the discriminator: + * 0 = pass-through (IMAP = previous OMAP = POSITION+COLOR; + * OMAP = PS IMAP superset = POSITION+COLOR) + * 1 = all IMAP/OMAP zero (the mesh-dummy contract) + * 2 = PS-exact (IMAP = POSITION.W; OMAP = POSITION.XYZW) + * 3 = IMAP POSITION+COLOR, OMAP POSITION.XYZW only + * A/B live: the sema service reads this per capture. + */ +static u32 gsp_phys_rmcall_gr_sph_gs_maps; +module_param_named(sema_gl_sph_gs_maps, gsp_phys_rmcall_gr_sph_gs_maps, + uint, 0644); +MODULE_PARM_DESC(sema_gl_sph_gs_maps, + "Synth GS SPH attribute-map variant: 0=pass-through(POS+COLOR both), 1=zero maps, 2=PS-exact(POS.W/POS.XYZW), 3=IMAP POS+COLOR / OMAP POS.XYZW (default 0)"); + +u32 gsp_phys_rmcall_gr_sph_gs_maps_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_gs_maps); +} + +/* + * SESSION 155 CONT 13: GS-mode context variant for the synthesized GS SPH. + * The mesh per-primitive dummy-GS contract (the only driver precedent for + * a fake GS SPH) requires ENABLE_FAST_G_S in the header and writes + * SET_GS_MODE(FAST_GS) + POST_VTG_SHADER_ATTRIBUTE_SKIP_MASK. Our replay + * channel never receives those (the client has no GS program). Bit j: + * 1 = set FASTGS in the SPH header and emit + * 0x0208 <- 1 (SET_GS_MODE FAST_GS) + * 0x1240..0x125c <- 0 (POST_VTG_SHADER_ATTRIBUTE_SKIP_MASK) + * 2 = enable the stage (final SET_PIPELINE_SHADER(4) = 0x41) + * Default 0 = the CONT 11 disabled/no-fastgs form. + */ +static u32 gsp_phys_rmcall_gr_sph_gs_mode; +module_param_named(sema_gl_sph_gs_mode, gsp_phys_rmcall_gr_sph_gs_mode, + uint, 0644); +MODULE_PARM_DESC(sema_gl_sph_gs_mode, + "Synth GS SPH mode bits: 1=FASTGS header + SET_GS_MODE(FAST_GS)+skip-mask, 2=enable stage (0x41); 0=disabled/no-fastgs (default)"); + +u32 gsp_phys_rmcall_gr_sph_gs_mode_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_gs_mode); +} + +/* + * SESSION 155 CONT 14: raster-enable prelude. The client's per-frame 3D + * window carries no SET_CT_WRITE / SET_DEPTH_TEST / SET_DEPTH_WRITE / + * SET_STENCIL_TEST / SET_SCISSOR_ENABLE - those are channel state on the + * client's own channel - so the replay channel keeps the hardware + * defaults. If CT_WRITE defaults to 0 every rasterized fragment is + * discarded and the surface stays black even though the draw executes. + * Emit the safe render-enable state ahead of each replayed 3D window: + * SET_CT_WRITE(0..7) 0x1a00+i*4 <- 0x1111 (R|G|B|A write enable) + * SET_DEPTH_TEST 0x12cc <- 0 + * SET_DEPTH_WRITE 0x12e8 <- 0 + * SET_STENCIL_TEST 0x1380 <- 0 + * SET_SCISSOR_ENABLE(0) 0x0e00 <- 0 + * Default 0 = legacy bytes (A/B). + */ +static bool gsp_phys_rmcall_gr_raster_pre; +module_param_named(sema_gl_raster_prelude, gsp_phys_rmcall_gr_raster_pre, + bool, 0644); +MODULE_PARM_DESC(sema_gl_raster_prelude, + "Emit CT_WRITE(0..7)=0x1111 + depth/stencil/scissor disable ahead of replayed 3D windows (default 0; A/B)"); + +bool gsp_phys_rmcall_gr_raster_pre_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_raster_pre); +} + +/* + * SESSION 155 CONT 18: follow a migrated screen page instead of skipping + * its write-back. The guest's video-memory manager migrates the X screen + * BO (the WGUARD-SKIP "used != now" case); the drawn content then sits in + * the bounce copy of the old page and is currently discarded. When this + * knob is on and the stale VA belongs to the captured screen surface, the + * page is written to the VA's CURRENT mapping (receipt + * WGUARD-STALE-WRITE) so the guest sees the frame. Off = the historical + * safe skip. + */ +static bool gsp_phys_gr_stale_write; +static u32 gsp_phys_gr_stale_log; /* CONT 18 */ +module_param_named(sema_gl_stale_write, gsp_phys_gr_stale_write, bool, + 0644); +MODULE_PARM_DESC(sema_gl_stale_write, + "Write a copy-out page to the screen VA's current mapping when the write guard reports a migration (default 0; safe skip)"); + +bool gsp_phys_rmcall_gr_stale_write_value(void) +{ + return READ_ONCE(gsp_phys_gr_stale_write); +} + +/* + * SESSION 155 CONT 22: extra raster/store-disable A/B. The client's window + * carries none of the blend/alpha/cull/point/ZCULL channel state; if the + * replay's defaults kill fragments (blend discard, cull, alpha, ZCULL), no + * color reaches the ROP. Emit the neutral disabling form ahead of the + * window (constant runs): + * 0x12e4 SET_BLEND_STATE_PER_TARGET <- 0 + * 0x1e00 SET_BLEND_PER_TARGET_SEPARATE<- 0 + * 0x12ec SET_ALPHA_TEST <- 0 + * 0x1308 D3D_SET_CULL_MODE <- 0 (no cull) + * 0x1520 SET_POINT_SPRITE <- 0 + * 0x02e4 SET_ZCULL_ROP_BYPASS <- 0 + * 0x02e8 SET_ZCULL_SUBREGION <- 0 + * 0x02fc ASSIGN_ZCULL_SUBREGIONS <- 0 + * Default 0 = legacy bytes. + */ +static bool gsp_phys_rmcall_gr_raster_extra; +module_param_named(sema_gl_raster_extra, gsp_phys_rmcall_gr_raster_extra, + bool, 0644); +MODULE_PARM_DESC(sema_gl_raster_extra, + "Emit blend/alpha/cull/point/ZCULL neutral-disabled state ahead of replayed 3D windows (default 0; A/B)"); + +bool gsp_phys_rmcall_gr_raster_extra_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_raster_extra); +} + +/* + * SESSION 155 CONT 15: where the synthesized GS SPH lives inside the + * program-family page. 0 = family+0x780 (below the window's own VTG bind + * at family+0x900), 1 = family+0xe80 (above it; the DS's PROGRAM_REGION + * snapshot reads the bind VA, so a region starting there does not cover + * 0x780). Default 0 = legacy. + */ +static u32 gsp_phys_rmcall_gr_sph_gs_addr; +module_param_named(sema_gl_sph_gs_addr, gsp_phys_rmcall_gr_sph_gs_addr, + uint, 0644); +MODULE_PARM_DESC(sema_gl_sph_gs_addr, + "Synth GS SPH offset in the family page: 0=+0x780 (legacy), 1=+0xe80 (above the VTG bind)"); + +u32 gsp_phys_rmcall_gr_sph_gs_addr_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_gs_addr); +} + +/* + * SESSION 155 CONT 15: GS context A/B words emitted with the synth GS bind. + * bit0: SET_MESH_CONTROL (0x114c) <- 0 + * bit1: SET_GS_MODE (0x0208) <- 0 (TYPE_ANY) + * The client's per-frame push carries neither; if the init push left them + * set, the GS macro/mesh path may treat the stage specially. Default 0. + */ +static u32 gsp_phys_rmcall_gr_sph_gs_ctx; +module_param_named(sema_gl_sph_gs_ctx, gsp_phys_rmcall_gr_sph_gs_ctx, + uint, 0644); +MODULE_PARM_DESC(sema_gl_sph_gs_ctx, + "Extra GS context with the synth bind: bit0=SET_MESH_CONTROL(0x114c)<-0, bit1=SET_GS_MODE(0x0208)<-0 (default 0)"); + +u32 gsp_phys_rmcall_gr_sph_gs_ctx_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_gs_ctx); +} +static bool gsp_phys_gr_mme_pairs_emitted; /* launch emitted a pair run */ +static u32 gsp_phys_gr_mme_pairs_gen; /* gen whose pairs retired OK */ +static u32 gsp_phys_gr_mme_launch_gen; /* last built launch's gen */ +static bool gsp_phys_gr_mme_pair_open; /* BEGIN..END open across windows */ +static u32 gsp_phys_gr_mme_pair_open_split; /* split mode of that BEGIN */ +static bool gsp_phys_gr_mme_pair_open_entry; /* open seen when THIS window was fixed */ +static u32 gsp_phys_gr_mme_pair_open_entry_split; /* split of the entry-open BEGIN */ +static u32 gsp_phys_gr_mme_pair_entry_elide_n; /* entry-open pair-run elisions */ + +/* + * True when a method run in [w, w+n) writes NVC597_BEGIN (0x1618) or + * NVC597_END (0x1614) - the per-launch proof the pair-elision extension + * gates on (pairs must have reached the FE once for the generation). + */ +static bool gsp_phys_gr_pair_present(const u32 *w, u32 n) +{ + u32 r = 0; + + while (r < n) { + u32 h = w[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u; + u32 m = (h & 0xfffu) << 2; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + if ((op == 1u || op == 3u || op == 4u || op == 5u) && + (m == 0x1618u || m == 0x1614u)) + return true; + r += span; + } + return false; +} + +/* + * SESSION 154-1: true when the method run in [w, w+n) already writes + * `method` (byte offset). Used to keep the channel-init re-emission from + * duplicating a write the captured window itself carries: the X channel's + * per-frame window contains the client's own SET_SHADER_EXCEPTIONS + * (0x1528) run (sema-pushraw `[79] off=0x5a4 op=4 addr=0x1528`), so the + * booter's extra INC-form write is the first method the FE sees and the + * one its trapped-method register reports (r1483 `trap=80001528`). + * + * The run grammar is the same as every window walk in this file: op 1/3/5 + * carry 1+count data words, op 4 is the client's one-word immd13 header + * (the 13-bit method field is in the header, no data word), everything + * else is a single header word. The low 13 bits are the method >> 2 for + * both the 0x2000 and 0x8000 header forms. + */ +static bool gsp_phys_gr_method_present(const u32 *w, u32 n, u32 method) +{ + u32 r = 0; + + while (r < n) { + u32 h = w[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 span = 1u; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + if (((h & 0x1fffu) << 2) == method) + return true; + r += span; + } + return false; +} + +/* + * SESSION 154-1: first SET_PIPELINE_PROGRAM_ADDRESS pair in an already + * relocated window. Returns the pair's engine-visible (shadow) VA. The + * DS validates each fetched SPH against the channel's PROGRAM REGION; + * the reference driver programs that region with the program's own GPU + * base (nv3dGetProgramGpuAddress -> NV9097_SET_PROGRAM_REGION_A/B), not + * with the whole shader heap. The replay's engine fetch goes through the + * relocated shadow VA, so the matching region base is the bound VA's 4K + * page (e.g. beef bind 0x19a0f00 -> region 0x19a0000), not the shadow + * pool's base (the r1483/r1484 forced 0x1500000, which does not contain + * the beef program's 0x19a0000 page and makes the DS reject the fetched + * SPH: r1484 `GR-STATE sph=0,019a0f00,019a0200,...,019a0000` together + * with `ds=86040646`). + * + * Only op 1 (INC) can carry the two-word A/B pair; the run grammar is the + * same as the window walkers. + */ +static bool gsp_phys_gr_first_prog_bind(const u32 *w, u32 n, u64 *va_out) +{ + u32 r = 0; + + while (r < n) { + u32 h = w[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 addr = (h & 0x1fffu) << 2; + u32 span = 1u; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + if (op == 1u && cnt >= 2u && addr >= 0x2014u && + addr < 0x4000u && (addr & 0x3fu) == 0x14u) { + u64 pva = ((u64)(w[r + 1u] & 0xffu) << 32) | + (u64)(w[r + 2u] & 0xfffffffcu); + + if (pva) { + *va_out = pva; + return true; + } + } + r += span; + } + return false; +} + +/* + * Apply the pair-chain rule to the final emitted window (prepends + * included). Returns the number of inserted drains; *np is updated. + * The scan is method-run aligned and insertion copies whole runs, so a + * run is never split and the BEGIN data word is never rewritten. + */ +static u32 gsp_phys_gr_pair_chain_fix(u32 *w, u32 *np, u32 cap) +{ + u32 n = *np, r = 0, begins = 0, ends = 0, drains = 0; + u32 chains = 0, orphans = 0, orphan_dropped = 0, nested = 0, badv = 0; + u32 entry_closes = 0; + bool open = READ_ONCE(gsp_phys_gr_mme_pair_open); + bool seen = false, begin_logged = false; + u32 open_split = READ_ONCE(gsp_phys_gr_mme_pair_open_split); + + WRITE_ONCE(gsp_phys_gr_mme_pair_open_entry, open); + WRITE_ONCE(gsp_phys_gr_mme_pair_open_entry_split, open_split); + + if (READ_ONCE(gsp_phys_rmcall_gr_pair_close_entry) && open) { + u32 expect = ((open_split == 1u || open_split == 2u) ? 1u : 0u); + + if (n + 2u <= cap) { + memmove(&w[2], &w[0], (size_t)n * 4u); + w[0] = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x1614u >> 2); + w[1] = expect; + n += 2u; + r = 2u; + open = false; + seen = true; + entry_closes++; + pr_info_ratelimited("vgsp-gr: PAIR-CLOSE-ENTRY split=%u ev=%u window=%u (carried pair closed before the window)\n", + open_split, expect, n); + } else { + pr_warn_ratelimited("vgsp-gr: PAIR-CLOSE-ENTRY skipped (window=%u cap=%u)\n", + n, cap); + } + } + + while (r < n) { + u32 h = w[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u; + u32 m = (h & 0xfffu) << 2; + bool is_begin = false, is_end = false; + u32 data = 0, split = 0, ev = 0; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + if (op == 4u) { + if (m == 0x1618u) { + is_begin = true; + data = (h >> 16) & 0x1fffu; + } else if (m == 0x1614u) { + is_end = true; + ev = (h >> 16) & 1u; + } + } else if ((op == 1u || op == 3u || op == 5u) && cnt && + m == 0x1618u) { + is_begin = true; + data = w[r + 1u]; + split = (data >> 29) & 3u; + } else if ((op == 1u || op == 3u || op == 5u) && cnt && + m == 0x1614u) { + is_end = true; + ev = w[r + 1u] & 1u; + } + if (is_begin) { + if (!begin_logged) { + begin_logged = true; + pr_info_ratelimited("vgsp-gr: PAIR-BEGIN-DATA off=%u data=%08x split=%u form=%s (full 32-bit data carried verbatim)\n", + r, data, split, + (op == 4u) ? "immd13(no-split)" : + "run32"); + } + if (!open && seen && split == 0u) { + /* + * EC 0x0d: a normal/normal pair begun while + * the previous pair is not drained is refused + * at its END. Insert the class's own stall + * (WAIT_FOR_IDLE <- 0) ahead of it. + */ + if (n + 2u <= cap) { + memmove(&w[r + 2u], &w[r], + (size_t)(n - r) * 4u); + w[r] = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | (1u << 16) | + (0x110u >> 2); + w[r + 1u] = 0u; + n += 2u; + r += 2u; + drains++; + pr_info_ratelimited("vgsp-gr: PAIR-DRAIN inserted WAIT_FOR_IDLE(0x110)<-0 before pair #%u at off=%u window=%u\n", + begins + 1u, r, n); + } else { + pr_warn_ratelimited("vgsp-gr: PAIR-DRAIN skipped (window=%u cap=%u): back-to-back pairs can still raise EC0x0d\n", + n, cap); + } + } + if (open) { + /* + * Legal only as a split chain (the previous + * BEGIN left its end open). + */ + if (open_split == 1u || open_split == 2u) + chains++; + else + nested++; + } else { + open = true; + seen = true; + } + open_split = split; + begins++; + } else if (is_end) { + if (!open) { + /* + * r1465 PAIR-ORPHAN-DROP: the opener's + * window was dropped, so the FE has no + * pair to close and would raise EC 0x0d + * on this END. Remove the whole END run + * (header 0x80000585/0x80000584 immd13 + * form or a 1+cnt run32) in place; never + * rewrite it. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_pair_orphan_drop) && + span <= n - r) { + memmove(&w[r], &w[r + span], + (size_t)(n - r - span) * 4u); + n -= span; + orphan_dropped++; + pr_info_ratelimited("vgsp-gr: PAIR-ORPHAN-DROP off=%u span=%u window=%u\n", + r, span, n); + continue; + } + orphans++; + } else { + u32 expect = ((open_split == 1u || + open_split == 2u) ? 1u : 0u); + + if (ev != expect) + badv++; + open = false; + ends++; + } + } + r += span; + } + if (begins > 1u || drains || chains || orphans || orphan_dropped || + nested || badv || open || entry_closes) + pr_info_ratelimited("vgsp-gr: PAIR-CHAIN begin=%u end=%u drains=%u chain=%u orphan_end=%u orphan_drop=%u nested=%u endv_mismatch=%u open_at_end=%u entry=%u close=%u window=%u\n", + begins, ends, drains, chains, orphans, orphan_dropped, + nested, badv, open ? 1u : 0u, + READ_ONCE(gsp_phys_gr_mme_pair_open_entry) ? 1u : 0u, + entry_closes, n); + *np = n; + WRITE_ONCE(gsp_phys_gr_mme_pair_open, open); + WRITE_ONCE(gsp_phys_gr_mme_pair_open_split, open_split); + return drains; +} + +/* + * SESSION 153-CONTINUED-79: parameter-conserving NOP for kept parking slots. + * The r1177 bisect: keep={0,2,57} (full 3089-word program resident, split + * emission) parks the PBDMA on the emitted block (MME-FAST events=0, grfifo + * frozen, warm sem WEDGED) while keep={0,2} retires every launch - slot 57's + * CALL_MME_MACRO(57) run is the r1142 park class. Rather than drop the run + * (which also removes its parameter words from the stream), rewrite the call + * run's header to target the CALL_MME_DATA port (method+4) so the parameter + * words are still written (conserved) but no macro executes: the trigger + * word becomes a plain port write and the PBDMA never opens a macro context + * whose parameter drain can back up. A slot named in this mask must also be + * in sema_gl_mme_keep_mask (kept = retained in the window). Default 0. + */ +static u64 gsp_phys_rmcall_gr_nop_param_mask; +module_param_named(sema_gl_mme_nop_param_mask, gsp_phys_rmcall_gr_nop_param_mask, + ullong, 0644); +MODULE_PARM_DESC(sema_gl_mme_nop_param_mask, + "Kept slots whose CALL_MME_MACRO trigger is rewritten to a plain CALL_MME_DATA port write (params conserved, macro not executed; u64 bitmask)"); + +/* + * SESSION 153-CONTINUED-80: the r1181 live A/B validated the retarget above + * and it PARKS the PBDMA exactly like the raw kept call - with + * keep=5|1<<57 + nop=1<<57 the first post-change launch froze + * (MME-FAST begin grfifo=0x00000001, end events=0, no poison, warm sem + * WEDGED) while keep=5 (slot 57 call+params elided) retired in 94-106 us + * on the same boot (r1176/r1181 baseline). The bare DATA-port run without an + * open macro context is not a safe landing for the trigger. With this knob + * set, a nop-mask slot joins the drop path instead (the r1144-r1177 + * proven-safe behavior: whole run elided, retires clean, params NOT + * conserved). Use for the all-real-macro A/B: sa_stub_all=0 leaves + * 34/39/40/48/51 real while this drops only 57. Default 0 preserves the + * retarget behavior so the r1179 transform evidence stays reproducible. + */ +static u64 gsp_phys_rmcall_gr_nop_drop; +module_param_named(sema_gl_mme_nop_drop, gsp_phys_rmcall_gr_nop_drop, ullong, + 0644); +MODULE_PARM_DESC(sema_gl_mme_nop_drop, + "Slots whose call runs are dropped entirely (proven-safe retire, SA[slot] stubbed) instead of retargeted to the CALL_MME_DATA port (u64 bitmask; default 0)"); + +/* + * SESSION 153-CONTINUED-82: hold real CALL runs until the prelude latch. + * The r1220 (all-real, nop_drop=1<<57, sa_stub_all=N) wedge: the window's + * real CALL runs executed in the SAME launch that emitted only the first + * PART of a fresh capture generation (`MME-PRELUDE part gen=3 off=1776 + * words=1776/3083 ... (continuation next launch)`), so the called macros + * fetched IR code that was not resident yet - `MME-FAST launch ... + * pc_valid=1 pc=513`, priv-error poison, `Shader Program Header 18 Error`, + * `Class Error ... Offset 0000238c, Data 00020020` (the r1116 lane-poison + * class) and the boot wedged. The prelude latch (gsp_phys_gr_mme_prelude_* + * gen/h) is set only when the FINAL part has gone out; with this knob the + * launch builder holds (elides, the proven-safe drop shape) every remaining + * CALL_MME_MACRO/DATA run of the emitted window while EITHER the latch for + * this generation+channel is not set after this launch, OR this launch + * uploaded generation code for the first time (latch invalid before the + * emission). The r1222 all-real boot proved the second clause necessary: + * the drain-completing launch (part-2 IR uploads + window real calls in the + * SAME block, latch set by its own emission) still hit ILLEGAL_OPCODE - + * real calls resume on the next launch, when only the idempotent keep-forced + * re-emission of the already-resident prefix is in flight. Default 1 (the + * fix); A/B with 0 to reproduce the r1209/r1220/r1222 wedge. + */ +static bool gsp_phys_rmcall_gr_mme_hold_calls = true; +module_param_named(sema_gl_mme_hold_calls, gsp_phys_rmcall_gr_mme_hold_calls, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_calls, + "Hold (elide) window CALL_MME_MACRO/DATA runs until the MME prelude latches for the current generation+channel (default 1; 0 = legacy, real calls run against a partial prelude)"); + +/* + * SESSION 153-CONTINUED-107: separately control the CONT-104 FE-idle clause + * of the hold (see the call site). With the IRAM gate proving residency the + * clause's only value is the r1359 race, and r1373 showed the FE wedged at + * 0x07071602 for minutes while every calls-kept window was held. Default 1 + * (pre-CONT-107 behavior); 0 releases the calls as soon as the prelude + * latch + IRAM gate allow. + */ +static bool gsp_phys_rmcall_gr_mme_hold_fe_idle = true; +module_param_named(sema_gl_mme_hold_fe_idle, + gsp_phys_rmcall_gr_mme_hold_fe_idle, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_fe_idle, + "Also hold the real CALL runs while the front-end is not settled (0x400504 != 1; default 1)"); + +/* + * SESSION 153-CONTINUED-84: emit NVC597_SET_MME_VERSION=1 (0x234) with the + * prelude step. The channel-init defaults the client pushes once per channel + * (Turing+) are never replayed, so the replayed context's MME stays in its + * reset/disabled state and every CALL_MME_MACRO run parks the front-end + * waiting for a macro that cannot start (r1245: call-bearing launches hang + * with all MME registers idle). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_mme_version = true; +module_param_named(sema_gl_mme_version, gsp_phys_rmcall_gr_mme_version, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_version, + "Emit SET_MME_VERSION=1 ahead of the replayed MME prelude (default 1; the Turing channel-init method the guest pushes once and the capture never carries)"); + +/* + * SESSION 153-CONTINUED-99: IR-only prelude drain. The call-bearing launch + * must carry the macros' SA pairs + IR words IN THE SAME BLOCK as the + * CALL_MME_MACRO runs: the split drain ships the full 3083-word capture + * across two launches (r1337 W1/W2), so the calls only ever ran on a launch + * whose block held part of the program - and every such launch faults MME + * ILLEGAL_OPCODE (relayed ESR 0x404490=0x80000004) the moment the first + * CALL executes (r1336/r1337/r1339/r1340, four boots; with {48,51,57} + * dropped in r1340 the remaining {0,2,34,39,40} calls fault identically). + * Shipping the FULL capture in one block does not work either: the 3441- + * and 3623-word blocks of r1341 never retired at all (MME-FAST begin + * grfifo=0d0d0001/00000001, no ch_rel, timeout ladder). The SA pairs + + * IR-load runs of the whole capture are only ~1277 of the 3083 words; the + * remainder is the setup family the macros do not fetch. With this knob the + * drain emits ONLY the runs targeting the MME SA/IR ports (0x114/0x118/ + * 0x11c/0x120), whole runs, so a single <=2048-word block can carry SA+IR + * (1277) + SA re-arm + window and the calls execute against the program + * the same launch uploaded. Default 0 (A/B; the r1337 drain behavior). + */ +static bool gsp_phys_rmcall_gr_mme_prelude_ir_only; +module_param_named(sema_gl_mme_prelude_ir_only, + gsp_phys_rmcall_gr_mme_prelude_ir_only, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_prelude_ir_only, + "Emit only the MME SA/IR port runs (0x114/0x118/0x11c/0x120) from the prelude drain so SA+IR fit one working-size block (default 0)"); + +/* + * SESSION 153-CONTINUED-99b: packed (partitioned) prelude. The captured + * prelude buffer is the sum of the client's overlapping MME upload runs: + * each run writes its macro's body at its SA entry, and the runs overlap + * (runs for later slots rewrite shared tail regions). Replaying them + * verbatim in capture order therefore leaves the IR in a state where a + * macro's own body tail has been overwritten with the NEXT macro's bytes - + * executing it fetches non-code and the MME raises ILLEGAL_OPCODE + * (relayed ESR 0x404490=0x80000004; r1336/r1337/r1339/r1340/r1342, five + * boots). Verified against the driver's own generated macro source: the + * slot-34 run equals mmeCodeOGL_C597_DRAW_IAB word-for-word (75/75) but + * the capture-order final image differs at 50 of them. + * + * This knob rebuilds the program the way the engine must see it: each IR + * address takes the value from the run of the slot whose SA region owns it + * (run j clipped at the next SA entry), then emits the whole IR as ONE + * contiguous 0x114 ONE_INC run plus the full SA table as [slot, entry] + * pairs - about 1270 words, so a single <=2048-word block carries SA+IR + + * SA re-arm + window and the window's CALL_MME_MACRO runs execute against + * the program the same launch uploaded. Default 0 (capture-order drain). + */ +static bool gsp_phys_rmcall_gr_mme_prelude_pack; +module_param_named(sema_gl_mme_prelude_pack, + gsp_phys_rmcall_gr_mme_prelude_pack, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_prelude_pack, + "Rebuild the MME IR as the partitioned program (each SA region from its own run) and emit it compactly with the SA table (default 0)"); + +/* Packed IR image; indexed by IR word address (<= 0x420 in captures). */ +static u32 gsp_phys_gr_mme_ir_pack[4096]; + +/* + * CONT-112: A/B IR-word override for the packed MME program. The offline + * replay of the captured SA stream (mmeref/, sim_run.py) shows slot 34 (the + * draw macro) emitting `0x1618<-0x61000008, 0xd74<-0, 0xd78<-count(0x3c), + * 0x1614<-0` from the CLIENT'S OWN call params - i.e. the emitted sequence + * is parameter-correct offline. This knob overrides ONE word of the packed + * image (the exact words the engine loads into its IRAM) so a live A/B can + * mutate the emitted method words one at a time (BEGIN data / SVS / DRAW + * count / END) and observe the engine's writes: the packed image is rebuilt + * per launch, so the value is live-tunable without a reload. Off by + * default (~0u); both params runtime-settable. + */ +static unsigned int gsp_phys_gr_mme_irpatch_off = ~0u; +module_param_named(sema_gl_mme_irpatch_off, gsp_phys_gr_mme_irpatch_off, + uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_irpatch_off, + "IR word index in the packed MME program to override for A/B (default 4294967295 = off)"); +static unsigned int gsp_phys_gr_mme_irpatch_val; +module_param_named(sema_gl_mme_irpatch_val, gsp_phys_gr_mme_irpatch_val, + uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_irpatch_val, + "Value written at sema_gl_mme_irpatch_off in the packed MME program (default 0)"); + +/* + * SESSION 153-CONTINUED-107: IRAM chronology + load-gate helpers. + * + * CONT-106 proved the trap-time fetch at PC 0x364 (macro 57 entry) reads + * ZERO IR words while the host-side pack is verbatim (CONT-105), and the + * IRAM debug port at 0x4044d0 also reads zero at 0x364 on that boot while + * the same port read the golden first words at pc=0 on r1364. The one + * unproven link is the LOAD: did the emitted 0x114 pack writes land in the + * MME's IRAM before the macro executed? These helpers read the live IRAM + * back and (a) log the timeline around each launch's fault-register + * transitions, (b) gate the real CALL runs on the readback matching the + * packed image at every covered SA entry. + * + * The probe reads the macro-57 entry group (0x364) under BOTH port address + * conventions - instruction base (write 0x364) and word base (write + * 3*0x364 = 0xA2C) - plus word 0, and prints the live words next to the + * pack-image expectation, so one line answers "is the program resident + * right now and which address unit does the port use". + */ +static void gsp_phys_mme_iram_probe(void __iomem *bar0, const char *tag, + u32 h_object) +{ + u32 a[3], bi[3], bw[3]; + + if (!bar0) + return; + gsp_phys_mme_iram_read3(bar0, 0u, a); + gsp_phys_mme_iram_read3(bar0, 0x364u, bi); + gsp_phys_mme_iram_read3(bar0, 3u * 0x364u, bw); + pr_info("vgsp-gr: MME-IRAM %s h=0x%x w0=%08x,%08x,%08x pack0=%08x,%08x,%08x i364=%08x,%08x,%08x w364=%08x,%08x,%08x pack364=%08x,%08x,%08x\n", + tag, h_object, a[0], a[1], a[2], + gsp_phys_gr_mme_ir_pack[0], + gsp_phys_gr_mme_ir_pack[1], + gsp_phys_gr_mme_ir_pack[2], + bi[0], bi[1], bi[2], + bw[0], bw[1], bw[2], + gsp_phys_gr_mme_ir_pack[3u * 0x364u], + gsp_phys_gr_mme_ir_pack[3u * 0x364u + 1u], + gsp_phys_gr_mme_ir_pack[3u * 0x364u + 2u]); +} + +/* + * SESSION 153-CONTINUED-107 gate knob (see the helper comment above). + * Default 0: A/B flip - with the gate on, a launch whose readback shows the + * IRAM not holding the packed image holds its real CALL runs and owes a + * full re-emission (prelude latch cleared; pack/drain cursors rewound), so + * the calls only ever execute against a verified-resident program. + */ +static bool gsp_phys_rmcall_gr_mme_iram_gate; +module_param_named(sema_gl_mme_iram_gate, gsp_phys_rmcall_gr_mme_iram_gate, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_iram_gate, + "Read the MME IRAM back (0x4044d0 port) and hold/re-emit the packed prelude until it reads back the packed image at the SA entries (default 0; the CONT-106 all-zero fetch)"); + +/* + * SESSION 153-CONTINUED-108: site control for the IRAM-gate readbacks. + * + * r1374 chronology (first gate boot that released calls): the carrier + * launch (window dropped, no calls) read grfifo=01010001 (empty) at +5 ms; + * the poll-break gate then wrote the MME debug port (0x4044d0) DURING that + * live launch; the first sample of the next launch 2.9 s later read + * grfifo=07071602 (not-empty + FULL, COUNT=22) with NO launch active in + * between - the front end never drained again for the rest of the boot + * (26 launches, all beginning FULL). CONT-107 already recorded the sibling + * evidence: every probe-bearing cycle (r1369..r1371) parked the engine on + * the first X-channel launches while the readback-free cycle (r1368) ran + * the full lane. Both gate sites write the same port, so each gets its own + * knob for the A/B: + * - break (default 0): the poll-break readback after the carrier + * release. Off by default - it is the r1374 suspect. Skipping it does + * not skip the live-gen promotion (that keys on the engine GP_GET + * fetch witness); the build site still gates the calls. + * - build (default 1): the pre-doorbell readback that authorizes the + * calls-bearing launch. Kept on; flip to 0 at runtime to isolate it. + */ +static bool gsp_phys_rmcall_gr_mme_iram_gate_break; +module_param_named(sema_gl_mme_iram_gate_break, + gsp_phys_rmcall_gr_mme_iram_gate_break, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_iram_gate_break, + "Do the IRAM-gate readback at the launch poll break (writes the debug port while the carrier is finishing; default 0 - the r1374 FIFO-fill suspect)"); + +static bool gsp_phys_rmcall_gr_mme_iram_gate_build = true; +module_param_named(sema_gl_mme_iram_gate_build, + gsp_phys_rmcall_gr_mme_iram_gate_build, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_iram_gate_build, + "Do the IRAM-gate readback in the launch builder before the doorbell (default 1; the calls-authorizing check)"); + +/* + * SESSION 153-CONTINUED-107: separate knob for the per-launch chronology + * probe lines. They write the MME debug port DURING a live launch, and the + * first three probe-bearing cycles (r1369..r1371) all parked the engine on + * the first X-channel launches while the last probe-free cycle (r1368) ran + * the full lane; until the port's side effects are cleared, keep the probe + * off by default and enable it only on a known-good boot (A/B). + */ +module_param_named(sema_gl_mme_iram_probe, gsp_phys_rmcall_gr_mme_iram_probe, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_iram_probe, + "Per-launch MME-IRAM chronology probe lines in the post-doorbell spin (writes the debug port during a live launch; default 0)"); +MODULE_PARM_DESC(sema_gl_mme_iram_probe, + "Per-launch MME-IRAM chronology probe lines in the post-doorbell spin (writes the debug port during a live launch; default 0)"); + +/* + * Read every covered SA entry from the live IRAM and compare with the packed + * image. Auto-calibrates the port's address unit once (word base 3*entry vs + * instruction base entry - the r1364 validation could only settle address 0). + * Returns true when every checked entry matches (or there is nothing to + * check); false on any mismatch (all-zero included). Bounded logging. + */ +static bool gsp_phys_gr_mme_iram_gate(void __iomem *bar0, u32 h_object, + u32 gen, const char *tag) +{ + static u32 gate_run_n, gate_fail_n, gate_pass_n; + static int base_sel = -1; /* -1 unresolved; 0 word; 1 instruction */ + u32 sa_ent[64], i; + u32 checked = 0, match = 0, zero = 0; + u32 bad_slot = 0xffffffffu, bad_entry = 0, checked_to_bad = 0; + u32 got0 = 0, got1 = 0, got2 = 0, exp0 = 0, exp1 = 0, exp2 = 0; + u64 sa_mask; + + if (!bar0) + return true; + sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + for (i = 0; i < 64u; i++) { + u32 entry, base, g0, g1, g2, got[3]; + + if (!(sa_mask & (1ull << i)) || !sa_ent[i]) + continue; + entry = sa_ent[i]; + if (entry * 3u + 2u >= 4096u) + continue; + g0 = gsp_phys_gr_mme_ir_pack[entry * 3u]; + g1 = gsp_phys_gr_mme_ir_pack[entry * 3u + 1u]; + g2 = gsp_phys_gr_mme_ir_pack[entry * 3u + 2u]; + /* + * Zero-gold entries are stub-covered (no captured code) or + * not part of the packed image; they carry nothing to gate. + */ + if (!g0 && !g1 && !g2) + continue; + if (base_sel < 0 && entry) { + /* + * One-time unit calibration on the first non-zero + * entry: try the word base, then the instruction + * base. Leaving it unresolved simply keeps the word + * base (the documented form). + */ + gsp_phys_mme_iram_read3(bar0, 3u * entry, got); + if (got[0] == g0 && got[1] == g1 && got[2] == g2) { + base_sel = 0; + pr_info("vgsp-gr: MME-IRAM-BASE resolved=word (3*entry) at entry=0x%x\n", + entry); + } else { + gsp_phys_mme_iram_read3(bar0, entry, got); + if (got[0] == g0 && got[1] == g1 && + got[2] == g2) { + base_sel = 1; + pr_info("vgsp-gr: MME-IRAM-BASE resolved=instruction (entry) at entry=0x%x\n", + entry); + } + } + } + base = (base_sel == 1) ? entry : 3u * entry; + gsp_phys_mme_iram_read3(bar0, base, got); + checked++; + if (got[0] == g0 && got[1] == g1 && got[2] == g2) { + match++; + continue; + } + if (!got[0] && !got[1] && !got[2]) + zero++; + if (bad_slot == 0xffffffffu) { + bad_slot = i; + bad_entry = entry; + checked_to_bad = checked; + got0 = got[0]; got1 = got[1]; got2 = got[2]; + exp0 = g0; exp1 = g1; exp2 = g2; + } + } + if (!checked || match == checked) { + if (gate_pass_n < 8u) { + gate_pass_n++; + pr_info("vgsp-gr: MME-IRAM-GATE pass %s h=0x%x gen=%u base=%d checked=%u run=%u\n", + tag, h_object, gen, base_sel, checked, + gate_run_n); + } + gate_run_n++; + return true; + } + gate_run_n++; + if (gate_fail_n < 24u) { + gate_fail_n++; + pr_warn("vgsp-gr: MME-IRAM-GATE FAIL %s h=0x%x gen=%u base=%d checked=%u match=%u zero=%u bad slot=%u(entry=0x%x @%u) got=%08x,%08x,%08x want=%08x,%08x,%08x run=%u\n", + tag, h_object, gen, base_sel, checked, match, zero, + bad_slot, bad_entry, checked_to_bad, + got0, got1, got2, exp0, exp1, exp2, gate_run_n); + } + return false; +} + +/* + * SESSION 153-CONTINUED-86: the missing SPH (shader-program-header) state. + * + * The executed beef launches fault `Graphics Exception: Shader Program + * Header 9/11/18 Error` + `DS_HWW_ESR 0x405840=0xa2040a00` (r1272): the DS + * unit validates the header of every pipeline program bound when a draw + * issues, including the SPH VERSION against the channel's programmed + * window. The Turing channel-init (the 3D driver's own + * _nv3dInitChannelTuring) writes the SPH version pair ONCE per channel: + * + * INC run @0x16a4, count 2: + * METHOD 0x16a4 (SET_SPH_VERSION) <- 4 | (4 << 16) = 0x00040004 + * METHOD 0x16a8 (CHECK_SPH_VERSION) <- 0x00040004 + * + * (fields: CURRENT 15:0, OLDEST_SUPPORTED 31:16; the value 4 is the SPH V4 + * the Turing shader images carry). The replayed context never receives it: + * the capture families (MME prelude 0x114..0x124/0x234/0x550..0x568/ + * 0x3400..0x34fc, attr prelude 0x1100..0x11fc/0x2000..0x205c) do not cover + * 0x16a4, and the captured 3D windows contain no such write (verified in the + * emitted-window decode and the raw guest-RAM stream). With the register + * unset (0/0) every fetched SPH fails its version window -> per-SPH error + * bits -> 0 GR-STATE retires and no stores. The client's own emitted init + * pair was located verbatim in the guest pushbuffer image (guest PA + * 0x8672d5b4: `200205a9 00040004 00040004`), so the launch builder emits + * exactly that run ahead of the window (right after SET_OBJECT, beside the + * MME version write). 3 words, idempotent; emitted on every replayed 3D + * window. Default 1; A/B with 0 reproduces the SPH fault. + */ +static bool gsp_phys_rmcall_gr_sph_version = true; +module_param_named(sema_gl_sph_version, gsp_phys_rmcall_gr_sph_version, + bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_version, + "Emit the Turing channel-init SET_SPH_VERSION/CHECK_SPH_VERSION pair (0x16a4/0x16a8 <- 0x00040004) ahead of each replayed 3D window (default 1; A/B with 0 reproduces the Shader Program Header 9/11/18 fault)"); + +/* + * SESSION 153-CONTINUED-93: forced SPH version-window value. 0 = auto (the + * blob-derived window, see the emission site); nonzero = emit exactly this + * 32-bit value for both 0x16a4/0x16a8. Live A/B for the DS version-window + * hypothesis: the emitted 4/4 window may reject the fetched SPHs while a + * wider/older window (e.g. 0x00040003) accepts them. + */ +static u64 gsp_phys_rmcall_gr_sph_version_val; +module_param_named(sema_sph_version_val, gsp_phys_rmcall_gr_sph_version_val, + ullong, 0644); +MODULE_PARM_DESC(sema_sph_version_val, + "Forced SET_SPH_VERSION/CHECK_SPH_VERSION value (0 = auto from the loaded blobs)"); + +/* + * SESSION 153-CONTINUED-90: SET_SHADER_EXCEPTIONS (0x1528 <- 0). + * + * NVIDIA's own Turing 3D channel init (_nv3dInitChannelTuring) pushes this + * FIRST, with the comment "Disable shader exceptions. This matches OpenGL + * driver behavior." The client's captured init stream carries the write too + * (sema-pushraw `off=0x5a4 op=4 addr=0x1528 v0=0`). The replayed context + * never receives it: the emitted push (PB-FAULT-DUMP) goes SET_OBJECT -> + * SPH version pair -> MME version with no 0x1528 run, so the DS keeps + * raising "Graphics Exception: Shader Program Header 5/9/10/11 Error" on + * every draw whose fetched SPH it would tolerate with exceptions disabled - + * and the exception poisons the channel before any store retires. An + * executing r1294 launch names the consequence: the first post-exception + * method write reports `ChID 0001, Class 0000c597, Offset 0000022c, Data + * 00000800` (SET_GO_IDLE_TIMEOUT poisoned by the exception state). + * Emitted right after SET_OBJECT, exactly like the reference init order. + * Default 1; A/B with 0 restores the old behavior. + */ +static bool gsp_phys_rmcall_gr_shader_exc = true; +module_param_named(sema_gl_shader_exc, gsp_phys_rmcall_gr_shader_exc, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shader_exc, + "Emit SET_SHADER_EXCEPTIONS (0x1528) <- 0 after SET_OBJECT like _nv3dInitChannelTuring (default 1)"); + +/* + * SESSION 154-1: encoding of the emitted SET_SHADER_EXCEPTIONS write. + * + * The client (and NVIDIA's _nv3dInitChannelTuring, which uses + * nvPushImmedVal) writes this method in the one-word immd13 form: header + * 0x80000000 | (value << 16) | (method >> 2), value 0, no data word - the + * captured init pushraw line is `off=0x5a4 op=4 addr=0x1528 cnt=0 v0=0`. + * The booter historically emitted the two-word INC form + * `0x2001054a 0x00000000`. Both encode the same register write, but the + * INC form is the method the FE's trapped-method register names + * (r1483 `trap=80001528`, mthd=0x1528) on every replay, while the + * client's own immd13 write is accepted (the X window's later 0x22c is + * what the trap advances to). Default 1 emits the client's exact form; + * A/B with 0 restores the legacy INC pair. + */ +static bool gsp_phys_rmcall_gr_shader_exc_immd = true; +module_param_named(sema_gl_shader_exc_immd, + gsp_phys_rmcall_gr_shader_exc_immd, bool, 0644); +MODULE_PARM_DESC(sema_gl_shader_exc_immd, + "Emit SET_SHADER_EXCEPTIONS (0x1528) in the client's immd13 form (0x8000054a, one word) instead of the legacy INC pair (default 1)"); + +/* + * SESSION 154-1: do not duplicate a window-carried SET_SHADER_EXCEPTIONS. + * The X channel's captured per-frame window carries the client's own + * 0x1528 run (the full channel-init block is inside it), so emitting the + * booter's copy again just adds an extra method ahead of the client's + * init. When this is on and the replayed window already contains a 0x1528 + * write, the emission is skipped (logged). Default 1; A/B with 0 keeps + * the unconditional emission. + */ +static bool gsp_phys_rmcall_gr_shader_exc_skip = true; +module_param_named(sema_gl_shader_exc_skip, + gsp_phys_rmcall_gr_shader_exc_skip, bool, 0644); +MODULE_PARM_DESC(sema_gl_shader_exc_skip, + "Skip the emitted SET_SHADER_EXCEPTIONS when the captured window already carries a 0x1528 write (default 1)"); + +/* + * SESSION 153-CONTINUED-91: clear DS_HWW_REPORT_MASK/_2 (0x405844/0x40584c) + * ahead of every replayed 3D window (host write + in-stream MME PRI R/W + * macro 5) so a DS SPH validation miss latches without poisoning the PRI + * block and the GSP-RM exception handler killing the channel. See the + * emission-site comment. Default 1; A/B with 0 restores the old behavior + * (every SPH miss = priv-error poison + channel wedge). + */ +static bool gsp_phys_rmcall_gr_ds_mask_clear = true; +module_param_named(sema_ds_mask_clear, gsp_phys_rmcall_gr_ds_mask_clear, + bool, 0644); +MODULE_PARM_DESC(sema_ds_mask_clear, + "Clear DS_HWW_REPORT_MASK/_2 ahead of replayed 3D windows (default 1; 0 = leave the all-report mask)"); + +/* + * SESSION 153-CONTINUED-91: emit the Turing program-region pair (the MME + * shadow scratch 0x78/0x79 = NVC397_SET_MME_PROGRAM_REGION) with the + * launch's own program family page as the shader-heap base, so the DS's + * SPH validation has a region to accept the fetched SPHs against. See the + * emission-site comment. Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sph_region = true; +module_param_named(sema_gl_sph_region, gsp_phys_rmcall_gr_sph_region, + bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_region, + "Emit the Volta+ program-region pair (SET_MME_SHADOW_SCRATCH 0x78/0x79 <- program family page) ahead of replayed 3D windows (default 1)"); + +/* + * Runtime-selectable program-region base. 0 = auto (the launch's own + * program family page, the guest-VA heap base the real driver would use); + * a nonzero value forces that base - the engine-side shadow-VA space needs + * its own experiment (the SPH_INFO snapshot shows the DS fetched through + * shadow VAs 0x620900/0x620200/0x620000, so a guest-VA base may be the + * mismatch). Writable at runtime for a live A/B. + */ +static u64 gsp_phys_rmcall_gr_sph_region_base; +module_param_named(sema_sph_region_base, gsp_phys_rmcall_gr_sph_region_base, + ullong, 0644); +MODULE_PARM_DESC(sema_sph_region_base, + "Program-region base override (0 = launch program family page; nonzero = explicit engine-side base)"); + +/* + * SESSION 154-1: prefer the bound program's own engine-visible VA page + * as the program region base. The reference driver sets the region to + * the program's GPU address (nv3dGetProgramGpuAddress); the replay's + * program is fetched at the relocated shadow VA, so the region must be + * that VA's 4K page. The forced sema_sph_region_base (0x1500000 in the + * campaign args) is the shadow pool's base: the X program's shadow + * 0x15c0000 lands inside it, but the beef program's 0x19a0000 does not, + * and the DS rejects the fetched SPH (r1484 `GR-STATE sph` stages 1/2/5 + * = 0x19a0f00/0x19a0200/0x19a0000 together with `ds=86040646`). With + * this on, the builder scans the already-relocated window for the first + * SET_PIPELINE_PROGRAM_ADDRESS pair and uses that VA's page as the base + * (beef: 0x19a0000), falling back to the override/auto only when the + * window carries no program bind. Default 1; A/B with 0 uses the old + * forced/auto base. + */ +static bool gsp_phys_rmcall_gr_sph_region_bind = true; +module_param_named(sema_gl_sph_region_bind, + gsp_phys_rmcall_gr_sph_region_bind, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_region_bind, + "Use the first bound program's relocated shadow page as the program-region base (default 1; 0 = forced/auto base)"); + +/* + * SESSION 153-CONTINUED-101: MME fixed-constant-buffer scratch seed + * (SET_MME_SHADOW_SCRATCH(0x18), method 0x3460). + * + * mmeCodeOGL_C597_SETUP_FIXED_CONST (macro slot 0) reads shadow scratch + * 0x18/0xd18 via STATE and unconditionally emits its high/low halves as + * SET_CONSTANT_BUFFER_SELECTOR_B/C (the compiled C597 form carries NO + * bug2548370 branch: the config table says no972630). With the replay + * context's scratch at 0 the macro emits selector B/C = 0 and the draw's + * constant fetch has no address. The client's own write (gf100Init- + * ConstantBuffers: (u32)(fixed CB VA >> 8)) is a once-per-channel event + * the ladder never receives, so the builder seeds it ahead of the + * window. Value source: sema_gl_mme_seed_val when nonzero, else the + * client's own captured write (gsp_sema_mme_scratch18_get). With + * sema_gl_mme_seed_xlat (default 1) the page is translated through the + * launch's own page set / shadow mapping first (the same relocator the + * window operand pairs use) so the emitted selector resolves in the + * engine's address space; on MISS the raw value is emitted (the A/B + * control). Default off; live-tunable. + */ +static bool gsp_phys_rmcall_gr_mme_seed; +module_param_named(sema_gl_mme_seed, gsp_phys_rmcall_gr_mme_seed, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_seed, + "Seed MME shadow scratch 0x18 (method 0x3460) with the fixed-CB base>>8 ahead of replayed 3D windows (default 0)"); +static u32 gsp_phys_rmcall_gr_mme_seed_val; +module_param_named(sema_gl_mme_seed_val, gsp_phys_rmcall_gr_mme_seed_val, + uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_seed_val, + "Explicit scratch 0x18 value (fixed CB VA>>8); 0 = use the client's captured 0x3460 write"); +static bool gsp_phys_rmcall_gr_mme_seed_xlat = true; +module_param_named(sema_gl_mme_seed_xlat, gsp_phys_rmcall_gr_mme_seed_xlat, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_seed_xlat, + "Translate the seed value's page to the engine-side shadow/identity VA before emitting (default 1; 0 = raw)"); + +/* + * SESSION 153-CONTINUED-105: golden MME runtime state seed, STATE + * 0xD26..0xD2A (SET_MME_SHADOW_SCRATCH 0x26..0x2a, methods 0x3498/0x349c/ + * 0x34a0/0x34a4/0x34a8). The bug2548370 loadConstantBuffer macro (slot 57 / + * macro address 0x39) branches at its entry on these five words: + * + * 0xD26 BUG2548370_PAGE_ALLOCATION_COUNT + * 0xD27 BUG2548370_ZERO_VIEW_OFFSET + * 0xD28 BUG2548370_LAST_UPDATED_CONSTANT_BUFFER_SELECTOR_B_C + * 0xD29 BUG2548370_LOAD_CONSTANT_BUFFER_OFFSET + * 0xD2A BUG2548370_LAST_UPDATED_LOAD_CONSTANT_BUFFER_OFFSET + * + * (verified against the driver's generated macro source: the entry reads + * exactly 0xd28/0xd19/0xd29/0xd2a/0xd26/0xd27 and the WAR path compares + * 0xd26 against 30 and 0xd27 against zero). The client's own prelude + * writes only 0xd2a <- 0x80000000 / 0xd29 <- 0 ("reset tracking after each + * draw") and the per-channel 0x3464/0x3460 writes; 0xd26/0xd27/0xd28 carry + * over from the client's own context and never appear in the captured + * stream. The replayed context therefore reads whatever the replay left + * there. Emit the five writes ahead of every replayed 3D window (values + * from sema_mme_state_d26..d2a; all-zero default = the zero-valid form). + * Default off; live-tunable A/B. + */ +static bool gsp_phys_rmcall_gr_mme_state_seed; +module_param_named(sema_gl_mme_state_seed, gsp_phys_rmcall_gr_mme_state_seed, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_state_seed, + "Seed MME runtime state 0xD26..0xD2A (methods 0x3498..0x34a8) ahead of replayed 3D windows (default 0)"); +static u32 gsp_phys_rmcall_gr_mme_state_d26; +module_param_named(sema_mme_state_d26, gsp_phys_rmcall_gr_mme_state_d26, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d26, "Seed value for STATE 0xD26 PAGE_ALLOCATION_COUNT (method 0x3498)"); +static u32 gsp_phys_rmcall_gr_mme_state_d27; +module_param_named(sema_mme_state_d27, gsp_phys_rmcall_gr_mme_state_d27, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d27, "Seed value for STATE 0xD27 ZERO_VIEW_OFFSET (method 0x349c)"); +static u32 gsp_phys_rmcall_gr_mme_state_d28; +module_param_named(sema_mme_state_d28, gsp_phys_rmcall_gr_mme_state_d28, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d28, "Seed value for STATE 0xD28 LAST_UPDATED_CB_SELECTOR_B_C (method 0x34a0)"); +static u32 gsp_phys_rmcall_gr_mme_state_d29; +module_param_named(sema_mme_state_d29, gsp_phys_rmcall_gr_mme_state_d29, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d29, "Seed value for STATE 0xD29 LOAD_CONSTANT_BUFFER_OFFSET (method 0x34a4)"); +static u32 gsp_phys_rmcall_gr_mme_state_d2a; +module_param_named(sema_mme_state_d2a, gsp_phys_rmcall_gr_mme_state_d2a, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d2a, "Seed value for STATE 0xD2A LAST_UPDATED_LOAD_CONSTANT_BUFFER_OFFSET (method 0x34a8)"); +static u32 gsp_phys_rmcall_gr_mme_state_d19; +module_param_named(sema_mme_state_d19, gsp_phys_rmcall_gr_mme_state_d19, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d19, "Seed value for STATE 0xD19 CONSTANT_BUFFER_SELECTOR_B_C (method 0x3464, CB VA>>8); 0 = use the client's captured 0x3464 write"); +static u32 gsp_phys_rmcall_gr_mme_state_d24; +module_param_named(sema_mme_state_d24, gsp_phys_rmcall_gr_mme_state_d24, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d24, "Seed value for STATE 0xD24 slow-path input (method 0x3490); 0 = use the client's captured 0x3490 write"); +static u32 gsp_phys_rmcall_gr_mme_state_d25; +module_param_named(sema_mme_state_d25, gsp_phys_rmcall_gr_mme_state_d25, + uint, 0644); +MODULE_PARM_DESC(sema_mme_state_d25, "Seed value for STATE 0xD25 BUG2548370_CONSTANT_BUFFER_SELECTOR_A / CB size (method 0x3494); 0 = use the client's captured 0x3494 write"); + +/* + * SESSION 155-CONT-5: engine-visible constant-buffer selector seed. + * + * The STATE seed above only writes MME *shadow scratch* (0x3460/0x3464/ + * 0x3498..0x34a8). The bug2548370 loadConstantBuffer macro (slot 57) + * only *reads* that scratch to track page changes; it never emits + * SET_CONSTANT_BUFFER_SELECTOR_A/B/C itself. The selector is programmed + * by SETUP_FIXED_CONST (slot 0) from scratch 0x18. When a replayed + * window carries a CALL macro 57 run but no CALL macro 0 (the driver + * legitimately elides it while the selector is already bound, e.g. the + * r1541 68-word tail), the engine executes LOAD_CONSTANT_BUFFER with no + * valid selector and the C597 class raises + * "Offset 00002390, Data bf800000, ErrorCode 00000003" (the r1456/ + * r1485/r1541 poison). With this knob the builder emits the same + * 4-method group macro 0 would emit - SET_CONSTANT_BUFFER_SELECTOR_A + * (size = MME_VPC_CONSTANT_BUFFER_SIZE 0x7000), B/C (the fixed-CB + * base>>8, translated through the launch's page set exactly like the + * scratch seed) and LOAD_CONSTANT_BUFFER_OFFSET (sema_mme_state_d29) - + * ahead of every replayed 3D window. Idempotent when macro 0 is also in + * the stream (it re-programs the same selector). Default off; requires + * sema_gl_mme_state_seed; legacy bytes unchanged when off. + */ +static bool gsp_phys_rmcall_gr_mme_selector_seed; +module_param_named(sema_gl_mme_selector_seed, + gsp_phys_rmcall_gr_mme_selector_seed, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_selector_seed, + "Emit SET_CONSTANT_BUFFER_SELECTOR_A/B/C + LOAD_CONSTANT_BUFFER_OFFSET (0x2380..0x238c) from the fixed-CB scratch ahead of replayed 3D windows (default 0; requires sema_gl_mme_state_seed)"); + +/* + * SESSION 153-CONTINUED-105: split the packed MME prelude emission across + * launches. The CONT-99b pack rebuilds the partitioned IR and emits it as + * ONE contiguous ONEINC run; for the current 55-slot capture that stream is + * 2 + maxa + 3*npair = ~2966 words (maxa = 2799, npair = 55), over the + * 2048-word per-launch block cap, so the pack silently falls back to the + * raw capture-order drain (which is only correct when the capture's runs + * do not overlap). With this knob the packed stream is drained with a + * generation-keyed cursor: the SA table goes out once, the IR goes out as + * whole-instruction chunks in consecutive launches (each chunk = one + * ONEINC 0x114 run with its own instruction-address pointer), and the + * prelude latch is set only when the last chunk is emitted. Calls stay + * held across the chunks (hold_calls + the FE-idle witness). Default 0. + */ +static bool gsp_phys_rmcall_gr_mme_pack_split = true; +module_param_named(sema_gl_mme_pack_split, gsp_phys_rmcall_gr_mme_pack_split, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_pack_split, + "Split the packed MME prelude (SA table + IR chunks) across launches when it exceeds the block cap (default 1; 0 = all-or-nothing CONT-99b pack)"); + +/* + * SESSION 153-CONTINUED-93: additionally emit the legacy program-region + * pair (0x1608/0x160c) beside the MME shadow pair (see the emission site). + * Default 0 (A/B; the DS snapshot registers share the legacy pair's names). + */ +static bool gsp_phys_rmcall_gr_sph_region_legacy; +module_param_named(sema_sph_region_legacy, gsp_phys_rmcall_gr_sph_region_legacy, + bool, 0644); +MODULE_PARM_DESC(sema_sph_region_legacy, + "Also emit the legacy SET_PROGRAM_REGION_A/B (0x1608/0x160c) pair with the same base (default 0)"); + +/* + * SESSION 153-CONTINUED-87: the missing TESS_CONTROL/PIXEL program binds. + * The client's real channel state binds every pipeline stage once per + * context; the captured init-push image (guest RAM scrape) carries + * `20020825 [0, 04400200]` (j=2 TESS_CONTROL) and + * `20020855 [0, 04400000]` (j=5 PIXEL) beside the SPH version pair - + * but the captured per-frame window carries only the j=1 (VTG) pair + * (0x4400900/0x4400300). The DS validates the SPH of every pipeline + * stage the draw activates, so the unbound stages latch their header + * errors (DS_HWW_ESR 0xa2040a00 = SPH9|SPH11|SPH18) while the V3 blob + * passes wherever a stage IS bound. Emit both binds ahead of the + * window at the window's own program-family page (all four VAs share + * one 4K page: 0x4400000 j=5, 0x4400200 j=2, 0x4400300/0x4400900 j=1), + * pushed through the same relocation pass as the window - a MISS skips + * the bind entirely (never a raw client VA). Beef client channels only + * (the family layout is the client's). Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_sph_bind = true; +module_param_named(sema_gl_sph_bind, gsp_phys_rmcall_gr_sph_bind, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_bind, + "Emit the client's missing TESS_CONTROL (j=2) and PIXEL (j=5) pipeline program binds ahead of replayed 3D windows (program-family page of the window's prog VA; default 1)"); + +static u32 gsp_phys_rmcall_gr_sph_bind_mask; /* bit j = synth slot j */ +module_param_named(sema_sph_bind_mask, gsp_phys_rmcall_gr_sph_bind_mask, + uint, 0644); +MODULE_PARM_DESC(sema_sph_bind_mask, + "Synth SPH-BIND slot mask (bit j = slot j; 0x20 = PS only, 0x01/0x10 = VSA/GS VTG image, 0x25 = VSA+PS) used when sema_gl_sph_bind=0"); + +bool gsp_phys_rmcall_gr_sph_bind_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_bind); +} + +/* + * SESSION 155 CONT 25: the synth slot mask must be readable by the sema + * service so it authors exactly the stage images the booter binds. With + * sema_gl_sph_bind=0 + sema_sph_bind_mask=0x35 the booter emits j0/j2/j5, + * but the service's PS/TCS authoring loop was gated on the all-slots bool, + * so the PS image could be left stale/zero at draw time (r1575). + */ +u32 gsp_phys_rmcall_gr_sph_bind_mask_value(void) +{ + return READ_ONCE(gsp_phys_rmcall_gr_sph_bind_mask); +} + +/* + * SESSION 154c (viewport wall): the replayed 3D window carries no + * viewport family state - the colour target, raster enable, shader + * binds, attributes, index buffer and draws are present, but the + * 0x0a00 scale/offset block, the 0x1924/0x192c/0x193c pixel and clip + * controls and the 0x0c00 clip rect are not (the client's captured + * push never writes them; the slot-31 viewport macro exists in the + * capture but is never called). In the fresh replayed context a + * missing viewport clips all geometry, so the engine executes the + * DRAW but stores no fragments. This knob synthesizes the missing + * state as a preamble ahead of each replayed 3D window. + * + * Words verified against the window decoder (op = hdr[31:29], + * count = hdr[28:16], method = hdr[12:0] << 2): + * 0x20060280 INC cnt=6 method 0x0a00 -> SET_VIEWPORT_SCALE_X/ + * Y/Z(0) and SET_VIEWPORT_OFFSET_X/Y/Z(0) + * 0x80000649 IMM data=0 method 0x1924 -> SET_VIEWPORT_PIXEL + * 0x8001064b IMM data=1 method 0x192c -> SET_VIEWPORT_SCALE_OFFSET + * 0x8001064f IMM data=1 method 0x193c -> SET_VIEWPORT_CLIP_CONTROL + * 0x20040300 INC cnt=4 method 0x0c00 -> SET_VIEWPORT_CLIP_ + * HORIZONTAL/VERTICAL/MIN_Z/MAX_Z(0) + * Mode 1 = scale/offset block; mode 2 adds the immediate controls; + * mode 3 adds the 1280x720 clip rect. Mode 4 is the same words with + * the clip-rect encoding verified against the class definitions + * (X0/Y0 in bits 15:0, WIDTH/HEIGHT in bits 31:16, integer pixel + * units) and the producer's (size << 16) | origin packing, so a + * 1280x720 target at a top-left origin emits 0x05000000, + * 0x02d00000, 0.0f (0x00000000) and 1.0f (0x3f800000); min_z/max_z + * are IEEE-754 floats. None of the words is an address operand, so + * the sequence is emitted verbatim (no relocation pass). Default 0 + * keeps the legacy window bytes. + * + * Mode 5 is the driver's own viewport macro path: CALL_MME_MACRO(33) + * (0x3908) with (x,y,w,h) = (0,0,1280,720) followed by the macro's + * appended state. Mode 6 is the raw equivalent without the CALL + * (scale/offset included, scaleY = +360). Header check against the + * decoder above: 0xa0040e42 = op 5 ONE_INC, cnt 4, method 0x3908 + * (four data words follow); 0x8001064b = op 4 IMM, data 1 in the + * count field, method 0x192c (single word, consumes no data); + * 0x2001064f + 0x00021800 = op 1 INC, cnt 1, method 0x193c (one + * data word). + */ +#define GSP_PHYS_GR_VP_PRE_WORDS 15u +static const u32 gsp_phys_gr_vp_pre[GSP_PHYS_GR_VP_PRE_WORDS] = { + /* mode 1: viewport scale/offset */ + 0x20060280u, 0x44200000u, 0xc3b40000u, 0x3f000000u, + 0x44200000u, 0x43b40000u, 0x3f000000u, + /* mode 2: pixel centre, scale/offset enable, clip control */ + 0x80000649u, 0x8001064bu, 0x8001064fu, + /* mode 3: clip rect (x0=0, w=1280, y0=0, h=720, z 0..1) */ + 0x20040300u, 0x05000000u, 0x02d00000u, 0x00000000u, + 0x3f800000u, +}; + +#define GSP_PHYS_GR_VP_PRE_M5_WORDS 17u +static const u32 gsp_phys_gr_vp_pre_m5[GSP_PHYS_GR_VP_PRE_M5_WORDS] = { + 0xa0040e42u, 0x00000000u, 0x00000000u, 0x00000500u, 0x000002d0u, + 0x20010282u, 0x3f000000u, 0x20010285u, 0x3f000000u, + 0x20020302u, 0x00000000u, 0x3f800000u, + 0x20010649u, 0x00000000u, 0x8001064bu, + 0x2001064fu, 0x00021800u, +}; + +#define GSP_PHYS_GR_VP_PRE_M6_WORDS 19u +static const u32 gsp_phys_gr_vp_pre_m6[GSP_PHYS_GR_VP_PRE_M6_WORDS] = { + 0x20020280u, 0x44200000u, 0x43b40000u, + 0x20040282u, 0x3f000000u, 0x44200000u, 0x43b40000u, 0x3f000000u, + 0x20020300u, 0x05000000u, 0x02d00000u, + 0x20020302u, 0x00000000u, 0x3f800000u, + 0x20010649u, 0x00000000u, 0x8001064bu, + 0x2001064fu, 0x00021800u, +}; + +/* + * SESSION 154 r1506: mode 7 = the CLIENT'S OWN window viewport (800x600 at + * the root origin) instead of the captured RT's 1280x720 screen viewport. + * The glxgears window is 800x600+0+0 on the 1280x720 root; its own GL + * viewport is (0,0,800,600) -> scale (400,300), offsets (400,300), clip + * 800x600. Mode 6 maps the geometry across the whole X screen; if the + * client's real draw args are window-relative, mode 7 is the faithful + * transform. Same encoding as mode 6 (scale/offset block, then clip, + * z range, pixel centre, scale-offset enable, clip control). + */ +#define GSP_PHYS_GR_VP_PRE_M7_WORDS 19u +static const u32 gsp_phys_gr_vp_pre_m7[GSP_PHYS_GR_VP_PRE_M7_WORDS] = { + 0x20020280u, 0x43c80000u, 0x43960000u, /* scale 400, 300 */ + 0x20040282u, 0x3f000000u, 0x43c80000u, 0x43960000u, 0x3f000000u, + 0x20020300u, 0x03200000u, 0x02580000u, /* clip 800x600 at 0,0 */ + 0x20020302u, 0x00000000u, 0x3f800000u, + 0x20010649u, 0x00000000u, 0x8001064bu, + 0x2001064fu, 0x00021800u, +}; + +static u32 gsp_phys_gr_vp_pre_len(int mode) +{ + switch (mode) { + case 1: + return 7u; + case 2: + return 10u; + case 3: + case 4: + return GSP_PHYS_GR_VP_PRE_WORDS; + case 5: + return GSP_PHYS_GR_VP_PRE_M5_WORDS; + case 6: + return GSP_PHYS_GR_VP_PRE_M6_WORDS; + case 7: + return GSP_PHYS_GR_VP_PRE_M7_WORDS; + default: + return 0u; + } +} + +static const u32 *gsp_phys_gr_vp_pre_src(int mode) +{ + switch (mode) { + case 5: + return gsp_phys_gr_vp_pre_m5; + case 6: + return gsp_phys_gr_vp_pre_m6; + case 7: + return gsp_phys_gr_vp_pre_m7; + default: + return gsp_phys_gr_vp_pre; + } +} + +/* + * Insertion word offset for the viewport prelude inside an emitted 3D + * window. The walk is the same run grammar as the MME-WINRUN census and + * the CALL drop pass (op = hdr[31:29], count = hdr[28:16] for op 1/3/5, + * run method = hdr[11:0] << 2; multi-word INC/NON_INCR runs cover + * m + 4k, IMM/ONE_INCR entries cover only their own method). Runs are + * never split. pos 1 stops at the first run touching the CALL_MME + * family 0x3800..0x39fc (after the window's state binds, ahead of its + * macro calls); pos 2 stops at the first run touching 0x3910. A window + * without the requested run returns n so the caller can skip it + * (non-draw window). + */ +static u32 gsp_phys_gr_vp_insert_off(const u32 *win, u32 n, int pos) +{ + u32 r = 0; + + if (pos != 1 && pos != 2) + return n; + while (r < n) { + u32 h = win[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 span = 1u, k; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + for (k = 0; k < span; k++) { + u32 m = ((h & 0xfffu) << 2) + 4u * k; + + if (k && op != 1u && op != 3u) + break; + if (pos == 1) { + if (m >= 0x3800u && m <= 0x39fcu) + return r; + } else if (m == 0x3910u) { + return r; + } + } + r += span; + } + return n; +} + +/* + * First-run offset of a method in a window, using the same run grammar + * as the scanners above (op 1/3/5 carry 1+cnt data words; op 5 writes + * only its header method, op 4 IMM carries data in the count field and + * is a single word). Returns n when no run touches the method. + */ +static u32 gsp_phys_gr_method_run_off(const u32 *win, u32 n, u32 method) +{ + u32 r = 0; + + while (r < n) { + u32 h = win[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 span = 1u, k; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + for (k = 0; k < span; k++) { + u32 m = ((h & 0xfffu) << 2) + 4u * k; + + if (k && op != 1u && op != 3u) + break; + if (m == method) + return r; + } + r += span; + } + return n; +} + +/* + * The client's pixel shader has MRT_ENABLE=0 and writes RT0 (0x800), + * while its init push selects CT target0=0 (0x121c = 0x0fac6881) and + * the replayed draw window binds only RT1 (0x840) - so the colour + * write has no enabled target. When sema_gl_rt0_bind=1 and the window + * does not write 0x800 itself, insert the missing RT0 bind (same + * 1280x720 X8R8G8B8 target at B=0x0a000000) plus a count-2 CT select + * (target0=0, target1=1) ahead of the window's first 0x840 run, and + * only when the window carries a CALL run (0x3800..0x39fc); a non-draw + * window is left unchanged. + * Constant words, no relocation. Default 0 is a strict no-op. + */ +#define GSP_PHYS_GR_RT0_BIND_WORDS 9u +static const u32 gsp_phys_gr_rt0_bind[GSP_PHYS_GR_RT0_BIND_WORDS] = { + 0x20060200u, 0x00000000u, 0x0a000000u, 0x00000500u, 0x000002d0u, + 0x000000e6u, 0x00000040u, 0x20010487u, 0x0fac6882u, +}; + +/* + * SESSION 155-CONT-6: the synthesized RT0 bind above writes + * SET_COLOR_TARGET_A..MEMORY (0x800..0x814) but not + * SET_COLOR_TARGET_THIRD_DIMENSION (0x818). The client's own full + * colour-target group is the 8-method form (gf100pushbuf.c / the + * gf100commandList.c 6-word push from WIDTH through ARRAY_PITCH, which + * includes THIRD_DIMENSION and ARRAY_PITCH); those load-only windows are + * not replayed, so the channel's 0x818 stays 0 while 0x814 carries + * THIRD_DIMENSION_CONTROL = DEFINES_ARRAY_SIZE. On the first native + * DRAW_INDEX_BUFFER (0x17e0) the C597 class then reports + * "ErrorCode 000000d2" = + * NV_PGRAPH_CLASS_ERROR_CODE_ERROR_ILLEGAL_COLOR_TARGET0_LAYER_OFFSET_VS_THIRD_DIMENSION + * (tu104/dev_graphics_nobundle.h:563): layer offset 0 is not below the + * array size 0. The driver's layer-mode ONE path writes depth 1 with + * ARRAY_SIZE control (gf100commandList.c:1232-1235), so widen the + * inserted group to the 8-method form with THIRD_DIMENSION(0)=1 and + * ARRAY_PITCH(0)=0. Default off; legacy bytes and count unchanged. + */ +#define GSP_PHYS_GR_RT0_BIND_EXT_WORDS 11u +static const u32 gsp_phys_gr_rt0_bind_ext[GSP_PHYS_GR_RT0_BIND_EXT_WORDS] = { + 0x20080200u, 0x00000000u, 0x0a000000u, 0x00000500u, 0x000002d0u, + 0x000000e6u, 0x00000040u, 0x00000001u, 0x00000000u, + 0x20010487u, 0x0fac6882u, +}; + +static bool gsp_phys_rmcall_gr_rt0_third_dim; +module_param_named(sema_gl_rt0_third_dim, + gsp_phys_rmcall_gr_rt0_third_dim, bool, 0644); +MODULE_PARM_DESC(sema_gl_rt0_third_dim, + "Extend the synthesized RT0 bind with SET_COLOR_TARGET_THIRD_DIMENSION(0)=1 and ARRAY_PITCH(0)=0 (0x818/0x81c) so DRAW_INDEX_BUFFER does not raise class error 0xd2 (default 0)"); + + +static int gsp_phys_rmcall_gr_rt0_bind; +module_param_named(sema_gl_rt0_bind, gsp_phys_rmcall_gr_rt0_bind, int, + 0644); +MODULE_PARM_DESC(sema_gl_rt0_bind, + "Insert the missing RT0 colour-target bind and count-2 SET_CT_SELECT ahead of a replayed window's first 0x840 (RT1) run when the window does not bind 0x800 and carries a CALL run (non-draw windows are left unchanged; default 0 = off)"); + +static int gsp_phys_rmcall_gr_vp_prelude; +module_param_named(sema_gl_vp_prelude, gsp_phys_rmcall_gr_vp_prelude, + int, 0644); +MODULE_PARM_DESC(sema_gl_vp_prelude, + "Synthesize the missing viewport state ahead of replayed 3D windows (0 = off (default); 1 = scale/offset; 2 = + pixel centre and clip control; 3 = + clip rect; 4 = same clip rect with the verified pixel-unit encoding; 5 = driver viewport macro CALL(33) 0x3908 + appended state; 6 = raw mode-5 state without the CALL; 7 = mode-6 encoding for the client window viewport 800x600 at 0,0)"); + +static int gsp_phys_rmcall_gr_vp_prelude_pos = 1; +module_param_named(sema_gl_vp_prelude_pos, gsp_phys_rmcall_gr_vp_prelude_pos, + int, 0644); +MODULE_PARM_DESC(sema_gl_vp_prelude_pos, + "Insertion point of the synthesized viewport prelude in a replayed 3D window (0 = window head (legacy); 1 = before the first CALL run 0x3800..0x39fc, after the state binds (default); 2 = before the first 0x3910 run); pos 1/2 leave a window without the requested run unchanged (non-draw window), only 0 inserts without a CALL run)"); + +/* + * r1512 diagnostic export: the sema service's draw-coverage receipt needs + * the viewport state the replay will inject for the current launch. Pure + * read of the module parameter + the literal table (no side effects). + */ +void gsp_phys_gr_vp_state_get(int *mode, u32 *words, u32 *n) +{ + int m = READ_ONCE(gsp_phys_rmcall_gr_vp_prelude); + u32 len = gsp_phys_gr_vp_pre_len(m); + const u32 *src = gsp_phys_gr_vp_pre_src(m); + + if (mode) + *mode = m; + if (n) + *n = len; + if (words && len && src) + memcpy(words, src, (size_t)len * 4u); +} + +/* + * CONT-119 (r1450): the BINDING words of the synthesized stage-2/5 + * REGISTER_COUNT runs. CONT-90 emitted each as a 2-word + * REGISTER_COUNT/BINDING run (`20020823 0000001a 00000000` for j=2, the + * j=5 twin on 0x214c) so the DS SPH validation saw a full stage-1-shaped + * pair. The FE walking our replayed window then rejects the synthesized + * `0x2090 <- 0` write with Class Error EC4 - it is the only 0x2090 writer + * in the stream (the client never emits a stage-2/5 BINDING), and it + * wedges the slot-57 macro at mpc 899/900. Default 0 drops the BINDING + * data words and emits only the 1-word REG_COUNT runs + * (`20010823 0000001a`, `20010853 0000001a`); 1 keeps the legacy CONT-90 + * bytes for a same-boot A/B. + */ +static bool gsp_phys_rmcall_gr_sph_bind_binding; +module_param_named(sema_gl_mme_sph_bind_binding, + gsp_phys_rmcall_gr_sph_bind_binding, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_sph_bind_binding, + "1 = legacy emit BINDING(2)/(5)=0 which the FE rejects EC4 @0x2090 when walking the window; 0 = drop the BINDING writes, keep the REG_COUNT runs (default 0)"); + +/* + * SESSION 153-CONTINUED-92: leave program-family operands at their ORIGINAL + * guest VAs when the operand's page is resolved and identity-mappable. + * + * The DS I-fetch reaches the delivered images through the shadow VAs the + * window pass picks (r1298-r1306: stages fetched at 0x620900/0x620200/ + * 0x620000) and rejects them with the bytes provably correct at those VAs + * (INLINE-RB rb=020a0481; canary proves the fetch follows the mapped page's + * content). The one variable left is the address space the DS validates in: + * the client programmed its own guest-VA program heap, and the shadow VAs + * (0x2xxxxx..0x6xxxxx pools) sit outside any region the client's state + * describes - a region/space mismatch rejects all bound stages regardless + * of content (rr=0/0, ds=a400e20 at every retire). With this knob the + * window pass first tries the Tier-1 identity mapper on the operand's own + * guest VA (launch page set first, the sema resolver else): when the guest + * VA is mapped to the resolved HPA, the operand is left verbatim, so the + * DS fetches and validates the SPH at the same VA space the client's heap + * uses. A page without an identity mapping keeps the shadow rewrite exactly + * as before (fail-open), and the identity-kept pairs still count toward the + * pass's relocated census so the SPH-BIND guard (orel == nrun) is stable. + * r1307 RESULT (experiment executed, negative): with the knob on, the + * beef lane's program pairs stayed at their guest VAs (0x4350900/0x4350f00/ + * 0x4350200/0x4350000, PROG-GUEST-VA receipts) and the engine faulted + * `MMU Fault: ENGINE GRAPHICS HUBCLIENT_PD faulted @ 0x0_04350000. FAULT_PTE + * ACCESS_TYPE_VIRT_READ` within 2.5 s of every such launch - the Tier-1 + * identity map installs (the PDE window is walked: FAULT_PTE, not + * FAULT_PDE) yet the engine still cannot fetch at the client's own VA. + * This reconfirms the r929 v2 lesson: guest VAs are not engine-fetchable on + * this replay channel, so the "validate in guest VA space" hypothesis is + * not testable by operand rewriting (the fetch dies before any DS check). + * Default 0 (baseline shadow rewrite); 1 reproduces the FAULT_PTE experiment. + */ +static bool gsp_phys_rmcall_gr_prog_guest_va; +module_param_named(sema_gl_prog_guest_va, gsp_phys_rmcall_gr_prog_guest_va, + bool, 0644); +MODULE_PARM_DESC(sema_gl_prog_guest_va, + "Leave program-family (0x2014-family) operands at their original guest VAs when identity-mappable (default 0; 1 = the r1307 FAULT_PTE experiment - guest VAs are not engine-fetchable, do not keep)"); + +/* + * SESSION 153-CONTINUED-84: SET_MME_SHADOW_RAM_CONTROL mode before the + * window. The vendor comment (mesh shader bind path) is explicit: + * "MME filtering swallows them all if old and new state values are the + * same" - METHOD_TRACK_WITH_FILTER (1) suppresses every method write whose + * value equals the shadowed one. Our replay repeats the same window every + * launch, so after the first replay every redundant write - including the + * CALL_MME_MACRO/CALL_MME_DATA parameter writes - matches the shadow and + * is swallowed; the macro call never reaches the MME (r1236-r1250: calls + * in the emitted window, all MME registers idle, some launches park the + * front-end). The mode-bearing writes live in the captured PRELUDE + * (0x124 x4 in the client's setup push), which the win_patch window sweep + * does not touch; emit METHOD_TRACK (0 = filtering OFF, semantics + * unchanged: the write simply applies) right before the SA re-arm and the + * window. Value = sema_gl_mme_track (0 = TRACK; 1/2/3 = other modes; -1 = + * do not emit). + */ +static int gsp_phys_rmcall_gr_mme_track; +module_param_named(sema_gl_mme_track, gsp_phys_rmcall_gr_mme_track, + int, 0644); +MODULE_PARM_DESC(sema_gl_mme_track, + "SET_MME_SHADOW_RAM_CONTROL mode emitted before each replayed window (0=TRACK filtering off (default); 1=TRACK_WITH_FILTER; 2=PASSTHROUGH; 3=REPLAY; -1=off)"); + +/* + * SESSION 153-CONTINUED-85: A/B knob for the SA re-arm (the r1237 fix). + * Before CONT-84 the calls in the replayed window resolved to the SA state + * the prelude left behind (its last captured SA write re-clobbers the slot; + * r1231: SA[34]=0 lands last, every draw call resolved to macro 0 and + * emitted nothing - and the r1228-phase launches retired). With the re-arm + * the calls resolve to the captured IR entries; every call-bearing launch + * since (r1245/r1250/r1265/r1270) parks or poisons the front-end. This knob + * restores the pre-arm resolution in one boot so the two states can be A/B'd + * without a rebuild (0 = do not re-point; default 1 = re-arm). + */ +static bool gsp_phys_rmcall_gr_mme_sa_rearm = true; +module_param_named(sema_gl_mme_sa_rearm, gsp_phys_rmcall_gr_mme_sa_rearm, + bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_sa_rearm, + "Re-emit LOAD_MME_START_ADDRESS_RAM pairs before each replayed window so CALL_MME_MACRO resolves to the captured SA entries (default 1; 0 = pre-arm resolution, A/B)"); + +/* + * The same stubbed-slot predicate the launch builder applies (see the + * sa_stub emit decision): a slot is stubbed when stubbing is on and either + * the all-slots knob is set, the capture holds no SA entry, or the IR + * instruction the entry names never entered the replay. A kept slot + * (sema_gl_mme_keep_mask) is never stubbed - unless the capture cannot + * serve it at all (keep_clamp, see the CONT-113 param comment). + */ +static bool gsp_phys_gr_slot_kept(u32 j, u64 sa_mask, const u32 *sa_ent) +{ + if (!(READ_ONCE(gsp_phys_rmcall_gr_keep_mask) & (1ull << j))) + return false; + if (READ_ONCE(gsp_phys_rmcall_gr_nop_drop) & (1ull << j)) + return false; + /* + * r1448 STRATEGY 3.3: golden-only means the kept-slot exemption + * cannot resurrect a body outside the golden corpus (fail-closed). + */ + if (gsp_phys_gr_slot_golden_blank(j)) + return false; + if (!READ_ONCE(gsp_phys_rmcall_gr_keep_clamp)) + return true; + if ((sa_mask & (1ull << j)) && gsp_sema_mme_ir_covered(sa_ent[j])) + return true; + /* + * SESSION 153-CONTINUED-114: a keep_mask slot the golden library + * serves is kept without capture coverage - the pack overlay + * carries its code (see the include comment). + */ + { + u32 gent, gn; + const u32 *gw; + + if (gsp_phys_gr_golden_slot(j, &gent, &gw, &gn)) + return true; + } + if (gsp_phys_gr_keep_clamp_log < 16u) { + gsp_phys_gr_keep_clamp_log++; + pr_warn("vgsp-gr: MME-KEEP-CLAMP slot=%u (keep_mask set but capture has no SA entry/coverage; stubbed + calls dropped, never run) entry=0x%x covered=%u\n", + j, sa_ent[j], + (unsigned)gsp_sema_mme_ir_covered(sa_ent[j])); + } + return false; +} + +static bool gsp_phys_gr_slot_stubbed(u32 j, u64 sa_mask, const u32 *sa_ent) +{ + if (j >= 64u || !gsp_sema_gr_sa_stub_value()) + return false; + /* + * SESSION 153-CONTINUED-80: a nop-drop slot is force-stubbed even + * when it is also kept or covered - the r1186 evidence shows a kept + * slot 57 loses its SA end-noop stub (the stub builder skips kept + * slots) and any call-run encoding the drop filter misses then + * executes the real macro 57 (the park class). Dropping the runs and + * stubbing SA[57] together is the safe combination. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_nop_drop) & (1ull << j)) + return true; + /* + * r1448 STRATEGY 3.3: no golden body, no execution - force-stub the + * slot so its captured body can never run (its CALL runs are then + * dropped and its SA pair points at the end-noop). + */ + if (gsp_phys_gr_slot_golden_blank(j)) + return true; + if (gsp_phys_gr_slot_kept(j, sa_mask, sa_ent)) + return false; + /* + * SESSION 153-CONTINUED-114: a slot the golden library serves is + * never stubbed (its CALL runs stay in the window and the pack + * carries its code; see the include comment). + */ + { + u32 gent, gn; + const u32 *gw; + + if (gsp_phys_gr_golden_slot(j, &gent, &gw, &gn)) + return false; + } + if (READ_ONCE(gsp_phys_rmcall_gr_sa_stub_all)) + return true; + if (!(sa_mask & (1ull << j))) + return true; + return !gsp_sema_mme_ir_covered(sa_ent[j]); +} + +/* + * In-place filter for the relocated 3D window: remove every run that + * writes the CALL_MME_MACRO(j)/CALL_MME_DATA(j) method pair of a stubbed + * slot. Pure run-level removal (runs are self-contained: op 1/3/5 carry + * 1+cnt data words; 0/2/4/6 are single words), so the surviving method + * sequence is unchanged and both the F2A window walk and the engine see + * the same stream shape with the calls elided. + */ +static u32 gsp_phys_gr_mme_drop_calls(u32 *win, u32 n, u64 sa_mask, + const u32 *sa_ent, u64 *dropped_mask) +{ + u32 r = 0, w = 0, dropped = 0, transform = 0; + u64 dmask = 0; + + while (r < n) { + u32 h = win[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u, k, m; + bool iscall = false; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + r + 1u + cnt <= n) + span = 1u + cnt; + /* + * Decode the touched methods the way the engine walks them: + * multi-word runs land on m + 4k; single-word entries carry + * their method in the low bits (IMMD and the SDM family both + * put it at 11:0 for our purposes). + */ + for (k = 0; k < span; k++) { + m = ((h & 0xfffu) << 2) + 4u * k; + if (k && op != 1u && op != 3u) + break; + if (m >= 0x3800u && m <= 0x39fcu) { + u32 j = (m - 0x3800u) / 8u; + + if ((READ_ONCE(gsp_phys_rmcall_gr_keep_mask) & + (1ull << j)) && + (READ_ONCE(gsp_phys_rmcall_gr_nop_param_mask) & + (1ull << j))) { + if (READ_ONCE(gsp_phys_rmcall_gr_nop_drop)) { + /* + * SESSION 153-CONTINUED-80: the + * retarget parks the PBDMA + * live (see the + * sema_gl_mme_nop_drop + * comment) - drop the whole + * run instead. + */ + iscall = true; + dmask |= 1ull << j; + break; + } + /* + * SESSION 153-CONTINUED-79: kept + + * nop-param slot - keep the run but + * retarget its header to the DATA + * port so the params still land + * (see the sema_gl_mme_nop_param_mask + * comment). + */ + u32 nm = 0x3804u + 8u * j; + + win[r] = (h & ~0xfffu) | + ((nm >> 2) & 0xfffu); + dmask |= 1ull << j; + transform++; + } else if (gsp_phys_gr_slot_stubbed(j, sa_mask, + sa_ent)) { + iscall = true; + dmask |= 1ull << j; + break; + } + } + } + if (iscall) { + dropped += span; + r += span; + continue; + } + if (w != r) + memmove(&win[w], &win[r], (size_t)span * 4u); + w += span; + r += span; + } + if (transform) + pr_info("vgsp-gr: MME-CALL-NOP %u call run(s) retargeted to the CALL_MME_DATA port (params conserved)\n", + transform); + if (dropped_mask) + *dropped_mask = dmask; + return dropped; +} + +/* + * SESSION 153-CONTINUED-61: generation of the MME prelude already emitted + * into the engine context (0 = none). The r1110 evidence: the prelude was + * emitted once on the first beef launch (gen=1, 512 words), that launch + * priv-errored (grfifo=0xbadf1201) and WEDGE-RECOVER ran TSG preempt + + * resched - which resets the engine context and with it the resident MME + * program - yet the latch kept suppressing re-emission, so every later + * window replay ran its macros against an unloaded MME (mme_pc=513 class + * stall, no retire). The recovery clears the latch so the next launch + * re-preloads the context. + * + * SESSION 153-CONTINUED-68: the latch is keyed by (generation, channel). + * A bare generation latch is SHARED across channels: the X channel's + * first post-maturity launch consumed the emission for the whole system + * and every beef launch then shipped without the program (r1140: only + * h=0x1000013 logged "MME-PRELUDE emitted gen=3", every beef launch + * words=400 prelude-less). Re-emitting per channel is idempotent. + */ +static u32 gsp_phys_gr_mme_prelude_gen; +static u32 gsp_phys_gr_mme_prelude_h; +static u32 gsp_phys_gr_mme_pre_full_dump; + +/* + * CONT-118: one-launch-delay release for the real CALL runs. The known + * poison class is the first calls-kept launch after a fresh split pack + * completes (the r1359 half-loaded-IR race): the drain-completing carrier + * holds itself (the r1222 clause), but the very next window would run the + * real calls against a program whose carrier only just executed. With this + * knob that first post-latch launch is held too - it arms the release - so + * the real calls resume one full launch AFTER the completing carrier in + * every case (the r1443b first-generation shape, reproduced for recreated + * generations too). Default 1; A/B with 0 to release on the first + * post-latch launch (legacy behavior). + */ +static u32 gsp_phys_gr_mme_release_gen; +static u32 gsp_phys_gr_mme_release_h; +static bool gsp_phys_rmcall_gr_mme_hold_settle = true; +module_param_named(sema_gl_mme_hold_settle, + gsp_phys_rmcall_gr_mme_hold_settle, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_settle, + "CONT-118: hold real CALL runs for one extra launch after a generation's split prelude latches (default 1; 0 = release on the first post-latch launch)"); + +/* + * r1445 3b: strict variant of the CONT-118 settled release. With this on, + * the armed release additionally requires a launch that carried NO prelude + * words, an idle front-end (0x400504 == 1) and the carrier's FETCH-WIT + * promotion (live_gen == pl_gen), so the real calls resume only after the + * completing carrier has provably executed against a settled MME. The + * rg/rh gen-h match machinery is unchanged; default 1, A/B with 0. + */ +static bool gsp_phys_rmcall_gr_mme_hold_strict = true; +module_param_named(sema_gl_mme_hold_strict, + gsp_phys_rmcall_gr_mme_hold_strict, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_strict, + "CONT-118 strict: release the held CALL runs only on a no-emission launch with the front-end idle and the carrier's generation proven live (default 1; 0 = the plain settled release)"); + +/* + * r1447: the first post-latch ("arming") launch has poisoned in every boot + * that reached it (r1444 103322, r1446 #4, r1446c #4) while window+calls- + * kept launches ran 22x clean (r1443b). The arming launch carries the + * window with its CALL runs elided - the one shape that never retired. Hold + * the WINDOW on the arming launch too (the calls are already held by the + * settle clause); the first window execution then is the proven + * calls+window-kept shape, one release later. The hold is keyed by + * generation+channel and persists until the release actually fires, so a + * strict-starved release cannot ride the window against elided CALL runs + * either; it is cleared at every residency reset (gate fail, wedge + * recovery, park recovery) and when the release fires. Bounded: the set + * happens only while the release is not yet armed, so at most one arming + * window-drop per generation. Default 1; 0 = legacy arming launch carries + * the calls-held window. + */ +static u32 gsp_phys_gr_mme_window_hold_gen; +static u32 gsp_phys_gr_mme_window_hold_h; +static bool gsp_phys_rmcall_gr_mme_hold_window = true; +module_param_named(sema_gl_mme_hold_window, + gsp_phys_rmcall_gr_mme_hold_window, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_hold_window, + "r1447: drop the 3D window on the one arming launch per generation until the held CALL runs release (default 1; 0 = legacy arming launch carries the calls-held window)"); + +/* + * r1447: bounded fail-open for the strict release. The strict clause + * requires gsp_phys_gr_fe_idle() == 0x00000001 exactly, a value the GR + * FIFO status does not produce under client traffic (observed settled: + * 0x0e0e0001-class), so with strict=1 the armed release can starve the + * calls forever. After this many strict-held launches in one generation + * the FE term fails open (the release proceeds) and logs why. + * Default 8; 0 = legacy never-fail-open (the strict release stays strictly + * gated on 0x400504 == 1 and can starve). + */ +static u32 gsp_phys_gr_mme_strict_holds; +static u32 gsp_phys_rmcall_gr_mme_strict_grace = 8u; +module_param_named(sema_gl_mme_strict_grace, + gsp_phys_rmcall_gr_mme_strict_grace, uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_strict_grace, + "r1447: fail the strict release's FE-idle term open after this many strict-held launches per generation (default 8; 0 = legacy never-fail-open)"); + +/* + * SESSION 153-CONTINUED-107: readback-failure counter + give-up bound for + * the load-only starvation (see the mme_load_only comment). 0 = unlimited; + * default 24. + */ +static u32 gsp_phys_gr_mme_iram_gate_fails; +static u32 gsp_phys_rmcall_gr_mme_iram_gate_max = 24u; + +/* + * SESSION 153-CONTINUED-107: gate decision shared by both IRAM-gate call + * sites (see gsp_phys_gr_mme_iram_gate). The readback failed, so hold the + * calls and owe a full re-emission. Clearing the prelude latch makes the + * next launch emit again (the builder's emit condition); rewinding the pack + * and drain cursors makes that re-emission restart from the first word. The + * pending-gen latch is cleared so the poll break cannot promote the + * generation to "resident". + */ +static void gsp_phys_gr_mme_iram_gate_fail(void) +{ + gsp_phys_gr_mme_iram_gate_fails++; + WRITE_ONCE(gsp_phys_gr_mme_pending_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_sa, false); + /* + * CONT-118: residency is unproven again, so a settled release armed + * for this generation is void; the re-emission re-earns it. + */ + WRITE_ONCE(gsp_phys_gr_mme_release_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_release_h, 0); + /* + * r1447: same for the arming window-hold and the strict grace budget. + */ + WRITE_ONCE(gsp_phys_gr_mme_window_hold_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_window_hold_h, 0); + WRITE_ONCE(gsp_phys_gr_mme_strict_holds, 0); +} + +module_param_named(sema_gl_mme_iram_gate_max, + gsp_phys_rmcall_gr_mme_iram_gate_max, uint, 0644); +MODULE_PARM_DESC(sema_gl_mme_iram_gate_max, + "Give up the IRAM-gate load-only starvation after this many readback failures (0 = never; default 24)"); + +static bool gsp_phys_gr_mme_iram_gate_giveup(void) +{ + u32 m = READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate_max); + + return m && READ_ONCE(gsp_phys_gr_mme_iram_gate_fails) >= m; +} + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude. The r1091 + * full-MME-prelude boot cleared MME (mme_esr=80000000) but the first beef + * window then latched a draw-shader program-header error + * (NV_PGRAPH_PRI_DS_HWW_ESR 0x405840 = 0xa2040a00: SPH9|SPH11|SPH18, + * M_PIPE_STAGE = VSB|PS) and a class error at SET_VERTEX_ATTRIBUTE_A(2) + * (0x1168, INVALID_ENUM): the replayed window is a mid-stream state cut - + * the attribute/pipeline record the client set in EARLIER windows (the + * missing stages' SET_PIPELINE_PROGRAM_ADDRESS binds and the attribute + * format state) was never replayed, so the DS validated program headers + * that were never bound. With this knob the launch builder preplays the + * captured runs from the service's attr prelude buffer (see + * svc_attr_prelude_capture), relocated exactly like the window itself + * (the patched copy is computed by the caller - it needs the launch page + * set and the live shadow counter), run-aligned, drained across the first + * launches of a capture generation. Declared before build_gr_launch (its + * reader); the matching capture-side knob is sema_attr_prelude_cap. + * Default 1; A/B with 0. + */ +static bool gsp_phys_rmcall_gr_attr_prelude = true; +module_param_named(sema_gl_attr_prelude, gsp_phys_rmcall_gr_attr_prelude, + bool, 0644); +MODULE_PARM_DESC(sema_gl_attr_prelude, + "Preplay the captured client attr/pipeline runs (0x1100..0x11fc, 0x2000..0x205c, relocation applied) ahead of the first 3D window replays of a capture generation (default 1)"); + +/* + * CONT-120 (r1451): drop the attr-prelude's SET_PIPELINE_SHADER(2) runs + * (run header method field encoding method 0x2080) from the attr copy. + * The r1450c wall #8 proved the writer of `Offset 0x2080 Data 0x20 + * ErrorCode 4` is the guest's own captured one-shot attr emission + * (`20010820 00000020` = SET_PIPELINE_SHADER(2) <- 0x20, a cold disable + * word) replayed on a windowless LOAD-ONLY carrier: with no SPH-BIND + * stage-2 enable ahead of it the FE refuses the write contextually. The + * same word rides window-bearing launches clean because our SPH-BIND + * block (0x2080 <- 0x21) is prepended. This knob drops only those runs + * (ops 1/3/5, cnt != 0, method == 0x2080), cursor advancing exactly like + * the SDM drop (no raw fallback, dropped for the generation). Default 1; + * A/B with 0 = legacy byte-for-byte. + */ +static bool gsp_phys_rmcall_gr_attr_sph_drop = true; +module_param_named(sema_gl_attr_sph_drop, gsp_phys_rmcall_gr_attr_sph_drop, + bool, 0644); +MODULE_PARM_DESC(sema_gl_attr_sph_drop, + "Drop the attr-prelude SET_PIPELINE_SHADER(2) runs (method 0x2080, the guest's cold 0x20 disable word) the FE refuses with EC4 on a windowless LOAD-ONLY carrier (default 1; 0 = legacy)"); +static u32 gsp_phys_gr_attr_sph_drop_count; /* runs dropped (all gens) */ +static u32 gsp_phys_gr_attr_sph_drop_gen; /* gen of the last drop log */ + +/* + * The patched (relocation-applied) attr prelude for the current capture + * generation, built by the submit path just before build_gr_launch. Single + * caller (the ordered single-inflight sema work item), so plain statics. + */ +#define GSP_PHYS_GR_ATTR_PRE_WORDS 512u +static u32 gsp_phys_gr_attr_pre[GSP_PHYS_GR_ATTR_PRE_WORDS]; +static u32 gsp_phys_gr_attr_pre_n; +static u32 gsp_phys_gr_attr_pre_gen; /* capture gen the patch is from */ +static u32 gsp_phys_gr_attr_emit_gen; /* gen whose runs are draining */ +static u32 gsp_phys_gr_attr_emit_off; /* word cursor into attr_pre */ + +/* + * SESSION 153-CONTINUED-62: surface (render-target) prelude. The beef + * client's per-frame draw windows are mid-stream cuts: the colour-target + * bind is sticky context state (the client and the X server write it once + * per context), so the replayed windows carry none - and every replayed + * draw rasterized into an unbound target and stored nothing (the observed + * 0/N retire). The service captures the last C597 colour-target run from + * any trusted push (svc_surf_prelude_capture; the observed carrier is the + * X server's 1280x720 screen bind) and this site replays it - aliased and + * relocated exactly like the window - ahead of any window that carries no + * usable target bind of its own. Emitted per capture generation (the + * patched buffer is reused while the captured run is unchanged, and the + * alias VA is stable per boot); a fresh generation re-patches. A/B via + * the sema_gl_surf_prelude knob (default 1). + */ +#define GSP_PHYS_GR_SURF_PRE_WORDS 32u +static u32 gsp_phys_gr_surf_pre[GSP_PHYS_GR_SURF_PRE_WORDS]; /* patched */ +static u32 gsp_phys_gr_surf_pre_in[GSP_PHYS_GR_SURF_PRE_WORDS]; +static u32 gsp_phys_gr_surf_pre_n; /* patched words, 0 = none */ +static u32 gsp_phys_gr_surf_pre_gen; /* capture gen the patch is from */ +static bool gsp_phys_rmcall_gr_surf_prelude = true; +module_param_named(sema_gl_surf_prelude, gsp_phys_rmcall_gr_surf_prelude, + bool, 0644); +MODULE_PARM_DESC(sema_gl_surf_prelude, + "Replay the captured C597 colour-target bind ahead of 3D windows that carry no render target of their own (default 1)"); + +/* + * SESSION 153-CONTINUED-62: does this window already carry a usable + * colour-target bind? A run with va == 0 (the all-zero degenerate bind + * seen on the client's 0x402094c push) does not count - the surface + * prelude is injected for it too. + */ +static bool gsp_phys_gr_window_has_surface(const u32 *win, u32 n){ + u32 off = 0; + + while (off < n) { + u32 hdr = win[off]; + u32 op = (hdr >> 29) & 7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 span = 1u; + + if ((op == 1u || op == 3u || op == 5u) && cnt && + off + 1u + cnt <= n) + span = 1u + cnt; + else { + off++; + continue; + } + if (op == 1u && addr >= 0x800u && addr < 0xc00u && + (addr & 0x3fu) == 0u && cnt >= 2u) { + u64 va = ((u64)(win[off + 1] & 0xffu) << 32) | + (u64)(win[off + 2] & 0xfffff000u); + + if (va) + return true; + } + off += span; + } + return false; +} + + +/* * GR launch method block: the inline QMD run at its original shape and the * channel REPORT_SEMAPHORE release. The stock guest push does NOT emit * SET_OBJECT before the inline run (the compute class is bound on the * channel long before the launch), so the replay must not either. * Returns the block size in dwords. */ +/* SESSION 153-CONTINUED-101: forward declaration - the seed emission in + * gsp_phys_rmcall_build_gr_launch (below) translates the seed page through + * the same relocator the window operand pairs use (defined further down). */ +static bool gsp_phys_rmcall_gr_3d_reloc(u32 *up_w, u32 *lo_w, + const struct gsp_phys_gr_submit *req, + u32 *n_shadow); + +/* + * SESSION 153-CONTINUED-104: FE idle witness for the calls-kept gate. + * The r1358/r1359 A/B (identical config; only the fixed-CB seed value + * differed) exposes the r1222-class race directly: + * - r1358: the calls-kept launch was submitted with the FE settled + * (MME-FAST begin grfifo=0x00000001) and the whole real-call window + * executed - the macro emissions landed (the wall became the CB read + * FAULT_PTE at fixed-CB+0x4000, i.e. the raw constant path now works); + * - r1359: the same launch was submitted while the FE still showed + * grfifo=0x0d0d1602 (the previous prelude-carrying launch's SA/IR + * method runs still draining) and the first macro execution poisoned + * with a REAL MME ILLEGAL_OPCODE (ESR 0x404490=0x80000004). + * The MME consumes the prelude's SA/IR port runs asynchronously, so a + * CALL_MME_MACRO decoded while those writes are in flight fetches + * half-loaded IR. Hold the real CALL runs until 0x400504 reads the idle + * value (0x00000001): the held launches still retire (the proven-safe + * drop shape) and a later window retries; once idle the calls run + * against a settled program. Fail-open when BAR0 is unavailable. + */ +static bool gsp_phys_gr_fe_idle(void) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + + return !bar0 || ioread32(bar0 + 0x00400504u) == 0x00000001u; +} + +/* + * r1447: raw GR_FIFO_STATUS word for the strict-grace log (0 when BAR0 is + * unavailable). Read-only diagnostic; the release decision still uses + * gsp_phys_gr_fe_idle(). + */ +static u32 gsp_phys_gr_fe_status(void) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + + return bar0 ? ioread32(bar0 + 0x00400504u) : 0u; +} + +/* + * r1445 5c: submit-side FE-busy deferral of the load-only carrier. The + * r1444 first-poison class submitted its first load-only carrier while the + * front-end was mid-fetch and the packed-program fetch poisoned the lane. + * This predicate is evaluated at the service's request gate BEFORE any + * emission/submission: when the launch would become a load-only carrier + * (the sema_gl_mme_iram_gate decision in build_gr_launch) and the FE is NOT + * idle, the request is busy-deferred (candidate survives, pump retries; + * nothing is emitted and no release is authored) and the carrier goes out + * once the FE has settled. Bounded: at most GSP_PHYS_GR_MME_FE_DEFER_MAX + * consecutive defers per prelude generation - the next attempt submits the + * carrier as before, so a never-idle FE cannot starve the lane. The caller + * passes a 3D stream launch (h_object / win3d_n); knob 0 = no-op. + */ +#define GSP_PHYS_GR_MME_FE_DEFER_MAX 4u +static bool gsp_phys_rmcall_gr_mme_fe_defer = true; +module_param_named(sema_gl_mme_fe_defer, + gsp_phys_rmcall_gr_mme_fe_defer, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_fe_defer, + "Busy-defer a 3D launch that would be a load-only MME carrier while the GR front-end is not idle, so the packed program is not loaded mid-fetch (default 1; 0 = submit the carrier as before)"); +static u32 gsp_phys_gr_mme_fe_defer_gen; +static u32 gsp_phys_gr_mme_fe_defer_n; +static u32 gsp_phys_gr_mme_fe_defer_giveup_gen; + +bool gsp_phys_gr_mme_fe_defer_wanted(u32 h_object, u32 win3d_n) +{ + u32 pl_n = 0, pl_gen = 0; + bool latch_now, live_now; + + if (!READ_ONCE(gsp_phys_rmcall_gr_mme_fe_defer) || + !gsp_phys_rmcall_gr_3d) + return false; + if (!win3d_n || win3d_n + 13u > GSP_PHYS_RMCALL_GR_METHOD_WORDS) + return false; + if (!READ_ONCE(gsp_phys_rmcall_gr_mme_prelude) || + !READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate) || + !READ_ONCE(gsp_phys_rmcall_gr_mme_prelude_pack) || + !READ_ONCE(gsp_phys_rmcall_gr_mme_hold_calls)) + return false; + if (gsp_phys_gr_mme_iram_gate_giveup()) + return false; + if (!gsp_sema_mme_prelude_get(&pl_n, &pl_gen) || !pl_n || !pl_gen) + return false; + if (gsp_phys_gr_mme_fe_defer_gen != pl_gen) { + gsp_phys_gr_mme_fe_defer_gen = pl_gen; + gsp_phys_gr_mme_fe_defer_n = 0; + } + /* + * Same (post-emission-equivalent) load-only predicate as + * build_gr_launch: a launch is a load-only carrier until its + * generation is both latched and proven live. + */ + latch_now = (pl_gen == READ_ONCE(gsp_phys_gr_mme_prelude_gen) && + h_object == READ_ONCE(gsp_phys_gr_mme_prelude_h)); + live_now = (READ_ONCE(gsp_phys_rmcall_gr_mme_live_skip) && + READ_ONCE(gsp_phys_gr_mme_live_gen) == pl_gen); + if (latch_now && live_now) { + gsp_phys_gr_mme_fe_defer_n = 0; + return false; /* calls-bearing launch, no carrier */ + } + if (gsp_phys_gr_fe_idle()) { + gsp_phys_gr_mme_fe_defer_n = 0; + return false; /* FE settled: the carrier may go out now */ + } + if (gsp_phys_gr_mme_fe_defer_n >= GSP_PHYS_GR_MME_FE_DEFER_MAX) { + if (gsp_phys_gr_mme_fe_defer_giveup_gen != pl_gen) { + gsp_phys_gr_mme_fe_defer_giveup_gen = pl_gen; + pr_warn("vgsp-gr: MME-FE-DEFER-GIVEUP gen=%u h=0x%x n=%u (front-end still busy; loading anyway)\n", + pl_gen, h_object, gsp_phys_gr_mme_fe_defer_n); + } + return false; /* bounded: historical carrier stands */ + } + gsp_phys_gr_mme_fe_defer_n++; + pr_info_ratelimited("vgsp-gr: MME-FE-DEFER gen=%u h=%#x n=%u\n", + pl_gen, h_object, gsp_phys_gr_mme_fe_defer_n); + return true; +} + +/* + * r1487 STREAM-DAMAGE: a client 3D section is sliced into method-aligned + * chunks replayed as separate launches (svc_gl_stream_chunks). The channel + * state accumulates across those launches, so a chunk that is *dropped* + * (MME-LOAD-ONLY carrier while the prelude is not resident, or the r1447 + * arming window-hold) permanently removes that slice of the method stream. + * A later chunk of the same push then executes CALL_MME_MACRO runs against a + * channel missing the dropped state: r1485's 81-word tail called + * LOAD_CONSTANT_BUFFER_BUG2548370 after chunks 1..3 had been dropped, the + * macro emitted method 0x2390 (LOAD_CONSTANT_BUFFER(0)) without a valid + * constant-buffer selector, the 3D class raised + * "Offset 00002390, Data bf800000, ErrorCode 00000003", and the FECS + * priv-error sentinel 0xbadf1201 wedged the lane (pc=513/mac=0xf1 are + * artifacts of that sentinel: 0xbadf1201 & 0xfff == 0x201). + * + * Fail closed per stream: remember the lowest dropped stream-relative offset + * for a push; while a later chunk of that push arrives, force the calls-held + * emission (CALL runs elided, the proven-safe shape) so no macro can run + * against the incomplete state. A chunk at stream_rel 0 is the start of a + * fresh client section and clears the record, so the next frame replays its + * whole window in order and draws normally. + */ +#define GSP_PHYS_GR_STREAM_DAMAGE_SLOTS 8 +struct gsp_phys_gr_stream_damage { + u64 push_va; + u32 rel; + bool valid; +}; +static struct gsp_phys_gr_stream_damage + gsp_phys_gr_stream_damage[GSP_PHYS_GR_STREAM_DAMAGE_SLOTS]; +static u32 gsp_phys_gr_stream_damage_next; +static u32 gsp_phys_gr_stream_damage_hits; +static u32 gsp_phys_gr_stream_damage_notes; + +static bool gsp_phys_gr_stream_damaged(u64 push_va, u32 rel) +{ + u32 i; + + for (i = 0; i < GSP_PHYS_GR_STREAM_DAMAGE_SLOTS; i++) { + struct gsp_phys_gr_stream_damage *d = + &gsp_phys_gr_stream_damage[i]; + + if (d->valid && d->push_va == push_va) + return rel > d->rel; + } + return false; +} + +static void gsp_phys_gr_stream_damage_note(u64 push_va, u32 rel) +{ + u32 i; + + for (i = 0; i < GSP_PHYS_GR_STREAM_DAMAGE_SLOTS; i++) { + struct gsp_phys_gr_stream_damage *d = + &gsp_phys_gr_stream_damage[i]; + + if (d->valid && d->push_va == push_va) { + if (rel < d->rel) + d->rel = rel; + return; + } + } + i = gsp_phys_gr_stream_damage_next++ % + GSP_PHYS_GR_STREAM_DAMAGE_SLOTS; + gsp_phys_gr_stream_damage[i].push_va = push_va; + gsp_phys_gr_stream_damage[i].rel = rel; + gsp_phys_gr_stream_damage[i].valid = true; +} + +static void gsp_phys_gr_stream_damage_clear(u64 push_va) +{ + u32 i; + + for (i = 0; i < GSP_PHYS_GR_STREAM_DAMAGE_SLOTS; i++) { + struct gsp_phys_gr_stream_damage *d = + &gsp_phys_gr_stream_damage[i]; + + if (d->valid && d->push_va == push_va) + d->valid = false; + } +} + static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, const struct gsp_phys_gr_submit *req, const u32 *qmd, u64 qmd_va, - u64 sem_va, u32 sem_payload) + u64 sem_va, u32 sem_payload, + const u32 *win3d, u32 win3d_n, + const u32 *attr_pre, + u32 attr_pre_n) { u32 *p = pb; int prelude = gsp_phys_rmcall_gr_prelude_value(); @@ -16644,6 +20886,2332 @@ static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, int two_step = gsp_phys_rmcall_gr_pcas_two_step_value(); /* + * r929 (Path A): a captured 3D method window replays verbatim. Bind + * TURING_A (0xC597) on the channel subchannel (the same bind the + * window itself carries - belt and braces, a rebind is harmless), + * then the window words (address operands relocated to shadow-pool + * VAs by gsp_phys_rmcall_gr_3d_patch_window), then the channel + * release tail. No compute init, no QMD, no PCAS: the engine walks + * real 3D methods (SET_OBJECT C597, state binds, LINE_LENGTH_IN + + * LAUNCH_DMA inline uploads, program/tex binds). Completion is the + * trailing REPORT_SEMAPHORE alone (the submit path leaves qmd_rel + * off for this mode). + */ + if (gsp_phys_rmcall_gr_3d && req && win3d && win3d_n && + win3d_n + 2u + 3u + 8u <= GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + /* + * SESSION 153-CONTINUED-53: MME context-setup prelude (see the + * sema_gl_mme_prelude knob above). Preplay the captured client + * setup runs once per capture generation, right after the + * SET_OBJECT C597 bind and ahead of the window, so + * CALL_MME_MACRO() runs execute against a loaded macro + * program instead of pc=0 -> ILLEGAL_OPCODE. The read is + * seqchecked (generation before/after): a concurrent append + * only defers emission to the next launch. A window too large + * to also carry the prelude replays without it (logged) - the + * emission then happens on the next launch that fits. + */ + bool emit_prelude = false; + bool pre_done = false; + bool mme_hold_calls = false; + bool mme_hold_window = false; + bool mme_hold_fe_busy = false; + bool mme_load_only = false; + bool latch_before = false; + bool stream_damaged = false; + u32 pre_words_out = 0; + u32 win_emit_off = 0; + const u32 *pl = NULL; + u32 pl_n = 0, pl_gen = 0, pl_room = 0; + u32 block_cap = READ_ONCE(gsp_phys_rmcall_gr_block_cap); + + if (block_cap < 64u || + block_cap > GSP_PHYS_RMCALL_GR_METHOD_WORDS) + block_cap = GSP_PHYS_RMCALL_GR_METHOD_WORDS; + /* + * SESSION 153-CONTINUED-86: harden the block cap against the + * emissions the block-cap arithmetic below does not fully + * count (the SA re-arm, 3 words per slot, up to 32 slots - + * see the MME-SA-REARM site). Clamping the cap keeps every + * combination inside the 4096-word method buffer; with the + * default 2048 cap this changes nothing. + */ + if (block_cap > GSP_PHYS_RMCALL_GR_METHOD_WORDS - 128u) + block_cap = GSP_PHYS_RMCALL_GR_METHOD_WORDS - 128u; + /* + * r1487 STREAM-DAMAGE: stream_rel 0 starts a fresh client + * section, so any dropped-chunk record for this push is + * stale; a later chunk keeps the record and is emitted + * calls-held below (see the table comment). + */ + if (req->stream_rel == 0u) + gsp_phys_gr_stream_damage_clear(req->push_va); + stream_damaged = gsp_phys_gr_stream_damaged(req->push_va, + req->stream_rel); + /* + * SESSION 153-CONTINUED-63: MME START-ADDRESS RAM completion. + * The prelude capture is capped at 512 words, so the client's + * SA table (macro slot -> IR address) may be incomplete; on a + * fresh context the missing slots read 0 and CALL_MME_MACRO(j) + * jumps to IR 0 - the slot-0 macro - which with the caller's + * args emitted LOAD_CONSTANT_BUFFER_OFFSET <- 0x20020 and + * poisoned the lane (r1116; verified offline with the MME64 + * simulator). Stubbing every CALLED slot the capture's SA + * table does not cover with a harmless end-noop makes the + * unknown call a no-op instead of a lane-killer. The called + * slots come from the window being emitted (IMMD writes to + * 0x3800+j*8 and INC/ONEINC runs writing any 0x3800..0x39f8 + * method, the .Macro half of each pair). + */ + u32 sa_ent[64]; + u64 sa_mask = 0; + u32 sa_stub[40]; + u32 sa_stub_ent[40]; + u32 sa_cov[40]; + u32 sa_cov_ent[40]; + u32 sa_nstub = 0, sa_ncov = 0, sa_block = 0; + + if (gsp_sema_gr_sa_stub_value() && win3d && win3d_n) { + u32 so = 0; + + sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + while (so < win3d_n && (sa_nstub < 40u || sa_ncov < 40u)) { + u32 h = win3d[so]; + u32 sop = (h >> 29) & 7u; + u32 scnt = (h >> 16) & 0x1fffu; + u32 sspan = 1u; + u32 k, slo, shi; + + if ((sop == 1u || sop == 3u || sop == 5u) && + scnt && so + 1u + scnt <= win3d_n) { + sspan = 1u + scnt; + slo = 0; + /* + * SESSION 153-CONTINUED-68: ONE_INC + * (op 5) writes only data[0] to the + * header method; the rest go to + * method+4 (the data port, see + * nv3dLoadConstantsHeader). Walking + * mthd+4k for op=5 invented ghost + * CALL_MME_MACRO slots (r1134/1135 + * list). + */ + shi = (sop == 5u) ? 0u : scnt - 1u; + } else if (sop == 4u) { + slo = 0; + shi = 0; + } else { + so++; + continue; + } + for (k = slo; k <= shi; k++) { + u32 mthd = ((h & 0xfffu) << 2) + 4u * k; + u32 j, t; + bool have = false; + + if (mthd < 0x3800u || mthd >= 0x3a00u || + (mthd & 7u)) + continue; + j = (mthd - 0x3800u) / 8u; + if (j >= 64u) + continue; + /* + * SESSION 153-CONTINUED-70: a kept slot + * (sema_gl_mme_keep_mask) is never + * stubbed - its real macro must run + * against the context exactly as the + * client's own SA/IR uploads left it. + * Logged in the covered list for the + * census; no stub pair is emitted. + * SESSION 153-CONTINUED-113: unless + * the capture cannot serve it at all + * (keep_clamp) - then it falls into + * the stub + call-drop path below. + */ +if (gsp_phys_gr_slot_kept(j, sa_mask, + sa_ent)) { + for (t = 0; t < sa_ncov; t++) + if (sa_cov[t] == j) + have = true; + if (!have && sa_ncov < 40u) { + sa_cov[sa_ncov] = j; + sa_cov_ent[sa_ncov] = + sa_ent[j]; + sa_ncov++; + } + continue; + } + /* + * SESSION 153-CONTINUED-114: a called + * slot outside the capture that the + * golden library serves counts as + * covered - the pack overlay emits its + * SA pair + code, so no stub pair is + * emitted and the CALL runs survive + * the drop pass (see slot_stubbed). + */ + { + u32 gent, gn; + const u32 *gw; + + if (gsp_phys_gr_golden_slot( + j, &gent, &gw, + &gn)) { + for (t = 0; t < sa_ncov; + t++) + if (sa_cov[t] == j) + have = true; + if (!have && + sa_ncov < 40u) { + sa_cov[sa_ncov] = j; + sa_cov_ent[sa_ncov] = + gent; + sa_ncov++; + } + continue; + } + } + /* + * SESSION 153-CONTINUED-68: a slot is only + * usable when the capture holds its SA entry + * AND the IR instruction that entry names + * actually entered the replay (see + * gsp_sema_mme_ir_covered); a last-wins/ + * truncated mismatch makes the engine fetch + * macro data that is not there + * (MISSING_MACRO_DATA). Called-and-covered + * slots are logged too (r1134 ILLEGAL_OPCODE + * attribution: which macros actually run). + */ + if (sa_mask & (1ull << j)) { + if (!READ_ONCE(gsp_phys_rmcall_gr_sa_stub_all) && + gsp_sema_mme_ir_covered(sa_ent[j]) && + !gsp_phys_gr_slot_golden_blank(j) && + !(READ_ONCE(gsp_phys_rmcall_gr_nop_drop) & + (1ull << j))) { + for (t = 0; t < sa_ncov; t++) + if (sa_cov[t] == j) + have = true; + if (!have && sa_ncov < 40u) { + sa_cov[sa_ncov] = j; + sa_cov_ent[sa_ncov] = + sa_ent[j]; + sa_ncov++; + } + continue; + } + } + for (t = 0; t < sa_nstub; t++) + if (sa_stub[t] == j) + have = true; + if (!have && sa_nstub < 40u) { + sa_stub[sa_nstub] = j; + sa_stub_ent[sa_nstub] = + (sa_mask & (1ull << j)) ? + sa_ent[j] : 0u; + sa_nstub++; + } + } + so += sspan; + } + if (sa_nstub) + sa_block = 8u + 3u * sa_nstub; + } + + /* + * SESSION 153-CONTINUED-68: full run-header census of the 3D + * window being replayed (bounded; first 4 windows). The + * r1134 ILLEGAL_OPCODE attribution needs the real call + * structure: which methods the window writes, how many data + * words each run carries, and the CALL_MME_MACRO/DATA + * pattern (op 1 INC vs op 5 ONE_INC) around them. + */ + { + static u32 win_dump_n; + static u32 win_dump_beef; + + /* + * SESSION 153-CONTINUED-69: the first-4 budget was + * consumed by the X channel's early windows in every + * boot, so the beef draw-class window (the one that + * faults) never got its run-header census. Two extra + * dumps are reserved for beef-family windows. + */ + if (win3d && win3d_n && + (win_dump_n < 4u || + (((req->h_object & 0xfffff000u) == 0xbeef0000u) && + win_dump_beef < 2u))) { + u32 wo = 0, wr = 0; + + win_dump_n++; + if ((req->h_object & 0xfffff000u) == 0xbeef0000u) + win_dump_beef++; + pr_info("vgsp-gr: MME-WINRUN dump h=0x%x push=0x%llx window=%u\n", + req->h_object, req->push_va, win3d_n); + while (wo < win3d_n && wr < 200u) { + u32 h = win3d[wo]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0x1fffu; + u32 addr = (h & 0xfffu) << 2; + u32 span = 1u; + u32 d0 = (wo + 1u < win3d_n) ? + win3d[wo + 1u] : 0u; + u32 d1 = (wo + 2u < win3d_n) ? + win3d[wo + 2u] : 0u; + u32 d2 = (wo + 3u < win3d_n) ? + win3d[wo + 3u] : 0u; + u32 d3 = (wo + 4u < win3d_n) ? + win3d[wo + 4u] : 0u; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && wo + 1u + cnt <= win3d_n) + span = 1u + cnt; + pr_info("vgsp-gr: MME-WINRUN off=%x op=%u cnt=%u mthd=%x d0=%x d1=%x d2=%x d3=%x\n", + wo * 4u, op, cnt, addr, d0, d1, + d2, d3); + wo += span; + wr++; + } + } + } + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_prelude)) { + pl = gsp_sema_mme_prelude_get(&pl_n, &pl_gen); + /* + * r1453 PAIR-CHAIN: remember the generation this + * launch was built against; the submit's success + * edge promotes it to gsp_phys_gr_mme_pairs_gen + * once a pair-bearing window retires (the held + * elision extension's install proof). + */ + gsp_phys_gr_mme_launch_gen = pl_gen; + /* + * SESSION 153-CONTINUED-72: with keep_mask bits set a + * kept macro can execute, and it needs the MME program + * resident. The r1152 evidence: the prelude was carried + * by one launch of the generation only; when that launch + * did not complete (engine fetch NONE / timeout), the + * later kept-macro launches ran an UNLOADED program and + * priv-errored (ILLEGAL_OPCODE -> poison). Force + * re-emission on every launch while any slot is kept + * (idempotent per the emission contract). + * + * SESSION 153-CONTINUED-79: the emission is drained in + * per-launch parts capped at block_cap words (see the + * sema_gl_mme_block_cap comment; oversized blocks never + * start). The latch (gsp_phys_gr_mme_prelude_gen/h) is + * set only when the final part has gone out. + */ + if (pl && pl_n && pl_gen) { + /* + * Fixed overhead: SET_OBJECT bind (2), SA-stub + * block, window, and the 4x2-word release tail. + */ + u32 fixed = 2u + sa_block + win3d_n + 8u; + + /* + * SESSION 153-CONTINUED-82: latch state BEFORE + * this launch's emission (see the hold decision + * below). A launch that first uploads a + * generation's MME code must not run real CALL + * runs in the same block (the r1222 evidence: + * the drain-completing launch bundled part-2's + * IR uploads with the window's real calls and + * hit ILLEGAL_OPCODE even though the latch was + * set by its own emission step). + */ + latch_before = + (pl_gen == READ_ONCE( + gsp_phys_gr_mme_prelude_gen) && + req->h_object == READ_ONCE( + gsp_phys_gr_mme_prelude_h)); + + pl_room = (fixed + 24u <= block_cap) ? + block_cap - fixed : 0u; + if (pl_room > pl_n) + pl_room = pl_n; + /* + * (Re)start the drain on a new generation/ + * channel, or when the request is forced (kept + * slots; a context-wiping wedge recovery clears + * the latch) and nothing is left to drain. + */ + if (pl_gen != + READ_ONCE(gsp_phys_gr_mme_pre_emit_gen) || + req->h_object != + READ_ONCE(gsp_phys_gr_mme_pre_emit_h) || + (READ_ONCE(gsp_phys_gr_mme_pre_emit_off) >= + pl_n && + ((READ_ONCE(gsp_phys_rmcall_gr_keep_mask) && + !(READ_ONCE( + gsp_phys_rmcall_gr_mme_live_skip) && + READ_ONCE(gsp_phys_gr_mme_live_gen) == + pl_gen)) || + pl_gen != READ_ONCE( + gsp_phys_gr_mme_prelude_gen)))) { + /* + * r1447: an actual generation change + * restarts the strict grace budget + * with the new residency attempt. + */ + if (pl_gen != + READ_ONCE( + gsp_phys_gr_mme_pre_emit_gen)) + WRITE_ONCE( + gsp_phys_gr_mme_strict_holds, + 0); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_gen, + pl_gen); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_h, + req->h_object); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + /* + * CONT-105: the packed split shares this + * restart edge; reset its cursor too so a + * wedged-recover latch clear replays the + * whole packed stream. + */ + WRITE_ONCE(gsp_phys_gr_mme_pack_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_sa, + false); + } + if (pl_room >= 24u) { + emit_prelude = + (pl_gen != READ_ONCE( + gsp_phys_gr_mme_prelude_gen) || + req->h_object != READ_ONCE( + gsp_phys_gr_mme_prelude_h) || + READ_ONCE( + gsp_phys_gr_mme_pre_emit_off) < + pl_n); + } else { + static u32 pre_defer_log; + + if (pre_defer_log < 8) { + pre_defer_log++; + pr_info("vgsp-gr: MME-PRELUDE deferred (window=%u prelude=%u sa=%u cap=%u) h=0x%x\n", + win3d_n, pl_n, sa_block, + block_cap, + req->h_object); + } + } + } + } + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | (1u << 16) | + (0u << 13) | + (GSP_PHYS_RMCALL_GR_SET_OBJECT >> 2); + *p++ = GSP_PHYS_RMCALL_CLASS_GR_3D; + /* + * SESSION 153-CONTINUED-90: SET_SHADER_EXCEPTIONS (0x1528) + * <- 0, the first write of NVIDIA's own Turing 3D channel + * init (see the sema_gl_shader_exc knob): with shader + * exceptions enabled the DS raises "Shader Program Header + * 5/9/10/11" on the draws whose SPHs it would otherwise + * tolerate, and the raised exception poisons the channel + * before any store retires. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_shader_exc)) { + static u32 shader_exc_log; + bool in_window = + READ_ONCE(gsp_phys_rmcall_gr_shader_exc_skip) && + gsp_phys_gr_method_present(win3d, win3d_n, + 0x1528u); + + if (in_window) { + if (shader_exc_log < 8u) { + shader_exc_log++; + pr_info("vgsp-gr: SHADER-EXC skipped (window carries 0x1528) h=0x%x push=0x%llx\n", + req->h_object, + req->push_va); + } + } else if (READ_ONCE( + gsp_phys_rmcall_gr_shader_exc_immd)) { + /* + * SESSION 154-1: the client's own + * encoding (nvPushImmedVal): immd13 + * header with the 13-bit value in bits + * 16..28 and NO following data word. + * Value 0 = ENABLE FALSE. + */ + *p++ = 0x80000000u | (0x1528u >> 2); + if (shader_exc_log < 8u) { + shader_exc_log++; + pr_info("vgsp-gr: SHADER-EXC emitted 0x1528 <- 0 (1 word immd13 %08x; pos=%u h=0x%x push=0x%llx)\n", + *(p - 1), (u32)(p - pb), + req->h_object, req->push_va); + } + } else { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (1u << 16) | (0x1528u >> 2); + *p++ = 0u; + if (shader_exc_log < 8u) { + shader_exc_log++; + pr_info("vgsp-gr: SHADER-EXC emitted 0x1528 <- 0 (2 words INC; pos=%u h=0x%x push=0x%llx)\n", + (u32)(p - pb), + req->h_object, + req->push_va); + } + } + } + /* + * SESSION 153-CONTINUED-91: DS HWW report suppression. + * + * The DS validates the SPH of every bound pipeline program at + * draw time and reports failures through DS_HWW_REPORT_MASK + * (0x405844, observed live = 0x00ffffff = every SPH<n>_ERR + * reported; mask_2 = 0x77). A reported failure poisons the + * whole PRI block (reads return 0xbadf1201 - the priv-error + * marker already documented for the MME latch) and the GSP-RM + * exception handler kills the channel: every draw attempt + * wedges at the first SPH error storm even when the SPH bytes + * at the bound VAs are the client's own real V4 images (r1298: + * INLINE-RB rb=020a0481 at the bind VA, DS still 5/9/10/11). + * The reference driver's own diagnostic path (Aftermath) + * explicitly ENABLES these reports to capture them, i.e. the + * reports are a diagnostic tap, not the shader-execute gate. + * Clear the mask so a SPH miss latches (readable between + * faults) without poisoning the channel. Host write only: + * an in-stream CALL_MME_MACRO(5) (the NVN PRI R/W macro) + * faulted with MME ILLEGAL_OPCODE (pc=513 mac=0xf1) on + * r1300 - the Turing MME's loaded macro program does not + * carry that GF100-era slot - so the register is written + * directly (it persists: the next beef launch reads 0/0). + * A/B with sema_ds_mask_clear=0. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_ds_mask_clear)) { + static u32 ds_mask_log; + void __iomem *ds_bar0 = gsp_phys_gpu_bar0(); + u32 m0 = 0, m2 = 0; + + if (ds_bar0) { + m0 = ioread32(ds_bar0 + 0x00405844u); + m2 = ioread32(ds_bar0 + 0x0040584cu); + if (m0 || m2) { + iowrite32(0u, ds_bar0 + 0x00405844u); + iowrite32(0u, ds_bar0 + 0x0040584cu); + } + /* + * SESSION 153-CONTINUED-92: clear the DS HWW + * ESR latch (W1C) ahead of every window so the + * retire readback names EXACTLY this launch's + * failed SPH checks instead of the union of + * every prior draw's. The latch was observed + * sticky across launches (r1308: a6000e20 + * constant over 25 s of draws), which made + * "did this window's checks change?" unreadable. + * Log the value being cleared (the previous + * window's set) so the A/B sequence is in the + * ring either way. + */ + { + u32 esr = ioread32(ds_bar0 + 0x00405840u); + + if (esr) + iowrite32(esr | 0x40000000u, + ds_bar0 + 0x00405840u); + if (ds_mask_log < 24u) { + ds_mask_log++; + pr_info("vgsp-gr: DS-ESR pre-clear=%08x (prev window's latch; this window starts clean) h=0x%x push=0x%llx\n", + esr, req->h_object, + req->push_va); + } + } + } + if (ds_mask_log < 24u) { + ds_mask_log++; + pr_info("vgsp-gr: DS-MASK clear mask=%08x/%08x rb=%08x/%08x h=0x%x push=0x%llx\n", + m0, m2, + ds_bar0 ? ioread32(ds_bar0 + 0x00405844u) : 0u, + ds_bar0 ? ioread32(ds_bar0 + 0x0040584cu) : 0u, + req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-86: the Turing channel-init SPH version + * pair (see the sema_gl_sph_version knob). Emitted on EVERY + * replayed 3D window (3 words, idempotent): the DS validates + * each bound program's SPH version against this window at draw + * time, and the replay context never received the client's + * once-per-channel write. Emitted before the prelude-room + * computations below so the live (p - pb) accounting covers + * it exactly. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_version)) { + /* + * SESSION 153-CONTINUED-87: the window's accepted + * version range must COVER the images the engine + * will fetch. The client's own init push writes + * 4|(4<<16), but our authored (V3) images are + * rejected by a 4/4 window (SPH9|11|18) while the + * r950-era acceptance - same V3 image - ran with + * NO version window. Emit oldest=V(blob), + * current=max(V(blob),4) so both the V3 image and + * a future V4 re-tag stay inside the range. + */ + u32 bv = (u32)gsp_sema_gl_sph_blob_version(); + u32 val = 0x00040004u; + u64 ov = READ_ONCE(gsp_phys_rmcall_gr_sph_version_val); + + if (ov) + val = (u32)ov; + else if (bv >= 1u && bv <= 15u) + val = ((bv < 4u ? 4u : bv) << 16) | bv; + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (2u << 16) | (0x16a4u >> 2); + *p++ = val; + *p++ = val; + } + /* + * SESSION 153-CONTINUED-91: the Turing program region. + * + * The DS's SPH validation checks each fetched SPH against the + * channel's PROGRAM REGION. On pre-Volta that was + * SET_PROGRAM_REGION_A/B (0x1608/0x160c); on Volta+ those + * methods are obsolete and the driver writes the same 64-bit + * shader-code-heap base through the MME shadow scratch pair + * (nvTuring: NVC397_SET_MME_PROGRAM_REGION = SET_MME_SHADOW_ + * SCRATCH(MME_SCRATCH_PROGRAM_REGION_A=0x78/B=0x79), + * gv100ResendProgramAddresses/`gf100Init`; method 0x3400 + + * i*4 -> 0x35e0/0x35e4). Observed live at the faulting draw: + * DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A/B = 0/0 and DS latches + * SPH errors for every bound stage whose bytes are provably + * correct (r1298 INLINE-RB rb=020a0481 at the bind VA; + * r1303 GR-STATE sph stages 1/2/5 = 0x620900/0x620200/0x620000 + * with ds=a4000e20 and rr=0/0) - a region mismatch rejects all + * of them regardless of content. Emit the pair (A = upper + * 7:0, B = lower 31:0) with the launch's own program family + * page as the heap base. A/B with sema_gl_sph_region=0. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_region) && req->prog_va) { + static u32 sph_reg_log; + u64 base = READ_ONCE(gsp_phys_rmcall_gr_sph_region_base); + u64 bind_va = 0; + bool bind_base = + READ_ONCE(gsp_phys_rmcall_gr_sph_region_bind) && + gsp_phys_gr_first_prog_bind(win3d, win3d_n, + &bind_va); + + /* + * SESSION 154-1: the DS compares the fetched SPH + * address against this region. The engine fetches + * the program through its relocated shadow VA, so + * the correct base is the bound pair's own shadow + * page; the forced override/shadow-pool base is + * only the fallback when the window binds nothing. + */ + if (bind_base) + base = bind_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + else if (!base) + base = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x35e0u >> 2); + *p++ = (u32)(base >> 32) & 0xffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x35e4u >> 2); + *p++ = (u32)base; + if (sph_reg_log < 16u) { + sph_reg_log++; + pr_info("vgsp-gr: SPH-REGION emitted base=0x%llx (scratch 0x78/0x79; src=%s bind=0x%llx) h=0x%x push=0x%llx\n", + base, bind_base ? "bind-shadow" : + "override/auto", bind_va, + req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-101: MME fixed-constant-buffer scratch + * seed (see the sema_gl_mme_seed* knobs). mmeCodeOGL_C597_ + * SETUP_FIXED_CONST (slot 0) emits SET_CONSTANT_BUFFER_ + * SELECTOR_B/C from shadow scratch 0x18/0xd18; the replay + * context never received the client's once-per-channel write + * (gf100InitConstantBuffers pushes method 0x3460), so the + * macro read 0 and the selector went empty. Emit the seed + * ahead of the window; the value is the fixed CB address>>8, + * page-translated to the launch's shadow/identity mapping when + * sema_gl_mme_seed_xlat is set (see gsp_phys_rmcall_gr_3d_ + * reloc - same machinery as the operand pairs). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed)) { + static u32 mme_seed_log; + u32 sv = READ_ONCE(gsp_phys_rmcall_gr_mme_seed_val); + bool have = sv != 0u; + + if (!have) + have = gsp_sema_mme_scratch18_get(&sv); + if (have && sv && + (u32)(p - pb) + 8u < GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u64 va = (u64)sv << 8; + bool xl = false; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed_xlat) + && va) { + u32 up_w = (u32)((va >> 32) & 0xffu); + u32 lo_w = (u32)va & 0xfffffffcu; + u32 n_sh = 0; + + if (gsp_phys_rmcall_gr_3d_reloc( + &up_w, &lo_w, req, &n_sh)) { + va = ((u64)(up_w & 0xffu) + << 32) | + (u64)(lo_w & 0xfffffffcu); + sv = (u32)(va >> 8); + xl = true; + } + } + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x3460u >> 2); + *p++ = sv; + if (mme_seed_log < 64u) { + mme_seed_log++; + pr_info("vgsp-gr: MME-SEED scratch 0x18 <- 0x%08x (va=0x%llx xlat=%u) h=0x%x push=0x%llx\n", + sv, va, xl, req->h_object, + req->push_va); + } + } else if (mme_seed_log < 64u) { + mme_seed_log++; + pr_info("vgsp-gr: MME-SEED no value (client 0x3460 not seen yet; set sema_gl_mme_seed_val) h=0x%x\n", + req->h_object); + } + } + /* + * SESSION 153-CONTINUED-115: STATE 0xD19 seed (shadow scratch + * 0x19, method 0x3464 = SET_MME_SHADOW_SCRATCH(0x19): the + * constant-buffer selector B/C address>>8). The bug2548370 + * loadConstantBuffer macro's in-band update emits the selector + * from d19 once the fast path is refused (see the state seed + * below); the replay context never received the client's + * once-per-channel 0x3464 write, so d19 read 0 and the + * consumer fetched constants from address 0. Value: + * sema_mme_state_d19 when nonzero, else the captured write + * (0x3464); fail-closed - a missed shadow translation skips + * the emission instead of writing an untranslated selector. + * Gated by sema_gl_mme_state_seed (STATE class). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_state_seed)) { + static u32 mme_state_d19_log; + u32 sv = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d19); + bool have = sv != 0u; + + if (!have) + have = gsp_sema_mme_scratch19_get(&sv); + if (have && sv && + (u32)(p - pb) + 8u < GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u64 va = (u64)sv << 8; + bool xl = false; + bool ok = true; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed_xlat) + && va) { + u32 up_w = (u32)((va >> 32) & 0xffu); + u32 lo_w = (u32)va & 0xfffffffcu; + u32 n_sh = 0; + + if (gsp_phys_rmcall_gr_3d_reloc( + &up_w, &lo_w, req, + &n_sh)) { + va = ((u64)(up_w & 0xffu) + << 32) | + (u64)(lo_w & 0xfffffffcu); + sv = (u32)(va >> 8); + xl = true; + } else { + ok = false; + } + } + if (ok) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (1u << 16) | (0x3464u >> 2); + *p++ = sv; + if (mme_state_d19_log < 64u) { + mme_state_d19_log++; + pr_info("vgsp-gr: MME-STATE-SEED d19=0x%08x (va=0x%llx xlat=%u) h=0x%x push=0x%llx\n", + sv, va, xl, + req->h_object, + req->push_va); + } + } + } + } + /* + * SESSION 153-CONTINUED-116: STATE 0xD24 seed (shadow scratch + * 0x24, method 0x3490 = SET_MME_SHADOW_SCRATCH(0x24)). The + * bug2548370 slow path reads the 0x349x family beside the d19 + * selector; the replay context never received the client's + * once-per-channel 0x3490 write, so d24 read whatever the + * replay left (r1440: ILLEGAL_OPCODE in the same post-slot-57 + * window). Value: sema_mme_state_d24 when nonzero, else the + * captured write (0x3490); fail-closed - a missed shadow + * translation skips the emission instead of writing an + * untranslated value. Gated by sema_gl_mme_state_seed. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_state_seed)) { + static u32 mme_state_d24_log; + u32 sv = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d24); + bool have = sv != 0u; + + if (!have) + have = gsp_sema_mme_scratch24_get(&sv); + if (have && sv && + (u32)(p - pb) + 8u < GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u64 va = (u64)sv << 8; + bool xl = false; + bool ok = true; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed_xlat) + && va) { + u32 up_w = (u32)((va >> 32) & 0xffu); + u32 lo_w = (u32)va & 0xfffffffcu; + u32 n_sh = 0; + + if (gsp_phys_rmcall_gr_3d_reloc( + &up_w, &lo_w, req, + &n_sh)) { + va = ((u64)(up_w & 0xffu) + << 32) | + (u64)(lo_w & 0xfffffffcu); + sv = (u32)(va >> 8); + xl = true; + } else { + ok = false; + } + } + if (ok) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (1u << 16) | (0x3490u >> 2); + *p++ = sv; + if (mme_state_d24_log < 64u) { + mme_state_d24_log++; + pr_info("vgsp-gr: MME-STATE-SEED d24=0x%08x (va=0x%llx xlat=%u) h=0x%x push=0x%llx\n", + sv, va, xl, + req->h_object, + req->push_va); + } + } + } + } + /* + * SESSION 153-CONTINUED-116: STATE 0xD25 seed (shadow scratch + * 0x25, method 0x3494 = SET_MME_SHADOW_SCRATCH(0x25)), the + * next unseeded input on the same slow path; same value + * policy, translation and fail-closed rule as d24 above. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_state_seed)) { + static u32 mme_state_d25_log; + u32 sv = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d25); + bool have = sv != 0u; + + if (!have) + have = gsp_sema_mme_scratch25_get(&sv); + if (have && sv && + (u32)(p - pb) + 8u < GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u64 va = (u64)sv << 8; + bool xl = false; + bool ok = true; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed_xlat) + && va) { + u32 up_w = (u32)((va >> 32) & 0xffu); + u32 lo_w = (u32)va & 0xfffffffcu; + u32 n_sh = 0; + + if (gsp_phys_rmcall_gr_3d_reloc( + &up_w, &lo_w, req, + &n_sh)) { + va = ((u64)(up_w & 0xffu) + << 32) | + (u64)(lo_w & 0xfffffffcu); + sv = (u32)(va >> 8); + xl = true; + } else { + ok = false; + } + } + if (ok) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (1u << 16) | (0x3494u >> 2); + *p++ = sv; + if (mme_state_d25_log < 64u) { + mme_state_d25_log++; + pr_info("vgsp-gr: MME-STATE-SEED d25=0x%08x (va=0x%llx xlat=%u) h=0x%x push=0x%llx\n", + sv, va, xl, + req->h_object, + req->push_va); + } + } + } + } + /* + * SESSION 153-CONTINUED-105: golden MME runtime state seed + * (see the sema_gl_mme_state_seed comment). Five idempotent + * shadow-scratch writes ahead of the window so the + * bug2548370 loadConstantBuffer macro (slot 57) branches on + * the seeded values instead of whatever the replay context + * happens to hold (the captured stream never carries + * 0xd26/0xd27/0xd28). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_state_seed)) { + static u32 mme_state_seed_log; + u32 mv[5]; + u32 mi; + + mv[0] = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d26); + mv[1] = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d27); + mv[2] = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d28); + mv[3] = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d29); + mv[4] = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d2a); + if ((u32)(p - pb) + 10u < + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + for (mi = 0; mi < 5u; mi++) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (1u << 16) | + ((0x3498u + 4u * mi) >> 2); + *p++ = mv[mi]; + } + if (mme_state_seed_log < 64u) { + mme_state_seed_log++; + pr_info("vgsp-gr: MME-STATE-SEED d26=%08x d27=%08x d28=%08x d29=%08x d2a=%08x h=0x%x push=0x%llx\n", + mv[0], mv[1], mv[2], mv[3], + mv[4], req->h_object, + req->push_va); + } + } + } + /* + * SESSION 155-CONT-5: engine-visible CB selector seed (see + * the sema_gl_mme_selector_seed comment). Emits the exact + * 4-method group SETUP_FIXED_CONST would emit, so a window + * that calls macro 57 without a preceding macro 0 has a + * valid selector instead of poisoning at 0x2390. Size is + * MME_VPC_CONSTANT_BUFFER_SIZE (0x7000, gf100mme.h); B/C + * carry the same fixed-CB base>>8 and translation as the + * 0x3460 scratch seed; the offset is the seeded d29. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_selector_seed) && + READ_ONCE(gsp_phys_rmcall_gr_mme_state_seed)) { + static u32 mme_sel_seed_log; + u32 sv = READ_ONCE(gsp_phys_rmcall_gr_mme_seed_val); + bool have = sv != 0u; + + if (!have) + have = gsp_sema_mme_scratch18_get(&sv); + if (have && sv && + (u32)(p - pb) + 8u < + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u64 va = (u64)sv << 8; + bool xl = false; + u32 off; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_seed_xlat) + && va) { + u32 up_w = (u32)((va >> 32) & 0xffu); + u32 lo_w = (u32)va & 0xfffffffcu; + u32 n_sh = 0; + + if (gsp_phys_rmcall_gr_3d_reloc( + &up_w, &lo_w, req, &n_sh)) { + va = ((u64)(up_w & 0xffu) + << 32) | + (u64)(lo_w & 0xfffffffcu); + sv = (u32)(va >> 8); + xl = true; + } + } + off = READ_ONCE(gsp_phys_rmcall_gr_mme_state_d29); + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x2380u >> 2); + *p++ = 0x7000u; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x2384u >> 2); + *p++ = (sv >> 24) & 0xffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x2388u >> 2); + *p++ = sv & 0xffffffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x238cu >> 2); + *p++ = off & 0xffffu; + if (mme_sel_seed_log < 64u) { + mme_sel_seed_log++; + pr_info("vgsp-gr: MME-SELECTOR-SEED size=0x7000 addr=0x%06x off=0x%04x (va=0x%llx xlat=%u) h=0x%x push=0x%llx\n", + sv & 0xffffffu, off & 0xffffu, + va, xl, req->h_object, + req->push_va); + } + } + } + /* + * SESSION 153-CONTINUED-93: the LEGACY program-region pair + * (0x1608/0x160c = the pre-Volta SET_PROGRAM_REGION_A/B whose + * names the DS's own snapshot registers + * DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A/B carry). Those + * snapshots read 0/0 on every SPH-faulting window while the + * MME shadow pair (0x78/0x79) is provably emitted, so the + * DS's validation region may be the legacy pair's state, not + * the MME scratch. The methods are "obsolete" on Volta+ (not + * in the c597 class table), so this is an A/B: a PBDMA that + * refuses them faults the launch visibly, an accepting one + * tells us via the latch. Knob sema_sph_region_legacy + * (default 0). Emitted beside the MME pair with the same + * base. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_region) && req->prog_va && + READ_ONCE(gsp_phys_rmcall_gr_sph_region_legacy)) { + static u32 sph_reg_leg_log; + u64 base = READ_ONCE(gsp_phys_rmcall_gr_sph_region_base); + u64 bind_va = 0; + bool bind_base = + READ_ONCE(gsp_phys_rmcall_gr_sph_region_bind) && + gsp_phys_gr_first_prog_bind(win3d, win3d_n, + &bind_va); + + if (bind_base) + base = bind_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + else if (!base) + base = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x1608u >> 2); + *p++ = (u32)(base >> 32) & 0xffu; + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x160cu >> 2); + *p++ = (u32)base; + if (sph_reg_leg_log < 16u) { + sph_reg_leg_log++; + pr_info("vgsp-gr: SPH-REGION-LEGACY emitted base=0x%llx (0x1608/0x160c; src=%s bind=0x%llx) h=0x%x push=0x%llx\n", + base, bind_base ? "bind-shadow" : + "override/auto", bind_va, + req->h_object, req->push_va); + } + } + /* + * SESSION 153-CONTINUED-84: SET_MME_VERSION=1 (0x234). + * + * The Turing channel-init defaults push + * `nvcMethodPusher: NVC597_SET_MME_VERSION <- 1` once per + * channel (class c597, TURING_OR_NEWER), and the replayed + * context otherwise never sees it: the captured MME setup + * families carry IR/SA/shadow-scratch loads but no 0x234 run + * (verified in the injected full-SA stream too). With the + * version register unset the MME never starts a macro - the + * r1245 evidence: call-bearing launches (real {0,2,34,39,40}, + * full 55-slot prelude, SA re-armed before the window) hang + * the front-end (`MME-FAST ... events=0` -> `GR LAUNCH FAILED: + * timeout after 2500 ms` with qmd_rel=0/ch_rel=0) while every + * MME register stays idle (`pc_valid=0`, `gen0=0`, + * `mme_esr=80000000`): the FE waits forever for a macro the + * halted MME cannot run. Emitted with the prelude step (the + * resident program is (re)loaded right after), ahead of the + * window, so the version select has landed before the first + * CALL_MME_MACRO run. Knob: sema_gl_mme_version (default 1; + * A/B with 0 reproduces the parked launches). + */ + if (emit_prelude && READ_ONCE(gsp_phys_rmcall_gr_mme_version)) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | + (0x234u >> 2); + *p++ = 1u; + } + if (emit_prelude) { + u32 g0 = pl_gen, g1 = 0, sdrop = 0, pwords = 0; + u32 doff = READ_ONCE(gsp_phys_gr_mme_pre_emit_off); + u32 room = (block_cap > (u32)(p - pb) + win3d_n + 8u + + sa_block) ? + block_cap - (u32)(p - pb) - win3d_n - 8u - + sa_block : 0u; + + /* + * SESSION 153-CONTINUED-79: run-aligned drain at the + * generation cursor; whole runs only, SDM-family + * single-word entries skipped exactly like the legacy + * sanitize (see gsp_phys_rmcall_gr_sdm_drop). + */ + /* + * SESSION 153-CONTINUED-99b: packed emission (see the + * sema_gl_mme_prelude_pack comment). Rebuild the + * partitioned IR, then emit the SA table + one + * contiguous 0x114 run; the drain loop is skipped. + */ + bool packed = false; + + if (READ_ONCE(gsp_phys_rmcall_gr_mme_prelude_pack)) { + u32 sa_ent[64]; + u64 sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + u32 maxa = 0, r = 0, nruns = 0, npair = 0; + u32 need, kk; + u32 *pp; + + memset(gsp_phys_gr_mme_ir_pack, 0, + sizeof(gsp_phys_gr_mme_ir_pack)); + while (r < pl_n) { + u32 h = pl[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt && + r + 1u + cnt <= pl_n) ? + 1u + cnt : 1u; + + if (r + span > pl_n) + span = pl_n - r; + if (op == 5u && + ((h & 0x1fffu) << 2) == 0x114u && + cnt >= 2u) { + u32 ptr = pl[r + 1u]; + u32 blen = cnt - 1u; + u32 nxt = 0, k; + + for (kk = 0; kk < 64u; kk++) + if ((sa_mask & (1ull << kk)) && + sa_ent[kk] > ptr && + (!nxt || + sa_ent[kk] < nxt)) + nxt = sa_ent[kk]; + /* + * CONT-100 units fix: `ptr` is an + * MME INSTRUCTION address (the + * driver advances codeOffset by + * codeSize/3, mmeDownload.cpp), + * and each instruction is 3 IR + * words stored at 3*ptr..3*ptr+2. + * Indexing the contiguous word + * image at ptr+k (the pre-CONT-100 + * form) misplaced every macro and + * left 2/3 of the program + * unreachable (r1343/r1344 poison, + * only ~312 of 933 instructions + * covered). Clamp the tail at the + * next SA entry in INSTRUCTIONS. + */ + if (nxt && + ptr + (blen + 2u) / 3u > nxt) + blen = 3u * (nxt - ptr); + for (k = 0; k < blen && + ptr * 3u + k < 4096u; k++) + gsp_phys_gr_mme_ir_pack[ptr * 3u + k] = + pl[r + 2u + k]; + if (ptr * 3u + blen > maxa) + maxa = ptr * 3u + blen; + nruns++; + } + r += span; + } + /* + * SESSION 153-CONTINUED-114: golden overlay. + * After the capture fill, merge the golden + * library's words into the packed image and + * the local SA pair table for every slot the + * capture cannot serve (mode 1), or every + * slot (mode 2 / force). Written after the + * capture fill so the golden words win any + * overlap; a complete capture is untouched + * in mode 1 (the overlay then only logs). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_golden)) { + u32 gj, goldn = 0, ginsp = 0; + u32 gblank = 0; + + for (gj = 0; gj < 64u; gj++) { + u32 gent, gn, q; + const u32 *gw; + + if (!gsp_phys_gr_golden_slot( + gj, &gent, &gw, + &gn)) + continue; + ginsp++; + if (!READ_ONCE(gsp_phys_rmcall_gr_mme_golden_force)) { + /* + * Mode 1: only slots the + * capture cannot serve + * (no SA entry, or the + * entry's IR never + * entered the replay). + */ + if ((sa_mask & + (1ull << gj)) && + sa_ent[gj] && + gsp_sema_mme_ir_covered( + sa_ent[gj])) + continue; + } + for (q = 0; + q < gn && + gent * 3u + q < 4096u; + q++) + gsp_phys_gr_mme_ir_pack[gent * 3u + q] = + gw[q]; + if (gent * 3u + gn > maxa) + maxa = gent * 3u + gn; + sa_mask |= 1ull << gj; + sa_ent[gj] = gent; + goldn++; + } + /* + * r1448 STRATEGY 3.3: force-blank + * the captured bodies of every slot + * the golden library cannot serve so + * no capture word can survive - only + * golden bodies may execute + * (fail-closed). The zero range is + * bounded by the next captured SA + * entry, exactly the bound the + * capture fill above applied. + * Clearing the local sa_mask/sa_ent + * pair also unpublishes the slot + * from the SA re-arm; its CALL runs + * are dropped via the stubbed-slot + * predicate (slot_stubbed -> + * golden_blank). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_golden_force) && + READ_ONCE(gsp_phys_rmcall_gr_mme_golden_only)) { + for (gj = 0; gj < 64u; gj++) { + u32 gent, gn, q, kk2; + const u32 *gw; + u32 bstart, bend; + u64 nmask; + + if (!(sa_mask & (1ull << gj)) || + !sa_ent[gj]) + continue; + if (gsp_phys_gr_golden_slot( + gj, &gent, &gw, + &gn)) + continue; + bstart = 3u * sa_ent[gj]; + bend = 4096u; + nmask = sa_mask & + ~(1ull << gj); + for (kk2 = 0; kk2 < 64u; kk2++) { + if (!(nmask & + (1ull << kk2)) || + !sa_ent[kk2]) + continue; + if (3u * sa_ent[kk2] > bstart && + 3u * sa_ent[kk2] < bend) + bend = 3u * sa_ent[kk2]; + } + for (q = bstart; q < bend; q++) + gsp_phys_gr_mme_ir_pack[q] = 0; + sa_mask &= ~(1ull << gj); + sa_ent[gj] = 0; + gblank++; + } + } + if (goldn && gsp_phys_gr_golden_log < 8u) { + gsp_phys_gr_golden_log++; + pr_info("vgsp-gr: MME-GOLDEN overlay %u slot(s) inspected=%u blank=%u maxa=%u h=0x%x push=0x%llx\n", + goldn, ginsp, gblank, + maxa, + req->h_object, + req->push_va); + } + } + /* + * CONT-112: single-word A/B override of the + * packed program (see the sema_gl_mme_irpatch + * comment). Applied after the image is built so + * the override is what the macro executes and + * what the IRAM gate compares against. + */ + if (gsp_phys_gr_mme_irpatch_off < maxa) { + pr_info("vgsp-gr: MME-IRPATCH word[%u] = 0x%08x (was 0x%08x, maxa=%u)\n", + gsp_phys_gr_mme_irpatch_off, + gsp_phys_gr_mme_irpatch_val, + gsp_phys_gr_mme_ir_pack[ + gsp_phys_gr_mme_irpatch_off], + maxa); + gsp_phys_gr_mme_ir_pack[ + gsp_phys_gr_mme_irpatch_off] = + gsp_phys_gr_mme_irpatch_val; + } + for (kk = 0; kk < 64u; kk++) + if ((sa_mask & (1ull << kk)) && + sa_ent[kk]) + npair++; + need = 2u + maxa + 3u * npair; + /* + * CONT-105: chunked emission. The CONT-99b + * all-or-nothing form only fires when the + * whole packed stream fits one block (need <= + * room); the current 55-slot capture needs + * ~2966 words against ~1600 available, so it + * silently fell back to the raw drain. The + * split form drains SA (once) + whole- + * instruction IR chunks across consecutive + * launches with the generation cursor reset + * alongside the drain cursor (see + * gsp_phys_gr_mme_pack_off). + */ + { + bool split = READ_ONCE( + gsp_phys_rmcall_gr_mme_pack_split); + u32 poff = READ_ONCE( + gsp_phys_gr_mme_pack_off); + bool sa_done = READ_ONCE( + gsp_phys_gr_mme_pack_sa); + u32 left, chunk; + bool can; + + /* + * Reserve the window's SA re-arm + * budget (<= 32 entries * 3 words) so + * the block stays under block_cap. + */ + left = (split && room > 96u) ? + room - 96u : (split ? 0u : room); + can = maxa >= 3u && poff < maxa && + (split ? left >= 5u : + need <= room); + if (can && split && !sa_done && + 3u * npair > left) + can = false; + if (can) { + pp = p; + if (!sa_done) { + for (kk = 0; kk < 64u; + kk++) { + if (!(sa_mask & + (1ull << + kk)) || + !sa_ent[kk]) + continue; + *pp++ = ((u32) + GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (2u << 16) | + (0x11cu >> 2); + *pp++ = kk; + *pp++ = + sa_ent[kk]; + } + left -= 3u * npair; + WRITE_ONCE( + gsp_phys_gr_mme_pack_sa, + true); + } + chunk = maxa - poff; + if (chunk > left - 2u) + chunk = left - 2u; + chunk -= chunk % 3u; + if (chunk >= 3u) { + *pp++ = ((u32) + GSP_PHYS_RMCALL_METHOD_ONEINC + << 29) | + ((chunk + 1u) << 16) | + (0x114u >> 2); + *pp++ = poff / 3u; + memcpy(pp, + &gsp_phys_gr_mme_ir_pack[poff], + (size_t)chunk * 4u); + pp += chunk; + poff += chunk; + WRITE_ONCE( + gsp_phys_gr_mme_pack_off, + poff); + } + pwords = (u32)(pp - p); + p = pp; + packed = true; + if (poff >= maxa) { + doff = pl_n; + pr_info("vgsp-gr: MME-PRELUDE pack ir_words=%u runs=%u sa=%u h=0x%x push=0x%llx\n", + maxa, nruns, + npair, + req->h_object, + req->push_va); + } else { + pr_info("vgsp-gr: MME-PRELUDE pack-part ir_words=%u/%u sa=%u split=%u h=0x%x push=0x%llx\n", + poff, maxa, + npair, + split ? 1u : 0u, + req->h_object, + req->push_va); + } + } else if (split && maxa >= 3u) { + /* + * Split mode never falls back + * to the raw drain: an + * un-emittable chunk simply + * waits for a launch with + * room (the cursor is + * untouched; the prelude + * latch stays unset so the + * calls stay held). + */ + packed = true; + } + } + } + while (!packed && doff < pl_n && + pwords + 1u <= room) { + u32 h = pl[doff]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt && + doff + 1u + cnt <= pl_n) ? + 1u + cnt : 1u; + + if (doff + span > pl_n) + span = pl_n - doff; + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + h && (op == 0u || op == 2u || op == 6u)) { + sdrop += span; + doff += span; + continue; + } + /* + * SESSION 153-CONTINUED-99: IR-only drain (see + * the sema_gl_mme_prelude_ir_only comment). + * Keep only the runs aimed at the MME SA/IR + * ports; skip everything else with the same + * run-aligned accounting as the SDM drop. + */ + if (READ_ONCE( + gsp_phys_rmcall_gr_mme_prelude_ir_only)) { + u32 ma = (h & 0x1fffu) << 2; + + if (ma != 0x114u && ma != 0x118u && + ma != 0x11cu && ma != 0x120u) { + sdrop += span; + doff += span; + continue; + } + } + if (pwords + span > room) + break; + memcpy(p, &pl[doff], (size_t)span * 4u); + p += span; + pwords += span; + doff += span; + } + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, doff); + pre_done = (doff >= pl_n); + pre_words_out = pwords; + /* seqcheck: a mid-copy append must not be recorded as + * emitted (the copy may be a torn prefix). */ + gsp_sema_mme_prelude_get(NULL, &g1); + if (pre_done && g1 == g0) { + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, g0); + WRITE_ONCE(gsp_phys_gr_mme_prelude_h, + req->h_object); + /* CONT-103: this launch carries the final + * part; its poll outcome (executed or not) + * decides whether the program is live. */ + WRITE_ONCE(gsp_phys_gr_mme_pending_gen, g0); + pr_info("vgsp-gr: MME-PRELUDE emitted gen=%u words=%u window=%u h=0x%x push=0x%llx sdm_drop=%u cap=%u\n", + g0, pl_n, win3d_n, req->h_object, + req->push_va, sdrop, block_cap); + /* + * SESSION 153-CONTINUED-70: one full dump of the + * replayed MME context-setup program per boot + * (the offline MME64 sim needs the same words the + * engine will execute). Also list the SA table + * with per-entry IR coverage so the slot-by-slot + * macro re-enable has its proof in one place. + */ + if (!gsp_phys_gr_mme_pre_full_dump) { + u32 k, sj; + u32 sa_ent[64]; + u64 sa_mask; + u32 cov_n = 0; + + gsp_phys_gr_mme_pre_full_dump = 1; + sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + for (sj = 0; sj < 64u; sj++) + if ((sa_mask & (1ull << sj)) && + gsp_sema_mme_ir_covered(sa_ent[sj])) + cov_n++; + pr_info("vgsp-gr: MME-PRE-DUMP words=%u gen=%u h=0x%x ir_end=0x%x sa_mask=0x%llx covered=%u code_words=%u keep_mask=0x%llx\n", + pl_n, g0, req->h_object, + gsp_sema_mme_ir_end_get(), + (unsigned long long)sa_mask, + cov_n, gsp_sema_mme_code_words_get(), + (unsigned long long)READ_ONCE(gsp_phys_rmcall_gr_keep_mask)); + for (k = 0; k + 4u <= pl_n; k += 4u) + pr_info("vgsp-gr: MME-PRE[%03u] %08x %08x %08x %08x\n", + k, pl[k], pl[k + 1], + pl[k + 2], pl[k + 3]); + for (sj = 0; sj < 64u; sj++) + if (sa_mask & (1ull << sj)) + pr_info("vgsp-gr: MME-PRE-SA[%u] entry=0x%x cov=%u\n", + sj, sa_ent[sj], + gsp_sema_mme_ir_covered(sa_ent[sj])); + } + } else if (pre_done) { + pr_info("vgsp-gr: MME-PRELUDE copy raced an append (gen %u->%u); will re-emit next launch\n", + g0, g1); + } else { + pr_info("vgsp-gr: MME-PRELUDE part gen=%u off=%u words=%u/%u window=%u h=0x%x push=0x%llx sdm_drop=%u cap=%u (continuation next launch)\n", + g0, doff, pwords, pl_n, win3d_n, + req->h_object, req->push_va, sdrop, + block_cap); + } + } + /* + * SESSION 153-CONTINUED-107: prelude-only LOAD carriers while + * the IRAM gate has not verified the packed program. + * + * r1372's first instruction (decisive): the calls-held pack + * carrier still poisoned (`DS-FAST ... ds a2040a00->badf1201 + * ... MME-FAST end launch events=1 (poison)` -> WEDGE-RECOVER + * wipes the context) because the block also carried the WINDOW + * - the window's own SPH state trips the DS validation and the + * FECS poison episode interrupts the 0x114/0x118 IRAM load + * mid-flight. Holding the calls is not enough; the LOAD must + * ride a carrier whose methods cannot fault, so while the + * generation is not verified live (live_gen != pl_gen) emit + * the pack/SA/release WITHOUT the window. The carrier retires, + * the poll-break gate reads the IRAM back; on a pass live_gen + * is promoted and the very next launch carries the window + * again (latch && live). Bounded by + * sema_gl_mme_iram_gate_max so a broken readback cannot starve + * the lane forever. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate) && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude_pack) && + READ_ONCE(gsp_phys_rmcall_gr_mme_hold_calls) && + pl && pl_n && pl_gen && + !gsp_phys_gr_mme_iram_gate_giveup()) { + bool latch_now = + (pl_gen == + READ_ONCE(gsp_phys_gr_mme_prelude_gen) && + req->h_object == + READ_ONCE(gsp_phys_gr_mme_prelude_h)); + bool live_now = + (READ_ONCE(gsp_phys_rmcall_gr_mme_live_skip) && + READ_ONCE(gsp_phys_gr_mme_live_gen) == pl_gen); + + if (!(latch_now && live_now)) + mme_load_only = true; + } + /* + * SESSION 153-CONTINUED-82: hold real CALL runs until the + * prelude latch for this generation+channel is set (see the + * sema_gl_mme_hold_calls comment). Evaluated AFTER the + * emission step above, so a launch that completed the drain + * in this same block has just written the latch. Real calls + * stay held when EITHER + * - the latch for this generation+channel is not valid after + * this launch (partial drain, raced append, or a fresh + * generation whose code is not fully resident), or + * - this launch uploaded generation code for the FIRST time + * (emitted words with the latch invalid before the step) - + * the r1222 evidence: the drain-completing launch bundled + * part-2's IR uploads with the window's real calls and hit + * ILLEGAL_OPCODE even though its own emission set the + * latch. A later launch (latch already valid, only the + * idempotent keep-forced re-emission of the prefix) runs + * the real calls against the now-resident program. + */ + if (pl && pl_n && pl_gen && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude) && + READ_ONCE(gsp_phys_rmcall_gr_mme_hold_calls)) { + u32 lg = READ_ONCE(gsp_phys_gr_mme_prelude_gen); + u32 lh = READ_ONCE(gsp_phys_gr_mme_prelude_h); + bool latch_after = (pl_gen == lg && + req->h_object == lh); + + { + u32 rg = READ_ONCE(gsp_phys_gr_mme_release_gen); + u32 rh = READ_ONCE(gsp_phys_gr_mme_release_h); + bool settle = READ_ONCE( + gsp_phys_rmcall_gr_mme_hold_settle); + bool strict = READ_ONCE( + gsp_phys_rmcall_gr_mme_hold_strict); + bool fe_idle_now = gsp_phys_gr_fe_idle(); + /* + * r1447: the strict FE-idle term fails open + * after sema_gl_mme_strict_grace held launches + * of this generation (0 = never fail open). + */ + bool grace_used = + (READ_ONCE( + gsp_phys_rmcall_gr_mme_strict_grace) != 0u && + READ_ONCE(gsp_phys_gr_mme_strict_holds) >= + READ_ONCE( + gsp_phys_rmcall_gr_mme_strict_grace)); + bool strict_ok = (!strict || + (pre_words_out == 0 && + READ_ONCE( + gsp_phys_gr_mme_live_gen) == + pl_gen && + (fe_idle_now || grace_used))); + + /* + * r1447: an armed window-hold for this + * generation+channel stays effective until the + * release actually fires (keyed exactly like + * the CONT-118 release statics; cleared at the + * residency resets and at release - see the + * sema_gl_mme_hold_window comment). + */ + mme_hold_window = + (READ_ONCE( + gsp_phys_gr_mme_window_hold_gen) == + pl_gen && + READ_ONCE( + gsp_phys_gr_mme_window_hold_h) == + req->h_object); + + /* + * r1445 3b: the strict variant strengthens the + * settled release - it only releases on a launch + * that emitted nothing, with the FE idle and the + * generation proven live (FETCH-WIT). strict=0 + * reduces to the exact rg/rh clause. + */ + mme_hold_calls = !latch_after || + (pre_words_out && !latch_before) || + (settle && + !(rg == pl_gen && + rh == req->h_object && + strict_ok)); + /* + * r1447: the arming launch - latch valid, no + * emission, release not yet armed - also drops + * the window (the calls are held by the settle + * clause above). The arm below writes rg/rh, so + * this set can fire at most once per generation; + * the first window execution is then the proven + * calls+window-kept shape. When the param is 0 + * this block is a no-op (legacy shape). + */ + if (settle && latch_after && !pre_words_out && + !(rg == pl_gen && rh == req->h_object) && + !stream_damaged && + READ_ONCE( + gsp_phys_rmcall_gr_mme_hold_window)) { + WARN_ON_ONCE(mme_load_only); + mme_hold_window = true; + WRITE_ONCE( + gsp_phys_gr_mme_window_hold_gen, + pl_gen); + WRITE_ONCE( + gsp_phys_gr_mme_window_hold_h, + req->h_object); + } + /* + * r1447: bounded fail-open for the strict + * release - count the launches the exact + * 0x400504 == 1 predicate held after the + * release was armed; at the grace bound the + * FE term fails open on the next launch. + */ + if (strict && settle && latch_after && + mme_hold_calls && + rg == pl_gen && rh == req->h_object && + !strict_ok) { + WRITE_ONCE( + gsp_phys_gr_mme_strict_holds, + gsp_phys_gr_mme_strict_holds + 1u); + if (READ_ONCE( + gsp_phys_rmcall_gr_mme_strict_grace) && + READ_ONCE( + gsp_phys_gr_mme_strict_holds) == + READ_ONCE( + gsp_phys_rmcall_gr_mme_strict_grace)) + pr_warn("vgsp-gr: STRICT-GRACE gen=%u n=%u h=%#x 0x400504=%08x (FE never read idle; releasing on the next launch) push=0x%llx\n", + pl_gen, + READ_ONCE( + gsp_phys_gr_mme_strict_holds), + req->h_object, + gsp_phys_gr_fe_status(), + req->push_va); + } + /* + * CONT-118: a launch that carries no prelude + * words but sees a valid latch is the first + * settled launch; arm the release so the NEXT + * launch runs the real calls. The completing + * carrier (pre_words_out > 0) must not arm it + * (the r1222 clause already holds it). + */ + if (settle && latch_after && !pre_words_out) { + WRITE_ONCE( + gsp_phys_gr_mme_release_gen, + pl_gen); + WRITE_ONCE( + gsp_phys_gr_mme_release_h, + req->h_object); + } + } + /* + * CONT-104: also hold while the front-end is not + * settled (the r1359 MME ILLEGAL_OPCODE race; see + * gsp_phys_gr_fe_idle). The held launch retires and + * a later window retries once 0x400504 reads idle. + * SESSION 153-CONTINUED-107: separately knob-gated - + * with the IRAM gate proving residency, the residual + * value of this clause is only the r1359 race; r1373 + * left the FE at 0x07071602 for minutes and starved + * every calls-kept window, so allow an explicit A/B + * (sema_gl_mme_hold_fe_idle=0). + */ + if (!mme_hold_calls && + READ_ONCE(gsp_phys_rmcall_gr_mme_hold_fe_idle) && + !gsp_phys_gr_fe_idle()) { + mme_hold_calls = true; + mme_hold_fe_busy = true; + } + } + /* + * r1487 STREAM-DAMAGE: an earlier chunk of this same push was + * dropped (LOAD-ONLY carrier or the arming window hold), so + * the channel no longer holds the state that chunk carried. + * Run this chunk's CALL_MME_MACRO/DATA runs held (elided) - + * the same proven-safe shape as the prelude-hold path - so no + * macro can emit LOAD_CONSTANT_BUFFER against the incomplete + * state and class-error into the FECS priv-error poison. The + * window's non-CALL state methods still execute, and the next + * client section (stream_rel 0) clears the record. + */ + if (stream_damaged && !mme_load_only) { + if (!mme_hold_calls && gsp_phys_gr_stream_damage_hits < 32u) { + gsp_phys_gr_stream_damage_hits++; + pr_info("vgsp-gr: STREAM-DAMAGE h=0x%x push=0x%llx rel=%u window=%u (calls held: earlier chunk dropped)\n", + req->h_object, req->push_va, + req->stream_rel, win3d_n); + } + mme_hold_calls = true; + } + /* + * SESSION 153-CONTINUED-107: IRAM readback gate (see the + * gsp_phys_gr_mme_iram_gate comment). The calls would run this + * launch; verify the live IRAM actually holds the packed image + * at the SA entries first. On a mismatch (the CONT-106 + * all-zero fetch) hold the calls and owe a full re-emission - + * the r1368 evidence says the real CALL runs against a + * non-resident program poison the lane (ILLEGAL_OPCODE at the + * macro entry) instead of drawing. Only meaningful on the + * packed path (the pack image is the comparator). + */ + if (!mme_hold_calls && pl && pl_n && pl_gen && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude) && + READ_ONCE(gsp_phys_rmcall_gr_mme_hold_calls) && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude_pack) && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate) && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate_build)) { + void __iomem *gbar0 = gsp_phys_gpu_bar0(); + + if (gbar0 && + !gsp_phys_gr_mme_iram_gate(gbar0, req->h_object, + pl_gen, "build")) { + mme_hold_calls = true; + gsp_phys_gr_mme_iram_gate_fail(); + } + } + /* + * SESSION 153-CONTINUED-63: emit the SA-RAM completion computed + * above - a one-instruction end-noop stub in the IR plus one + * ONE_INC 0x11c run loading [slot, stub] for every called slot + * the capture's SA table does not cover (see the sa_stub + * comment at the emit decision). Emitted with the prelude (same + * generation latch), so it lands once per capture generation + * and the context keeps it for every later window. + */ + if (emit_prelude && sa_block) { + u32 t, stub_pc = gsp_sema_mme_ir_end_get(); + + /* + * SESSION 153-CONTINUED-68: the stub PC must be past the + * replayed program's last instruction - the window also + * calls COVERED slots, and a fixed PC inside a captured + * macro body would clobber its code. Fall back to 0xC0 + * when the capture reports nothing yet. + */ + if (stub_pc < GSP_PHYS_RMCALL_GR_MME_STUB_PC) + stub_pc = GSP_PHYS_RMCALL_GR_MME_STUB_PC; + /* + * SESSION 153-CONTINUED-69: explicit stub-PC override + * (sema_gl_mme_stub_pc). The ir_end-derived PC sits at + * the first instruction past the replayed program; if + * the MME's IR allocation/ONEINC pointer semantics + * differ (instruction vs word units, allocation + * boundary), the stub body may land where the engine + * does not look. This knob moves the stub (e.g. to + * 0xf0/0x100, or back to the 0xC0 floor) so the + * landing can be A/B-tested without a rebuild. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_stub_pc)) + stub_pc = READ_ONCE(gsp_phys_rmcall_gr_mme_stub_pc); + /* + * The IR upload MUST use the client's own SEC_OP (5, + * ONE_INC): with op=1 the method auto-increment walked + * the data words onto 0x11c/0x120 and clobbered SA[0] + * while writing no usable stub code (r1131). count = 7 + * = 1 pointer + 6 code words (two 3-word NOPs; see the + * gsp_phys_gr_mme_stub_code comment). + */ + *p++ = (GSP_PHYS_RMCALL_METHOD_ONEINC << 29) | + (7u << 16) | (0x114u >> 2); + *p++ = stub_pc; + *p++ = gsp_phys_gr_mme_stub_code[0]; + *p++ = gsp_phys_gr_mme_stub_code[1]; + *p++ = gsp_phys_gr_mme_stub_code[2]; + *p++ = gsp_phys_gr_mme_stub_code[3]; + *p++ = gsp_phys_gr_mme_stub_code[4]; + *p++ = gsp_phys_gr_mme_stub_code[5]; + /* + * Each [slot, stub] pair is its own INC 0x11c run: the + * pair methods are re-written per slot (a single longer + * run would walk onto 0x124 SetMmeShadowRamControl and + * beyond). + */ + for (t = 0; t < sa_nstub; t++) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (2u << 16) | (0x11cu >> 2); + *p++ = sa_stub[t]; + *p++ = stub_pc; + } + pr_info("vgsp-gr: MME-SA-STUB slots=%u words=%u pc=0x%x h=0x%x push=0x%llx\n", + sa_nstub, sa_block, stub_pc, req->h_object, + req->push_va); + { + char sb[192]; + u32 k, o = 0; + + sb[0] = '\0'; + for (k = 0; k < sa_nstub && o + 8u < sizeof(sb); k++) + o += scnprintf(sb + o, sizeof(sb) - o, + "%s%u:%x", k ? "," : "", + sa_stub[k], + sa_stub_ent[k]); + pr_info("vgsp-gr: MME-SA-STUB list=[%s]\n", sb); + } + { + char sb[192]; + u32 k, o = 0; + + sb[0] = '\0'; + for (k = 0; k < sa_ncov && o + 8u < sizeof(sb); k++) + o += scnprintf(sb + o, sizeof(sb) - o, + "%s%u:%x", k ? "," : "", + sa_cov[k], + sa_cov_ent[k]); + pr_info("vgsp-gr: MME-SA-COVERED list=[%s]\n", + sb); + } + } + /* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude (see the + * sema_gl_attr_prelude knob above). Emit the patched captured + * runs after the MME prelude and ahead of the window, + * run-aligned, draining across launches when the remaining + * block budget cannot take the whole remainder. The cursor + * resets when a new capture generation is patched. A window + * with no room at all defers the whole step to the next launch + * (logged by the patch site). + */ + if (attr_pre && attr_pre_n && + READ_ONCE(gsp_phys_rmcall_gr_attr_prelude)) { + u32 avail = block_cap - + (u32)(p - pb) - (win3d_n + 2u + 8u); + u32 off, copied = 0; + + /* + * SESSION 153-CONTINUED-79: the attr prelude shares + * the same per-launch block cap (see + * sema_gl_mme_block_cap); unsigned underflow would + * hand it the whole module buffer on an oversized + * window. + */ + if ((u32)(p - pb) + win3d_n + 2u + 8u >= block_cap) + avail = 0; + + if (gsp_phys_gr_attr_emit_gen != + READ_ONCE(gsp_phys_gr_attr_pre_gen)) { + WRITE_ONCE(gsp_phys_gr_attr_emit_gen, + READ_ONCE(gsp_phys_gr_attr_pre_gen)); + WRITE_ONCE(gsp_phys_gr_attr_emit_off, 0); + } + off = READ_ONCE(gsp_phys_gr_attr_emit_off); + while (off < attr_pre_n) { + u32 hdr = attr_pre[off]; + u32 op = (hdr >> 29) & 7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt) ? + 1u + cnt : 1u; + + if (off + span > attr_pre_n) + span = attr_pre_n - off; + /* + * SESSION 153-CONTINUED-59: the same SDM + * sanitize as the MME prelude (see + * gsp_phys_rmcall_gr_sdm_drop); the cursor + * still advances so the drop is permanent. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + hdr && (op == 0u || op == 2u || + op == 6u)) { + off += span; + continue; + } + /* + * CONT-120 (r1451): drop the attr-prelude + * SET_PIPELINE_SHADER(2) run(s) - the guest's + * cold `0x2080 <- 0x20` word is refused by the + * FE with Class Error EC4 when its one-shot + * attr emission rides a windowless LOAD-ONLY + * carrier (no SPH-BIND enable ahead of it; see + * sema_gl_attr_sph_drop). Cursor advances like + * the SDM drop: no raw fallback. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_attr_sph_drop) && + (op == 1u || op == 3u || op == 5u) && cnt && + ((hdr & 0xfffu) << 2) == 0x2080u) { + gsp_phys_gr_attr_sph_drop_count++; + if (gsp_phys_gr_attr_sph_drop_gen != + READ_ONCE(gsp_phys_gr_attr_pre_gen)) { + gsp_phys_gr_attr_sph_drop_gen = + READ_ONCE(gsp_phys_gr_attr_pre_gen); + pr_info("vgsp-gr: ATTR-PRELUDE dropped SET_PIPELINE gen=%u mthd=0x%x drops=%u\n", + gsp_phys_gr_attr_sph_drop_gen, + (u32)((hdr & 0xfffu) << 2), + gsp_phys_gr_attr_sph_drop_count); + } + off += span; + continue; + } + if (copied + span > avail) + break; + memcpy(&p[copied], &attr_pre[off], + (size_t)span * 4u); + copied += span; + off += span; + } + if (copied) { + p += copied; + WRITE_ONCE(gsp_phys_gr_attr_emit_off, off); + pr_info("vgsp-gr: ATTR-PRELUDE emitted gen=%u words=%u/%u window=%u h=0x%x push=0x%llx%s\n", + READ_ONCE(gsp_phys_gr_attr_pre_gen), + off, attr_pre_n, win3d_n, + req->h_object, req->push_va, + off < attr_pre_n ? + " (continuation next launch)" : ""); + } else if (off < attr_pre_n) { + static u32 attr_defer_log; + + if (attr_defer_log < 8) { + attr_defer_log++; + pr_info("vgsp-gr: ATTR-PRELUDE deferred (no room: window=%u avail=%u) h=0x%x\n", + win3d_n, avail, + req->h_object); + } + } + } + /* + * SESSION 153-CONTINUED-84: shadow-RAM mode before the window + * (see the sema_gl_mme_track comment). METHOD_TRACK (0) is + * emitted by default so the repeated CALL/DATA parameter + * writes are not swallowed by the redundant-method filter. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_track) >= 0 && + p + 2u <= pb + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + *p++ = ((u32)GSP_PHYS_RMCALL_METHOD_IMM << 29) | + (1u << 16) | (0x124u >> 2); + *p++ = (u32)READ_ONCE(gsp_phys_rmcall_gr_mme_track); + } + /* + * SESSION 153-CONTINUED-84: SA re-arm BEFORE the window. + * The r1236 evidence: this block was emitted AFTER the + * window copy (contradicting its own "before the window" + * log), so the window's CALL_MME_MACRO(j) runs executed + * against the SA table state left by the prelude - whose + * last captured SA write re-clobbers the slot (r1231: + * SA[34]=0 lands last, every draw call resolved to macro + * 0 and emitted nothing). Because the prelude is + * re-emitted on every launch while any slot is kept, the + * late re-arm was re-clobbered before the next window + * anyway; the re-point only ever held between a window + * and the next prelude re-emission. Emitting the re-arm + * ahead of the window makes every launch's calls resolve + * to the captured SA entry. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_sa_rearm)) { + u32 sa_ent[64]; + u64 sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + u32 r = 0, rearm = 0; + + while (r < win3d_n && rearm < 32u) { + u32 h = win3d[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = ((op == 1u || op == 3u || + op == 5u) && cnt && + r + 1u + cnt <= win3d_n) ? + 1u + cnt : 1u; + u32 mb = (h & 0xfffu) << 2; + + if (mb >= 0x3800u && mb <= 0x39fcu) { + u32 j = (mb - 0x3800u) / 8u; + + if (j < 64u && + (sa_mask & (1ull << j)) && + sa_ent[j]) { + *p++ = ((u32) + GSP_PHYS_RMCALL_METHOD_IMM + << 29) | + (2u << 16) | + (0x11cu >> 2); + *p++ = j; + *p++ = sa_ent[j]; + rearm++; + } + } + r += span; + } + if (rearm) + pr_info("vgsp-gr: MME-SA-REARM %u slot(s) re-pointed before the window (h=0x%x push=0x%llx)\n", + rearm, req->h_object, req->push_va); + } + /* + * r1447: the release actually fires on this launch (the calls + * are kept) - the arming window-hold is consumed here, so the + * release launch carries the CALL runs and the window together + * (the r1443b shape); the strict grace budget restarts with the + * next generation cycle. + */ + if (mme_hold_window && !mme_load_only && + (!mme_hold_calls || stream_damaged)) { + WRITE_ONCE(gsp_phys_gr_mme_window_hold_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_window_hold_h, 0); + WRITE_ONCE(gsp_phys_gr_mme_strict_holds, 0); + mme_hold_window = false; + } + /* + * r1453 PAIR-CHAIN: the window was already drained in the + * submit path (after the SPH-BIND/SURF-PRELUDE prepends, + * once per launch - see gsp_phys_gr_pair_chain_fix). The + * emitted-pair proof is recomputed from the final branch + * output below. + */ + win_emit_off = (u32)(p - pb); + if (mme_load_only || mme_hold_window) { + /* + * r1487 STREAM-DAMAGE: this window slice never reaches + * the engine. Record the push/offset so the remaining + * chunks of the same client section hold their CALL + * runs instead of executing a macro against the lost + * state (see the damage table comment). + */ + gsp_phys_gr_stream_damage_note(req->push_va, + req->stream_rel); + if (gsp_phys_gr_stream_damage_notes < 32u) { + gsp_phys_gr_stream_damage_notes++; + pr_info("vgsp-gr: STREAM-DAMAGE-NOTE h=0x%x push=0x%llx rel=%u window=%u (%s)\n", + req->h_object, req->push_va, + req->stream_rel, win3d_n, + mme_hold_window ? "window-hold" : + "load-only"); + } + /* + * r1465 PAIR-ORPHAN-DROP: the FE consumed none of + * this window, so the pair projection is a no-op: + * restore the launch-entry snapshot the fixer took + * before the walk (MME-WINDOW-HOLD and MME-LOAD-ONLY + * both land here; the held-call branch still emits + * the window, so its pairs are FE-visible). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_pair_orphan_drop)) { + WRITE_ONCE(gsp_phys_gr_mme_pair_open, + READ_ONCE(gsp_phys_gr_mme_pair_open_entry)); + WRITE_ONCE(gsp_phys_gr_mme_pair_open_split, + READ_ONCE(gsp_phys_gr_mme_pair_open_entry_split)); + } + if (mme_hold_window && !mme_load_only) { + pr_info_ratelimited("vgsp-gr: MME-WINDOW-HOLD gen=%u h=%#x window=%u rel=%u dropped (r1447 arming hold) push=0x%llx\n", + pl_gen, req->h_object, win3d_n, + req->stream_rel, req->push_va); + } else { + /* + * SESSION 153-CONTINUED-107: no window on a + * LOAD carrier (see the mme_load_only + * comment). The block is SET_OBJECT + + * prelude/SA/pack + release only; the engine + * can retire it without any DS/shader state + * to validate, so the IRAM load cannot be + * interrupted by the SPH poison. + */ + static u32 load_only_log; + + load_only_log++; + if (load_only_log <= 24u) + pr_info("vgsp-gr: MME-LOAD-ONLY carrier h=0x%x gen=%u window=%u rel=%u dropped until the IRAM gate verifies (log %u) push=0x%llx\n", + req->h_object, pl_gen, + win3d_n, req->stream_rel, + load_only_log, + req->push_va); + } + } else if (mme_hold_calls) { + /* + * SESSION 153-CONTINUED-82: the prelude latch for + * this generation+channel is not set - elide every + * remaining CALL_MME_MACRO/DATA run of the window + * (the proven-safe drop shape; see the + * sema_gl_mme_hold_calls comment). Runs touching + * the MME macro method space 0x3800..0x39fc are the + * ones that execute a macro; the drop decision is + * the same walk gsp_phys_gr_mme_drop_calls uses. + * + * r1453 PAIR-CHAIN: the held window is the 0x2058 + * "other writer" (r1453 strategy §3.2) - it copies + * everything except CALL runs, including the + * program-address pair runs, and the r1452b #33 + * launch then traps 0x2058 at Data 0x19b0900 / + * EC 0x0d. Extend the sema_gl_mme_nop_pair elision + * here ONLY once a pair-bearing window of this + * same generation has retired (pairs_gen == pl_gen, + * set on the submit success edge): eliding before + * that starves the binds (r1452 D1, HUBCLIENT_PD + * @ 0x0). The elision still requires the nop_pair + * opt-in, so the default held shape is unchanged. + */ + u32 r = 0, w = 0, held = 0, pelided = 0; + bool pair_installed = + READ_ONCE(gsp_phys_rmcall_gr_mme_nop_pair) && + READ_ONCE(gsp_phys_rmcall_gr_pair_chain) && + pl_gen && READ_ONCE(gsp_phys_gr_mme_pairs_gen) == + pl_gen; + + while (r < win3d_n) { + u32 h = win3d[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u; + bool iscall = false, ispair = false; + u32 k, m0, mend; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && r + 1u + cnt <= win3d_n) + span = 1u + cnt; + m0 = (h & 0xfffu) << 2; + mend = m0 + 4u * (span - 1u); + if (pair_installed && + (op == 0u || op == 1u || op == 3u || + op == 4u || op == 5u) && + m0 <= 0x2158u && mend >= 0x2054u) + ispair = true; + for (k = 0; k < span && !ispair; k++) { + u32 m = ((h & 0xfffu) << 2) + + 4u * k; + + if (k && op != 1u && op != 3u) + break; + if (m >= 0x3800u && m <= 0x39fcu) { + iscall = true; + break; + } + } + if (iscall || ispair) { + if (iscall) + held += span; + if (ispair) + pelided += span; + r += span; + continue; + } + memcpy(&p[w], &win3d[r], (size_t)span * 4u); + w += span; + r += span; + } + if (held) + pr_info("vgsp-gr: MME-CALL-HOLD %u word(s) of real CALL runs held (prelude gen=%u not latched for h=0x%x; window=%u->%u) push=0x%llx%s\n", + held, pl_gen, req->h_object, + win3d_n, w, req->push_va, + mme_hold_fe_busy ? + " [FE-not-idle (CONT-104)]" : ""); + if (pelided) + pr_info_ratelimited("vgsp-gr: PAIR-CHAIN held-elide %u word(s) of program-pair runs dropped (pairs installed gen=%u h=0x%x; 0x2058 other-writer) push=0x%llx\n", + pelided, pl_gen, req->h_object, + req->push_va); + p += w; + } else if (READ_ONCE(gsp_phys_rmcall_gr_mme_nop_pair)) { + /* + * CONT-103: the calls-kept launch now starts (the + * prelude-live skip shrank it under the ~2050-word + * wall) and the engine decodes the window, but the + * FE then TRAPS on the relocated program-address + * pair write (trap=0x80002058 data=<shadow VA>, + * CONT-101: the relocated 0x2054-pair operand) and + * parks the front-end (grfifo 00000001). The binds + * persist on the channel from the earlier + * calls-held launches of the same capture, so + * eliding the pair re-binds removes the trapping + * write while keeping the program bound. Drops any + * method run overlapping the program-pair space + * [0x2054, 0x2158] (0x2054/0x2058, 0x2094/0x2098, + * 0x2154/0x2158 - the client window's own pairs and + * the synthesized SPH binds alike; this walk runs on + * the final window, prepends included). + */ + u32 r = 0, w = 0, elided = 0; + + while (r < win3d_n) { + u32 h = win3d[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u, m0, mend; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && r + 1u + cnt <= win3d_n) + span = 1u + cnt; + m0 = (h & 0xfffu) << 2; + mend = m0 + 4u * (span - 1u); + if ((op == 0u || op == 1u || op == 3u || + op == 4u || op == 5u) && + m0 <= 0x2158u && mend >= 0x2054u) { + elided += span; + r += span; + continue; + } + memcpy(&p[w], &win3d[r], (size_t)span * 4u); + w += span; + r += span; + } + if (elided) + pr_info("vgsp-gr: MME-PAIR-DROP %u word(s) of program-pair runs elided (calls-kept launch; binds persist from the held-call launches) h=0x%x push=0x%llx window=%u->%u\n", + elided, req->h_object, req->push_va, + win3d_n, w); + p += w; + } else { + memcpy(p, win3d, (size_t)win3d_n * 4); + p += win3d_n; + } + /* + * r1454 PAIR-CHAIN-ENTRY-ELIDE: a window that ENTERS with a + * BEGIN..END pair already open (carried from the previous + * emitted window) must not re-write the program-address pair + * runs - the FE refuses the first state method with EC0x0d + * BEGIN_END_ACTIVE (r1453b #11: 0x2058 <- shadow VA at words + * 18-20, before the orphan END that closes the carried pair). + * Re-walk the FINAL emitted window with the same run math as + * the nop_pair branch and compact away every run overlapping + * [0x2054, 0x2158]. Fail closed: requires the pair_chain + * knob, the entry-open snapshot, and the non-zero pairs_gen + * retire proof (set only when a pair-bearing window retires), + * so launch #1 still writes its pairs (prev_open=0) and D1 + * stays structurally dead. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_pair_chain) && + READ_ONCE(gsp_phys_gr_mme_pair_open_entry) && + READ_ONCE(gsp_phys_gr_mme_pairs_gen) != 0u && + (u32)(p - pb) > win_emit_off) { + u32 *ww = &pb[win_emit_off]; + u32 wn = (u32)(p - ww); + u32 r = 0, w = 0, elided = 0; + + while (r < wn) { + u32 h = ww[r]; + u32 op = (h >> 29) & 7u; + u32 cnt = (h >> 16) & 0xfffu; + u32 span = 1u, m0, mend; + + if ((op == 1u || op == 3u || op == 5u) && + cnt && r + 1u + cnt <= wn) + span = 1u + cnt; + m0 = (h & 0xfffu) << 2; + mend = m0 + 4u * (span - 1u); + if ((op == 0u || op == 1u || op == 3u || + op == 4u || op == 5u) && + m0 <= 0x2158u && mend >= 0x2054u) { + elided += span; + r += span; + continue; + } + if (w != r) + memmove(&ww[w], &ww[r], + (size_t)span * 4u); + w += span; + r += span; + } + if (elided) { + gsp_phys_gr_mme_pair_entry_elide_n++; + p -= elided; + pr_info_ratelimited("vgsp-gr: PAIR-CHAIN-ENTRY-ELIDE gen=%u n=%u total=%u (pair open at launch start; window=%u->%u)\n", + READ_ONCE(gsp_phys_gr_mme_pairs_gen), + elided, + gsp_phys_gr_mme_pair_entry_elide_n, + wn, w); + } + } + /* + * r1453 PAIR-CHAIN: the install proof for the held-call + * elision extension. Recomputed from the FINAL emitted window + * (after the CALL/pair elision branches above), so a launch + * that dropped every pair never marks its generation as + * pair-installed. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_pair_chain) && + (u32)(p - pb) > win_emit_off) + gsp_phys_gr_mme_pairs_emitted = + gsp_phys_gr_pair_present(&pb[win_emit_off], + (u32)(p - pb) - win_emit_off); + /* + * SESSION 153-CONTINUED-84: the SA re-arm block used to be + * emitted HERE (after the window). It was moved ahead of the + * window above; see that block's comment for the r1236 + * evidence. An SA[slot]=entry re-point behind the call that + * needs it cannot help that call, and the per-launch + * prelude re-emission re-clobbers it before the next window. + */ + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_A, + (u32)(sem_va >> 32)); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_B, + (u32)sem_va); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_C, + sem_payload); + p = gsp_phys_rmcall_push_gr(p, + GSP_PHYS_RMCALL_GR_SET_REPORT_SEMAPHORE_D, + GSP_PHYS_RMCALL_GR_SEM_RELEASE_D); + { + static u32 gr3d_log; + + if (gr3d_log < 16) { + gr3d_log++; + pr_info("vgsp-gr: 3D STREAM replay h=0x%x push=0x%llx class=0xc597 window=%u words=%u verbatim (no prelude/QMD)\n", + req->h_object, req->push_va, + win3d_n, (u32)(p - pb)); + } + /* + * SESSION 153-CONTINUED-45 diag: the exact block the + * PBDMA fetches (first/last window words). The beef + * family's replay schedule-stalls (grfifo frozen) while + * the X family's retires; this dump is the content A/B + * behind that divergence. Bounded per boot. + */ + if (gr3d_log < 16) { + u32 wi; + + pr_info("vgsp-gr: 3DSTREAM hdr=%08x %08x win0..9=", + pb[0], pb[1]); + for (wi = 0; wi < 10 && wi < win3d_n; wi++) + pr_cont("%08x ", win3d[wi]); + pr_cont("| tail="); + for (wi = (win3d_n > 10 ? win3d_n - 10 : 0); + wi < win3d_n; wi++) + pr_cont("%08x ", win3d[wi]); + pr_cont("\n"); + } + } + return (u32)(p - pb); + } + + /* * r425 A/B: with gsp_phys_rmcall_gr_init_once the compute init preamble * is emitted only on the first forwarded launch (the stock driver's * per-channel shape); subsequent launches start at the inline-QMD @@ -16659,6 +23227,25 @@ static u32 gsp_phys_rmcall_build_gr_launch(u32 *pb, GSP_PHYS_RMCALL_GR_SET_QMD_VIRT_CONTROL, GSP_PHYS_RMCALL_GR_QMD_VIRT_IQ2M_SEND_PCAS); /* + * r597: the prelude's acquire groups (if any) go before the QMD run + * so the engine cannot read the launch's data pages until the copy + * channel's semaphore has advanced. + */ + { + u32 w = gsp_phys_rmcall_gr_emit_acquires(p, req, + 128u); + static u32 acq_log; + + if (w) { + p += w; + if (acq_log < 16) { + acq_log++; + pr_info("vgsp-gr: prelude acquire groups emitted words=%u h=0x%x push=0x%llx\n", + w, req->h_object, req->push_va); + } + } + } + /* * Verbatim replay: copy the captured GRCE prelude and the guest's * own inline-QMD run, keeping the method headers, and relocate only * the ADDRESS_A/B data words and the 64 QMD words exactly like the @@ -16833,7 +23420,7 @@ static int gsp_phys_rmcall_gr_map_va_locked(void __iomem *bar0, u64 guest_va, pt_va = gsp_phys_chunk_page_va(pt, 0); if (!pt_va) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt_va + (size_t)pte_idx * 8); dma_wmb(); boot_stage.rmcall_gr_win[slot].pte[i].idx = pte_idx; @@ -16843,6 +23430,58 @@ static int gsp_phys_rmcall_gr_map_va_locked(void __iomem *bar0, u64 guest_va, return 0; } +/* + * r1497: install a VID-aperture PTE at the guest VA so the engine reads + * the real FBMEM page (fb_off) instead of an empty shadow page. Same + * never-rewrite discipline; the cache key is tagged so a VID entry can + * never be mistaken for a host-PA alias. + */ +static int gsp_phys_rmcall_gr_map_va_vid_locked(void __iomem *bar0, + u64 guest_va, u64 fb_off, + u32 epoch) +{ + u64 page = guest_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u32 pde_index = (u32)(page >> 21); + u32 pte_idx = (u32)((page >> 12) & 0x1ffu); + u32 line = pte_idx / GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES; + u64 key = (fb_off & ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) | + (1ULL << 63); + struct gsp_phys_boot_chunk *pt; + u8 *pt_va; + u32 slot, i; + int ret; + + ret = gsp_phys_rmcall_gr_win_slot_locked(bar0, pde_index, &slot); + if (ret) + return ret; + for (i = 0; i < boot_stage.rmcall_gr_win[slot].pte_count; i++) { + if (boot_stage.rmcall_gr_win[slot].pte[i].idx == pte_idx) { + if (boot_stage.rmcall_gr_win[slot].pte[i].hpa == key) + return 0; + return -EALREADY; + } + if (boot_stage.rmcall_gr_win[slot].pte[i].epoch != epoch && + boot_stage.rmcall_gr_win[slot].pte[i].idx / + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES == line) + return -EBUSY; + } + if (boot_stage.rmcall_gr_win[slot].pte_count >= + GSP_PHYS_RMCALL_GR_PT_ENTRIES) + return -ENOSPC; + pt = &boot_stage.rmcall_gr_pt[boot_stage.rmcall_gr_win[slot].pt_slot]; + pt_va = gsp_phys_chunk_page_va(pt, 0); + if (!pt_va) + return -EIO; + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf_vid(fb_off), + pt_va + (size_t)pte_idx * 8); + dma_wmb(); + boot_stage.rmcall_gr_win[slot].pte[i].idx = pte_idx; + boot_stage.rmcall_gr_win[slot].pte[i].epoch = epoch; + boot_stage.rmcall_gr_win[slot].pte[i].hpa = key; + boot_stage.rmcall_gr_win[slot].pte_count++; + return 0; +} + static void __iomem *gsp_phys_gr_scratch_bar0; /* @@ -17001,7 +23640,7 @@ static int gsp_phys_rmcall_gr_map_shadow_locked(u32 slot, u64 hpa, pt = gsp_phys_rmcall_guest_pt_page_va(page); if (!pt) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page) * 8); dma_wmb(); @@ -17048,7 +23687,7 @@ static int gsp_phys_rmcall_gr_map_shadow_contig_locked(u32 line, u32 pte, pt = gsp_phys_rmcall_guest_pt_page_va(page); if (!pt) return -EIO; - put_unaligned_le64(gsp_phys_rmcall_pte(hpa), + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa), pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * line_in_page + pte) * 8); dma_wmb(); @@ -17389,6 +24028,54 @@ static struct gsp_phys_boot_chunk *gsp_phys_gr_data_chunk(void) } /* + * r452: scan the whole active GR data-bounce chunk for an 8-byte pattern + * (write "0x<pattern>" to /sys/module/vgsp/parameters/gsp_phys_gr_scan). + * Verifies by content where a lost engine store actually landed: the chunk + * is the only bulk memory the operand window PTEs map, so a store that + * executed but is absent from the writer's tracked pages must be findable + * here (a stale-PTE landing) or nowhere (a dropped store). + */ +static int gsp_phys_gr_scan_set(const char *val, + const struct kernel_param *kp) +{ + struct gsp_phys_boot_chunk *c = gsp_phys_gr_data_chunk(); + u64 pattern = 0; + u32 i, hits = 0; + + if (kstrtoull(val, 0, &pattern)) + return -EINVAL; + if (!c->pages) + return -ENODEV; + for (i = 0; i < c->pages; i++) { + const u8 *p = gsp_phys_chunk_page_va(c, i); + u32 off; + + if (!p) + continue; + for (off = 0; off + 8 <= GSP_PHYS_PAGE_SIZE; off += 8) { + if (get_unaligned_le64(p + off) == pattern) { + if (hits < 32) + pr_info("vgsp-gr: CHUNK-SCAN found 0x%llx at bi=%u off=0x%x (va=0x%llx)\n", + pattern, i, off, + GSP_PHYS_RMCALL_GR_DATA_BOUNCE_VA0 + + ((u64)i << 12) + off); + hits++; + } + } + } + pr_info("vgsp-gr: CHUNK-SCAN 0x%llx done: %u hit(s) over %u pages\n", + pattern, hits, c->pages); + return 0; +} + +static const struct kernel_param_ops gsp_phys_gr_scan_ops = { + .set = gsp_phys_gr_scan_set, +}; +module_param_cb(gsp_phys_gr_scan, &gsp_phys_gr_scan_ops, NULL, 0200); +MODULE_PARM_DESC(gsp_phys_gr_scan, + "Scan the GR data-bounce chunk for an 8-byte pattern (hex)"); + +/* * B-2: page index of a relocated operand VA inside the engine-visible data * region, replacing the five inlined (reloc_va - 0x110000) >> 12 derivations * and their MAIL_SLOT bounds. Grown region: VAs from @@ -17434,6 +24121,45 @@ static u32 gsp_phys_gr_data_bi(u64 reloc_va) * gsp-phys-work item is the only caller, reset per submission exactly like * bounce_csum_in. */ +/* + * r564: minimum data records for a forwardable launch. Healthy vadd-class + * launches resolve 968-969 records (the client's whole arena); a launch + * with only a handful after the fill's drops has an unresolved argument + * set and must not be handed to the engine (see the submit guard). + * + * SESSION 127: the Vulkan inline-QMD launch (the C5C0 translator) is a + * compute dispatch whose operand set is one descriptor page and one + * parameter bank - a handful of records is healthy for it. Tunable module + * parameter (default 16 preserves the CUDA-era threshold); the translator + * cycles pass gsp_phys_gr_min_data=2. + */ +#define GSP_PHYS_GR_MIN_DATA_RECS 16u +static unsigned int gsp_phys_gr_min_data_param = GSP_PHYS_GR_MIN_DATA_RECS; +module_param_named(gsp_phys_gr_min_data, gsp_phys_gr_min_data_param, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_min_data, + "GR submit minimum data records (degenerate-set refusal threshold, 0..4096; 0 = explicit bypass for the no-external-data launch class; default 16)"); + +/* + * r801 (SESSION 143): 0 is an explicit BYPASS, not a clamp to 1. The + * Vulkan launch's single QMD pointer pair names its own QMD window and its + * params are scalars, so the launch has no external data by construction; + * the bare n_data>=min gate wrongly classifies it degenerate. The bypass is + * safe now that gsp_phys_chunk_page_va answers NULL for an unallocated + * chunk (r800): before that guard the empty-set post-mortem print + * dereferenced a garbage pointer and oopsed at + * gsp_phys_backend_guest_gr_submit+0x4394. The proper predicate (all + * candidates launch-internal-filtered, none unresolved => not degenerate) + * replaces the bare threshold next; until then 0 stays an explicit, + * opt-in debug/test value and the default remains 16. + */ +static unsigned int gsp_phys_gr_min_data_value(void) +{ + unsigned int v = READ_ONCE(gsp_phys_gr_min_data_param); + + if (v > 4096) + v = 4096; + return v; +} #define GSP_PHYS_GR_GAP_MAX 4096u /* total late pages per launch */ /* * relfix PART 3 (r217 span containment): the late path authors ONLY the @@ -17838,7 +24564,7 @@ static bool gsp_phys_gr_reloc_fix_word(const struct gsp_phys_gr_submit *req, */ static noinline u32 gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, - const u32 *csum_in, u32 *n_tracked, u64 *hash_out) + const u64 *csum_in, u32 *n_tracked, u64 *hash_out) { static u32 checked[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); @@ -17884,15 +24610,24 @@ gsp_phys_gr_scan_pages(const struct gsp_phys_gr_submit *req, *n_tracked = n_all; for (j = 0; j < n_all; j++) { const u8 *src = gsp_phys_chunk_page_va(dchunk, checked[j]); - u32 ci, csum = 0; + u64 fp; if (!src) continue; - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum != csum_in[checked[j]]) + /* + * r452: FNV-1a per page, not a 32-bit additive sum. The + * add-sum was PROVABLY blind to a power-of-two in-place + * scale: doubling every word of a 4 KB page shifts each + * exponent by +1, i.e. +2^23 per word, and 1024 words give + * 2^33 == 0 (mod 2^32) - so c[i] *= 2.0f left the checksum + * bit-identical, the page classified "unchanged", and the + * client read the pre-scale values (the deterministic + * streams2/scale class). FNV-1a has no such structure. + */ + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp != csum_in[checked[j]]) changed++; - hash ^= csum; + hash ^= fp; hash *= 1099511628211ULL; } if (hash_out) @@ -17928,7 +24663,7 @@ static void gsp_phys_gr_qmd64_post(const u8 *deposit, const char *tag) static noinline u32 gsp_phys_gr_output_retire_wait(const struct gsp_phys_gr_submit *req, - const u32 *csum_in, u32 *n_tracked) + const u64 *csum_in, u32 *n_tracked) { u32 n_all = 0, iter; u32 changed = 0; @@ -18082,7 +24817,7 @@ static u32 gsp_phys_gr_idem_n; static u64 gsp_phys_gr_launch_fp(const u32 *qmd, const struct gsp_phys_gr_submit *req, - const u32 *csum_in) + const u64 *csum_in) { u64 h = 1469598103934665603ULL; u32 i; @@ -18160,6 +24895,22 @@ static void gsp_phys_gr_page_spot_check(const u8 *dst, const u8 *src, } /* + * r452: the legacy 32-bit additive page sum, kept ONLY for the all-zero + * filter. It is never a change-detection fingerprint any more: doubling every + * word of a 4 KB page leaves the sum bit-identical (1024 x 2^23 == 2^33 == 0 + * mod 2^32), which silently classified an in-place x2 scale as "no engine + * store" and shipped stale values (the deterministic streams2 class). + */ +static inline u32 gsp_phys_gr_page_sum32(const u8 *p) +{ + u32 ci, csum = 0; + + for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) + csum += get_unaligned_le32(p + ci); + return csum; +} + +/* * COPYWIT r226 zero filter: an all-zero page is never a meaningful alias * partner. A fresh client's fill stages long runs of zero pages * (uninitialized cudaMalloc buffers), and consecutive zero pages produced @@ -18205,7 +24956,7 @@ static u32 gsp_phys_gr_in_census_log; * again). Returns the number of pages staged. */ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, - u32 *csum_in, u64 *staged_hpa) + u64 *csum_in, u64 *staged_hpa) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); struct { @@ -18242,7 +24993,9 @@ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, "gr-bounce-copy-in"); for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) csum += get_unaligned_le32(dst + ci); - csum_in[bi] = csum; + /* r452: the change baseline is the FNV-1a fingerprint (the + * add-sum above survives only as the all-zero test). */ + csum_in[bi] = gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); zero = gsp_phys_gr_page_all_zero(csum, dst); if (!zero && gsp_phys_gr_copywit_enabled()) { u32 j; @@ -18291,7 +25044,7 @@ static u32 gsp_phys_gr_fill_bounce(const struct gsp_phys_gr_submit *req, * retry, so a retried launch starts from the guest inputs in both sets. * Returns the number of pages staged. */ -static u32 gsp_phys_gr_fill_gap_bounce(u32 *csum_in, u64 *staged_hpa) +static u32 gsp_phys_gr_fill_gap_bounce(u64 *csum_in, u64 *staged_hpa) { struct gsp_phys_boot_chunk *dchunk = gsp_phys_gr_data_chunk(); u32 gi, copied = 0; @@ -18299,7 +25052,6 @@ static u32 gsp_phys_gr_fill_gap_bounce(u32 *csum_in, u64 *staged_hpa) for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { const u8 *src = (const u8 *)__va(gsp_phys_gr_gap[gi].hpa); u8 *dst = gsp_phys_chunk_page_va(dchunk, gsp_phys_gr_gap[gi].bi); - u32 ci, csum = 0; if (!src || !dst) continue; @@ -18309,9 +25061,9 @@ static u32 gsp_phys_gr_fill_gap_bounce(u32 *csum_in, u64 *staged_hpa) staged_hpa[gsp_phys_gr_gap[gi].bi] = gsp_phys_gr_gap[gi].hpa; gsp_phys_gr_page_spot_check(dst, src, gsp_phys_gr_gap[gi].va, "gr-bounce-copy-in"); - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(dst + ci); - csum_in[gsp_phys_gr_gap[gi].bi] = csum; + /* r452: FNV-1a baseline (see gsp_phys_gr_scan_pages). */ + csum_in[gsp_phys_gr_gap[gi].bi] = + gsp_phys_gr_fnv1a64(dst, GSP_PHYS_PAGE_SIZE); copied++; } dma_wmb(); @@ -18408,7 +25160,8 @@ static bool gsp_phys_gr_win0_retry_rearm(u32 *qmd, */ static noinline void gsp_phys_gr_poll_break_log(u32 w0, u32 want_qmd, u32 w1, u32 since_us, - const struct gsp_phys_rmcall_ce_submit *sub) + const struct gsp_phys_rmcall_ce_submit *sub, + u32 h_object) { struct gsp_phys_boot_chunk *ud_c = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; @@ -18416,16 +25169,180 @@ gsp_phys_gr_poll_break_log(u32 w0, u32 want_qmd, u32 w1, u32 since_us, u32 usr_gpget = ud ? get_unaligned_le32(ud + gsp_phys_rmcall_userd_ctl_off() + GSP_PHYS_RMCALL_USERD_GPGET_OFF) : 0; + /* + * SESSION 153-CONTINUED-47: re-read the ring slot the launch + * published (the entry AS THE ENGINE SEES IT) after the wait, so an + * X-vs-beef diff on one boot compares the exact e0/e1 the PBDMA + * fetched, plus the doorbell-visible GP_GET/GP_PUT movement. + */ + struct gsp_phys_boot_chunk *r_c = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_GPFIFO]; + u8 *ring = r_c->pages ? gsp_phys_chunk_page_va(r_c, 0) : NULL; + u32 re0 = 0, re1 = 0; + + if (ring && sub->slot < GSP_PHYS_RMCALL_GPFIFO_ENTRIES) { + re0 = get_unaligned_le32(ring + sub->slot * 8); + re1 = get_unaligned_le32(ring + sub->slot * 8 + 4); + } - pr_info("vgsp-gr: poll break w0=0x%x want=0x%x ch=0x%x after %u us usr_gpget=0x%x gpget=0x%x gpput=0x%x\n", + pr_info("vgsp-gr: poll break w0=0x%x want=0x%x ch=0x%x after %u us usr_gpget=0x%x gpget=0x%x gpput=0x%x ring[%u]=%08x %08x (pub %08x %08x)\n", w0, want_qmd, w1, (u32)ktime_to_us(ktime_get()) - since_us, - usr_gpget, sub->gpget, sub->gpput); + usr_gpget, sub->gpget, sub->gpput, + sub->slot, re0, re1, sub->e0, sub->e1); + /* + * CONT-103: engine-fetch witness + prelude-liveness bookkeeping. + * GP_GET advancing past the published slot is the PBDMA consuming + * this launch's GPFIFO entry - the fetch witness the QMD sentinel + * cannot provide on the stream3d lane (no QMD deposit there). A + * break with GP_GET frozen is a stale/software-authored release. + */ + pr_info("vgsp-gr: FETCH-WIT h=0x%x slot=%u gpget %u->%u %s\n", + h_object, sub->slot, sub->gpget, usr_gpget, + usr_gpget != sub->gpget ? "engine-fetched" : + "GP_GET-frozen (software ack)"); + if (usr_gpget != sub->gpget && + READ_ONCE(gsp_phys_gr_mme_pending_gen)) { + WRITE_ONCE(gsp_phys_gr_mme_live_gen, + READ_ONCE(gsp_phys_gr_mme_pending_gen)); + WRITE_ONCE(gsp_phys_gr_mme_pending_gen, 0); + } /* r425: the healthy-launch SKED/FECS baseline for the wedge compare. */ gsp_phys_gr_sked_wit(gsp_phys_gpu_bar0(), "post-poll"); } /* + * CONT-103: fetch-witness for a launch that did NOT break - GP_GET frozen + * means the PBDMA never consumed the entry (the >2050-word fetch wall / + * parked front-end class), so the carrier's generation is not live and the + * next launch must re-emit the prelude. noinline: keeps the submit frame. + */ +static noinline void +gsp_phys_gr_fetch_miss_log(u32 h_object, + const struct gsp_phys_rmcall_ce_submit *sub, + const char *tag) +{ + struct gsp_phys_boot_chunk *ud_c = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; + u8 *ud = ud_c->pages ? gsp_phys_chunk_page_va(ud_c, 0) : NULL; + u32 usr_gpget = ud ? + get_unaligned_le32(ud + gsp_phys_rmcall_userd_ctl_off() + + GSP_PHYS_RMCALL_USERD_GPGET_OFF) : 0; + + pr_info("vgsp-gr: FETCH-WIT %s h=0x%x slot=%u gpget %u->%u %s\n", + tag, h_object, sub->slot, sub->gpget, usr_gpget, + usr_gpget != sub->gpget ? "engine-fetched" : + "GP_GET-frozen (NOT fetched: parked or over-cap block)"); + /* + * SESSION 153-CONTINUED-109: readback-free live promotion on the + * timeout path. The poll-break path promotes the generation whose + * final prelude part was carried when GP_GET advanced (the engine + * consumed the entry). A launch can also time out the w0/w1 release + * wait AFTER the engine fetched it (r1377: `FETCH-WIT timeout ... + * gpget 5->6 engine-fetched` on the load-only carriers - their + * releases only matched ~10 s later through the ladder), and the + * historical code cleared pending_gen here, permanently losing the + * promotion: live_gen never caught pl_gen, every later launch stayed + * window-elided, and the MME never got a calls-bearing launch + * (`pc_valid=0` on all samples). Promote on the same GP_GET witness + * the break path trusts; only a genuinely unfetched carrier clears. + */ + if (usr_gpget != sub->gpget && + READ_ONCE(gsp_phys_gr_mme_pending_gen)) { + pr_info("vgsp-gr: MME-LIVE promote via GP_GET advance (%s) gen=%u h=0x%x\n", + tag, READ_ONCE(gsp_phys_gr_mme_pending_gen), + h_object); + WRITE_ONCE(gsp_phys_gr_mme_live_gen, + READ_ONCE(gsp_phys_gr_mme_pending_gen)); + WRITE_ONCE(gsp_phys_gr_mme_pending_gen, 0); + } else { + WRITE_ONCE(gsp_phys_gr_mme_pending_gen, 0); + } +} + +/* + * SESSION 153-CONTINUED-72: GR-STATE retire readback. The keep-mask A/B + * must answer "does the kept macro execute at all" before chasing the + * 0/903 stores. The FE decode-state EX records (0x40a630..3c) name the + * last MME-GENERATED method write: bit31 GENERATED, 21:12 macro address, + * 11:0 pc - gen0=1 with the kept slot's macro address at the retire is + * the live witness that the engine fetched and ran the kept macro. The + * MME HWW latch (0x404490 + INFO/IR), the trapped method (0x40704 offset + * + bit20 mmegen, data 0x40708) and the DS latch catch the write when it + * faulted. One line per retire; bounded per boot (beef up to 48, others + * 4) so a battery boot cannot flood. + */ +static noinline void gsp_phys_gr_state_rb_log(u32 h_object) +{ + static u32 gs_beef_n, gs_other_n; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + bool beef = (h_object & 0xfffff000u) == 0xbeef0000u; + u32 fed0, fed1, fed2, fed3, esr, info, ir2, trap, data, ds, gf; + /* + * SESSION 153-CONTINUED-91: with the DS HWW report mask cleared + * (sema_ds_mask_clear) a SPH validation miss latches in DS_HWW_ESR + * without the GSP-RM fault handler poisoning the PRI block, so the + * per-stage SPH_INFO snapshot is finally readable: STAGE0..5 START + * (0x405c00+si*4) / UPPER (0x405c24+si*4) name the exact SPH VA each + * pipeline stage fetched, and PROGRAM_REGION_A/B (0x405c1c/c20) the + * region the DS validated against (the r1277+ wall's missing datum). + */ + u32 sph[6], sphu[6], rr_a, rr_b; + u32 si; + + if (!bar0) + return; + if (beef ? gs_beef_n >= 48u : gs_other_n >= 4u) + return; + if (beef) + gs_beef_n++; + else + gs_other_n++; + for (si = 0; si < 6u; si++) { + sph[si] = ioread32(bar0 + 0x00405c00u + si * 4u); + sphu[si] = ioread32(bar0 + 0x00405c24u + si * 4u); + } + rr_a = ioread32(bar0 + 0x00405c1cu); + rr_b = ioread32(bar0 + 0x00405c20u); + fed0 = ioread32(bar0 + 0x0040a630u); + fed1 = ioread32(bar0 + 0x0040a634u); + fed2 = ioread32(bar0 + 0x0040a638u); + fed3 = ioread32(bar0 + 0x0040a63cu); + esr = ioread32(bar0 + 0x00404490u); + info = ioread32(bar0 + 0x00404494u); + /* SESSION 153-CONTINUED-106: INFO2 is 0x40449c (0x404498 is CONFIG). */ + ir2 = ioread32(bar0 + 0x0040449cu); + trap = ioread32(bar0 + 0x00400704u); + data = ioread32(bar0 + 0x00400708u); + ds = ioread32(bar0 + 0x00405840u); + gf = ioread32(bar0 + 0x00400504u); + pr_info("vgsp-gr: GR-STATE retire h=0x%x fed=%08x,%08x,%08x,%08x (pc0=%u mac0=0x%x gen0=%u) mme_esr=%08x info=%08x ir=%08x trap=%08x(mthd=0x%x mmegen=%u) data=%08x ds=%08x grfifo=%08x\n", + h_object, fed0, fed1, fed2, fed3, + fed0 & 0xfffu, (fed0 >> 12) & 0x3ffu, (fed0 >> 31) & 1u, + esr, info, ir2, trap, trap & 0x3ffcu, (trap >> 20) & 1u, + data, ds, gf); + pr_info("vgsp-gr: GR-STATE sph h=0x%x ds2=%08x sph=%08x,%08x,%08x,%08x,%08x,%08x sphU=%02x,%02x,%02x,%02x,%02x,%02x rr=%08x,%08x\n", + h_object, ioread32(bar0 + 0x00405848u), + sph[0], sph[1], sph[2], sph[3], sph[4], sph[5], + sphu[0] & 0xffu, sphu[1] & 0xffu, sphu[2] & 0xffu, + sphu[3] & 0xffu, sphu[4] & 0xffu, sphu[5] & 0xffu, + rr_a, rr_b); + /* + * SESSION 153-CONTINUED-91: did the emitted program-region pair + * (SET_MME_SHADOW_SCRATCH 0x78/0x79 = method 0x35e0/0x35e4 through + * class 0xc597) actually land in the MME shadow? The shadow read + * port is the same one the store-side probe uses; a zero readback + * means the write never took effect and every region experiment so + * far was a no-op. + */ + if (beef) + pr_info("vgsp-gr: GR-STATE region-shadow h=0x%x scratch[0x78]=%08x scratch[0x79]=%08x\n", + h_object, + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x35e0u), + gsp_phys_mme_shadow_read(bar0, 0xc597u, 0x35e4u)); +} + +/* * F4-b: one-shot deposit readback at the first timeout (noinline for the same * frame reason as above). Reads the qmdwin/shadow destination (or the guest * QMD page) with the booter's existing unaligned host readback; w63 is the @@ -18574,6 +25491,7 @@ static u32 gsp_phys_m91_fault_start; static u32 gsp_phys_m91_cap_used; static bool gsp_phys_m91_cap_suppressed; static bool gsp_phys_m91_icache_log; +static bool gsp_phys_r449_ccache_log; static bool gsp_phys_gr_rel_membar_log; static noinline void gsp_phys_gr_m91_fault_cap(const struct gsp_phys_gr_submit *req, @@ -18663,7 +25581,7 @@ gsp_phys_gr_m91_fault_cap(const struct gsp_phys_gr_submit *req, fn = &gsp_phys_fault_notes[(gsp_phys_fault_note_next + GSP_PHYS_FAULT_NOTE_MAX - 1u) % GSP_PHYS_FAULT_NOTE_MAX]; - pr_info("vgsp-m91: FAULT-CAP ok=%u faults=%u retry=%s attempts=%u qmd=0x%llx qmd64=%s prog_w48=0x%08x w49=0x%08x prog_va=0x%llx prog_pa=0x%llx code_src=%s off=0x%x code64=%s fault_et=0x%x fault_chid=%u fault_err=\"%.48s\"\n", + pr_info("vgsp-m91: FAULT-CAP ok=%u faults=%u retry=%s attempts=%u qmd=0x%llx qmd64=%s prog_w48=0x%08x w49=0x%08x prog_va=0x%llx prog_pa=0x%llx code_src=%s off=0x%x code64=%s fault_et=0x%x fault_chid=%u fault_err=\"%.200s\"\n", ok ? 1u : 0u, faults, fast ? "fast-rearm" : (rearmed ? "resched/fresh-entry" : "none"), attempts, qmd_dst, qhex, qmd[48], qmd[49], pva, ppa, src, off, @@ -18857,6 +25775,2457 @@ static void gsp_phys_gr_stall_log(const char *reason, u64 push_va, reason, push_va, jiffies_to_msecs(jiffies - start), action); } +/* + * SESSION 153-CONTINUED-27: big-surface alias bookkeeping. A replayed + * 3D window's colour target (C597 INC 0x800+j*0x40 [upper, lower, w, h, + * fmt] and its TWOD twin INC 0x218 [w, h, upper, lower]) can be a surface + * far larger than the launch's 16-page set (the X screen 0x120400000, + * 1280x720x4 = 900 pages). Two earlier routes are dead ends for it: + * - one-page shadow relocation (r992: `pair addr=0x840 va=0x120400000 + * -> MISS` then FAULT_PDE @ +0xa000 - the raster walk crosses the + * single alias page); + * - Tier-1 windows at the surface's own VAs (CONT.27 r993: PDE_UPDATE + * for pdeIndex 2306/2307 status=0x0, yet GPC2/GPC5 still fault + * FAULT_PDE @ +0x4000/+0xa000 - same class as the r929 v2 FE faults; + * the GPC engines only see the scratch-PT pool). + * The surviving route is the proven one: a CONTIGUOUS run of never-used + * scratch-PT fetch lines (one line = 16 PTEs = 64 KB VA; lines are + * monotonic and consecutive lines are VA-contiguous inside each pool + * block), one PTE per surface page, so the engine reaches all 900 pages + * through one contiguous alias VA and the raster walk never crosses into + * an unmapped page. The pair operands are rewritten to <alias>|<offset> + * by gsp_phys_rmcall_gr_3d_reloc. The table is per-boot: the alias PTEs + * are never rewritten (fetch-line rule) and the screen's HPAs are fixed + * by its single allocation, so every later launch is a pure rewrite. + */ +struct gsp_phys_gr_surf_keep { + u64 va; /* guest surface base VA (page aligned) */ + u64 alias; /* engine-visible contiguous shadow alias base */ + u64 npages; /* pages aliased (declared + slack tail) */ +}; +static struct gsp_phys_gr_surf_keep gsp_phys_gr_surf[4]; +static u32 gsp_phys_gr_surf_n; + +/* + * r1490: the effective engine target of the captured surface ON THIS LAUNCH. + * The captured-surface rec set (gsp_sema_gr_surf_recs) is global - it is the + * last C597 colour-target bind a trusted channel emitted - while a launch's + * own data recs only cover it when the fill resolved the WHOLE span through + * the captured bind's VAS. Several clients share the same 0x1200xxxxx VAs: + * the beef client holds a ~10-page SYS allocation at 0x120400000, the X + * server's 1280x720 screen of the same VA is a 900-page FB allocation. A + * partial run must never win the rec-over-alias preference: the engine would + * be pointed at the client's code/operand buffer, its stores would land in + * (and its copy-in would read) method/SASS words instead of the screen, and + * the surface would look like it received the method stream. + * + * Complete rec run (contiguous base..last reloc VA) -> its bounce base, the + * path that lets the retire wait and copy-out see the pixels. Otherwise the + * per-boot contiguous alias built from the captured bind's VAS pages is the + * target: the engine writes the guest screen HPAs directly, exactly as the + * pre-r1483 alias path did. 0 = neither is available (caller keeps the + * per-page path). + */ +static u64 gsp_phys_gr_surf_launch_target(const struct gsp_phys_gr_submit *req) +{ + u64 sva = 0, last_va, base = 0, last = 0; + u32 sn = 0, di; + bool have_base = false, have_last = false; + + if (!req) + return 0; + if (gsp_sema_gr_surf_recs(&sva, &sn) && sva && sn >= 2u) { + for (di = 0; di < req->n_data; di++) { + if (req->data[di].va == sva && + req->data[di].reloc_va) { + base = req->data[di].reloc_va; + have_base = true; + break; + } + } + last_va = sva + ((u64)(sn - 1u) << 12); + for (di = 0; di < req->n_data; di++) { + if (req->data[di].va == last_va && + req->data[di].reloc_va) { + last = req->data[di].reloc_va; + have_last = true; + break; + } + } + if (have_base && have_last && + last == base + ((u64)(sn - 1u) << 12)) + return base; + } + if (gsp_sema_surf_va_get(&sva, &sn)) { + for (di = 0; di < gsp_phys_gr_surf_n; di++) + if (gsp_phys_gr_surf[di].va == sva) + return gsp_phys_gr_surf[di].alias; + } + return 0; +} + +/* + * SESSION 153-CONTINUED-27: slack tail for the alias. r994 measured the + * replay's colour writes landing past the declared w*h*4 footprint + * (1280x720x4 = 0x384000) by up to +0xE000 (11.2 rows) - FAULT_PTE on + * the last fetch line's unwritten PTE slots (0x5d6000..0x5e2000 with the + * alias at 0x250000). The tail pages are aliased too: a page that still + * resolves (inside the guest allocation's own padding - exactly where + * native overdraw would land) maps the real HPA; one that does not + * resolves to a dedicated zeroed module page so the stray writes vanish + * instead of faulting. The chunk is allocated lazily and lives until + * module unload (4 KB). + */ +#define GSP_PHYS_GR_SURF_SLACK_PAGES 32u +static struct gsp_phys_boot_chunk gsp_phys_gr_surf_dummy; + +static u64 gsp_phys_gr_surf_dummy_dma_locked(void) +{ + struct gsp_phys_boot_chunk *d = &gsp_phys_gr_surf_dummy; + + if (!d->pages) { + if (gsp_phys_chunk_alloc(boot_stage.pdev, d, 1u)) + return 0; + gsp_phys_chunk_zero(d); + } + return (u64)gsp_phys_chunk_page_dma(d, 0); +} + +/* + * r929 v3 (Path A): relocate one 3D-window address operand to a shadow-pool + * VA. The guest VAs the 3D methods name (uploads 0x1_214f8f00.., vertex, + * program, tex pools) are installed in the ladder page tree by the Tier-1 + * mapper yet the engine's FE/PD walks still fault FAULT_PDE on them (r929 + * v2 evidence: "ENGINE GRAPHICS HUBCLIENT_FE faulted @ 0x1_214f8000 ... + * FAULT_PDE"); the shadow pool lines (map_shadow_locked) ARE the proven + * engine-visible region (every CUDA data/page relocation lives there). The + * page offset is preserved; a VA whose 4 KB page has no resolved hpa stays + * verbatim (it faults later and is named in the drain). Called with + * gsp_phys_rm_lock held. + */ +/* + * SESSION 153-CONTINUED-59: window/attr operand-miss state for the + * pre-doorbell abort (see gsp_phys_rmcall_gr_3d_reloc's level-2 fallback + * and the MISS-ABORT site in gsp_phys_backend_guest_gr_submit). Reset at + * the start of each 3D-window patch pass; a nonzero count at the submit + * point means the emitted block still carries a raw client VA the physical + * engine cannot resolve, which is the r1104/r1105 lane-killing class. + * sema_gl_miss_abort=0 restores the historical submit-anyway behaviour. + */ +static u32 gsp_phys_rmcall_gr_3d_miss_n; +static u64 gsp_phys_rmcall_gr_3d_junk_n; +static bool gsp_phys_rmcall_gr_miss_abort = true; +static u64 gsp_phys_gr_miss_abort_n; +module_param_named(sema_gl_miss_abort, gsp_phys_rmcall_gr_miss_abort, bool, + 0644); +MODULE_PARM_DESC(sema_gl_miss_abort, + "Abort a 3D-window launch before the doorbell while a window/attr operand relocation MISSes (page demanded via the pin path; retried on the next forward; fail-open after 3000 aborts). Default 1"); + +/* + * SESSION 153-CONTINUED-73: direct-write witness over the launch's OWN + * relocated operand destinations. The retirement scan only sees stores into + * the data-rec bounce copies; a pair rewritten to a shadow or surface-alias + * VA makes the engine store STRAIGHT into the guest page behind it (the + * 3d_reloc alias comment calls this class "direct guest writes, invisible + * to the witness"). Every such rewrite records its destination guest page + * here (with the source VA for the report); the submit path fingerprints + * the set right before the doorbell and the retirement wait re-fingerprints + * it, so one healthy run answers: do the draw's stores land in the bounce + * (counted today), in guest pages through a relocated operand (this + * witness), or in NEITHER (the draw rasterized nothing and the missing + * pipeline/vertex state is the wall). The MME-family payload rewrites + * (sema_gl_mme_reloc -> mme-reloc-rsv) feed the same set, so the MME + * shadow-scratch destinations are covered too. Static scratch: the only + * caller is the single-inflight ordered gsp-phys-work item. + */ +#define GSP_PHYS_GR_WIT_MAX 4096u +static u64 gsp_phys_gr_wit_pa[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_va[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_csum[GSP_PHYS_GR_WIT_MAX]; +static u64 gsp_phys_gr_wit_first8[GSP_PHYS_GR_WIT_MAX]; +static u32 gsp_phys_gr_wit_n; +static u32 gsp_phys_gr_wit_rep_log; + +/* + * CONT-111: the changed-page line budget was a hardcoded 4; a store-rich + * launch (r1378 reached 305/903 relocated-operand pages) needs the whole + * list to find the render target among the operand pages. 0 keeps the + * summary line only. + */ +static u32 gsp_phys_gr_wit_shown = 4; +module_param_named(sema_gl_wit_shown, gsp_phys_gr_wit_shown, uint, 0644); +MODULE_PARM_DESC(sema_gl_wit_shown, + "Max GUEST-WIT changed va/pa lines per report (default 4; CONT-111 RT hunt)"); + +static void gsp_phys_gr_wit_reset(void) +{ + gsp_phys_gr_wit_n = 0; +} + +static void gsp_phys_gr_wit_note(u64 va, u64 hpa) +{ + u32 k; + + if (!hpa) + return; + hpa &= ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + for (k = 0; k < gsp_phys_gr_wit_n; k++) + if (gsp_phys_gr_wit_pa[k] == hpa) + return; + if (gsp_phys_gr_wit_n >= GSP_PHYS_GR_WIT_MAX) + return; + gsp_phys_gr_wit_pa[gsp_phys_gr_wit_n] = hpa; + gsp_phys_gr_wit_va[gsp_phys_gr_wit_n] = + va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + gsp_phys_gr_wit_n++; +} + +static void gsp_phys_gr_wit_capture(const struct gsp_phys_gr_submit *req) +{ + u32 k; + + /* + * Also witness the guest pages behind the launch's tracked data + * records (and the late-resolved gap pages): a store through a + * surface ALIAS VA lands in the surface's own guest HPA - invisible + * to the bounce scan even when the same page is a tracked record - + * so the direct-write pass needs these HPAs in the set. The + * pre-doorbell timing (after every module-side writer) makes the + * baseline the guest content the engine starts from. + */ + for (k = 0; req && k < req->n_data; k++) + gsp_phys_gr_wit_note(req->data[k].va, req->data[k].hpa); + for (k = 0; k < gsp_phys_gr_gap_n; k++) + gsp_phys_gr_wit_note(gsp_phys_gr_gap[k].va, + gsp_phys_gr_gap[k].hpa); + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + + gsp_phys_gr_wit_first8[k] = p ? + get_unaligned_le64(p) : 0; + gsp_phys_gr_wit_csum[k] = p ? + gsp_phys_gr_fnv1a64(p, GSP_PHYS_PAGE_SIZE) : 0; + } +} + +static u32 gsp_phys_gr_wit_report(const char *where) +{ + u32 k, changed = 0, shown = 0; + u64 first_va = 0, first_pa = 0; + + if (!gsp_phys_gr_wit_n) + return 0; + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + u64 fp = p ? + gsp_phys_gr_fnv1a64(p, GSP_PHYS_PAGE_SIZE) : 0; + + if (fp != gsp_phys_gr_wit_csum[k]) { + changed++; + if (!first_pa) { + first_pa = gsp_phys_gr_wit_pa[k]; + first_va = gsp_phys_gr_wit_va[k]; + } + /* + * CONT-73 attribution: the pre/post first word of up + * to four changed pages. A QMD/descriptor word or an + * MME-scratch datum names the writer; freshly built + * vertex/float data means the guest CPU rewrote its + * buffer while the engine ran. + */ + if (shown < gsp_phys_gr_wit_shown) { + shown++; + pr_info("vgsp-gr: GUEST-WIT %s changed va=0x%llx pa=0x%llx pre=0x%016llx post=0x%016llx\n", + where, + gsp_phys_gr_wit_va[k], + gsp_phys_gr_wit_pa[k], + gsp_phys_gr_wit_first8[k], + p ? get_unaligned_le64(p) : 0); + } + } + } + /* + * One line per launch while the report budget lasts (a full boot can + * carry hundreds of X-channel launches; the beef rows are the ones + * being chased). + */ + if (gsp_phys_gr_wit_rep_log < 512u) { + gsp_phys_gr_wit_rep_log++; + pr_info("vgsp-gr: GUEST-WIT %s: %u/%u relocated-operand guest page(s) changed (direct shadow/alias stores%s)\n", + where, changed, gsp_phys_gr_wit_n, + changed ? "" : " - NONE (no direct guest writes)"); + if (changed) + pr_info("vgsp-gr: GUEST-WIT %s first changed va=0x%llx pa=0x%llx\n", + where, first_va, first_pa); + } + /* + * CONT-73 settle control: the engine is retired by now, so a second + * scan after a short quiet delay separates engine stores (one-shot, + * will not grow) from LIVE guest-CPU rewrites (still growing). A + * nonzero "late" count here means the changed rows above are guest + * noise, not engine writes. + */ + if (changed && gsp_phys_gr_wit_rep_log <= 128u) { + u32 late = 0; + + msleep(150); + for (k = 0; k < gsp_phys_gr_wit_n; k++) { + const u8 *p = + (const u8 *)__va(gsp_phys_gr_wit_pa[k]); + u64 fp = p ? + gsp_phys_gr_fnv1a64(p, + GSP_PHYS_PAGE_SIZE) : 0; + + if (fp != gsp_phys_gr_wit_csum[k]) + late++; + } + pr_info("vgsp-gr: GUEST-WIT %s settle: %u page(s) changed at +150 ms (was %u)%s\n", + where, late, changed, + late > changed ? + " - GUEST-ACTIVE (growing; noise)" : + " - quiescent (engine-final writes)"); + } + return changed; +} + +/* + * SESSION 153-CONTINUED-90: SPH-alias receipt. The DS I-fetch of a bound + * stage reaches the program-family VA through the shadow VA the relocation + * picked, and the PTE of that shadow VA names the physical page the engine + * reads. Log the mapping choice (launch page set vs trusted resolver), the + * shadow VA and the mapped HPA, then read the fetch words back through the + * host direct map of that very page (for a RAM-backed HPA the direct map is + * content-identical to the engine's PTE-based read), so "which physical page + * does the DS I-fetch actually read" is provable from the log alone. Bounded + * per boot; foreign (non-RAM) HPAs are named but not dereferenced. + */ +static void gsp_phys_gr_sph_alias_receipt(u64 page, u64 va, u64 sh, u64 hpa, + const char *via) +{ + static u32 sph_alias_n; + u32 w[8] = { 0 }; + u32 c[4] = { 0 }; + u32 off = (u32)(va & (GSP_PHYS_PAGE_SIZE - 1u)); + u32 coff = off + 0x80u; /* SPH code starts 0x80 B into the image */ + bool rd = false; + + if (sph_alias_n >= 48u) + return; + sph_alias_n++; + if (pfn_valid((unsigned long)(hpa >> PAGE_SHIFT))) { + const u8 *p = (const u8 *)__va(hpa + off); + u32 i; + + for (i = 0; i < 8u; i++) + w[i] = get_unaligned_le32(p + (size_t)i * 4u); + if (coff + sizeof(c) <= GSP_PHYS_PAGE_SIZE) { + const u8 *cp = (const u8 *)__va(hpa + coff); + + for (i = 0; i < 4u; i++) + c[i] = get_unaligned_le32(cp + (size_t)i * 4u); + } + rd = true; + } + pr_info("vgsp-gr: SPH-ALIAS-%s page=0x%llx va=0x%llx off=0x%x shadow=0x%llx hpa=0x%llx rd=%u w8=%08x %08x %08x %08x %08x %08x %08x %08x code4=%08x %08x %08x %08x\n", + via, page, va, off, sh, hpa, rd ? 1u : 0u, + w[0], w[1], w[2], w[3], w[4], w[5], w[6], w[7], + c[0], c[1], c[2], c[3]); +} + +/* + * SESSION 153-CONTINUED-90: A/B canary for the DS fetch path. When set, + * every program-family operand the window relocates maps a dedicated canary + * page (a deterministic 0xcc00cc00|idx pattern) instead of the resolved + * guest page. The DS error set either changes (the DS does fetch through + * this shadow VA) or stays byte-identical (something else supplies the + * stage start, e.g. the QMD program pointer path). Live-flippable. + */ +static bool gsp_phys_gr_sph_canary; +module_param_named(sema_gl_sph_canary, gsp_phys_gr_sph_canary, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_canary, + "Map program-family shadow VAs to a canary page (SPH fetch-path A/B)"); +static struct gsp_phys_boot_chunk sph_canary_chunk; +static bool sph_canary_ready; + +static u64 gsp_phys_gr_sph_canary_hpa(void) +{ + if (!sph_canary_ready) { + u8 *p; + u32 i; + + if (!boot_stage.pdev || + gsp_phys_chunk_alloc(boot_stage.pdev, &sph_canary_chunk, 1)) + return 0; + p = gsp_phys_chunk_page_va(&sph_canary_chunk, 0); + if (!p) + return 0; + for (i = 0; i < GSP_PHYS_PAGE_SIZE / 4u; i++) + ((u32 *)p)[i] = 0xcc00cc00u | (i & 0xffu); + sph_canary_ready = true; + } + return (u64)gsp_phys_chunk_page_dma(&sph_canary_chunk, 0); +} + +/* + * SESSION 153-CONTINUED-93: 8K program-pair shadow line (mission task 2 + * fallback). The DS validates a bound program through its fetched SPH; a + * per-page shadow line maps exactly one 4 KB page at PTE0 and leaves the + * line's PTE1 invalid, so any read past the first 4 KB of the program page + * (the SASS tail of a bind whose delivered length crosses 0x1000, or any + * DS full-program walk) leaves the authored bytes behind. Map the program + * family page (PTE0) and family+0x1000 (PTE1) into ONE fresh 128 B fetch + * line BEFORE the engine can fetch it (a line's contents are frozen once + * fetched), so a contiguous 8 KB is engine-visible from its first window + * on. The line is cached per hpa0 (pair cache) so every later window reuses + * the identical never-rewritten mapping; the alias cache is re-pointed at + * the pair line so every other consumer (program binds, code-fetch + * operands) sees the same 8 KB view. Live A/B via sema_gl_sph_8k. Called + * with gsp_phys_rm_lock held. + */ +static bool gsp_phys_rmcall_gr_sph_8k; +module_param_named(sema_gl_sph_8k, gsp_phys_rmcall_gr_sph_8k, bool, 0644); +MODULE_PARM_DESC(sema_gl_sph_8k, + "Map the program family page and family+0x1000 in one contiguous 8K shadow line (fresh fetch line; default 0)"); +#define GSP_PHYS_RMCALL_GR_PAIR_CACHE 64u +static u64 gsp_phys_gr_pair_hpa[GSP_PHYS_RMCALL_GR_PAIR_CACHE]; +static u64 gsp_phys_gr_pair_va[GSP_PHYS_RMCALL_GR_PAIR_CACHE]; +static u32 gsp_phys_gr_pair_n; + +/* + * Resolve the HPA of `page0 + 4K` through the same sources the relocation + * chain uses: the launch's own page set (the pages the copy-in authored), + * the tracked data recs, the sema service's trusted resolver, then the + * late-resolved gap pages. 0 = not resolvable. + */ +static u64 gsp_phys_rmcall_gr_prog_next_hpa_locked( + const struct gsp_phys_gr_submit *req, u64 page0) +{ + u64 page1 = page0 + GSP_PHYS_PAGE_SIZE; + u32 i; + + for (i = 0; i < req->n_pages; i++) + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + req->pages[i].hpa) + return req->pages[i].hpa; + for (i = 0; i < req->n_data; i++) + if ((req->data[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + req->data[i].hpa) + return req->data[i].hpa; + { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(page1, &r) && r.hpa) + return r.hpa; + } + for (i = 0; i < gsp_phys_gr_gap_n; i++) + if ((gsp_phys_gr_gap[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page1 && + gsp_phys_gr_gap[i].hpa) + return gsp_phys_gr_gap[i].hpa; + return 0; +} + +/* + * r1494: resolve one guest page through the same chain the relocation pass + * uses: the launch's own page set (copy-in authored pages), the tracked + * data recs, the GR ledger/pin/walk resolver, then the late-resolved gap + * pages. 0 = not resolvable. Unlike prog_next_hpa this takes any page, not + * just page0+4K. + */ +static u64 gsp_phys_rmcall_gr_stream_page_hpa_locked( + const struct gsp_phys_gr_submit *req, u64 page) +{ + u32 i; + + /* + * r1497: a vertex/index fetch page can resolve to an FB page whose + * emulated shadow is empty while the same VA has a sysmem mapping + * with the real bytes (r1496: the active indexed draw's 0x44fea00 + * stream read all-zero). Populate the empty shadow page before the + * stream line is built. Non-empty pages and SYS answers are no-ops. + */ + gsp_sema_gr_fetch_backing(page); + for (i = 0; i < req->n_pages; i++) + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page && + req->pages[i].hpa) + return req->pages[i].hpa; + for (i = 0; i < req->n_data; i++) + if ((req->data[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page && + req->data[i].hpa) + return req->data[i].hpa; + { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(page, &r) && r.hpa) + return r.hpa; + } + for (i = 0; i < gsp_phys_gr_gap_n; i++) + if ((gsp_phys_gr_gap[i].va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page && + gsp_phys_gr_gap[i].hpa) + return gsp_phys_gr_gap[i].hpa; + return 0; +} + +static void gsp_phys_rmcall_gr_alias_cache_set_locked(u64 hpa, u64 va) +{ + u32 i; + + for (i = 0; i < boot_stage.rmcall_gr_alias_count; i++) + if (boot_stage.rmcall_gr_alias_hpa[i] == hpa) { + boot_stage.rmcall_gr_alias_va[i] = va; + return; + } + if (boot_stage.rmcall_gr_alias_count < GSP_PHYS_RMCALL_GR_ALIAS_CACHE) { + u32 ci = boot_stage.rmcall_gr_alias_count; + + boot_stage.rmcall_gr_alias_hpa[ci] = hpa; + boot_stage.rmcall_gr_alias_va[ci] = va; + boot_stage.rmcall_gr_alias_count = ci + 1u; + } +} + +/* + * r1494: a vertex stream in the MME payload is + * [stream, end_hi, end_lo, base_hi, base_lo]; END and BASE are both client + * VAs and the MME derives the stream length from end - base. The generic + * per-word payload pass relocated each end independently to its own + * single-page shadow fetch line, so any stream that crosses a 4 KB + * boundary (the client's vertex buffers here are 0x600..0x2d2c bytes and + * several cross) came out with its two ends in unrelated shadow windows: + * end - base went negative, and the VTG's fetch past the base page hit an + * invalid PTE (the 0xbadf1201 miss=1 priv-error seen once the real VS + * started executing). Map the whole [base, end) range into ONE fresh + * fetch line (up to 16 x 4 KB) and rewrite BOTH words from that line, so + * the size the MME computes is the client's real size and every page of + * the stream is engine-visible at base+offset. A page that cannot be + * resolved maps the module zero page: a zero attribute is a degenerate + * vertex, never a fault. Returns the line base VA; caller consumes one + * shadow slot. + */ +static u64 gsp_phys_rmcall_gr_stream_line_locked( + const struct gsp_phys_gr_submit *req, u64 base_va, u64 end_va, + u32 slot) +{ + struct gsp_phys_boot_chunk *sh = &boot_stage.rmcall_gr_shadow; + u64 base_page = base_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 end_page = (end_va - 1u) & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 hpa[GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES]; + u64 pte[GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES]; + u64 fb_off[GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES]; + bool real[GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES]; + u64 va0, dummy = 0; + u32 npages, i, line, page, line_in_page, dummy_n = 0, vid_n = 0; + u8 *pt; + int ret; + + if (end_va <= base_va) + return 0; + npages = (u32)((end_page - base_page) >> 12) + 1u; + if (npages > GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES) + return 0; + for (i = 0; i < npages; i++) { + u64 pa = base_page + ((u64)i << 12); + + real[i] = false; + fb_off[i] = 0; + /* + * r1497: the active draw's vertex/index page resolves to a + * vidmem page whose emulated shadow is empty. Point the + * fetch line at the real FBMEM page (the guest's own page + * table names the offset) instead of delivering zeros. + */ + if (gsp_phys_gr_fetch_vid_off_locked(pa, &fb_off[i])) { + pte[i] = gsp_phys_rmcall_gr_pte_leaf_vid(fb_off[i]); + hpa[i] = 0; + real[i] = true; + vid_n++; + continue; + } + hpa[i] = gsp_phys_rmcall_gr_stream_page_hpa_locked(req, pa); + if (!hpa[i]) { + if (!dummy) + dummy = gsp_phys_gr_surf_dummy_dma_locked(); + if (!dummy) + return 0; + hpa[i] = dummy; + dummy_n++; + } + pte[i] = gsp_phys_rmcall_gr_pte_leaf(hpa[i]); + } + if (!sh->pages) { + ret = gsp_phys_chunk_alloc(boot_stage.pdev, sh, + GSP_PHYS_RMCALL_GR_SHADOW_PAGES); + if (ret) + return 0; + gsp_phys_chunk_zero(sh); + } + if (slot >= sh->pages || !boot_stage.rmcall_va_guest_pt) + return 0; + line = boot_stage.rmcall_guest_pt_next_line; + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) + return 0; + page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + ret = gsp_phys_rmcall_gr_scratch_win_install_locked(page); + if (ret) + return 0; + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) + return 0; + for (i = 0; i < npages; i++) + put_unaligned_le64(pte[i], pt + (size_t)( + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + line_in_page + i) * 8); + dma_wmb(); + boot_stage.rmcall_guest_pt_next_line = line + 1u; + va0 = gsp_phys_rmcall_guest_pt_win_va(page) + + ((u64)line_in_page << 16); + for (i = 0; i < npages; i++) + if (!real[i]) + gsp_phys_rmcall_gr_alias_cache_set_locked(hpa[i], + va0 + ((u64)i << 12)); + { + static u32 stream_line_log; + u32 w[8] = { 0 }, k; + u64 hpa0 = hpa[0]; + + if (!real[0] && + pfn_valid((unsigned long)(hpa0 >> PAGE_SHIFT))) { + const u8 *p = (const u8 *)__va( + hpa0 + (base_va & (GSP_PHYS_PAGE_SIZE - 1))); + + for (k = 0; k < 8u; k++) + w[k] = get_unaligned_le32(p + (size_t)k * 4u); + } + if (stream_line_log < 32u) { + stream_line_log++; + pr_info("vgsp-gr: MME-VSTREAM-LINE h=0x%x base=0x%llx end=0x%llx pages=%u dummy=%u vid=%u fb0=0x%llx line=0x%llx hpa0=0x%llx first8=%08x %08x %08x %08x %08x %08x %08x %08x\n", + req ? req->h_object : 0u, base_va, end_va, + npages, dummy_n, vid_n, fb_off[0], va0, hpa0, + w[0], w[1], w[2], w[3], w[4], w[5], w[6], + w[7]); + } + } + return va0; +} + +/* + * Allocate (or reuse) the pair line. Returns the line base VA on success, + * 0 when the second half cannot be resolved or the pool is exhausted (the + * caller then falls through to the single-page path). The fresh line is + * allocated on first use for an hpa0; the pair cache reuses it afterwards + * (the PTEs are identical on every window by construction). + */ +static u64 gsp_phys_rmcall_gr_sph_pair_locked( + const struct gsp_phys_gr_submit *req, u64 page0, u64 hpa0, u32 slot) +{ + struct gsp_phys_boot_chunk *sh = &boot_stage.rmcall_gr_shadow; + u32 i, line, page, line_in_page; + u64 hpa1, va0; + u8 *pt; + int ret; + + for (i = 0; i < gsp_phys_gr_pair_n; i++) + if (gsp_phys_gr_pair_hpa[i] == hpa0) + return gsp_phys_gr_pair_va[i]; + hpa1 = gsp_phys_rmcall_gr_prog_next_hpa_locked(req, page0); + if (!hpa1) + return 0; + if (!sh->pages) { + ret = gsp_phys_chunk_alloc(boot_stage.pdev, sh, + GSP_PHYS_RMCALL_GR_SHADOW_PAGES); + if (ret) + return 0; + gsp_phys_chunk_zero(sh); + } + if (slot >= sh->pages || !boot_stage.rmcall_va_guest_pt) + return 0; + line = boot_stage.rmcall_guest_pt_next_line; + if (line >= GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) + return 0; + page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + line_in_page = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + if (gsp_phys_rmcall_gr_scratch_win_install_locked(page)) + return 0; + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) + return 0; + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa0), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + line_in_page) * 8); + put_unaligned_le64(gsp_phys_rmcall_gr_pte_leaf(hpa1), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + line_in_page + 1u) * 8); + dma_wmb(); + boot_stage.rmcall_guest_pt_next_line = line + 1u; + va0 = gsp_phys_rmcall_guest_pt_win_va(page) + + ((u64)line_in_page << 16); + gsp_phys_rmcall_gr_alias_cache_set_locked(hpa0, va0); + gsp_phys_rmcall_gr_alias_cache_set_locked(hpa1, + va0 + GSP_PHYS_PAGE_SIZE); + if (gsp_phys_gr_pair_n < GSP_PHYS_RMCALL_GR_PAIR_CACHE) { + gsp_phys_gr_pair_hpa[gsp_phys_gr_pair_n] = hpa0; + gsp_phys_gr_pair_va[gsp_phys_gr_pair_n] = va0; + gsp_phys_gr_pair_n++; + } + pr_info("vgsp-gr: SPH-PAIR line=0x%llx hpa0=0x%llx hpa1=0x%llx (8K contiguous program page) h=0x%x\n", + va0, hpa0, hpa1, req->h_object); + return va0; +} + +/* + * SESSION 153-CONTINUED-92: identity-fetch keep test (see the + * sema_gl_prog_guest_va knob above). Resolve the operand's page (the + * launch's own page set first - the pages the copy-in authored - then the + * sema service's trusted resolver), install the Tier-1 identity PTE at the + * guest VA itself and report whether the operand may stay verbatim. The + * receipt names the exact VA the DS will now fetch through (the guest VA, + * shadow column = the guest VA) and reads the fetch words back through the + * host direct map of the mapped HPA, so "the DS reads the authored page at + * its own heap VA" is provable from the log alone. Anything the identity + * mapper refuses (no window slot, stale fetch line, unresolved HPA) returns + * false and the caller falls back to the shadow rewrite. Called with + * gsp_phys_rm_lock held. + */ +static bool gsp_phys_rmcall_gr_prog_keep_locked(void __iomem *bar0, + u32 epoch, u32 up_w, u32 lo_w, + const struct gsp_phys_gr_submit *req) +{ + u64 va = ((u64)(up_w & 0xffu) << 32) | (u64)(lo_w & 0xfffffffcu); + u64 page = va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 hpa = 0; + u32 i; + + if (!va || !page) + return false; + for (i = 0; i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == page && + req->pages[i].hpa) { + hpa = req->pages[i].hpa; + break; + } + } + if (!hpa) { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(page, &r) && r.hpa) + hpa = r.hpa; + } + if (!hpa) + return false; + if (gsp_phys_rmcall_gr_map_va_locked(bar0, page, hpa, epoch)) + return false; + gsp_phys_gr_sph_alias_receipt(page, va, va, hpa, "prog-guest"); + return true; +} + +static bool gsp_phys_rmcall_gr_3d_reloc(u32 *up_w, u32 *lo_w, + const struct gsp_phys_gr_submit *req, + u32 *n_shadow) +{ + u64 va = ((u64)(*up_w & 0xffu) << 32) | (u64)(*lo_w & 0xfffffffcu); + u64 page = va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u64 sh = 0; + u32 line = 0, i; + + if (!va || !page) + return false; + /* + * SESSION 153-CONTINUED-90: must run before every resolution stage so + * the canary cannot be shadowed by an earlier (surd) alias hit. + */ + if (READ_ONCE(gsp_phys_gr_sph_canary) && req->prog_va && + page == (req->prog_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 chpa = gsp_phys_gr_sph_canary_hpa(); + u32 line3 = 0; + + if (chpa && + *n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && + !gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, chpa, + &sh, &line3)) { + if (line3 != 0xffffffffu) + (*n_shadow)++; + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + gsp_phys_gr_sph_alias_receipt(page, va, sh, chpa, + "canary"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + /* + * SESSION 153-CONTINUED-63: rec-over-alias preference. When the fill + * tracked the render target through the bounce lane (every page + * resolved into contiguous data recs), rewrite the operand to the + * rec's bounce VA so the engine's stores land where the retire wait + * can count them and the copy-out can ship them back to the guest's + * own pages. The completeness gate (gsp_sema_gr_surf_recs) makes a + * partial run impossible to point at. Runs before the alias so the + * identity alias (direct guest writes, invisible to the witness) + * cannot win for a tracked target. + */ + { + u64 sva = 0; + u32 sn = 0; + + if (gsp_sema_gr_surf_recs(&sva, &sn) && + va >= sva && va < sva + ((u64)sn << 12)) { + for (i = 0; i < req->n_data; i++) { + if (req->data[i].va != page || + !req->data[i].reloc_va) + continue; + sh = req->data[i].reloc_va | + (va & (GSP_PHYS_PAGE_SIZE - 1)); + *lo_w = (*lo_w & 0x3u) | + ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + } + /* + * SESSION 153-CONTINUED-27: a big surface (the X screen) is aliased + * as ONE contiguous run by the pre-pass; rewrite the operand to the + * alias base | offset. A one-page shadow relocation of it would + * break the raster walk at the first page crossing (r992), and the + * raw VA has no PTE chain the GPC engines can see (r993). + */ + for (i = 0; i < gsp_phys_gr_surf_n; i++) { + if (va >= gsp_phys_gr_surf[i].va && + va < gsp_phys_gr_surf[i].va + + (gsp_phys_gr_surf[i].npages << 12)) { + /* + * CONT-73: the alias VA maps the surface's guest + * HPAs 1:1 (the alias PTEs were built from + * req->pages[] first, the trusted resolver else), so + * a store through it lands straight in that guest + * page - record it for the direct-write witness. + */ + { + u32 pi; + + for (pi = 0; pi < req->n_pages; pi++) { + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != + page || + !req->pages[pi].hpa) + continue; + gsp_phys_gr_wit_note(page, + req->pages[pi].hpa); + break; + } + if (pi == req->n_pages) { + struct gsp_sema_gr_reloc ra; + + if (!gsp_sema_gr_reloc_resolve(page, + &ra) && + ra.hpa) + gsp_phys_gr_wit_note(page, + ra.hpa); + } + } + sh = gsp_phys_gr_surf[i].alias + + (va - gsp_phys_gr_surf[i].va); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + for (i = 0; i < req->n_pages; i++) { + if ((req->pages[i].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != page || + !req->pages[i].hpa) + continue; + if (*n_shadow >= GSP_PHYS_RMCALL_GR_SHADOW_PAGES) + return false; + /* + * SESSION 153-CONTINUED-93: 8K program-pair line (see the + * pair helper). Only the program family page rides this + * path; an unresolvable second half falls through to the + * single-page map below. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sph_8k) && req->prog_va && + page == (req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) { + u64 pv = gsp_phys_rmcall_gr_sph_pair_locked( + req, page, req->pages[i].hpa, *n_shadow); + + if (pv) { + (*n_shadow)++; + sh = pv | (va & + (GSP_PHYS_PAGE_SIZE - 1)); + gsp_phys_gr_wit_note(page, req->pages[i].hpa); + gsp_phys_gr_sph_alias_receipt(page, va, pv, + req->pages[i].hpa, "pair"); + *lo_w = (*lo_w & 0x3u) | + ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | + ((u32)(sh >> 32) & 0xffu); + return true; + } + } + if (gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, + req->pages[i].hpa, + &sh, &line)) + return false; + if (line != 0xffffffffu) + (*n_shadow)++; /* alias hit consumes no slot */ + /* CONT-73: shadow VA maps this guest HPA 1:1. */ + gsp_phys_gr_wit_note(page, req->pages[i].hpa); + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + if (req->prog_va && + page == (req->prog_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + gsp_phys_gr_sph_alias_receipt(page, va, sh, + req->pages[i].hpa, + "pages"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | ((u32)(sh >> 32) & 0xffu); + return true; + } + /* + * SESSION 153-CONTINUED-59: level-2 late resolution, the same trusted + * chain the MME payload branch uses (ipin/pin/lat/map/walk then shadow + * map). The r1104 first beat window MISSed the program pair 0x2054 + * (va=0x4350900) and the r1105 windows MISSed the index-buffer pairs + * (0x17c8, va=0x44fedf0...) whose pages were not in the launch page + * set at build time; the raw guest VA then let the engine fault the + * MMU (HUBCLIENT_PD) and the guest driver reset the channel. Resolve + * the page here if any stage knows it, so a MISS only survives when + * every source fails; a surviving MISS is recorded for the pin demand + * set and counts toward the pre-doorbell abort. + */ + { + struct gsp_sema_gr_reloc r; + u32 line2 = 0; + + if (!gsp_sema_gr_reloc_resolve(va, &r) && r.hpa && + *n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES && + !gsp_phys_rmcall_gr_map_shadow_locked(*n_shadow, r.hpa, + &sh, &line2)) { + if (line2 != 0xffffffffu) + (*n_shadow)++; + /* CONT-73: shadow VA maps this guest HPA 1:1. */ + gsp_phys_gr_wit_note(page, r.hpa); + sh |= va & (GSP_PHYS_PAGE_SIZE - 1); + if (req->prog_va && + page == + (req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + gsp_phys_gr_sph_alias_receipt(page, va, sh, + r.hpa, + "resolve"); + *lo_w = (*lo_w & 0x3u) | ((u32)sh & 0xfffffffcu); + *up_w = (*up_w & ~0xffu) | ((u32)(sh >> 32) & 0xffu); + return true; + } + } + /* + * SESSION 153-CONTINUED-61: junk-operand filter. The window + * recognizer is byte-shape based, so float/immediate DATA words in + * the method stream also parse as address pairs. The observed junk + * class sits in the 256 MB..4 GB numeric range (0x3d000000..0xc2...... + * float bit patterns: 0x3f800000 = 1.0f, 0xbf800000 = -1.0f, ...) + * and can NEVER resolve - r1109 measured miss=20 constant across + * every attempt, all of it this class, which pinned the abort census + * nonzero forever and blocked every submit. Real client VAs of this + * driver live below 0x5000000 (the 64-80 MB GL/heap range, e.g. + * 0x4250000/0x4350000/0x462fc40) or at/above 0x1_00000000 (the 4G+ + * push-buffer/UVM ranges). A miss outside both is a data-word + * misidentification, not a raw client VA the engine would read as an + * address: it is neither demanded (the demand set was 60 % junk by + * r1109, diverting the pin cycle) nor counted toward the pre-doorbell + * abort. Witnessed per class, bounded. + */ + if (page >= 0x10000000ULL && page < 0x100000000ULL) { + static u32 gr_missjunk_log; + + gsp_phys_rmcall_gr_3d_junk_n++; + if (gr_missjunk_log < 24 || + !(gsp_phys_rmcall_gr_3d_junk_n % 512u)) { + gr_missjunk_log++; + pr_info("vgsp-gr: MISS-JUNK va=0x%llx (data-word class; not demanded, not counted toward the abort) n=%llu\n", + page, (u64)gsp_phys_rmcall_gr_3d_junk_n); + } + return false; + } + /* + * Unresolvable: demand the page for the pin path and count the miss + * (the submit path aborts pre-doorbell while the count is nonzero). + */ + gsp_phys_rmcall_gr_3d_miss_n++; + gsp_sema_demand_note(page); + return false; +} + +/* + * r929 v3: copy the 3D window and relocate every address operand a 3D-class + * method run carries (same recognizers as the sema service's svc_gl_field: + * LINE_LENGTH_IN 0x180 OFFSET_OUT pair, SET_VERTEX_STREAM_SUBSTITUTE 0xf84, + * SET_TEX_SAMPLER_POOL 0x155c, SET_TEX_HEADER_POOL 0x1574 and the + * SET_PIPELINE_PROGRAM_ADDRESS 0x2014/0x2018+j*0x40 pairs). Returns the + * number of relocated pair(s). Run widths follow the service's decoder: + * op 1/3/5 are 1+count dwords, everything else resyncs one dword. + */ +/* + * SESSION 153-CONTINUED-13 (draw-window strip experiment): comma-separated + * hex NVC597 method byte-addresses to REMOVE from a captured 3D window + * before the verbatim replay (whole methods - a dropped run takes its + * count data words with it). Answers whether a DRAW window's non-retire + * lives in the draw content (BEGIN/END/immediate vertices/DRAW_ARRAY) or + * in the state binds/method-stream shape. Empty = off (historical + * behaviour); live-tunable via sysfs so one boot can sweep the variants. + */ +/* + * SESSION 153-CONTINUED-47 (A/B knob): gates the newly recognised pair + * classes (SET_INDEX_BUFFER_A/B 0x17c8 and the vertex-stream banks + * 0x1c20..0x1cf0) in the verbatim-replay relocation pass. Default 0 keeps + * the r1035-r1041 baseline relocation set exactly; flipping it live + * (`echo Y > /sys/module/vgsp/parameters/sema_gl_idx_reloc`) rewrites the + * next launches' index/vertex operands to shadow VAs, so one boot can A/B + * the effect instead of comparing across boots. + */ +static bool gsp_phys_rmcall_gr_idx_reloc; +module_param_named(sema_gl_idx_reloc, gsp_phys_rmcall_gr_idx_reloc, bool, 0644); +MODULE_PARM_DESC(sema_gl_idx_reloc, + "Relocate SET_INDEX_BUFFER (0x17c8) and vertex-stream banks 0x1c20..0x1cf0 operands to shadow VAs in 3D replays (default 0 = baseline set only)"); + +/* + * SESSION 153-CONTINUED-48: MME-family payload relocation. The C597 class + * carries CALL_MME_MACRO(j) = 0x3800+j*8 and CALL_MME_DATA(j) = 0x3804+j*8 + * (class/clc597.h); the GLX client's macro(2) run (observed at byte + * address 0x3810, cnt=5) inlines raw 32-bit client VAs as macro arguments + * (`... 0450f800 ... 0450f000` in the first beef window of r1042/r1043). + * Those words are the last raw-operand class left in the faulting window + * after the index/vertex relocation: the engine walks the macro data + * through its own fetch path, and a raw client VA has no PTE chain the + * GPC engines can see (the r993/r1043 FAULT_PDE->PRI-fault class). The + * pass rewrites any payload word of a run whose byte address lies in + * [0x3800, 0x3a00) through the same vetted shadow mapping the pair + * classes use: only a word whose page the launch's own page set (or the + * big-surface alias) resolves is rewritten (MISS stays verbatim), the + * rewrite maps the SAME HPA at a shadow VA so the value is + * content-equivalent, and a shadow VA that would need an upper byte (a + * 64-bit alias in a 32-bit slot) is refused. Default 0 (baseline windows + * byte-identical); live-tunable so one boot can A/B. + */ +static bool gsp_phys_rmcall_gr_mme_reloc; +module_param_named(sema_gl_mme_reloc, gsp_phys_rmcall_gr_mme_reloc, bool, 0644); +MODULE_PARM_DESC(sema_gl_mme_reloc, + "Relocate CALL_MME_MACRO/DATA(j) run payload words whose page resolves in the launch page set to shadow VAs in 3D replays (default 0 = off)"); + +/* + * SESSION 153-CONTINUED-58: torn-read zero guard for the vertex-attribute + * families. The r1098/r1099 first-launch class error + * (`Class Error: ChId 0001, Class 0000c597, Offset 00001168, + * Data 00000000, ErrorCode 00000005` = SET_VERTEX_ATTRIBUTE_A(2) written + * with NUMERICAL_TYPE=0, the header's UNUSED_ENUM_DO_NOT_USE value -> + * INVALID_ENUM) is written by a replayed window run + * `20030458 38400000 38200040 00000000`: a cnt=3 INC run at method 0x1160 + * whose THIRD data word is exactly zero and is immediately followed by a + * 52-word zero region - the read caught the client's push buffer after + * part of it had been reused/cleared. The same client writes valid A(2) + * values in the other windows of the same push (0x38200040/0x38200600/ + * 0x38400600), so the zero is a torn read, not client content; and + * NUMERICAL_TYPE=0 is never a legitimate client write. Guard: for INC + * runs in the SET_VERTEX_ATTRIBUTE_A/B families (0x1160..0x11dc) drop a + * data word that is exactly zero, splitting the run so the remaining + * words keep their methods (the attribute then keeps the value the + * ATTR-PRELUDE or the previous frame wrote). Default 1; live-tunable. + */ +static u32 gsp_phys_rmcall_gr_attr_zero_fix = 1; +module_param_named(sema_gl_attr_zero_fix, gsp_phys_rmcall_gr_attr_zero_fix, uint, 0644); +MODULE_PARM_DESC(sema_gl_attr_zero_fix, + "Drop torn-read zero data words in the SET_VERTEX_ATTRIBUTE_A/B runs (0x1160..0x11dc) of 3D replays, splitting the run (default 1)"); + +/* (SESSION 153-CONTINUED-53 MME prelude knob is declared before + * gsp_phys_rmcall_build_gr_launch, its reader.) */ + +/* + * SESSION 153-CONTINUED-48: ladder-wedge recovery. CONT-47 measured that + * the FIRST beef launch per boot whose window carries an unresolvable raw + * operand PRI-faults the GR: every SKED/FIFO register then reads the + * priv-error poison (cp. kernel_falcon_ga102.c `privErrVal = 0xbadf0000`) + * and the ladder channel stays wedged for the whole boot (engine GP_GET + * stuck; every later publish unconsumed, even an emptied window) - which + * makes content bisection a one-shot per boot. When enabled, the launch + * poll watches the GR FIFO status (0x400504, the sked recorder's + * `grfifo`) for the 0xbadf-tagged poison and performs ONE bounded + * recovery per launch: the r445 hard re-arm (TSG PREEMPT bWait=1 + the + * m155 lane-enable tail + GPFIFO/TSG schedules + doorbell) followed by + * the warm GR semaphore probe, exactly the mechanism behind + * gsp_phys_gr_wedge_rearm. Bounded: one detection per launch, poison + * gated, all RPCs keep their own timeouts. Default 1 (the poisoned state + * is already a guaranteed-dead boot; the recovery only adds the chance + * that later launches in the same boot still execute). + */ +static bool gsp_phys_gr_wedge_recover = true; +module_param_named(sema_gr_wedge_recover, gsp_phys_gr_wedge_recover, bool, + 0644); +MODULE_PARM_DESC(sema_gr_wedge_recover, + "On the 0xbadf priv-error poison in the GR FIFO status, run one bounded TSG-preempt+resched hard re-arm per launch (default 1)"); + +/* + * SESSION 153-CONTINUED-93: parked-engine recovery on the warm-sem ladder + * gate. The r1311-r1313 class: after the first few DS-validated draws the + * front-end stops fetching (grfifo=0x00000001, no poison, no trap) and the + * warm GR semaphore probe times out on every later launch - the ladder + * then fails fast and the whole boot is dead for bisection. When this knob + * is set, a wedged probe triggers ONE bounded hard re-arm (the proven + * r445 TSG-preempt bWait=1 + resched + probe) before giving up; an ALIVE + * probe after the re-arm lets the fresh-entry retries resume, and the MME + * prelude latch is cleared so the re-preload is owed (the preempt reset + * the context). Bounded: one per launch; all RPCs keep their timeouts. + * Default 1; A/B with 0 restores the fail-fast ladder. + */ +static bool gsp_phys_gr_park_recover = true; +module_param_named(sema_gr_park_recover, gsp_phys_gr_park_recover, bool, + 0644); +MODULE_PARM_DESC(sema_gr_park_recover, + "On a wedged warm-sem probe in the launch timeout ladder, run one bounded TSG-preempt+resched hard re-arm per launch and retry (default 1)"); + +static char *gsp_phys_rmcall_gr_win_drop; +module_param_named(sema_gl_win_drop, gsp_phys_rmcall_gr_win_drop, charp, 0644); +MODULE_PARM_DESC(sema_gl_win_drop, + "Hex method byte-addresses (comma-separated) to strip from a captured 3D window before replay (draw isolation; default empty)"); + +static bool gsp_phys_rmcall_gr_win_drop_match(u32 addr) +{ + const char *p = gsp_phys_rmcall_gr_win_drop; + u32 n = 0; + + if (!p || !*p) + return false; + while (*p && n < 64) { + u32 v = 0; + u32 digits = 0; + + while (*p == ' ' || *p == ',' || *p == ':' || *p == ';') + p++; + /* + * SESSION 153-CONTINUED-46: accept an optional 0x/0X prefix. + * The original parser scanned hex digits only, so the natural + * "0x3810,0x39c8" input parsed as 0 (the 'x' ended the digit + * run after the leading '0') and silently never matched - + * cost a full bisect round. + */ + if (p[0] == '0' && (p[1] == 'x' || p[1] == 'X')) + p += 2; + for (;;) { + char c = *p; + u32 d; + + if (c >= '0' && c <= '9') + d = c - '0'; + else if (c >= 'a' && c <= 'f') + d = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + d = c - 'A' + 10; + else + break; + if (v <= 0xfffffffu) + v = (v << 4) | d; + p++; + digits++; + } + if (!digits) + break; + /* + * SESSION 153-CONTINUED-47: `lo-hi` drops every method + * byte-address in the closed range (both ends accept the + * 0x prefix). The content bisect needs whole families + * (`0x3800-0x3a00` MME, `0x3400-0x3500` MME scratch) and + * the emptied-window control (`0-0x7ffc`): with the window + * stripped whole, a launch that still retires proves the + * SET_OBJECT + release tail path is healthy and that the + * wedge lives in the stripped content. + */ + if (*p == '-') { + const char *q = p + 1; + u32 hi = 0; + u32 hd = 0; + + if (q[0] == '0' && (q[1] == 'x' || q[1] == 'X')) + q += 2; + for (;;) { + char c = *q; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (hi <= 0xfffffffu) + hi = (hi << 4) | h; + q++; + hd++; + } + if (hd) { + p = q; + n++; + if (addr >= v && addr <= hi) + return true; + continue; + } + } + n++; + if (v == addr) + return true; + } + return false; +} + +/* + * SESSION 153-CONTINUED-27: auto-drop of the software-emulated lanes from + * every verbatim 3D replay. Three method triggers in a captured window + * cause the physical engine to repeat work the sema service already + * executes in software - and, on the replay channel, with the WRONG + * context state (no previous-window sticky registers): + * - 0x1b0 LAUNCH_DMA (C597 CE lane): triggers both the CE copy and the + * LOAD_INLINE_DATA upload. The service authors every LID payload into + * the guest pages itself (svc_gl_lid/svc_grce_inline_write) and + * emulates the CE strips (svc_op2d); physically re-running them with + * an unset source/context walks unbounded memory (r992-r996: the + * "march" - faults at whatever page the alias cap ended, e.g. + * 0x5d6000/0x5f4000, after having written every mapped page). + * - 0x5e0 (TWOD RECTS fill trigger): the second point pair of the + * client's screen clear. The service's FERMI_TWOD executor runs every + * fill (r986: 46x `sema-twod-fill ... done=3686400 rc=0`); the + * physical re-run uses the replay channel's unset dst PITCH and + * marches (same fault class). + * - 0x10c NOTIFY: the completion write; without the preceding + * SET_NOTIFY_A/B (which live in other pushes) the engine writes + * address 0 (r996: `faulted @ 0x0_00000000 FAULT_PTE VIRT_WRITE`). + * The kick-notify lane + the backend's release tail author every + * completion the client actually waits on. + * All three are pure triggers - their setup methods stay in the window, + * so dropping the run can never misalign display state. The emulated + * lanes are the live ones (the r986 pixel proof: the screen paints via + * svc_twod fills, and every CE strip retires rc=0 in software). Default + * on; set 0 to replay the raw window unmodified. + */ +static bool gsp_phys_gr_win_drop_auto = 1; +module_param_named(sema_gl_win_drop_auto, gsp_phys_gr_win_drop_auto, bool, + 0644); +MODULE_PARM_DESC(sema_gl_win_drop_auto, + "Drop the software-emulated trigger methods (0x10c NOTIFY, 0x1b0 LAUNCH_DMA, 0x5e0 TWOD fill) from verbatim 3D replays (default 1)"); + +static bool gsp_phys_rmcall_gr_win_drop_auto_match(u32 addr) +{ + if (!READ_ONCE(gsp_phys_gr_win_drop_auto)) + return false; + /* + * SESSION 153-CONTINUED-48: NVC597_SET_REPORT_SEMAPHORE_A=0x1b00 - + * the C597 twin of the 0x10c NOTIFY trigger. Every glcore draw + * window ends with 4-word runs [A=upper(7:0), B=lower(31:0), + * C=payload, D=flags] whose B word is a raw client VA (observed + * 0x41a5bb0/0x759e870/0x251800; payloads = frame counters 0xb4f3+). + * The replayed engine executing them writes a client VA the GPC + * engines cannot reach - the VIRT_WRITE fault class (r1061: the + * kept [0,0x2000) window whose only fetch/write triggers besides + * the index binds are these raw 0x1b00 writes is exactly the one + * that still faulted with grfifo=0xbadf1201; r1062: the full window + * stalls the front-end right after the PBDMA consumes it and + * fecs_cur never loads). Their page cannot join the launch page set + * (n_ptr is capped at 7 by the QMD pointer region and the draw's + * program/index/vertex/MME pages already fill it), so relocation is + * impossible; dropping is the same call the 0x10c NOTIFY drop + * already makes: the module appends its own SET_REPORT_SEMAPHORE + * tail and the host-side completion lanes author every completion + * the client actually waits on (the r1061 keep-prefix window + * retired with ch_rel=0xc0ffee02 and the client kept pushing + * frames without any 0x1b00 in the replay). + */ + return addr == 0x010cu || addr == 0x01b0u || addr == 0x05e0u || + addr == 0x1b00u; +} + +/* + * SESSION 153-CONTINUED-14 (FECS residency / draw-stall isolation): patch + * the DATA of one method inside a replayed 3D window before it reaches the + * engine. Format: comma-separated `addr:data` hex pairs, addr = method + * byte-address (e.g. 1618 = BEGIN, 1614 = END), data = the replacement + * value. Header-only methods (IMMD) get their 13-bit data field rewritten + * in place; count-carrying methods (INC/NON_INC/INC_ONCE) get their FIRST + * payload word replaced. Live-tunable via sysfs so one boot can sweep + * variants (BEGIN OP=7 QUADS -> TRIANGLES etc.) against the draw-stall. + */ +static char *gsp_phys_rmcall_gr_win_patch; +module_param_named(sema_gl_win_patch, gsp_phys_rmcall_gr_win_patch, charp, + 0644); +MODULE_PARM_DESC(sema_gl_win_patch, + "Hex addr:data pairs (comma-separated) patched into a captured 3D window method before replay (default empty)"); + +static bool gsp_phys_rmcall_gr_win_patch_find(u32 addr, u32 *out_data) +{ + const char *p = gsp_phys_rmcall_gr_win_patch; + u32 n = 0; + + if (!p || !*p) + return false; + while (*p && n < 64) { + u32 v = 0, d = 0; + u32 digits = 0; + bool have_data = false; + + while (*p == ' ' || *p == ',' || *p == ':' || *p == ';') + p++; + for (;;) { + char c = *p; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (v <= 0xfffffffu) + v = (v << 4) | h; + p++; + digits++; + } + if (!digits) + break; + if (*p == ':' || *p == '=') { + p++; + for (;;) { + char c = *p; + u32 h; + + if (c >= '0' && c <= '9') + h = c - '0'; + else if (c >= 'a' && c <= 'f') + h = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + h = c - 'A' + 10; + else + break; + if (d <= 0xfffffffu) + d = (d << 4) | h; + p++; + have_data = true; + } + } + n++; + if (have_data && v == addr) { + *out_data = d; + return true; + } + } + return false; +} + +static u32 gsp_phys_rmcall_gr_3d_last_subdrop; +static u32 gsp_phys_rmcall_gr_3d_last_sdmdrop; + +/* + * SESSION 153-CONTINUED-62: an MME-family payload word that neither the + * launch page set nor the level-2 resolver could translate has to stay + * verbatim in the emitted stream (never guessed) - but a raw client VA in + * an 0x3810 macro payload is exactly the operand the engine faults on + * (`MMU Fault HUBCLIENT_PD`). Until this helper the word was left silent: + * the abort census stayed 0 and the launch went out to fault seconds + * later (r1115: fault @0x0_04350000, the 0x4350900 program-page payload). + * Now the surviving miss counts toward the pre-doorbell MISS-ABORT and + * joins the pin demand set, so the launch is held (fail-closed, bounded) + * until the page resolves, exactly like a window pair miss. The float/ + * data-word class (0x10000000..0x100000000) is excluded - it can never + * resolve and must not pin the census nonzero (the r1109 lesson). + */ +static void gsp_phys_gr_3d_mme_miss(u32 wv) +{ + u64 page = (u64)wv & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + if (page >= 0x10000000ULL && page < 0x100000000ULL) + return; + gsp_phys_rmcall_gr_3d_miss_n++; + gsp_sema_demand_note(page); +} + + +static u32 gsp_phys_rmcall_gr_3d_last_progkeep; +static u32 gsp_phys_rmcall_gr_3d_patch_window(const u32 *in, u32 n, u32 *out, + u32 *out_n, u32 *n_dropped, + const struct gsp_phys_gr_submit *req, + u32 *n_shadow, + void __iomem *bar0, u32 epoch) +{ + u32 off = 0, wout = 0, reloc = 0, dropped = 0, subdrop = 0, sdm_drop = 0; + u32 wp_match = 0, wp_lowrun = 0, prog_keep = 0; + + *out_n = n; + *n_dropped = 0; + while (off < n) { + u32 hdr = in[off]; + u32 op = (hdr >> 29) & 0x7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 sub = (hdr >> 13) & 0x7u; + u32 span = 1u; + + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) + span = 1u + cnt; + /* + * SESSION 153-CONTINUED-52: drop METHOD WRITES aimed at a + * non-zero subchannel. The glcore draw window ends with + * `2001c00d 70000000/68000000` = INC m=0x34 cnt=1 SUBCHANNEL 6 + * (channel-class MEM_OP_D lane) two-word runs. The ladder + * channel has nothing bound on subchannel 6, so the physical + * PBDMA hands those methods to its host-software lane and + * raises NV_PPBDMA_INTR_0_METHOD (bit 21): the r1075 relay + * records `Channel ID 00000004 intr 00200000` at every launch, + * and the real host driver's software-method handler answers + * "Invalid host sofware method received" -> rcRequired -> + * robust-channel error recovery kills the channel mid-window + * (fifo_gk104.c). The methods are channel-level barriers whose + * effects the module's own release/fence lanes already author, + * so dropping them from the replay is the same call the 0x10c + * NOTIFY / 0x1b0 LAUNCH_DMA auto-drops make. Only count- + * carrying writes (op 1/3/5) are candidates: the single-word + * TERT encodings (op 0/2/4/7) reuse bits 15:13 as mask/type + * payload. + */ + if ((op == 1u || op == 3u || op == 5u) && sub != 0u) { + dropped++; + subdrop++; + off += span; + continue; + } + /* + * SESSION 153-CONTINUED-59: single-word SDM-type entries + * (SEC_OP 0/2/6, see gsp_phys_rmcall_gr_sdm_drop). The + * retiring X-family windows never yield one; this guards a + * window that carries the client's SDM markers (the + * 0x1fff0-class marker words or the prelude 0x046x6dXX + * family), which the physical PBDMA refuses with PBENTRY. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_sdm_drop) && + hdr && (op == 0u || op == 2u || op == 6u)) { + /* + * SESSION 153-CONTINUED-86: name every dropped SDM + * entry (bounded) so the A/B arm can be audited: the + * "keep the drop only for entries proven unrelated" + * decision needs the actual words, not the count. + */ + static u32 sdm_log; + + if (sdm_log < 24u) { + sdm_log++; + pr_info("vgsp-gr: SDM-DROP in[%u]=%08x op=%u addr=%#x sub=%u cnt/val=%#x (span=%u)\n", + off, hdr, op, addr, sub, + (cnt ? cnt : ((hdr >> 16) & 0x1fffu)), + span); + } + dropped++; + sdm_drop++; + off += span; + continue; + } + if (gsp_phys_rmcall_gr_win_drop_match(addr) || + gsp_phys_rmcall_gr_win_drop_auto_match(addr)) { + dropped++; + off += span; + continue; + } + memcpy(&out[wout], &in[off], (size_t)span * 4); + /* + * SESSION 153-CONTINUED-58: torn-read zero guard - see the + * sema_gl_attr_zero_fix comment. Only INC runs (op 1) whose + * data words advance one method each; the attribute runs are + * exactly this shape. Rebuild the run in place, skipping + * exactly-zero words and re-targeting the surviving words' + * methods (a zero word's method is simply never written, so + * the attribute keeps its prelude/previous-frame value). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_attr_zero_fix) && + op == 1u && cnt && addr >= 0x1160u && addr <= 0x11dcu) { + u32 k, k0 = 0, L = 0, wr = wout; + u32 subch = (hdr >> 13) & 0x7u; + + for (k = 0; k <= cnt; k++) { + u32 wv = (k < cnt) ? out[wout + 1u + k] : 0u; + /* + * SESSION 153-CONTINUED-62: the bad predicate + * also covers a torn ACTIVE attribute whose + * COMPONENT_BIT_WIDTHS enum is 0 (no such + * encoding exists): the r1113 setup-push + * capture carried + * `20030458 38400000 38200040 000081a4` + * (A(2)=0x81a4, widths=0) and the replayed + * prelude write trapped the engine - + * `Class Error: Class 0000c597, Offset + * 00001168, Data 000081a4, ErrorCode 0xc` + * -> grfifo=0xbadf1201 poison the moment the + * surf prelude let the draw execute. An + * inactive attribute (SOURCE=1, bit 6) keeps + * its word; only the invalid active encoding + * is dropped, so the attribute keeps its + * prelude/previous value until the window's + * own (valid) write lands. + */ + bool bad = (k < cnt) && + (wv == 0u || + (((wv >> 21) & 0x3fu) == 0u && + !(wv & 0x40u))); + + if (!bad && k < cnt) { + if (!L) + k0 = k; + out[wr + 1u] = out[wout + 1u + k]; + wr++; + L++; + continue; + } + if (L) { + u32 naddr = addr + 4u * k0; + u32 nhdr = (1u << 29) | + (L << 16) | + ((naddr >> 2) & 0x1fffu) | + (subch << 13); + + out[wr - L] = nhdr; + L = 0; + } + if (bad) { + static u32 az_log; + + if (az_log < 64) { + az_log++; + pr_info("vgsp-gr: ATTR-ZERO addr=0x%x word=%u/%u off=%u dropped (torn zero/invalid-width; attribute keeps its prelude value)\n", + addr + 4u * k, k, + cnt, off); + } + } + } + /* +1: the rebuilt run still needs its (new) header word */ + span = (wr - wout) + 1u; + } + { + /* + * SESSION 153-CONTINUED-14: live data patch (BEGIN + * OP sweep etc.). Header-only forms rewrite the 13-bit + * data field; count-carrying forms rewrite the first + * payload word. + */ + u32 pv; + + if (gsp_phys_rmcall_gr_win_patch_find(addr, &pv)) { + static u32 patch_log; + u32 old; + + wp_match++; + if (span > 1u) { + old = out[wout + 1]; + out[wout + 1] = pv; + } else { + old = (out[wout] >> 16) & 0x1fffu; + out[wout] = (out[wout] & ~0x1fff0000u) | + ((pv & 0x1fffu) << 16); + } + if (patch_log < 32) { + patch_log++; + pr_info("vgsp-gr: 3D win-patch addr=0x%x off=%u data 0x%x -> 0x%x\n", + addr, off, old, pv); + } + } else if (addr >= 0x100u && addr < 0x200u) { + /* + * SESSION 153-CONTINUED-84: the win_patch + * knob (sema_gl_win_patch=124:1) reaches + * this pass only if the window carries a + * run at the patch address; log the low-page + * runs so a decode mismatch is visible + * instead of silent. + */ + wp_lowrun++; + if (wp_lowrun <= 4u && + gsp_phys_rmcall_gr_win_patch && + *gsp_phys_rmcall_gr_win_patch) + pr_info("vgsp-gr: WIN-PATCH lowrun addr=0x%x op=%u cnt=%u hdr=%08x\n", + addr, op, cnt, hdr); + } + } + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) { + u32 *d = &out[wout + 1]; + u32 lo_i = 0; + bool pair = false; + + if (op == 1 && addr == 0x0180 && cnt >= 4) { + lo_i = 2; /* OFFSET_OUT is v2/v3 */ + pair = true; + } else if (op == 1 && addr == 0x0f84 && cnt >= 2) { + pair = true; + } else if (op == 1 && addr == 0x155c && cnt >= 3) { + pair = true; + } else if (op == 1 && addr == 0x1574 && cnt >= 3) { + pair = true; + } else if (op == 1 && addr >= 0x2014 && + addr < 0x4000 && (addr & 0x3fu) == 0x14 && + cnt >= 2) { + pair = true; + } else if (op == 1 && (addr == 0x1c00 || addr == 0x1c10) && + cnt >= 3) { + /* SESSION 153-CONTINUED-10: SET_VERTEX_STREAM_ + * A_FORMAT(j)+BASE(j) run [fmt, upper, lower] + * (j*0x10 stride) - the vertex data the draw + * reads. Observed [00001000 1 2150b790]. */ + lo_i = 1; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr >= 0x1c20 && addr <= 0x1cf0 && + (addr & 0xfu) == 0u && cnt >= 3) { + /* SESSION 153-CONTINUED-47 (A/B knob): widen + * the vertex-stream banks from j=0/1 to every + * FORMAT slot (0x1c20..0x1cf0). The GLX + * client's 3D window binds streams j>=2 that + * svc_gl_field already collects into the page + * set but the window never had rewritten. */ + lo_i = 1; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr == 0x17c8 && cnt >= 2) { + /* SESSION 153-CONTINUED-47 (A/B knob): SET_ + * INDEX_BUFFER_A/B [ADDRESS_UPPER(7:0), + * ADDRESS_LOWER(31:0)] - the indexed draw + * fetches its indices through this VA. The + * r1042 boot showed an X window carrying + * index binds PRI-stalling once relocated + * (relocation of exactly this pair is the + * only kernel delta vs the r1035 baseline + * where index-bearing windows did not exist + * and every X window retired), so the + * rewrite is gated off by default and A/B-ed + * live per boot. Observed [00000000 + * 0450fcc8] -> VA 0x450fcc8. */ + lo_i = 0; + pair = true; + } else if (READ_ONCE(gsp_phys_rmcall_gr_idx_reloc) && + op == 1 && addr == 0x1b00 && cnt >= 2) { + /* SESSION 153-CONTINUED-48: NVC597_SET_REPORT_ + * SEMAPHORE_A/B..D (0x1b00) - the engine's own + * completion write [upper(7:0), lower(31:0), + * payload, flags] (observed [0, 0x4181bb0, + * 0x4f9f6, 0x1000f010] at the end of every + * glcore window). With the VA raw the replayed + * engine writes a client VA the GPC engines + * cannot reach - the VIRT_WRITE fault class + * (r1061: the kept [0,0x2000) window that + * faults is exactly the one still carrying the + * raw 0x1b00 VAs). Rewrite the pair to the + * shadow VA (same HPA, page collected by + * svc_gl_field 0x1b00), so the write lands in + * the client's own page - its wait then sees a + * real hardware completion. MISS stays + * verbatim like every other pair. */ + lo_i = 0; + pair = true; + } else if (op == 1 && addr == 0x2380 && cnt >= 3) { + /* SET_CONSTANT_BUFFER_SELECTOR_A/B: + * [size, upper, lower]; observed + * [00010000 1 2150a500]. */ + lo_i = 1; + pair = true; + } else if (op == 1 && addr >= 0x0800 && addr < 0x0c00 && + (addr & 0x3fu) == 0u && cnt >= 2) { + /* SET_COLOR_TARGET_A/B(j) offset pair: + * [upper, lower, w, h, fmt]; the surface the + * draw rasterizes into (must live in guest + * RAM for the present lane to see pixels). + * Observed [1 222fa200 40 4 f7 0]. */ + lo_i = 0; + pair = true; + } else if (op == 1 && addr == 0x218 && cnt >= 4 && + (d[2] & ~0xffu) == 0u && + (d[3] & 0xfffu) == 0u) { + /* SESSION 153-CONTINUED-27: the TWOD twin + * of the colour target - INC 0x218 cnt>=4 + * covers 0x218..0x224 = [w, h, upper, + * lower] (the emulator's SET_DST sequence; + * the glxgears client window fills the X + * screen through it). Shape-checked so a + * non-surface run can never be rewritten. */ + lo_i = 2; + pair = true; + } + if (pair) { + u64 pva = ((u64)(d[lo_i] & 0xffu) << 32) | + (u64)(d[lo_i + 1] & 0xfffffffcu); + static u32 pair_log; + bool ok = false, kept = false; + + /* + * SESSION 153-CONTINUED-92: the program-family + * space experiment (see sema_gl_prog_guest_va). + * The DS rejects every bound stage whose SPH it + * fetches through a relocated shadow VA while + * the bytes at those VAs are provably correct; + * leave the operand at its ORIGINAL guest VA + * whenever the page resolves + identity-maps, + * so the DS fetches/validates in the same VA + * space the client's program heap describes. + * Kept pairs count as relocated so the SPH-BIND + * guard (orel == nrun) is unaffected; a page the + * identity mapper refuses falls through to the + * shadow rewrite exactly as before. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_prog_guest_va) && + bar0 && req->prog_va && + addr >= 0x2014u && addr < 0x4000u && + (addr & 0x3fu) == 0x14u && + gsp_phys_rmcall_gr_prog_keep_locked( + bar0, epoch, d[lo_i], d[lo_i + 1], + req)) { + static u32 progva_log; + + kept = true; + prog_keep++; + if (progva_log < 24u) { + progva_log++; + pr_info("vgsp-gr: PROG-GUEST-VA keep addr=0x%x va=0x%llx (Tier-1 identity; operand left verbatim) h=0x%x\n", + addr, pva, + req->h_object); + } + } else { + ok = gsp_phys_rmcall_gr_3d_reloc( + &d[lo_i], &d[lo_i + 1], req, + n_shadow); + } + if (ok || kept) + reloc++; + if (!kept && pair_log < 48) { + pair_log++; + pr_info("vgsp-gr: pair addr=0x%x va=0x%llx -> %s0x%llx\n", + addr, pva, ok ? "" : + "MISS ", + ((u64)(d[lo_i] & 0xffu) << 32) | + (u64)(d[lo_i + 1] & 0xfffffffcu)); + } + } + /* + * SESSION 153-CONTINUED-48: MME-family payload + * relocation (see the sema_gl_mme_reloc param). The + * C597 MME macro/data family spans 0x3800..0x3a00; + * its runs carry the macro arguments as raw payload + * words, which include 32-bit client VAs + * (0x450f800/0x450f000 observed in the beef macro(2) + * run at 0x3810). Rewrite only a word whose page the + * launch's own page set resolves (the same vetted + * shadow mapping the pairs use); a MISS, an + * unaligned/too-small word, or a shadow VA that + * cannot fit the 32-bit slot stays verbatim. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_mme_reloc) && + addr >= 0x3800u && addr < 0x3a00u) { + u32 wi; + bool stream_pair = false; + + /* + * r1494: VERTEX_STREAM macro(2) carries + * [stream, end_hi, end_lo, base_hi, base_lo] + * where END and BASE are client VAs (the MME + * computes the stream length as end - base). + * Relocating the two words independently put + * a page-crossing stream's ends in unrelated + * shadow windows: end - base went negative + * and the VTG fetch past the base page read + * an invalid PTE -> 0xbadf1201 miss=1 once + * the real VS ran. Map the whole range into + * one fresh contiguous line and rewrite both + * words from it; fall back to the historical + * per-word pass only when the range cannot be + * mapped. + */ + if (addr == 0x3810u && cnt >= 5u && + !d[1] && !d[3]) { + u64 bva = (u64)d[4]; + u64 eva = (u64)d[2]; + u64 lva; + + if (bva >= 0x10000u && eva > bva && + (eva - bva) <= 0x10000u && + !(bva & 3u) && !(eva & 3u) && + *n_shadow < + GSP_PHYS_RMCALL_GR_SHADOW_PAGES && + (lva = gsp_phys_rmcall_gr_stream_line_locked( + req, bva, eva, + *n_shadow)) != 0) { + u64 bpage = bva & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + + d[2] = (u32)(lva + (eva - bpage)); + d[4] = (u32)(lva + (bva & + (GSP_PHYS_PAGE_SIZE - 1))); + (*n_shadow)++; + stream_pair = true; + reloc += 2; + } + } + for (wi = 0; wi < cnt; wi++) { + u32 wv = d[wi]; + u32 up = 0, lo = wv; + static u32 mme_log; + + if (stream_pair && + (wi == 2u || wi == 4u)) + continue; + if (wv < 0x10000u || (wv & 3u)) + continue; + if (!gsp_phys_rmcall_gr_3d_reloc( + &up, &lo, req, n_shadow)) { + /* + * SESSION 153-CONTINUED-51: + * the payload page missed the + * launch's page set (the QMD + * pointer region caps the set + * far below a full window's + * operand count - r1071: the + * macro payload pages + * 0x44fd200..0x44fde00 were + * ~12 distinct pages while the + * set held 7; the words stayed + * RAW and the engine fetched an + * untranslatable guest VA on + * every full window - the + * 0xbadf1201 priv-error + * poison). Resolve the page + * through the SAME trusted + * ledger/pin/walk chain the + * launch pages use (the + * RELOC-FIXED resolver) and + * shadow-map it exactly like a + * matched page. A resolve/map + * miss or a shadow VA that + * cannot fit the 32-bit slot + * stays verbatim (never + * guessed) - and, SESSION + * 153-CONTINUED-62, it now + * COUNTS toward the abort + * census and joins the pin + * demand set: r1115's residual + * fault was exactly this class + * (`MMU Fault HUBCLIENT_PD @ + * 0x0_04350000` 2.5 s after a + * launch whose census read 0 + * because an 0x3810 payload + * word 0x4350900 missed here + * silently). + */ + struct gsp_sema_gr_reloc r; + u64 va2 = (u64)wv; + u64 sh2 = 0; + u32 line2 = 0; + + if (gsp_sema_gr_reloc_resolve( + va2, &r) || + !r.hpa) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (*n_shadow >= + GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (gsp_phys_rmcall_gr_map_shadow_locked( + *n_shadow, r.hpa, + &sh2, &line2)) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + if (line2 != 0xffffffffu) + (*n_shadow)++; + /* CONT-73: MME shadow-scratch + * destination maps this guest + * HPA 1:1; record it so a store + * through the MME path is + * witnessed too. */ + gsp_phys_gr_wit_note( + va2 & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1), + r.hpa); + sh2 |= va2 & + (GSP_PHYS_PAGE_SIZE - 1); + up = (u32)(sh2 >> 32) & 0xffu; + lo = (wv & 0x3u) | + ((u32)sh2 & 0xfffffffcu); + if (up != 0) { + gsp_phys_gr_3d_mme_miss( + wv); + continue; + } + d[wi] = lo; + reloc++; + if (mme_log < 32) { + mme_log++; + pr_info("vgsp-gr: mme-reloc-rsv addr=0x%x wi=%u w=0x%x -> 0x%x\n", + addr, wi, wv, + lo); + } + continue; + } + if (up != 0) { + gsp_phys_gr_3d_mme_miss(wv); + continue; + } + d[wi] = lo; + reloc++; + if (mme_log < 32) { + mme_log++; + pr_info("vgsp-gr: mme-reloc addr=0x%x wi=%u w=0x%x -> 0x%x\n", + addr, wi, wv, lo); + } + } + /* + * r1491: the VERTEX_STREAM macro + * (CALL_MME_MACRO(2)=0x3810) carries + * [stream, limit_hi, limit_lo, addr_hi, + * addr_lo]; log the post-relocation words so + * the VTG stream base the engine will program + * is provable from the log. Bounded. + */ + if (addr == 0x3810u && cnt >= 5u) { + static u32 vstream_log; + + if (vstream_log < 24u) { + vstream_log++; + pr_info("vgsp-gr: MME-VSTREAM h=0x%x stream=%u end=%08x:%08x base=%08x:%08x (post-reloc)\n", + req ? req->h_object : 0u, + d[0], d[1], d[2], + d[3], d[4]); + } + } + } + wout += span; + off += 1u + cnt; + continue; + } + off++; /* header-only / resync, like the service decoder */ + wout++; + } + /* + * SESSION 153-CONTINUED-84: prove the win_patch knob reached the + * pass (the r1236 "never logged" was a late write, not a decode + * gap - this line makes the distinction explicit on every boot). + */ + if (gsp_phys_rmcall_gr_win_patch && *gsp_phys_rmcall_gr_win_patch) { + static u32 wp_dbg; + + if (wp_dbg < 24u) { + wp_dbg++; + pr_info("vgsp-gr: WIN-PATCH sweep knob=%s runs=%u matched=%u lowruns=%u h=0x%x\n", + gsp_phys_rmcall_gr_win_patch, n, wp_match, + wp_lowrun, + req ? req->h_object : 0u); + } + } + *out_n = wout; + *n_dropped = dropped; + gsp_phys_rmcall_gr_3d_last_subdrop = subdrop; + gsp_phys_rmcall_gr_3d_last_sdmdrop = sdm_drop; + gsp_phys_rmcall_gr_3d_last_progkeep = prog_keep; + return reloc; +} + +/* + * SESSION 153-CONTINUED-27: alias-build for a big colour target. Consumes + * ceil(npages/16) never-used scratch-PT fetch lines and writes one PTE per + * surface page, so the whole surface is reachable through one CONTIGUOUS + * alias VA (lines are monotonic; consecutive lines are VA-contiguous + * inside a pool block - both pool blocks are checked). Every page's HPA + * comes from the launch's own page set when present, else from the + * service's trusted resolver - the identical vetted translation the + * shadow relocation uses. Lines are consumed even on failure (they are + * never reused, so a half-written line can never be shadowed later); + * the caller then falls back to the existing per-page path. Called with + * gsp_phys_rm_lock held. + */ +static bool gsp_phys_gr_surface_alias_locked(const struct gsp_phys_gr_submit *req, + u64 va, u64 npages, u64 *alias_out) +{ + u64 map_pages = npages + GSP_PHYS_GR_SURF_SLACK_PAGES; + u32 nlines; + u32 base_line; + u32 i, j; + u64 first_va, last_va; + + if (map_pages > 4096u) + map_pages = 4096u; + nlines = (u32)((map_pages + + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES - 1) / + GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES); + base_line = boot_stage.rmcall_guest_pt_next_line; + if (!nlines || base_line + nlines > GSP_PHYS_RMCALL_GUEST_PT_TOTAL_LINES) + return false; + first_va = gsp_phys_rmcall_guest_pt_win_va( + base_line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + + ((u64)(base_line % + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) << 16); + last_va = gsp_phys_rmcall_guest_pt_win_va( + (base_line + nlines - 1u) / + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) + + ((u64)((base_line + nlines - 1u) % + GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE) << 16); + /* one contiguous run: never straddle a pool block boundary */ + if (last_va != first_va + ((u64)(nlines - 1u) << 16)) + return false; + /* + * Consume the lines before writing: a failure mid-way leaves the + * partially written lines consumed (never reused), so the monotonic + * line invariant - a cached line can never shadow a later mapping - + * holds unconditionally. + */ + boot_stage.rmcall_guest_pt_next_line = base_line + nlines; + for (i = 0; i < nlines; i++) { + u32 line = base_line + i; + u32 page = line / GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + u32 lip = line % GSP_PHYS_RMCALL_GUEST_PT_LINES_PER_PAGE; + u8 *pt; + + if (gsp_phys_rmcall_gr_scratch_win_install_locked(page)) + return false; + pt = gsp_phys_rmcall_guest_pt_page_va(page); + if (!pt) + return false; + for (j = 0; j < GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES; j++) { + u64 k = (u64)i * GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES + + j; + u64 pv, hpa = 0; + u32 pi; + + if (k >= map_pages) + break; + pv = va + (k << 12); + for (pi = 0; pi < req->n_pages; pi++) { + if ((req->pages[pi].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == pv && + req->pages[pi].hpa) { + hpa = req->pages[pi].hpa; + break; + } + } + if (!hpa) { + struct gsp_sema_gr_reloc r; + + if (gsp_sema_gr_reloc_resolve(pv, &r) || + !r.hpa) { + /* + * A declared page must resolve; a + * slack-tail page past the guest + * allocation gets the dummy so the + * engine's overdraw vanishes. + */ + if (k < npages) + return false; + hpa = gsp_phys_gr_surf_dummy_dma_locked(); + if (!hpa) + return false; + } else { + hpa = r.hpa; + } + } + put_unaligned_le64( + gsp_phys_rmcall_pte(hpa), + pt + (size_t)(GSP_PHYS_RMCALL_GUEST_PT_CACHE_PTES * + lip + + j) * + 8); + } + dma_wmb(); + } + *alias_out = first_va; + return true; +} + +/* + * SESSION 153-CONTINUED-27: pre-pass over a raw 3D window. Scans the + * colour-target runs (C597 INC 0x800+j*0x40 [upper, lower, w, h, fmt, + * blksz] and - same surface, shape seen on the TWOD subchannel - INC + * 0x218 cnt>=4 [w, h, upper, lower]) and aliases each big surface once + * per boot so the relocation pass can rewrite its operands to the alias + * (r992/r993 evidence: the X screen 0x120400000, 1280x720 fmt 0xe6, is + * the draw's render target AND the TWOD fill's destination). Small + * surfaces keep the historical one-pair shadow relocation. Called with + * gsp_phys_rm_lock held. + */ +static void gsp_phys_gr_surface_alias_scan_locked( + const u32 *win, u32 n, const struct gsp_phys_gr_submit *req) +{ + u32 off = 0; + + while (off < n) { + u32 hdr = win[off]; + u32 op = (hdr >> 29) & 0x7u; + u32 cnt = (hdr >> 16) & 0xfffu; + u32 addr = (hdr & 0x1fffu) << 2; + u32 span = 1u; + + if ((op == 1 || op == 3 || op == 5) && cnt && + off + 1u + cnt <= n) { + span = 1u + cnt; + } else { + off++; + continue; + } + if (op == 1 && ((addr >= 0x0800 && addr < 0x0c00 && + (addr & 0x3fu) == 0u && cnt >= 4) || + (addr == 0x218 && cnt >= 4))) { + const u32 *d = &win[off + 1u]; + u64 va; + u32 w, h, fmt; + u64 bytes, npages; + u32 sdi; + u64 alias = 0; + + if (addr == 0x218) { + va = ((u64)(d[2] & 0xffu) << 32) | + (u64)(d[3] & 0xfffff000u); + w = d[0]; + h = d[1]; + fmt = 0xe6u; /* TWOD run carries fmt on + * its own IMMD; the surface + * class here is 32bpp */ + } else { + va = ((u64)(d[0] & 0xffu) << 32) | + (u64)(d[1] & 0xfffff000u); + w = d[2]; + h = d[3]; + fmt = d[4]; + } + if (!va || (va & 0xfffu) || fmt != 0xe6u || + !w || !h || w > 8192u || h > 8192u) + goto next; + bytes = (u64)w * 4u * (u64)h; + npages = (bytes + GSP_PHYS_PAGE_SIZE - 1) >> 12; + if (npages < 2u || npages > 4096u) + goto next; + for (sdi = 0; sdi < gsp_phys_gr_surf_n; sdi++) + if (gsp_phys_gr_surf[sdi].va == va) { + /* r1510: content-gated screen + * code-heap quarantine (no-op when + * the surface is clean). */ + (void)gsp_sema_surf_clean_restore( + va, npages); + goto next; /* already aliased */ + } + if (gsp_phys_gr_surface_alias_locked(req, va, npages, + &alias)) { + if (gsp_phys_gr_surf_n < + ARRAY_SIZE(gsp_phys_gr_surf)) { + gsp_phys_gr_surf[gsp_phys_gr_surf_n].va = + va; + gsp_phys_gr_surf[gsp_phys_gr_surf_n] + .alias = alias; + gsp_phys_gr_surf[gsp_phys_gr_surf_n] + .npages = npages; + gsp_phys_gr_surf_n++; + } + pr_info("vgsp-gr: SURFACE alias va=0x%llx pages=%llu (%ux%u fmt=0x%x) -> 0x%llx (contiguous shadow run; operands rewritten)\n", + va, npages, w, h, fmt, alias); + } else { + pr_info("vgsp-gr: SURFACE alias FAILED va=0x%llx (%ux%u fmt=0x%x): operands fall to the per-page path\n", + va, w, h, fmt); + } + } +next: + off += span; + } +} + +/* + * SESSION 153-CONTINUED-66: write a built method block into the module + * pushbuffer. The block may exceed one 4 KB page (GSP_PHYS_RMCALL_GR_METHOD_ + * WORDS is 2048), so copy page-wise: page 0 is the registered BUF_PB chunk, + * page 1 is the PTE-planted rmcall_gr_pb2 page at VA_PB2. The two pages are + * contiguous in GPU VA (VA_PB..VA_PB+0x2000), so the GPFIFO entry's single + * fetch address + length walks both. A short/absent page-2 plant leaves the + * tail unwritten and the caller's bound check keeps such launches <= 4 KB. + */ +static void gsp_phys_gr_pb_copy(const u32 *methods, u32 words) +{ + struct gsp_phys_boot_chunk *pb = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_PB]; + u32 per = GSP_PHYS_PAGE_SIZE / 4u; + u32 done = 0; + + while (done < words) { + struct gsp_phys_boot_chunk *c; + u32 page = done / per; + u32 off = done % per; + u32 n = words - done; + u8 *dst; + + if (n > per - off) + n = per - off; + if (!page) + c = pb; + else if (page == 1u) + c = &boot_stage.rmcall_gr_pb2; + else + return; + dst = gsp_phys_chunk_page_va(c, 0); + if (page == 2u || page == 3u) { + /* + * SESSION 153-CONTINUED-77: pages 1 and 2 of the pb2 + * chunk are the third/fourth pushbuffer pages + * (VA_PB3 slot 0x103, VA_PB4 slot 0x104). + */ + if (boot_stage.rmcall_gr_pb2.pages < page) + return; + dst = gsp_phys_chunk_page_va(&boot_stage.rmcall_gr_pb2, + page - 1u); + } + if (!dst) + return; + memcpy(dst + (size_t)off * 4u, methods + done, (size_t)n * 4u); + done += n; + } +} + +/* + * SESSION 153-CONTINUED-95: CTX-BIND launch receipt - is the client's + * promoted GR context buffer engine-visible on the replay channel? + * + * The guest driver promotes its per-channel context buffers through + * NV2080_CTRL_CMD_GPU_PROMOTE_CTX entries (MAIN bufferId 0 and PATCH + * bufferId 2 carry bInitialize=1 on a fresh context). The emulated GSP-RM + * answers SUCCESS and records the statements (gsp_sema_service_ctx_ + * promote_note), but it does not install them anywhere: the replay + * channel's engine context is the buffer the kernel's own ladder built + * (server FBMEM MAIN, mapped at GSP_PHYS_RMCALL_VA_GR_CTX and committed by + * the mode-35 lane promote). This receipt logs, at the first launches of a + * boot, the exact promote statements next to the engine's binding so the + * "context state never replayed" hypothesis is decided by bytes: the + * client's MAIN/PATCH pa/va/size/bInitialize, and the first words of the + * client MAIN buffer read through the GPA->HPA map (a zeroed buffer is the + * expected state before a server-side ctx init that the emulation never + * performs). + */ +static void gsp_phys_ctx_bind_receipt(void) +{ + static u32 receipt_n; + u64 cpa = 0, cva = 0, csz = 0, ppa = 0, pva = 0, psz = 0; + u32 cattr = 0, cinit = 0, pattr = 0, pinit = 0; + bool have_main, have_patch; + + if (receipt_n >= 4u) + return; + receipt_n++; + have_main = gsp_sema_service_ctx_promote_get(GSP_PHYS_RMCALL_ENGINE_GR0, + 0u, &cpa, &cva, &csz, + &cattr, &cinit) == 0; + have_patch = gsp_sema_service_ctx_promote_get(GSP_PHYS_RMCALL_ENGINE_GR0, + 2u, &ppa, &pva, &psz, + &pattr, &pinit) == 0; + pr_info("vgsp-gr: CTX-BIND client-main=%s pa=0x%llx va=0x%llx size=0x%llx attr=0x%x binit=%u; client-patch=%s pa=0x%llx va=0x%llx size=0x%llx binit=%u; replay-ctx pa=0x%llx size=0x%x fbmem=%u va=0x%llx srv_ctxva=%u; installed=0 (engine ctx is the ladder's buffer, NOT the client promote)\n", + have_main ? "seen" : "none", cpa, cva, csz, cattr, cinit, + have_patch ? "seen" : "none", ppa, pva, psz, pinit, + boot_stage.rmcall_ctx_pa, boot_stage.rmcall_ctx_size, + boot_stage.rmcall_ctx_fbmem ? 1u : 0u, + boot_stage.rmcall_ctx_va, boot_stage.rmcall_srv_ctxva ? 1u : 0u); + if (have_main && cpa) { + u64 hpa = 0; + + if (gsp_phys_gpa_to_hpa_vgpu(NULL, cpa & ~(u64)(GSP_PHYS_PAGE_SIZE - 1), &hpa) == 0 && + pfn_valid(hpa >> PAGE_SHIFT)) { + const u32 *w = (const u32 *)__va(hpa); + + pr_info("vgsp-gr: CTX-BIND client-main content[0..7]=%08x %08x %08x %08x %08x %08x %08x %08x (gpa=0x%llx hpa=0x%llx)\n", + get_unaligned_le32((const u8 *)&w[0]), + get_unaligned_le32((const u8 *)&w[1]), + get_unaligned_le32((const u8 *)&w[2]), + get_unaligned_le32((const u8 *)&w[3]), + get_unaligned_le32((const u8 *)&w[4]), + get_unaligned_le32((const u8 *)&w[5]), + get_unaligned_le32((const u8 *)&w[6]), + get_unaligned_le32((const u8 *)&w[7]), + cpa, hpa); + } else { + pr_info("vgsp-gr: CTX-BIND client-main content: GPA 0x%llx -> HPA translation unavailable (no guest RAM registration yet)\n", + cpa); + } + } +} + +/* + * SESSION 153-CONTINUED-95: engine context readback (gsp-phys-ctx-dump). + * + * What the DS validates SPHs against is the *engine's* context buffer, not + * this host's view of anything the client promoted. The replay channel's + * context is the ladder's buffer (server FBMEM MAIN) mapped into the + * channel VAS at GSP_PHYS_RMCALL_VA_GR_CTX with VIDMEM PTEs. The host has + * no aperture to FBMEM (no BAR1 window), so the copy engine reads it: one + * 4 KB CE copy per page from the ctx VA to a host page placed on a fresh + * never-reused scratch fetch line, polled on the module semaphore exactly + * like the P6a guest copies. Runs once per boot at the first launch (the + * channel is idle: single-inflight work item), while the ladder's channel + * is not server-memory and the identity DMA domain holds. Bounded: stops + * at the first refused/timed-out page and records how many bytes landed. + */ +#define GSP_PHYS_CTX_RB_BYTES 0x140000u /* MAIN 0x133000 + PATCH 0x5000 + slack */ + +static u8 *gsp_phys_ctx_rb_buf; +static u32 gsp_phys_ctx_rb_len; +static bool gsp_phys_ctx_rb_tried; + +static const u8 *gsp_phys_ctx_readback_get(u32 *len) +{ + if (len) + *len = READ_ONCE(gsp_phys_ctx_rb_len); + return gsp_phys_ctx_rb_buf; +} + +/* + * SESSION 153-CONTINUED-96: re-arm the one-shot readback. A write to the + * gsp-phys-ctx-read debugfs file clears the tried flag (and the length), so + * the next forwarded launch re-runs the copy with the current server state. + * Safe from any context: the readback itself only runs in the single-inflight + * launch path and re-checks every gate. + */ +static void gsp_phys_ctx_readback_rearm(void) +{ + WRITE_ONCE(gsp_phys_ctx_rb_len, 0); + WRITE_ONCE(gsp_phys_ctx_rb_tried, false); +} + +static void gsp_phys_ctx_readback_try(void) +{ + struct gsp_phys_boot_chunk *sem = + &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_SEM]; + void __iomem *bar0; + struct page *pg; + u64 hpa; + u32 i, bytes = 0; + u32 ctx_pages, patch_pages = 0; + + if (READ_ONCE(gsp_phys_ctx_rb_tried)) + return; + if (!boot_stage.rmcall_ctx_valid || !boot_stage.rmcall_ctx_pa) + return; + /* + * SESSION 153-CONTINUED-96: the copies run on the secondary COPY0 + * lane (gsp_phys_rmcall_ce_lane_bringup). CE class methods are never + * legal on the GR0 ladder channel, so the lane's readiness (alloc + + * BIND/schedules + its own token) is the hard requirement here; the + * GR channel pieces are not used by the copy. Gate on the pieces the + * lane submission needs: claimed, RM init-done, not unloading. + */ + if (!boot_stage.rmcall_ce_lane_valid || !boot_stage.rmcall_ce_token) + return; + if (gsp_phys_backend_state() != GSP_PHYS_BACKEND_READY) + return; + if (!gsp_phys_gpu_claimed() || !boot_stage.rm_init_done || + boot_stage.rm_unload_submitted) + return; + if (gsp_phys_dma_domain_identity() != 1) + return; + bar0 = gsp_phys_gpu_bar0(); + if (!bar0) + return; + if (!gsp_phys_ctx_rb_buf) { + gsp_phys_ctx_rb_buf = vmalloc(GSP_PHYS_CTX_RB_BYTES); + if (!gsp_phys_ctx_rb_buf) + return; + } + /* + * SESSION 153-CONTINUED-96: copy the WHOLE MAIN context (the golden + * register image the engine restores) and the PATCH buffer (the + * context patch area the DS's SPH region may live in), in that order, + * after the 64 KB r1323/r1324 first run only proved the head. The + * dump file serves MAIN first, then PATCH; lengths are logged. + */ + ctx_pages = DIV_ROUND_UP(boot_stage.rmcall_ctx_size, + GSP_PHYS_PAGE_SIZE); + if (ctx_pages > (GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE)) + ctx_pages = GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE; + if (boot_stage.rmcall_golden_fb_aux && boot_stage.rmcall_patch_size) { + patch_pages = DIV_ROUND_UP(boot_stage.rmcall_patch_size, + GSP_PHYS_PAGE_SIZE); + if (ctx_pages + patch_pages > + GSP_PHYS_CTX_RB_BYTES / GSP_PHYS_PAGE_SIZE) + patch_pages = GSP_PHYS_CTX_RB_BYTES / + GSP_PHYS_PAGE_SIZE - ctx_pages; + } + pg = alloc_page(GFP_KERNEL); + if (!pg) + return; + hpa = (u64)page_to_phys(pg); + for (i = 0; i < ctx_pages + patch_pages; i++) { + u64 src_base = (i < ctx_pages) ? + GSP_PHYS_RMCALL_VA_GR_CTX : + GSP_PHYS_RMCALL_VA_GR_PATCH; + u64 page_off = (i < ctx_pages) ? + (u64)i * GSP_PHYS_PAGE_SIZE : + (u64)(i - ctx_pages) * GSP_PHYS_PAGE_SIZE; + u64 copy_src_va = src_base + page_off; + u64 map_src_va = 0, dst_va = 0, sem_va = 0; + u32 poll = 0; + unsigned long deadline; + int ret; + + mutex_lock(&gsp_phys_rm_lock); + ret = gsp_phys_rmcall_guest_map_locked(hpa, hpa, 0, &map_src_va, + &dst_va, &sem_va); + if (ret) { + mutex_unlock(&gsp_phys_rm_lock); + break; + } + memset((u8 *)__va(hpa), 0xab, GSP_PHYS_PAGE_SIZE); + ret = gsp_phys_rmcall_ce_lane_submit_locked(bar0, + copy_src_va, dst_va, GSP_PHYS_RMCALL_VA_SEM, + GSP_PHYS_PAGE_SIZE, + GSP_PHYS_RMCALL_SEM_PAYLOAD, NULL); + mutex_unlock(&gsp_phys_rm_lock); + if (ret) { + pr_warn("vgsp-gr: CTX-READBACK page %u submit refused (%d) after %u bytes; stopping\n", + i, ret, bytes); + break; + } + deadline = jiffies + msecs_to_jiffies(250); + for (;;) { + poll = READ_ONCE(*(u32 *)gsp_phys_chunk_page_va(sem, 0)); + if (poll == GSP_PHYS_RMCALL_SEM_PAYLOAD) + break; + if (time_after_eq(jiffies, deadline)) { + poll = 0; + break; + } + gsp_phys_rmcall_wait_quantum(); + } + if (poll != GSP_PHYS_RMCALL_SEM_PAYLOAD) { + pr_warn("vgsp-gr: CTX-READBACK page %u (src VA 0x%llx) no CE release within 250 ms after %u bytes; stopping\n", + i, copy_src_va, bytes); + break; + } + memcpy(gsp_phys_ctx_rb_buf + (size_t)i * GSP_PHYS_PAGE_SIZE, + (const u8 *)__va(hpa), GSP_PHYS_PAGE_SIZE); + bytes += GSP_PHYS_PAGE_SIZE; + } + __free_page(pg); + WRITE_ONCE(gsp_phys_ctx_rb_len, bytes); + WRITE_ONCE(gsp_phys_ctx_rb_tried, true); + pr_info("vgsp-gr: CTX-READBACK engine ctx pa=0x%llx copied=%u bytes on the CE lane (MAIN %u pages then PATCH %u pages; server/own buffer, not the client promote; WPR2 untouched)\n", + boot_stage.rmcall_ctx_pa, bytes, ctx_pages, patch_pages); +} + int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out) { @@ -18874,6 +28243,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * under the warning bound. */ static u32 methods[GSP_PHYS_RMCALL_GR_METHOD_WORDS]; + static u32 stream_patch[GSP_PHYS_RMCALL_GR_METHOD_WORDS]; static u32 qmd[GSP_PHYS_GR_QMD_WORDS]; static struct gsp_phys_fault_info fault; unsigned long start = jiffies; @@ -18884,6 +28254,34 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, const u8 *guest_param_pg; u32 want_qmd = 0, want_ch = 0; bool qmd_rel = false, ch_rel = false; + /* + * r929 (Path A): this submission is a raw 3D method window. The QMD + * machinery (qmdwin relocation, QMD prewrite, release0 redirect, + * PCAS) does not apply: the block is SET_OBJECT C597 + the window + + * the channel release tail, and completion is the channel release + * alone (qmd_rel stays off so the poll does not wait for a QMD + * release word no engine will write). + */ + bool stream3d = gsp_phys_rmcall_gr_3d && req && req->stream_n; + /* + * CONT-73: fresh direct-write witness for this submission. The + * window/attr patch passes below record every relocated operand + * destination's guest page; the pre-doorbell capture and the retire + * report read exactly this launch's set (single-inflight, statics). + */ + gsp_phys_gr_wit_reset(); + const u32 *win3d = NULL; + u32 win3d_n = 0; + /* + * r1483: the SURF prelude emitted for THIS launch. The RT0 bind + * (sema_gl_rt0_bind) used to carry a constant 0xa000000 target; + * that is only correct when this launch's data-rec base is the 0xa + * half. The PS writes RT0 while the client binds RT1, so RT0 must + * name the same surface as the SURF prelude or half the launches + * store into the other half's (untracked) bounce window. When set, + * the RT0 run copies the emitted SURF run's target words. + */ + bool surf_pre_emitted = false; bool qmd_rel_redirected = false; /* r420b: release0 -> module sem */ u64 qmd_rel_hpa = 0, ch_rel_hpa = 0; u64 ch_sem_va = GSP_PHYS_RMCALL_VA_SEM + 4; @@ -18891,9 +28289,10 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, u8 *qmd_dst_mem = NULL; u32 qmd_rel_off = 0, ch_rel_off = 0; u32 poll = 0, w0 = 0, w1 = 0, grpoll_n = 0; + static u32 post_spin_n; bool sem_gated = false; - static u32 sked_prev[11]; - static u32 sked_sample[11]; + static u32 sked_prev[28]; + static u32 sked_sample[28]; static u32 sked_tick; static bool sked_first; /* @@ -18906,7 +28305,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * stack). Safe as static: the only caller is the single-inflight * ordered gsp-phys-work item. */ - static u32 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; + static u64 bounce_csum_in[GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES]; /* * r378: which guest page's content each bounce slot holds this * launch (set by the copy-in fills, cleared per submission). The @@ -18920,6 +28319,14 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, sked_tick = 0; sked_first = true; guest_param_pg = NULL; + /* + * SESSION 153-CONTINUED-95: prove the client's promoted GR context + * buffer binding at the first launches, then read the engine's own + * context buffer back for byte-level inspection (both bounded, both + * diagnostic; no behavior change to the launch itself). + */ + gsp_phys_ctx_bind_receipt(); + gsp_phys_ctx_readback_try(); u32 qmd_win_pde = 0, qmd_win_installed = 0; int qmd_win_ret = 0; bool qmd_win_diag = false; @@ -18929,6 +28336,22 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, bool rearmed = false; u32 attempts = 0; /* + * SESSION 153-CONTINUED-48: one bounded priv-error recovery attempt + * per launch (sema_gr_wedge_recover), with a per-boot tally for the + * report. Static like the other submit scratch: the single-inflight + * ordered work item is the only caller. + */ + bool wedge_recover_done = false; + static u32 wedge_recoveries; + /* + * SESSION 153-CONTINUED-72: the MME-FAST pre-reset spin can observe + * the 0xbadf poison and then the FECS clears it before the poll + * loop's first recovery check reads grfifo (the r1149 race: recovery + * missed -> prelude latch stuck -> engine wedged for the boot). The + * observation is latched here and OR-ed into the WEDGE-RECOVER gate. + */ + bool fast_poison = false; + /* * F2/F4 state: the fast first-timeout re-arm runs once per launch; the * deposit readback is one-shot per launch (the first timeout is the * evidence window - the old r198 post-mortem only ran after attempt 3, @@ -18953,6 +28376,17 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, fast_rearmed = false; deposit_rb_done = false; + gsp_phys_gr_mme_pairs_emitted = false; /* r1453 PAIR-CHAIN */ + gsp_phys_gr_mme_launch_gen = 0; + /* + * r1454 PAIR-CHAIN-ENTRY-ELIDE: per-launch scratch only - the + * cross-launch tracker (pair_open/split) must survive this site + * (it is reset at the two context wipes instead). Clearing the + * entry snapshot here is fail-closed: a launch whose window never + * reaches gsp_phys_gr_pair_chain_fix cannot elide. + */ + gsp_phys_gr_mme_pair_open_entry = false; + gsp_phys_gr_mme_pair_open_entry_split = 0; gsp_phys_gr_codehash_done = false; /* CODEHASH: once per launch */ /* * M91 FAULT-CAP: snapshot the fault epoch per launch, so the capture @@ -19004,6 +28438,26 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, gsp_phys_gr_refused++; return -EINVAL; } + /* + * r564: refuse a degenerate data set BEFORE the engine ever sees it. + * r565 evidence: the fill dropped 4 argument bases (GRFILL-DROP), + * RDW showed nrecs=2 where healthy launches run 968-969, the launch + * was still submitted and the backend timed out after ~19 s + * ("GR forward failed rc=-110"), after which the tail authored the + * software release and the client kept spinning on a completion for a + * kernel that never ran - the worst of both (20 s of backend + * occupancy plus a false completion). Refusing up front fails fast + * into the same software-completion fallback without touching the + * engine, and cannot wedge it for the next client. + */ + if (req->n_data < gsp_phys_gr_min_data_value()) { + gsp_phys_gr_refused++; + pr_err_ratelimited("vgsp-gr: GR submit refused: degenerate data set n_data=%u (min %u) launch_va=0x%llx\n", + req->n_data, + gsp_phys_gr_min_data_value(), + req->push_va); + return -ENODATA; + } if (!gsp_phys_backend_gr_ready() || !gsp_phys_guest_mem_enabled() || gsp_phys_dma_domain_identity() != 1 || !gsp_phys_guest_ram_active()) { gsp_phys_gr_refused++; @@ -19023,6 +28477,41 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, ret = -EIO; goto out; } + /* + * SESSION 153-CONTINUED-60: all-zero 3D-window skip. The client's push + * page is cyclically zeroed/rewritten (CONT-45); a capture that lands + * on the zero phase still forwards an all-zero method window. Live + * r1107: such a window launched (3D raw dump words=252, every word 0), + * the engine trapped SET_VERTEX_ATTRIBUTE_A(2) (trap=00001168 + * data_lo=0x50) and escalated to a priv-error poison (grfifo= + * 0xbadf1201, mme_pc=513) that wedged the GR lane for the rest of the + * boot; r1031 had already seen the same class fault the PBDMA on the + * zero lead (PB-FAULT-DUMP P0). A zero window carries no method to + * execute and nothing to store: skip the hardware handoff entirely - + * no shadow relocation, no engine contact, no wedge - and return the + * distinct rc -EBADMSG so the service authors the software release and + * suppresses the synthetic fault (a frame with no content deserves no + * client-visible engine error). + */ + if (stream3d) { + u32 zwi; + bool zerowin = req->stream_n != 0; + + for (zwi = 0; zwi < req->stream_n; zwi++) { + if (req->stream[zwi]) { + zerowin = false; + break; + } + } + if (zerowin) { + gsp_phys_gr_zerowin_skips++; + pr_warn_ratelimited("vgsp-gr: ZEROWIN-SKIP h=0x%x push=0x%llx window=%u all-zero (no engine submit; software release) n=%llu\n", + req->h_object, req->push_va, req->stream_n, + gsp_phys_gr_zerowin_skips); + ret = -EBADMSG; + goto out; + } + } gsp_phys_gr_submitted++; gsp_phys_gr_override_apply_param(); @@ -19084,6 +28573,32 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, } } /* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default + * 0 = off): the same cache-control word's bit 31 (the V01_06-layout + * SHADER_CONSTANT_CACHE position) is never set by the captured guest + * QMDs. The forwarded param page is rewritten in place between launches + * (the two-bank win0 pool), so a stale constant-cache line can serve + * the PREVIOUS launch's parameters and the current launch's output + * pages stay untouched - the post-dispatch no-store signature. Mode 1 + * sets bit 31; mode 2 forces the whole invalidate set (bits 24..31). + * Only when the QMD carries the driver's standard cache-control + * signature (bits 26..29) so an unknown QMD revision is never touched. + */ + if (gsp_phys_gr_ccache_inv_value() && + (qmd[5] & 0x3c000000u) == 0x3c000000u) { + u32 mode = gsp_phys_gr_ccache_inv_value(); + + if (mode == 2) + qmd[5] |= 0xff000000u; + else + qmd[5] |= 1u << 31; + if (!gsp_phys_r449_ccache_log) { + gsp_phys_r449_ccache_log = true; + pr_info("vgsp-gr: r449 constant-cache invalidate mode=%u applied (w5=0x%08x)\n", + mode, qmd[5]); + } + } + /* * r433 RELEASE_MEMBAR_TYPE A/B (gsp_phys_gr_rel_membar): force the * forwarded QMD's RELEASE_MEMBAR_TYPE to FE_SYSMEMBAR (QMDV00_06 * word 11 bit 14) so the front-end flushes its pending system-memory @@ -19117,13 +28632,16 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * a fresh shadow line so the guest observes its own QMD release; else * point release0 at the module semaphore so the host observes engine * completion. A reduction release is never redirected (its arithmetic - * would change). + * would change). r929: skipped entirely for a 3D stream replay - + * there is no QMD, so there is no release0 to point anywhere. */ - if (((qmd[4] >> 10) & 1)) { + if (!stream3d && ((qmd[4] >> 10) & 1)) { u32 red_en = (qmd[24] >> 26) & 1; u32 one_word = (qmd[24] >> 31) & 1; bool rel_mod = gsp_phys_rmcall_gr_rel_mod_value() != 0; + int rel_map_ret = 0; + if (!rel_mod && req->rel_hpa && one_word && !red_en && n_shadow < GSP_PHYS_RMCALL_GR_SHADOW_PAGES) { u64 sh_va = 0; @@ -19143,6 +28661,7 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, 0xffu); qmd[25] = req->rel_payload; } else { + rel_map_ret = ret; ret = 0; } } @@ -19171,6 +28690,23 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, * completed on the guest word (the r414 saxpy hang). */ qmd_rel_redirected = true; + /* + * r633 diag: name the exact redirect cause. rel_hpa==0 + * means the caller could not resolve the release page + * (capture resolution or PA->HPA); map_ret!=0 means the + * page was resolved but its shadow mapping failed. + */ + { + static u32 qmdrel_redir_log; + + if (qmdrel_redir_log < 32) { + qmdrel_redir_log++; + pr_info("vgsp-gr: qmd-rel redirect: rel_hpa=0x%llx map_ret=%d one_word=%u red_en=%u rel_mod=%d n_shadow=%u pay=%u\n", + req->rel_hpa, rel_map_ret, + one_word, red_en, rel_mod ? 1 : 0, + n_shadow, req->rel_payload); + } + } } } @@ -19268,7 +28804,53 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, pg_va = pg->guest_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); pr_info("vgsp-gr: page[%u] va=0x%llx hpa=0x%llx qmd_word=%u\n", i, pg->guest_va, pg->hpa, pg->qmd_word); - if (pg->hpa) { + /* + * r1497/r1498: an FB *fetch* page (a QMD pointer pair, not + * the data-fill surface) whose shadow is empty is the active + * indexed draw's vertex/index class. Map the real FBMEM page + * through a VID leaf instead of the empty shadow. The + * render-target/surface pages (QMD word NONE) must keep + * their shadow identity: the engine's stores have to land in + * the emulated page so the copy-out can ship them back to + * the guest (r1498: mapping the surface to real FB made the + * frame invisible to the guest). + */ + if (pg->hpa && pg->qmd_word != 48u && + pg->qmd_word != GSP_PHYS_GR_QMD_WORD_NONE) { + u64 fb_off = 0; + + if (gsp_phys_gr_fetch_vid_off_locked(pg_va, + &fb_off)) { + ret = gsp_phys_rmcall_gr_map_va_vid_locked( + bar0, pg_va, fb_off, epoch); + if (!ret) { + mapped = true; + gsp_phys_gr_vid_map_pages++; + if (gsp_phys_gr_vid_log < 32u) { + gsp_phys_gr_vid_log++; + pr_info("vgsp-gr: FETCH-VID-MAP va=0x%llx fb=0x%llx qmd=%u h=0x%x push=0x%llx\n", + pg_va, fb_off, + pg->qmd_word, + req->h_object, + req->push_va); + } + } else if (ret == -ENOSPC) { + gsp_phys_gr_pool_exhausted++; + } + } + } + /* + * SESSION 149 R15 (M2-S1): with the window-0 shadow path + * enabled, force the program page (QMD word 48) OFF the + * post-commit Tier-1 install. r801b proved that install is + * ineffective for a late GR page (the engine kept reading + * the empty guest page: CODEHASH h0=0, FAULT-CAP + * code_src=guest-hpa); skipping it lets the page fall to + * the win0 shadow copy + w48/49 rewrite line, which is the + * proven one. Every other page keeps its Tier-1 route. + */ + if (!mapped && pg->hpa && !(pg->qmd_word == 48u && + gsp_phys_rmcall_gr_shadow_win0_value())) { ret = gsp_phys_rmcall_gr_map_va_locked(bar0, pg_va, pg->hpa, epoch); if (!ret) { @@ -19375,6 +28957,80 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, pg->guest_va, sh_va, slot); } } + /* + * SESSION 153-CONTINUED-89: inline records that land on a + * qmd_word-NONE page. The record-apply loop below belongs to + * the QMD-word shadow-*copy* path and never runs for these + * pages; they are resolved to the guest HPA directly (Tier-1 + * identity, contiguous shadow-line extension, or the alias + * above), so authoring the record into pg->hpa is exactly + * what the engine will read. The concrete case: the real + * captured upload at 0x4350f00 (1408 B) crosses its 4 KB + * page - the head (256 B) lands on the program page 0x4350000 + * and applies, but the tail page 0x4351000 is qmd_word=NONE + * and was silently dropped, leaving the draw's winning j=1 + * program (the later bind) truncated at 256 B at the engine + * (the DS rejects the SASS it cannot fetch: SPH 9/11). + */ + if (mapped && pg->hpa && + pg->qmd_word == GSP_PHYS_GR_QMD_WORD_NONE) { + u32 jx; + + for (jx = 0; jx < req->n_inl; jx++) { + const struct gsp_phys_gr_inline *inl = + &req->inl[jx]; + u32 len; + + if ((inl->dst_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) != pg_va) + continue; + len = inl->len; + if (len > GSP_PHYS_GR_INLINE_BYTES) + len = GSP_PHYS_GR_INLINE_BYTES; + if ((u64)(inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)) + len > + GSP_PHYS_PAGE_SIZE) + continue; + memcpy((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)), + inl->data, len); + { + static u32 inl_alias_log; + static u32 inl_alias_writer_log; + + if (inl_alias_log < 12u) { + inl_alias_log++; + pr_info("vgsp-gr: INLINE-APPLY-ALIAS dst=0x%llx len=%u pa=0x%llx first=%08x (qmd_word=NONE page authored)\n", + inl->dst_va, len, + pg->hpa, + len >= 4 ? + get_unaligned_le32(inl->data) : + 0u); + } + if (inl_alias_writer_log < 16u && + len >= 128u) { + inl_alias_writer_log++; + pr_info("vgsp-gr: INLINE-WRITER-ALIAS h=0x%x push=0x%llx dst=0x%llx len=%u pg_va=0x%llx pa=0x%llx ap=%u fb=0x%llx hpa=0x%llx first8=%08x %08x %08x %08x\n", + req->h_object, req->push_va, + inl->dst_va, len, pg_va, + pg->reserved & + 0x3fffffffuLL, + (pg->reserved >> 31) ? + 2u : 1u, + (pg->reserved >> 31) ? + (u64)(pg->reserved & + 0x3fffffffu) : + 0ULL, + pg->hpa, + len >= 4 ? get_unaligned_le32(inl->data) : 0u, + len >= 8 ? get_unaligned_le32(inl->data + 4) : 0u, + len >= 12 ? get_unaligned_le32(inl->data + 8) : 0u, + len >= 16 ? get_unaligned_le32(inl->data + 12) : 0u); + } + } + } + } if (!mapped && pg->qmd_word != GSP_PHYS_GR_QMD_WORD_NONE && pg->qmd_word + 1 < GSP_PHYS_GR_QMD_WORDS) { struct gsp_phys_boot_chunk *sh = @@ -19556,6 +29212,118 @@ int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, memcpy(sh_page + (inl->dst_va & (GSP_PHYS_PAGE_SIZE - 1)), inl->data, len); + /* + * SESSION 153-CONTINUED-10: a stream3d (3D + * method-window) submission maps the engine's + * VAs straight at the GUEST page HPA + * (gsp_phys_rmcall_gr_map_shadow_locked + * installs the guest PTE - there is no + * shadow-chunk copy in that path), so the + * engine's I-fetch reads the guest page, not + * sh_page. The GLX program page is never + * written by the guest itself (the driver's + * uploads never ran), and the r945 evidence is + * exactly that: "Graphics Exception: Shader + * Program Header 11/18 Error" on the draw + * whose inline prog-hex blob had only reached + * this shadow copy. Author the inline bytes + * into the guest page too, emulating the + * upload that never executed. + */ + /* + * SESSION 154: a program page whose CODESRC + * verdict keeps it at the guest VA is read by + * the engine directly (code_keep_guest sets + * mapped without a QMD rewrite). The inline + * image MUST land in the guest HPA there even + * when the window was not classified + * stream3d: the shadow copy is then unused and + * the engine fetches the empty guest page. + * r1470: the complete 0x41c0000 window's VS + * applied guest=0 into the shadow while the + * guest page stayed zero (the win0 shadow copy + * then restored the zero page as well). + */ + if ((stream3d || code_keep_guest) && pg->hpa) + memcpy((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1)), + inl->data, len); + /* + * SESSION 153-CONTINUED-11: bounded witness of + * every inline record applied (which program + * page received which image), so the SPH + * delivery is auditable from the log. + */ + { + static u32 inl_apply_log; + static u32 inl_writer_log; + + if (inl_apply_log < 24) { + inl_apply_log++; + pr_info("vgsp-gr: INLINE-APPLY dst=0x%llx len=%u pg_va=0x%llx guest=%u first=%08x %08x\n", + inl->dst_va, len, pg_va, + (stream3d || code_keep_guest) && + pg->hpa ? 1u : 0u, + len >= 4 ? get_unaligned_le32(inl->data) : 0u, + len >= 8 ? get_unaligned_le32(inl->data + 4) : 0u); + } + /* + * r1503 (mission task 1): decisive + * writer receipt for every image-sized + * inline delivery - names the emitting + * launch, the VA, the resolved PA / + * aperture and the first payload words, + * so a delivery that lands in a + * scanout/window-buffer page is + * provable from the log alone. + */ + if (inl_writer_log < 32u && len >= 128u) { + inl_writer_log++; + pr_info("vgsp-gr: INLINE-WRITER h=0x%x push=0x%llx dst=0x%llx len=%u pg_va=0x%llx pa=0x%llx ap=%u fb=0x%llx hpa=0x%llx first8=%08x %08x %08x %08x\n", + req->h_object, req->push_va, + inl->dst_va, len, pg_va, + pg->reserved & + 0x3fffffffuLL, + (pg->reserved >> 31) ? + 2u : 1u, + (pg->reserved >> 31) ? + (u64)(pg->reserved & + 0x3fffffffu) : + 0ULL, + pg->hpa, + len >= 4 ? get_unaligned_le32(inl->data) : 0u, + len >= 8 ? get_unaligned_le32(inl->data + 4) : 0u, + len >= 12 ? get_unaligned_le32(inl->data + 8) : 0u, + len >= 16 ? get_unaligned_le32(inl->data + 12) : 0u); + } + } + /* + * SESSION 153-CONTINUED-87: readback witness. + * For a stream3d window the engine's I-fetch + * walks the GUEST page (the inline bytes were + * just authored there above); read the exact + * engine-visible location back so the fetch + * content is provable from the log - if the DS + * still rejects the header, the bytes are known + * present at the VA it fetched. + */ + if ((stream3d || code_keep_guest) && pg->hpa) { + static u32 inl_rb_log; + const u32 *gw = (const u32 *) + ((u8 *)__va(pg->hpa) + + (inl->dst_va & + (GSP_PHYS_PAGE_SIZE - 1))); + + if (inl_rb_log < 12u) { + inl_rb_log++; + pr_info("vgsp-gr: INLINE-RB dst=0x%llx pa=0x%llx rb=%08x %08x %08x %08x %08x %08x %08x %08x\n", + inl->dst_va, pg->hpa, + gw[0], gw[1], gw[2], + gw[3], gw[4], gw[5], + gw[6], gw[7]); + } + } copied++; } /* @@ -19822,8 +29590,9 @@ gr_page_unmapped: * GPFIFO/PB/REPORT_SEM legs already use. The 2 MiB window is installed * by the ladder itself, so no window ensure is sent for it (a * pdeIndex-0 Tier-1 install would touch the ladder's own PDE). + * r929: a 3D stream replay has no inline QMD at all (skipped). */ - if (gsp_phys_rmcall_gr_qmdwin_value()) { + if (!stream3d && gsp_phys_rmcall_gr_qmdwin_value()) { u8 *qw = boot_stage.rmcall_gr_qmdwin.pages ? gsp_phys_chunk_page_va(&boot_stage.rmcall_gr_qmdwin, 0) : NULL; @@ -19850,9 +29619,9 @@ gr_page_unmapped: * use it as the destination when the caller resolved it; a Tier-1 * install keeps the guest VA (the guest sees its own QMD), a Tier-2 * alias keeps the launch runnable, and an unresolved page keeps the - * module scratch VA. + * module scratch VA. r929: skipped for a 3D stream replay. */ - if (!qmdwin_on && req->qmd_hpa) { + if (!stream3d && !qmdwin_on && req->qmd_hpa) { u64 qmd_page = req->qmd_va & ~(u64)(GSP_PHYS_PAGE_SIZE - 1); u32 qmd_off = (u32)(req->qmd_va & (GSP_PHYS_PAGE_SIZE - 1)); bool dst_ok = false; @@ -19970,10 +29739,1254 @@ gr_page_unmapped: GSP_PHYS_GR_QMD_SENTINEL); } + /* + * r929 v3 (Path A): relocate the 3D window's address operands to + * shadow-pool VAs the engine walkers can reach (see + * gsp_phys_rmcall_gr_3d_patch_window; the r929 v2 FE faulted + * FAULT_PDE on a Tier-1-mapped guest VA). Runs with the rm lock + * held, before the method block is built. + */ + if (stream3d) { + u32 win_n = req->stream_n; + u32 ndrop = 0; + u32 nrel; + + /* + * SESSION 153-CONTINUED-27: a big colour target (the X + * screen) is aliased as ONE contiguous shadow run BEFORE + * the relocation pass runs, so the pass rewrites its + * operands to the alias (see the table above). + */ + gsp_phys_gr_surface_alias_scan_locked(req->stream, + req->stream_n, req); + /* SESSION 153-CONTINUED-59: fresh miss census for this pass + * (window + attr patch below both feed it). */ + gsp_phys_rmcall_gr_3d_miss_n = 0; + nrel = gsp_phys_rmcall_gr_3d_patch_window( + req->stream, req->stream_n, stream_patch, &win_n, + &ndrop, req, &n_shadow, bar0, epoch); + + win3d = stream_patch; + win3d_n = win_n; + /* + * SESSION 153-CONTINUED-69: drop the CALL_MME_MACRO/DATA runs + * of stubbed slots (see gsp_phys_gr_mme_drop_calls). With the + * stub as a pure NOP terminator the parameter port no longer + * drains; the r1142 stall parked the PBDMA on the 16-arg + * CALL_MME_MACRO(57) run and the block never completed. The + * drop turns those calls into nothing instead of a call whose + * parameters have nowhere to go. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_call_drop) && + gsp_sema_gr_sa_stub_value()) { + u32 sa_ent[64]; + u64 sa_mask = gsp_sema_mme_sa_get(sa_ent, 64); + u64 dmask = 0; + u32 ndrop_c = gsp_phys_gr_mme_drop_calls( + stream_patch, win3d_n, sa_mask, sa_ent, + &dmask); + + if (ndrop_c) { + win3d_n -= ndrop_c; + pr_info("vgsp-gr: MME-CALL-DROP %u word(s) of stubbed-slot CALL runs (window=%u) dropped_mask=0x%llx h=0x%x push=0x%llx\n", + ndrop_c, win3d_n, + (unsigned long long)dmask, + req->h_object, req->push_va); + } + } + pr_info("vgsp-gr: 3D window relocated %u operand pair(s) to shadow VAs (window=%u shadow=%u dropped=%u subdrop=%u sdmdrop=%u progkeep=%u)\n", + nrel, win3d_n, n_shadow, ndrop, + gsp_phys_rmcall_gr_3d_last_subdrop, + gsp_phys_rmcall_gr_3d_last_sdmdrop, + gsp_phys_rmcall_gr_3d_last_progkeep); + /* + * SESSION 153-CONTINUED-87: emit the client's missing + * pipeline-stage program binds (see the sema_gl_sph_bind + * comment). The window carries only the j=1 (VTG) pair; the + * real client state also binds j=2 (TESS_CONTROL at + * family+0x200) and j=5 (PIXEL at family+0x0) - both present + * verbatim in the init-push image scraped from guest RAM + * (`20020825 [0,04400200]`, `20020855 [0,04400000]`). Build + * the two runs and push them through the same window + * relocation pass (miss -> the pair stays raw and the whole + * bind is skipped, never a raw client VA to the engine), + * then prepend them ahead of the window. Beef client + * channels only: the family layout is that client's. + */ + bool sph_bind_all = + READ_ONCE(gsp_phys_rmcall_gr_sph_bind); + u32 sph_bind_mask = + READ_ONCE(gsp_phys_rmcall_gr_sph_bind_mask); + + if ((sph_bind_all || sph_bind_mask) && + (req->h_object & 0xfffff000u) == 0xbeef0000u && + req->prog_va && + win3d_n + 96u <= GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + static u32 sph_bind_log; + u64 fam = req->prog_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1); + u32 in[96], out[96]; + u32 nfin = 0, nrun = 0, on = 0, odrop = 0, orel; + u32 bi; + bool bind_binding = + READ_ONCE(gsp_phys_rmcall_gr_sph_bind_binding); + bool pair_safe = + READ_ONCE(gsp_phys_rmcall_gr_sph_bind_pair_safe); + /* + * SESSION 155 CONT 10: per-stage disable mask + * (bit j = synth slot j). Only meaningful when the + * all-slots knob is off, matching the r1561 bool's + * !sph_bind_all gate. + */ + u32 stage_disable_mask = !sph_bind_all ? + (READ_ONCE(gsp_phys_rmcall_gr_sph_stage_disable) & + 0x3fu) : 0u; + bool stage_disable = stage_disable_mask != 0u; + bool emit_j2 = !(stage_disable_mask & 0x04u) && + (sph_bind_all || (sph_bind_mask & 0x04u)); + bool emit_j5 = !(stage_disable_mask & 0x20u) && + (sph_bind_all || (sph_bind_mask & 0x20u)); + /* + * SESSION 155 CONT 8: the DS flags the UNBOUND VTG-class + * stages (VSA/VSB/TCS/GS) in SPH bits 1/2/6/9/10/11/18. + * Mask bits 0x01 and 0x10 add the same synthesized + * per-stage record for VSA (j=0) and GS (j=4), binding + * the client's own VTG image (family+0x900) with the + * GS/VS-CULL types. Default off: bits 0/4 were unused. + * CONT 9: stage_disable suppresses j0/j2/j4 (a disabled + * stage has no program to bind). + */ + bool emit_j0 = !(stage_disable_mask & 0x01u) && + (sph_bind_all || (sph_bind_mask & 0x01u)); + bool emit_j4 = !(stage_disable_mask & 0x10u) && + (sph_bind_all || (sph_bind_mask & 0x10u)); + /* + * SESSION 155 CONT 11: synthesized GS image bind + * (see the sema_gl_sph_gs_img comment). Independent + * of the j4 slot mask: it replaces the disable-only + * GS state with a disabled stage that still has a + * valid SPH operand. + */ + bool emit_gsimg = + READ_ONCE(gsp_phys_rmcall_gr_sph_gs_img); + u32 gs_mode = + READ_ONCE(gsp_phys_rmcall_gr_sph_gs_mode); + bool emit_raster = + READ_ONCE(gsp_phys_rmcall_gr_raster_pre); + bool emit_raster_extra = + READ_ONCE(gsp_phys_rmcall_gr_raster_extra); + u32 gs_addr = READ_ONCE(gsp_phys_rmcall_gr_sph_gs_addr); + u32 gs_ctx = READ_ONCE(gsp_phys_rmcall_gr_sph_gs_ctx); + + /* + * SESSION 155 CONT 9: the prelude is prepended ahead of + * the window, so if the tracker still holds a pair open + * from the previous emitted window every synthesized + * method would run inside BEGIN/END (r1558 EC 0x0d at + * 0x2018). Emit the class's own END as the FIRST word + * of the prelude, matching the event the open BEGIN's + * split mode requires. The pair-chain walk later sees + * this END close the open pair and starts a fresh one at + * the window's own BEGIN. Skipped when + * sema_gl_pair_close_entry is on: that helper inserts + * the same END ahead of the whole window (preludes + * included), so emitting both would double-close. + */ + if (pair_safe && + !READ_ONCE(gsp_phys_rmcall_gr_pair_close_entry) && + READ_ONCE(gsp_phys_gr_mme_pair_open)) { + u32 osplit = + READ_ONCE(gsp_phys_gr_mme_pair_open_split); + u32 ev = (osplit == 1u || osplit == 2u) ? 1u : 0u; + + in[nfin++] = 0x20000000u | (1u << 16) | + (0x1614u >> 2); + in[nfin++] = ev; + pr_info_ratelimited("vgsp-gr: SPH-BIND-PAIR-SAFE close split=%u ev=%u h=0x%x push=0x%llx\n", + osplit, ev, req->h_object, req->push_va); + } + + /* + * SESSION 155 CONT 9: the window is a non-tessellated, + * non-geometry draw. The client's own state disables + * VSA/TCS/TES/GS; the replay channel inherits the + * setup-push records instead, so the DS validates VSB + * in tessellation mode (SPH5/7) and a stale GS program + * (SPH18). Emit the client's disable records FIRST (so + * no later synth bind re-enables them), with the exact + * values the driver's disabled-stage paths write. + * CONT 10: per-bit mask - only the stages whose bit is + * set get a record (bit1 VSB and bit5 PS have no + * unchanged-program record here; bit5 only suppresses + * the synth PS bind). + */ + if (stage_disable) { + if (stage_disable_mask & 0x01u) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2000u >> 2); + in[nfin++] = 0x00u; + } + if (stage_disable_mask & 0x04u) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2080u >> 2); + in[nfin++] = 0x20u; + } + if (stage_disable_mask & 0x08u) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x20c0u >> 2); + in[nfin++] = 0x30u; + } + if (stage_disable_mask & 0x10u) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2100u >> 2); + in[nfin++] = 0x40u; + } + pr_info_ratelimited("vgsp-gr: SPH-STAGE-DISABLE mask=0x%x h=0x%x push=0x%llx\n", + stage_disable_mask, req->h_object, + req->push_va); + } + + /* + * SESSION 153-CONTINUED-89 (2nd half): the client's + * full channel-init pipeline block, harvested from the + * guest-RAM init-push image (guest PA 0x81efc5b4, this + * boot) and identical in the X channel's own captured + * window (methods 0x21c, 0x155c, 0x1574 in that exact + * order). The per-frame window carries none of it - + * on real hardware the client wrote it ONCE at + * channel init, and the replay channel's fresh context + * never receives it, so the DS validates the fetched + * SPHs against unset context state (9/11). Emitted + * verbatim in the client's own order, ahead of the + * stage binds: + * 0x21c <- 0x1001 INVALIDATE_SHADER_CACHES + * (INSTRUCTION|CONSTANT) + * 0x155c <- 1, 0x214fa200, 0 + * TEX_SAMPLER_POOL A/B/C + * 0x1574 <- 1, 0x214fa220, 0x10 + * TEX_HEADER_POOL A/B/C + * 0x310 <- 0x705 SET_SPA_VERSION + * 0x22c <- 0x800 SET_GO_IDLE_TIMEOUT + * 0x1128 <- ffffffff, ffffffff + * DA_OUTPUT_ATTRIBUTE_SKIP_MASK + * ZCULL_BOUNDS(0x196c) is already carried by the + * window itself. All constant runs (no relocation). + */ + in[nfin++] = 0x20000000u | (1u << 16) | + (0x21cu >> 2); + in[nfin++] = 0x1001u; + { + /* + * The pool runs are address-bearing (the patch + * pass's own rules: addr 0x155c/0x1574, cnt >= + * 3) and count toward the relocated-pair + * guard, so emit them only when their page is + * in this launch's page set (r1290: emitted + * unconditionally -> unaccounted pairs -> + * `SPH-BIND skipped (reloc=6/4)` and the whole + * prelude was lost). Both VAs share one page. + */ + bool pool_pg = false; + u32 pk; + + for (pk = 0; pk < req->n_pages; pk++) + if ((req->pages[pk].guest_va & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1)) == + (0x214fa200u & + ~(u64)(GSP_PHYS_PAGE_SIZE - 1))) + pool_pg = true; + if (pool_pg) { + in[nfin++] = 0x20000000u | (3u << 16) | + (0x155cu >> 2); + in[nfin++] = 0x1u; + in[nfin++] = 0x214fa200u; + in[nfin++] = 0x0u; + in[nfin++] = 0x20000000u | (3u << 16) | + (0x1574u >> 2); + in[nfin++] = 0x1u; + in[nfin++] = 0x214fa220u; + in[nfin++] = 0x10u; + nrun += 2; + } + } + in[nfin++] = 0x20000000u | (1u << 16) | + (0x310u >> 2); + in[nfin++] = 0x705u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x22cu >> 2); + in[nfin++] = 0x800u; + in[nfin++] = 0x20000000u | (2u << 16) | + (0x1128u >> 2); + in[nfin++] = 0xffffffffu; + in[nfin++] = 0xffffffffu; + /* + * SESSION 155 CONT 14: the render-enable state the + * client's channel carried from its init push + * (see the sema_gl_raster_prelude comment). + */ + if (emit_raster) { + u32 ri5; + + in[nfin++] = 0x20000000u | (8u << 16) | + (0x1a00u >> 2); + for (ri5 = 0; ri5 < 8u; ri5++) + in[nfin++] = 0x1111u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x12ccu >> 2); + in[nfin++] = 0u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x12e8u >> 2); + in[nfin++] = 0u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x1380u >> 2); + in[nfin++] = 0u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x0e00u >> 2); + in[nfin++] = 0u; + } + /* + * SESSION 155 CONT 22: neutral blend/alpha/cull/ + * point/ZCULL state (see sema_gl_raster_extra). + */ + if (emit_raster_extra) { + static const u32 rxm[] = { + 0x12e4u, 0x1e00u, 0x12ecu, 0x1308u, + 0x1520u, 0x02e4u, 0x02e8u, 0x02fcu, + }; + u32 ri6; + + for (ri6 = 0; ri6 < 8u; ri6++) { + in[nfin++] = 0x20000000u | + (1u << 16) | + (rxm[ri6] >> 2); + in[nfin++] = 0u; + } + } + + /* + * SESSION 153-CONTINUED-89: the client's init-push + * pipeline state for the window's own stage 1. The + * guest-RAM init-push image (live at guest PA + * 0x81efc5b4, this boot) carries, beside the SPH + * version pair: + * 0x2040 <- 0x11 SET_PIPELINE_SHADER(1) + * ENABLE|TYPE=VERTEX + * 0x2054 <- 1, 0x21201b00 program address pair + * 0x204c <- 0x1a, 0x0 REGISTER_COUNT(1)=26, + * BINDING(1)=0 + * The per-frame window carries only the address pair, + * so the replay channel's TYPE/COUNT/BINDING stage + * registers stay unset - the state the DS validates + * the fetched SPH against (9/11). Emit the TYPE run + * before the binds and the COUNT/BINDING pair after + * them, in the client's own order. Both are constant + * runs (no relocation). + */ + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2040u >> 2); + in[nfin++] = 0x11u; + + /* + * SESSION 153-CONTINUED-87 (second half): the window's + * OWN binds first. The captured per-frame window + * carries draws (BEGIN at word 93) BEFORE its program + * binds (j=1 pair at word 184): on real hardware the + * bind happened in an earlier push, but the replay + * channel's fresh context has nothing bound when the + * early draws validate their stages - the + * Shader-Program-Header poison (r1277+ 9/11). Re-emit + * every bind the window itself carries ahead of it. + */ + for (bi = 0; sph_bind_all && + bi < req->n_prog_binds && bi < 8u; bi++) { + u64 va = req->prog_bind_va[bi]; + u32 sl = req->prog_bind_slot[bi] & 0x1fu; + + if (!va || nfin + 3u > 96u) + continue; + in[nfin++] = 0x20000000u | (2u << 16) | + ((0x2014u + sl * 0x40u) >> 2); + in[nfin++] = (u32)(va >> 32) & 0xffu; + in[nfin++] = (u32)va & 0xfffffffcu; + nrun++; + } + /* + * Then the client's missing stage binds (see the + * sema_gl_sph_bind comment): TESS_CONTROL (j=2 at + * family+0x200) and PIXEL (j=5 at family+0x0), both + * present verbatim in the init-push image scraped + * from guest RAM (`20020825 [0,04400200]`, + * `20020855 [0,04400000]`). + */ + if (nfin + 3u + + (emit_j0 ? (bind_binding ? 8u : 7u) : 0u) + + (emit_j2 ? (bind_binding ? 8u : 7u) : 0u) + + (emit_j4 ? (bind_binding ? 8u : 7u) : 0u) + + (emit_j5 ? (bind_binding ? 8u : 7u) : 0u) + + (emit_gsimg ? (9u + ((gs_mode & 1u) ? 11u : 0u) + + ((gs_ctx & 1u) ? 2u : 0u) + + ((gs_ctx & 2u) ? 2u : 0u)) : 0u) + + (emit_raster ? 17u : 0u) + + (emit_raster_extra ? 16u : 0u) + <= 96u) { + /* + * SESSION 155 CONT 9: pair_safe emits each + * synthesized stage's SET_PIPELINE_SHADER + * TYPE and REGISTER_COUNT *before* its SPH + * address pair, the order the client's own + * stage-1 init push uses. The legacy path + * below stays address-first (bytes + * unchanged when the knob is off). + */ + if (pair_safe) { + if (emit_j0) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2000u >> 2); + in[nfin++] = 1u | (0u << 4); + in[nfin++] = 0x20000000u | + ((bind_binding ? 2u : 1u) << 16) | + (0x200cu >> 2); + in[nfin++] = 0x1au; + if (bind_binding) + in[nfin++] = 0x0u; + } + if (emit_j4) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2100u >> 2); + in[nfin++] = 1u | (4u << 4); + in[nfin++] = 0x20000000u | + ((bind_binding ? 2u : 1u) << 16) | + (0x210cu >> 2); + in[nfin++] = 0x1au; + if (bind_binding) + in[nfin++] = 0x0u; + } + if (emit_j2) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2080u >> 2); + in[nfin++] = 1u | (2u << 4); + in[nfin++] = 0x20000000u | + ((bind_binding ? 2u : 1u) << 16) | + (0x208cu >> 2); + in[nfin++] = 0x1au; + if (bind_binding) + in[nfin++] = 0x0u; + } + if (emit_j5) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2140u >> 2); + in[nfin++] = 1u | (5u << 4); + in[nfin++] = 0x20000000u | + ((bind_binding ? 2u : 1u) << 16) | + (0x214cu >> 2); + in[nfin++] = 0x1au; + if (bind_binding) + in[nfin++] = 0x0u; + } + } + if (emit_j0) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2014u >> 2); + in[nfin++] = (u32)((fam + 0x900u) >> 32) & 0xffu; + in[nfin++] = (u32)(fam + 0x900u) & 0xfffffffcu; + } + if (emit_j4) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2114u >> 2); + in[nfin++] = (u32)((fam + 0x900u) >> 32) & 0xffu; + in[nfin++] = (u32)(fam + 0x900u) & 0xfffffffcu; + } + if (emit_j2) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2094u >> 2); + in[nfin++] = (u32)((fam + 0x200u) >> 32) & 0xffu; + in[nfin++] = (u32)(fam + 0x200u) & 0xfffffffcu; + } + if (emit_j5) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2154u >> 2); + in[nfin++] = (u32)(fam >> 32) & 0xffu; + in[nfin++] = (u32)fam & 0xfffffffcu; + } + if (!pair_safe) { + /* + * SESSION 153-CONTINUED-88: the per-slot + * SET_PIPELINE_SHADER(j) TYPE state. The + * client's init push writes it once per + * channel (init-push image: `20010810 + * 00000011` = SET_PIPELINE_SHADER(1) <- + * ENABLE|TYPE=VTG); the captured per-frame + * windows carry only the program-address + * pairs, so the replay context's stage + * registers for the synthesized binds would + * stay unset. Emit them for the two + * synthesized slots with the known types + * (j=2 TESS_CONTROL -> 2, j=5 PIXEL -> 5). + */ + if (emit_j0) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2000u >> 2); + in[nfin++] = 1u | (0u << 4); + } + if (emit_j4) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2100u >> 2); + in[nfin++] = 1u | (4u << 4); + } + if (emit_j2) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2080u >> 2); + in[nfin++] = 1u | (2u << 4); + } + if (emit_j5) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2140u >> 2); + in[nfin++] = 1u | (5u << 4); + } + /* + * SESSION 153-CONTINUED-90: the per-stage + * REGISTER_COUNT/BINDING records for the + * synthesized j=2/j=5 slots - the DS SPH + * validation's last missing per-stage input; + * same shape as the captured stage-1 pair + * (reg count 26, group 0). + * CONT-119 (r1450): the client never emits a + * stage-2/5 BINDING, and the FE walking the + * window rejects our synthesized BINDING=0 + * word (0x2090) with Class Error EC4. Drop + * the BINDING data writes by default and keep + * the 1-word REG_COUNT runs only; the knob + * restores the legacy CONT-90 2-word runs. + */ + if (bind_binding) { + if (emit_j0) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x200cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + } + if (emit_j4) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x210cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + } + if (emit_j2) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x208cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + } + if (emit_j5) { + in[nfin++] = 0x20000000u | (2u << 16) | + (0x214cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + } + } else { + if (emit_j0) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x200cu >> 2); + in[nfin++] = 0x1au; + } + if (emit_j4) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x210cu >> 2); + in[nfin++] = 0x1au; + } + if (emit_j2) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x208cu >> 2); + in[nfin++] = 0x1au; + } + if (emit_j5) { + in[nfin++] = 0x20000000u | (1u << 16) | + (0x214cu >> 2); + in[nfin++] = 0x1au; + } + } + } + /* + * SESSION 153-CONTINUED-89: the client's + * init-push stage-1 REGISTER_COUNT/BINDING + * pair (`20020813 0000001a 00000000` in the + * guest-RAM init image): REGISTER_COUNT(1) = + * 26 (the window SPH's reg count), BINDING(1) + * = group 0. Emitted after the address pair, + * exactly as the client's own push orders it. + */ + in[nfin++] = 0x20000000u | (2u << 16) | + (0x204cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x0u; + /* + * SESSION 155 CONT 11: the synthesized + * header-only GEOMETRY SPH (128 B at + * family+0x780, authored by the sema + * service). Bind it to stage 4 and keep the + * stage disabled: the DS then validates a + * real GS SPH (its SPH18 check2 attribute + * map) while the engine does not execute the + * header-only image. Emitted after the + * stage-1 pair, before the address-pair + * census. The pair is the only relocatable + * run here (nrun++ below). + */ + if (emit_gsimg) { + u64 gva = fam + (gs_addr ? 0xe80u : + 0x780u); + + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2100u >> 2); + in[nfin++] = 0x40u; + in[nfin++] = 0x20000000u | (1u << 16) | + (0x210cu >> 2); + in[nfin++] = 0x1au; + in[nfin++] = 0x20000000u | (2u << 16) | + (0x2114u >> 2); + in[nfin++] = (u32)(gva >> 32) & 0xffu; + in[nfin++] = (u32)gva & 0xfffffffcu; + /* + * CONT 13: the mesh-dummy GS + * contract also writes the GS mode + * and the post-VTG attribute skip + * mask (both zero default state). + */ + if (gs_mode & 1u) { + in[nfin++] = 0x20000000u | + (1u << 16) | + (0x0208u >> 2); + in[nfin++] = 0x1u; + in[nfin++] = 0x20000000u | + (8u << 16) | + (0x1240u >> 2); + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + in[nfin++] = 0u; + } + if (gs_ctx & 1u) { + in[nfin++] = 0x20000000u | + (1u << 16) | + (0x114cu >> 2); + in[nfin++] = 0u; + } + if (gs_ctx & 2u) { + in[nfin++] = 0x20000000u | + (1u << 16) | + (0x0208u >> 2); + in[nfin++] = 0u; + } + in[nfin++] = 0x20000000u | (1u << 16) | + (0x2100u >> 2); + in[nfin++] = (gs_mode & 2u) ? 0x41u : + 0x40u; + } + /* + * nrun counts only the ADDRESS-bearing + * binds: the guard below matches it + * against the relocation pass's + * relocated-pair count, and the constant + * type runs can never be relocated. + */ + if (emit_j0) + nrun++; + if (emit_j2) + nrun++; + if (emit_j4) + nrun++; + if (emit_j5) + nrun++; + if (emit_gsimg) + nrun++; + } + orel = gsp_phys_rmcall_gr_3d_patch_window( + in, nfin, out, &on, &odrop, req, &n_shadow, + bar0, epoch); + if (orel == nrun && on == nfin) { + memmove(&stream_patch[nfin], stream_patch, + (size_t)win3d_n * 4u); + memcpy(stream_patch, out, (size_t)nfin * 4u); + win3d_n += nfin; + if (sph_bind_log < 64u) { + sph_bind_log++; + pr_info("vgsp-gr: SPH-BIND emitted fam=0x%llx win-binds=%u j0=0x%llx j2=0x%llx j4=0x%llx j5=0x%llx j1-type=0x11 rc=0x1a initblk=17 win=%u (pre=%u) h=0x%x push=0x%llx\n", + fam, req->n_prog_binds, + fam + 0x900u, fam + 0x200u, + fam + 0x900u, fam, + win3d_n, nfin, + req->h_object, + req->push_va); + } + } else if (sph_bind_log < 64u) { + sph_bind_log++; + pr_info("vgsp-gr: SPH-BIND skipped (reloc=%u/%u kept=%u/%u fam=0x%llx h=0x%x push=0x%llx)\n", + orel, nrun, on, nfin, fam, + req->h_object, req->push_va); + } + pr_info("vgsp-gr: SPH-BIND-MASK mask=0x%x all=%u j0=%u j2=%u j4=%u j5=%u h=0x%x push=0x%llx\n", + sph_bind_mask, sph_bind_all ? 1u : 0u, + emit_j0 ? 1u : 0u, emit_j2 ? 1u : 0u, + emit_j4 ? 1u : 0u, emit_j5 ? 1u : 0u, + req->h_object, req->push_va); + } + /* + * SESSION 153-CONTINUED-62: surface (render-target) prelude - + * see the gsp_phys_gr_surf_pre comment. Injected only when the + * window carries no usable colour-target bind of its own; the + * alias scan + relocation run exactly as they would for a + * window-carried bind (the captured VA is the surface the + * machine's draws target, X screen observed). The miss census + * feeds the MISS-ABORT below, so an unresolvable surface pair + * holds the launch (fail-closed) instead of faulting. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_surf_prelude) && + !gsp_phys_gr_window_has_surface(req->stream, + req->stream_n)) { + const u32 *sp; + u32 spn = 0, spg = 0; + + /* + * r1510: if the guest's code-heap upload overlapped + * the captured X screen, restore the screen from the + * capture-time baseline before this launch's replay. + * Content-gated (see the sema service's + * gsp_sema_surf_clean_restore). + */ + { + u64 cva = 0; + u32 cnp = 0; + + if (gsp_sema_surf_va_get(&cva, &cnp)) + (void)gsp_sema_surf_clean_restore(cva, + cnp); + } + sp = gsp_sema_surf_prelude_get(&spn, &spg); + if (sp && spn && spg) { + bool repatch = + spg != READ_ONCE(gsp_phys_gr_surf_pre_gen); + u64 want = 0; + + /* + * r1483/r1490: the rec-over-alias preference + * below rewrites the pair to THIS launch's + * data-rec bounce VA, and the fill alternates + * its relocation half per capture push. A + * buffer patched once per capture generation + * would keep the first launch's target + * forever - including the X server's 900-page + * rec lane when the next launch is the beef + * client whose partial 10-page allocation sits + * at the same VA. Re-patch whenever this + * launch's EFFECTIVE surface target (complete + * rec run or the contiguous alias, see + * gsp_phys_gr_surf_launch_target) differs + * from the cached words, so a partial launch + * always falls back to the real screen alias + * and never to its own code/operand buffer. + */ + if (!repatch && gsp_phys_gr_surf_pre_n >= 3) { + want = gsp_phys_gr_surf_launch_target( + req); + if (want) { + u64 have = + ((u64)(gsp_phys_gr_surf_pre[1] & + 0xffu) << 32) | + (u64)(gsp_phys_gr_surf_pre[2] & + 0xfffffffcu); + + if (have != want) { + static u32 retarget_log; + + repatch = true; + if (retarget_log < 16u) { + retarget_log++; + pr_info("vgsp-gr: SURF-PRELUDE retarget have=0x%llx want=0x%llx h=0x%x push=0x%llx\n", + have, want, + req->h_object, + req->push_va); + } + } + } + } + if (repatch) { + u32 sdrop = 0, sn = spn; + u32 srel; + + if (sn > GSP_PHYS_GR_SURF_PRE_WORDS) + sn = GSP_PHYS_GR_SURF_PRE_WORDS; + memcpy(gsp_phys_gr_surf_pre_in, sp, + (size_t)sn * 4u); + gsp_phys_gr_surface_alias_scan_locked( + gsp_phys_gr_surf_pre_in, sn, + req); + srel = gsp_phys_rmcall_gr_3d_patch_window( + gsp_phys_gr_surf_pre_in, sn, + gsp_phys_gr_surf_pre, + &gsp_phys_gr_surf_pre_n, + &sdrop, req, &n_shadow, + bar0, epoch); + WRITE_ONCE(gsp_phys_gr_surf_pre_gen, + spg); + { + static u32 surf_patch_log; + + if (surf_patch_log < 16u) { + surf_patch_log++; + pr_info("vgsp-gr: SURF-PRELUDE patched gen=%u raw=%u kept=%u dropped=%u reloc=%u target=0x%llx h=0x%x push=0x%llx\n", + spg, sn, + gsp_phys_gr_surf_pre_n, + sdrop, srel, + ((u64)(gsp_phys_gr_surf_pre[1] & 0xffu) << 32) | + (u64)(gsp_phys_gr_surf_pre[2] & 0xfffffffcu), + req->h_object, + req->push_va); + } + } + } + if (gsp_phys_gr_surf_pre_n && + win3d_n + gsp_phys_gr_surf_pre_n + 2u + + 8u <= + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + static u32 surf_emit_log; + + memmove(&stream_patch[gsp_phys_gr_surf_pre_n], + stream_patch, + (size_t)win3d_n * 4u); + memcpy(stream_patch, + gsp_phys_gr_surf_pre, + (size_t)gsp_phys_gr_surf_pre_n * + 4u); + win3d_n += gsp_phys_gr_surf_pre_n; + surf_pre_emitted = true; + if (surf_emit_log < 16) { + surf_emit_log++; + pr_info("vgsp-gr: SURF-PRELUDE emitted words=%u window=%u h=0x%x push=0x%llx\n", + gsp_phys_gr_surf_pre_n, + win3d_n, + req->h_object, + req->push_va); + } + } + } + } + /* + * SESSION 154c: synthesized viewport preamble (see the + * sema_gl_vp_prelude comment). Insert the chosen literal + * sequence at sema_gl_vp_prelude_pos: the window head + * (legacy), immediately ahead of the first CALL run + * (0x3800..0x39fc) so every state bind - including the + * 0x2080 SET_PIPELINE(2)=0x21 run - has landed before the + * macro calls, or ahead of the first 0x3910 run. The scan + * uses the MME-WINRUN run grammar; a window without the + * requested run is left unchanged (non-draw window); only + * pos 0 inserts at the head without a CALL run. The words + * carry no address operand, so no relocation pass is + * applied - they are emitted verbatim. Same block-size + * guard as SPH-BIND / SURF-PRELUDE. + */ + { + int vp_mode = READ_ONCE(gsp_phys_rmcall_gr_vp_prelude); + int vp_pos = READ_ONCE(gsp_phys_rmcall_gr_vp_prelude_pos); + u32 vpn = gsp_phys_gr_vp_pre_len(vp_mode); + const u32 *vp_src = gsp_phys_gr_vp_pre_src(vp_mode); + + if (vpn && vp_src) { + u32 vp_at = 0; + bool vp_skip = false; + + if (vp_pos == 1 || vp_pos == 2) { + vp_at = gsp_phys_gr_vp_insert_off( + stream_patch, win3d_n, vp_pos); + if (vp_at >= win3d_n) { + static bool vp_nd_log; + + vp_skip = true; + if (!vp_nd_log) { + vp_nd_log = true; + pr_info("vgsp-gr: VP-PRELUDE skipped (no CALL run; non-draw window) window=%u pos=%d mode=%d\n", + win3d_n, vp_pos, + vp_mode); + } + } + } + if (!vp_skip && + win3d_n + vpn + 2u + 8u <= + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + static u32 vp_emit_log; + u32 vp_old = win3d_n; + + memmove(&stream_patch[vp_at + vpn], + &stream_patch[vp_at], + (size_t)(win3d_n - vp_at) * + 4u); + memcpy(&stream_patch[vp_at], + vp_src, + (size_t)vpn * 4u); + win3d_n += vpn; + if (vp_emit_log < 16u) { + vp_emit_log++; + pr_info("vgsp-gr: VP-PRELUDE mode=%d pos=%d words=%u window=%u->%u at=%u\n", + vp_mode, vp_pos, vpn, + vp_old, win3d_n, + vp_at); + } + } else if (!vp_skip) { + static u32 vp_skip_log; + + if (vp_skip_log < 8u) { + vp_skip_log++; + pr_info("vgsp-gr: VP-PRELUDE skipped (no room) mode=%d pos=%d words=%u window=%u cap=%u\n", + vp_mode, vp_pos, vpn, + win3d_n, + GSP_PHYS_RMCALL_GR_METHOD_WORDS); + } + } + } + } + /* + * Missing RT0 bind (see the sema_gl_rt0_bind comment). + * Insert the RT0 colour-target run and the count-2 CT + * select ahead of the window's first 0x840 (RT1) run. + * Skip when the window binds 0x800 itself; skip when it + * has no 0x840 anchor. Same run grammar and block-size + * guard as the viewport prelude; constant words, no + * relocation pass. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_rt0_bind)) { + static u32 rt0_emit_log; + static u32 rt0_skip_log; + static bool rt0_nd_log; + u32 rt0_at = gsp_phys_gr_method_run_off( + stream_patch, win3d_n, 0x840u); + u32 rt0_have = gsp_phys_gr_method_run_off( + stream_patch, win3d_n, 0x800u); + u32 rt0_call = gsp_phys_gr_vp_insert_off( + stream_patch, win3d_n, 1); + + if (rt0_call >= win3d_n) { + if (!rt0_nd_log) { + rt0_nd_log = true; + pr_info("vgsp-gr: RT0-BIND skipped (non-draw window) window=%u\n", + win3d_n); + } + } else if (rt0_have < win3d_n) { + if (rt0_skip_log < 8u) { + rt0_skip_log++; + pr_info("vgsp-gr: RT0-BIND skipped (window already binds 0x800) window=%u at=%u\n", + win3d_n, rt0_at); + } + } else if (rt0_at >= win3d_n) { + if (rt0_skip_log < 8u) { + rt0_skip_log++; + pr_info("vgsp-gr: RT0-BIND skipped (no 0x840 run) window=%u\n", + win3d_n); + } + } else if (win3d_n + + (READ_ONCE(gsp_phys_rmcall_gr_rt0_third_dim) ? + GSP_PHYS_GR_RT0_BIND_EXT_WORDS : + GSP_PHYS_GR_RT0_BIND_WORDS) + + 2u + 8u <= + GSP_PHYS_RMCALL_GR_METHOD_WORDS) { + u32 rt0_old = win3d_n; + bool rt0_td = + READ_ONCE(gsp_phys_rmcall_gr_rt0_third_dim); + u32 rt0_n = rt0_td ? + GSP_PHYS_GR_RT0_BIND_EXT_WORDS : + GSP_PHYS_GR_RT0_BIND_WORDS; + u32 rt0_words[GSP_PHYS_GR_RT0_BIND_EXT_WORDS]; + + if (rt0_td) + memcpy(rt0_words, gsp_phys_gr_rt0_bind_ext, + sizeof(gsp_phys_gr_rt0_bind_ext)); + else + memcpy(rt0_words, gsp_phys_gr_rt0_bind, + sizeof(gsp_phys_gr_rt0_bind)); + /* + * r1483: RT0 and RT1 must name the SAME + * surface. The SURF prelude just emitted + * ahead of this run already carries the + * launch's real target after relocation + * (the data-rec bounce base when the + * surface rec set is complete, the + * contiguous alias otherwise); copy its + * geometry words so the PS's RT0 stores + * land in the tracked target on BOTH + * relocation halves. Without the SURF + * prelude (the window binds its own + * target) keep the historical constant. + */ + if (surf_pre_emitted && + gsp_phys_gr_surf_pre_n >= 7u) + memcpy(&rt0_words[1], + &gsp_phys_gr_surf_pre[1], + 6u * sizeof(u32)); + memmove(&stream_patch[rt0_at + rt0_n], + &stream_patch[rt0_at], + (size_t)(win3d_n - rt0_at) * 4u); + memcpy(&stream_patch[rt0_at], rt0_words, + rt0_n * 4u); + win3d_n += rt0_n; + if (rt0_emit_log < 16u) { + rt0_emit_log++; + pr_info("vgsp-gr: RT0-BIND emitted window=%u->%u at=%u target=0x%llx surf=%u td=%u\n", + rt0_old, win3d_n, rt0_at, + ((u64)(rt0_words[1] & 0xffu) << 32) | + (u64)(rt0_words[2] & 0xfffffffcu), + surf_pre_emitted ? 1u : 0u, + rt0_td ? 1u : 0u); + } + } else { + if (rt0_skip_log < 8u) { + rt0_skip_log++; + pr_info("vgsp-gr: RT0-BIND skipped (no room) window=%u cap=%u\n", + win3d_n, + GSP_PHYS_RMCALL_GR_METHOD_WORDS); + } + } + } + /* + * SESSION 153-CONTINUED-57: patch the attr/pipeline prelude for + * this launch (relocation exactly like the window: the + * SET_PIPELINE_PROGRAM_ADDRESS pairs and any other recognised + * operand land in engine-visible shadow VAs; sub!=0 and + * win_drop runs are removed by the same pass). Only done when + * the capture generation moved - the shadow VAs are stable + * per HPA for the boot, so an older patch stays valid while it + * drains across launches. See gsp_phys_gr_attr_pre*. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_attr_prelude)) { + const u32 *ap; + u32 apn = 0, apg = 0; + + ap = gsp_sema_attr_prelude_get(&apn, &apg); + if (ap && apn && apg && + apg != READ_ONCE(gsp_phys_gr_attr_pre_gen)) { + u32 adrop = 0, an = apn; + + if (an > GSP_PHYS_GR_ATTR_PRE_WORDS) + an = GSP_PHYS_GR_ATTR_PRE_WORDS; + gsp_phys_rmcall_gr_3d_patch_window(ap, an, + gsp_phys_gr_attr_pre, + &gsp_phys_gr_attr_pre_n, + &adrop, req, &n_shadow, + bar0, epoch); + if (gsp_phys_gr_attr_pre_n) { + WRITE_ONCE(gsp_phys_gr_attr_pre_gen, + apg); + pr_info("vgsp-gr: ATTR-PRELUDE patched gen=%u raw=%u kept=%u dropped=%u h=0x%x push=0x%llx\n", + apg, an, gsp_phys_gr_attr_pre_n, + adrop, req->h_object, + req->push_va); + } + } + } + if (ndrop) { + static u32 gr3d_drop_log; + + if (gr3d_drop_log < 32) { + gr3d_drop_log++; + pr_info("vgsp-gr: 3D win-drop h=0x%x push=0x%llx raw=%u kept=%u dropped=%u list=[%s]\n", + req->h_object, req->push_va, + req->stream_n, win3d_n, ndrop, + gsp_phys_rmcall_gr_win_drop ? + gsp_phys_rmcall_gr_win_drop : ""); + } + } + /* + * SESSION 153-CONTINUED-59: pre-doorbell MISS abort (see the + * gsp_phys_rmcall_gr_miss_abort comment). A window/attr + * operand whose relocation stayed a raw client VA after the + * level-2 resolver would make the physical engine read an + * unmapped page (r1104 MMU Fault HUBCLIENT_PD @0x4350000 -> + * guest channel reset; r1105 index-buffer misses). Ride the + * same failure path as RELOC-GAP-STRICT: the launch is not + * submitted, the caller falls back and authors the software + * release, the page is already in the pin demand set, and + * the next forward retries. Bounded: after 3000 aborts the + * check fails open so a page the resolver can never see + * cannot livelock the lane forever. + */ + if (READ_ONCE(gsp_phys_rmcall_gr_miss_abort) && + gsp_phys_rmcall_gr_3d_miss_n && + gsp_phys_gr_miss_abort_n < 3000u) { + static u32 miss_abort_log; + + gsp_phys_gr_miss_abort_n++; + if (miss_abort_log < 16u || + !(gsp_phys_gr_miss_abort_n % 256u)) { + miss_abort_log++; + pr_info("vgsp-gr: MISS-ABORT h=0x%x push=0x%llx miss=%u n=%llu (operand page unresolvable; demanded via pin path, launch not submitted - retry next forward)\n", + req->h_object, req->push_va, + gsp_phys_rmcall_gr_3d_miss_n, + gsp_phys_gr_miss_abort_n); + } + mutex_unlock(&gsp_phys_rm_lock); + gsp_phys_gr_refused++; + /* + * SESSION 153-CONTINUED-61: the dedicated verdict lets + * the sema service keep the captured candidate armed + * and re-forward it (re-arm) instead of consuming it + * with a software completion: the demanded pages + * resolve through the pin path and the retry then runs + * the real launch. + */ + ret = GSP_PHYS_GR_RC_MISSABORT; + goto out; + } + /* + * SESSION 153-CONTINUED-56: bounded RAW (pre-drop, pre-reloc) + * window dump so the emitted window can be diffed against the + * client's authentic bytes. r1091 faulted at method 0x1168 + * (SET_VERTEX_ATTRIBUTE_A(2)) with Data 00000000 in the emitted + * stream; this copy tells whether that zero is the client's own + * value or an emission artifact (drop/reloc/align). + */ + { + static u32 gr3d_raw_log; + u32 wi; + + if (gr3d_raw_log < 8) { + gr3d_raw_log++; + pr_info("vgsp-gr: 3D raw dump h=0x%x push=0x%llx words=%u\n", + req->h_object, req->push_va, + req->stream_n); + for (wi = 0; + wi + 16 <= req->stream_n && wi < 448; + wi += 16) + pr_info("vgsp-gr: 3D raw[%3u]: %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + wi, + req->stream[wi + 0], + req->stream[wi + 1], + req->stream[wi + 2], + req->stream[wi + 3], + req->stream[wi + 4], + req->stream[wi + 5], + req->stream[wi + 6], + req->stream[wi + 7], + req->stream[wi + 8], + req->stream[wi + 9], + req->stream[wi + 10], + req->stream[wi + 11], + req->stream[wi + 12], + req->stream[wi + 13], + req->stream[wi + 14], + req->stream[wi + 15]); + } + } + /* + * SESSION 153-CONTINUED-11: bounded raw window dump - the + * exact C597 method stream the engine walks (program binds + * 0x2014/0x2018+j*0x40, SET_PIPELINE_SHADER 0x2000+j*0x40, + * draws 0x1618/0xd74). Two windows per boot. + */ + { + static u32 gr3d_win_log; + u32 wi; + + if (gr3d_win_log < 48) { + gr3d_win_log++; + pr_info("vgsp-gr: 3D window dump h=0x%x push=0x%llx words=%u\n", + req->h_object, req->push_va, win3d_n); + for (wi = 0; wi < win3d_n && wi < 384; + wi += 16) + pr_info("vgsp-gr: 3D win[%3u]: %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + wi, + win3d[wi + 0], win3d[wi + 1], + win3d[wi + 2], win3d[wi + 3], + win3d[wi + 4], win3d[wi + 5], + win3d[wi + 6], win3d[wi + 7], + win3d[wi + 8], win3d[wi + 9], + win3d[wi + 10], win3d[wi + 11], + win3d[wi + 12], win3d[wi + 13], + win3d[wi + 14], win3d[wi + 15]); + } + } + } + + /* + * r1453 PAIR-CHAIN: drain the final window ONCE per launch, after + * every prepend (SPH-BIND, SURF-PRELUDE) and the relocation pass, + * so the same bytes feed both the first build and the outcome + * retry. The pass only inserts WAIT_FOR_IDLE(0x110) runs ahead of + * back-to-back normal/normal BEGIN pairs; the guest's BEGIN data + * words are carried verbatim. Knob 0 = no pass (legacy bytes). + */ + if (READ_ONCE(gsp_phys_rmcall_gr_pair_chain) && win3d && win3d_n) { + u32 pn = win3d_n; + + (void)gsp_phys_gr_pair_chain_fix((u32 *)win3d, &pn, + GSP_PHYS_RMCALL_GR_METHOD_WORDS); + win3d_n = pn; + } words = gsp_phys_rmcall_build_gr_launch(methods, req, qmd, qmd_dst, - ch_sem_va, want_ch); - memcpy(gsp_phys_chunk_page_va(pb, 0), methods, words * 4); + ch_sem_va, want_ch, + win3d, win3d_n, + gsp_phys_gr_attr_pre, + gsp_phys_gr_attr_pre_n); + gsp_phys_gr_pb_copy(methods, words); + /* SESSION 153-CONTINUED-12: identify this block to a later PBDMA dump. */ + WRITE_ONCE(gsp_phys_gr_last_words, words); + WRITE_ONCE(gsp_phys_gr_last_h, req->h_object); + WRITE_ONCE(gsp_phys_gr_last_push_va, req->push_va); + WRITE_ONCE(gsp_phys_gr_last_pb_dma, + (u64)gsp_phys_chunk_page_dma(pb, 0)); + /* + * SESSION 153-CONTINUED-52: stream validator for the block the engine + * will walk. The r1072 fault class ("Channel ID 4 intr0 00040000" = + * PBDMA PBENTRY, "invalid method encoding used in method", + * fifo_tu102.c) is raised by a method header the PBDMA refuses, or by + * a run whose data is not fully present so the walk desyncs into + * following words. Walk the exact published dwords with the FIFO DMA + * grammar (dev_ram.ref NV_FIFO_DMA_*: SEC_OP 31:29; 1/3/5 carry 1+cnt + * data words, 4/7 and TERT mask ops are single-word, 6 is RESERVED) + * and flag the first occurrences of a reserved encoding or a run + * overrunning the block. Read-only; bounded per boot. + */ + { + static u32 blkval_log; + const u32 *bv = methods; + u32 bo = 0, nbad = 0; + + if (blkval_log < 48) { + while (bo < words && nbad < 4) { + u32 h = bv[bo]; + u32 bop = (h >> 29) & 7u; + u32 bcnt = (h >> 16) & 0xfffu; + u32 bspan = 1u; + + if (bop == 1u || bop == 3u || bop == 5u) + bspan = 1u + bcnt; + if (bop == 6u) { + nbad++; + pr_warn("vgsp-gr: BLOCKVAL reserved-encoding word=%u/%u val=%08x (h=%x push=0x%llx prev=%08x %08x)\n", + bo, words, h, req->h_object, + req->push_va, + bo ? bv[bo - 1] : 0, + bo + 1 < words ? + bv[bo + 1] : 0); + } + if (bo + bspan > words) { + nbad++; + pr_warn("vgsp-gr: BLOCKVAL run-overrun word=%u/%u hdr=%08x span=%u (needs %u more) (h=%x push=0x%llx)\n", + bo, words, h, bspan, + bo + bspan - words, + req->h_object, req->push_va); + } + bo += bspan; + } + blkval_log++; + if (nbad) + pr_warn("vgsp-gr: BLOCKVAL done words=%u bad=%u (h=%x push=0x%llx)\n", + words, nbad, req->h_object, + req->push_va); + } + } pr_info("vgsp-gr: qmd w0..3=%08x %08x %08x %08x w4..7=%08x %08x %08x %08x w23..25=%08x %08x %08x w32/33=%08x %08x w48/49=%08x %08x\n", qmd[0], qmd[1], qmd[2], qmd[3], qmd[4], qmd[5], qmd[6], qmd[7], qmd[23], qmd[24], qmd[25], qmd[32], qmd[33], qmd[48], qmd[49]); @@ -20160,6 +31173,22 @@ gr_page_unmapped: (u64)gcopied << 12); } + /* + * CONT-73: fingerprint the direct-write witness set (relocated + * operand destinations that map guest pages) immediately before the + * doorbell - after every pre-launch writer (copy-in, mirrors) has + * run, so the retire re-scan measures the ENGINE's stores alone. + */ + gsp_phys_gr_wit_capture(req); + + if (gsp_phys_gr_vidmem_bracket) { + int hret = gsp_sema_service_vidmem_hash_snapshot(); + + if (hret) + pr_warn_ratelimited("vgsp-gr: VIDMEM-BRACKET snapshot failed rc=%d\n", + hret); + } + ret = gsp_phys_rmcall_publish_locked(bar0, boot_stage.rmcall_va_pb, words, &sub); mutex_unlock(&gsp_phys_rm_lock); @@ -20293,19 +31322,53 @@ gr_page_unmapped: break; } } - pr_info("vgsp-gr: GR LAUNCH submitted h=0x%x push=0x%llx len=%u pages=%u shadow=%u qmd_va=0x%llx slot=%u words=%u prelude=%d qmdvirt=%d token=0x%x; polling qmd_rel=%u ch_rel=0x%08x (<=%u ms)%s\n", + pr_info("vgsp-gr: GR LAUNCH submitted h=0x%x push=0x%llx len=%u pages=%u shadow=%u qmd_va=0x%llx slot=%u gpget=%u gpput=%u words=%u e0=0x%08x e1=0x%08x prelude=%d qmdvirt=%d token=0x%x; polling qmd_rel=%u ch_rel=0x%08x (<=%u ms)%s\n", req->h_object, req->push_va, req->push_len, req->n_pages, - n_shadow, req->qmd_va, sub.slot, words, + n_shadow, req->qmd_va, sub.slot, sub.gpget, sub.gpput, words, + sub.e0, sub.e1, gsp_phys_rmcall_gr_prelude_value(), gsp_phys_rmcall_gr_qmdvirt_value(), boot_stage.rmcall_token, qmd_rel, want_ch, poll_ms, qmdwin_note); + { + /* + * r596 diag: the raw-prelude geometry the verbatim gate tests. + * The verbatim replay needs + * raw && raw_words >= hdr + 67 && + * raw_qmd_end == hdr + 67 && + * INIT_WORDS + hdr + 67 + TAIL_WORDS <= METHOD_WORDS + * Bounded to the first eight launches so the geometry is on + * record without flooding a battery run. + */ + static u32 prelude_geom_log; + + if (gsp_phys_rmcall_gr_prelude_value() && prelude_geom_log < 8) { + prelude_geom_log++; + pr_info("vgsp-gr: prelude geom raw=%u raw_words=%u hdr=%u end=%u want_end=%u v=%d\n", + req->raw ? 1u : 0u, req->raw_words, + req->raw_qmd_hdr, req->raw_qmd_end, + req->raw_qmd_hdr + + GSP_PHYS_RMCALL_GR_QMD_WINDOW_WORDS, + gsp_phys_rmcall_gr_prelude_value()); + } + } if (qmd_win_diag) pr_info("vgsp-gr: qmd dest va=0x%llx pdeIndex=%u installed=%u win_mask=0x%llx ensure_ret=%d\n", qmd_dst, qmd_win_pde, qmd_win_installed, (unsigned long long)boot_stage.rmcall_guest_pt_win_mask, qmd_win_ret); + /* + * SESSION 153-CONTINUED-69: pre-reset MME fault sample. The engine + * faults within the first tens of ms of the launch; sample the MME + * ESR/INFO/INFO2 + DS ESR + GRFIFO/trap regs at MMIO rate from the + * moment the doorbell rang (in publish) so the faulting values are + * caught before the GSP-RM clears them. + */ + if (stream3d && bar0) + fast_poison = gsp_phys_mme_fast_sample(bar0, "launch", + req->h_object, 0); + poll_start_us = (u32)ktime_to_us(ktime_get()); deadline = jiffies + msecs_to_jiffies(poll_ms); for (;;) { @@ -20334,10 +31397,80 @@ gr_page_unmapped: sked_sample[8] = ioread32(bar0 + 0x00409b00u); sked_sample[9] = ioread32(bar0 + 0x00409b04u); sked_sample[10] = ioread32(bar0 + 0x00409400u); + /* + * SESSION 153-CONTINUED-14: FECS/FE ctxsw handshake + * lanes. fe0 = FECS_CTXSW_STATUS_FE_0 (0x409c00), + * fehost = FECS_HOST_INT_STATUS (0x409c18), fecsirq + * = FECS_IRQSTAT (0x409008). If the FECS dies in a + * switch, it should die visibly here first. + */ + sked_sample[11] = ioread32(bar0 + 0x00409c00u); + sked_sample[12] = ioread32(bar0 + 0x00409c18u); + sked_sample[13] = ioread32(bar0 + 0x00409008u); + /* + * SESSION 153-CONTINUED-52: in-flight fault registers. + * The DS shader-program-header error latch (0x405840), + * the MME exception latch (0x404490) with its PC/macro + * address (0x404494) and instruction register + * (0x40449c, fixed in SESSION 153-CONTINUED-106) are + * cleared by the real RM within ~ms of + * the fault, so the post-timeout reads are all clean. + * Sampling them every poll tick records the faulting + * values at the first change. + */ + sked_sample[14] = ioread32(bar0 + 0x00405840u); + sked_sample[15] = ioread32(bar0 + 0x00404490u); + sked_sample[16] = ioread32(bar0 + 0x00404494u); + /* SESSION 153-CONTINUED-106: INFO2 0x40449c (was CONFIG). */ + sked_sample[17] = ioread32(bar0 + 0x0040449cu); + /* + * SESSION 153-CONTINUED-52: the PBDMA host-software + * METHOD latch (NV_PPBDMA_METHOD0 0x4000c0: ADDR + * 13:2, SUBCH 18:16, INCR 0) - the method whose + * delivery raises NV_PPBDMA_INTR_0_METHOD (bit 21). + * The real driver clears the entry when it processes + * the interrupt, so only an in-flight sample can name + * it. + */ + sked_sample[18] = ioread32(bar0 + 0x004000c0u); + sked_sample[19] = ioread32(bar0 + 0x004000c4u); + /* + * SESSION 153-CONTINUED-57: the class-error trapped + * register set (gr_error_gk104.c grGetExceptionChannel): + * NV_PGRAPH_TRAPPED_ADDR 0x40704 (MTHD offset + SUBCH + + * INVALID status bit), TRAPPED_DATA_LOW 0x40708 (the + * class error's "Data" field), TRAPPED_DATA_HIGH + * 0x4070c, and TRAPPED_DATA_MME 0x40710 (PC 11:0 + + * MACRO_ADDRESS 19:12) - the last one names whether the + * offending write came from MME macro execution. The + * real RM clears these within ~ms of the fault; a + * per-tick sample catches the first change. The + * persistent 0x1168/Data=0 class error (r1091/r1098) + * needs exactly this decode. + */ + sked_sample[20] = ioread32(bar0 + 0x00400704u); + sked_sample[21] = ioread32(bar0 + 0x00400708u); + sked_sample[22] = ioread32(bar0 + 0x0040070cu); + sked_sample[23] = ioread32(bar0 + 0x00400710u); + /* + * SESSION 153-CONTINUED-58: the front-end DECODE + * STATE EX registers (NV_PGRAPH_PRI_FE_DECODE_STATE_EX + * 0x40a630..0x40a63c, 4 slots): MME_PC 11:0, + * MME_MACRO_ADDR 21:12, MME_GENERATED bit 31 - the + * front end's own record of the last MME-generated + * method write, which names the macro (address) and + * its pc for the class error even when the trapped + * registers are already back to the 0xbadf priv + * poison. Sampled per tick like the trap set. + */ + sked_sample[24] = ioread32(bar0 + 0x0040a630u); + sked_sample[25] = ioread32(bar0 + 0x0040a634u); + sked_sample[26] = ioread32(bar0 + 0x0040a638u); + sked_sample[27] = ioread32(bar0 + 0x0040a63cu); if (sked_first || memcmp(sked_sample, sked_prev, sizeof(sked_sample))) { - pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x\n", + pr_info("vgsp-gr: sked rec +%ums cwd0=%08x cwd1=%08x cwd2=%08x info1=%08x info2=%08x act=%08x esr=%08x grfifo=%08x fecs_cur=%08x fecs_new=%08x ctxsw1=%08x fe0=%08x fehost=%08x fecsirq=%08x ds_esr=%08x mme_esr=%08x mme_pc=%u mme_mac=0x%x mme_ir=%08x mth0=%08x mth1=%08x(pbaddr=0x%x sub=%u incr=%u) trap=%08x(data_lo=%08x hi=%08x sub=%u mmegen=%u dhv=%u) mmepc=%u mmemac=0x%x\n", sked_tick * GSP_PHYS_RMCALL_POLL_MS, sked_sample[0], sked_sample[1], @@ -20345,12 +31478,150 @@ gr_page_unmapped: sked_sample[4], sked_sample[5], sked_sample[6], sked_sample[7], sked_sample[8], sked_sample[9], - sked_sample[10]); + sked_sample[10], sked_sample[11], + sked_sample[12], sked_sample[13], + sked_sample[14], sked_sample[15], + sked_sample[16] & 0xfffu, + (sked_sample[16] >> 12) & 0xffu, + sked_sample[17], + sked_sample[18], sked_sample[19], + (sked_sample[18] >> 2) << 2, + (sked_sample[18] >> 16) & 7u, + sked_sample[18] & 1u, + sked_sample[20], sked_sample[21], + sked_sample[22], + (sked_sample[20] >> 16) & 7u, + (sked_sample[20] >> 20) & 1u, + (sked_sample[20] >> 24) & 1u, + sked_sample[23] & 0xfffu, + (sked_sample[23] >> 12) & 0xffu); + if (sked_sample[24] || sked_sample[25] || + sked_sample[26] || sked_sample[27]) + pr_info("vgsp-gr: sked fed0=%08x fed1=%08x fed2=%08x fed3=%08x (pc0=%u mac0=0x%x gen0=%u)\n", + sked_sample[24], + sked_sample[25], + sked_sample[26], + sked_sample[27], + sked_sample[24] & 0xfffu, + (sked_sample[24] >> 12) & 0x3ffu, + (sked_sample[24] >> 31) & 1u); memcpy(sked_prev, sked_sample, sizeof(sked_sample)); sked_first = false; } } + /* + * SESSION 153-CONTINUED-48: priv-error (0xbadf-tagged) wedge + * recovery - see sema_gr_wedge_recover. The poison latch is + * checked before the completion break and the ladder so a + * launch whose window faulted the GR hands the engine to the + * next launch recovered instead of wedged. One attempt per + * launch (local flag); the hard re-arm is the r445 mechanism, + * bounded by its own RPC and sem-probe timeouts. + */ + if (READ_ONCE(gsp_phys_gr_wedge_recover) && bar0 && + !wedge_recover_done) { + u32 g0 = ioread32(bar0 + 0x00400504u); + + /* + * SESSION 153-CONTINUED-72: either the poison is still + * asserted here, or the MME-FAST spin caught it before + * the FECS cleared it (fast_poison latch). Both mean + * the window priv-errored and the engine context is + * gone; the earlier live-register-only check missed + * the self-cleared case (r1149) and wedged the boot. + */ + if ((g0 & 0xffff0000u) == 0xbadf0000u || fast_poison) { + u32 g1; + + wedge_recover_done = true; + wedge_recoveries++; + pr_warn("vgsp-gr: WEDGE-RECOVER priv-error poison seen (grfifo=0x%08x%s) h=0x%x push=0x%llx - TSG preempt + resched\n", + g0, + fast_poison && + (g0 & 0xffff0000u) != 0xbadf0000u ? + " via MME-FAST latch (self-cleared)" : + "", + req->h_object, req->push_va); + gsp_phys_mme_regs_log(bar0, "wedge-poison"); + mutex_lock(&gsp_phys_rm_lock); + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + g1 = ioread32(bar0 + 0x00400504u); + /* + * SESSION 153-CONTINUED-61: the TSG preempt + + * resched above resets the engine context, so + * the MME program the emitted prelude loaded is + * gone. Clear the emission latch: the next + * launch that can carry the prelude re-preloads + * it (without this every later replay ran its + * macros against an unloaded MME and stalled + * pre-retire - the r1110 mme_pc=513 class). + */ + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, 0); + /* + * SESSION 153-CONTINUED-79: also rewind the + * split-emission drain cursor, else a cleared + * latch with an exhausted cursor would skip + * the owed re-preload. + */ + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + /* + * SESSION 153-CONTINUED-109: rewind the PACK + * cursor too (the r1377 deadlock: recovery reset + * pre_emit_off but left pack_off at its end, so + * the split emitter hit `poff >= maxa` -> can=0 + * -> packed=1 with zero words on every launch: + * `MME-PRELUDE part gen=3 off=0 words=0/3083` + * forever, latch never re-set, gates never + * release, MME never starts). The gate-fail + * path (gsp_phys_gr_mme_iram_gate_fail) already + * rewinds both cursor pairs. + */ + WRITE_ONCE(gsp_phys_gr_mme_pack_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_sa, false); + /* + * SESSION 153-CONTINUED-109: the hard re-arm + * wiped the engine context, so the generation + * whose carrier executed is no longer resident. + * Invalidate it; the re-preload's own GP_GET + * advance (readback-free witness) re-promotes + * (fetch_miss/poll-break paths). Without this + * the stale live_gen would mark a wiped context + * live and the next window+calls launch would + * run against an unloaded MME. + */ + WRITE_ONCE(gsp_phys_gr_mme_live_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_release_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_release_h, 0); + /* r1453 PAIR-CHAIN: context wipe voids the + * pair-install proof; re-earn it. */ + WRITE_ONCE(gsp_phys_gr_mme_pairs_gen, 0); + /* r1454 PAIR-CHAIN-ENTRY-ELIDE: the wiped + * context has no open FE pair; clear the + * cross-launch tracker too. */ + WRITE_ONCE(gsp_phys_gr_mme_pair_open, false); + WRITE_ONCE(gsp_phys_gr_mme_pair_open_split, + 0); + WRITE_ONCE(gsp_phys_gr_mme_pair_open_entry, + false); + WRITE_ONCE( + gsp_phys_gr_mme_pair_open_entry_split, + 0); + /* + * r1447: the wiped context voids the arming + * window-hold and the strict grace budget too. + */ + WRITE_ONCE(gsp_phys_gr_mme_window_hold_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_window_hold_h, 0); + WRITE_ONCE(gsp_phys_gr_mme_strict_holds, 0); + pr_warn("vgsp-gr: WEDGE-RECOVER after re-arm grfifo=0x%08x (%s); recovery #%u this boot; MME prelude latch cleared (context re-preload owed)\n", + g1, + (g1 & 0xffff0000u) == 0xbadf0000u ? + "still poisoned" : "cleared", + wedge_recoveries); + } + } if (qmd_rel) { if (qmd_rel_hpa) w0 = READ_ONCE(*(u32 *)((u8 *)__va(qmd_rel_hpa) + @@ -20358,15 +31629,162 @@ gr_page_unmapped: else w0 = READ_ONCE(*(u32 *)sem_va); } + /* + * SESSION 153-CONTINUED-59: PBENTRY first-sighting watch. + * The r1102 fault was only captured via the GSP's 0x1006 + * relay (seconds later, intr0 already cleared by the RM): + * here every 5 ms poll of a 3D-stream launch scans the 12 + * PBDMA instances for NV_PPBDMA_INTR_0 bit 18 (PBENTRY) and + * on the FIRST sighting dumps the live PB_HEADER / PB_COUNT + * / PB_DATA0 / HDR_SHADOW set plus the raw 0x100..0x120 + * block for the faulting instance, before any re-arm or RM + * clear can overwrite them. Bounded (8 lines per boot); + * read-only, no behavior change to the launch. + */ + if (stream3d && bar0) { + static u32 pbentry_log; + u32 pi2; + + for (pi2 = 0; pi2 < 12 && pbentry_log < 8; pi2++) { + u32 b2 = 0x40000u + pi2 * 0x2000u; + u32 i0 = ioread32(bar0 + b2 + 0x108u); + + if (!(i0 & 0x00040000u)) + continue; + pbentry_log++; + pr_warn("vgsp-gr: PBENTRY-FIRST-SIGHTING P%u intr0=%08x gpf=%u pbfetch=0x%llx hdr=%08x cnt=%08x data0=%08x pbhd=%08x h=%x push=0x%llx words=%u tick=%u\n", + pi2, i0, + ioread32(bar0 + b2 + 0x050u), + ((u64)(ioread32(bar0 + b2 + 0x058u) & + 0xffu) << 32) | + (ioread32(bar0 + b2 + 0x054u) & + ~3u), + ioread32(bar0 + b2 + 0x118u), + ioread32(bar0 + b2 + 0x088u), + ioread32(bar0 + b2 + 0x08cu), + ioread32(bar0 + b2 + 0x084u), + req->h_object, req->push_va, words, + sked_tick); + pr_info("vgsp-gr: PBENTRY-FIRST-SIGHTING P%u raw[0x100]=%08x [104]=%08x [10c]=%08x [110]=%08x [114]=%08x [11c]=%08x [120]=%08x\n", + pi2, + ioread32(bar0 + b2 + 0x100u), + ioread32(bar0 + b2 + 0x104u), + ioread32(bar0 + b2 + 0x10cu), + ioread32(bar0 + b2 + 0x110u), + ioread32(bar0 + b2 + 0x114u), + ioread32(bar0 + b2 + 0x11cu), + ioread32(bar0 + b2 + 0x120u)); + gsp_phys_pbdma_fault_dump(0xffffffffu); + break; + } + } + /* + * SESSION 153-CONTINUED-13: early PBDMA fetch probe for a 3D + * stream launch - where is the front end while a DRAW window + * does not retire? One shot per launch at ~150 ms and ~1.2 s + * (5 ms poll), bounded per boot. pbfetch inside the module + * pushbuffer page = the walk/exec is stuck in our block; + * pbfetch past it = the whole stream was decoded and the + * stall is in execution/completion. + */ + if (stream3d && bar0 && + (sked_tick == 30 || sked_tick == 240)) { + static u32 winprobe_log; + u32 pi; + + if (winprobe_log < 64) { + winprobe_log++; + pr_info("vgsp-gr: WINPROBE +%ums push=0x%llx words=%u block_dma=0x%llx grfifo=%08x trapped=%08x trapped_lo=%08x\n", + sked_tick * GSP_PHYS_RMCALL_POLL_MS, + req->push_va, words, sub.pb_dma, + ioread32(bar0 + 0x00400504u), + ioread32(bar0 + 0x00400704u), + ioread32(bar0 + 0x00400708u)); + for (pi = 0; pi < 12; pi++) { + u32 b = 0x40000u + pi * 0x2000u; + u32 intr0 = ioread32(bar0 + b + 0x108u); + u32 gpf = ioread32(bar0 + b + 0x050u); + u32 pbf = ioread32(bar0 + b + 0x054u); + u32 pbfhi = ioread32(bar0 + b + 0x058u); + u32 hdrsh = ioread32(bar0 + b + 0x118u); + u32 wbh = ioread32(bar0 + b + 0x084u); + u32 wbcnt = ioread32(bar0 + b + 0x088u); + u32 wbd0 = ioread32(bar0 + b + 0x08cu); + u64 fetch64 = + ((u64)(pbfhi & 0xffu) << 32) | + (u64)(pbf & ~3u); + s64 woff = -1; + + if (!intr0 && !gpf && !pbf && !hdrsh) + continue; + if (gsp_phys_gr_last_pb_dma && + fetch64 >= gsp_phys_gr_last_pb_dma && + fetch64 < + gsp_phys_gr_last_pb_dma + + (u64)gsp_phys_gr_last_words * + 4u) + woff = (s64)(fetch64 - + gsp_phys_gr_last_pb_dma) / + 4; + pr_info("vgsp-gr: WINPROBE P%u intr0=%08x gp_put=%u pbfetch=0x%llx off=%lld hdr=%08x(%s m=0x%x cnt=%u sub=%u) pbhd=%08x(type=%s) pbcnt=%08x pbd0=%08x\n", + pi, intr0, gpf, fetch64, woff, hdrsh, + gsp_phys_pbdma_type_name(hdrsh >> 29), + (hdrsh & 0x1fffu) << 2, + (hdrsh >> 16) & 0xfffu, + (hdrsh >> 13) & 7u, wbh, + gsp_phys_pbdma_type_name(wbh >> 29), + wbcnt, wbd0); + } + } + } if (ch_rel_hpa) w1 = READ_ONCE(*(u32 *)((u8 *)__va(ch_rel_hpa) + ch_rel_off)); else w1 = READ_ONCE(*(u32 *)(sem_va + 4)); if ((!qmd_rel || w0 == want_qmd) && w1 == want_ch) { + /* + * r1453 PAIR-CHAIN: this launch retired. If its + * emitted window carried BEGIN/END runs, the pair + * runs are installed on the channel for this + * prelude generation - the proof the held-call + * elision extension waits for. A context wipe + * (WEDGE-RECOVER / park recovery) clears this so a + * later generation re-earns it from a drain-carrying + * window. + */ + if (gsp_phys_gr_mme_pairs_emitted && + gsp_phys_gr_mme_launch_gen && + READ_ONCE(gsp_phys_rmcall_gr_pair_chain)) + WRITE_ONCE(gsp_phys_gr_mme_pairs_gen, + gsp_phys_gr_mme_launch_gen); + /* + * SESSION 153-CONTINUED-107: if THIS launch carried + * the final packed prelude part, gate its residency on + * the IRAM readback BEFORE the break log promotes the + * generation to live (see gsp_phys_gr_mme_iram_gate). + * A failed readback clears the pending gen, the + * prelude latch and both emission cursors: the calls + * stay held and the next launch re-emits from word 0. + */ + if (bar0 && + READ_ONCE(gsp_phys_gr_mme_pending_gen) && + READ_ONCE(gsp_phys_rmcall_gr_mme_prelude_pack) && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate) && + READ_ONCE(gsp_phys_rmcall_gr_mme_iram_gate_break) && + !gsp_phys_gr_mme_iram_gate(bar0, req->h_object, + READ_ONCE(gsp_phys_gr_mme_pending_gen), + "break")) + gsp_phys_gr_mme_iram_gate_fail(); /* F4-a: first poll break + USERD witnesses. */ gsp_phys_gr_poll_break_log(w0, want_qmd, w1, - poll_start_us, &sub); + poll_start_us, &sub, + req->h_object); + /* + * SESSION 153-CONTINUED-72: GR-STATE readback at the + * retire (fed MME-generated-write records + latches). + */ + gsp_phys_gr_state_rb_log(req->h_object); break; } /* @@ -20384,6 +31802,9 @@ gr_page_unmapped: pr_err("vgsp-gr: GR LAUNCH FAILED: wait budget crossed (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx; ladder stopped\n", w0, want_qmd, w1, want_ch, req->h_object, req->push_va); + gsp_phys_mme_regs_log(bar0, "wait-max"); + gsp_phys_gr_fetch_miss_log(req->h_object, &sub, + "wait-max"); ret = -ETIMEDOUT; goto out; } @@ -20395,6 +31816,23 @@ gr_page_unmapped: pr_err("vgsp-gr: GR LAUNCH FAILED: timeout after %u ms (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x want=0x%08x) h=0x%x push=0x%llx\n", poll_ms, w0, want_qmd, w1, want_ch, req->h_object, req->push_va); + gsp_phys_mme_regs_log(bar0, "timeout"); + gsp_phys_gr_fetch_miss_log(req->h_object, &sub, + "timeout"); + /* + * SESSION 153-CONTINUED-106: the r1363 trap sat in the + * ~2.6 s window AFTER this timeout (launch failed + * 403088.34, ILLEGAL_OPCODE relayed 403090.96). Re-spin + * once here, bounded to the first few timeouts of the + * boot, so the latch is caught at first sight with its + * INFO/IR word instead of being missed between spins. + */ + if (gsp_phys_mme_post_ms && post_spin_n < 8u) { + post_spin_n++; + (void)gsp_phys_mme_fast_sample(bar0, "post", + req->h_object, + gsp_phys_mme_post_ms); + } /* * F4-b: one-shot deposit readback at the FIRST timeout * (the old post-mortem only ran after attempt 3, which @@ -20499,6 +31937,80 @@ gr_page_unmapped: mutex_lock(&gsp_phys_rm_lock); sp_ret = gsp_phys_rmcall_gr_sem_push_locked(bar0); mutex_unlock(&gsp_phys_rm_lock); + /* + * SESSION 153-CONTINUED-93: parked-engine + * recovery (see the sema_gr_park_recover + * knob). A wedged probe on a poison-free, + * trap-free front-end is the r1311-r1313 + * park class: the context is gone and only a + * context reload brings it back. One bounded + * hard re-arm per launch; ALIVE afterwards + * resumes the fresh-entry retries. + */ + if (sp_ret && bar0 && + READ_ONCE(gsp_phys_gr_park_recover)) { + static u32 park_rec_n; + + mutex_lock(&gsp_phys_rm_lock); + sp_ret = + gsp_phys_gr_wedge_hard_rearm_locked(bar0); + mutex_unlock(&gsp_phys_rm_lock); + park_rec_n++; + if (!sp_ret) { + WRITE_ONCE(gsp_phys_gr_mme_prelude_gen, 0); + WRITE_ONCE(gsp_phys_gr_mme_pre_emit_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_off, 0); + WRITE_ONCE(gsp_phys_gr_mme_pack_sa, false); + WRITE_ONCE(gsp_phys_gr_mme_live_gen, 0); + WRITE_ONCE( + gsp_phys_gr_mme_release_gen, + 0); + WRITE_ONCE( + gsp_phys_gr_mme_release_h, + 0); + /* r1453 PAIR-CHAIN: same + * context wipe - the pair + * install proof is void. */ + WRITE_ONCE( + gsp_phys_gr_mme_pairs_gen, + 0); + /* r1454 PAIR-CHAIN-ENTRY- + * ELIDE: same wipe - no + * open FE pair remains. */ + WRITE_ONCE( + gsp_phys_gr_mme_pair_open, + false); + WRITE_ONCE( + gsp_phys_gr_mme_pair_open_split, + 0); + WRITE_ONCE( + gsp_phys_gr_mme_pair_open_entry, + false); + WRITE_ONCE( + gsp_phys_gr_mme_pair_open_entry_split, + 0); + /* + * r1447: same context wipe - + * void the arming window-hold + * and the strict grace budget. + */ + WRITE_ONCE( + gsp_phys_gr_mme_window_hold_gen, + 0); + WRITE_ONCE( + gsp_phys_gr_mme_window_hold_h, + 0); + WRITE_ONCE( + gsp_phys_gr_mme_strict_holds, + 0); + pr_warn("vgsp-gr: PARK-RECOVER #%u warm sem ALIVE after TSG preempt + resched; fresh-entry retries resume push=0x%llx\n", + park_rec_n, req->push_va); + } else { + pr_warn("vgsp-gr: PARK-RECOVER #%u still wedged after hard re-arm (rc=%d) push=0x%llx\n", + park_rec_n, sp_ret, + req->push_va); + } + } if (sp_ret) { gsp_phys_gr_sem_wedged++; pr_info("vgsp-gr: timeout ladder: warm sem WEDGED (rc=%d) - failing fast instead of fresh-entry retries\n", @@ -20793,6 +32305,16 @@ gr_page_unmapped: ioread32(bar0 + 0x00405b00u), ioread32(bar0 + 0x00400100u), ioread32(bar0 + 0x0040013cu)); + if (bar0) + pr_info("vgsp-gr: FECS readback: CUR=0x%08x NEW=0x%08x ST1=0x%08x FE0=0x%08x HOSTINT=0x%08x IRQSTAT=0x%08x CPUCTL=0x%08x OS=0x%08x\n", + ioread32(bar0 + 0x00409b00u), + ioread32(bar0 + 0x00409b04u), + ioread32(bar0 + 0x00409400u), + ioread32(bar0 + 0x00409c00u), + ioread32(bar0 + 0x00409c18u), + ioread32(bar0 + 0x00409008u), + ioread32(bar0 + 0x00409100u), + ioread32(bar0 + 0x00409080u)); if (bar0) { u32 g; @@ -20830,7 +32352,7 @@ gr_page_unmapped: ret = -ETIMEDOUT; goto out; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } gsp_phys_gr_ok++; @@ -20878,6 +32400,14 @@ gr_page_unmapped: u32 wrote = gsp_phys_gr_output_retire_wait(req, bounce_csum_in, &n_tracked); + if (gsp_phys_gr_vidmem_bracket) + gsp_sema_service_vidmem_hash_diff("launch", 64); + + /* CONT-73: where did the draw write? bounce count above; + * direct guest stores (relocated alias/shadow operands and + * MME shadow-scratch destinations) here. */ + gsp_phys_gr_wit_report("retire"); + /* * r434: post-launch hardware-only QMD dump (w50/w51 INNER * get/put = the SKED's CTA dispatch counters) so the @@ -20898,9 +32428,16 @@ gr_page_unmapped: if (wrote == 0 && n_tracked) { bool ff = gsp_phys_gr_qmd_fetched(qmd_dst_mem); - pr_info("vgsp-gr: first-attempt witness: engine fetch %s (qmd sentinel %s) tracked=%u\n", - ff ? "OK" : "NONE", - ff ? "overwritten" : "intact", n_tracked); + /* CONT-103: no QMD deposit on the stream3d lane, so the + * sentinel is vacuous there; the GP_GET FETCH-WIT is the + * fetch witness for that lane. */ + if (qmd_dst_mem) + pr_info("vgsp-gr: first-attempt witness: engine fetch %s (qmd sentinel %s) tracked=%u\n", + ff ? "OK" : "NONE", + ff ? "overwritten" : "intact", n_tracked); + else + pr_info("vgsp-gr: first-attempt witness: engine fetch n/a (no QMD deposit on this lane - sentinel vacuous; see FETCH-WIT) tracked=%u\n", + n_tracked); } /* * r427 idempotent fast-accept: a zero-write launch whose identity @@ -21166,9 +32703,12 @@ gr_page_unmapped: words = gsp_phys_rmcall_build_gr_launch( methods, req, qmd, qmd_dst, ch_sem_va, - want_ch); - memcpy(gsp_phys_chunk_page_va(pb, 0), - methods, words * 4); + want_ch, win3d, + win3d_n, + gsp_phys_gr_attr_pre, + gsp_phys_gr_attr_pre_n); + gsp_phys_gr_pb_copy(methods, + words); } if (qmdwin_swap) pr_info("vgsp-gr: outcome retry %u: release words zeroed + fresh qmdwin bank %u dest 0x%llx + win0 bank %u%s + PB rebuilt\n", @@ -21188,6 +32728,9 @@ gr_page_unmapped: bounce_staged_hpa); gsp_phys_gr_fill_gap_bounce(bounce_csum_in, bounce_staged_hpa); + /* CONT-73: refingerprint the direct-write + * witness set for this retry publish. */ + gsp_phys_gr_wit_capture(req); mutex_lock(&gsp_phys_rm_lock); ret = gsp_phys_rmcall_publish_locked( bar0, boot_stage.rmcall_va_pb, @@ -21235,7 +32778,7 @@ gr_page_unmapped: } if (time_after_eq(jiffies, dl)) break; - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } pr_info("vgsp-gr: outcome retry %u: fresh ack %s (qmd_rel=0x%08x want=0x%08x ch_rel=0x%08x after %u ms)\n", nw_attempts, @@ -21275,6 +32818,8 @@ gr_page_unmapped: } wrote = gsp_phys_gr_output_retire_wait( req, bounce_csum_in, &n_tracked); + /* CONT-73: direct-write witness per retry. */ + gsp_phys_gr_wit_report("retry-retire"); gsp_phys_gr_qmd64_post(qmd_dst_mem, "retry-retire"); pr_info("vgsp-gr: outcome retry %u done: engine wrote %u page(s)\n", @@ -21329,7 +32874,8 @@ gr_page_unmapped: fetched ? "AMBIGUOUS" : "CONFIRMED", req->h_object, req->push_va, req->n_data, n_tracked, nw_attempts, want_qmd, - fetched ? "OK" : "NONE", + qmd_dst_mem ? (fetched ? "OK" : "NONE") : + "n/a(noQMD-stream3d)", READ_ONCE(gsp_phys_gr_nw_streak) + 1u, gsp_phys_gr_wedge_quiet, gsp_phys_gr_wedge_heals); @@ -21414,14 +32960,12 @@ gr_page_unmapped: const u8 *pp = gsp_phys_chunk_page_va( gsp_phys_gr_data_chunk(), gsp_phys_gr_prev_out_bi); - u32 pc = 0, pi; + u64 pc; if (pp) { - for (pi = 0; pi < GSP_PHYS_PAGE_SIZE; - pi += 4) - pc += get_unaligned_le32(pp + pi); + pc = gsp_phys_gr_fnv1a64(pp, GSP_PHYS_PAGE_SIZE); if (pc != gsp_phys_gr_prev_out_csum) - pr_warn("vgsp-gr: PREV-OUT-CHANGED bi=%u csum %08x -> %08x (late store after its launch shipped)\n", + pr_warn("vgsp-gr: PREV-OUT-CHANGED bi=%u fp %016llx -> %016llx (late store after its launch shipped)\n", gsp_phys_gr_prev_out_bi, gsp_phys_gr_prev_out_csum, pc); @@ -21447,6 +32991,18 @@ gr_page_unmapped: late, late_tracked, late ? "LATE-STORES - copy-out will ship them" : "NO-STORES - dispatch lost or read-only kernel"); + /* + * SESSION 153-CONTINUED-83: the no-writes moment + * is the store wall's fingerprint - snapshot the + * DS/SPH/MME/FE-decode evidence here so the + * stage that never received a program header (or + * the MME PC that never emitted the draw) is on + * the same line as the failing launch. + */ + if (!late) + gsp_phys_store_side_log(gsp_phys_gpu_bar0(), + "no-writes", + req->h_object); } if (qmd_rel_hpa) { @@ -21514,11 +33070,12 @@ gr_page_unmapped: u32 aliased = 0, unchanged = 0, skipped = 0; u32 chg[8] = { }; u32 outwit = 0; /* COPYWIT: bounded per-launch witness lines */ + u32 outwit_chg = 0; /* CONT-112: own budget, changed pages */ for (wbi = 0; wbi < req->n_data; wbi++) { u32 bi = gsp_phys_gr_data_bi(req->data[wbi].reloc_va); const u8 *src; - u32 ci, csum = 0; + u64 fp; if (bi == GSP_PHYS_GR_DATA_BI_NONE || !req->data[wbi].hpa) @@ -21539,9 +33096,8 @@ gr_page_unmapped: aliased++; continue; } - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum == bounce_csum_in[bi]) { + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { /* * r378: the engine did not touch this page, * so the guest page still holds exactly this @@ -21550,6 +33106,37 @@ gr_page_unmapped: * stale-content corruption at worst - skip. */ unchanged++; + /* + * r452: witness the UNCHANGED page's chunk + * content (first 4 words) and the guest + * page's, bounded per launch. This is the + * only place the two can be compared for a + * page the csum rule classified as "engine + * did not touch" - it separates "the store + * never landed in the chunk" from "the + * store landed but the csum rule missed + * it" (or "the reader resolves a different + * page than the writer wrote"). + */ + if (gsp_phys_gr_copywit_enabled() && + outwit < GSP_PHYS_GR_COPYWIT_MAX) { + const u8 *gp = req->data[wbi].hpa ? + (const u8 *)__va(req->data[wbi].hpa) : + NULL; + + outwit++; + pr_info("vgsp-gr: OUT-UNCH bi=%u va=0x%llx fp=%016llx pre=%016llx chunk=%08x %08x %08x %08x guest=%08x %08x %08x %08x\n", + bi, req->data[wbi].va, fp, + bounce_csum_in[bi], + get_unaligned_le32(src), + get_unaligned_le32(src + 4), + get_unaligned_le32(src + 8), + get_unaligned_le32(src + 12), + gp ? get_unaligned_le32(gp) : 0, + gp ? get_unaligned_le32(gp + 4) : 0, + gp ? get_unaligned_le32(gp + 8) : 0, + gp ? get_unaligned_le32(gp + 12) : 0); + } continue; } { @@ -21584,7 +33171,7 @@ gr_page_unmapped: * sema-bigcopy resolution lines. */ if (gsp_phys_gr_copywit_enabled() && - outwit < GSP_PHYS_GR_COPYWIT_MAX) { + outwit_chg < GSP_PHYS_GR_COPYWIT_CHG_MAX) { /* * r391 diagnostic (sum=0 class): the * landing-table resolution of this VA @@ -21599,14 +33186,14 @@ gr_page_unmapped: req->data[wbi].va, &rpa, &rap, &rval); - outwit++; - pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%08x post=%08x res=%s pa=0x%llx ap=%u val8=%016llx vbase=%px\n", + outwit_chg++; + pr_info("vgsp-gr: OUT-PAGE-WIT va=0x%llx hpa=0x%llx bi=%u first8=%016llx last8=%016llx pre=%016llx post=%016llx res=%s pa=0x%llx ap=%u val8=%016llx vbase=%px\n", req->data[wbi].va, req->data[wbi].hpa, bi, f8, (u64)get_unaligned_le64( src + GSP_PHYS_PAGE_SIZE - 8), - bounce_csum_in[bi], csum, + bounce_csum_in[bi], fp, rhave ? "ledger" : "none", rpa, rap, rval, (void *)gsp_sema_gr_shadow_base()); @@ -21621,7 +33208,8 @@ gr_page_unmapped: * evidence). */ if (gsp_phys_gr_copywit_enabled() && - !gsp_phys_gr_page_all_zero(csum, src)) { + !gsp_phys_gr_page_all_zero( + gsp_phys_gr_page_sum32(src), src)) { u32 wi; for (wi = 0; wi < gsp_phys_gr_outwit_n; @@ -21657,7 +33245,7 @@ gr_page_unmapped: */ if (changed == 0) { gsp_phys_gr_prev_out_bi = bi; - gsp_phys_gr_prev_out_csum = csum; + gsp_phys_gr_prev_out_csum = fp; gsp_phys_gr_prev_out_valid = true; } changed++; @@ -21673,7 +33261,49 @@ gr_page_unmapped: if (!gsp_sema_gr_write_guard(req->data[wbi].va, req->data[wbi].hpa, "gr-bounce-copy-out")) { - skipped++; + /* + * CONT 18: a migrated screen page - follow + * the VA's current mapping instead of + * dropping the frame (see the knob). + */ + bool followed = false; + + if (READ_ONCE(gsp_phys_gr_stale_write)) { + u64 sva = 0; + u32 snp = 0; + + if (gsp_sema_surf_va_get(&sva, &snp) && + sva && + req->data[wbi].va >= sva && + req->data[wbi].va < + sva + ((u64)snp << 12)) { + u64 rpa = 0, rval = 0; + u32 rap = 0; + + if (gsp_sema_gr_va_snap( + req->data[wbi].va, + &rpa, &rap, &rval) && + rpa) { + memcpy((u8 *)__va(rpa), + src, + GSP_PHYS_PAGE_SIZE); + followed = true; + if (gsp_phys_gr_stale_log < + 64u) { + gsp_phys_gr_stale_log++; + pr_info("vgsp-gr: WGUARD-STALE-WRITE va=0x%llx used=0x%llx now=0x%llx h=0x%x push=0x%llx\n", + req->data[wbi].va, + req->data[wbi].hpa, + rpa, + req->h_object, + req->push_va); + } + wrote++; + } + } + } + if (!followed) + skipped++; continue; } memcpy((u8 *)__va(req->data[wbi].hpa), src, @@ -21688,13 +33318,13 @@ gr_page_unmapped: * write target and the reader's page are different * (aperture/staleness disagreement), not a lost write. */ - if (outwit < GSP_PHYS_GR_COPYWIT_MAX) { + if (outwit_chg < GSP_PHYS_GR_COPYWIT_CHG_MAX) { u64 rpa2 = 0, rval2 = 0; u32 rap2 = 0; if (gsp_sema_gr_va_snap(req->data[wbi].va, &rpa2, &rap2, &rval2)) { - outwit++; + outwit_chg++; pr_info("vgsp-gr: OUT-PAGE-POST va=0x%llx hpa=0x%llx res_pa=0x%llx ap=%u val8=%016llx vbase=%px\n", req->data[wbi].va, req->data[wbi].hpa, rpa2, @@ -21738,8 +33368,7 @@ gr_page_unmapped: if (bi == GSP_PHYS_GR_DATA_BI_NONE) continue; wp = gsp_phys_chunk_page_va(dchunk, bi); - for (k = 0; k < GSP_PHYS_PAGE_SIZE; k += 4) - wc += get_unaligned_le32(wp + k); + (void)k; (void)wc; pr_info("vgsp-gr: window[%u] reloc=0x%llx first=%08x %08x %08x %08x changed=%u\n", bi, req->data[wi].reloc_va, @@ -21747,7 +33376,9 @@ gr_page_unmapped: get_unaligned_le32(wp + 4), get_unaligned_le32(wp + 8), get_unaligned_le32(wp + 12), - wc != bounce_csum_in[bi]); + gsp_phys_gr_fnv1a64( + wp, GSP_PHYS_PAGE_SIZE) != + bounce_csum_in[bi]); } } pr_info("vgsp-gr: bounce copy-out: %u pages written back, %u changed by engine, %u unchanged-skipped, %u alias-skipped, %u stale-skipped (first bi: %u %u %u %u %u %u %u %u)\n", @@ -21770,7 +33401,7 @@ gr_page_unmapped: for (gi = 0; gi < gsp_phys_gr_gap_n; gi++) { u32 bi = gsp_phys_gr_gap[gi].bi; const u8 *src = gsp_phys_chunk_page_va(dchunk, bi); - u32 ci, csum = 0; + u64 fp; if (!src) continue; @@ -21783,9 +33414,8 @@ gr_page_unmapped: gskipped++; continue; } - for (ci = 0; ci < GSP_PHYS_PAGE_SIZE; ci += 4) - csum += get_unaligned_le32(src + ci); - if (csum == bounce_csum_in[bi]) { + fp = gsp_phys_gr_fnv1a64(src, GSP_PHYS_PAGE_SIZE); + if (fp == bounce_csum_in[bi]) { /* Engine did not touch it: nothing to ship. */ gunchanged++; continue; @@ -22009,6 +33639,181 @@ static void gsp_phys_rmcall_ctx_diag_decode(const char *tag) ramfc_valid); } +/* + * SESSION 153-CONTINUED-96: secondary COPY0 lane enable. On a GR0 ladder + * the ladder channel speaks GR and CE class methods are never legal on it, + * so the ladder's own FBMEM reads (gsp_phys_ctx_readback_try) need a second + * engine. Default 1 builds the lane (best-effort: a failed build only + * disables the readback, the GR ladder is unchanged); 0 skips it entirely. + */ +static bool gsp_phys_rmcall_ce_lane = 1; +module_param_named(gsp_phys_rmcall_ce_lane, gsp_phys_rmcall_ce_lane, bool, + 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_ce_lane, + "Session 153-CONTINUED-96: build and schedule the secondary COPY0 lane (COPY0 TSG + channel on the ladder VAS, engineType 9) on a GR0 ladder so the ladder's own CE copy blocks (engine FBMEM readback) have a legal engine (default 1; 0 disables the lane, the GR ladder is unchanged)"); + +static bool gsp_phys_rmcall_ce_lane_value(void) +{ + return gsp_phys_rmcall_ce_lane; +} + +/* + * SESSION 153-CONTINUED-96: secondary COPY0 lane bringup (GR0 ladders only). + * + * The ladder's own FBMEM readback (and any later diagnostic copy of engine + * state the host cannot aperture) needs a CE engine, but CE class methods + * are never legal on the GR0 ladder channel. This builds the smallest + * second lane on the SAME VAS: its own COPY0 TSG (parent device, same + * ladder VAS) and its own V5-shape channel (engineType 9, TSG parent, + * lane memdescs, GPFIFO at VA_CE_GPFIFO), then runs the historical + * COPY0 stage-4 sequence - BIND(COPY0) -> GPFIFO_SCHEDULE -> + * TSG_GPFIFO_SCHEDULE -> notif index (advisory) -> GET_WORK_SUBMIT_TOKEN - + * and stores the lane's own work-submit token. Best-effort by design: a + * failure is logged and the GR ladder continues unchanged (no stage-4 + * abort, no retry); rmcall_ce_lane_valid stays false and the readback is + * skipped. The PDB slots for the lane's GPFIFO/PB pages were planted in + * pdb_build (stage 3, before any engine fetch). + */ +static int gsp_phys_rmcall_ce_lane_bringup(void __iomem *bar0) +{ + struct gsp_phys_rmcall_tsg_params tsg; + struct gsp_phys_rmcall_channel_params ch; + struct gsp_phys_rmcall_channel_variant v; + u32 token = 0, rpc_result = 0, status = 0; + u8 sched_params[2] = { 1, 0 }; + u32 bind_params = GSP_PHYS_RMCALL_ENGINE_COPY0; + u32 notif_index = GSP_PHYS_RMCALL_NOTIF_INDEX; + int ret; + + if (!gsp_phys_rmcall_engine_is_gr()) + return 0; /* COPY0 ladder: the lane already is it */ + if (!gsp_phys_rmcall_ce_lane_value()) + return 0; + if (boot_stage.rmcall_ce_lane_valid) + return 0; + + /* (1) TSG on the ladder VAS bound to the COPY0 engine. */ + memset(&tsg, 0, sizeof(tsg)); + tsg.h_va_space = cpu_to_le32(boot_stage.rmcall_vas); + tsg.engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_COPY0); + ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client, + boot_stage.rmcall_device, + GSP_PHYS_RMCALL_HANDLE_CE_TSG, + GSP_PHYS_RMCALL_CLASS_TSG, &tsg, sizeof(tsg), + false, NULL, "KEPLER_CHANNEL_GROUP_A (CE lane)", + true, NULL, NULL); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: COPY0 TSG alloc failed (%d); the lane is not built and the GR ladder continues unchanged\n", + ret); + return ret; + } + boot_stage.rmcall_ce_tsg = GSP_PHYS_RMCALL_HANDLE_CE_TSG; + + /* (2) The lane channel: V5 shape, COPY0 engine, lane memdescs. */ + memset(&v, 0, sizeof(v)); + v.name = "ce-lane-tsg-vas-pdb-skip-scrubber"; + v.id = 5; + v.tsg_parent = true; + v.flags = GSP_PHYS_RMCALL_CHANNEL_FLAG_SKIP_SCRUBBER; + v.internal_flags = GSP_PHYS_RMCALL_CHANNEL_INTERNALFLAGS_PRIV_KERNEL; + v.server_mem = false; + v.handle = GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL; + gsp_phys_rmcall_channel_params_init(&ch, &v); + ch.engine_type = cpu_to_le32(GSP_PHYS_RMCALL_ENGINE_COPY0); + ch.h_context_share = 0; /* a CE channel never joins a GR share */ + ch.gp_fifo_offset = cpu_to_le64(GSP_PHYS_RMCALL_VA_CE_GPFIFO); + ch.instance_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_INSTANCE); + ch.userd_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_USERD); + ch.ramfc_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_RAMFC); + ch.mthdbuf_mem = gsp_phys_rmcall_mem_desc( + GSP_PHYS_RMCALL_BUF_CE_MTHDBUF); + ret = gsp_phys_rmcall_alloc(bar0, boot_stage.rmcall_client, + boot_stage.rmcall_ce_tsg, + GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL, + GSP_PHYS_RMCALL_CLASS_CHANNEL, &ch, sizeof(ch), + false, NULL, "CE lane channel", true, NULL, + &status); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: channel alloc failed (%d, status=0x%x); the lane is not built and the GR ladder continues unchanged\n", + ret, status); + return ret; + } + boot_stage.rmcall_ce_channel = GSP_PHYS_RMCALL_HANDLE_CE_CHANNEL; + + /* (3) BIND to the COPY0 engine (the historical stage-4 sequence). */ + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_BIND_CHANNEL, + &bind_params, sizeof(bind_params), 16, NULL, 0, + &rpc_result, &status, + "CE lane BIND_CHANNEL COPY0"); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: BIND_CHANNEL (engineType=0x%x) failed (rpc_result=0x%x status=0x%x); lane unusable, GR ladder continues\n", + bind_params, rpc_result, status); + return ret; + } + + /* (4) GPFIFO_SCHEDULE + TSG_GPFIFO_SCHEDULE. */ + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_GPFIFO_SCHEDULE, + sched_params, sizeof(sched_params), 16, NULL, 0, + &rpc_result, &status, "CE lane GPFIFO_SCHEDULE"); + if (ret) { + pr_warn("vgsp-boot: rm-call: CE lane: GPFIFO_SCHEDULE failed (rpc_result=0x%x status=0x%x); lane unusable, GR ladder continues\n", + rpc_result, status); + return ret; + } + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_tsg, + GSP_PHYS_RMCALL_CTRL_TSG_GPFIFO_SCHEDULE, + sched_params, sizeof(sched_params), 16, NULL, 0, + &rpc_result, &status, "CE lane TSG_GPFIFO_SCHEDULE"); + if (ret) + pr_warn("vgsp-boot: rm-call: CE lane: TSG_GPFIFO_SCHEDULE advisory failure (rpc_result=0x%x status=0x%x); continuing\n", + rpc_result, status); + + /* (5) notif index (advisory) + the lane's own work-submit token. */ + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX, + ¬if_index, sizeof(notif_index), 16, NULL, 0, + &rpc_result, &status, + "CE lane SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX"); + if (ret) + pr_info("vgsp-boot: rm-call: CE lane: SET_WORK_SUBMIT_TOKEN_NOTIF_INDEX advisory failure (rpc_result=0x%x status=0x%x); continuing to the token\n", + rpc_result, status); + token = 0; + rpc_result = 0; + status = 0; + ret = gsp_phys_rmcall_control(bar0, boot_stage.rmcall_ce_channel, + GSP_PHYS_RMCALL_CTRL_GET_WORK_SUBMIT_TOKEN, + &token, sizeof(token), 28, &token, sizeof(token), + &rpc_result, &status, "CE lane GET_WORK_SUBMIT_TOKEN"); + if (ret && token != 0) { + pr_warn("vgsp-boot: rm-call: CE lane: GET_WORK_SUBMIT_TOKEN failed (rpc_result=0x%x status=0x%x) but the echoed params carry token=0x%x; proceeding\n", + rpc_result, status, token); + ret = 0; + } + if (ret || !token) { + pr_warn("vgsp-boot: rm-call: CE lane: no work-submit token (ret=%d rpc_result=0x%x status=0x%x token=0x%x); lane unusable, GR ladder continues\n", + ret, rpc_result, status, token); + return ret ? ret : -EIO; + } + boot_stage.rmcall_ce_token = token; + boot_stage.rmcall_ce_lane_valid = true; + pr_info("vgsp-boot: rm-call: CE lane READY (secondary COPY0 on the ladder VAS): hTsg=0x%x hChannel=0x%x token=0x%x (runlistId=%u chId=0x%x) gpFifoVa=0x%llx pbVa=0x%llx; CE copies on it read FBMEM the host cannot aperture\n", + boot_stage.rmcall_ce_tsg, boot_stage.rmcall_ce_channel, token, + (token >> 16) & 0x7fu, token & 0xfffu, + GSP_PHYS_RMCALL_VA_CE_GPFIFO, GSP_PHYS_RMCALL_VA_CE_PB); + return 0; +} + static int gsp_phys_rmcall_stage4(void __iomem *bar0) { struct gsp_phys_boot_chunk *userd_c = &boot_stage.rmcall_chunks[GSP_PHYS_RMCALL_BUF_USERD]; @@ -22190,6 +33995,22 @@ static int gsp_phys_rmcall_stage4(void __iomem *bar0) boot_stage.rmcall_token = token; /* + * SESSION 153-CONTINUED-96: on a GR0 ladder, build the secondary + * COPY0 lane (own TSG/channel on the same VAS, own token) so the + * ladder's own CE copy blocks - the engine-context FBMEM readback - + * have a legal engine. Best-effort: a failure disables the readback + * only; the GR ladder below is byte-identical. On a COPY0 ladder the + * bringup is a no-op (the ladder channel already is the CE lane). + */ + { + int ce_ret = gsp_phys_rmcall_ce_lane_bringup(bar0); + + if (ce_ret) + pr_warn("vgsp-boot: rm-call: CE lane bringup failed (%d); engine FBMEM readback unavailable this boot (GR ladder unchanged)\n", + ce_ret); + } + + /* * H1: a GR0 ladder releases a semaphore on the compute engine instead * of running a CE copy. The dedicated helper performs the same guarded * submission (gr=true), doorbell and module-sem poll, logs REAL GR SEM @@ -22259,7 +34080,7 @@ static int gsp_phys_rmcall_stage4(void __iomem *bar0) gsp_phys_status_drain(NULL); return -ETIMEDOUT; } - msleep(GSP_PHYS_RMCALL_POLL_MS); + gsp_phys_rmcall_wait_quantum(); } if (memcmp(gsp_phys_chunk_page_va(src, 0), @@ -22736,6 +34557,8 @@ static void gsp_phys_rmcall_fini(struct pci_dev *pdev) gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_pt[i]); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_shadow); gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_qmdwin); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_pb2); + gsp_phys_chunk_free(pdev, &boot_stage.rmcall_gr_probe); /* * B-2: the old 208-page bounce window (previously leaked here) and the * grown data-bounce region chunks. The region's PT chunk and data chunk @@ -22880,6 +34703,17 @@ static int gsp_phys_state_show(struct seq_file *m, void *v) boot_stage.rmcall_gr_promote_step, boot_stage.rmcall_gr_promote_size, boot_stage.rmcall_gr_hvirt); + /* SESSION 153-CONTINUED-96: secondary COPY0 lane (GR0 ladders only). */ + if (boot_stage.rmcall_ce_tsg || boot_stage.rmcall_ce_lane_valid) + seq_printf(m, + "rm_call_ce_lane: valid=%u hTsg=0x%x hChannel=0x%x token=0x%x gpFifoVa=0x%llx pbVa=0x%llx ceLaneParam=%u\n", + boot_stage.rmcall_ce_lane_valid, + boot_stage.rmcall_ce_tsg, + boot_stage.rmcall_ce_channel, + boot_stage.rmcall_ce_token, + GSP_PHYS_RMCALL_VA_CE_GPFIFO, + GSP_PHYS_RMCALL_VA_CE_PB, + gsp_phys_rmcall_ce_lane_value() ? 1u : 0u); seq_printf(m, "rm_call_guest: gpa=0x%llx hpa=0x%llx g2h_ok=%u w2g_ok=%u selftest_ok=%u gpa_valid=%u phys_domain_identity=%d\n", boot_stage.rmcall_guest_gpa, boot_stage.rmcall_guest_hpa, @@ -23769,6 +35603,91 @@ static const struct file_operations gsp_phys_guest_gpa_fops = { }; /* + * gsp-phys-shared-bit0 (write-only, mode 0200): development-harness assist + * for the NVIDIA X driver's per-client shared-data slot. + * + * Every direct GLX client receives a shared memory page from the X driver + * (nvidia_drv.so creates a memfd and passes it over a unix socket) whose + * first words are {u32 zero, u32 cookie, u32 state, u32 zero, u32 hClient, + * u32 hObject}; glcore's device init reads the state word and skips its own + * class-c46f channel creation while bit 0 of it is set. In this emulated + * environment the X driver publishes state=3 (bits 0+1) and direct clients + * therefore never allocate a channel and never render. The value is written + * by the X driver before the fd reaches the client, so it cannot be + * corrected from the client side. + * + * Input: "0x<hex GPA>" (4 KB aligned) naming the guest page that carries the + * slot. The GPA must fall inside a range registered through + * gsp-phys-guest-ram; it is translated with gsp_phys_gpa_to_hpa_vgpu() + * (first open mdev) and the state word is cleared of bit 0 through the + * kernel direct map (the translator validates pfn_valid, so __va() is a + * valid kernel mapping for the shmem-backed guest RAM). As a sanity guard + * the write is only performed when the page's +0x10 word looks like an RM + * client handle (0xc1d0xxxx / 0xc1e0xxxx) or the token carries a "force" + * suffix. Old and new values are always logged. + * + * This is a diagnostic aid for the development VM, matching the spirit of + * the existing guest-GPA pin; it is not part of the product behaviour. + */ +static ssize_t gsp_phys_shared_bit0_write(struct file *file, + const char __user *ubuf, size_t len, + loff_t *ppos) +{ + char buf[48]; + u64 gpa, hpa = 0; + u32 *word; + u32 h1032; + bool force = false; + int ret; + + if (len == 0 || len >= sizeof(buf)) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + buf[len] = '\0'; + + { + char *p = strim(buf); + char *tok = strsep(&p, " \t"); + + if (p && strstr(p, "force")) + force = true; + if (kstrtoull(tok, 0, &gpa) || !PAGE_ALIGNED(gpa)) { + pr_info("vgsp-boot: gsp-phys-shared-bit0: write ignored: expected \"0x<hex GPA>\" (4 KB aligned, optional 'force')\n"); + return -EINVAL; + } + } + + ret = gsp_phys_gpa_to_hpa_vgpu(NULL, gpa, &hpa); + if (ret) { + pr_err("vgsp-boot: gsp-phys-shared-bit0: GPA 0x%llx translation failed (%d: no open mdev / GPA not inside a gsp-phys-guest-ram range / not RAM)\n", + gpa, ret); + return -ENOENT; + } + + word = (u32 *)__va(hpa + 8); + h1032 = *(u32 *)__va(hpa + 0x10); + if (!force && (h1032 >> 16) != 0xc1d0 && (h1032 >> 16) != 0xc1e0) { + pr_info("vgsp-boot: gsp-phys-shared-bit0: gpa=0x%llx word=0x%x but +0x10=0x%x does not look like an RM client handle; refusing (append 'force' to override)\n", + gpa, *word, h1032); + return -EINVAL; + } + + pr_info("vgsp-boot: gsp-phys-shared-bit0: gpa=0x%llx hpa=0x%llx hClient=0x%x word=0x%x -> 0x%x\n", + gpa, hpa, h1032, *word, *word & ~1u); + *word &= ~1u; + return len; +} + +static const struct file_operations gsp_phys_shared_bit0_fops = { + .owner = THIS_MODULE, + .open = simple_open, + .write = gsp_phys_shared_bit0_write, + .llseek = no_llseek, +}; + + +/* * gsp-phys-guest-ram (write-only, mode 0200): configure a guest RAM backing * file that gsp_phys_gpa_to_hpa_vgpu() translates through (QEMU * memory-backend-file,...,share=yes). One write is @@ -23971,12 +35890,54 @@ static int gsp_phys_reg_probe_show(struct seq_file *m, void *unused) 0x0040070cu); fe_hww_esr = gsp_phys_reg_probe_line(m, bar0, "FE_HWW_ESR", 0x00404000u); + /* + * SESSION 153-CONTINUED-52: MME exception latches (see + * gsp_phys_mme_regs_log). + */ + (void)gsp_phys_reg_probe_line(m, bar0, "PGRAPH_PRI_MME_HWW_ESR", + 0x00404490u); + (void)gsp_phys_reg_probe_line(m, bar0, + "PGRAPH_PRI_MME_HWW_ESR_INFO", + 0x00404494u); + (void)gsp_phys_reg_probe_line(m, bar0, + "PGRAPH_PRI_MME_HWW_ESR_INFO2", + 0x0040449cu); fe_chip_def = gsp_phys_reg_probe_line(m, bar0, "FE_CHIP_DEF_INFO", 0x00404030u); fe_go_idle = gsp_phys_reg_probe_line(m, bar0, "FE_GO_IDLE_INFO", 0x00404194u); sked_hww_esr = gsp_phys_reg_probe_line(m, bar0, "SKED_HWW_ESR", 0x00407020u); + /* + * SESSION 153-CONTINUED-11: the shader-program-header fault window. + * DS_HWW_ESR bit n = "the SPH with index n failed validation" (hwref: + * NV_PGRAPH_PRI_DS_HWW_ESR_SPH0_ERR..SPH23_ERR, ESR_2 for 24..30); + * the SPH_INFO registers snapshot the START address of each stage's + * faulting SPH, so a post-fault read names the exact VA the engine + * fetched. + */ + (void)gsp_phys_reg_probe_line(m, bar0, "DS_HWW_ESR", + 0x00405840u); + (void)gsp_phys_reg_probe_line(m, bar0, "DS_HWW_ESR_2", + 0x00405848u); + (void)gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_A", + 0x00405c1cu); + (void)gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_PROGRAM_REGION_B", + 0x00405c20u); + { + u32 si; + + for (si = 0; si < 6u; si++) { + gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_STAGE_START", + 0x00405c00u + si * 4u); + gsp_phys_reg_probe_line(m, bar0, + "DS_HWW_ESR_SPH_INFO_STAGE_START_UPPER", + 0x00405c24u + si * 4u); + } + } sked_activity = gsp_phys_reg_probe_line(m, bar0, "SKED_ACTIVITY", 0x00407054u); sked_cwd0 = gsp_phys_reg_probe_line(m, bar0, @@ -24131,6 +36092,306 @@ static const struct file_operations gsp_phys_reg_probe_fops = { .release = single_release, }; +/* + * SESSION 153-CONTINUED-91: DS SPH fault spy. + * + * The DS_HWW_ESR SPH<n>_ERR bits (bit n = "the SPH with index n failed + * validation", hwref NV_PGRAPH_PRI_DS_HWW_ESR_SPH0_ERR..SPH23_ERR) and the + * per-stage SPH_INFO snapshot (STAGE0..STAGE5 START/UPPER at 0x405c00+si*4 / + * 0x405c24+si*4, PROGRAM_REGION_A/B at 0x405c1c/c20) are latched by the DS + * at fault time and consumed+cleared by the real GSP-RM exception handler + * before our relayed fault packet arrives (the fault-time register read in + * gsp_phys_fault_regs_log sees ESR=0x80000000 and all-zero stage starts + * one millisecond later). The RM's own diagnostic prints those STAGE + * registers (grErrorPrintDsSphInfo_GM200), so they name the exact SPH VAs + * the DS rejected - the missing datum behind the r1277+ Shader Program + * Header wall (CONT-90/91 could only move the fault SET by content, never + * identify the SPH entries). + * + * Reading this file runs a bounded tight poll of DS_HWW_ESR and snapshots + * the ESR pair, the six stage starts and the region pair on every sample + * that has any SPH bit set. Development-only, read-only, no state touched; + * run `cat /sys/kernel/debug/vgsp/gsp-phys-ds-spy` while the client draws. + * Duration via sema_ds_spy_ms (default 20 s). + */ +static u32 gsp_phys_ds_spy_ms = 20000; +module_param_named(sema_ds_spy_ms, gsp_phys_ds_spy_ms, uint, 0644); +MODULE_PARM_DESC(sema_ds_spy_ms, + "gsp-phys-ds-spy sampling duration in milliseconds (default 20000)"); + +#define GSP_PHYS_DS_ESR 0x00405840u +#define GSP_PHYS_DS_ESR_2 0x00405848u +#define GSP_PHYS_DS_REPORT_MASK 0x00405844u +#define GSP_PHYS_DS_REPORT_MASK_2 0x0040584cu +#define GSP_PHYS_DS_SPH_STAGE_LO 0x00405c00u +#define GSP_PHYS_DS_SPH_STAGE_UP 0x00405c24u +#define GSP_PHYS_DS_SPH_REGION_A 0x00405c1cu +#define GSP_PHYS_DS_SPH_REGION_B 0x00405c20u + +static int gsp_phys_ds_spy_show(struct seq_file *m, void *unused) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + u64 t0; + u32 hits = 0, samples = 0, maskholds = 0, mask0_last = 0, mask2_last = 0; + u32 ms = READ_ONCE(gsp_phys_ds_spy_ms); + + if (!bar0) + return -ENODEV; + if (!ms) + ms = 20000; + seq_printf(m, "# DS-SPY begin duration_ms=%u (poll DS_HWW_ESR 0x%08x)\n", + ms, GSP_PHYS_DS_ESR); + seq_printf(m, "# DS-SPY masks esr=%08x esr2=%08x mask=%08x mask2=%08x\n", + readl(bar0 + GSP_PHYS_DS_ESR), + readl(bar0 + GSP_PHYS_DS_ESR_2), + readl(bar0 + GSP_PHYS_DS_REPORT_MASK), + readl(bar0 + GSP_PHYS_DS_REPORT_MASK_2)); + t0 = ktime_get_ns(); + while (ktime_get_ns() - t0 < (u64)ms * 1000000ull) { + u32 esr = readl(bar0 + GSP_PHYS_DS_ESR); + u32 m0 = readl(bar0 + GSP_PHYS_DS_REPORT_MASK); + u32 m2 = readl(bar0 + GSP_PHYS_DS_REPORT_MASK_2); + + /* + * SESSION 153-CONTINUED-97: hold the HWW report masks + * cleared while sampling. A pending report POISONS the + * whole PRI block (every read returns 0xbadf1201), which + * hid the SPH stage snapshot at the validation instant + * (r1325 spy2: all 16+ hits badf1201, masks back at + * 00ffffff). The mask is (re)armed by the context load / + * the RM setup, not only by our replay path, so re-clear + * it every sample; the re-clear count is logged so the + * capture's provenance is visible. NOTE (r1326): poison + * still appeared on one pass despite 29 re-clears, so a + * clean latch is not guaranteed and the store-side read + * remains the authoritative fault-time datum. + * Development-only diagnostic tap. + */ + if (m0 || m2) { + iowrite32(0u, bar0 + GSP_PHYS_DS_REPORT_MASK); + iowrite32(0u, bar0 + GSP_PHYS_DS_REPORT_MASK_2); + maskholds++; + mask0_last = m0; + mask2_last = m2; + } + samples++; + if (esr & 0x7fffffffu) { + u32 esr2 = readl(bar0 + GSP_PHYS_DS_ESR_2); + u32 esr_a, s[6], u[6], ra, rb; + u32 si; + + for (si = 0; si < 6u; si++) { + s[si] = readl(bar0 + + GSP_PHYS_DS_SPH_STAGE_LO + + si * 4u); + u[si] = readl(bar0 + + GSP_PHYS_DS_SPH_STAGE_UP + + si * 4u); + } + ra = readl(bar0 + GSP_PHYS_DS_SPH_REGION_A); + rb = readl(bar0 + GSP_PHYS_DS_SPH_REGION_B); + esr_a = readl(bar0 + GSP_PHYS_DS_ESR); + hits++; + seq_printf(m, + "DS-SPY hit %u t=%lldus esr=%08x esr2=%08x esr_after=%08x region=%08x/%08x stage=%08x,%08x,%08x,%08x,%08x,%08x upper=%02x,%02x,%02x,%02x,%02x,%02x\n", + hits, div_u64(ktime_get_ns() - t0, 1000), + esr, esr2, esr_a, ra, rb, + s[0], s[1], s[2], s[3], s[4], s[5], + u[0] & 0xffu, u[1] & 0xffu, u[2] & 0xffu, + u[3] & 0xffu, u[4] & 0xffu, u[5] & 0xffu); + udelay(20); + } + } + seq_printf(m, "# DS-SPY end samples=%u hits=%u maskholds=%u last_mask=%08x/%08x\n", + samples, hits, maskholds, mask0_last, mask2_last); + return 0; +} + +static int gsp_phys_ds_spy_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_ds_spy_show, NULL); +} + +static const struct file_operations gsp_phys_ds_spy_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_ds_spy_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* + * SESSION 153-CONTINUED-93: live PGRAPH register lab (debugfs + * gsp-phys-pgraph-rw). Read: a fixed snapshot of the DS/PD/FIFO registers + * the SPH investigation reads. Write: "0xADDR [0xVALUE]" (hex) inside the + * PGRAPH PRI window [0x400000,0x410000) - reads the old value, writes the + * new one (when given) and reads back, logging both. Diagnostic only; + * development bring-up tool. Writes outside the window are refused. + */ +static int gsp_phys_pgraph_show(struct seq_file *m, void *unused) +{ + void __iomem *bar0 = gsp_phys_gpu_bar0(); + + if (!bar0) + return -ENODEV; + seq_printf(m, "# PGRAPH register snapshot\n"); + seq_printf(m, "PD_CYA 0x00406020 %08x\n", readl(bar0 + 0x00406020u)); + seq_printf(m, "PD_HWW_ESR 0x00406018 %08x\n", readl(bar0 + 0x00406018u)); + seq_printf(m, "DS_HWW_ESR 0x00405840 %08x\n", readl(bar0 + 0x00405840u)); + seq_printf(m, "DS_HWW_ESR_2 0x00405848 %08x\n", readl(bar0 + 0x00405848u)); + seq_printf(m, "DS_REPORT_MASK 0x00405844 %08x\n", readl(bar0 + 0x00405844u)); + seq_printf(m, "DS_REPORT_MASK2 0x0040584c %08x\n", readl(bar0 + 0x0040584cu)); + seq_printf(m, "DS_SPH_REGION_A 0x00405c1c %08x\n", readl(bar0 + 0x00405c1cu)); + seq_printf(m, "DS_SPH_REGION_B 0x00405c20 %08x\n", readl(bar0 + 0x00405c20u)); + seq_printf(m, "DS_SPH_STAGE1 0x00405c04 %08x\n", readl(bar0 + 0x00405c04u)); + seq_printf(m, "DS_SPH_STAGE2 0x00405c08 %08x\n", readl(bar0 + 0x00405c08u)); + seq_printf(m, "DS_SPH_STAGE5 0x00405c14 %08x\n", readl(bar0 + 0x00405c14u)); + seq_printf(m, "PGRAPH_STATUS 0x00400700 %08x\n", readl(bar0 + 0x00400700u)); + seq_printf(m, "GR_FIFO_STATUS 0x00400504 %08x\n", readl(bar0 + 0x00400504u)); + return 0; +} + +static int gsp_phys_pgraph_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_pgraph_show, NULL); +} + +static ssize_t gsp_phys_pgraph_write(struct file *file, + const char __user *ubuf, size_t len, + loff_t *off) +{ + char buf[64]; + unsigned long long addr = 0, val = 0; + void __iomem *bar0 = gsp_phys_gpu_bar0(); + int n, has_val; + u32 old, now; + + if (!bar0) + return -ENODEV; + if (len == 0 || len >= sizeof(buf)) + return -EINVAL; + if (copy_from_user(buf, ubuf, len)) + return -EFAULT; + buf[len] = '\0'; + n = sscanf(buf, "%llx %llx", &addr, &val); + if (n < 1) + return -EINVAL; + has_val = (n >= 2); + if (addr < 0x00400000ull || addr >= 0x00410000ull) + return -EINVAL; + old = readl(bar0 + addr); + if (has_val) { + writel((u32)val, bar0 + addr); + now = readl(bar0 + addr); + pr_warn("vgsp-boot: pgraph-rw 0x%08llx old=%08x new=%08x (wrote %08llx)\n", + addr, old, now, val); + } else { + pr_info("vgsp-boot: pgraph-rw 0x%08llx = %08x (read)\n", + addr, old); + } + return (ssize_t)len; +} + +static const struct file_operations gsp_phys_pgraph_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_pgraph_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, + .write = gsp_phys_pgraph_write, +}; + +/* + * SESSION 153-CONTINUED-94: gsp-phys-init-push - the first large beef push + * captured at decode time (see svc_init_push_scan). Prints a header with + * the push address/length, the local-memory-run count (SET_SHADER_LOCAL_ + * MEMORY_A..D seen in ANY beef push so far; 0 = the client never emitted + * the download-state local-memory block) and the bytes as 16-byte rows. + */ +static int gsp_phys_init_push_show(struct seq_file *m, void *unused) +{ + const u8 *buf; + u32 len, h, lmem, off; + u64 va; + + buf = gsp_sema_init_cap_get(&len, &h, &va, &lmem); + if (!len) { + seq_puts(m, "vgsp-init-push: empty (no >=4 KiB beef push with non-zero head captured yet)\n"); + return 0; + } + seq_printf(m, "vgsp-init-push: h=0x%x va=0x%llx len=%u lmem_runs=%u\n", + h, va, len, lmem); + for (off = 0; off + 4u <= len; off += 16u) { + u32 k, n = min_t(u32, 4u, (len - off) / 4u); + + seq_printf(m, "%06x:", off); + for (k = 0; k < n; k++) { + u32 w; + + memcpy(&w, buf + off + k * 4u, 4); + seq_printf(m, " %08x", w); + } + seq_putc(m, '\n'); + } + return 0; +} + +static int gsp_phys_init_push_open(struct inode *inode, struct file *file) +{ + return single_open(file, gsp_phys_init_push_show, NULL); +} + +static const struct file_operations gsp_phys_init_push_fops = { + .owner = THIS_MODULE, + .open = gsp_phys_init_push_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* + * SESSION 153-CONTINUED-95: gsp-phys-ctx-dump - raw binary readback of the + * engine context buffer (see gsp_phys_ctx_readback_try). Empty until the + * first launch of the boot has run the CE readback. + */ +static ssize_t gsp_phys_ctx_dump_read(struct file *filp, char __user *ubuf, + size_t count, loff_t *ppos) +{ + const u8 *buf; + u32 len = 0; + + buf = gsp_phys_ctx_readback_get(&len); + if (!buf || !len) + return 0; + return simple_read_from_buffer(ubuf, count, ppos, buf, len); +} + +static const struct file_operations gsp_phys_ctx_dump_fops = { + .owner = THIS_MODULE, + .read = gsp_phys_ctx_dump_read, + .llseek = default_llseek, +}; + +/* + * SESSION 153-CONTINUED-96: gsp-phys-ctx-read - write-only re-arm for the + * one-shot engine-context readback. `echo 1 > .../gsp-phys-ctx-read` clears + * the tried flag; the next forwarded launch re-copies MAIN+PATCH with the + * current server state. The copy itself still runs only in the launch path. + */ +static ssize_t gsp_phys_ctx_read_write(struct file *filp, + const char __user *ubuf, + size_t count, loff_t *ppos) +{ + gsp_phys_ctx_readback_rearm(); + pr_info("vgsp-gr: CTX-READBACK re-armed; the next forwarded launch re-copies the engine ctx (MAIN+PATCH)\n"); + return count; +} + +static const struct file_operations gsp_phys_ctx_read_fops = { + .owner = THIS_MODULE, + .write = gsp_phys_ctx_read_write, +}; + int gsp_phys_boot_debugfs_init(void) { struct dentry *file; @@ -24223,6 +36484,20 @@ int gsp_phys_boot_debugfs_init(void) err); return (int)err; } + file = debugfs_create_file("gsp-phys-shared-bit0", 0200, + gsp_phys_debugfs_root, NULL, + &gsp_phys_shared_bit0_fops); + if (IS_ERR_OR_NULL(file)) { + long err = IS_ERR(file) ? PTR_ERR(file) : -ENODEV; + + /* + * The shared-slot bit0 assist is optional; the other + * triggers and the state snapshot stay registered. + */ + pr_warn("vgsp-boot: debugfs file gsp-phys-shared-bit0 creation failed: %ld (other triggers and state snapshot stay available)\n", + err); + return (int)err; + } file = debugfs_create_file("gsp-phys-guest-ram", 0200, gsp_phys_debugfs_root, NULL, &gsp_phys_guest_ram_fops); @@ -24279,7 +36554,57 @@ int gsp_phys_boot_debugfs_init(void) err); return (int)err; } - pr_info("vgsp-boot: debugfs files /sys/kernel/debug/vgsp/{gsp-phys-state read-only, gsp-phys-log-dump read-only, gsp-phys-status-drain read-only, gsp-phys-reg-probe read-only, gsp-phys-unload-rpc write-only, gsp-phys-rm-probe write-only, gsp-phys-rm-call write-only, gsp-phys-guest-gpa write-only, gsp-phys-guest-ram write-only} ready\n"); + /* + * SESSION 153-CONTINUED-91: the DS SPH fault spy (see the show fn). + * Diagnostic only; failure to create it must not fail the init. + */ + file = debugfs_create_file("gsp-phys-ds-spy", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ds_spy_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ds-spy creation failed (DS SPH fault spy unavailable)\n"); + /* + * SESSION 153-CONTINUED-93: live PGRAPH register lab (read snapshot + + * bounded writes inside the PGRAPH PRI window). Diagnostic only. + */ + file = debugfs_create_file("gsp-phys-pgraph-rw", 0644, + gsp_phys_debugfs_root, NULL, + &gsp_phys_pgraph_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-pgraph-rw creation failed (PGRAPH register lab unavailable)\n"); + /* + * SESSION 153-CONTINUED-94: the captured first large beef push (the + * GL client's channel-init stream) verbatim, so the local-memory + * census and any other method in it can be verified offline after + * the guest's transient pushbuffer is gone. Diagnostic only. + */ + file = debugfs_create_file("gsp-phys-init-push", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_init_push_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-init-push creation failed (init-push capture unavailable)\n"); + /* + * SESSION 153-CONTINUED-95: raw binary dump of the engine context + * buffer readback (64 KB from GSP_PHYS_RMCALL_VA_GR_CTX, taken once + * per boot at the first launch). 0444; empty until the first launch + * happens. Read with `dd if=...gsp-phys-ctx-dump of=/tmp/ctx.bin`. + */ + file = debugfs_create_file("gsp-phys-ctx-dump", 0444, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ctx_dump_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ctx-dump creation failed (engine ctx readback unavailable)\n"); + /* + * SESSION 153-CONTINUED-96: write-only re-arm for the readback (see + * gsp_phys_ctx_read_write). `echo 1 >` it to re-copy MAIN+PATCH at + * the next forwarded launch without a module reload. + */ + file = debugfs_create_file("gsp-phys-ctx-read", 0200, + gsp_phys_debugfs_root, NULL, + &gsp_phys_ctx_read_fops); + if (IS_ERR_OR_NULL(file)) + pr_warn("vgsp-boot: debugfs file gsp-phys-ctx-read creation failed (readback re-arm unavailable)\n"); + pr_info("vgsp-boot: debugfs files /sys/kernel/debug/vgsp/{gsp-phys-state read-only, gsp-phys-log-dump read-only, gsp-phys-status-drain read-only, gsp-phys-reg-probe read-only, gsp-phys-ds-spy read-only, gsp-phys-ctx-dump read-only, gsp-phys-ctx-read write-only, gsp-phys-unload-rpc write-only, gsp-phys-rm-probe write-only, gsp-phys-rm-call write-only, gsp-phys-guest-gpa write-only, gsp-phys-guest-ram write-only} ready\n"); return 0; } @@ -24633,8 +36958,8 @@ static_assert(offsetof(struct gsp_phys_rmcall_channel_params, process_id) == 296 "NV_CHANNEL_ALLOC_PARAMS.ProcessID must be at offset 296"); static_assert(offsetof(struct gsp_phys_rmcall_channel_params, tpc_config_id) == 360, "NV_CHANNEL_ALLOC_PARAMS.tpcConfigID must be at offset 360"); -static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 81, - "the rm-call ladder registers exactly eighty-one DMA buffers (14 ladder + 16 single-page guest scratch PTs + 48 single-page extension scratch PTs + 3 golden9g VAS levels)"); +static_assert(GSP_PHYS_RMCALL_BUF_COUNT == 87, + "the rm-call ladder registers exactly eighty-seven DMA buffers (14 ladder + 16 single-page guest scratch PTs + 48 single-page extension scratch PTs + 3 golden9g VAS levels + 6 secondary CE lane buffers)"); static_assert((GSP_PHYS_RMCALL_HANDLE_MEM_BASE + (GSP_PHYS_RMCALL_BUF_COUNT - 1) * GSP_PHYS_RMCALL_HANDLE_MEM_STRIDE) < 0xcaf80000u, @@ -24648,11 +36973,26 @@ static_assert(((GSP_PHYS_RMCALL_VA_BASE >> 47) & 0x3) == 0 && ((GSP_PHYS_RMCALL_VA_BASE >> 21) & 0xff) == 0 && ((GSP_PHYS_RMCALL_VA_SEM >> 21) & 0xff) == 0, "the channel PDB VA window must sit in root/PD2/PD1/PD0 index 0 (GP10X 48:47/46:38/37:29/28:21 splits)"); -static_assert(((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) != - ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) && - (((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) + 4) == - ((GSP_PHYS_RMCALL_VA_SEM >> 12) & 0x1ff), - "the GPFIFO, pushbuffer and CE src/dst/semaphore VAs must take the five consecutive small-PT slots 0x100..0x104"); +static_assert(((GSP_PHYS_RMCALL_VA_GPFIFO >> 12) & 0x1ff) == 0x100 && + ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) == 0x101 && + ((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ff) == 0x102 && + ((GSP_PHYS_RMCALL_VA_PB3 >> 12) & 0x1ff) == 0x103 && + ((GSP_PHYS_RMCALL_VA_PB4 >> 12) & 0x1ff) == 0x104, + "the GPFIFO + four pushbuffer pages must occupy the consecutive small-PT slots 0x100..0x104 for the engine's linear fetch walk (SESSION 153-CONTINUED-77: CE src/dst/sem moved off the walk to 0x109/0x10b/0x10c)"); +/* + * SESSION 153-CONTINUED-66: the two-page pushbuffer and the moved CE SRC / + * probe pages must stay inside the same 2 MB PD0 slot and the same 128 B PTE + * fetch line as the ladder, and must not collide with each other or with the + * QMD-window pair (0x107/0x108). + */ +static_assert(((GSP_PHYS_RMCALL_VA_PB2 >> 12) & 0x1ff) == + ((GSP_PHYS_RMCALL_VA_PB >> 12) & 0x1ff) + 1, + "the pushbuffer's second page must sit directly above the first (one contiguous GPU VA fetch)"); +static_assert(((GSP_PHYS_RMCALL_VA_SRC >> 21) & 0xff) == 0 && + ((GSP_PHYS_RMCALL_VA_GR_PROBE >> 21) & 0xff) == 0 && + ((GSP_PHYS_RMCALL_VA_SRC >> 12) & 0x1ff) == 0x109 && + ((GSP_PHYS_RMCALL_VA_GR_PROBE >> 12) & 0x1ff) == 0x10a, + "the moved CE SRC page and the warm-semaphore probe page must take the free slots 0x109/0x10a in the ladder's PTE line"); static_assert(((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 47) & 0x3) == 0 && ((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 38) & 0x1ff) == 0 && ((GSP_PHYS_RMCALL_VA_GUEST_PT_BASE >> 29) & 0x1ff) == 0 && @@ -24689,18 +37029,20 @@ static_assert(GSP_PHYS_RMCALL_VA_GR_CTX + static_assert(sizeof(struct gsp_phys_gr_page) == 24, "the GR forwarding page descriptor must stay {u64,u64,u32,u32} = 24 bytes"); static_assert(offsetof(struct gsp_phys_gr_inline, data) == 12 && - sizeof(struct gsp_phys_gr_inline) == 80, - "the GR forwarding inline record must stay {u64,u32,u8[64]} padded to 80 bytes"); + sizeof(struct gsp_phys_gr_inline) >= + 12 + GSP_PHYS_GR_INLINE_BYTES, + "the GR forwarding inline record must stay {u64,u32,u8[N]}"); static_assert((2u + 1u + GSP_PHYS_RMCALL_GR_QMD_INLINE_WORDS + 8u) <= GSP_PHYS_RMCALL_GR_METHOD_WORDS, "the GR launch method block (compute init + SET_OBJECT + inline-QMD run + PCAS + REPORT_SEMAPHORE) must fit the bound"); -static_assert(GSP_PHYS_RMCALL_GR_INIT_WORDS + GSP_PHYS_GR_RAW_WORDS + +static_assert(GSP_PHYS_RMCALL_GR_INIT_WORDS + + GSP_PHYS_RMCALL_GR_QMD_WINDOW_WORDS + GSP_PHYS_RMCALL_GR_TAIL_WORDS <= GSP_PHYS_RMCALL_GR_METHOD_WORDS, - "the verbatim GR launch block (compute init + raw tail + PCAS + REPORT_SEMAPHORE) must fit the bound"); + "the verbatim GR launch block minimum (compute init + window + PCAS + REPORT_SEMAPHORE) must fit the bound; the per-launch runtime check bounds raw_qmd_hdr for larger preludes"); static_assert(GSP_PHYS_RMCALL_GR_METHOD_WORDS * 4u <= - GSP_PHYS_RMCALL_COPY_SIZE, - "the GR launch method block must fit one module pushbuffer page"); + 4u * GSP_PHYS_RMCALL_COPY_SIZE, + "the GR launch method block may span the four module pushbuffer pages (SESSION 153-CONTINUED-77: prelude cap 3584 + largest chunked window)"); /* * GspSystemInfo wire ABI, pinned against the reference header compile diff --git a/kernel/gsp_phys_booter.h b/kernel/gsp_phys_booter.h index 80beb54..f821a85 100644 --- a/kernel/gsp_phys_booter.h +++ b/kernel/gsp_phys_booter.h @@ -164,6 +164,16 @@ int gsp_phys_backend_static_wait(struct gsp_phys_backend_static_config *out, void gsp_phys_backend_static_invalidate(void); /* + * Mode-35 KGR static-data snapshots (see gsp_phys_booter.c): the live GSP's + * replies to the real client's KGR query set, captured at backend boot and + * served byte-identically by the guest-side emulation. Returns 0 with + * *out_len set, -ENODATA when cmd was not captured for the current backend + * generation, -ENOSPC when the snapshot exceeds dst_len, -EAGAIN when the + * backend is not READY. + */ +int gsp_phys_kgr_snap_get(u32 cmd, void *dst, u32 dst_len, u32 *out_len); + +/* * Run one fn1+fn65 probe on a READY backend to fill the static snapshot * cache. Takes gsp_phys_rm_lock inside (via gsp_phys_rm_probe), so it must * run on a work item, never inline under a caller-held rm lock or on the mdev @@ -344,7 +354,13 @@ void gsp_phys_backend_relay_counters(u64 *requested, u64 *ok, u64 *verify_fail, */ #define GSP_PHYS_GR_MAX_PAGES 16u #define GSP_PHYS_GR_MAX_INLINE 8u -#define GSP_PHYS_GR_INLINE_BYTES 64u +/* + * SESSION 153-CONTINUED-88: 256 -> 2048 (must stay equal to the sema + * service's SVC_GR_INLINE_BYTES and the bridge's GSP_BRIDGE_INLINE_BYTES - + * static_asserts enforce it). The GL window's real program uploads are + * 1536 B / 1408 B and must be carried whole to the engine-visible page. + */ +#define GSP_PHYS_GR_INLINE_BYTES 2048u #define GSP_PHYS_GR_QMD_WORDS 64u /* * IQ2M sentinel: written to the inline-QMD destination's word 63 after the @@ -358,7 +374,30 @@ void gsp_phys_backend_relay_counters(u64 *requested, u64 *ok, u64 *verify_fail, #define GSP_PHYS_GR_QMD_SENTINEL 0xdeadbeefu #define GSP_PHYS_GR_QMD_SENTINEL_WORD 63u #define GSP_PHYS_GR_QMD_WORD_NONE 0xffffffffu -#define GSP_PHYS_GR_RAW_WORDS 256u +/* + * SESSION 153-CONTINUED-61: dedicated backend verdict for the 3D-window + * MISS-ABORT (gsp_phys_rmcall_gr_miss_abort): the launch was NOT submitted + * because a window/attr operand's page is still unresolved. The sema + * service uses this code to keep the captured candidate armed (re-arm) + * instead of consuming it with a software completion, so the forward path + * retries the same chunk once the pin demand resolves the page. -ENOMSG + * is returned by no other path on this interface. + */ +#define GSP_PHYS_GR_RC_MISSABORT (-ENOMSG) +/* + * r929: grown 256 -> 512 dwords so a whole GL 3D section push (the observed + * X server one-shot is 1524 B = 381 dwords) survives in the captured raw + * tail; the raw window is the source of the 3D method-stream replay. + * SESSION 153-CONTINUED-45: 512 -> 2048 dwords (8 KB) for the DIRECT GL + * client's 5016-7144 B render pushes (glcore); the raw window must carry + * the whole push for the section capture's `len == push_n*4` gate. + * SESSION 154-1: 2048 -> 4096 dwords (16 KB) for the client's full-frame + * 12288-B pushes (r1483 `3D mark failed ... raw_ok=0 fit_ok=0 + * rawlen=3072 pushn=2048`), so the chunked window replay + * (svc_gl_stream_chunks) can run instead of falling back. Must equal + * SVC_GR_PUSH_WORDS (static_assert in gsp_sema_service.c). + */ +#define GSP_PHYS_GR_RAW_WORDS 4096u /* * B-2 (large-operand data path): the grown, boot-fixed GR data-bounce region. @@ -441,6 +480,48 @@ struct gsp_phys_gr_submit { u64 push_va; /* guest push identity (logging only) */ u32 push_len; u32 h_object; /* guest GR channel handle (logging only) */ + /* + * r929 (GL 3D path): the captured raw 3D method window - the + * SET_OBJECT C597 section from its class bind through the section + * close (state binds, LINE_LENGTH_IN + LAUNCH_DMA uploads, program/ + * tex binds; the prelude before the bind is excluded - its host-sem + * VAs belong to the client's VM and are unmapped in the replay + * channel, r929 v1 FAULT_PDE). When stream_n > 0 and + * gsp_phys_rmcall_gr_3d is on, the launch block replays the window + * verbatim after a SET_OBJECT TURING_A bind instead of the compute + * prelude + QMD; the engine then executes the REAL 3D methods. + * stream points into the caller's raw buffer; it is a separate + * pointer so a caller can pass any window. + */ + const u32 *stream; + u32 stream_n; /* dwords */ + /* Stream-relative dword offset of this window inside the client's + * 3D section (0 = the section's first chunk). Used to detect that an + * earlier chunk of the same push was dropped by a LOAD-ONLY carrier + * or the r1447 window hold: the remaining chunks must then elide + * their CALL runs (the lost chunk's state - e.g. the constant-buffer + * selector - is no longer on the channel, so a macro that emits + * LOAD_CONSTANT_BUFFER would class-error -> FECS priv-error poison). + */ + u32 stream_rel; + /* + * SESSION 153-CONTINUED-87: the first program VA the captured + * window binds (SET_PIPELINE_PROGRAM_ADDRESS pair; 0 when none). + * The launch builder uses its 4K page as the program family for + * the client's missing stage binds (sema_gl_sph_bind). + */ + u64 prog_va; + /* + * SESSION 153-CONTINUED-87: every program bind the captured window + * itself carries (VA + pipeline slot). The launch builder re-emits + * them AHEAD of the window: the window's own draws (BEGIN 0x1618) + * precede its binds (measured word 93 vs 184), and on the replay + * channel's fresh context the early draws would validate an unbound + * stage and latch the Shader Program Header fault. + */ + u32 n_prog_binds; + u64 prog_bind_va[8]; + u32 prog_bind_slot[8]; }; /* True when gsp_phys_gr_forward is set and the ladder is a live GR channel @@ -460,6 +541,15 @@ bool gsp_phys_gr_quiet_hold_active(void); int gsp_phys_backend_guest_gr_submit(const struct gsp_phys_gr_submit *req, struct gsp_phys_backend_copy_result *out); +/* + * r1445 5c: true when a 3D stream launch (h_object / win3d_n identify the + * window) would become a load-only MME carrier while the GR front-end is + * not idle; the service's request gate then busy-defers (nothing emitted + * or submitted, no release authored, candidate retried). Bounded per + * prelude generation internally; sema_gl_mme_fe_defer=0 makes it a no-op. + */ +bool gsp_phys_gr_mme_fe_defer_wanted(u32 h_object, u32 win3d_n); + /* GR forwarding counters (submitted/verified/cancelled), plain reads. */ void gsp_phys_backend_gr_counters(u64 *submitted, u64 *ok, u64 *refused, u64 *timed_out, u64 *pool_exhausted); diff --git a/kernel/gsp_phys_gpu.c b/kernel/gsp_phys_gpu.c index b61929e..f1d72f7 100644 --- a/kernel/gsp_phys_gpu.c +++ b/kernel/gsp_phys_gpu.c @@ -617,6 +617,25 @@ int gsp_phys_rmcall_gr_prelude_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_prelude_value); /* + * r597: emit the SEM_EXECUTE ACQUIRE groups found in the captured guest + * prelude ahead of the rebuilt inline-QMD window (default 0). The verbatim + * prelude replay breaks the launch (qmd_rel=0 after an 8.9 s stall, r599), + * so the acquire cannot be taken by replaying the prelude; this gate takes + * only the wait method groups and leaves the proven window in place. + */ +static int gsp_phys_rmcall_gr_acq_param; +module_param_named(gsp_phys_rmcall_gr_acq, gsp_phys_rmcall_gr_acq_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_rmcall_gr_acq, + "Emit the captured prelude's SEM_EXECUTE ACQUIRE groups ahead of the rebuilt GR window (default 0)"); + +int gsp_phys_rmcall_gr_acq_value(void) +{ + return gsp_phys_rmcall_gr_acq_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_acq_value); + +/* * QMD virtualization control gate (default 0 after m26 falsified the * hypothesis: on Turing the context init value is 0, native CUDA never * writes 0x29c, and the nonzero enable alone does not schedule). When set, @@ -734,20 +753,30 @@ int gsp_phys_rmcall_gr_qmdwin_value(void) EXPORT_SYMBOL_GPL(gsp_phys_rmcall_gr_qmdwin_value); /* - * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod; default 0). - * When set, a forwarded GR launch ignores the resolved guest release page and - * points QMD release0 at the module window-0 semaphore VA - * (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host VA. - * Discriminant for "engine completed but the shadow release write was lost" - * vs "engine never completed": with the module release, a completed QMD is - * always observable. Also makes the existing module-release fallback - * observable (it previously set no poll source, so it could never complete). - */ -static int gsp_phys_rmcall_gr_rel_mod_param; + * m64 forced module-semaphore release (gsp_phys_rmcall_gr_rel_mod). + * + * DEFAULT 1 since r453. When set, a forwarded GR launch ignores the resolved + * guest release page and points QMD release0 at the module window-0 semaphore + * VA (GSP_PHYS_RMCALL_VA_SEM), which the host polls at the SEM chunk's host + * VA; the GUEST-VISIBLE release word is authored by the host only on the + * completion edge - after the retire wait and after the bounce copy-out has + * shipped the engine's stores to the guest pages. Evidence (r453, the FNV + * change-detection fix on top): + * - with 0: the guest's release could be satisfied by the engine's own + * early write while the host copy-out was still pending, so a client's + * stream-ordered readback (streams2's c2, memcpy_roundtrip's memset + * phase, streams_events' chain) read PRE-launch content: 11/15 battery. + * - with 1: 13/15 battery (streams_events and memcpy_roundtrip flipped + * green), no stalls across the full battery, mail+saxpy clean. + * The r414 stall hole (a launch that never reached the hold edge left the + * guest word unauthored) is closed by the r420b per-outcome authoring path; + * one-line rollback: echo 0 > .../parameters/gsp_phys_rmcall_gr_rel_mod. + */ +static int gsp_phys_rmcall_gr_rel_mod_param = 1; module_param_named(gsp_phys_rmcall_gr_rel_mod, gsp_phys_rmcall_gr_rel_mod_param, int, 0644); MODULE_PARM_DESC(gsp_phys_rmcall_gr_rel_mod, - "Force the forwarded GR QMD release0 to the module semaphore (default 0; 1 = m64 engine-completion discriminant)"); + "Force the forwarded GR QMD release0 to the module semaphore; guest release authored only on the post-copy-out completion edge (default 1 since r453; 0 = historical engine-authored guest release)"); int gsp_phys_rmcall_gr_rel_mod_value(void) { @@ -1198,6 +1227,36 @@ int gsp_phys_gr_wedge_rearm_value(void) EXPORT_SYMBOL_GPL(gsp_phys_gr_wedge_rearm_value); /* + * r449 SHADER-CONSTANT-CACHE invalidate (gsp_phys_gr_ccache_inv; default 0). + * The forwarded QMD's cache-control block (QMDV02_03 word 5) carries the + * per-cache invalidate bits; the captured guest QMDs always set bits 26..29 + * (texture header/sampler/data + shader DATA) and the M91 fix adds bit 30 + * (INSTRUCTION), but bit 31 (SHADER CONSTANT - where a kernel's launch + * parameters live) is never set. The forwarded launch's param page is + * rewritten in place between launches (the two-bank win0 pool), so a stale + * constant-cache line can serve the PREVIOUS launch's parameters - the + * kernel then computes with the previous launch's pointers/sizes and the + * current launch's tracked output pages stay untouched ("no stores"). + * Mode: 0 = off, 1 = set bit 31 only, 2 = force the full 0xff000000 + * invalidate set. Read at submission time, so it can be A/B'd at runtime. + */ +static int gsp_phys_gr_ccache_inv_param; +module_param_named(gsp_phys_gr_ccache_inv, gsp_phys_gr_ccache_inv_param, + int, 0644); +MODULE_PARM_DESC(gsp_phys_gr_ccache_inv, + "r449: forwarded-QMD shader-constant-cache invalidate: 0=off, 1=bit31, 2=all invalidate bits (default 0)"); + +unsigned int gsp_phys_gr_ccache_inv_value(void) +{ + if (gsp_phys_gr_ccache_inv_param < 0) + return 0; + if (gsp_phys_gr_ccache_inv_param > 2) + return 2; + return (unsigned int)gsp_phys_gr_ccache_inv_param; +} +EXPORT_SYMBOL_GPL(gsp_phys_gr_ccache_inv_value); + +/* * r410 wedge quiet window (gsp_phys_gr_wedge_quiet_ms; default 40000, clamped * [0,120000]; 0 disables). The sticky per-lane no-execute wedge (100% of * launches no-write while kept busy) is proven to CLEAR when the lane stays @@ -1895,6 +1954,41 @@ bool gsp_phys_guest_ram_active(void) EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_active); /* + * Highest GPA any registered guest-RAM range covers. Conservative global + * bound (max across the legacy and every per-mdev record) for consumers that + * only need to reject translations pointing past RAM - e.g. the sema push + * read's PA ceiling (r1127: a stale VAS walk answered 0x23921d000 = 8.89 GB + * for the beef setup push; with the 8/10 GB guest layout the answer is + * inside the registered ranges and reads padding zeros, so no bound alone + * fixes it - the repeated-zero invalidation does - but any answer ABOVE + * this top can never read content). 0 when nothing is registered. + */ +u64 gsp_phys_guest_ram_top(void) +{ + u64 top = 0; + unsigned int i, j; + + mutex_lock(&guest_ram_lock); + for (i = 0; i < GSP_PHYS_GUEST_RAM_MDEVS + 1; i++) { + const struct gsp_phys_guest_ram_rec *rec = + (i == 0) ? &guest_ram_legacy : &guest_ram_mdevs[i - 1]; + + if (!rec->file) + continue; + for (j = 0; j < rec->ranges_n; j++) { + u64 end = rec->ranges[j].gpa_base + + rec->ranges[j].size; + + if (end > top) + top = end; + } + } + mutex_unlock(&guest_ram_lock); + return top; +} +EXPORT_SYMBOL_GPL(gsp_phys_guest_ram_top); + +/* * Lock-safe one-line snapshot for the gsp-phys-state dump: active flag, * path, range count and every range (gpa/offset/size). With only the legacy * record configured (the one-mdev campaign) the output is byte-identical to diff --git a/kernel/gsp_phys_gpu.h b/kernel/gsp_phys_gpu.h index 46d841d..d84b080 100644 --- a/kernel/gsp_phys_gpu.h +++ b/kernel/gsp_phys_gpu.h @@ -295,6 +295,7 @@ int gsp_phys_rmcall_gr_promote_value(void); * init+window+PCAS+sem shape); 1 replays the verbatim prelude. */ int gsp_phys_rmcall_gr_prelude_value(void); +int gsp_phys_rmcall_gr_acq_value(void); /* * gsp_phys_rmcall_gr_qmdvirt (default 1): emit @@ -339,6 +340,66 @@ int gsp_phys_rmcall_faultbuf_sys_value(void); * (proven small-PT line) instead of the window-1 scratch range (default 0). */ int gsp_phys_rmcall_gr_shadow_win0_value(void); +/* + * SESSION 153-CONTINUED-87: emit the client's missing TESS_CONTROL (j=2) + * and PIXEL (j=5) pipeline program binds ahead of replayed 3D windows + * (sema_gl_sph_bind, default 1). The sema service uses the same value to + * author the matching program images at those family VAs. + */ +bool gsp_phys_rmcall_gr_sph_bind_value(void); +u32 gsp_phys_rmcall_gr_sph_bind_mask_value(void); + +/* + * SESSION 155 CONT 11: bind a synthesized header-only GEOMETRY SPH to + * stage 4 with ENABLE=0 (sema_gl_sph_gs_img, default 0). The sema service + * uses this value to author the 128-byte image at family+0x780. + */ +bool gsp_phys_rmcall_gr_sph_gs_img_value(void); + +/* + * SESSION 155 CONT 12: attribute-map variant for the synthesized GS SPH + * (0=pass-through, 1=zero maps, 2=PS-exact, 3=IMAP POS+COLOR / OMAP + * POS.XYZW). The sema service reads this per capture. + */ +u32 gsp_phys_rmcall_gr_sph_gs_maps_value(void); + +/* + * SESSION 155 CONT 13: GS-mode bits for the synthesized GS SPH + * (1=FASTGS header + SET_GS_MODE/skip-mask, 2=enable the stage). + */ +u32 gsp_phys_rmcall_gr_sph_gs_mode_value(void); + +/* + * SESSION 155 CONT 14: raster-enable prelude (CT_WRITE + depth/stencil/ + * scissor disable) for replayed 3D windows. + */ +bool gsp_phys_rmcall_gr_raster_pre_value(void); + +/* + * SESSION 155 CONT 18: follow a migrated screen page in the copy-out + * (sema_gl_stale_write). + */ +bool gsp_phys_rmcall_gr_stale_write_value(void); + +/* + * SESSION 155 CONT 22: extra neutral raster state (blend/alpha/cull/point/ + * ZCULL) for replayed 3D windows. + */ +bool gsp_phys_rmcall_gr_raster_extra_value(void); + +/* + * SESSION 155 CONT 15: synthesized GS SPH placement/maps/context A/B. + */ +u32 gsp_phys_rmcall_gr_sph_gs_addr_value(void); +u32 gsp_phys_rmcall_gr_sph_gs_ctx_value(void); + +/* + * r1512: the viewport state the 3D replay injects for the current launch + * (sema_gl_vp_prelude mode + its literal words). Read-only; used by the + * sema service's draw-coverage receipt. + */ +void gsp_phys_gr_vp_state_get(int *mode, u32 *words, u32 *n); + /* m91: keep the forwarded QMD pointer pairs at the guest VAs (default 0). */ int gsp_phys_rmcall_gr_ptr_guest_value(void); @@ -403,6 +464,7 @@ int gsp_phys_rmcall_gr_cwd_slot_value(void); * attempt counter (default 3, clamped [0,4]). */ int gsp_phys_gr_nowrite_retries_value(void); int gsp_phys_gr_wedge_rearm_value(void); +unsigned int gsp_phys_gr_ccache_inv_value(void); /* STALLFREE: absolute per-launch GR submit wall-clock budget in ms, clamped * [10000,600000]; 0 = only the built-in per-ladder bounds (default 60000). @@ -514,6 +576,15 @@ bool gsp_phys_guest_ram_active(void); void gsp_phys_guest_ram_state(char *buf, size_t len); /* + * Highest GPA any registered guest-RAM range covers (max of gpa_base+size + * across every active record; 0 when nothing is registered). A GPA at or + * above this can never hold guest content - the backing file's padding + * reads as zeros - so translation consumers use it to reject walk/pin + * answers that point past the registered RAM. + */ +u64 gsp_phys_guest_ram_top(void); + +/* * Open-mdev helpers for the per-mdev debugfs selector and the single-guest * guards: count the mdevs whose VFIO device is open, or resolve one open * mdev by its device name (the mdev UUID). The returned device is not diff --git a/kernel/gsp_rm_emulation.c b/kernel/gsp_rm_emulation.c index cd9cbf2..0cef9e7 100644 --- a/kernel/gsp_rm_emulation.c +++ b/kernel/gsp_rm_emulation.c @@ -19,6 +19,7 @@ #include "gsp_sema_service.h" #include "gsp_phys_booter.h" #include "gsp_phys_gpu.h" +#include "gsp_display_engine.h" /* ============================================================ * Guest-memory access ("virtual DMA" on behalf of the device) @@ -511,11 +512,14 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, : GSPRM_EVENT_BURST_MAX; /* m171: expire the budget by wall time so a long-ago - * flood can never starve a fresh completion post. */ + * flood can never starve a fresh completion post. Only an + * admitted record moves the window: a refused attempt that + * retried (the display completion retry runs every 250 ms) + * must not keep the window perpetually fresh, or the budget + * would never expire and the retry would livelock. */ if (time_after(jiffies, rm->event_burst_at + GSPRM_EVENT_BURST_WINDOW)) rm->event_burst = 0; - rm->event_burst_at = jiffies; if (rm->event_burst >= cap) { rm->events_dropped++; if (rm->events_dropped <= 8 || @@ -528,6 +532,7 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, kfree(buf); return -EAGAIN; } + rm->event_burst_at = jiffies; rm->event_burst++; } else { rm->event_burst = 0; @@ -566,8 +571,22 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, u32 peer_read = 0; bool have_peer = false; int w; + int wait_iters = 50; + + /* + * r1615: critical event wake-ups (channel completions, and in + * particular the display/mode-set completion the X server's + * 2D sync waits for) must not be dropped just because the + * guest is briefly behind. Pump the consumption cursor for up + * to 200 ms so the guest's RM thread can drain; only drop if + * the ring is still full after that (the 0.5 s hint re-post + * retries). Non-critical event wake-ups keep the old + * drop-immediately policy; replies keep their 50 ms wait. + */ + if (function == GSPRM_RPC_EVENT_POST_EVENT) + wait_iters = critical ? 200 : 0; - for (w = 0; w < 51; w++) { + for (w = 0; w <= wait_iters; w++) { u32 used, space, rxoff = 0; u8 shdr[32]; @@ -617,9 +636,20 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, * two fn=0x1 replies dropped at boot -> the device * never comes up). */ - if (function == GSPRM_RPC_EVENT_POST_EVENT || w == 50) + if (w == wait_iters) break; mutex_unlock(&rm->stat_lock); + /* + * r1668: a full status ring means the guest's RM + * consumer is blocked, and the usual cause is the + * CeUtils scrub wait holding the RM lock the event + * callback needs. Authoring the scrubber's finish + * payload is the only way out, but the pump cannot + * reach its next pass while this post waits, so run + * the scrub step here (service thread only; no-op + * elsewhere). + */ + gsp_sema_service_scrub_pump(ctx->vgpu); msleep(1); mutex_lock(&rm->stat_lock); } @@ -707,11 +737,15 @@ static int gsp_rm_send_flags(struct gsp_emu_context *ctx, u32 function, return ret; } - /* Notify the guest (MSI-X vector 0). The driver's bottom-half drains - * events under the GPU lock; the rpcRecvPoll poller holds the same - * lock from send to poll, so a reply can't be consumed by the event - * drain while a poller waits for it (same as real GSP-RM). */ - gsp_emu_inject_completion_interrupt(ctx, rm->stat_seq); + /* Notify the guest. An event must also raise the GSP engine interrupt + * (falcon SWGEN0 + CPU_INTR leaf/top) so the guest's ISR drains the + * status queue and nv_post_event() wakes a client blocked in poll() on + * /dev/nvidia0 (e.g. a GLX swap). Replies only need the MSI-X ping + * because the client polls the queue for them. */ + if (function == GSPRM_RPC_EVENT_POST_EVENT) + gsp_emu_raise_gsp_event_irq(ctx); + else + gsp_emu_inject_completion_interrupt(ctx, rm->stat_seq); mutex_unlock(&rm->stat_lock); return 0; @@ -726,6 +760,117 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, } /* + * r576 DIAG: event-post payload witness (gsp_rm_ev_log=N budget per boot). + * Every EVENT_POST_EVENT record actually sent is logged with its full + * payload and send result: when a client parks waiting for a completion + * wake-up that never arrives, this shows whether the post was generated, + * with which hClient/hEvent, and whether the status-queue write accepted it + * (rc=0) or dropped it (burst cap / ring full => rc<0). The companion + * sentinel fires only when post_all_events found valid events but could not + * post any of them while none were dedup-skipped - the "client waits, host + * thinks it posted" anomaly. 0 (default) = off. + */ +unsigned gsp_rm_ev_log_max; +static unsigned gsp_rm_ev_log; + +static void gsp_rm_evlog_post(struct nv_vgpu_device *vgpu, + const struct gsp_rm_emu *rm, u32 idx, + const char *src, int rc) +{ + if (!gsp_rm_ev_log_max || gsp_rm_ev_log >= gsp_rm_ev_log_max) + return; + gsp_rm_ev_log++; + nv_vgpu_info(vgpu, + "vgsp-rm: EVPOST %s ev[%u] cl=0x%x ev=0x%x nidx=0x%x data=0x%x rc=%d", + src, idx, rm->events[idx].h_client, + rm->events[idx].h_event, rm->events[idx].notify_index, + rm->events[idx].data, rc); +} + +/* + * r577 experiment knobs (runtime-writable; default 0 = legacy behaviour). + * The r576 live experiment showed the guest consumes our POST_EVENT records + * (rmstate cursors match) yet never enters nv_post_event (clean single-probe + * window, forced rmpost rounds -> entries-written 0): the guest RM drops + * them somewhere between the status-queue read and osNotifyEvent. These + * knobs let one boot A/B the record contents without rebuilding: + * gsp_rm_ev_seq - when nonzero, event records carry the status-queue + * sequence as the RPC-level sequence (legacy: 0) + * gsp_rm_ev_nlist - when nonzero, bNotifyList (payload byte 28) is set, + * selecting the driver's notifier-list dispatch path + * gsp_rm_ev_cl - when nonzero, overrides the payload hClient (e.g. to + * the kernel RM client) for the CliGetEventInfo lookup + */ +unsigned gsp_rm_ev_seq; +unsigned gsp_rm_ev_nlist; +unsigned gsp_rm_ev_cl; +/* + * r578: reachability probe for the guest-RM event drop. When nonzero, event + * records carry this RPC function instead of POST_EVENT (0x1003). Posting a + * function with an unmistakable guest-side side effect (e.g. + * NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT 0x100C with an unsupported ucodeEngDesc + * trips NV_ASSERT_FAILED in _kgspRpcUcodeLibosPrint, or + * NV_VGPU_MSG_EVENT_OS_ERROR_LOG 0x1006 logs an errString) proves whether + * the guest ever DISPATCHES our status-queue event records: visible side + * effect => the dispatch works and POST_EVENT is dropped inside + * _kgspRpcPostEvent; no side effect => the records are consumed without + * dispatch. + */ +unsigned gsp_rm_ev_func; +/* + * r1767: gate the r1765 flip-event association (hSrcResource == 0 -> + * alloc-header hParent). Default 0 keeps the pre-r1766 event semantics; + * the RCA exonerated the association from the r1766 soft-locks but not + * from the r1766 ISR crash class, so it stays off until validated + * end-to-end. + */ +static bool gsp_rm_ev_assoc; /* default 0: pre-r1766 event semantics */ +module_param_named(ev_assoc, gsp_rm_ev_assoc, bool, 0644); +MODULE_PARM_DESC(ev_assoc, + "associate hSrcResource==0 events with their alloc parent (r1765 flip-event fix; default 0)"); + +static void gsp_rm_fill_ev_payload(const struct gsp_rm_emu *rm, u32 i, + u8 *payload) +{ + u32 h_client = rm->events[i].h_client; + /* + * r579 THE EVENT-DROP ROOT CAUSE: NV0005_ALLOC_PARAMETERS.notifyIndex + * is OVERLOADED (cl0005.h): INDEX=bits[15:0], SUBDEVICE=bits[23:16], + * FLAGS=bits[31:24]. The registration word (e.g. 0x3c00000c) must not + * be posted verbatim: the guest's _kgspRpcPostEvent hands the payload + * notifyIndex straight to gpuNotifySubDeviceEvent_IMPL, which indexes + * its notifier array by it - posting 0x3c00000c produced a wild + * pointer (base + 0x3c00000c*4 + 0x1508) and a kernel Oops + * ("BUG: unable to handle page fault ... gpuNotifySubDeviceEvent_IMPL + * +0x21f", RAX=0x3c00000c) on every dispatched event, which is why + * nv_post_event never fired and clients waiting on event wakes + * wedged. Post the INDEX field only. + */ + u32 notify_index = rm->events[i].notify_index & 0xffffu; + + memset(payload, 0, 32); + if (gsp_rm_ev_cl) + h_client = gsp_rm_ev_cl; + memcpy(payload + 0, &h_client, 4); + memcpy(payload + 4, &rm->events[i].h_event, 4); + memcpy(payload + 8, ¬ify_index, 4); + memcpy(payload + 12, &rm->events[i].data, 4); + /* info16@16, status@20, eventDataSize@24 = 0 */ + if (gsp_rm_ev_nlist) + payload[28] = 1; /* bNotifyList */ +} + +static u32 gsp_rm_ev_sequence(const struct gsp_rm_emu *rm) +{ + return gsp_rm_ev_seq ? rm->stat_seq : 0; +} + +static u32 gsp_rm_ev_function(void) +{ + return gsp_rm_ev_func ? gsp_rm_ev_func : GSPRM_RPC_EVENT_POST_EVENT; +} + +/* * Post the GPU event that tells the guest's CPU-side RM a channel's * submitted work is complete. A real GPU raises an interrupt, the host GSP * turns it into an NV_VGPU_MSG_EVENT_POST_EVENT record for the client's @@ -735,7 +880,8 @@ static int gsp_rm_send(struct gsp_emu_context *ctx, u32 function, * Safe to call from the progress service's thread: the status queue write is * serialized by rm->stat_lock and the guest is woken with MSI-X vector 0. */ -int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) +static int gsp_rm_post_channel_event_common(struct nv_vgpu_device *vgpu, + u32 h_src, bool force, bool strict) { struct gsp_emu_context *ctx = vgpu ? vgpu->gsp_comm_ctx : NULL; struct gsp_rm_emu *rm; @@ -757,20 +903,50 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) if (!rm->events[i].valid || rm->events[i].h_src != h_src) continue; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[i].h_client, 4); - memcpy(payload + 4, &rm->events[i].h_event, 4); - memcpy(payload + 8, &rm->events[i].notify_index, 4); - memcpy(payload + 12, &rm->events[i].data, 4); - /* info16@16, status@20, eventDataSize@24, bNotifyList@28 = 0 */ - if (gsp_rm_send_flags(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload), true)) + /* + * r1780: the directly-associated post is the flood path. The + * X channel's completion edge fires per consumed submission and + * the 48/s burst cap then drops the wake-ups a blocked client is + * waiting for (live: "event burst capped (48/48), dropped + * total=3000", DDX "Failed to idle DMA"). A re-post within + * GSPRM_EVENT_HINT_DEDUP adds no wake-up - the client re-reads + * channel state on any event and the 0.5 s hint re-posts late + * waiters - so apply the dedup the fallback loop already uses. + */ + if (!force && + time_before(jiffies, + rm->events[i].last_post + + GSPRM_EVENT_HINT_DEDUP)) continue; + + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send_flags(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, + payload, sizeof(payload), + true); + + gsp_rm_evlog_post(vgpu, rm, i, "chan", rc); + if (rc) + continue; + } rm->events[i].last_post = jiffies; posted++; } /* + * r1765: strict mode is the display engine's 1:1 contract - a + * channel-scoped completion posts ONLY the events registered on the + * channel (a real RM fires the channel's own event list on AWAKEN). + * Never fall through to the all-events shotgun for such a post: the + * shotgun coalesces flip completions and lets unrelated completions + * nudge the flip event (stray FLIP_OCCURRED -> the nvidia-drm + * dequeue WARN). The caller's retry pump re-posts a refused edge. + */ + if (!posted && strict) + return -ENOENT; + + /* * Fallback when the event's source object could not be associated with * the channel: nudge every registered event. The client's dispatcher * re-reads each channel's state on any event, so a spurious wake-up is @@ -780,15 +956,45 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) for (i = 0; i < rm->events_n; i++) { if (!rm->events[i].valid) continue; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[i].h_client, 4); - memcpy(payload + 4, &rm->events[i].h_event, 4); - memcpy(payload + 8, &rm->events[i].notify_index, 4); - memcpy(payload + 12, &rm->events[i].data, 4); - if (gsp_rm_send_flags(ctx, GSPRM_RPC_EVENT_POST_EVENT, - 0, 0, payload, - sizeof(payload), true)) + /* + * r1765: display-channel events are driven 1:1 by + * the display engine and must not be nudged by + * unrelated completions. The predicate answers + * false in the legacy default mode (gated on + * disp_events_strict), so this is inert unless the + * strict path is enabled. + */ + if (gsp_disp_engine_owns_object(vgpu, + rm->events[i].h_src)) + continue; + /* + * r580: apply the hint dedup to the per-completion + * shotgun. Every completion posts ALL registered + * events (the ABI gives no source association), so a + * completion burst burns the burst budget 8 records + * per completion and real wake-ups get dropped + * (r579 boot: events_dropped=724). A redundant + * re-post within 250 ms adds no wake-up - the + * client's dispatcher re-reads channel state on any + * event - and the 0.5 s hint re-posts late waiters. + */ + if (!force && + time_before(jiffies, rm->events[i].last_post + + GSPRM_EVENT_HINT_DEDUP)) continue; + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send_flags(ctx, + gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), + 0, payload, + sizeof(payload), + true); + + gsp_rm_evlog_post(vgpu, rm, i, "chanfb", rc); + if (rc) + continue; + } rm->events[i].last_post = jiffies; posted++; } @@ -805,6 +1011,86 @@ int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) return posted ? 0 : -ENOENT; } +int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src) +{ + return gsp_rm_post_channel_event_common(vgpu, h_src, false, false); +} + +/* + * r1765 strict variants (display engine, disp_events_strict=1): exact-match + * only - no all-events fallback; -ENOENT rather than a shotgun when the + * channel has no registered event (yet). + */ +int gsp_rm_post_channel_event_strict(struct nv_vgpu_device *vgpu, u32 h_src) +{ + return gsp_rm_post_channel_event_common(vgpu, h_src, false, true); +} + +/* + * Force a completion wake-up past the hint dedup. The 250 ms dedup assumes a + * recent post already woke the client, but a waiter that arrived after that + * post misses it, and a busy event stream keeps refreshing every event's + * stamp so the coalesced completion is never delivered (r1649: the X server's + * display UPDATE completion was refused - the coincident hint returned + * -EAGAIN because every event was inside the window - and the flip wait spun + * in the nvidia ioctl forever). Callers use this only for a refused + * completion edge, rate-limited by their own retry window; the event burst + * cap and the status-ring free-space discipline still apply inside + * gsp_rm_send_flags(). + */ +int gsp_rm_post_channel_event_force(struct nv_vgpu_device *vgpu, u32 h_src) +{ + return gsp_rm_post_channel_event_common(vgpu, h_src, true, false); +} + +int gsp_rm_post_channel_event_force_strict(struct nv_vgpu_device *vgpu, + u32 h_src) +{ + return gsp_rm_post_channel_event_common(vgpu, h_src, true, true); +} + +/* + * r623 experiment: deliver a synthetic OS_ERROR_LOG (0x1006) record to the + * guest's RM queue - the error report a real GSP sends into the client's + * message queue when an engine faults. r618's synth-fault writes the channel + * notifier and posts a generic event; after it, the NEXT client stuck + * pre-submission (r620 atomics; r616b saxpy/math). If the guest's fault + * handling marks the device recoverable only when the 0x1006 report arrives, + * this record is the missing half. Payload mirrors the GSP ABI decode in + * gsp_phys_booter.c (GSP_PHYS_OSERR_OFF_*): exceptType u32, runlistId u32, + * chid u32, errString[256], prevXid u32. Manual/debug only. + */ +int gsp_rm_send_os_error_log(struct nv_vgpu_device *vgpu, u32 except_type, + u32 runlist_id, u32 chid, const char *engine) +{ + struct gsp_emu_context *ctx = vgpu ? vgpu->gsp_comm_ctx : NULL; + struct gsp_rm_emu *rm; + u8 payload[272] = {0}; + int rc; + + if (!ctx) + return -ENODEV; + rm = ctx->rm_emu; + if (!rm || !rm->parsed) + return -EAGAIN; + + memcpy(payload + 0, &except_type, 4); + memcpy(payload + 4, &runlist_id, 4); + memcpy(payload + 8, &chid, 4); + snprintf((char *)payload + 12, 255, "ENGINE %s", + engine && *engine ? engine : "GR"); + /* prevXid stays 0 */ + + rc = gsp_rm_send(ctx, 0x1006, (u32)rm->stat_seq + 1, 0, payload, + sizeof(payload)); + if (!rc) + nv_vgpu_info(vgpu, + "vgsp-rm: OS_ERROR_LOG sent (synthetic) except=0x%x rl=%u chid=%u engine=%s\n", + except_type, runlist_id, chid, + engine && *engine ? engine : "GR"); + return rc; +} + /* * Debug: post the event record for one registered event (or all of them with * idx 0xffffffff). Used to test the guest's event path on demand without @@ -819,14 +1105,10 @@ static int gsp_rm_post_event_idx(struct gsp_emu_context *ctx, u32 idx) if (idx >= rm->events_n || !rm->events[idx].valid) return -EINVAL; - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[idx].h_client, 4); - memcpy(payload + 4, &rm->events[idx].h_event, 4); - memcpy(payload + 8, &rm->events[idx].notify_index, 4); - memcpy(payload + 12, &rm->events[idx].data, 4); + gsp_rm_fill_ev_payload(rm, idx, payload); - rc = gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload)); + rc = gsp_rm_send(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, payload, sizeof(payload)); if (!rc) rm->events[idx].last_post = jiffies; return rc; @@ -898,24 +1180,48 @@ int gsp_rm_post_all_events(struct nv_vgpu_device *vgpu) for (i = 0; i < rm->events_n; i++) { if (!rm->events[i].valid) continue; + /* + * r1765: the hint must not nudge display-channel events - + * they are driven 1:1 by the display engine's UPDATE + * completions. False in the legacy default mode (the + * predicate is gated on disp_events_strict), so the hint is + * unchanged unless the strict path is enabled. + */ + if (gsp_disp_engine_owns_object(vgpu, rm->events[i].h_src)) + continue; if (time_before(jiffies, rm->events[i].last_post + GSPRM_EVENT_HINT_DEDUP)) { skipped++; continue; } - memset(payload, 0, sizeof(payload)); - memcpy(payload + 0, &rm->events[i].h_client, 4); - memcpy(payload + 4, &rm->events[i].h_event, 4); - memcpy(payload + 8, &rm->events[i].notify_index, 4); - memcpy(payload + 12, &rm->events[i].data, 4); - if (gsp_rm_send(ctx, GSPRM_RPC_EVENT_POST_EVENT, 0, 0, - payload, sizeof(payload))) - continue; + gsp_rm_fill_ev_payload(rm, i, payload); + { + int rc = gsp_rm_send(ctx, gsp_rm_ev_function(), + gsp_rm_ev_sequence(rm), 0, + payload, sizeof(payload)); + + gsp_rm_evlog_post(vgpu, rm, i, "all", rc); + if (rc) + continue; + } rm->events[i].last_post = jiffies; n++; } + /* + * r576 DIAG: the anomaly sentinel - valid events existed, none were + * dedup-skipped, yet not one post could be sent. Every send rc is in + * the EVPOST lines above (budget permitting). + */ + if (!n && !skipped && gsp_rm_ev_log_max && + gsp_rm_ev_log < gsp_rm_ev_log_max) { + gsp_rm_ev_log++; + nv_vgpu_info(vgpu, + "vgsp-rm: EVPOST all: no postable event (events_n=%u dropped=%llu)", + rm->events_n, rm->events_dropped); + } + if (n) return 0; /* -EAGAIN: valid events existed but every one was dedup-skipped @@ -1169,6 +1475,7 @@ static const char *gsp_rm_rpc_name(u32 function) case GSPRM_RPC_FUNC_GSP_RM_CONTROL: return "GSP_RM_CONTROL"; case GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER: return "UNLOADING_GUEST_DRIVER"; case GSPRM_RPC_FUNC_UPDATE_BAR_PDE: return "UPDATE_BAR_PDE"; + case GSPRM_RPC_FUNC_DUP_OBJECT: return "DUP_OBJECT"; default: return "UNKNOWN"; } } @@ -1739,7 +2046,20 @@ static void gsp_rm_handle_bus_get_pci_bar_info(struct gsp_emu_context *ctx, * 36 bytes (observed paramsSize): feHwSysCap(0) windowPresentMask(4) * bFbRemapperEnabled(8) numHeads(12) i2cPort(16) internalDispActiveMask(20) * embeddedDisplayPortMask(24) muxSupported(28) numDispChannels(32). - * Report a minimal headless display: 1 head, 1 (core) channel. + * + * r651: this reply is also the gate for the WINDOW-channel alloc path. + * Live gdb on the guest nvidia.ko pinned the display path 0x56 to + * kdispGetChannelNum_v03_00: for dispChnClass_Win it requires + * pStaticInfo->windowPresentMask & NVBIT32(instance) and returns + * NV_ERR_NOT_SUPPORTED (0x56) when the bit is clear; the Curs class reads + * HEAD_EXISTS bits from feHwSysCap the same way (0x49+instance channel num). + * The previous reply left both masks zero, so nvRmApiAlloc(0xc57e) died in + * the guest's local resource construct (clientAllocResource -> + * resservResourceFactory -> dispchnConstruct -> kdispGetChannelNum) before + * any GSP-RM RPC was issued. Report the same one-head / 8-window model the + * NV0073 shim and the caps notifier expose (feHwSysCap head-0 exists, + * windowPresentMask bits 0..7, numDispChannels covering the cursor channel + * numbers 73..80, i.e. 81 entries). */ static void gsp_rm_handle_display_get_static_info(struct gsp_emu_context *ctx, u32 sequence, @@ -1751,15 +2071,205 @@ static void gsp_rm_handle_display_get_static_info(struct gsp_emu_context *ctx, memset(out_params, 0, 36); + /* + * feHwSysCap: NV_PDISP_FE_HW_SYS_CAP_HEAD_EXISTS(i) occupies bit i + * (dev_disp.h v03_00); head 0 exists on the emulated board. + */ + val32 = 0x1; + memcpy(out_params + 0, &val32, 4); + + /* + * windowPresentMask: bit i = window instance i present. Mirrors the + * caps notifier's SYS_CAPB=0xff and the NV0073 window assignment + * (windows 0/1 flexible) - eight usable windows. + */ + val32 = 0xff; + memcpy(out_params + 4, &val32, 4); + val32 = 1; /* numHeads */ memcpy(out_params + 12, &val32, 4); - val32 = 1; /* numDispChannels (core channel) */ + + /* + * numDispChannels sizes pClientChannelTable[], indexed by the packed + * channel number kdispGetChannelNum returns: CORE=0, WIN(i)=1+i, + * WINIM(i)=33+i, CURS(i)=73+i. Cover CURS(7)=80 -> 81 entries so any + * instance the guest allocates stays in bounds. + */ + val32 = 81; memcpy(out_params + 32, &val32, 4); *out_params_size = 36; *out_status = 0; - nv_vgpu_info(ctx->vgpu, "vgsp-rm: DISPLAY_GET_STATIC_INFO -> heads=1 dispChannels=1"); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY_GET_STATIC_INFO -> heads=1 dispChannels=81 winMask=0xff feHwSysCap=0x1"); +} + +/* + * INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES (0x20800a5d) + * + * dispswConstruct_IMPL (disp_sw.c) asks for {displayMask, numHeads} and + * rejects every logicalHeadId >= numHeads with "invalid logical head + * number: N" + NV_ERR_INSUFFICIENT_RESOURCES; the generic all-zero reply + * made numHeads = 0, so nvkms' DISPLAY_SW objects could not be constructed + * on any head (observed r653: heads 0..3 all rejected -> vkCreateDevice -3). + * Report the same one-head model as DISPLAY_GET_STATIC_INFO: display id 0 + * active, one head. + * + * Params ABI (NV2080_CTRL_INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES_PARAMS, + * 8 bytes): displayMask(0) numHeads(4). + */ +static void gsp_rm_handle_display_active_devices(struct gsp_emu_context *ctx, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 displayMask = 0x1; /* display id 0 */ + u32 numHeads = 1; + + memcpy(out_params + 0, &displayMask, 4); + memcpy(out_params + 4, &numHeads, 4); + + *out_params_size = 8; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY_GET_ACTIVE_DISPLAY_DEVICES -> displayMask=0x1 numHeads=1"); +} + +/* + * NVC370_CTRL_CMD_GET_CHANNEL_INFO (0xc3700104) + * + * nvRMSyncEvoChannel() (nvkms-rm.c) polls this control through the display + * object after every EVO channel kickoff and waits up to 2 s for the channel + * to report NVC370_CTRL_GET_CHANNEL_INFO_STATE_IDLE, otherwise it logs + * "Idling display engine timed out: <class>:<instance>:<sd>:<token>" and + * fails the whole modeset init (observed on the r652 window path: 0xc57d and + * 0xc57e instances 0/1). The emulated display engine has no pushbuffer + * consumer, so the state would never leave BUSY; report IDLE immediately. + * + * Params ABI (NVC370_CTRL_CMD_GET_CHANNEL_INFO_PARAMS, 20 bytes): + * subdeviceIndex(0) channelClass(4) channelInstance(8) + * IsChannelInDebugMode(12) channelState(16 = NVBIT(0) IDLE). + * Echo the request so class/instance round-trip and set the state word. + */ +static void gsp_rm_handle_display_channel_info(struct gsp_emu_context *ctx, + const u8 *in_params, u32 in_size, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 idle = 1; /* NVC370_CTRL_GET_CHANNEL_INFO_STATE_IDLE */ + u32 class_id = 0, instance = 0; + u32 size = in_size; + + if (size > 64) + size = 64; /* defensive cap; the real params are 20 bytes */ + if (size >= 20) { + memcpy(out_params, in_params, size); + memcpy(&class_id, in_params + 4, 4); + memcpy(&instance, in_params + 8, 4); + memcpy(out_params + 16, &idle, 4); + } else { + memset(out_params, 0, 20); + memcpy(out_params + 16, &idle, 4); + size = 20; + } + + *out_params_size = size; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: GET_CHANNEL_INFO class=0x%x instance=%u -> IDLE", + class_id, instance); +} + +/* + * NV0080_CTRL_CMD_FIFO_GET_ENGINE_CONTEXT_PROPERTIES (0x801707) + * + * The Vulkan/GL client queries the device for the size and alignment of an + * engine's context save region while building its GR channel context + * (observed: engineId 0x8 = GRAPHICS_ZCULL, one query at the end of every + * vkCreateDevice attempt). The generic all-zero reply handed back size = 0; + * the client's context-buffer allocation then failed (memdesc of size 0) + * and device creation aborted with VK_ERROR_INITIALIZATION_FAILED + * (observed r661: four create attempts, each ending at this control, then + * teardown and VKFAIL -3). + * + * The sizes below mirror the real 570.144 GSP-RM geometry captured on this + * TU104 (rm-call golden runs); entries marked "fill" were not covered by + * the capture and use nonzero page-aligned placeholders - revisit them when + * a real capture for the remaining engine ids exists. + * + * Params ABI (NV0080_CTRL_FIFO_GET_ENGINE_CONTEXT_PROPERTIES_PARAMS, + * 12 bytes): engineId(0, in) alignment(4, out) size(8, out). + */ +static const struct gsp_rm_engine_ctx_geom { + u32 engine_id; + u32 alignment; + u32 size; +} gsp_rm_engine_ctx_geoms[] = { + { 0x00, 0x1000, 0xf2d00 }, /* GRAPHICS (captured) */ + { 0x08, 0x1000, 0x10000 }, /* GRAPHICS_ZCULL (fill) */ + { 0x09, 0x1000, 0xd100 }, /* GRAPHICS_PM (captured) */ + { 0x0a, 0x1000, 0x1000 }, /* COMPUTE_PREEMPT (fill) */ + { 0x0b, 0x1000, 0x1000 }, /* GRAPHICS_PREEMPT (fill) */ + { 0x0c, 0x1000, 0x1000 }, /* GRAPHICS_SPILL (fill) */ + { 0x0d, 0x100, 0x20000 }, /* GRAPHICS_PAGEPOOL (fill) */ + { 0x0e, 0x100, 0x8000 }, /* GRAPHICS_BETACB (fill) */ + { 0x0f, 0x100, 0x80000 }, /* GRAPHICS_RTV (fill) */ + { 0x10, 0x1000, 0x5000 }, /* GRAPHICS_PATCH (captured) */ + { 0x11, 0x100, 0x3000 }, /* GRAPHICS_BUNDLE_CB (captured) */ + { 0x12, 0x100, 0x20000 }, /* GRAPHICS_PAGEPOOL_GLOBAL (captured) */ + { 0x13, 0x1000, 0x7d7400 }, /* GRAPHICS_ATTRIBUTE_CB (captured) */ + { 0x14, 0x100, 0x80000 }, /* GRAPHICS_RTV_CB_GLOBAL (captured) */ + { 0x15, 0x100, 0x20000 }, /* GRAPHICS_GFXP_POOL (fill) */ + { 0x16, 0x100, 0x1000 }, /* GRAPHICS_GFXP_CTRL_BLK (fill) */ + { 0x17, 0x1000, 0x10000 }, /* GRAPHICS_FECS_EVENT (captured) */ + { 0x18, 0x1000, 0x80000 }, /* GRAPHICS_PRIV_ACCESS_MAP (captured) */ + { 0x19, 0x100, 0x1000 }, /* GRAPHICS_SETUP (fill) */ +}; + +static void gsp_rm_handle_fifo_engine_ctx_props(struct gsp_emu_context *ctx, + const u8 *in_params, u32 in_size, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 engine_id = ~0u; + u32 alignment = 0x1000, size = 0x1000; + u32 i; + bool found = false; + + if (in_size >= 4) + memcpy(&engine_id, in_params + 0, 4); + + for (i = 0; i < ARRAY_SIZE(gsp_rm_engine_ctx_geoms); i++) { + if (gsp_rm_engine_ctx_geoms[i].engine_id == engine_id) { + alignment = gsp_rm_engine_ctx_geoms[i].alignment; + size = gsp_rm_engine_ctx_geoms[i].size; + found = true; + break; + } + } + + if (engine_id > 0x19) { + /* Same rejection the real RM's VF handler uses for ids + * outside the engine context properties range. */ + *out_status = 0x56; /* NV_ERR_NOT_SUPPORTED */ + *out_params_size = 0; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: FIFO_GET_ENGINE_CONTEXT_PROPERTIES engineId=0x%x -> NOT_SUPPORTED", + engine_id); + return; + } + + memcpy(out_params + 0, &engine_id, 4); + memcpy(out_params + 4, &alignment, 4); + memcpy(out_params + 8, &size, 4); + *out_params_size = 12; + *out_status = 0; + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: FIFO_GET_ENGINE_CONTEXT_PROPERTIES engineId=0x%x -> alignment=0x%x size=0x%x%s", + engine_id, alignment, size, found ? "" : " (default)"); } /* @@ -1785,6 +2295,231 @@ static void gsp_rm_handle_display_get_ip_version(struct gsp_emu_context *ctx, } /* + * NV0073 display-common heads + one static HDMI connector + * (0x730102 / 0x730287 / 0x7302ad / 0x730107 / 0x73028b / 0x730250 / + * 0x731140) + * + * nvidia-modeset's nvkms-rm.c ProbeHeadCountAndWindowAssignment asks the + * display-common object for the head count (NV0073_CTRL_CMD_SYSTEM_ + * GET_NUM_HEADS = 0x730102) and a zero reply makes it log "No heads found on + * board!" and fail -- which kills Xorg's ScreenInit and every Vulkan + * vkCreateDevice (measured: the emulated device replies all-zero through the + * generic path, so nvkms never opens). The kernel driver already sees 1 head + * (DISPLAY_GET_STATIC_INFO reports numHeads=1); report the same 1 head here, + * a single-head mask for GET_ALL_HEAD_MASK (0x730287), and a fully-flexible + * window assignment for windows 0/1 (0xFF = FLEXIBLE; windows above the head + * count stay 0 = not present). + * + * The next wall after the heads (measured r643): nvRmAllocDisplays fails on + * NoValidDisplays() - nvkms requires at least one valid display. Report one + * static TMDS/HDMI connector on display id 0: GET_SUPPORTED (0x730107) + * displayMask=1|DDC=1, SPECIFIC_OR_GET_INFO (0x73028b) SOR / + * SOR_SINGLE_TMDS_A (TMDS avoids the whole DP-AUX lib), SPECIFIC_ + * GET_CONNECTOR_DATA (0x730250) one HDMI_A entry, DFP_GET_INFO (0x731140) + * TMDS flags. + * + * r1617: the connector was still reported disconnected. NVKMS + * nvRmGetConnectedDpys() polls SYSTEM_GET_CONNECT_STATE (0x730108) and a zero + * displayMask makes it see no displays; the driver then sets the NoScanout + * NULL mode and the NVKMS deferred display close bumps the shared nvClip + * modeNum past the GL client's mode-switch test (which requires bit0 clear). + * Report display 0 connected, its type as DFP (0x730240) and a 600 MHz pclk + * limit (0x73028a); the SOR caps seed already gives 600 MHz TMDS. The EDID + * itself is served by gsp_rm_handle_display_get_edid() (0x730245). + */ +static void gsp_rm_handle_display_heads(struct gsp_emu_context *ctx, u32 cmd, + u32 req_size, const u8 *req_params, + u32 *out_status, u8 *out_params, + u32 *out_params_size) +{ + u32 one = 1; + u32 sz = req_size ? req_size : 72; + u32 v; + u32 req_w0 = 0, req_w1 = 0, req_w2 = 0; + + memset(out_params, 0, sz); + if (req_params) { + if (req_size >= 4) + memcpy(&req_w0, req_params, 4); + if (req_size >= 8) + memcpy(&req_w1, req_params + 4, 4); + if (req_size >= 12) + memcpy(&req_w2, req_params + 8, 4); + } + switch (cmd) { + case 0x730102: /* SYSTEM_GET_NUM_HEADS: {subDev, flags, numHeads} */ + if (sz >= 12) + memcpy(out_params + 8, &one, 4); + break; + case 0x730287: /* SPECIFIC_GET_ALL_HEAD_MASK: {subDev, headMask} */ + if (sz >= 8) + memcpy(out_params + 4, &one, 4); + break; + case 0x7302ad: /* SPECIFIC_GET_VALID_HEAD_WINDOW_ASSIGNMENT: + * {subDev, windowHeadMask[32]} - windows 0 and 1 + * flexible (0xFF), the rest absent (0). */ + if (sz >= 36) { + out_params[4] = 0xff; + out_params[5] = 0xff; + } + break; + case 0x730107: /* SYSTEM_GET_SUPPORTED: {subDev, displayMask, + * displayMaskDDC} - display id 0 supported + DDC. */ + if (sz >= 12) { + memcpy(out_params + 4, &one, 4); + memcpy(out_params + 8, &one, 4); + } + break; + case 0x730108: /* SYSTEM_GET_CONNECT_STATE: {subDev, flags, + * displayMask, retryTimeMs} - report the requested + * display 0 connected (NVKMS polls this ~126x while + * opening the display; zero made it set NoScanout). */ + if (sz >= 16) { + u32 mask = 0; + + if (req_params) + memcpy(&mask, req_params + 8, 4); + mask &= 1; + if (!mask) + mask = 1; + memcpy(out_params + 8, &mask, 4); + } + break; + case 0x730240: /* SPECIFIC_GET_TYPE: {subDev, displayId, + * displayType} - display 0 is our static DFP. */ + if (sz >= 12) { + v = 2; /* NV0073_CTRL_SPECIFIC_DISPLAY_TYPE_DFP */ + memcpy(out_params + 8, &v, 4); + } + break; + case 0x73028a: /* SPECIFIC_GET_PCLK_LIMIT: {subDev, displayId, + * pclkLimit, orPclkLimit, vbPclkLimit} in kHz. The + * TMDS SOR caps (600 MHz) are the real limit; a sane + * RM value also keeps non-TMDS validation working. */ + if (sz >= 20) { + v = 600000; + memcpy(out_params + 8, &v, 4); + memcpy(out_params + 12, &v, 4); + memcpy(out_params + 16, &v, 4); + } + break; + case 0x73028b: /* SPECIFIC_OR_GET_INFO: {subDev, displayId, index, + * type, protocol, ditherType, ditherAlgo, location, + * rootPortId, dcbIndex, vbiosAddress, bIsLitByVbios, + * bIsDispDynamic} */ + if (sz >= 52) { + v = 2; /* NV0073_CTRL_SPECIFIC_OR_TYPE_SOR */ + memcpy(out_params + 12, &v, 4); + v = 1; /* SOR_SINGLE_TMDS_A (no DP aux) */ + memcpy(out_params + 16, &v, 4); + /* location=0 (chip), remaining out-fields stay 0: + * not lit by vbios, not dynamic. */ + } + break; + case 0x730250: /* SPECIFIC_GET_CONNECTOR_DATA: {subDev, + * displayId, flags, DDCPartners, count, + * data[4]{index,type,location}, platform} */ + if (sz >= 72) { + v = 1; /* CONNECTOR_DATA_FLAGS_PRESENT_YES */ + memcpy(out_params + 8, &v, 4); + v = 1; /* count */ + memcpy(out_params + 16, &v, 4); + /* data[0].index = 0 */ + v = 0x61; /* CONNECTOR_DATA_TYPE_HDMI_A */ + memcpy(out_params + 24, &v, 4); + /* data[0].location = 0 */ + } + break; + case 0x73010b: /* SYSTEM_GET_HEAD_ROUTING_MAP: {subDev, + * displayMask, oldDisplayMask, oldHeadRoutingMap, + * headRoutingMap} - the generic zero reply dropped + * displayMask (0), so the boot display init's head- + * routing step saw no display. Keep display 0 routed + * to head 0 (headRoutingMap=0). */ + if (sz >= 20) { + u32 mask = 0; + + if (req_params) + memcpy(&mask, req_params + 4, 4); + mask &= 1; + if (!mask) + mask = 1; + memcpy(out_params + 4, &mask, 4); + /* headRoutingMap stays 0: display 0 -> head 0 */ + } + break; + case 0x731140: /* DFP_GET_INFO: {subDev, displayId, flags, + * UHBRSupportedByDfp} - flags signal bits = TMDS(0) */ + break; /* zeros are the TMDS signal value */ + } + *out_params_size = sz; + *out_status = 0; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DISPLAY shim cmd=0x%x -> 1 head / 1 HDMI connector (size %u req=%#x/%#x/%#x)", + cmd, sz, req_w0, req_w1, req_w2); +} + +/* + * NV0073_CTRL_CMD_SPECIFIC_GET_EDID_V2 (0x730245) + * + * nvidia-modeset's ReadEdidFromResman() reads the display EDID through this + * control, not through the I2C_TRANSACTION (0x402c0105) path. A zero or + * invalid reply leaves the DFP with no modes, so the driver falls back to + * the NoScanout NULL mode and the NVKMS deferred close bumps the shared + * nvClip modeNum. Serve a 128-byte EDID 1.3 base block with a 1920x1080@60 + * detailed timing plus the 640x480/800x600/1024x768 established timings + * (checksum valid, no extension block). + * + * Params (NV0073_CTRL_SPECIFIC_GET_EDID_V2_PARAMS): subDeviceInstance@0, + * displayId@4, bufferSize@8, flags@12, edidBuffer@16[2048]. Echo the whole + * request and patch bufferSize + edidBuffer (the request is zero-initialized + * by the caller; the reply must stay the same serialized length). + */ +static void gsp_rm_handle_display_get_edid(struct gsp_emu_context *ctx, + u32 req_size, + const u8 *req_params, + u32 *out_status, + u8 *out_params, + u32 *out_params_size) +{ + static const u8 edid[128] = { + 0x00, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, + 0x3a, 0xc4, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x01, 0x22, 0x01, 0x03, 0x80, 0x33, 0x1d, 0x78, + 0x0a, 0xee, 0x91, 0xa3, 0x54, 0x4c, 0x99, 0x26, + 0x0f, 0x50, 0x54, 0x21, 0x08, 0x00, 0x01, 0x01, + 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, + 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x3a, + 0x80, 0x18, 0x71, 0x38, 0x2d, 0x40, 0x58, 0x2c, + 0x45, 0x00, 0xfd, 0x1e, 0x11, 0x00, 0x00, 0x1a, + 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0xfd, 0x00, 0x32, + 0x4b, 0x1e, 0x53, 0x11, 0x00, 0x0a, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x10, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, + }; + u32 buffer_size = sizeof(edid); + u32 display_id = 0; + + if (!req_params || req_size < 16) { + *out_params_size = 0; + *out_status = 0x1; /* NV_ERR_INVALID_ARGUMENT */ + return; + } + memcpy(out_params, req_params, req_size); + memcpy(out_params + 8, &buffer_size, 4); + memcpy(out_params + 16, edid, sizeof(edid)); + memcpy(&display_id, out_params + 4, 4); + *out_params_size = req_size; + *out_status = 0; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: GET_EDID_V2 displayId=%u -> %u-byte EDID (1920x1080@60)", + display_id, buffer_size); +} + +/* * CE_GET_FAULT_METHOD_BUFFER_SIZE (0x20802a08) * * kchangrpInit allocates one fault method buffer per runqueue of this @@ -2063,6 +2798,18 @@ static void gsp_rm_handle_kgr_get_info(struct gsp_emu_context *ctx, info->infoList[TU104_GR_INFO_IDX_MAX_LEGACY_SUBCONTEXT_COUNT].data = TU104_MAX_SUBCONTEXT_COUNT; info->infoList[TU104_GR_INFO_IDX_MAX_PER_ENGINE_SUBCONTEXT_COUNT].data = TU104_MAX_SUBCONTEXT_COUNT; + /* + * GR identity for the nvidia-3d library. SM_VERSION in particular + * gates nv3dAllocDevice (GetSpaVersion -> FALSE on + * SM_VERSION_NONE), which makes nvHsAllocDevice fail silently and + * nvidia-drm report "Failed to allocate NvKmsKapiDevice". + */ + info->infoList[TU104_GR_INFO_IDX_SM_VERSION].data = TU104_GR_INFO_SM_VERSION_7_5; + info->infoList[TU104_GR_INFO_IDX_MAX_WARPS_PER_SM].data = TU104_MAX_WARPS_PER_SM; + info->infoList[TU104_GR_INFO_IDX_MAX_THREADS_PER_WARP].data = TU104_MAX_THREADS_PER_WARP; + info->infoList[TU104_GR_INFO_IDX_THREAD_STACK_SCALING_FACTOR].data = + TU104_THREAD_STACK_SCALING_FACTOR; + *out_params_size = TU104_GR_INFO_MAX_ENGINES * sizeof(struct tu104_gr_static_info); *out_status = 0; @@ -2161,11 +2908,16 @@ static void gsp_rm_handle_kgr_get_context_buffers_info( for (e = 0; e < TU104_CTX_GPU_MAX_ENGINES; e++) { for (i = 0; i < TU104_CTX_ENGINE_ID_COUNT; i++) { - /* engine IDs the chip does not support report - * NV_U32_MAX (the client skips those entries) */ + /* r1781/CONT.435: use 0 (not NV_U32_MAX) for buffer + * kinds this chip does not support. kgraphics treats + * size==0 as "not supported" and skips the buffer + * (kernel_graphics.c:1785); an NV_U32_MAX byte length + * would instead round up to exactly 4 GiB in the + * page-count computation. The reserve-size sum also + * guards on NV_U32_MAX, so 0 is both safe and honest. */ if (sizes[i].size == 0) { - info->engineContextBuffersInfo[e].engine[i].size = 0xFFFFFFFF; - info->engineContextBuffersInfo[e].engine[i].alignment = 0xFFFFFFFF; + info->engineContextBuffersInfo[e].engine[i].size = 0; + info->engineContextBuffersInfo[e].engine[i].alignment = 0; continue; } info->engineContextBuffersInfo[e].engine[i].size = sizes[i].size; @@ -2191,6 +2943,20 @@ static void gsp_rm_handle_kgr_get_context_buffers_info( * L2/LTC topology from here. Echo the requested indices and fill in the * emulated TU104 values. */ +/* + * m113: FB_GET_INFO_V2 COMPRESSION_SIZE (index 0x01, bytes). The header: + * "A return value of 0 indicates the GPU does not support compression" - + * but the guest driver still attempts a COMPR-enabled, 64 KB-aligned + * comptag allocation (vendor VidHeap alloc, attr COMPR=2) and fails it + * NV_ERR_NO_MEMORY (0x51) when the comptag pool has no space, which + * libcuda maps to VK_ERROR_OUT_OF_DEVICE_MEMORY (the vkAllocateMemory -2 + * gate). Report a comptag pool sized like the real Turing parts (~fb/16). + * 0 = the historic "no compression" reply. + */ +static uint gsp_rm_fb_compr_mb = 128; +module_param_named(wall2_fb_compr_mb, gsp_rm_fb_compr_mb, uint, 0644); +MODULE_PARM_DESC(wall2_fb_compr_mb, "FB_GET_INFO_V2 COMPRESSION_SIZE in MiB (default 128, 0 = none)"); + static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) { u64 fb = nv_vgpu_fb_emul_size(ctx->vgpu); @@ -2199,18 +2965,28 @@ static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) case TU104_FB_INFO_IDX_TILE_REGION_COUNT: return 0; case TU104_FB_INFO_IDX_COMPRESSION_SIZE: - return 0; + /* m113: non-zero comptag pool (bytes) - see gsp_rm_fb_compr_mb. */ + return gsp_rm_fb_compr_mb << 20; case TU104_FB_INFO_IDX_PARTITION_COUNT: return 1; + /* + * r1781/CONT.435: every size index below is documented in KB + * (ctrl2080fb.h: RAM_SIZE/TOTAL_RAM_SIZE/USABLE_RAM_SIZE/HEAP_SIZE/ + * MAPPABLE_HEAP_SIZE/HEAP_FREE/BAR1_SIZE/BAR1_AVAIL_SIZE are all + * "in kilobytes"). Returning the byte count over-reported the + * emulated FB size by 1024x to the X DDX (FbTotalSizeKb = + * 0x20000000 = 512 GB). COMPRESSION_SIZE and L2CACHE_SIZE stay + * in bytes per the same header. + */ case TU104_FB_INFO_IDX_BAR1_SIZE: - return (u32)fb; + return (u32)(fb >> 10); case TU104_FB_INFO_IDX_RAM_SIZE: case TU104_FB_INFO_IDX_TOTAL_RAM_SIZE: case TU104_FB_INFO_IDX_USABLE_RAM_SIZE: - return (u32)fb; + return (u32)(fb >> 10); case TU104_FB_INFO_IDX_HEAP_SIZE: case TU104_FB_INFO_IDX_MAPPABLE_HEAP_SIZE: - return (u32)fb; + return (u32)(fb >> 10); case TU104_FB_INFO_IDX_BUS_WIDTH: return 256; case TU104_FB_INFO_IDX_RAM_TYPE: @@ -2223,7 +2999,7 @@ static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) case TU104_FB_INFO_IDX_PARTITION_MASK: return 1; case TU104_FB_INFO_IDX_HEAP_FREE: - return (u32)fb; + return (u32)(fb >> 10); case TU104_FB_INFO_IDX_RAM_LOCATION: return TU104_RAM_LOCATION_GPU_DEDICATED; case TU104_FB_INFO_IDX_FBP_COUNT: @@ -2236,7 +3012,7 @@ static u32 tu104_fb_info_data(struct gsp_emu_context *ctx, u32 index) return TU104_MEMORYINFO_VENDOR_ID_SAMSUNG; case TU104_FB_INFO_IDX_BAR1_AVAIL_SIZE: case TU104_FB_INFO_IDX_BAR1_MAX_CONTIG_AVAIL_SIZE: - return (u32)fb; + return (u32)(fb >> 10); case TU104_FB_INFO_IDX_LTC_COUNT: return TU104_LTC_COUNT; case TU104_FB_INFO_IDX_LTS_COUNT: @@ -2356,6 +3132,13 @@ static void gsp_rm_handle_clk_domain_info(struct gsp_emu_context *ctx, u32 cmd, * sysmem connection type"). Report a PCIe Gen3 x16 link for the emulated * vGPU and echo every requested index. */ +static u32 gsp_rm_bus_gart_mb = 256; +module_param_named(bus_gart_mb, gsp_rm_bus_gart_mb, uint, 0644); +MODULE_PARM_DESC(bus_gart_mb, + "BUS_GET_INFO_V2 GPU_GART_SIZE in MiB (default 256; 0 = historical zero reply, A/B). " + "4096 made the guest allocate a 4 GiB GART sysmem object at GL init (c15-c22 OOM/crash); " + "256 keeps the GART well below guest RAM."); + static u32 tu104_bus_info_data(u32 index) { switch (index) { @@ -2376,6 +3159,14 @@ static u32 tu104_bus_info_data(u32 index) return TU104_PCIE_LINK_CTRL_STATUS_GEN3_X16; case TU104_BUS_INFO_IDX_SYSMEM_CONNECTION_TYPE: return TU104_BUS_INFO_SYSMEM_CONN_PCIE; + case TU104_BUS_INFO_IDX_GPU_GART_SIZE: + return READ_ONCE(gsp_rm_bus_gart_mb); + case TU104_BUS_INFO_IDX_GPU_GART_SIZE_ALT: + return 0; + case TU104_BUS_INFO_IDX_GPU_GART_DMA_FLAGS_0: + case TU104_BUS_INFO_IDX_GPU_GART_DMA_FLAGS_1: + /* GPUGART | CTXDMA (bits 0 | 2). */ + return 0x5u; default: return 0; } @@ -2456,6 +3247,7 @@ static void gsp_rm_handle_bus_get_info_v2(struct gsp_emu_context *ctx, #define GSPRM_COPY_PDES_MAX_LEVELS 6 static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, + u32 h_object, const u8 *params, u32 params_size) { struct gsp_rm_emu *rm = ctx->rm_emu; @@ -2506,6 +3298,46 @@ static void gsp_rm_handle_copy_server_pdes(struct gsp_emu_context *ctx, "vgsp-rm: pt level[%u] phys=0x%llx size=0x%llx aperture=%u pageShift=%u", i, phys, size, aperture, page_shift); } + + /* + * r496j: this control IS the kernel RM publishing the page-table + * chain of its internal VAS (the level[0..] instances with their + * phys addresses and page shifts). The chain can name the X/GL + * channel's pushbuffer pages and progress semaphore, for which no + * other translation source exists (the fn-54 SET_PAGE_DIRECTORY + * path is external-VAS/UVM only). It is NOT registered directly: + * a blast registration of every client's directory let a stale + * kernel directory answer another client's VA in the walker's + * first-success order (the r459 stale-directory class; observed + * as a CUDA client hanging after DEVICE with r496g). Instead the + * root is offered as a CANDIDATE; the progress service registers + * it only after it has been VERIFIED against a content-validated + * pair of that very client (the ring found by the RAM content + * scan). + */ + if (num_levels >= 1 && rm->pt.levels[0].phys && + rm->pt.levels[0].size >= 16 && + rm->pt.levels[0].size <= 4096 * 8) { + /* + * The level phys values are guest PAs on this platform + * (observed 4+ GiB, above any FB shadow); aperture 3 in the + * message does not match the walker's FB/SYS encoding, so + * classify by address: values inside the emulated VRAM are + * FB, everything else sysmem. The walker also retries the + * other aperture, so a misclassification only costs a walk. + */ + bool vidmem = rm->pt.levels[0].phys < + (u64)ctx->vgpu->bars[1].size; + + gsp_sema_service_vaspace_candidate_add( + h_object, rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), vidmem); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: VAS 0x%x page directory offered as candidate pdb=0x%llx entries=%u (%s; registered only after a content-verified ring walk)", + h_object, (u64)rm->pt.levels[0].phys, + (u32)(rm->pt.levels[0].size / 8), + vidmem ? "vidmem" : "sysmem"); + } } static bool gsp_rm_alloc_interesting(u32 class_id); @@ -2522,8 +3354,21 @@ static bool gsp_rm_is_channel_class(u32 class_id) case 0x906f: /* MAXWELL_CHANNEL_GPFIFO_A */ case 0xa06f: /* KEPLER_CHANNEL_GPFIFO_A */ case 0xb06f: /* PASCAL_CHANNEL_GPFIFO_A */ - case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A */ + case 0xc06f: /* VOLTA_CHANNEL_GPFIFO_A (PRE-TURING NAME) */ + case 0xc36f: /* VOLTA_CHANNEL_GPFIFO_A */ + /* + * r828: the RC watchdog's channel allocates 0xc36f on this + * device (the driver's gpfifoMapping loop picks the first + * class the gpu reports supported) - without it in this set + * the channel never reaches the progress service, its class2d + * notifier cycle (SET_NOTIFY_A/B + NOTIFY) never executes and + * the guest's krcWatchdog declares the GPU locked after its + * 7 s timeout as soon as robust channels are in play. + */ case 0xc46f: /* TURING_CHANNEL_GPFIFO_A */ + case 0xc56f: /* AMPERE_CHANNEL_GPFIFO_A */ + case 0xc86f: /* HOPPER_CHANNEL_GPFIFO_A */ + case 0xc96f: /* BLACKWELL_CHANNEL_GPFIFO_A */ return true; default: return false; @@ -2539,14 +3384,317 @@ static bool gsp_rm_is_channel_class(u32 class_id) #define GSPRM_CHAN_USERD_MEM 168 #define GSPRM_CHAN_MEM_APERTURE 16 /* offset of addressSpace in the desc */ #define GSPRM_CHAN_NOTIFIER_MEM 248 +#define GSPRM_CHAN_HUSERD_MEMORY 32 /* NvHandle hUserdMemory[NV_MAX_SUBDEVICES] */ +#define GSPRM_CHAN_USERD_OFFSET 64 /* NvU64 userdOffset[NV_MAX_SUBDEVICES] */ + +/* + * CONT.438: client memory-object ALLOCs - the objects hUserdMemory can name. + * + * NV_MEMORY_ALLOCATION_PARAMS (nvos.h:1631-1668; ABI asserted by + * gsp_phys_booter.c static_asserts): 128 bytes, u64 fields 8-byte aligned. + * attr @24 (u32) location bits 1:0 (NVOS32_ATTR_LOCATION_VIDMEM = 0, + * NVOS32_ATTR_LOCATION_PCI = 1) + * size @64 (u64) [IN/OUT] + * offset @80 (u64) [IN/OUT] - the server-returned GPU physical base for + * sysmem (class 0x3e NV01_MEMORY_SYSTEM) or the FB + * offset for vidmem (class 0x40 NV01_MEMORY_LOCAL_USER). + * Both classes share this layout (cl003e.h:33, + * cl0040.h:34; the allocation call is nv_gpu_ops.c: + * 4644). + * + * NV_MEMORY_VIRTUAL_ALLOCATION_PARAMS (cl0070.h:66-70) is only 24 bytes + * { offset @0, limit @8, hVASpace @16 } - a VA-range reservation, not a + * physical allocation, so it must never be turned into a PA. Class 0x50a0 + * NV50_MEMORY_VIRTUAL is allocated with NV_MEMORY_ALLOCATION_PARAMS and also + * returns a VA in .offset (nv_gpu_ops.c:4727-4730); same rule. + */ +#define GSPRM_MEM_ALLOC_SIZE 64 +#define GSPRM_MEM_ALLOC_OFFSET 80 +#define GSPRM_MEM_VIRT_OFFSET 0 +#define GSPRM_MEM_VIRT_LIMIT 8 + +static struct gsp_rm_mem_object * +gsp_rm_mem_object_find(struct gsp_rm_emu *rm, u32 h_client, u32 h_object) +{ + u32 i; + + for (i = 0; i < rm->mem_objects_n; i++) { + struct gsp_rm_mem_object *e = &rm->mem_objects[i]; + + if (e->valid && e->h_client == h_client && + e->h_object == h_object) + return e; + } + return NULL; +} + +/* Reuse a retired slot, append while the table has room, else evict. */ +static struct gsp_rm_mem_object *gsp_rm_mem_object_slot(struct gsp_rm_emu *rm) +{ + struct gsp_rm_mem_object *e; + u32 i, slot; + + for (i = 0; i < rm->mem_objects_n; i++) { + if (!rm->mem_objects[i].valid) + return &rm->mem_objects[i]; + } + if (rm->mem_objects_n < GSPRM_MAX_MEM_OBJECTS) + return &rm->mem_objects[rm->mem_objects_n++]; + + slot = rm->mem_objects_next++ % GSPRM_MAX_MEM_OBJECTS; + e = &rm->mem_objects[slot]; + if (rm->mem_object_evictions < 8) { + rm->mem_object_evictions++; + pr_info("vgsp-rm: mem-object table full; evicting (hClient=0x%x hObject=0x%x class=0x%x) for a new allocation\n", + e->h_client, e->h_object, e->class_id); + } + return e; +} + +static void gsp_rm_mem_object_note(struct gsp_rm_emu *rm, u32 h_client, + u32 h_parent, u32 h_object, u32 class_id, + const u8 *params, u32 params_size) +{ + struct gsp_rm_mem_object *e; + u64 base = 0, size = 0; + u32 address_space; + + switch (class_id) { + case 0x3e: /* NV01_MEMORY_SYSTEM */ + case 0x40: /* NV01_MEMORY_LOCAL_USER */ + if (params_size < GSPRM_MEM_ALLOC_OFFSET + 8) + return; + memcpy(&base, params + GSPRM_MEM_ALLOC_OFFSET, 8); + if (params_size >= GSPRM_MEM_ALLOC_SIZE + 8) + memcpy(&size, params + GSPRM_MEM_ALLOC_SIZE, 8); + address_space = class_id == 0x3e ? GSPRM_MEM_AS_SYSMEM + : GSPRM_MEM_AS_VIDMEM; + break; + case 0x70: /* NV01_MEMORY_VIRTUAL (VA range) */ + if (params_size < 16) + return; + memcpy(&base, params + GSPRM_MEM_VIRT_OFFSET, 8); + memcpy(&size, params + GSPRM_MEM_VIRT_LIMIT, 8); + address_space = GSPRM_MEM_AS_VIRTUAL; + break; + case 0x50a0: /* NV50_MEMORY_VIRTUAL (VA in .offset) */ + if (params_size < GSPRM_MEM_ALLOC_OFFSET + 8) + return; + memcpy(&base, params + GSPRM_MEM_ALLOC_OFFSET, 8); + if (params_size >= GSPRM_MEM_ALLOC_SIZE + 8) + memcpy(&size, params + GSPRM_MEM_ALLOC_SIZE, 8); + address_space = GSPRM_MEM_AS_VIRTUAL; + break; + default: + return; + } + + e = gsp_rm_mem_object_find(rm, h_client, h_object); + if (!e) + e = gsp_rm_mem_object_slot(rm); + e->h_object = h_object; + e->h_client = h_client; + e->h_parent = h_parent; + e->class_id = class_id; + e->base = base; + e->size = size; + e->address_space = (u8)address_space; + e->valid = true; + + if ((class_id == 0x3e || class_id == 0x40) && + rm->mem_object_logs < 64) { + rm->mem_object_logs++; + nv_vgpu_info(rm->vgpu, + "vgsp-rm: mem-object hClient=0x%x hParent=0x%x hObject=0x%x class=0x%x base=0x%llx size=0x%llx as=%u\n", + h_client, h_parent, h_object, class_id, base, + size, address_space); + } +} -static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, - const u8 *params, u32 params_size) +static void gsp_rm_mem_object_retire(struct gsp_rm_emu *rm, u32 h_root, + u32 h_object) +{ + u32 i, retired = 0; + + for (i = 0; i < rm->mem_objects_n; i++) { + struct gsp_rm_mem_object *e = &rm->mem_objects[i]; + + if (!e->valid) + continue; + /* + * A normal FREE names the object (hObjectOld) and the freeing + * client (hRoot); a root FREE names the client itself. Handles + * are recycled across sessions, so both must match. + */ + if ((h_object && h_object != h_root && + e->h_object == h_object && + (!h_root || e->h_client == h_root)) || + (h_object && h_object == h_root && + e->h_client == h_root)) { + e->valid = false; + retired++; + } + } + if (retired && gsp_sema_dbg) + nv_vgpu_info(rm->vgpu, + "vgsp-rm: retired %u mem-object record(s) on FREE hObject=0x%x hRoot=0x%x\n", + retired, h_object, h_root); +} + +static bool gsp_rm_mem_object_alias(struct gsp_rm_emu *rm, u32 h_dst_client, + u32 h_dst_object, u32 h_src_client, + u32 h_src_object) +{ + struct gsp_rm_mem_object *src, *dst; + + src = gsp_rm_mem_object_find(rm, h_src_client, h_src_object); + if (!src) + return false; + + dst = gsp_rm_mem_object_find(rm, h_dst_client, h_dst_object); + if (!dst) + dst = gsp_rm_mem_object_slot(rm); + + *dst = *src; + dst->h_client = h_dst_client; + dst->h_object = h_dst_object; + dst->valid = true; + return true; +} + +/* + * CONT.438: resolve a GSP-client channel's client-allocated USERD. + * + * On GSP, bClientAllocatedUserD channels carry hUserdMemory[subdev] + * (NV_CHANNEL_ALLOC_PARAMS @32; gsp_abi_check.c:1764) and + * userdOffset[subdev] (@64), while the userdMem memdesc (@168) stays zero. + * The real server resolves the handle to a memory object and uses + * memdescGetPhysAddr(..., userdOffset) (kernel_channel_gv100.c:204). + * We can do the same only when the object's ALLOC already taught us a + * physical base; otherwise leave USERD zero and say why. Never invent a PA: + * the service later writes GP_GET through the adopted block. + */ +static void gsp_rm_resolve_channel_userd(struct gsp_emu_context *ctx, + struct sema_channel_desc *desc, + u32 h_client, u32 h_object, + u32 h_userd, u64 userd_offset) +{ + struct gsp_rm_emu *rm = ctx->rm_emu; + struct gsp_rm_mem_object *e; + u64 pa; + + e = gsp_rm_mem_object_find(rm, h_client, h_userd); + if (!e) { + struct gsp_rm_mem_object *alias = NULL; + u32 i; + + for (i = 0; i < rm->mem_objects_n; i++) { + struct gsp_rm_mem_object *cand = &rm->mem_objects[i]; + + if (cand->valid && cand->h_object == h_userd) { + alias = cand; + break; + } + } + if (alias) + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x off=0x%llx UNRESOLVED (handle exists under hClient=0x%x class=0x%x base=0x%llx, channel hClient=0x%x); leaving USERD zero\n", + h_object, h_userd, + (unsigned long long)userd_offset, + alias->h_client, alias->class_id, + (unsigned long long)alias->base, + h_client); + else + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x off=0x%llx UNRESOLVED (no memory object in table, n=%u); leaving USERD zero\n", + h_object, h_userd, + (unsigned long long)userd_offset, + rm->mem_objects_n); + return; + } + + if (e->address_space == GSPRM_MEM_AS_VIRTUAL) { + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x class=0x%x is a virtual range (base=0x%llx); refusing to guess a PA, leaving USERD zero\n", + h_object, h_userd, e->class_id, + (unsigned long long)e->base); + return; + } + if (!e->base) { + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x class=0x%x base=0 (server did not echo a physical base); leaving USERD zero\n", + h_object, h_userd, e->class_id); + return; + } + /* A physical base is page-granular; anything else is not a PA we know. */ + if (e->base & 0xfff) { + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x class=0x%x unaligned base=0x%llx; leaving USERD zero\n", + h_object, h_userd, e->class_id, + (unsigned long long)e->base); + return; + } + /* GP_GET @0x88 / GP_PUT @0x8c must lie inside the object. */ + if (e->size && userd_offset + 0x90 > e->size) { + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x off=0x%llx outside size=0x%llx; leaving USERD zero\n", + h_object, h_userd, + (unsigned long long)userd_offset, + (unsigned long long)e->size); + return; + } + + pa = e->base + userd_offset; + if (e->address_space == GSPRM_MEM_AS_VIDMEM && + pa + 0x90 > nv_vgpu_fb_emul_size(ctx->vgpu)) { + pr_info_ratelimited("vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x pa=0x%llx beyond emulated FB (0x%llx); leaving USERD zero\n", + h_object, h_userd, (unsigned long long)pa, + (unsigned long long)nv_vgpu_fb_emul_size(ctx->vgpu)); + return; + } + + desc->userd_pa = pa; + desc->userd_as = e->address_space == GSPRM_MEM_AS_VIDMEM ? 2 : 1; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: chan USERD client-alloc hObject=0x%x hUserd=0x%x class=0x%x base=0x%llx off=0x%llx as=%u -> pa=0x%llx (memory-object table)\n", + h_object, h_userd, e->class_id, + (unsigned long long)e->base, + (unsigned long long)userd_offset, + desc->userd_as, (unsigned long long)pa); +} + +static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_client, + u32 h_object, const u8 *params, + u32 params_size) { struct sema_channel_desc desc; + u64 userd_wire_base = 0; + + /* + * CONT.405: name the registration attempt for the direct-GL client's + * channel families. A "beef"/"bfef" handle that appears more than + * once is the client's alloc -> setup -> free -> retry loop; the + * matching beef step trace in gsp_rm_handle_rm_control names the + * last control it issued before the retry. Bounded. + */ + if (((h_object & 0xffff0000u) == 0xbeef0000u || + (h_object & 0xffffff00u) == 0xbfef0000u)) { + static u32 beef_reg_log; + + if (beef_reg_log < 64) { + beef_reg_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: beef chan register #%u h=0x%x (setup step: alloc_channel params=%u; next client step is BIND/GPFIFO_SCHEDULE, then its first kick)", + beef_reg_log, h_object, params_size); + } + } - if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) + if (params_size < GSPRM_CHAN_NOTIFIER_MEM + 8) { + /* + * r828 diag: a channel-class alloc whose params are shorter + * than the notifier field is not registered, silently. The + * RC watchdog channel (hObject 0x31415900) never appeared in + * the sema table; name every such skip with its size so the + * shape question is answered by a log, not inference. + */ + pr_info("vgsp-rm: channel alloc hObject=0x%x params_size=%u too short (< %u); NOT registered\n", + h_object, params_size, GSPRM_CHAN_NOTIFIER_MEM + 8); return; + } memset(&desc, 0, sizeof(desc)); desc.vgpu = ctx->vgpu; @@ -2554,19 +3702,300 @@ static void gsp_rm_register_channel(struct gsp_emu_context *ctx, u32 h_object, memcpy(&desc.gpfifo_va, params + GSPRM_CHAN_GPFIFO_OFFSET, 8); memcpy(&desc.gpfifo_entries, params + GSPRM_CHAN_GPFIFO_ENTRIES, 4); memcpy(&desc.engine_type, params + GSPRM_CHAN_ENGINE_TYPE, 4); + /* + * The wire carries NV2080_ENGINE_TYPE_*: NULL = 0, GR0 (== GRAPHICS) + * = 1, COPY0 = 9, NVDEC0 = 0x13, ... The progress service's + * convention is "engine_type == 0 means the GR channel" (CUDA's + * kernel-RM GR channels arrive with engineType NULL), so an X/GL 3D + * channel - which carries the real NV2080_ENGINE_TYPE_GRAPHICS = 1 - + * was misclassified as a CE and its push never executed ("Failed to + * initialize DMA" in the X driver's pushbuffer NOP test). Normalize + * GR0..GR7 (1..8) to 0: the emulated device has a single GR and every + * other engine id is >= COPY0 (9), so this cannot capture a CE. + */ + if (desc.engine_type >= 1 && desc.engine_type <= 8) + desc.engine_type = 0; + /* + * r835: the channel's own address space (NV_CHANNEL_ALLOC_PARAMS: + * hContextShare @24, hVASpace @28 - confirmed by this struct's own + * known offsets: engineType @128 = 32 + 8 handles + 8 subdevice + * handles + 8 u64 userd offsets). Two clients map different objects + * at overlapping VAs (everyone's RM window sits at 0x1200xxxxx), so + * the global VA-keyed translation maps conflate them: the X server's + * ring VA resolved through the GL client's page tables (walk-probe: + * vas h=0xcaf00005 answered 0x120090000 with a zero page) and its + * push stream decoded as empty, wedging X exactly like the RC + * watchdog's ring. The service resolves a channel's VAs through this + * VAS first. + */ + memcpy(&desc.h_vaspace, params + 28, 4); memcpy(&desc.cid, params + GSPRM_CHAN_CID, 4); memcpy(&desc.instance_pa, params + GSPRM_CHAN_INSTANCE_MEM, 8); memcpy(&desc.userd_pa, params + GSPRM_CHAN_USERD_MEM, 8); + userd_wire_base = desc.userd_pa; + /* + * SESSION 153-CONTINUED-38: the USERD control page sits at + * userdMem.address + userdOffset[subDeviceId] (NV_CHANNEL_ALLOC_PARAMS + * userdOffset[NV_MAX_SUBDEVICES], @64; subDeviceId @136, the + * one-hot bit index is normally 0 but the X client-share channel + * carries 1). The GL client's kernel channels declare userdMem fb + * 0x18fe00 with userdOffset[0]=0x7fe00, so the service was + * probing/monitoring 0x18fe00 while the real control page + * (GPGet/GPPut) lives at 0x20fc00 - both the registered candidate + * and every FB observation were at the wrong offset, which is why + * the direct client's channels never resolved a USERD (CONT-37 + * alloc/free retry loop). Index by the subdevice. + */ + /* + * CONT.404: read the aperture before the flags log so the logged + * /u value is the real one (it used to print 0 because the memcpy + * sat after the block). + */ memcpy(&desc.userd_as, params + GSPRM_CHAN_USERD_MEM + GSPRM_CHAN_MEM_APERTURE, 4); + { + u32 sd = 0; + u64 uo = 0; + u32 flags = 0; + + if (params_size >= 140) + memcpy(&sd, params + 136, 4); + /* + * CONT.402: subDeviceId is a one-hot bitmask + * (alloc_channel.h), so the userdOffset[] slot is its first + * set bit, not the raw value. + */ + sd = sd ? (u32)__builtin_ctz(sd) : 0; + if (sd >= 8) + sd = 0; + if (params_size >= 64 + (sd + 1) * 8) { + memcpy(&uo, params + 64 + (u64)sd * 8, 8); + desc.userd_pa += uo; + } + /* + * CONT.402: NVOS04_FLAGS_CHANNEL_USERD_INDEX_VALUE (bits 10:8) + * and USERD_INDEX_PAGE_VALUE (bits 20:12) select the 0x200 + * control block inside the declared userdMem page. Logged so + * the block offset can be validated before it is applied. + * + * CONT.404: apply it. The kernel driver sets idx = ChID % 8 and + * page = ChID / 8 (kernel_channel.c, IS_GSP_CLIENT path), so a + * channel whose ChID is nonzero has its GPGet/GPPut 0x200-aligned + * block somewhere inside the declared userdMem. Reading base+0 + * made the guest's GP_PUT advances invisible - X's channel got + * ChID=3 in r1697 and timed out in "Failed to initialize DMA" + * with the host watching fill at the wrong block. + */ + if (params_size >= 24) { + u64 base = desc.userd_pa; + u32 idx, pg; + + memcpy(&flags, params + 20, 4); + idx = (flags >> 8) & 0x7; + pg = (flags >> 12) & 0x1ff; + if (idx != 0 || pg != 0) + desc.userd_pa += (u64)pg * 0x1000 + + (u64)idx * 0x200; + pr_info("vgsp-rm: chan userd flags=0x%x idx=%u page=%u base=0x%llx block=0x%llx/%u hObject=0x%x\n", + flags, idx, pg, base, desc.userd_pa, + desc.userd_as, h_object); + } + /* + * CONT.438: a GSP-client kernel channel carries its USERD as + * hUserdMemory[subdev] (@32), not in the userdMem memdesc + * (@168): the wire memdesc base is zero while userdOffset[] + * may be nonzero. Do not let base 0 + offset/flags masquerade + * as a small PA - resolve the real memory object, or leave + * USERD zero (the service must never write a guessed block). + */ + if (userd_wire_base == 0) { + u64 wire_pa = desc.userd_pa; + u32 wire_as = desc.userd_as; + u32 h_userd = 0; + + desc.userd_pa = 0; + desc.userd_as = 0; + if (params_size >= + GSPRM_CHAN_USERD_OFFSET + (sd + 1) * 8 && + params_size >= + GSPRM_CHAN_HUSERD_MEMORY + + (sd + 1) * 4) { + memcpy(&h_userd, + params + GSPRM_CHAN_HUSERD_MEMORY + + (u64)sd * 4, 4); + if (h_userd) + gsp_rm_resolve_channel_userd(ctx, + &desc, + h_client, + h_object, + h_userd, + uo); + } + /* + * CONT.438b: the X/GLX kernel channel needs its + * wire-derived block until the client memory-object + * table can name the real USERD. c24 (V30) cleared + * it and X failed "Failed to initialize DMA"/ + * "Failed to allocate push buffer"; c23 (V28) used + * base 0 + userdOffset and X started. Keep the old + * value as a fallback, but only when it is a real + * 4K-backed offset (the resolver still wins when it + * has a verified memory object). + */ + if (!desc.userd_pa && wire_pa) { + desc.userd_pa = wire_pa; + desc.userd_as = wire_as; + pr_info_ratelimited("vgsp-rm: chan USERD hObject=0x%x hUserd=0x%x unresolved; fallback to wire base+offset=0x%llx as=%u (no guessed table PA)\n", + h_object, h_userd, + wire_pa, wire_as); + } + } + } memcpy(&desc.notifier_pa, params + GSPRM_CHAN_NOTIFIER_MEM, 8); memcpy(&desc.notifier_as, params + GSPRM_CHAN_NOTIFIER_MEM + GSPRM_CHAN_MEM_APERTURE, 4); + /* + * CONT.404 diag: the registered block is not always the block the + * client writes - X's live GP_PUT was adopted by the shape scan at + * fb 0x910000 while the wire declared 0x190000 (+0x600), and the + * direct-GL channel's declared page stayed zero. Name the whole + * wire memory-descriptor set and peek the registered block's + * GP_GET/GP_PUT, so the next cycle separates "wire base is not the + * client's block" from "client never writes FB". Bounded. + */ + { + static u32 md_log; + u64 inst_b = 0, ramfc_b = 0; + u32 inst_sz = 0, ramfc_sz = 0; + u32 inst_as = 0, ramfc_as = 0; + u32 g0 = 0, p0 = 0; + + if (md_log < 32) { + md_log++; + memcpy(&inst_b, params + GSPRM_CHAN_INSTANCE_MEM, 8); + memcpy(&inst_sz, params + GSPRM_CHAN_INSTANCE_MEM + 8, 4); + memcpy(&inst_as, params + GSPRM_CHAN_INSTANCE_MEM + + GSPRM_CHAN_MEM_APERTURE, 4); + memcpy(&ramfc_b, params + 192, 8); + memcpy(&ramfc_sz, params + 200, 4); + memcpy(&ramfc_as, params + 192 + + GSPRM_CHAN_MEM_APERTURE, 4); + if (desc.userd_as == 2 && ctx->vgpu && + ctx->vgpu->bars[1].vaddr && + desc.userd_pa + 0x90 <= + nv_vgpu_fb_emul_size(ctx->vgpu)) { + memcpy(&g0, ctx->vgpu->bars[1].vaddr + + desc.userd_pa + 0x88, 4); + memcpy(&p0, ctx->vgpu->bars[1].vaddr + + desc.userd_pa + 0x8c, 4); + } + pr_info("vgsp-rm: chan mds hObject=0x%x inst=0x%llx/%u/%u userd=0x%llx/%u ramfc=0x%llx/%u/%u notif=0x%llx/%u getput=%08x/%08x\n", + h_object, inst_b, inst_sz, inst_as, + desc.userd_pa, desc.userd_as, + ramfc_b, ramfc_sz, ramfc_as, + desc.notifier_pa, desc.notifier_as, g0, p0); + } + } + + /* + * SESSION 153-CONTINUED-15 (draw attribution): every channel-class + * allocation is named with its wire shape before registration, so a + * channel the service silently refuses (gpfifo VA 0 / entries < 2) + * cannot hide behind an absent "registered" line. Bounded. + */ + { + static u32 chan_alloc_log; + + if (chan_alloc_log < 32) { + chan_alloc_log++; + pr_info("vgsp-rm: chan alloc hObject=0x%x gpfifo=0x%llx ent=%u eng=%u vas=0x%x cid=%u userd=0x%llx/%u notif=0x%llx/%u\n", + h_object, desc.gpfifo_va, desc.gpfifo_entries, + desc.engine_type, desc.h_vaspace, desc.cid, + desc.userd_pa, desc.userd_as, + desc.notifier_pa, desc.notifier_as); + } + } + /* + * SESSION 153-CONTINUED-38: name the per-subdevice userdOffset (the + * kernel channels' control page may sit inside the declared userdMem + * at a nonzero offset; the service currently assumes address+0). + * Bounded. + */ + { + static u32 uo_log; + + if (uo_log < 8 && params_size >= 80) { + u64 uo0 = 0; + u32 sd = 0; + + memcpy(&uo0, params + 64, 8); + if (params_size >= 140) + memcpy(&sd, params + 136, 4); + uo_log++; + pr_info("vgsp-rm: chan userdOff[0]=0x%llx subDevId=0x%x hObject=0x%x (userd memdesc=0x%llx/%u)\n", + uo0, sd, h_object, desc.userd_pa, + desc.userd_as); + } + } gsp_sema_service_channel_add(&desc); } /* + * CONT.406: GR class objects (TURING_A 0xc597 and TURING_COMPUTE_A 0xc5c0) + * are allocated on a channel and the client frees them again when the next + * step of its GR context setup fails. Keep the last few allocations so the + * FREE side can name the object class and how long the client kept it: that + * turns "the client freed the 3D object" into a measured keep time, and + * separates "the alloc reply was rejected" (keep time ~0) from "the client + * failed later and cleaned up" (keep time ~80 ms, the observed r1700 case). + * All alloc/free RPC handling is serialized on the emulation thread, so the + * table needs no lock. Bounded. + */ +#define GSPRM_GR3D_LOG_MAX 24 +#define GSPRM_MAX_GROBJS 16 + +static struct { + u32 h; + u32 class_id; + unsigned long at; +} gsp_rm_grobjs[GSPRM_MAX_GROBJS]; +static u32 gsp_rm_grobjs_next; + +static void gsp_rm_grobj_note(u32 h, u32 class_id) +{ + u32 i; + + for (i = 0; i < GSPRM_MAX_GROBJS; i++) { + if (gsp_rm_grobjs[i].h == h) { + gsp_rm_grobjs[i].class_id = class_id; + gsp_rm_grobjs[i].at = jiffies; + return; + } + } + i = gsp_rm_grobjs_next++ % GSPRM_MAX_GROBJS; + gsp_rm_grobjs[i].h = h; + gsp_rm_grobjs[i].class_id = class_id; + gsp_rm_grobjs[i].at = jiffies; +} + +static bool gsp_rm_grobj_retire(u32 h, u32 *class_id, unsigned long *at) +{ + u32 i; + + for (i = 0; i < GSPRM_MAX_GROBJS; i++) { + if (gsp_rm_grobjs[i].h != h || !gsp_rm_grobjs[i].class_id) + continue; + *class_id = gsp_rm_grobjs[i].class_id; + *at = gsp_rm_grobjs[i].at; + gsp_rm_grobjs[i].class_id = 0; + gsp_rm_grobjs[i].h = 0; + return true; + } + return false; +} + +/* * GSP_RM_ALLOC (103) - rpc_gsp_rm_alloc_v03_00 (32 byte header) followed by * the class-specific allocation parameters. * @@ -2606,11 +4035,37 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, "vgsp-rm: GSP_RM_ALLOC class=0x%x paramsSize=%u flags=0x%x hClient=0x%x hParent=0x%x hObject=0x%x", class_id, params_size, flags, hc, hp, ho); + /* + * CONT.405: same bounded step trace as gsp_rm_handle_rm_control, for + * the direct-GL client's object allocations. Channel classes get a + * retry counter per handle so the alloc/free loop is visible without + * gsp_sema_dbg. + */ + { + bool beef = (hc >= 0xc1d00010u && hc <= 0xc1d0002fu) || + (ho & 0xffff0000u) == 0xbeef0000u || + (ho & 0xffffff00u) == 0xbfef0000u; + + if (beef) { + static u32 beef_alloc_log; + + if (beef_alloc_log < 128) { + beef_alloc_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: beef alloc step #%u class=0x%x hClient=0x%x hParent=0x%x hObject=0x%x sz=%u%s", + beef_alloc_log, class_id, hc, hp, ho, + params_size, + gsp_rm_is_channel_class(class_id) ? + " (channel)" : ""); + } + } + } + /* Full param dump for memory/channel classes and any class we have not * classified yet (rate-limited). %phN is capped at 64 bytes per print, * so chunk it. */ if (gsp_sema_dbg && gsp_rm_alloc_interesting(class_id) && - rm->alloc_param_dumps < 60) { + rm->alloc_param_dumps < 500) { u32 shown = min(params_size, (u32)384); u32 off; @@ -2625,17 +4080,51 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, } } + /* + * CONT.438: remember client memory objects (sysmem/vidmem/virtual) + * so a GSP-client channel's hUserdMemory[subdev] can be resolved to + * a physical USERD base later in this same serialized RPC stream. + */ + gsp_rm_mem_object_note(rm, hc, hp, ho, class_id, params, params_size); + /* Channels: hand the progress service everything it needs to run the * work submitted on this channel (GPFIFO ring + USERD). */ if (gsp_rm_is_channel_class(class_id)) - gsp_rm_register_channel(ctx, ho, params, params_size); + gsp_rm_register_channel(ctx, hc, ho, params, params_size); + + /* + * r1621: display channel objects (core 0xc57d, window 0xc57e, cursor + * 0xc57b, cursor-PIO base 0xc57a). Record class+instance -> hObject + * so the display engine can post the completion event the driver + * registered on the channel and name the channel in diagnostics. + * instance is the first u32 of both the DMA and PIO alloc params. + */ + if ((class_id == 0xc57d || class_id == 0xc57e || + class_id == 0xc57b || class_id == 0xc57a) && + params_size >= 4) { + u32 inst = 0; + + memcpy(&inst, params, 4); + gsp_disp_engine_note_object(ctx->vgpu, class_id, inst, ho); + } /* - * NV01_EVENT (allocated with class 0x79 in the guest ABI; params are - * NV0005_ALLOC_PARAMETERS {hParentClient, hSrcResource, hClass, - * notifyIndex, data}). The driver allocates one per channel and waits - * for the GPU to post the event before it considers submitted work - * complete - remember them so completed work can be signalled back. + * NV01_EVENT (params are NV0005_ALLOC_PARAMETERS {hParentClient, + * hSrcResource, hClass, notifyIndex, data}). The driver allocates one + * per channel and waits for the GPU to post the event before it + * considers submitted work complete - remember them so completed work + * can be signalled back. + * + * r682: the guest ABI uses THREE event classes and only the + * NV01_EVENT_OS_EVENT (0x79) shape was registered: libcuda's channel + * events arrive as 0x79, but the modern nvkms/Vulkan path allocates + * NV01_EVENT_KERNEL_CALLBACK_EX (0x7E, nvos.h) - observed live as + * `GSP_RM_ALLOC class=0x7e paramsSize=24 hParent=<subdevice> + * hObject=0xcaf00003` from the second RM client while its main + * thread sat in the vkCreateDevice wait loop (readiness word in a + * /dev/nvidiactl notification page, never signalled). Register 0x78 + * (KERNEL_CALLBACK) and 0x7E alongside 0x79 so the completion + * wake-ups reach those events too. * * The table is append+retire: FREE retires records (valid=false) but * never shrinks events_n, so long-lived boots with many client @@ -2648,7 +4137,8 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, * first retired slot for a new registration, and log loudly when the * table is genuinely full of live entries. */ - if (class_id == 0x79 && params_size >= 24) { + if ((class_id == 0x79 || class_id == 0x7e || class_id == 0x78) && + params_size >= 24) { u32 h_src = 0, notify_index = 0, hclass = 0; u64 data64 = 0; int hc_off = -1, i; @@ -2666,16 +4156,17 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, rm->events_n, ho); else { /* - * Locate hClass (0x79) inside the params instead of - * trusting a fixed offset: NV0005_ALLOC_PARAMETERS is - * {hParentClient, hSrcResource, hClass, notifyIndex, - * data}, with hSrcResource immediately before hClass. + * Locate hClass (0x78/0x79/0x7e) inside the params + * instead of trusting a fixed offset: + * NV0005_ALLOC_PARAMETERS is {hParentClient, + * hSrcResource, hClass, notifyIndex, data}, with + * hSrcResource immediately before hClass. */ for (i = 0; i + 4 <= (int)params_size; i += 4) { u32 v; memcpy(&v, params + i, 4); - if (v == 0x79) { + if (v == 0x79 || v == 0x7e || v == 0x78) { hc_off = i; break; } @@ -2694,6 +4185,27 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, memcpy(¬ify_index, params + 12, 4); memcpy(&data64, params + 16, 8); } + /* + * r1765: the modern NVKMS/kernel events are registered + * with hSrcResource = 0 in NV0005_ALLOC_PARAMETERS; + * their real source is the ALLOC parent object (the + * event is allocated ON the channel handle: + * hParent=0x1001b for hEvent=0x1007a, verified live). + * With h_src = 0 the exact-match post path never fired + * for display completions and every flip rode the + * all-events fallback, which coalesced/delayed flips + * and let ANY other completion nudge the flip event + * (stray FLIP_OCCURRED -> the nvidia-drm dequeue + * WARN). Fall back to the alloc header's hParent when + * the params name no source resource. + * + * r1767: gated behind ev_assoc (default off) until the + * flip fix is validated end-to-end - the r1767 RCA + * exonerated the association from the r1766 soft-lock + * class, but not from the r1766 ISR crash class. + */ + if (gsp_rm_ev_assoc && h_src == 0 && hp != 0 && hp != hc) + h_src = hp; rm->events[slot].h_src = h_src; rm->events[slot].h_event = ho; rm->events[slot].h_client = hc; @@ -2727,6 +4239,39 @@ static void gsp_rm_handle_rm_alloc(struct gsp_emu_context *ctx, memcpy(reply, p, reply_len); memset(reply + 16, 0, 4); /* status = NV_OK */ + /* + * CONT.406: the direct-GL client allocates a TURING_A (0xc597) 3D + * object on its channel immediately before it would submit its first + * kick, and then frees it again. Record the exact reply the client + * receives (bounded) and remember the object so the FREE side can say + * how long it was kept. The reply must echo the 16-byte + * NV_GR_ALLOCATION_PARAMETERS {version, flags, size, caps} with + * status=NV_OK; caps is a class-dependent *output* that has no + * defined bits for c597 in any 570 header, so the echoed 0 is the + * value the golden/physical path also sends and accepts. + */ + if (class_id == 0xc597u) { + u32 rp[4] = { 0, 0, 0, 0 }; + + if (echo_len >= 4) + memcpy(&rp[0], reply + GSPRM_ALLOC_HDR_SIZE, 4); + if (echo_len >= 8) + memcpy(&rp[1], reply + GSPRM_ALLOC_HDR_SIZE + 4, 4); + if (echo_len >= 12) + memcpy(&rp[2], reply + GSPRM_ALLOC_HDR_SIZE + 8, 4); + if (echo_len >= 16) + memcpy(&rp[3], reply + GSPRM_ALLOC_HDR_SIZE + 12, 4); + + gsp_rm_grobj_note(ho, class_id); + if (rm->gr3d_alloc_log < GSPRM_GR3D_LOG_MAX) { + rm->gr3d_alloc_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: alloc 0xc597 h=0x%x status=0x%x size=0x%x reply=%08x %08x %08x %08x (parent=0x%x client=0x%x)", + ho, 0, params_size, + rp[0], rp[1], rp[2], rp[3], hp, hc); + } + } + if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_ALLOC, sequence, 0, reply, reply_len)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: GSP_RM_ALLOC reply send failed"); @@ -2901,6 +4446,217 @@ gsp_rm_handle_promote_fault_method_buffers(struct gsp_emu_context *ctx, } /* + * r519 experiment switch, independent of the P3 whitelist: forward ONLY the + * BINAPI reserved-interface query 0x20810107 to the live GSP session. The P3 + * whitelist may be off (its mixed real/emulated device answers perturb the + * boot); this one command is what the GL surface init needs answered. + */ +static bool gsp_rm_binapi07_fwd_param; +module_param_named(gsp_rm_binapi07_fwd, gsp_rm_binapi07_fwd_param, bool, 0644); +MODULE_PARM_DESC(gsp_rm_binapi07_fwd, + "Forward only the guest GL core's BINAPI 0x20810107 query to the live GSP session (default 0; diagnostic/experiment)"); + +/* + * SESSION 153-CONTINUED-17: reply word (little-endian u16) for BINAPI + * 0x20810107. The guest GL client reads reply byte[1] as a feature flag + * (glcore+0xFC74E1: setne <ctx>+0x203 after cmpb $0x0,reply[1]) and the + * flag gates method emission at glcore+0xB5B48B (see the control handler). + * 0x0101 sets both bytes; 0x0100 sets byte[1] only. Default 0 = the + * historical {0,0} reply. + */ +static uint gsp_rm_binapi07_rep; +module_param_named(binapi07_rep, gsp_rm_binapi07_rep, uint, 0644); +MODULE_PARM_DESC(binapi07_rep, + "reply word (LE u16) for BINAPI cmd 0x20810107; the GL client reads byte[1] as a feature flag (default 0)"); + +/* + * session 75: reply words for the client-pending control 0x2080a097 (the + * libnvidia-eglcore engine-readiness poll). Runtime-tunable so one boot can + * sweep reply variants: the client reads +0x204, +0x488 and +0x48c back + * (file offsets 0xecd45f..0xecd475) before its engine loop re-issues the + * control. Defaults: flag=1 (the request sets it to 0), the other words 0. + */ +static uint gsp_rm_a097_flag = 1; +module_param_named(wall2_a097_flag, gsp_rm_a097_flag, uint, 0644); +MODULE_PARM_DESC(wall2_a097_flag, "0x2080a097 reply word at params+0x204 (default 1)"); +static uint gsp_rm_a097_v488; +module_param_named(wall2_a097_v488, gsp_rm_a097_v488, uint, 0644); +MODULE_PARM_DESC(wall2_a097_v488, "0x2080a097 reply word at params+0x488 (default 0)"); +static uint gsp_rm_a097_v48c; +module_param_named(wall2_a097_v48c, gsp_rm_a097_v48c, uint, 0644); +MODULE_PARM_DESC(wall2_a097_v48c, "0x2080a097 reply word at params+0x48c (default 0)"); + +/* + * session 76: the 0x2080a097 reply's RECORD area (params+4..+0x23). The + * real host RM handler overwrites up to four {u32, u32} mode pairs there + * (per-mode engine status + a x1000 clock/percent value); the client + * (count=1) copies back the first 16 bytes into its per-engine query + * buffer, and the second word of the first pair is consumed as the + * "engine progress" word (libnvidia-eglcore d5e3e0: *(inner+4) -> + * session+0xe104, compared against per-boot config thresholds). + * Each word defaults to 0xffffffff = leave the echoed request word + * untouched; any other value replaces it, so one boot can sweep the + * record matrix without rebuilding. + */ +static uint gsp_rm_a097_count = ~0u; +module_param_named(wall2_a097_count, gsp_rm_a097_count, uint, 0644); +MODULE_PARM_DESC(wall2_a097_count, "0x2080a097 reply count word at params+0 (0xffffffff = echo)"); +static uint gsp_rm_a097_rec0 = ~0u, gsp_rm_a097_rec1 = ~0u; +static uint gsp_rm_a097_rec2 = ~0u, gsp_rm_a097_rec3 = ~0u; +static uint gsp_rm_a097_rec4 = ~0u, gsp_rm_a097_rec5 = ~0u; +static uint gsp_rm_a097_rec6 = ~0u, gsp_rm_a097_rec7 = ~0u; +module_param_named(wall2_a097_rec0, gsp_rm_a097_rec0, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec0, "0x2080a097 reply word at params+0x4 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec1, gsp_rm_a097_rec1, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec1, "0x2080a097 reply word at params+0x8 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec2, gsp_rm_a097_rec2, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec2, "0x2080a097 reply word at params+0xc (0xffffffff = echo)"); +module_param_named(wall2_a097_rec3, gsp_rm_a097_rec3, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec3, "0x2080a097 reply word at params+0x10 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec4, gsp_rm_a097_rec4, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec4, "0x2080a097 reply word at params+0x14 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec5, gsp_rm_a097_rec5, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec5, "0x2080a097 reply word at params+0x18 (0xffffffff = echo)"); +module_param_named(wall2_a097_rec6, gsp_rm_a097_rec6, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec6, "0x2080a097 reply word at params+0x1c (0xffffffff = echo)"); +module_param_named(wall2_a097_rec7, gsp_rm_a097_rec7, uint, 0644); +MODULE_PARM_DESC(wall2_a097_rec7, "0x2080a097 reply word at params+0x20 (0xffffffff = echo)"); + +/* + * m112b: the NV04_VID_HEAP_CONTROL enumeration pair 0x90960106/0x90960107 + * (legacy NVOS32 VidHeap API, class 0x9096). libnvidia-eglcore (file offset + * 0x586d80) walks three heaps: per heap it queries 0x90960106 (12 bytes: + * {u32 firstIdx, u32 lastIdx}; the client then enumerates the heap's memory + * blocks by index via 0x90960107 (56 bytes) while firstIdx+1 <= idx <= + * lastIdx). The 56-byte block entry is read at +0x00/+0x08 (two u64s), +0x28 + * (a u32 type compared against a table at eglcore 0x101b1c0) and +0x30 (a + * flag BYTE: when 0 the heap/block is skipped). The generic zero reply makes + * every heap look empty, the client concludes out-of-device-memory and + * vkAllocateMemory fails -2 WITHOUT any alloc reaching the emulation + * (dmesg-corroborated: only 0x9096* controls + a class 0xc5c0 alloc). + * Runtime-tunable so one boot can sweep reply variants (same pattern as the + * wall2_a097_* words). + */ +static uint gsp_rm_9096_lo; /* 0x90960106 firstIdx */ +static uint gsp_rm_9096_hi = 1; /* 0x90960106 lastIdx */ +module_param_named(wall2_9096_lo, gsp_rm_9096_lo, uint, 0644); +MODULE_PARM_DESC(wall2_9096_lo, "0x90960106 reply firstIdx (default 0)"); +module_param_named(wall2_9096_hi, gsp_rm_9096_hi, uint, 0644); +MODULE_PARM_DESC(wall2_9096_hi, "0x90960106 reply lastIdx (default 1)"); +static uint gsp_rm_9096_ent_addr; /* 0x90960107 entry +0x00 (u64) */ +module_param_named(wall2_9096_ent_addr, gsp_rm_9096_ent_addr, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_addr, "0x90960107 entry word at +0x00 (default 0)"); +static uint gsp_rm_9096_ent_size = 0x80000000u; /* entry +0x08 (u64) */ +module_param_named(wall2_9096_ent_size, gsp_rm_9096_ent_size, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_size, "0x90960107 entry word at +0x08 (default 0x80000000)"); +static uint gsp_rm_9096_ent_type = 3; /* entry +0x28 (u32) */ +module_param_named(wall2_9096_ent_type, gsp_rm_9096_ent_type, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_type, "0x90960107 entry word at +0x28 (default 3)"); +static uint gsp_rm_9096_ent_flag = 1; /* entry +0x30 (u8) */ + +module_param_named(wall2_9096_ent_flag, gsp_rm_9096_ent_flag, uint, 0644); +MODULE_PARM_DESC(wall2_9096_ent_flag, "0x90960107 entry flag byte at +0x30 (default 1)"); + + +/* + * The live session's oracle client/object for the 0x20810107 experiment, + * created on demand; 0 until the first query builds it. One device per module + * load, so file-statics are enough; reset by unload. + */ +static u32 gsp_rm_binapi07_be_client; +static u32 gsp_rm_binapi07_be_object; + +/* + * One GSP_RM_ALLOC (fn 103) against the live backend, built exactly like the + * rm-call ladder's helper (rpc_gsp_rm_alloc_v03_00: hClient, hParent, + * hObject, hClass, status, paramsSize, flags, reserved[4] + params). + * object_is_out selects the reply read: a ROOT alloc returns the + * server-chosen client handle in params[0] (payload +32), anything else + * echoes the client-chosen hObject at +8; status is at +16 in both cases. + */ +static int gsp_rm_binapi07_backend_alloc(u32 h_client, u32 h_parent, + u32 h_object, u32 h_class, + const void *params, u32 params_len, + bool object_is_out, + u32 *out_object, u32 *out_status) +{ + u8 *areq; + void *arsp = NULL; + u32 arsp_len = 0, st = 0xffffffff, ho = 0; + int ret; + + areq = kzalloc(32 + params_len, GFP_KERNEL); + if (!areq) + return -ENOMEM; + memcpy(areq + 0, &h_client, 4); + memcpy(areq + 4, &h_parent, 4); + memcpy(areq + 8, &h_object, 4); + memcpy(areq + 12, &h_class, 4); + memcpy(areq + 20, ¶ms_len, 4); + if (params_len) + memcpy(areq + 32, params, params_len); + + ret = gsp_phys_backend_rpc(GSPRM_RPC_FUNC_GSP_RM_ALLOC, areq, + 32 + params_len, &arsp, &arsp_len, 500); + kfree(areq); + if (!ret && arsp) { + if (arsp_len >= 20) + memcpy(&st, (u8 *)arsp + 16, 4); + if (object_is_out) { + if (arsp_len >= 36) + memcpy(&ho, (u8 *)arsp + 32, 4); + } else if (arsp_len >= 12) { + memcpy(&ho, (u8 *)arsp + 8, 4); + } + } + kfree(arsp); + if (out_object) + *out_object = ho; + if (out_status) + *out_status = st; + return ret; +} + +/* + * Build the oracle object on the live session: a class-0x2081 child of the + * session's internal subdevice, since the server dispatches the method from + * the OBJECT's class (the subdevice attempt alone answered + * NV_ERR_NOT_SUPPORTED, 0x56, and a fresh root client at runtime answered + * NV_ERR_INVALID_CLIENT, 0x23 - the ladder "adopts" the server's own + * internal client instead of creating one). A small candidate list of + * likely-free handles in that client's namespace is tried; the first + * success wins. Returns 0 with *out_object set. + */ +static int gsp_rm_binapi07_backend_setup(struct gsp_emu_context *ctx, + struct gsp_phys_backend_static_config *snap, + u32 *out_object) +{ + static const u32 cand[] = { 0xabcd2081u, 0xabcd2001u, 0xabcd2a2au, + 0xabcd5f5fu, 0xabcd1234u }; + u8 p4[4] = { 0, 0, 0, 0 }; + u32 st = 0xffffffff, obj = 0; + int i, ret; + + for (i = 0; i < (int)ARRAY_SIZE(cand); i++) { + u32 want = cand[i]; + + obj = want; + ret = gsp_rm_binapi07_backend_alloc(snap->h_internal_client, + snap->h_internal_subdevice, + want, 0x2081, p4, 4, + false, &obj, &st); + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT oracle BINAPI alloc attempt %d: hObject=0x%x ret=%d status=0x%x", + i, want, ret, st); + if (!ret && st == 0 && obj) { + *out_object = obj; + return 0; + } + } + return -EIO; +} + +/* * Bounded logging for the P3 forward counters: the first four fallbacks (so a * session that cannot forward is visible) and every 64th event. Never called * when the parameter is off. @@ -2949,12 +4705,37 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, u32 out_params_size; int ret; + bool guest_handles = false; + /* Wave 1 whitelist: compile-time side-effect-free queries only. */ switch (req->cmd) { case TU104_CTRL_BUS_GET_PCI_BAR_INFO: case TU104_CTRL_BUS_GET_INFO_V2: case TU104_CTRL_FB_GET_INFO_V2: case TU104_CTRL_INTERNAL_GMMU_GET_STATIC_INFO: + if (!gsp_phys_ctrl_forward_enabled()) + return false; + break; + case 0x20810107: /* NV2081_BINAPI reserved-interface method 7 */ + /* + * EXPERIMENT (r519): the guest GL core allocates a BINAPI + * object (class 0x2081) and sends this 2-byte query during + * 3D-surface init; our {0,0} generic reply leaves the + * driver's feature flag clear and the EGL surface init tears + * down (EGL_BAD_ALLOC at eglMakeCurrent). The guest-side + * handles (hClient 0xc1d..., a 0xbeef object) have no backend + * counterpart: allocate a real BINAPI object (class 0x2081) on + * the live session once, then send the query to it. The first + * attempt (subdevice translation) came back NV_ERR_NOT_SUPPORTED + * (0x56) - the server dispatches the method by the OBJECT's + * class, so a subdevice cannot answer a BINAPI method. The + * reply is logged either way; an error falls back to the + * emulated generic success until the semantics are known. + * Gated by its own parameter so the P3 whitelist can stay off. + */ + if (!gsp_rm_binapi07_fwd_param) + return false; + guest_handles = true; break; default: return false; @@ -2966,24 +4747,44 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, if (req->paramsSize > GSPRM_MAX_SINGLE_ELEM_DATA - GSP_RM_CTRL_HDR_SIZE) goto fallback; + /* The experiment is scoped to the observed shape: 2-byte in/out. */ + if (guest_handles && req->paramsSize != 2) + goto fallback; + /* A valid snapshot implies a READY session and the adopted handles. */ if (gsp_phys_backend_static_get(&snap)) goto fallback; + if (guest_handles && !gsp_rm_binapi07_be_object) { + u32 obj = 0; + + if (gsp_rm_binapi07_backend_setup(ctx, &snap, &obj)) + goto fallback; + /* + * The object lives in the internal client's namespace; the + * control is sent as that client (snap.h_internal_client). + */ + gsp_rm_binapi07_be_client = snap.h_internal_client; + gsp_rm_binapi07_be_object = obj; + } + /* Translate the guest's emulated internal handles; any other handle * domain stays emulated. */ - if (req->hClient != TU104_H_INTERNAL_CLIENT || - (req->hObject != TU104_H_INTERNAL_DEVICE && - req->hObject != TU104_H_INTERNAL_SUBDEVICE)) + if (!guest_handles && + (req->hClient != TU104_H_INTERNAL_CLIENT || + (req->hObject != TU104_H_INTERNAL_DEVICE && + req->hObject != TU104_H_INTERNAL_SUBDEVICE))) goto fallback; fwd_req = kmalloc(GSP_RM_CTRL_HDR_SIZE + req->paramsSize, GFP_KERNEL); if (!fwd_req) goto fallback; memset(&fwd_hdr, 0, sizeof(fwd_hdr)); - fwd_hdr.hClient = snap.h_internal_client; - fwd_hdr.hObject = req->hObject == TU104_H_INTERNAL_DEVICE ? - snap.h_internal_device : snap.h_internal_subdevice; + fwd_hdr.hClient = guest_handles ? gsp_rm_binapi07_be_client : + snap.h_internal_client; + fwd_hdr.hObject = guest_handles ? gsp_rm_binapi07_be_object : + (req->hObject == TU104_H_INTERNAL_DEVICE ? + snap.h_internal_device : snap.h_internal_subdevice); fwd_hdr.cmd = req->cmd; fwd_hdr.paramsSize = req->paramsSize; fwd_hdr.flags = req->flags; @@ -3003,6 +4804,27 @@ static bool gsp_rm_ctrl_forward(struct gsp_emu_context *ctx, } /* + * r519 experiment evidence: log the live GSP's own answer to the + * BINAPI query (status + first params bytes) before deciding. + */ + if (req->cmd == 0x20810107) { + u32 st = 0xffffffff, ps = 0xffffffff; + u8 b[8] = { 0 }; + + if (rsp && rsp_len >= GSP_RM_CTRL_HDR_SIZE) { + memcpy(&st, (u8 *)rsp + 12, 4); + memcpy(&ps, (u8 *)rsp + 16, 4); + if (ps <= 8 && ps <= rsp_len - GSP_RM_CTRL_HDR_SIZE) + memcpy(b, (u8 *)rsp + GSP_RM_CTRL_HDR_SIZE, + ps); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI-EXPERIMENT 0x20810107 forwarded to live GSP: rsp_len=%u status=0x%x outSize=%u bytes=%02x %02x %02x %02x %02x %02x %02x %02x", + rsp_len, st, ps, b[0], b[1], b[2], b[3], b[4], + b[5], b[6], b[7]); + } + + /* * The reply is rpc_gsp_rm_control_v03_00: status at +12, paramsSize * at +16, echoed params at +24 - the same 24-byte prefix the emulated * handler builds below. The RPC-level rpc_result was already checked @@ -3048,6 +4870,101 @@ fallback: return false; } +/* + * r489: a GSP_RM_CONTROL reply whose rpc record exceeds the driver's + * pRpc->maxRpcSize must be sent with the GSP's continuation-record protocol + * (nv gsp_rpc.c _rpcWriteContinuationEntries): the FIRST record carries + * exactly maxRpcSize bytes of [rpc header + ctrl header + params]; each + * following NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD record carries up to + * maxRpcSize - sizeof(hdr) bytes of the remainder. The driver's + * _issueRpcLarge consumes the first frame as min(bufSize, maxRpcSize) and + * then requires one continuation record per remaining chunk - without them + * it times out ("rpcRecvPoll timedout for fn 71 continuation record") and + * the control fails (NV_ERR_INVALID_PARAMETER), which is how the GL VASpace + * construction died on cmd 0x2080a0a4 (paramsSize 67392 > maxRpcSize). + * + * maxRpcSize is the guest driver's GSP_MSG_QUEUE_RPC_SIZE_MAX; the value is + * pinned by the driver's own arithmetic: for the 0x2080a0a4 reply + * (full_param_size = 32 + 24 + 67392 = 67448) it reported remainingSize + * 0x940 = 2368 after the first frame, i.e. maxRpcSize = 67448 - 2368 = + * 65080. A build whose constant differs shows up immediately as a + * mismatched continuation count in the guest log; adjust here. + */ +#define GSPRM_CRM_CONT_MAXRPC 65080u + +static int gsp_rm_send_rm_control(struct gsp_emu_context *ctx, u32 sequence, + const void *data, u32 data_len) +{ + u32 total = sizeof(struct gsprm_rpc_header) + data_len; + const u8 *p = data; + u32 off = 0; + int ret; + + if (total <= GSPRM_CRM_CONT_MAXRPC) + return gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, + sequence, 0, data, data_len); + + /* Frame 1: rpc.length = maxRpcSize. */ + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, + p, GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)); + if (ret) + return ret; + off = GSPRM_CRM_CONT_MAXRPC - sizeof(struct gsprm_rpc_header); + + /* Continuations: rpc.length = hdr + chunk. */ + while (off < data_len) { + u32 chunk = data_len - off; + + if (chunk > GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header)) + chunk = GSPRM_CRM_CONT_MAXRPC - + sizeof(struct gsprm_rpc_header); + ret = gsp_rm_send(ctx, GSPRM_RPC_FUNC_CONTINUATION_RECORD, + sequence, 0, p + off, chunk); + if (ret) + return ret; + off += chunk; + } + return 0; +} + +/* + * Real-GSP KGR static-data snapshots (mode-35 boot capture; see + * gsp_phys_kgr_snap_get in gsp_phys_booter.c). The physical backend asks the + * live GSP the real client's KGR query set at boot and retains the replies; + * serve the identical bytes for the same commands so the guest's GL/3D stack + * reads the device's true GR static values. Concrete consumer: the X + * driver's 3D-engine selection maps GR_GET_INFO index SM_VERSION (0x0c) + * through a table and fails with "Failed to select a 3D engine" when it + * reads back as SM_VERSION_NONE (0); the live GSP's KGR_GET_INFO reply + * carries the real value (0x75 on TU104). Returns true when the reply was + * served from the snapshot (out_params_size set); false falls through to the + * emulated handler unchanged. + */ +static bool gsp_rm_kgr_snap_serve(struct gsp_emu_context *ctx, u32 cmd, + u32 req_params_size, + u8 *out_params, u32 out_len, + u32 *out_params_size) +{ + u32 n = 0; + int ret; + + ret = gsp_phys_kgr_snap_get(cmd, out_params, out_len, &n); + if (ret || !n) + return false; + + /* Never answer with more bytes than the guest asked for. */ + if (n > req_params_size) + return false; + + *out_params_size = n; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: KGR cmd=0x%x served from the real-GSP snapshot (%u bytes)", + cmd, n); + return true; +} + static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, const u8 *elem, u32 sequence) { @@ -3060,9 +4977,35 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, u32 out_status = 0; u32 out_params_size = 0; u64 t0 = ktime_get_ns(); + bool beef; memcpy(&req, req_data, GSP_RM_CTRL_HDR_SIZE); + /* + * CONT.405: bounded step trace for the direct-GL client's RM clients + * (the "beef" device/VAS/BINAPI client and the "bfef" channel + * client). Its init is a fixed control sequence; when it gives up it + * frees the channel and retries with a fresh client, so the LAST + * cmd this trace shows before the next client's device alloc is the + * failing step. Necessary because the default branch answers an + * unhandled control with generic SUCCESS and that is invisible at + * gsp_sema_dbg=0 - which is exactly where the retry loop lived. + */ + beef = (req.hClient >= 0xc1d00010u && req.hClient <= 0xc1d0002fu) || + (req.hObject & 0xffff0000u) == 0xbeef0000u || + (req.hObject & 0xffffff00u) == 0xbfef0000u; + if (beef) { + static u32 beef_step_log; + + if (beef_step_log < 128) { + beef_step_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: beef step #%u cmd=0x%x hClient=0x%x hObject=0x%x in=%u", + beef_step_log, req.cmd, req.hClient, + req.hObject, req.paramsSize); + } + } + if (gsp_sema_dbg) nv_vgpu_info(ctx->vgpu, "vgsp-rm: GSP_RM_CONTROL cmd=0x%x hClient=0x%x hObject=0x%x paramsSize=%u flags=0x%x", req.cmd, req.hClient, req.hObject, req.paramsSize, @@ -3082,13 +5025,26 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, * return means nothing was sent and the emulated switch below runs * byte-identically to the parameter-off path. */ - if (gsp_phys_ctrl_forward_enabled() && + if ((gsp_phys_ctrl_forward_enabled() || gsp_rm_binapi07_fwd_param) && gsp_rm_ctrl_forward(ctx, &req, req_data + GSP_RM_CTRL_HDR_SIZE, reply, reply_data_len, sequence)) { kfree(reply); return; } + /* + * KGR static-data snapshots (mode-35 boot capture): the live GSP's own + * replies to the real client's KGR query set are served byte-identically + * for the same commands, so the guest's GL/3D stack sees the device's + * true GR static values. Falls through to the emulated switch when no + * snapshot exists (backend not READY, other boot mode, cmd not captured). + */ + if (gsp_rm_kgr_snap_serve(ctx, req.cmd, req.paramsSize, + reply + GSP_RM_CTRL_HDR_SIZE, + reply_data_len - GSP_RM_CTRL_HDR_SIZE, + &out_params_size)) + goto rm_control_reply; + switch (req.cmd) { case TU104_CTRL_FIFO_GET_DEVICE_INFO_TABLE: gsp_rm_handle_fifo_device_table(ctx, sequence, &req, @@ -3098,7 +5054,8 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; case 0x90f10106: /* NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES */ - gsp_rm_handle_copy_server_pdes(ctx, req_data + GSP_RM_CTRL_HDR_SIZE, + gsp_rm_handle_copy_server_pdes(ctx, req.hObject, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize); out_status = 0; out_params_size = 0; @@ -3168,6 +5125,171 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); break; + case 0xc3700104: /* NVC370_CTRL_CMD_GET_CHANNEL_INFO */ + gsp_rm_handle_display_channel_info(ctx, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x20800a5d: /* NV2080_CTRL_CMD_INTERNAL_DISPLAY_GET_ACTIVE_DISPLAY_DEVICES */ + gsp_rm_handle_display_active_devices(ctx, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case TU104_CTRL_INTERNAL_DISPLAY_CHANNEL_PUSHBUFFER: { + /* + * r1621: the kernel RM programs each display channel's push + * buffer through this internal control right after the channel + * object is allocated. It carries the guest-physical push- + * buffer address, its size and the display class/instance - the + * exact inputs the host-side EVO engine needs (see + * gsp_display_engine.c). The reply stays a verbatim echo. + */ + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + + /* + * The real struct (NV2080_CTRL_INTERNAL_DISPLAY_CHANNEL_ + * PUSHBUFFER_PARAMS) is 56 bytes: addressSpace@0, + * physicalAddr@8, limit@16, cacheSnoop@24, hclass@28, + * channelInstance@32, valid@36 (NvBool = 1 byte!), + * pbTargetAperture@40, channelPBSize@44, subDeviceId@48. + * Read valid as a byte: the 3 padding bytes after it are not + * guaranteed zero, so a u32 load can turn valid=0 (PIO + * cursor channel) into a false positive. + */ + pr_info("rm-ctl: display channel pushbuffer hObject=0x%x size=%u pa=0x%llx limit=0x%llx class=0x%x inst=%u valid=%u\n", + req.hObject, req.paramsSize, + req.paramsSize >= 24 ? *(const u64 *)(pp + 8) : 0, + req.paramsSize >= 24 ? *(const u64 *)(pp + 16) : 0, + req.paramsSize >= 32 ? *(const u32 *)(pp + 28) : 0, + req.paramsSize >= 36 ? *(const u32 *)(pp + 32) : 0, + req.paramsSize >= 37 ? pp[36] : 0); + if (req.paramsSize >= 40 && + req.paramsSize + GSP_RM_CTRL_HDR_SIZE <= + GSPRM_MAX_SINGLE_ELEM_DATA) { + u32 address_space = 0, hclass = 0, instance = 0; + u8 valid = 0; + u64 pa = 0, limit = 0; + + memcpy(&address_space, pp + 0, 4); + memcpy(&pa, pp + 8, 8); + memcpy(&limit, pp + 16, 8); + memcpy(&hclass, pp + 28, 4); + memcpy(&instance, pp + 32, 4); + valid = pp[36]; + memcpy(rp, pp, req.paramsSize); + gsp_disp_engine_channel_add(ctx->vgpu, hclass, instance, + pa, limit, address_space, + valid != 0); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + + case TU104_CTRL_INTERNAL_DISPLAY_WRITE_INST_MEM: { + /* + * The kernel RM announces the display instance-memory block + * (8 KB context-DMA hash table + context-DMA entries) and + * its location. The host engine resolves the notifier + * context-DMA handle carried by SET_NOTIFIER_DMA through + * this block; without it the notifier PA stays unknown and + * the UPDATE FINISHED word is never written (r1627: UPDATE + * armed=1 handle=0x10013 -> pa=0x0). + * + * Params: instMemPhysAddr@0 (u64), instMemSize@8 (u64), + * instMemAddrSpace@16 (u32), instMemCpuCacheAttr@20 (u32). + */ + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + + pr_info("rm-ctl: display write inst mem size=%u pa=0x%llx space=%u\n", + req.paramsSize, + req.paramsSize >= 16 ? *(const u64 *)(pp + 0) : 0, + req.paramsSize >= 24 ? *(const u32 *)(pp + 16) : 0); + if (req.paramsSize >= 24 && + req.paramsSize + GSP_RM_CTRL_HDR_SIZE <= + GSPRM_MAX_SINGLE_ELEM_DATA) { + u64 pa = 0, size = 0; + u32 space = 0; + + memcpy(&pa, pp + 0, 8); + memcpy(&size, pp + 8, 8); + memcpy(&space, pp + 16, 4); + memcpy(rp, pp, req.paramsSize); + gsp_disp_engine_inst_mem(ctx->vgpu, pa, size, space); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + + case 0x801707: /* NV0080_CTRL_CMD_FIFO_GET_ENGINE_CONTEXT_PROPERTIES */ + gsp_rm_handle_fifo_engine_ctx_props(ctx, + req_data + GSP_RM_CTRL_HDR_SIZE, req.paramsSize, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x730102: /* NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS */ + case 0x730287: /* NV0073_CTRL_CMD_SPECIFIC_GET_ALL_HEAD_MASK */ + case 0x7302ad: /* NV0073_CTRL_CMD_SPECIFIC_GET_VALID_HEAD_WINDOW_ASSIGNMENT */ + case 0x730107: /* NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED */ + case 0x730108: /* NV0073_CTRL_CMD_SYSTEM_GET_CONNECT_STATE */ + case 0x730240: /* NV0073_CTRL_CMD_SPECIFIC_GET_TYPE */ + case 0x73028a: /* NV0073_CTRL_CMD_SPECIFIC_GET_PCLK_LIMIT */ + case 0x73028b: /* NV0073_CTRL_CMD_SPECIFIC_OR_GET_INFO */ + case 0x730250: /* NV0073_CTRL_CMD_SPECIFIC_GET_CONNECTOR_DATA */ + case 0x73010b: /* NV0073_CTRL_CMD_SYSTEM_GET_HEAD_ROUTING_MAP */ + case 0x731140: /* NV0073_CTRL_CMD_DFP_GET_INFO */ + gsp_rm_handle_display_heads(ctx, req.cmd, req.paramsSize, + req_data + GSP_RM_CTRL_HDR_SIZE, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0x730245: /* NV0073_CTRL_CMD_SPECIFIC_GET_EDID_V2 */ + gsp_rm_handle_display_get_edid(ctx, req.paramsSize, + req_data + GSP_RM_CTRL_HDR_SIZE, + &out_status, + reply + GSP_RM_CTRL_HDR_SIZE, &out_params_size); + break; + + case 0xc3720101: { /* NVC372_CTRL_CMD_IS_MODE_POSSIBLE (IMP) */ + /* + * nvidia-modeset's GPU IMP check: 2048-byte params + * (NVC372_CTRL_IS_MODE_POSSIBLE_PARAMS) with the mode inputs + * and the bIsPossible answer at byte offset 1904 (NvBool is + * one byte). The caller memsets the struct before filling + * the inputs, so echoing the request and flipping + * bIsPossible is coherent. The generic zero reply answered + * "not possible", which made every EDID mode fail with + * "GPU extended capability check failed" before any mode set. + */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 1905 && + req.paramsSize <= GSPRM_MAX_SINGLE_ELEM_DATA) { + memcpy(rp, pp, req.paramsSize); + rp[1904] = 1; /* bIsPossible */ + out_params_size = req.paramsSize; + } else { + out_params_size = 0; + } + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: IS_MODE_POSSIBLE in=%u -> possible (heads=%u windows=%u)", + req.paramsSize, + req.paramsSize > 5 ? pp[4] : 0, + req.paramsSize > 5 ? pp[5] : 0); + out_status = 0; + break; + } + case TU104_CTRL_CE_GET_FAULT_METHOD_BUFFER_SIZE: gsp_rm_handle_ce_get_fault_method_buffer_size(ctx, sequence, &out_status, @@ -3306,6 +5428,32 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, "vgsp-rm: promote[%u] bufferId=%u bInitialize=%u bNonmapped=%u physAttr=0x%x pa=0x%llx va=0x%llx size=0x%llx", i, bid, e[30], e[31], attr, pa, eva, esz); + /* + * r820: register the driver's own statement + * of this kernel buffer's backing (r817 + * Xorg GL init: the page-table walk answered + * a stale zero page for the pushbuffer VA + * while this entry named the real one; a + * userspace pin scanner can never cover + * kernel-owned buffers). See + * gsp_sema_service_promote_add(). + */ + gsp_sema_service_promote_add(eva, pa, esz, + attr, bid); + /* + * SESSION 153-CONTINUED-95: keep the latest + * MAIN (0) / PATCH (2) statement per engine + * for the launch-time CTX-BIND receipt (the + * bInitialize flag is the client's "server, + * initialize this buffer" request; the + * emulation answers OK but never installs + * the buffer on the engine channel - the + * receipt makes that visible). + */ + gsp_sema_service_ctx_promote_note(engine_type, + bid, pa, eva, + esz, attr, + e[30]); } } out_status = 0; @@ -3371,6 +5519,149 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + case 0x20800afe: { /* NV2080_CTRL_CMD_INTERNAL_INIT_USER_SHARED_DATA + * + * The guest kernel driver hands the GSP-RM the physical + * address of the RUSD (RM user shared data) page it + * allocated; on real hardware the GSP registers the + * memdesc and keeps the page's sequence-guarded fields + * (clocks, BAR1, utilisation, ...) populated, and 0 + * de-initialises it. Nothing here performs that upkeep, + * and consumers treat a zeroed sequence word as "data + * invalid" and skip the fields, so the correct emulated + * behaviour is to acknowledge the registration, echo the + * request (the caller may read the address back) and log + * the address once so the page can be correlated with + * guest RAM during diagnosis. */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 8) { + u64 phys_addr; + + memcpy(&phys_addr, pp, 8); + memcpy(rp, pp, req.paramsSize); + rm->rusd_phys_addr = phys_addr; + rm->rusd_init_seen = true; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: INIT_USER_SHARED_DATA physAddr=0x%llx (RUSD registered; fields left at sequence 0 = 'data invalid', consumers skip)", + phys_addr); + out_status = 0; + out_params_size = req.paramsSize; + } else { + out_status = 0; + out_params_size = 0; + } + break; + } + + case 0x2080a097: { /* client-pending engine-context query (session 75) + * + * libnvidia-eglcore issues this via NVOS54 and then + * POLLS the reply's params words: the request sets + * +0x204 = 0, and the caller re-issues the whole + * control every ~1.8 s while its readiness check + * stays negative (observed: 32-36 polls against the + * generic all-zero reply, then the client gives up + * and vkCreateDevice returns -4). The params are an + * in/out record block: u32 count at +0, then count + * 16-byte records, then +0x204, and the client reads + * back +0, +0x204, +0x488 and +0x48c after the call + * (libnvidia-eglcore.so.570.144 file offsets + * 0xecd45f..0xecd475). The readiness wait itself + * lives in the client's engine loop + * (0xd6ef40..0xd6f0ab polling byte +0x1b038 of the + * engine object, which a91ea0 recomputes from many + * state fields), so the reply words below are + * runtime-tunable for a single-cycle sweep without + * rebuilding: the wall2_a097_* module parameters. + * + * Reply: verbatim echo (the request records are the + * client's own query data) with the tunable words + * applied. Defaults keep the completion-style flag. */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 0x490 && + req.paramsSize + GSP_RM_CTRL_HDR_SIZE <= + GSPRM_MAX_SINGLE_ELEM_DATA) { + u32 count = 0, flag; + static const u32 *const recp[8] = { + &gsp_rm_a097_rec0, &gsp_rm_a097_rec1, + &gsp_rm_a097_rec2, &gsp_rm_a097_rec3, + &gsp_rm_a097_rec4, &gsp_rm_a097_rec5, + &gsp_rm_a097_rec6, &gsp_rm_a097_rec7, + }; + int i; + + memcpy(rp, pp, req.paramsSize); + memcpy(&count, pp, 4); + memcpy(&flag, pp + 0x204, 4); + if (gsp_rm_a097_count != ~0u) + *(u32 *)(rp + 0) = gsp_rm_a097_count; + for (i = 0; i < 8; i++) { + if (*recp[i] != ~0u) + *(u32 *)(rp + 4 + i * 4) = *recp[i]; + } + *(u32 *)(rp + 0x204) = gsp_rm_a097_flag; + *(u32 *)(rp + 0x488) = gsp_rm_a097_v488; + *(u32 *)(rp + 0x48c) = gsp_rm_a097_v48c; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x2080a097: echoed %u bytes (count=%u, req flag=%u) -> flag=%u v488=0x%x v48c=0x%x", + req.paramsSize, count, flag, + gsp_rm_a097_flag, gsp_rm_a097_v488, + gsp_rm_a097_v48c); + out_status = 0; + out_params_size = req.paramsSize; + } else { + out_status = 0; + out_params_size = 0; + } + break; + } + + case 0x90960106: { /* NV04_VID_HEAP_CONTROL block-index range (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 12) { + memcpy(rp, pp, req.paramsSize); + *(u32 *)(rp + 0) = gsp_rm_9096_lo; + *(u32 *)(rp + 4) = gsp_rm_9096_hi; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960106 -> first=%u last=%u", + gsp_rm_9096_lo, gsp_rm_9096_hi); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + + case 0x90960107: { /* NV04_VID_HEAP_CONTROL block entry by index (m112b) */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 56) { + memcpy(rp, pp, req.paramsSize); + *(u64 *)(rp + 0x00) = (u64)gsp_rm_9096_ent_addr; + *(u64 *)(rp + 0x08) = (u64)gsp_rm_9096_ent_size; + *(u32 *)(rp + 0x28) = gsp_rm_9096_ent_type; + *(u8 *)(rp + 0x30) = (u8)gsp_rm_9096_ent_flag; + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: ctrl 0x90960107 -> addr=0x%x size=0x%x type=%u flag=%u", + gsp_rm_9096_ent_addr, + gsp_rm_9096_ent_size, + gsp_rm_9096_ent_type, + gsp_rm_9096_ent_flag); + } + out_status = 0; + out_params_size = req.paramsSize; + break; + } + default: /* Unknown control command - return success with zeroed params. * Do NOT echo the request: pure queries pass uninitialized @@ -3393,6 +5684,52 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, out_params_size = req.paramsSize; break; + case 0x20810107: { /* NV2081_BINAPI reserved-interface method 7 */ + /* + * SESSION 153-CONTINUED-17: the guest GL client + * (libnvidia-glcore 570.144, file offset 0xFC74A6) sends this + * 2-byte query right after allocating its BINAPI object and + * CONSUMES REPLY BYTE [1] AS A FEATURE FLAG: + * + * glcore+0xFC74D9: cmpb $0x0, 0x131(%rsp) + * glcore+0xFC74E1: setne 0x203(%r14) ; flag = replies[1] + * + * and the flag gates 3D method emission into the command + * buffer: + * + * glcore+0xB5B48B: mov 0x1248(%r14),%rax + * cmpb $0x0,0x203(%rax) + * je <skip> + * glcore+0xB5B49B: movabs $0xf020010e66,%rdi + * mov %rdi,(0x80(%r14)) ; emit + advance + * + * With the historical {0,0} generic reply the flag stays 0 and + * the client's submission pipeline never activates (no GL + * channel, no pushes, black window). binapi07_rep selects the + * reply word (LE u16) at runtime so one boot can test the + * feature without a rebuild: 0x0101 = both bytes set, + * 0x0100 = byte[1] only. Default 0 reproduces {0,0}. + */ + u8 *rp = reply + GSP_RM_CTRL_HDR_SIZE; + const u8 *pp = req_data + GSP_RM_CTRL_HDR_SIZE; + + if (req.paramsSize >= 2 && req.paramsSize <= 16) { + memcpy(rp, pp, req.paramsSize); + rp[0] = gsp_rm_binapi07_rep & 0xff; + rp[1] = (gsp_rm_binapi07_rep >> 8) & 0xff; + out_params_size = req.paramsSize; + } else { + out_params_size = 0; + } + if (gsp_sema_dbg) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: BINAPI 0x20810107 reply in=%u rep=0x%04x bytes=%02x %02x", + req.paramsSize, gsp_rm_binapi07_rep, + rp[0], rp[1]); + out_status = 0; + break; + } + case TU104_CTRL_I2C_TRANSACTION: /* I2C transactions have variable-length FINN-serialized * params: an all-zero reply fails the driver's DeserializeUp @@ -3409,6 +5746,27 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, break; } + /* + * r487/r489 THE GL/VULKAN VASPACE WALL (Xorg/EGL/GZDOOM, every + * session to date): the reply must follow the GSP's own large-reply + * framing. r487 clamped the echo to fit one 64 KB record; the guest + * then refused the record ("Incorrect message length 65528") and, + * when it fit, waited for CONTINUATION records that never came + * ("rpcRecvPoll timedout for fn 71 continuation record + * (remainingSize=0x940)") - rpc.c's _issueRpcLarge consumes the + * first frame up to pRpc->maxRpcSize (65080) and then requires + * NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD frames for the rest + * (the GSP's own _rpcWriteContinuationEntries). The size is left + * unclamped here; gsp_rm_send_rm_control() frames it per that + * protocol. Root evidence: cmd 0x2080a0a4 (PERF_VPSTATES_INFO, + * paramsSize 67392, FINN-serialized) sent during Xorg VASpace + * construction; its failure made vmmCreateVaspace return ffff and + * killed GL - and, after the length was fixed, the 3D-engine + * selection ("Failed to select a 3D engine") until the framing was + * correct. + */ + +rm_control_reply: /* Build the reply header */ memcpy(reply + 0, &req.hClient, 4); memcpy(reply + 4, &req.hObject, 4); @@ -3426,10 +5784,25 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, req.cmd, req.hClient, req.hObject, req.paramsSize, out_params_size, out_status, dt_us); + /* + * CONT.405: close the beef step row with its outcome so the + * last completed step before the retry is unambiguous. + */ + if (beef) { + static u32 beef_done_log; + + if (beef_done_log < 128) { + beef_done_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: beef done cmd=0x%x -> status=0x%x out=%u", + req.cmd, out_status, + out_params_size); + } + } } - if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_GSP_RM_CONTROL, sequence, 0, - reply, GSP_RM_CTRL_HDR_SIZE + out_params_size)) + if (gsp_rm_send_rm_control(ctx, sequence, reply, + GSP_RM_CTRL_HDR_SIZE + out_params_size)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: RM_CONTROL reply send failed cmd=0x%x", req.cmd); else @@ -3439,6 +5812,87 @@ static void gsp_rm_handle_rm_control(struct gsp_emu_context *ctx, } /* + * r510gl: DUP_OBJECT (function 21) - NVOS55_PARAMETERS echo with status=NV_OK. + * + * The guest driver dups an object into a second client at EGL/GLX + * MakeCurrent time (memory and VASpace objects: the GL context's client + * shares the device's allocations), and the historical generic-SUCCESS + * default replied with an EMPTY payload - the driver's reply must carry the + * params struct back (rpc_dup_object_v03_00 = NVOS55_PARAMETERS + * {hClient, hParent, hObject, hClientSrc, hObjectSrc, flags, status}), so + * an empty reply failed the call and the driver tore the whole EGL session + * down (guest witness: 'RPC UNKNOWN(21) seq=0 len=60 -> generic SUCCESS', + * immediately followed by the session's FREE cascade and + * eglMakeCurrent -> EGL_BAD_ALLOC (0x3003), 2026-10-23 r510). + * + * The emulation does not model per-object state for these classes (controls + * and allocs are answered from the request), so the echo with NV_OK is the + * complete answer; the dup'd handle is used by later calls exactly like the + * source handle. + */ +static void gsp_rm_handle_dup_object(struct gsp_emu_context *ctx, + const u8 *elem, u32 sequence) +{ + struct gsp_rm_emu *rm = ctx->rm_emu; + const struct gsprm_rpc_header *hdr = + (const struct gsprm_rpc_header *)(elem + GSPRM_ELEM_HDR_SIZE); + const u8 *req = elem + GSPRM_ELEM_HDR_SIZE + sizeof(*hdr); + u8 params[28]; /* NVOS55_PARAMETERS_v03_00 */ + u32 hDstClient = 0, hDstObject = 0, hSrcClient = 0, hSrcObject = 0; + + if (hdr->length >= sizeof(*hdr) + sizeof(params)) + memcpy(params, req, sizeof(params)); + else + memset(params, 0, sizeof(params)); + + memcpy(&hDstClient, params + 0, 4); + memcpy(&hDstObject, params + 8, 4); + memcpy(&hSrcClient, params + 12, 4); + memcpy(&hSrcObject, params + 16, 4); + memset(params + 24, 0, 4); /* status = NV_OK */ + + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT hDstClient=0x%x hDstObject=0x%x hSrcClient=0x%x hSrcObject=0x%x seq=%u -> NV_OK echo", + hDstClient, hDstObject, hSrcClient, hSrcObject, sequence); + + /* + * CONT.407: when the duplicated object is a published VA space, the + * destination client's new handle names the SAME page directory (the + * GL client dups X's VAS 0x10005 as 0xbfef0001 and allocates its GR + * channel with that hVASpace). Register the alias so the channel- + * scoped lookup resolves the gpfifo VA through the client's own + * directory instead of the newest-first global walk (r1702/r1703 + * answered 0x1bcfff0/ap=2 because 0xbfef0001 was never known). + * Non-VAS dups simply return false and stay a pure handle echo. + */ + if (gsp_sema_service_vaspace_alias(hDstObject, hSrcObject)) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT VAS alias h=0x%x -> src h=0x%x (gpfifo VA now resolves through the source directory)\n", + hDstObject, hSrcObject); + + /* + * CONT.438: a duplicated memory object keeps the same physical base + * under the destination client's handle (the client may pass the dup + * as hUserdMemory). + */ + if (gsp_rm_mem_object_alias(rm, hDstClient, hDstObject, + hSrcClient, hSrcObject)) + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: DUP_OBJECT mem-object alias h=0x%x (client 0x%x) -> src h=0x%x (client 0x%x)\n", + hDstObject, hDstClient, hSrcObject, hSrcClient); + + /* Small delay before the reply: same reply-matching race mitigation as + * FREE/UNLOADING. */ + udelay(2000); + + if (gsp_rm_send(ctx, GSPRM_RPC_FUNC_DUP_OBJECT, sequence, 0, + params, sizeof(params))) + nv_vgpu_err(ctx->vgpu, "vgsp-rm: DUP_OBJECT reply send failed"); + else + rm->rpcs_replied++; +} + +/* * GSP_RM_FREE (function 10) - NVOS00_PARAMETERS echo with status=NV_OK. * * rpc_gsp_free: { hRoot, hObjectParent, hObjectOld, status }. Echo the @@ -3466,6 +5920,7 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem, memcpy(&hObjOld, params + 8, 4); memset(params + 12, 0, 4); /* status = NV_OK */ + gsp_rm_mem_object_retire(rm, hRoot, hObjOld); gsp_sema_service_channel_del(ctx->vgpu, hObjOld); /* @@ -3499,6 +5954,24 @@ static void gsp_rm_handle_free(struct gsp_emu_context *ctx, const u8 *elem, "vgsp-rm: retired %u event record(s) on FREE hObjOld=0x%x hRoot=0x%x", removed, hObjOld, hRoot); + /* CONT.406: if this is a GR object we noted at alloc time, name the + * class and the keep time. A ~0 ms keep means the alloc reply (or the + * client's params copy-out) was rejected; a longer keep means the + * client proceeded and cleaned up after a later step failed. */ + { + u32 gcls = 0; + unsigned long gat = 0; + + if (gsp_rm_grobj_retire(hObjOld, &gcls, &gat) && + rm->gr3d_free_log < GSPRM_GR3D_LOG_MAX) { + rm->gr3d_free_log++; + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: gr-obj free h=0x%x class=0x%x after %ums (client kept it)", + hObjOld, gcls, + jiffies_to_msecs(jiffies - gat)); + } + } + if (gsp_sema_dbg) nv_vgpu_info(ctx->vgpu, "vgsp-rm: FREE hObjOld=0x%x seq=%u cmd_read=%u stat_write=%u", hObjOld, sequence, rm->cmd_read, rm->stat_write); @@ -3599,6 +6072,8 @@ static void gsp_rm_handle_set_page_directory(struct gsp_emu_context *ctx, desc.pdb = phys; desc.num_entries = num_entries; desc.b_vidmem = (flags & 0x3) == 0; + /* CONT.400: RM's own SET_PAGE_DIRECTORY is authoritative. */ + desc.verified = true; gsp_sema_service_vaspace_add(&desc); /* Show the first entries of the directory: they are the ground @@ -3685,14 +6160,18 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) return; case GSPRM_RPC_FUNC_CONTINUATION_RECORD: - /* CONTINUATION_RECORD: driver expects a reply. Send SUCCESS. */ - nv_vgpu_info(ctx->vgpu, "vgsp-rm: async RPC %s(%u) seq=%u len=%u -> SUCCESS reply", - gsp_rm_rpc_name(function), function, sequence, - hdr->length); - if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) - nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); - else - rm->rpcs_replied++; + /* + * r489: a CONTINUATION_RECORD from the driver is ONE-WAY data + * - the tail of a multi-frame request (its _issueRpcLarge + * sends frame 1 then continuation records). The old reply + * here was a guess ("driver expects a reply"); it injected an + * extra fn-71 record into the reply stream that the driver + * drained as an event ("Unexpected RPC event from GPU0: 0x47") + * and it corrupted the framing of our own large replies + * (which use fn-71 continuations the other way). Consume + * silently: the driver waits only for the reply to frame 1. + */ + rm->rpcs_replied++; return; case GSPRM_RPC_FUNC_SET_GUEST_SYSTEM_INFO: @@ -3715,6 +6194,10 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) gsp_rm_handle_free(ctx, elem, sequence); return; + case GSPRM_RPC_FUNC_DUP_OBJECT: + gsp_rm_handle_dup_object(ctx, elem, sequence); + return; + case GSPRM_RPC_FUNC_SET_PAGE_DIRECTORY: gsp_rm_handle_set_page_directory(ctx, elem, sequence, hdr->length); return; @@ -3728,6 +6211,23 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) * For vGPU, we just acknowledge success. */ nv_vgpu_info(ctx->vgpu, "vgsp-rm: RPC UPDATE_BAR_PDE(%u) seq=%u len=%u -> SUCCESS", function, sequence, hdr->length); + { + const u8 *pl = (const u8 *)hdr + sizeof(*hdr); + u32 n = min_t(u32, hdr->length, 64); + char dump[256]; + int o = 0; + u32 i; + + for (i = 0; i < n; i += 4) { + u32 v = 0; + + memcpy(&v, pl + i, 4); + o += snprintf(dump + o, sizeof(dump) - o, + "%08x ", v); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: bar-pde payload: %s", dump); + } if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: UPDATE_BAR_PDE reply send failed"); else @@ -3756,6 +6256,31 @@ static void gsp_rm_handle_element(struct gsp_emu_context *ctx, const u8 *elem) nv_vgpu_info(ctx->vgpu, "vgsp-rm: RPC %s(%u) seq=%u len=%u -> generic SUCCESS reply (unknown #%llu)", gsp_rm_rpc_name(function), function, sequence, hdr->length, rm->unknown_rpcs); + /* + * Bounded payload dump for the first unknown RPCs: the vGPU + * guest maps its client VA spaces through some RPC; the + * mapping data (va pages / PAs / memdesc handles) must be + * visible here to be captured. Never echo it back (pure + * queries would eat their own stack garbage). + */ + if (rm->unknown_rpcs <= 96 && hdr->length) { + const u8 *pl = (const u8 *)hdr + sizeof(*hdr); + u32 n = min_t(u32, hdr->length, 64); + char dump[256]; + int o = 0; + u32 i; + + for (i = 0; i < n; i += 4) { + u32 v = 0; + + memcpy(&v, pl + i, 4); + o += snprintf(dump + o, sizeof(dump) - o, + "%08x ", v); + } + nv_vgpu_info(ctx->vgpu, + "vgsp-rm: unknown fn=%u seq=%u len=%u payload: %s", + function, sequence, hdr->length, dump); + } if (gsp_rm_send(ctx, function, sequence, 0, NULL, 0)) nv_vgpu_err(ctx->vgpu, "vgsp-rm: reply send failed fn=%u", function); else diff --git a/kernel/gsp_rm_emulation.h b/kernel/gsp_rm_emulation.h index 96ea16a..17154b7 100644 --- a/kernel/gsp_rm_emulation.h +++ b/kernel/gsp_rm_emulation.h @@ -45,10 +45,40 @@ struct gsp_emu_context; #define GSPRM_RPC_FUNC_GSP_RM_CONTROL 76 #define GSPRM_RPC_FUNC_GSP_RM_ALLOC 103 #define GSPRM_RPC_FUNC_FREE 10 +#define GSPRM_RPC_FUNC_DUP_OBJECT 21 #define GSPRM_RPC_FUNC_UNLOADING_GUEST_DRIVER 47 #define GSPRM_RPC_FUNC_UPDATE_BAR_PDE 70 #define GSPRM_RPC_EVENT_POST_EVENT 0x1003 #define GSPRM_MAX_EVENTS 128 +/* + * CONT.438: bounded client memory-object table. GSP-client kernel channels + * pass a client-allocated USERD as hUserdMemory[subdev] (NV_CHANNEL_ALLOC_PARAMS + * @32), not as the userdMem memdesc (@168), so the channel alloc alone cannot + * name the USERD PA. Remember the memory-object ALLOCs by (hClient, hObject) + * and resolve the channel's handle later. Sized well above the handful of + * USERD objects live at once; retired slots are reused. + */ +#define GSPRM_MAX_MEM_OBJECTS 128 + +/* Address-space tag for a remembered memory object (NV_ADDR_* values). */ +enum gsp_rm_mem_as { + GSPRM_MEM_AS_UNKNOWN = 0, + GSPRM_MEM_AS_SYSMEM = 1, /* NV_ADDR_SYSMEM: base is a guest PA */ + GSPRM_MEM_AS_VIDMEM = 2, /* NV_ADDR_FBMEM: base is an FB offset */ + GSPRM_MEM_AS_VIRTUAL = 3, /* VA range only; never a physical base */ +}; + +struct gsp_rm_mem_object { + u32 h_object; + u32 h_client; + u32 h_parent; + u32 class_id; + u64 base; /* PA (sysmem), FB offset (vidmem), or VA */ + u64 size; + u8 address_space; /* enum gsp_rm_mem_as */ + bool valid; +}; + /* Max event records written between two replies (status-ring safety). */ #define GSPRM_EVENT_BURST_MAX 24 #define GSPRM_RPC_EVENT_GSP_INIT_DONE 0x1001 @@ -175,6 +205,8 @@ struct gsp_rm_emu { u64 unknown_rpcs; u32 alloc_dumps; /* raw GSP_RM_ALLOC dumps emitted */ u32 alloc_param_dumps; /* alloc param dumps emitted */ + u32 gr3d_alloc_log; /* CONT.406 c597 alloc-reply witness lines */ + u32 gr3d_free_log; /* CONT.406 c597 free-after witness lines */ u32 ctrl_param_dumps; /* control param dumps emitted */ u64 static_live; /* static-info replies carrying live identity */ u64 static_fallback; /* overlay requested, no snapshot served */ @@ -183,6 +215,10 @@ struct gsp_rm_emu { u64 ctrl_fwd_fallback; /* forward attempt used emulation */ u64 ctrl_fwd_timeout; /* fallbacks caused by a backend timeout */ + /* NV2080_CTRL_CMD_INTERNAL_INIT_USER_SHARED_DATA bookkeeping */ + u64 rusd_phys_addr; /* last registered RUSD phys address */ + bool rusd_init_seen; /* a registration was acknowledged */ + /* * NV01_EVENT objects (class 0x5, allocated via class 0x79 in the * guest ABI). Each carries {hSrcResource, notifyIndex, data}; the @@ -204,12 +240,58 @@ struct gsp_rm_emu { u64 events_dropped; u32 event_burst; /* event records written in the current window */ unsigned long event_burst_at; /* m171: burst window start (time-based reset) */ + + /* + * CONT.438: client memory objects observed through GSP_RM_ALLOC, + * keyed by (hClient, hObject), used to turn a channel's + * hUserdMemory[subdev] into a physical USERD base. All alloc/free + * handling is serialized on the emulation thread, so no lock. + */ + struct gsp_rm_mem_object mem_objects[GSPRM_MAX_MEM_OBJECTS]; + u32 mem_objects_n; + u32 mem_objects_next; /* eviction cursor when full */ + u32 mem_object_logs; /* bounded mem-object witness budget */ + u32 mem_object_evictions; /* bounded diagnostic counter */ }; +/* + * r576 DIAG: per-boot witness budget for GSP event-post payloads (default 0 + * = off). Nonzero logs every EVENT_POST_EVENT record actually sent (slot, + * hClient, hEvent, notifyIndex, data, send rc) and the sentinel case of a + * post_all_events call that found nothing postable - the evidence needed + * when a client parks waiting for a completion wake-up that never arrives. + */ +extern unsigned gsp_rm_ev_log_max; + +/* + * r577 experiment knobs (runtime-writable; default 0 = legacy). See the + * definitions in gsp_rm_emulation.c: event record sequence, bNotifyList, + * hClient override - the A/B space for the guest-RM event drop. + */ +extern unsigned gsp_rm_ev_seq; +extern unsigned gsp_rm_ev_nlist; +extern unsigned gsp_rm_ev_cl; +extern unsigned gsp_rm_ev_func; + int gsp_rm_emu_init(struct nv_vgpu_device *vgpu, struct gsp_emu_context *ctx); void gsp_rm_emu_fini(struct gsp_emu_context *ctx); int gsp_rm_post_channel_event(struct nv_vgpu_device *vgpu, u32 h_src); +int gsp_rm_post_channel_event_force(struct nv_vgpu_device *vgpu, u32 h_src); +/* + * r1765 strict variants: post only to events registered on h_src; a miss + * returns -ENOENT instead of running the all-events fallback (the display + * engine's retry pump re-posts a refused completion). + */ +int gsp_rm_post_channel_event_strict(struct nv_vgpu_device *vgpu, u32 h_src); +int gsp_rm_post_channel_event_force_strict(struct nv_vgpu_device *vgpu, + u32 h_src); int gsp_rm_force_post(struct nv_vgpu_device *vgpu, u32 idx); +/* + * r623: deliver a synthetic OS_ERROR_LOG (0x1006) record to the guest's RM + * queue (the engine-fault report a real GSP sends). Debug/experiment only. + */ +int gsp_rm_send_os_error_log(struct nv_vgpu_device *vgpu, u32 except_type, + u32 runlist_id, u32 chid, const char *engine); int gsp_rm_post_all_events(struct nv_vgpu_device *vgpu); int gsp_rm_dump_state(struct nv_vgpu_device *vgpu); int gsp_rm_scan_guest_ram(struct nv_vgpu_device *vgpu, u64 needle, diff --git a/kernel/gsp_rm_tu104.h b/kernel/gsp_rm_tu104.h index b5ee4d7..815da3b 100644 --- a/kernel/gsp_rm_tu104.h +++ b/kernel/gsp_rm_tu104.h @@ -735,6 +735,8 @@ static const struct tu104_internal_device_info tu104_internal_device_table[] = { #define TU104_CTRL_BUS_GET_PCI_BAR_INFO 0x20801803 #define TU104_CTRL_INTERNAL_DISPLAY_GET_STATIC_INFO 0x20800a01 #define TU104_CTRL_INTERNAL_DISPLAY_GET_IP_VERSION 0x20800a4b +#define TU104_CTRL_INTERNAL_DISPLAY_CHANNEL_PUSHBUFFER 0x20800a58 +#define TU104_CTRL_INTERNAL_DISPLAY_WRITE_INST_MEM 0x20800a49 #define TU104_CTRL_I2C_TRANSACTION 0x402c0105 #define TU104_CTRL_CE_GET_FAULT_METHOD_BUFFER_SIZE 0x20802a08 #define TU104_CTRL_CE_GET_PHYSICAL_CAPS 0x20802a07 @@ -754,7 +756,18 @@ static const struct tu104_internal_device_info tu104_internal_device_table[] = { TU104_CE_CAPS0_SYSMEM | \ TU104_CE_CAPS0_P2P | \ TU104_CE_CAPS0_BL_GT_64K) -#define TU104_CE_CAPS1 0x00 +/* + * Caps byte 1: the 2D/3D copy path requires a CE that supports the + * blocklinear access modes (hardware bug 200288104): the X driver's + * nvCEChooseEngine skips every CE without CE_SUPPORTS_NONPIPELINED_BL + * (byte1 0x01; and PIPELINED_BL 0x02 for the preferred pass) and then + * fails with "Failed to find a suitable copy object" - the observed wall + * after the DMA pushbuffer test started passing. Turing's CEs are fully + * capable (pipelined + non-pipelined BL + pitchlinear): byte1 = 0x03. + * (Reference: ctrl2080ce.h NV2080_CTRL_CE_CAPS_CE_SUPPORTS_*_BL 1:0x01/ + * 1:0x02.) + */ +#define TU104_CE_CAPS1 0x03 /* ENGINE_TYPE_COPY0 = 0x9, COPY(i) = COPY0 + i (cl2080_notification.h) */ #define TU104_ENGINE_TYPE_COPY0 0x00000009 @@ -934,6 +947,17 @@ struct tu104_bus_get_info_v2_params { #define TU104_BUS_INFO_IDX_PCIE_DOWNSTREAM_LINK_CTRL 0x0a #define TU104_BUS_INFO_IDX_PCIE_GEN_INFO 0x2d #define TU104_BUS_INFO_IDX_SYSMEM_CONNECTION_TYPE 0x33 +/* + * SESSION 155 CONT 26: GPU GART size / DMA flags. The GL client's buffer + * allocator reads GPU_GART_SIZE and the GPUGART/CTXDMA capability flags + * from BUS_GET_INFO_V2 at context init; a zero reply makes gpuGartSize=0, + * the heap limits 0, and every glBufferData(data)/glMapBufferRange fail + * GL_OUT_OF_MEMORY (the GZDoom GL crash). + */ +#define TU104_BUS_INFO_IDX_GPU_GART_SIZE 0x0d +#define TU104_BUS_INFO_IDX_GPU_GART_SIZE_ALT 0x32 +#define TU104_BUS_INFO_IDX_GPU_GART_DMA_FLAGS_0 0x0b +#define TU104_BUS_INFO_IDX_GPU_GART_DMA_FLAGS_1 0x0c #define TU104_BUS_INFO_TYPE_PCI_EXPRESS 0x03 #define TU104_BUS_INFO_SYSMEM_CONN_PCIE 0x00 @@ -962,7 +986,11 @@ struct tu104_gr_static_info { /* infoList indices we populate (ctrl0080gr.h) */ #define TU104_GR_INFO_IDX_SHADER_PIPE_COUNT 0x07 +#define TU104_GR_INFO_IDX_THREAD_STACK_SCALING_FACTOR 0x08 #define TU104_GR_INFO_IDX_SHADER_PIPE_SUB_COUNT 0x09 +#define TU104_GR_INFO_IDX_SM_VERSION 0x0c +#define TU104_GR_INFO_IDX_MAX_WARPS_PER_SM 0x0d +#define TU104_GR_INFO_IDX_MAX_THREADS_PER_WARP 0x0e #define TU104_GR_INFO_IDX_LITTER_NUM_GPCS 0x14 #define TU104_GR_INFO_IDX_LITTER_NUM_TPC_PER_GPC 0x17 #define TU104_GR_INFO_IDX_GPU_CORE_COUNT 0x1d @@ -987,6 +1015,20 @@ struct tu104_gr_static_info { /* TU104 max subcontexts (NV_PGRAPH_PRI_FE_CHIP_DEF_INFO_MAX_VEID_COUNT_INIT) */ #define TU104_MAX_SUBCONTEXT_COUNT 64 +/* + * GR identity values the nvidia-3d library reads through GR_GET_INFO. + * SM_VERSION must be NV2080_CTRL_GR_INFO_SM_VERSION_7_5 (0x705): the 3D + * device allocation maps the SM version through a table and returns FALSE + * when it reads SM_VERSION_NONE (0). On the guest that surfaces as + * nvHsAllocDevice -> nvHs3dAllocDevice failing silently, and nvidia-drm + * reports "Failed to allocate NvKmsKapiDevice" (r1626 gdb: after_hs + * rax=0x0, hs_hs3d_fail_block). Warps/threads are the TU104 limits. + */ +#define TU104_GR_INFO_SM_VERSION_7_5 0x00000705 +#define TU104_MAX_WARPS_PER_SM 64 +#define TU104_MAX_THREADS_PER_WARP 32 +#define TU104_THREAD_STACK_SCALING_FACTOR 1 + /* NV2080_CTRL_INTERNAL_STATIC_GR_FLOORSWEEPING_MASKS (ctrl2080internal.h) * + STATIC_KGR_GET_FLOORSWEEPING_MASKS cmd = 0x20800a26. * Driver computes numGpcs from floorsweepingMasks.gpcMask - zero kills diff --git a/kernel/gsp_sema_service.c b/kernel/gsp_sema_service.c index b90c948..96e135b 100644 --- a/kernel/gsp_sema_service.c +++ b/kernel/gsp_sema_service.c @@ -44,11 +44,16 @@ #include <linux/workqueue.h> #include <linux/mm.h> #include <linux/vmalloc.h> +#include <linux/debugfs.h> +#include <linux/seq_file.h> +#include <linux/io.h> #include <asm/unaligned.h> #include "nvidia-vgpu-vfio.h" +#include "bar_emulation.h" #include "gsp_sema_service.h" #include "gsp_rm_emulation.h" +#include "gsp_display_engine.h" #include "gsp_phys_booter.h" #include "gsp_phys_gpu.h" #include "gsp_cuda_bridge.h" @@ -80,19 +85,49 @@ static inline u64 svc_fb_shadow_size(const struct nv_vgpu_device *vgpu) * hole), so the top of RAM is around 6 GB, not 4 GB. Anything below this is * RAM; the emulated vidmem lives in its own aperture and must not be confused * with it. - */ -#define SEMA_PA_MAX 0x180000000ULL + * + * r933: the 8 GiB GL guest's top of RAM is 0x280000000, and the GL client's + * kick/push/ring pages were measured in the 6-8 GiB band (kick gpa + * 0x20ad9b000 > the old 0x180000000 6 GiB cap), so every walk leaf there was + * rejected by the aperture bound (walk-probe rc=-2 pa=0x229c1a000) and only + * per-session pin injection could read them. The compile-time constant sizes + * the fixed tables (hot-page bitmap); the live acceptance bound is + * svc_pa_max(), driven by the sema_gr_scan_top module parameter so a cycle on + * a smaller guest shrinks both together. + */ +#define SEMA_PA_MAX 0x280000000ULL #define SEMA_SCAN_PAGES_MAX 0x20000 /* 512 MB worth of pages per pass */ /* Plausible GPU VA window for the client's system-memory objects (Turing). */ #define SEMA_VA_LO 0x100000000ULL #define SEMA_VA_HI 0x2000000000ULL +/* + * session 79: the GR/GL ucode init releases target the RM's LOW aperture + * (the same 0x4000000-0x4ffffff VA family as the push/ring windows): the + * live GL init push carries `2005c017 04923060 ... 02000001` (0x5c-form, + * va=0x4923060, payload 1, RELEASE) and `200426c0 00000000 04b19b10 + * 00000001 10000000` (REPORT_SEM group, va=0x4b19b10, payload 1, + * RELEASE) - both rejected by the sysmem-only range check, so the GR + * completion never applied and the NvOgl engine-init wait spun forever. + * Accept the low aperture for release decode (the pushes are only + * decoded from trusted ring reads, so the false-positive risk the + * sysmem bound guarded against does not apply here). + */ +#define SVC_LOW_REL_VA_LO 0x4000000ULL + /* Canonical kernel virtual address range (x86_64). */ #define KVA_LO 0xffff800000000000ULL #define KVA_HI 0xffffc00000000000ULL /* + * r534: guest-read granularity for the full-RAM discovery scan + * (svc_scan_pass). Hoisted here because the scan is defined before the + * push-apply window that used to own the define. + */ +#define SVC_SCAN_CHUNK (64 * 1024) /* 16 pages per guest read */ + +/* * INC_METHOD header for the CE semaphore group. Observed value 0x20038090: * bits [31:28] opcode/const, [23:16] count = 3, lower bits encode * (address >> 2) == 0x240 plus subchannel flags (0x8000 in the observed @@ -160,6 +195,121 @@ struct sema_map_entry { #define SVC_USERD_GP_GET 0x88 #define SVC_USERD_GP_PUT 0x8c +/* + * r496b: guest-RAM ring content scan (see gr_ram_cursor in struct + * svc_channel) and the VRAM USERD shape scan (svc_chan_userd). One scan + * slice per service pass so the service thread stays responsive. + */ +#define SVC_GR_RAM_SLICE (512ULL << 20) /* guest RAM per pass */ +/* + * SESSION 149 R15 (M1.1b): 24 slices cover two full sweeps of the 6 GB + * guest before the boot-time budget retires; the put-triggered keep-alive + * in the pump grants up to SVC_GR_RAM_REFRESH_MAX further budgets. + */ +#define SVC_GR_RAM_PASS_MAX 24 /* slices before a budget retires */ +#define SVC_GR_RAM_REFRESH_MAX 3 /* keep-alive budget refreshes */ +#define SVC_GR_RAM_REFRESH_MS 10000 /* minimum spacing between them */ +#define SVC_GR_RAM_WIN (2ULL << 20) /* allocation window pinned */ +/* + * r979 (SESSION 153-CONTINUED-21): the kick family's per-frame submit + * record pushbuffer VA. Constant across every measured boot (r908 + * documented the GL client ring pair {push=0x1203c0000,len=64} then + * {push=0x1203c0000,len=32}; r966-r978 every kick exec and every + * canonical push read named exactly this VA). Used to discriminate the + * kick ring page from shape lookalikes in the RAM ring scan - r978 + * latched a foreign table page (pa=0x104fa9000) whose entry pair passed + * the shape gate and named a bogus 256 KB push (0x12048b824/len 0x40000) + * while the true ring (pa=0x1168ce000, found by scanning for the + * record's own entry pair) sat later in RAM scan order. + */ +#define SVC_GL_KICK_PUSH_VA 0x1203c0000ULL +/* SESSION 154 (M-GL1.2): watched USERD candidates for the liveness latch. */ +#define SVC_GR_USERD_WATCH 64 +/* + * SESSION 153-CONTINUED-20: a kick-family (h 0xbfef00xx) watched candidate + * whose PUT does not move for this long is a frozen pool lookalike (the + * r972/r973 descriptor-pool class: put=7 forever) - dropped so the ~50 + * frozen entries can never crowd out the real advancing control block. + */ +#define SVC_KICK_WATCH_STALE_MS 20000 + +/* + * r908: guest-RAM ring-scan top (guest physical). The guest's usable RAM + * can extend above the old 6 GB service cap: the 8 GiB guest's e820 top is + * 0x280000000, and the GL client channel's GPFIFO ring page was measured + * at gpa 0x231e5d000 (readable: svc_read -> vfio_dma_rw answered there). + * Reads past mapped RAM (the 0x80000000-0x100000000 PCI hole, or a smaller + * guest) fail fast, so a too-large top costs a few failed probes per sweep. + * The module parameter is the per-cycle guard: shrink it to the boot's real + * RAM top. + * + * r933: this parameter is now the single live PA acceptance bound for the + * whole service (svc_pa_max), not just the ring scan - the walk leaves in + * the 6-8 GiB band were exactly what the old separate 6 GB SEMA_PA_MAX + * rejected (the GL client's kick pages). SEMA_PA_MAX remains the + * compile-time maximum that sizes fixed tables; the param clamps it. + */ +unsigned long long gsp_sema_gr_scan_top = 0x280000000ULL; +module_param_named(sema_gr_scan_top, gsp_sema_gr_scan_top, ullong, 0644); + +/* + * r1780 (CONT.430b): keep the direct-GL (beef GR) ring content scan alive + * while the ring is unresolved and the channel has no submissions yet, so + * a client that registers an EMPTY ring before its first kick still gets + * found when it finally submits. Default 1 (the validated pre-change path + * is unchanged once a ring is latched or a submission is observed). + */ +static bool gsp_sema_gr_ram_keep = true; +module_param_named(sema_gr_ram_keep, gsp_sema_gr_ram_keep, bool, 0644); +MODULE_PARM_DESC(sema_gr_ram_keep, + "Refresh the beef GR ring content-scan budget while the ring is unresolved and no submission was observed; default 1"); + +/* + * r1777 (V12): direct-GL CE-family ring probe gate. Defined here because + * the demote/promote paths use it before the probe block; the parameter + * description lives with the probe implementation. Default off. + */ +static bool gsp_sema_ce_ram; +#define SVC_GR_SCAN_TOP_MAX 0x400000000ULL /* 16 GiB sanity clamp */ + +/* + * Live guest-PA acceptance bound: min(param, compile-time max). Every + * runtime range check that used to compare against SEMA_PA_MAX uses this + * instead, so raising the ceiling for a bigger guest (r933: 8 GiB guest, + * GL client's pages above 6 GiB) is one module parameter, and shrinking it + * for a smaller guest shrinks the scans and the walk acceptance together. + * Never 0 (a zero param falls back to the compile-time max). + */ +static inline u64 svc_pa_max(void) +{ + u64 top = READ_ONCE(gsp_sema_gr_scan_top); + + if (!top || top > SEMA_PA_MAX) + return SEMA_PA_MAX; + return top; +} + +/* + * Ring-scan slice budget: two full sweeps at the configured top, with the + * original 24-slice floor and a u8-safe cap (gr_ram_passes). + */ +static u32 svc_gr_ram_pass_budget(void) +{ + u64 top = gsp_sema_gr_scan_top; + u32 slices; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top <= SVC_GR_RAM_SLICE) + return SVC_GR_RAM_PASS_MAX; + slices = (u32)(top / SVC_GR_RAM_SLICE) * 2; + if (slices < SVC_GR_RAM_PASS_MAX) + slices = SVC_GR_RAM_PASS_MAX; + if (slices > 250) + slices = 250; + return slices; +} + struct svc_channel { bool valid; u32 session_gen; /* m176: session generation this entry was @@ -175,12 +325,40 @@ struct svc_channel { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (doorbell token vector) */ + u32 h_vaspace; /* r835: the client VAS this channel was + * allocated in (NV_CHANNEL_ALLOC_PARAMS. + * hVASpace). Channel-scoped VA resolution + * consults this VAS's page tables FIRST: + * every client's RM window sits at the same + * VAs (0x1200xxxxx) and the global VA-keyed + * maps conflate them - the X server's ring + * VA resolved through the GL client's page + * tables (walk-probe vas h=0xcaf00005 + * answered 0x120090000) and its push stream + * decoded as zero pages, wedging X the same + * way the RC watchdog's ring was starved. */ u8 ce_index; /* rank among the live CE channels (creation order) */ u8 gr_index; /* rank among the live GR channels (creation order) */ + /* SESSION 154 r1509: the beef GL client's ring head carries its + * one-shot init uploads (LOAD_INLINE_DATA code/data images). The + * first observed GP_PUT can already be far ahead of those entries + * (r1508: put=8982 at registration), so the continuation sweep + * never sees them and the fetch pages stay zero. Sweep the head + * once per channel registration in inline-only mode. */ + bool head_inline_done; u64 gpfifo_va; u32 gpfifo_entries; u64 userd_pa; u32 userd_as; /* NV_ADDR_SYSMEM / NV_ADDR_FBMEM */ + /* + * CONT.405: the DECLARED FBMEM userd block as it arrived in the + * channel alloc params (userdMem.base + userdOffset + userd + * index/page). userd_pa is overwritten whenever a scan latches the + * live block; this field keeps the wire value so the BAR1 alias can + * be re-derived on every pass and so the alias bias can be learned + * live (wire -> BAR1 page). Zero for non-FBMEM channels. + */ + u64 userd_wire; bool userd_ok; /* userd_pa reads a plausible put/get */ bool userd_wait_fresh; /* ISSUE-0002 r48: the layout USERD was accepted @@ -302,6 +480,21 @@ struct svc_channel { * the end with no operand left pending, so a * replay suppresses every op. When clear, the * done bitmap decides per op (retry path). */ + bool ever_ops; /* r611: this channel has decoded at least one + * transfer op (copy/fill), i.e. it carries the + * client's stream-ordered memory work. The r607 + * stream-order watermark considers only such + * channels: a sibling GR LAUNCH channel's ring + * legitimately holds future launches, and + * deferring on those creates mutual deferral + * between the client's own launch channels + * (streams2: launches on h=0x5c00001f/23 each + * saw the other's unconsumed entry, burned the + * 15 s cap and were refused fail-closed - the + * r610c stream class red). Measured r606/r609/ + * r610c: op-decoding channels are exactly the + * copy ranks (0x5c000059 family) and the CE + * ranks; launch channels decode no ops. */ u64 slot_ops_hash[SVC_GPFIFO_MAX_BITS]; /* Content witness for slot_ops_ident: the * FNV-1a hash of the push bytes the identity @@ -322,6 +515,15 @@ struct svc_channel { unsigned long push_hold_until; /* absolute deadline of the bounded * work-before-release hold for an * unreadable CE entry (0 = none) */ + unsigned long gr_setup_hold_until; /* SESSION 153-CONTINUED-54: absolute + * deadline of the GR setup-push hold: + * an unreadable/all-zero MME-setup + * (or >= 12 KB) GR push stays pending + * past the 1-failed-rescue consume + * until the demanded guest pin lands + * (0 = none) */ + u64 gr_setup_hold_va; /* entry VA the setup hold was armed for */ + u32 gr_setup_hold_idx; /* GPFIFO slot the setup hold was armed for */ unsigned long defer_since; /* first failure of the pending GR entry * (0 = none): the entry is left pending * for the GR grace window so a late @@ -385,6 +587,16 @@ struct svc_channel { * even if still unresolved - bounded zeros, never * a hang); m156: re-held while the window is * live so a busy forward gets its real retry */ + /* + * r509: one-shot verdict of the absolute bound. The expiry call + * pushes the deadline (so the hold window restarts) and latches this + * flag; the r351 retry-forward block consumes it within the same + * pass. Without the latch the two call sites in one pass could not + * both observe the expiry (the first push of the deadline made the + * second call see a fresh window - r213-r215's arming bug in + * reverse). + */ + bool gr_hold_bound_hit; unsigned long gr_hold_deadline; /* GRFIX FIX1b: hard expiry of the data- * stalled hold for gr_hold_dead_va. * Fixed once on the first hold of that @@ -420,6 +632,209 @@ struct svc_channel { u64 gr_nocap_va; /* push VA the window was opened for */ unsigned long gr_nocap_since; /* absolute first-seen stamp */ /* + * r496b (X/GL kernel-RM sysmem channel): ring discovery by content in + * guest RAM. The CPU RM maps sysmem objects locally (no + * SET_PAGE_DIRECTORY, no rm_mem record carrying the ring VA), so + * svc_lookup_trusted cannot name the ring page; the driver's kickoff + * GPFIFO entries {pushbuffer VA, len} identify it instead. The scan + * is sliced (SVC_GR_RAM_SLICE per pass) and cursor-resumed so the + * service thread stays responsive. + */ + u64 gr_ram_cursor; /* resume point of the content scan */ + u8 gr_ram_passes; /* slices walked so far */ + bool gr_ram_logged; /* one discovery line per channel */ + /* + * SESSION 149 R15 (M1.1b keep-alive): the boot-time slice budget can + * exhaust long before the client creates its ring objects (r886: 16 + * slices spent, the GL client's ring never scanned, no pairing). A + * GPFIFO put advance proves the client is live, so grant a fresh + * bounded budget - spaced and capped so a never-found ring cannot + * keep the scan hot. + */ + u32 gr_ram_kick; /* put value at the last budget refresh */ + u8 gr_ram_refreshes; /* budget refreshes granted so far */ + unsigned long gr_ram_refresh_at; /* next eligible refresh */ + /* + * r912 (M-GL1.2): guest-RAM USERD shape scan for a GR channel whose + * ring was content-validated in RAM (gr_ram_ring). Such a client's + * USERD is a system-memory object like its ring (the kernel-RM/X GL + * path), so the FB-shadow shape scan can never reach it and the pump + * never iterates its entries. Same sliced/cursor discipline as the + * ring scan; the cursor is persisted immediately because the + * stalled-userd pump path skips the end-of-pass write-back. + */ + u64 gr_ram_userd_cursor; /* resume point of the USERD scan */ + u8 gr_ram_userd_passes; /* slices walked so far */ + bool gr_ram_userd_logged; /* one discovery/give-up line */ + /* + * SESSION 154 (M-GL1.2): liveness confirmation for the RAM USERD + * latch. The shape+entry gates alone admitted frozen or slowly + * churning lookalike pages (r913: pa 0x7c000 adopted at put=1 only + * survived by luck; r914: pa 0x9e200 adopted at put=12 get=8 and + * its put%2==0 collapsed the walk - submissions stayed 0 while the + * client retried for minutes). A candidate is WATCHED across passes + * instead: only a block whose PUT takes the client's own single +1 + * step is adopted as the live control block - the measured live + * block advances 1->2->3 while the lookalikes jump by larger + * amounts or churn to zero. + */ + u64 gr_ram_userd_watch[SVC_GR_USERD_WATCH]; /* watched PAs */ + u32 gr_ram_userd_watch_put[SVC_GR_USERD_WATCH]; /* last PUT seen */ + u8 gr_ram_userd_watch_hits[SVC_GR_USERD_WATCH]; /* +1/+2 streak */ + /* + * SESSION 153-CONTINUED-20: jiffies when the entry was added or last + * stepped; the stale-entry timeout drops a candidate whose PUT has + * not moved for SVC_KICK_WATCH_STALE_MS (the frozen descriptor-pool + * class the kick-family watch exists to defeat). + */ + unsigned long gr_ram_userd_watch_at[SVC_GR_USERD_WATCH]; + u8 gr_ram_userd_watch_n; + u32 gr_ram_userd_watch_log; /* bounded add-log counter */ + /* + * SESSION 153-CONTINUED-34 (BAR0-shadow USERD): the GL client's channel + * allocation declares its USERD in the FB aperture (0x18fc00-class + * offsets) but the guest's GP_PUT writes land in the emulated BAR0 + * shadow at that offset. Watch that block across passes: a nonzero PUT + * (or a PUT step) is the client's own live write; adopt it as the + * channel's USERD in the BAR0 aperture so the pump iterates the ring. + * bar0_userd_at is the first-sighting stamp; a stable nonzero PUT is + * adopted after a short patience window so a single-submission client + * (waiting on that submission) cannot deadlock the watch. + */ + u64 bar0_userd_watch_pa; /* watched offset (0 = none) */ + u32 bar0_userd_watch_put; /* PUT at the first sighting */ + unsigned long bar0_userd_watch_at; /* jiffies at the first sighting */ + u8 bar0_userd_log; /* bounded per-channel log counter */ + /* + * SESSION 153-CONTINUED-20: bounded budget re-arms for the + * kick-family watch sweep. When every watch entry has gone stale + * (frozen pool) and the slice budget is spent, one re-arm keeps the + * sweep alive so the real block can still be found; capped like the + * reject counter so a dead channel stops sweeping in bounded time. + */ + u8 gr_ram_userd_kick_sweeps; + u32 gr_ram_put_seeded; /* last PUT the walk seed consumed */ + /* + * SESSION 154 (M-GL1.2): speculative-latch confirmation state. The + * RAM USERD scan latches a gated candidate immediately; the pump + * confirms it from its own per-pass PUT read (a +1/+2 step), and + * unlatches a churning/frozen block (see svc_pump_channels). + */ + u32 gr_ram_userd_latch_put; /* PUT at the speculative latch */ + unsigned long gr_ram_userd_latch_at; /* jiffies at the latch */ + bool gr_ram_userd_confirmed; /* +1/+2 step observed */ + /* + * CONT.407: keep-across-put-reset. r1701-r1703 proved the kick + * family's live block with a +1 step (put 2 -> 3) and then the owner + * zeroed it (`speculative latch REJECTED (put=0 latch=3)`) 7-8 s + * later; dropping the USERD there destroyed a working control block + * and left the frame loop's next kick undecodable. A confirmed kick + * latch now KEEPS the block across a put reset, re-arms at the + * reset baseline (put=0) and lets the next authored step re-confirm. + * Bounded: after 64 keeps a block that keeps zeroing is treated as + * churn by the old path; the log emits at most 8 lines per channel. + */ + u8 gr_ram_userd_keeps; /* keeps applied (bound 64) */ + u8 gr_ram_userd_keep_log; /* keep log lines (bound 8) */ + /* + * SESSION 153-CONT-5: rejected-latch memory. A speculative latch the + * pump rejects (frozen or jumped PUT) used to exhaust the RAM scan + * budget permanently: r936's kick channel latched a FROZEN lookalike + * (put=7 get=1 on a driver struct page), the pump rejected it twice, + * the 20-slice scan budget retired, and the channel went dark with no + * re-latch path - kick retires (and their publish writes) stopped for + * the rest of the session. Remember the rejected PA and its PUT: the + * scan skips that exact (pa, put) pair silently, re-adopts the block + * the moment its PUT moves (content change = new evidence), and each + * rejection re-arms the scan budget so one stale window can never kill + * the pipeline. The re-arm count is bounded (64: the r937 rate of + * ~4 rejections/min makes that >15 min of continuous retire activity + * on a client that only ever wrote one kick, while a truly dead + * channel still stops sweeping in bounded time). + */ + u64 gr_ram_userd_reject_pa; /* PA of the rejected latch */ + u32 gr_ram_userd_reject_put; /* its PUT at rejection time */ + u8 gr_ram_userd_rejects; /* rejections seen (bound) */ + /* + * SESSION 154 (M-GL1.2): content anchor for a client push whose + * translation went stale (r913: a validated pin kept answering for + * a reused page and the wrong-VAS walk answer was implausible while + * the real push bytes sat in RAM on a page no directory covered). + * Scan guest RAM for the client's release-group signature (sliced, + * cursor-resumed, like the ring scans), validate the page and pin + * it so the pump's retry decodes the real bytes. + */ + u64 gr_anchor_cursor; /* resume point of the anchor scan */ + u64 gr_anchor_va; /* entry VA the sweep is for */ + u8 gr_anchor_sweeps; /* completed sweeps for that VA */ + bool gr_anchor_logged; /* one start line per channel */ + /* + * r496e: true when the ring was found by the RAM content scan (a + * kernel-RM sysmem channel - the X/GL 3D channel). Such a channel's + * pushes are 2D/3D method streams the emulation does not execute; + * the correct model is consume-and-apply-releases: no QMD capture, + * no launch forward, no no-capture hold. Also allows an all-zero + * NOP push (the driver's pushbuffer sanity test writes exactly one + * zero dword) to be consumed as the no-op it is. + */ + bool gr_ram_ring; + /* + * SESSION 153-CONTINUED-19 (kick-family USERD acceptance): ring + * liveness re-validation pace for the kick channel (h 0xbfef00xx). + * While the kick USERD stays unresolved this channel's ring latch is + * re-checked each pass (at most once a second) and re-armed when it + * no longer decodes entries - see svc_kick_ring_revalidate. + */ + unsigned long gr_kick_ring_reval_at; + /* + * r496b: the FB-scan USERD (see svc_userd_fb_scan). Latched when the + * registered/ring-derived candidates all fail; the scan validates the + * hit against the ring entries the put claims, so a foreign channel's + * USERD cannot be adopted. Bounded retries: the driver kicks off a + * few passes after channel registration, so a scan before the first + * put must not retire the discovery. + */ + u8 userd_scans; + /* + * SESSION 153-CONTINUED-37: time-paced re-run state for the FB USERD + * shape scan (r496b) and its pristine-block fallback. The old + * count-only cap spent every retry within ~64 ms of registration + * (pump pass cadence), while the X driver's pushbuffer test kicks its + * first GP_PUT hundreds of ms later and its VAS registers at ~1 s - + * so on unlucky timings the budget was gone before any candidate + * existed and the 10 s DMA test read GET=0 forever ("Failed to + * initialize DMA", r1013). One scan per 25 ms up to 512 keeps the + * discovery alive across the whole client window. + */ + unsigned long fb_userd_scan_at; /* jiffies of the last scan */ + u16 fb_userd_scan_n; /* scans spent since registration */ + /* + * r828: the RC watchdog's notifier VA, captured from the class2d + * SET_NOTIFY_A/B methods on subchannel 3 and zeroed at the NOTIFY + * (see svc_notify2d_write). Survives the pass snapshot via the + * write-back list. + */ + u64 notify2d_va; + /* r829: bounded count of zero-entry (bad-translation) holds; see the + * zero-entry hold in svc_pump_channels. Persists via the write-back. */ + u32 zero_held; + /* r831: the registered userd value is a real guest PA (the driver's + * own declaration; e.g. the RC watchdog channel). The ring auto-pin's + * ring+0x2000 convention must not overwrite it. */ + bool userd_reg; + /* + * r471: the completion-counter publish must never advance the + * guest's word past a push whose content was still unwritten when + * the entry was consumed (the streams_events staging-reuse + * corruption: the publish satisfied the client's D2H wait before + * its copy ran, and the readback drained a staging buffer that + * still held the PREVIOUS copy's bytes). Lowest ring slot consumed + * with all-zero content; 0 = none. Cleared when a later decode of + * a GR push reads non-zero content (the content landed - the rewinds + * re-read entries constantly, so this heals within a pass or two). + */ + u32 gr_bad_put; + /* * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. A * channel that cannot make forward progress (last_get pinned behind * put, or a residual session put that never drops) is detected here @@ -482,10 +897,23 @@ struct svc_pin { struct svc_vaspace { bool valid; + /* + * CONT.400: proven against the client's own content-validated ring (or + * published by RM). The bounded age-out registers UNVERIFIED; keeping + * the distinction lets svc_chan_vas_candidate_promote upgrade a parked + * directory to verified instead of giving up because it is "known". + */ + bool verified; u32 h_vaspace; u64 pdb; u32 entries; /* root entry count */ int ap; /* SVC_APERTURE_* of the directory itself */ + unsigned long stamp; /* r459: last registration/use; LRU evict + * when the table is full (a stale entry from + * a dead client otherwise blocks every new + * client after the 8th registration: the + * mid-boot walk failure that made the 10th+ + * client's copies fail rc=-2) */ }; struct svc_lat { @@ -515,6 +943,12 @@ static void svc_mark_hot(u64 pa) * push-buffer base) and the channel (its slot). */ #define SVC_PB_MAX 32 +/* + * R10 (patch A): svc_pb[] slot namespace for GR (eng==0) channels' + * content-anchored ring windows (svc_pb_pair_gr). CE slots are + * 4*ce_index (0..124), so the high bit keeps the namespaces apart. + */ +#define SVC_PB_GR_SLOT_BASE 0x80000000u struct svc_pb_hit { u64 pa_page; u64 va; /* push-buffer base VA, once paired with a channel */ @@ -548,21 +982,69 @@ static u32 svc_ring_n; */ static u8 svc_sess_flush; static u8 svc_guest_reset_req; /* m177: guest RM reload; wiped on svc thread */ +static u8 svc_sess_flush_forced; /* execute the next flush past the live-channel skip + * (ISSUE-0002 client-session boundary aid; see + * gsp_sema_service_sess_flush_force) */ +/* + * SESSION 148 perf tracing: sema_trace_h selects one channel handle; the + * pump then logs a compact state line for it once a second, and + * svc_counter_publish logs why it skipped (last_get gate / arming gate) or + * the value it wrote. This is the instrument for locating the 22.4 s/frame + * glxgears pacer without guessing at windows. + */ +unsigned long long gsp_sema_trace_h; +module_param_named(sema_trace_h, gsp_sema_trace_h, ullong, 0644); +MODULE_PARM_DESC(sema_trace_h, + "Trace one channel handle (0x...; 0=off): 1/s state + publish gates"); +static unsigned long svc_perf_last; static u8 svc_sess_active; /* a channel of a session has registered (until flush) */ static u32 svc_sess_gen; /* m176: bumped by every executed session flush; * stamps channels/scrubbers so per-session learned * state is never inherited by a recycled handle */ static struct completion svc_sess_flush_done; /* a pending flush has executed */ static unsigned long svc_ring_done; /* CE ranks whose ring is pinned (session state) */ +/* + * r1644: one-shot authoring of the UVM channel-pool init-push tracker + * release, keyed by CE rank. A UVM CE channel's first submission releases + * its own tracking semaphore -- pool slot rank (channel-creation order) in + * the page mapped at 0x121000000 -- with payload 1, and channel_init() + * waits on that value. + * The GPFIFO entry is regularly invisible at the registered VA/PA at the + * instant the guest waits for it (r1644: rank 1 read zero at 0x22b0000 and + * only later translated to 0x2640000, after the notifier clear), so the + * release is authored from the rank + pool mapping when the work-submit + * notifier edge proves the submission exists. Session state; re-armed per + * channel registration and cleared by svc_session_flush. + */ +static unsigned long svc_ce_initrel_done; +static unsigned long svc_ce_initrel_until[32]; /* re-fan-out window per rank */ +static unsigned long svc_ce_initrel_next[32]; /* next retry per rank */ +static u32 svc_ce_initrel_val[32]; /* last value authored per rank */ +static u32 svc_ce_initrel_log; /* Per-guest session records (one per open mdev; see SVC_MAX_CHANNELS for the channel cap). */ #define SVC_MAX_GUESTS 8 +/* + * SESSION 153-CONTINUED-34: PCFG doorbell window in the emulated BAR0 shadow. + * Work-submission doorbells the guest pokes after writing a GPFIFO entry can + * land anywhere in the PCFG window (0x88000..0x89000), not only at the + * virtual-function doorbell the trap path used to dispatch (0x88704). Because + * the BAR0 shadow is direct-mapped, most writes never reach the trap path at + * all, so the service thread also watches the window for changed words and + * feeds each new value to gsp_sema_service_doorbell (token format unchanged; + * the call only wakes the pump earlier - the pump itself is per-pass). + */ +#define SVC_BAR0_PCFG_LO 0x88000u +#define SVC_BAR0_PCFG_HI 0x89000u +#define SVC_BAR0_PCFG_WORDS ((SVC_BAR0_PCFG_HI - SVC_BAR0_PCFG_LO) / 4u) struct svc_guest_sess { struct nv_vgpu_device *vgpu; /* NULL = free slot */ bool active; /* a channel of this guest is * registered until this guest's * last channel closes */ u64 last_build; /* per-guest svc_rebuild freshness */ + u32 pcfg_snap[SVC_BAR0_PCFG_WORDS]; + bool pcfg_have; /* snapshot taken for this guest */ }; static struct svc_guest_sess svc_guests[SVC_MAX_GUESTS]; static DEFINE_SPINLOCK(svc_guest_lock); @@ -621,6 +1103,7 @@ static struct svc_guest_sess *svc_guest_get(struct nv_vgpu_device *vgpu, */ struct svc_scrub { struct nv_vgpu_device *vgpu; /* owning mdev (NULL = free slot) */ + bool registered; /* PA came from the guest scanner (vmap) */ bool userd_ok; /* a 4096-entry channel was registered */ u32 h_object; /* handle of the registered scrubber channel */ u64 userd_pa; @@ -630,41 +1113,191 @@ struct svc_scrub { bool pa_ok; /* scanner registered fin/hsem PAs */ u64 fin_pa; /* finish payload word (u32) */ u64 hsem_pa; /* channel host semaphore word (u32) */ + unsigned long pa_at; /* last successful walk re-resolution */ bool seeded; u32 put_last; + u32 consumed; /* CONT.408: last consumed entry (sema) */ u64 submitted; /* blocks submitted since seeding */ u64 authored; /* last authored finish value */ + u32 authored_gen; /* pa_gen at the last authored write */ + u32 fin_release; /* last authored finish value (real counts) */ u32 pa_gen; /* scanner registration generation (logged) */ u32 session_gen; /* m176: session generation of the last registration */ u32 log_budget; + /* + * CONT.419: GPFIFO submission ledger. The registered channel's ring + * (channel buffer + 0x64000) records every scrub work item the guest + * submitted; its valid entries are the ONLY host-observable source of + * the guest's lastSubmittedWorkId. fin is advanced to this count + * (forward-only), which mirrors real CE completions and can never + * exceed the guest's pending list. + */ + u32 ring_cum; /* cumulative pushes counted from the ring */ + unsigned long ring_at; /* last ring read (rate limit) */ + unsigned long ring_valid_at; /* last first-read validation attempt */ + u64 ring_pa; /* ring page resolved in the latest read */ + bool ring_page_ok; /* GPFIFO VA resolved this pass */ + bool ring_read_ok; /* the last ring read completed */ + bool ring_ready; /* snapshot initialized for this geometry */ + bool ring_valid; /* first-read prefix+terminus accepted */ + bool ring_logged; /* one-shot ring ledger log */ + bool ring_release_logged; /* one-shot ring release log */ + u32 ring_resets; /* bounded reset-log budget */ }; /* - * One scrubber record per guest: every guest's PMA has its own 4096-entry - * scrubber channel, and the authored words must be written through the - * owning guest's address space (writing guest A's PA through guest B's VFIO - * device would corrupt B). Keyed by vgpu pointer, freed by - * gsp_sema_service_vgpu_cleanup(). + * One scrubber record per guest-channel buffer: every guest's PMA has its + * own 4096-entry scrubber channel, and the X DDX brings its own CE/PMA + * channel buffer whose host semaphore + finish payload are separate words. + * All authored words must be written through the owning guest's address + * space (writing guest A's PA through guest B's VFIO device would corrupt + * B). Keyed by vgpu pointer, freed by gsp_sema_service_vgpu_cleanup(). + * + * CONT.409: more than one record per guest. The stash path below binds the + * CeUtils scrubber channel (4096 GPFIFO entries) it discovers in the RM + * alloc stream; the guest-side scanner registers additional channel-buffer + * vmap pages through sema_scrubbuf_add. The scanner's PA is the page the + * guest RM actually polls (its memdesc/transfer vmap), while the GPU-VAS + * walk can keep resolving a stale directory after the guest RM re-creates + * its kernel VAS - the r1712 forced-sema boot is the witness (walk target + * 0x126b60000 while the guest polled 0x124fec000). A registered PA is + * therefore authoritative and is never replaced by a walk result. + */ +#define SVC_SCRUB_RECS 4 +static struct svc_scrub svc_scrubs[SVC_MAX_GUESTS * SVC_SCRUB_RECS]; + +/* + * ALLOC-LOTTERY fix (r1765/r1766/r1767 RCA): the guest FOLDS the finish + * payload word into a 64-bit completed-work counter (ceutilsUpdateProgress) + * and _scrubCheckLocked() then allocates (completed - lastSeen) * sizeof + * (SCRUB_NODE) = count * 24 bytes at the next PMA allocation. Releasing + * the wait with 0xffffffff made the next PMA allocation request + * 24 * 0xffffffff bytes (vmap: 103079215088 = 24*0xffffffff + 8 staging) + * and fail - X died with "Failed to allocate VASpace object" BEFORE its + * alloc RPC (no host exchange). Any FABRICATED delta is equally unsafe: + * the guest's _scrubCopyListItems() decrements scrubListSize by the + * reported delta with no guard, so a delta larger than the pending list + * underflows the u64 and the guest spins ~2^64 in _searchScrubList under + * the scrubber mutex (r1766/r1767: Xorg 100% CPU soft-lock; PMA and + * kernel corruption; panics). Never fabricate work: mirror ONLY the + * submissions the guest actually made, so the reported delta can never + * exceed its pending list. Until then write back the word it already + * holds (a no-op). The 96 GiB fold is therefore avoided because we never + * write all-ones while sema_scrub_ramp=1: the value is either the current + * word (no-op) or a real submitted count. The host semaphore keeps the + * all-ones release (it is COMPARED against a putIndex, never folded). + * A/B: sema_scrub_ramp=0 restores the legacy all-ones release. + * + * Known limit: the value advances only when real submissions are observed; + * the proper long-term fix is to keep the scrubber USERD GP_PUT readable + * in every cycle and author exactly its count (tracked in the handoff). + */ +static bool gsp_sema_scrub_ramp = true; +module_param_named(sema_scrub_ramp, gsp_sema_scrub_ramp, bool, 0644); +MODULE_PARM_DESC(sema_scrub_ramp, + "scrub finish release: 1 = real submitted counts only (default), 0 = legacy all-ones"); + +/* + * r1779 (V13): P0 for the per-boot X/GL wedge. The ring ledger must never + * author a fin value for a ring it has not verified. sema_scrub_ring=0 is the + * A/B control: the ledger is skipped entirely and fin stays a no-op (a + * blocked scrub wait is recoverable; an underflowed scrubListSize is not). */ -static struct svc_scrub svc_scrubs[SVC_MAX_GUESTS]; +static bool gsp_sema_scrub_ring = true; +module_param_named(sema_scrub_ring, gsp_sema_scrub_ring, bool, 0644); +MODULE_PARM_DESC(sema_scrub_ring, + "scrub fin ledger from the channel GPFIFO ring: 1 = enabled (default), 0 = A/B off"); +static u32 svc_scrub_fin_next(struct svc_scrub *s, u32 cur) +{ + u32 base; + + /* + * Never fabricate completed scrub work (r1766/r1767 RCA): the + * guest's _scrubCopyListItems() decrements scrubListSize by the + * reported delta with no guard, so any fabricated delta larger + * than the pending list underflows the u64 and the guest spins + * ~2^64 in _searchScrubList under the scrubber mutex (Xorg 100% + * CPU soft-lock; PMA corruption; kernel panics). Mirror ONLY + * work the guest actually submitted; until then write back the + * word it already holds (a no-op). A/B: sema_scrub_ramp=0 still + * restores the legacy all-ones release. + */ + if (!s->submitted || s->submitted <= s->authored) + return cur; /* nothing new submitted: no-op */ + base = (u32)s->submitted; + if (base <= cur) + return cur; /* forward-only */ + s->fin_release = base; + return base; +} + +/* + * First record of this guest for the CeUtils stash path: prefer the record + * that already owns a stashed 4096-entry channel, else a non-registered + * slot, so a scanner registration is never clobbered by the stash. + */ static struct svc_scrub *svc_scrub_get(struct nv_vgpu_device *vgpu, bool create) { struct svc_scrub *s = NULL; + struct svc_scrub *any = NULL; int i; if (!vgpu) return NULL; - for (i = 0; i < SVC_MAX_GUESTS; i++) { - if (svc_scrubs[i].vgpu == vgpu) + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + if (svc_scrubs[i].vgpu != vgpu) + continue; + if (svc_scrubs[i].userd_ok) return &svc_scrubs[i]; - if (create && !s && !svc_scrubs[i].vgpu) + if (!svc_scrubs[i].registered && !any) + any = &svc_scrubs[i]; + } + if (any) + return any; + if (!create) + return NULL; + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + if (!svc_scrubs[i].vgpu) { s = &svc_scrubs[i]; + memset(s, 0, sizeof(*s)); + s->vgpu = vgpu; + return s; + } } - if (s) { - memset(s, 0, sizeof(*s)); - s->vgpu = vgpu; + return NULL; +} + +/* + * Register/replace the scanner's channel-buffer record for this guest. + * Matching on hsem_pa keeps a moved buffer as a new record (the old one is + * reused because a registered record without a stashed USERD is disposable + * once the scanner names a different page). + */ +static struct svc_scrub *svc_scrub_reg(struct nv_vgpu_device *vgpu, u64 hsem_pa) +{ + struct svc_scrub *s = NULL; + int i; + + if (!vgpu) + return NULL; + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + if (svc_scrubs[i].vgpu != vgpu) + continue; + if (svc_scrubs[i].hsem_pa == hsem_pa) + return &svc_scrubs[i]; + if (svc_scrubs[i].registered && !svc_scrubs[i].userd_ok && !s) + s = &svc_scrubs[i]; + } + if (s) + return s; + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + if (!svc_scrubs[i].vgpu) { + memset(&svc_scrubs[i], 0, sizeof(svc_scrubs[i])); + svc_scrubs[i].vgpu = vgpu; + return &svc_scrubs[i]; + } } - return s; + return NULL; } static u8 svc_pb_passes; static u64 svc_pb_last; @@ -681,7 +1314,23 @@ static u64 svc_pb_last; static u32 svc_pb_idle; /* consecutive passes with no new base */ static u32 svc_pb_last_n; /* bases before the active pass */ static unsigned long svc_pb_done; /* CE ranks whose push buffer is pinned */ +/* R10 (patch A): GR ranks whose ring-anchored window is paired. */ +static unsigned long svc_pb_gr_done; static unsigned long svc_gr_ring_done; /* GR ranks whose ring is pinned */ +/* SESSION 133: ranks that already consumed their one shadow ring scan. */ +static unsigned long svc_gr_ring_scan_done; +/* + * r1780 (CONT.430c): per-rank scan cadence. A single global timestamp let + * the first GR channel in pump order (descending handles; the RC watchdog + * 0x31415900 precedes the direct-GL 0xbeef0100) consume the window every + * 1.5 s, so the direct-GL call always returned at this gate and its ring + * content scan never ran - the ring stayed unresolved and the X DDX hit + * its idle timeout. Per-rank keeps the same bound per channel. + */ +static unsigned long svc_gr_ring_scan_last[32]; /* per-rank scan rate limit */ +/* SESSION 138: bounded witness for the GR request builder's page list and + * the fill's candidate set (one launch's wiring printed per budget slice). */ +static u32 svc_gr_pages_log; static bool svc_pb_first_push(const u8 *buf) { @@ -803,6 +1452,18 @@ struct sema_svc { u64 sem256_log; /* 256-byte op trace budget */ u64 op_tr_log; /* operand targeted-resolve budget */ u32 op_tr_pending; /* an operand's targeted scan is in progress */ + u32 op_owed; /* r540: a denied transfer op + * was never re-run because the + * per-slot pend cap was hit; the + * slot must stay INCOMPLETE so a + * later trusted/injected pass + * runs it (the r539 5-minute + * device-init stall class: the + * client's GRCE H2D was denied + * while untrusted, capped, then + * replay-suppressed forever and + * the copy never executed) */ + u64 op_owed_log; /* r540: bounded owe witness */ u64 copy_watch_log; /* sema_copy_watch log budget */ u64 copy_dst_hold; /* m169: device-dst copies held for * want of a pin (stale-table guard) */ @@ -812,6 +1473,7 @@ struct sema_svc { u64 copy_dst_log; /* sema_copy_dst log budget */ u32 op_tr_src; /* provenance of last svc_op_translate */ u32 sem_sw_log; /* sema-sw release-write log budget */ + u32 sem_sw_xg_log; /* sema-sw X/GL-window budget */ u64 push_pat_log; /* push-content pattern budget */ u64 dma_skip_log; /* unrecognized-DMA budget */ u64 xfer_raw_log; /* operandless transfer raw-dump budget */ @@ -819,6 +1481,7 @@ struct sema_svc { u64 pushtrace_all; /* fail4: de-windowed push decode * trace (every push, bounded) */ u64 opadd_log; /* fail4: every op appended */ + u64 op2d_raw_log; /* r458: raw 2D-shaped copy pushes */ u64 dmatrace_log; /* fail4: denied-transfer trace */ u64 pend_dbg_log; /* fail4 r305: pending/hold witness */ u64 snap_gate_log; /* fail4 r311: capture-gate witness */ @@ -841,9 +1504,22 @@ struct sema_svc { u64 entry_norel_log; u64 ops_denied; /* work refused: implausible stream */ u64 false_trans; /* resolvable VA mapped to non-push */ + u64 kick_badpg; /* r977: kick-family push reads + * that lacked the canonical + * submit-record prefix (wrong + * page; forced the rescue) */ + u64 rescue_map_ok; /* SESSION 148: content-verified + * map-rescue successes (GR + CE) */ u64 sem_write_reresolve; /* writes re-resolved after read-back */ u64 pool_alt_applied; /* pool releases fanned to a * displaced pin (ISSUE-0002) */ + u64 pool_dual_applied; /* r539: pool releases fanned + * to the secondary candidate page + * (pin/walk disagreement; the + * r538 init-stall class) */ + u64 pool_dual_log; /* r539 fan-out log budget */ + u64 pool_tracker_log; /* r1642 tracker-slot budget */ + u64 pool_vas_log; /* r1648 all-VAS fan-out budget */ u64 pool_residue; /* fresh-channel first releases whose * pool word read far ahead of the * payload (previous session's @@ -878,6 +1554,9 @@ struct sema_svc { * words but built no candidate */ u64 gr_nocap_consumed; /* m164: no-capture windows that * expired and consumed the entry */ + u64 gr_owed_resolved; /* r548: unwound deferred launches + * software-completed at hold expiry + * instead of consumed silently */ u64 gpfifo_starved; /* entries consumed without a * deliverable exact slot */ u64 counter_rel_applied; /* releases to registered counters */ @@ -956,6 +1635,8 @@ struct sema_svc { u64 fwd_copy_skipped; u64 fwd_copy_soft; u64 fwd_copy_fail; + /* r1497: empty FB fetch pages populated from their sysmem backing. */ + u64 fwd_fetch_backing; u64 fwd_rel_deferred; u64 fwd_rel_applied; u64 fwd_rel_fallback; @@ -1011,7 +1692,43 @@ struct sema_svc { u64 fwd_gr_ok; u64 fwd_gr_fallback; u64 fwd_gr_refused; + u32 fwd_gr_nocand_log; /* SESSION 127: first-N + * request-without-candidate + * witnesses */ + u64 fwd_gr_nordy; /* SESSION 127: backend-not- + * ready request refusals */ + u64 fwd_gr_holdbusy;/* SESSION 127: hold-active + * request busy verdicts */ + u64 fwd_gr_inflight;/* SESSION 127: inflight-bit + * request busy verdicts */ + u32 fwd_gr_enoe_log;/* SESSION 127: first-N fill + * ENOENT page witnesses */ u64 fwd_gr_defer; /* m150: held for data-arg retry */ + u64 fwd_gr_dstpend; /* r541: held for an unexecuted + * device write on a data page */ + u32 fwd_gr_dstpend_log; + u64 fwd_gr_closed; /* r583: fail-closed at the defer + * cap (dstpend still live) */ + u32 fwd_gr_closed_log; + u64 fwd_gr_argdefer; /* r543: held for a partial + * argument set (unresolved arg) */ + u32 fwd_gr_argdefer_log; + u64 fwd_gr_argfw; /* r612: non-degenerate fill + * forwarded at the cap (stalled + * candidates are decoys) */ + u32 fwd_gr_argfw_log; + u64 fwd_gr_cewm; /* r590: held on the CE ring's + * live submission watermark */ + u64 fwd_gr_cewm_cap; /* r590: watermark still + * backlogged at the defer cap */ + u32 fwd_gr_cewm_log; + u64 ce_mutual_skips; /* r638: watermark + * skipped a channel whose + * head entry is wait-gate + * held (a consumer-side + * mutual citation, not a + * producer backlog) */ + u32 ce_mutual_log; u64 fwd_gr_timeout; u64 fwd_gr_fault; u64 fwd_gr_bridge; @@ -1043,6 +1760,46 @@ struct sema_svc { #define SVC_STALLED_MAX 256 u64 stalled_vas[SVC_STALLED_MAX]; u32 stalled_va_count; + + /* + * SESSION 153-CONTINUED-56: the armed setup-hold's demand VA set + * (page-aligned, deduped per boot), exposed read-only to the demand + * tool via /sys/kernel/debug/vgsp/sema_setup_demand. State, not a + * log row: under vgsp-sema spam the `stalled VA #N` dmesg rows + * rotate out of the ring before the 3 s harvest reads them, so the + * setup-push pages were lost (r1090 needed a manual demand); the pin + * tool now reads this file directly each pass instead. + * + * SESSION 153-CONTINUED-60: the set is a RING. At 64 entries the + * r1106 MISS wall (push=0xcf3d724 miss=11..17/pass, abort loop + * n=2..9) never converged because every later gsp_sema_demand_note + * (the MISS pages themselves, the attr-prelude program pages) was + * silently dropped once the 64 slots were exhausted -> the pin tool + * never demanded them -> never resolved -> perpetual pre-doorbell + * abort. Capacity is raised to 512 and a full set overwrites the + * oldest slot instead of refusing the note, so the resident set is + * the demand-paging working set and a later note is never lost; + * total/evicted counters are published for observability. + */ +#define SVC_SETUP_DEMAND_MAX 512 + u64 setup_demand_vas[SVC_SETUP_DEMAND_MAX]; + /* + * SESSION 153-CONTINUED-64: the held entry's exact read window. + * The armed setup hold records its entry page with (off = entry va's + * in-page offset, len = the GPFIFO entry length); every other page + * keeps len = 0. The guest pin resolver uses the pair to validate a + * candidate page RANGE against the exact bytes the host will read + * [va, va+len) with the host's own full method-stream parse, so a pin + * is only injected when the host read is guaranteed to parse (kills + * the content-heuristic family learn that served a wrong area in + * r1120: `first8=00040000 00001000 ... plausible=0`). + */ + u8 setup_demand_off[SVC_SETUP_DEMAND_MAX]; + u32 setup_demand_len[SVC_SETUP_DEMAND_MAX]; + u32 setup_demand_count; + u32 setup_demand_head; /* ring write cursor (valid once full) */ + u64 setup_demand_total; /* notes accepted (dedup hits excluded) */ + u64 setup_demand_evicted; /* oldest-slot overwrites */ }; static struct sema_svc svc; @@ -1076,6 +1833,180 @@ static void svc_log_stalled_va(u64 va) } /* + * SESSION 153-CONTINUED-56: record one page-aligned VA of the armed + * setup-hold demand set. Same caller as svc_log_stalled_va (the service + * thread); the debugfs reader publishes entries with a write barrier on the + * count, so a reader can never observe a torn entry (the count never + * decreases and an entry is stable once its slot is covered by the count). + * + * SESSION 153-CONTINUED-60: full-set behaviour is ring overwrite (oldest + * slot) instead of drop, so the r1106 MISS pages can never be refused a + * slot by a stale 64-entry set; see the member comment. Entry stores are + * aligned u64 on a 64-bit kernel, so the in-place overwrite cannot tear for + * the seq reader (worst case it sees the previous resident VA once more). + */ +static void svc_setup_demand_record_ex(u64 va, u32 off, u32 len) +{ + u32 i, n = svc.setup_demand_count; + + if (!va) + return; + for (i = 0; i < n; i++) + if (svc.setup_demand_vas[i] == va) { + /* + * SESSION 153-CONTINUED-64: upgrade a plain entry to the + * anchored (off,len) window when the hold re-records it. + * off/len publish order keeps the reader consistent for + * new slots; an in-place upgrade may be seen one pass + * torn (off new, len old) - the guest then validates the + * wrong window once and retries, which is harmless. + */ + if (len && !svc.setup_demand_len[i]) { + svc.setup_demand_off[i] = (u8)off; + smp_wmb(); + svc.setup_demand_len[i] = len; + } + return; + } + if (n < SVC_SETUP_DEMAND_MAX) { + svc.setup_demand_vas[n] = va; + svc.setup_demand_off[n] = (u8)off; + svc.setup_demand_len[n] = len; + smp_wmb(); + svc.setup_demand_count = n + 1; + } else { + static u32 ring_log; + u32 w = svc.setup_demand_head; + + svc.setup_demand_vas[w] = va; + svc.setup_demand_off[w] = (u8)off; + svc.setup_demand_len[w] = len; + svc.setup_demand_head = (w + 1u) % SVC_SETUP_DEMAND_MAX; + svc.setup_demand_evicted++; + if (ring_log < 4u || !(svc.setup_demand_evicted % 4096u)) { + ring_log++; + pr_info("sema-svc: demand ring full; evicting oldest slot %u for va=0x%llx (evicted=%llu total=%llu)\n", + w, va, svc.setup_demand_evicted, + svc.setup_demand_total + 1u); + } + } + svc.setup_demand_total++; +} + +static void svc_setup_demand_record(u64 va) +{ + svc_setup_demand_record_ex(va, 0, 0); +} + +/* + * SESSION 153-CONTINUED-59: cross-file entry for the GR launch builder (see + * gsp_sema_service.h). Records the page of a VA whose window/attr operand + * relocation MISSed, so the pin demand path fetches it before the next + * forward; the builder then either resolves it (RELOC level-2 fallback) or + * aborts the launch pre-doorbell instead of letting the engine read a raw + * unmapped guest VA (the r1104 MMU Fault HUBCLIENT_PD class). + */ +void gsp_sema_demand_note(u64 va) +{ + if (va) + svc_setup_demand_record(va & ~(u64)0xfffu); +} + +/* + * Read-only view of the setup-hold demand set for the pin tool: + * # sema setup-hold demand VAs (page-aligned): N total=T evicted=E + * 0x4021000 + * 0x4001004 0x4 60216 <- anchored entry: entry va, in-page offset, len + * ... + * `total` counts every accepted note (dedup hits excluded); `evicted` counts + * ring overwrites of the oldest slot (SESSION 153-CONTINUED-60). An anchored + * line carries the exact [va, va+len) read window of the held GPFIFO entry + * (SESSION 153-CONTINUED-64). + */ +static int svc_setup_demand_show(struct seq_file *m, void *v) +{ + u32 i, n = READ_ONCE(svc.setup_demand_count); + + seq_printf(m, "# sema setup-hold demand VAs (page-aligned): %u total=%llu evicted=%llu\n", + n, (unsigned long long)svc.setup_demand_total, + (unsigned long long)svc.setup_demand_evicted); + for (i = 0; i < n && i < SVC_SETUP_DEMAND_MAX; i++) { + u32 dl = svc.setup_demand_len[i]; + + if (dl) + seq_printf(m, "0x%llx 0x%x %u\n", + (unsigned long long)svc.setup_demand_vas[i] + + svc.setup_demand_off[i], + (unsigned)svc.setup_demand_off[i], dl); + else + seq_printf(m, "0x%llx\n", + (unsigned long long)svc.setup_demand_vas[i]); + } + return 0; +} + +static int svc_setup_demand_open(struct inode *inode, struct file *file) +{ + return single_open(file, svc_setup_demand_show, NULL); +} + +static const struct file_operations svc_setup_demand_fops = { + .owner = THIS_MODULE, + .open = svc_setup_demand_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* True when this module created the vgsp debugfs root (mirrors cuda bridge). */ +static bool svc_setup_demand_root_owned; + +static int svc_setup_demand_debugfs_init(void) +{ + struct dentry *root, *f; + + svc_setup_demand_root_owned = false; + root = debugfs_lookup("vgsp", NULL); + if (IS_ERR(root)) + root = NULL; + if (!root) { + root = debugfs_create_dir("vgsp", NULL); + if (IS_ERR_OR_NULL(root)) { + pr_warn("sema-svc: debugfs dir vgsp unavailable; sema_setup_demand not exposed\n"); + return -ENOENT; + } + svc_setup_demand_root_owned = true; + } + f = debugfs_create_file("sema_setup_demand", 0444, root, NULL, + &svc_setup_demand_fops); + if (IS_ERR_OR_NULL(f)) { + pr_warn("sema-svc: debugfs sema_setup_demand creation failed\n"); + if (svc_setup_demand_root_owned) + debugfs_lookup_and_remove("vgsp", NULL); + svc_setup_demand_root_owned = false; + return -ENOMEM; + } + dput(root); + pr_info("sema-svc: debugfs /sys/kernel/debug/vgsp/sema_setup_demand ready\n"); + return 0; +} + +static void svc_setup_demand_debugfs_fini(void) +{ + struct dentry *root; + + root = debugfs_lookup("vgsp", NULL); + if (IS_ERR_OR_NULL(root)) + return; + debugfs_lookup_and_remove("sema_setup_demand", root); + dput(root); + if (svc_setup_demand_root_owned) { + debugfs_lookup_and_remove("vgsp", NULL); + svc_setup_demand_root_owned = false; + } +} + +/* * Deferred readability probe for unreadable CE pushes that were consumed via * the fake-release fallback. The pump fakes immediately (work-before-release * cannot hold the channel); this queue re-attempts the read later in the @@ -1136,6 +2067,17 @@ static int svc_write(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, size_t len /* NV_ADDR_SYSMEM / NV_ADDR_FBMEM as carried in the channel alloc params. */ #define SVC_APERTURE_SYS 1 #define SVC_APERTURE_FB 2 +/* + * SESSION 153-CONTINUED-34: the emulated BAR0 (register) shadow as a third + * aperture. The GL client's channel allocation reports its USERD block in the + * FB aperture (NV_ADDR_FBMEM, offset 0x18fc00/0x18fe00/0xfbfc00...) but the + * writes the guest actually makes to its GP_PUT land in the emulated BAR0 + * shadow at that same offset (the vGPU guest driver programs its usermode + * block through the register window). Every other source reads the FB shadow + * at those offsets and finds nothing, so the pump never iterates the + * channel's GPFIFO and the channel retires with submissions=0. + */ +#define SVC_APERTURE_BAR0 3 /* * Aperture selectors are codes (1=SYS, 2=FB), never booleans. The idiom @@ -1147,7 +2089,31 @@ static int svc_write(struct nv_vgpu_device *vgpu, u64 gpa, void *buf, size_t len */ static inline u32 svc_ap_of(int ap) { - return ap == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + if (ap == SVC_APERTURE_FB) + return SVC_APERTURE_FB; + /* SESSION 153-CONT-34: the BAR0 selector must survive normalisation. */ + if (ap == SVC_APERTURE_BAR0) + return SVC_APERTURE_BAR0; + return SVC_APERTURE_SYS; +} + +/* + * SESSION 153-CONT-34: the emulated BAR0 shadow (the buffer the guest's + * register writes actually land in - it is mmap'd into QEMU and also the + * trap-path write target). NULL when the device has no emulation shadow. + */ +static void *svc_bar0_shadow(struct nv_vgpu_device *vgpu, size_t *size) +{ + struct nv_vgpu_bar_emul_ctx *bctx; + + if (!vgpu) + return NULL; + bctx = vgpu->bar_emul_ctx; + if (!bctx || !bctx->bar0_shadow) + return NULL; + if (size) + *size = bctx->bar0_shadow_size; + return bctx->bar0_shadow; } /* @@ -1167,6 +2133,16 @@ static int svc_read_ap(struct nv_vgpu_device *vgpu, u64 addr, u32 aperture, memcpy(buf, base + addr, len); return 0; } + /* SESSION 153-CONT-34: emulated BAR0 shadow (register window). */ + if (aperture == SVC_APERTURE_BAR0) { + size_t size = 0; + u8 *base = svc_bar0_shadow(vgpu, &size); + + if (!base || addr + len > size) + return -EINVAL; + memcpy(buf, base + addr, len); + return 0; + } return svc_read(vgpu, addr, buf, len); } @@ -1182,10 +2158,357 @@ static int svc_write_ap(struct nv_vgpu_device *vgpu, u64 addr, u32 aperture, memcpy(base + addr, buf, len); return 0; } + /* SESSION 153-CONT-34: emulated BAR0 shadow (register window). */ + if (aperture == SVC_APERTURE_BAR0) { + size_t size = 0; + u8 *base = svc_bar0_shadow(vgpu, &size); + + if (!base || addr + len > size) + return -EINVAL; + memcpy(base + addr, buf, len); + return 0; + } return svc_write(vgpu, addr, buf, len); } /* + * CONT.419 - GPFIFO submission ledger for the scrub wait release. + * + * The guest's PMA scrubber (mem_scrub.c) submits one CE work item per + * scrubbed range; each item gets a sequential 32-bit payload + * (ceutilsMemset: payload = lastSubmittedPayload + 1) and one GPFIFO entry + * in the scrubber channel's ring. The guest's wait is + * + * _scrubWaitAndSave(): while (_scrubCheckProgress() < lastSeen + items) + * + * where _scrubCheckProgress() folds the channel's finish-payload word. The + * release the host must author is therefore "fin := lastSubmittedWorkId". + * The guest-side hard invariant (r1766/r1767 RCA, re-measured CONT.419): + * + * itemsToSave = fin_fold_delta; _scrubCopyListItems(): + * lastSeenIdByClient += itemsToSave; + * scrubListSize -= itemsToSave; (UNGUARDED u64) + * + * so any fin value beyond lastSubmittedWorkId underflows scrubListSize to + * ~2^64 and the guest spins in _searchScrubList / scrubCheckAndWaitForSize + * under the scrubber mutex (measured live: fin=12 with lastSubmitted=11 -> + * scrubListSize=0xffffffffffffffff, guest soft-locked; and a blind +1 tick + * has no host-observable "release" edge, so it inevitably overshoots). + * fin must never exceed the guest's pending count. + * + * The submission ledger IS observable: the channel's GPFIFO ring sits + * exactly one GPFIFO (entries * 8 bytes; 4096 * 8 = 0x8000) below the + * sema/fin page the scanner registers, and each entry's low dword is the + * push-buffer GPU VA (CeUtils scrubber window 0x120000000..0x120080000 -> + * low 32 in [0x20000000, 0x20200000), 4-byte aligned). Measured live + * (CONT.419): ring at 0x126c6b000 = fin page 0x126c73000 - 0x8000 held + * exactly 11 valid entries while the guest's lastSubmittedWorkId was 11. + * Counting valid entries therefore yields a fin value that can never + * exceed the pending list, and advancing fin to it releases every pending + * wait exactly once (delta == pending -> scrubListSize lands on 0). + * + * Wrap safety: entries are overwritten in place after 4096 pushes, so the + * per-entry previous content is snapshotted (16 KB per active record) and + * every change into a valid entry counts as one new push, making the + * cumulative count exact across wraps. The snapshot is reset whenever the + * record's geometry (PA) changes and freed in gsp_sema_service_stop(). + */ +#define SVC_SCRUB_RING_ENTRIES 4096u +#define SVC_SCRUB_RING_BYTES (SVC_SCRUB_RING_ENTRIES * 8u) +#define SVC_SCRUB_RING_RATE_MS 250u +/* r1779 (V13): re-run the prefix/terminus validation at most this often. */ +#define SVC_SCRUB_RING_VALID_MS 500u +/* r1779 (V13): a scrub push must lie within this window of the GPFIFO VA. */ +#define SVC_SCRUB_RING_PUSH_WIN 0x1000000u + +/* + * Per-record entry snapshot (lo | hi<<32), lazily allocated, freed at + * service stop. The hi dword carries the GPFIFO flags/length, so the whole + * 8-byte entry is snapshotted and validated (r1779/V13). + */ +static u64 *svc_scrub_ring_snap[SVC_MAX_GUESTS * SVC_SCRUB_RECS]; + +/* Defined with the stash geometry resolver below; reused for the GPFIFO VA. */ +static bool svc_scrub_resolve_va(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa, int *ap, u32 *hvas); + +static inline bool svc_scrub_push_lo(u32 lo) +{ + return lo >= 0x20000000u && lo < 0x20200000u && !(lo & 3u); +} + +/* + * r1779 (V13): strict GPFIFO entry shape for the CeUtils scrubber ledger. + * The permissive svc_scrub_push_lo() above only recognises the low dword's + * 2 MiB push window, which a dense data page can satisfy. Real scrubber + * entries are {push_va, 0x00006001-class flags} (r1768b dump): require the + * hardware flag bits, a plausible length field, and a push VA in a bounded + * window around this channel's own GPFIFO VA. + */ +static bool svc_scrub_entry_strict(struct svc_scrub *s, u32 lo, u32 hi) +{ + s32 off; + u32 len; + + if ((hi & 3u) != 1u) + return false; + len = ((hi >> 10) & 0x1fffffu) << 2; + if (len < 4u || len > 0x10000u) + return false; + if (lo & 3u) + return false; + off = (s32)(lo - (u32)s->gpfifo_va); + if (off > (s32)SVC_SCRUB_RING_PUSH_WIN || + off < -(s32)SVC_SCRUB_RING_PUSH_WIN) + return false; + return true; +} + +/* + * r1779 (V13): resolve the channel's GPFIFO ring page through the GPU page + * tables instead of deriving it as "hsem page - entries*8". The old + * arithmetic is only correct when the channel buffer's GPFIFO and sema + * pages are PA-contiguous; the GPFIFO VA is a different PTE, and in the + * r1775/r1777 boots the subtraction could land on a stale/dense page. The + * walk here is the same one the stash geometry path already trusts. + */ +static bool svc_scrub_ring_page(struct nv_vgpu_device *vgpu, + struct svc_scrub *s, u64 *ring_pa) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u32 hv = 0; + + if (!s->gpfifo_va) + return false; + if (!svc_scrub_resolve_va(vgpu, s->gpfifo_va, &wpa, &wap, &hv) || !wpa) + return false; + if (svc_ap_of(wap) != SVC_APERTURE_SYS) + return false; + *ring_pa = wpa & ~0xfffull; + return true; +} + +/* + * r1779 (V13): first-read validation. The guest pushes sequentially, so a + * genuine ring is a contiguous strict-valid prefix followed by a fully-zero + * slot and at least the next 32 zero slots (the tail is only written after + * a wrap). Anything else - an all-zero page, a dense data page, a stale + * ring with a dirty tail - is not trusted as a submission ledger. + */ +static bool svc_scrub_ring_validate(struct svc_scrub *s, const u64 *snap, + u32 *count) +{ + u32 n = 0, z; + bool tail = true; + + while (n < SVC_SCRUB_RING_ENTRIES && + svc_scrub_entry_strict(s, (u32)snap[n], (u32)(snap[n] >> 32))) + n++; + for (z = n; z <= n + 32 && z < SVC_SCRUB_RING_ENTRIES; z++) { + if (snap[z]) { + tail = false; + break; + } + } + *count = n; + return n > 0 && n < SVC_SCRUB_RING_ENTRIES && tail; +} + +/* + * A scanner vmap candidate can resolve the channel GPFIFO ring page + * (channel base + 0x64000) and register it as if it were the sema/fin + * page. Authoring the all-ones release pair there overwrites GPFIFO entry + * 0 of the real channel (measured: entry 0 clobbered to 0xffffffff) and + * corrupts the submission ledger. A page is a ring page when another + * registered record's sema page sits exactly one GPFIFO above it. + */ +static bool svc_scrub_is_ring_page(struct svc_scrub *s) +{ + int i; + + if (s->userd_ok || !s->registered || !s->pa_ok) + return false; + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + struct svc_scrub *w = &svc_scrubs[i]; + u64 span; + + if (w == s || w->vgpu != s->vgpu || !w->pa_ok || !w->registered) + continue; + span = (u64)w->entries * 8u; + if (!span || span > SVC_SCRUB_RING_BYTES) + continue; + if ((s->hsem_pa & ~0xfffull) == + (w->hsem_pa & ~0xfffull) - span) + return true; + } + return false; +} + +static void svc_scrub_ring_reset(struct svc_scrub *s) +{ + s->ring_cum = 0; + s->ring_at = 0; + s->ring_valid_at = 0; + s->ring_ready = false; + s->ring_valid = false; + s->ring_page_ok = false; + s->ring_read_ok = false; +} + +/* + * Read the channel's GPFIFO ring and return the cumulative push count + * (== the guest's lastSubmittedWorkId). Rate-limited to one read per + * SVC_SCRUB_RING_RATE_MS; returns the last count otherwise, 0 when the + * ring is unreadable or the geometry is not ring-capable (entries unknown + * or larger than the scrubber's 4096-entry ring). + */ +static u32 svc_scrub_ring_cum(struct nv_vgpu_device *vgpu, struct svc_scrub *s) +{ + u64 *snap = svc_scrub_ring_snap[s - svc_scrubs]; + u64 ring_pa; + u8 *chunk; + u32 i; + bool ok = true; + + s->ring_read_ok = false; + if (!s->pa_ok || !s->entries || s->entries > SVC_SCRUB_RING_ENTRIES) + return s->ring_cum; + if (time_before(jiffies, + s->ring_at + msecs_to_jiffies(SVC_SCRUB_RING_RATE_MS))) + return s->ring_cum; + s->ring_at = jiffies; + if (!snap) { + snap = kzalloc(SVC_SCRUB_RING_ENTRIES * sizeof(u64), GFP_KERNEL); + if (!snap) + return s->ring_cum; + svc_scrub_ring_snap[s - svc_scrubs] = snap; + } + chunk = kmalloc(4096, GFP_KERNEL); + if (!chunk) + return s->ring_cum; + ring_pa = s->ring_pa; + for (i = 0; i < SVC_SCRUB_RING_ENTRIES; i += 512) { + u32 j; + + if (svc_read_ap(vgpu, ring_pa + (u64)i * 8u, + SVC_APERTURE_SYS, chunk, 4096)) { + ok = false; + break; + } + for (j = 0; j < 512; j++) { + u64 e = *(u64 *)(chunk + j * 8); + u32 k = i + j; + + /* + * Incremental counting only after the first read was + * validated: a new or overwritten (wrap) strict entry + * is one push. Before validation the snapshot only + * records content for the prefix check below. + */ + if (s->ring_valid && e != snap[k] && + svc_scrub_entry_strict(s, (u32)e, + (u32)(e >> 32))) + s->ring_cum++; + snap[k] = e; + } + } + kfree(chunk); + if (!ok) + return s->ring_cum; + s->ring_read_ok = true; + if (!s->ring_valid) { + u32 n = 0; + + /* + * Bound the validation cost: it reads only the in-memory + * snapshot, but it runs from the per-pass authoring path, so + * re-attempt at most every SVC_SCRUB_RING_VALID_MS. A fresh + * geometry (ring_ready=false) validates immediately. + */ + if (s->ring_ready && + time_before(jiffies, s->ring_valid_at + + msecs_to_jiffies(SVC_SCRUB_RING_VALID_MS))) + return s->ring_cum; + s->ring_valid_at = jiffies; + s->ring_ready = true; + if (!svc_scrub_ring_validate(s, snap, &n)) { + pr_info_ratelimited("sema-svc: scrub ring ledger disabled fin_pa=0x%llx ring_pa=0x%llx (prefix/terminus rejected n=%u)\n", + s->fin_pa, ring_pa, n); + s->ring_cum = 0; + return 0; + } + s->ring_cum = n; + if (!s->ring_logged) { + s->ring_logged = true; + pr_info("sema-svc: scrub ring ledger fin_pa=0x%llx ring_pa=0x%llx pushes=%u\n", + s->fin_pa, ring_pa, s->ring_cum); + } + s->ring_valid = true; + } + return s->ring_cum; +} + +/* + * CONT.419 fin advance: forward-only to the ring ledger count. Never + * fabricates progress - the count is the guest's own submissions, so the + * reported delta can never exceed the pending list (see the invariant + * above). With no observed pushes the word is left untouched. + * + * r1779 (V13) P0 guards: + * - a regressed fin word means the guest reset its scrubber object while + * the resolved DA stayed the same; re-applying the pre-reset absolute + * count underflows scrubListSize (r1775 c6: 4076 into an empty list -> + * 0xFFFFFFFFFFFFF014 -> scrubCheckAndWaitForSize spin, _nv052600rm+0x7b); + * - the ring value is used only when the GPFIFO page resolved this pass + * AND its first read passed the prefix/terminus validation this session. + */ +static u32 svc_scrub_fin_next_ring(struct nv_vgpu_device *vgpu, + struct svc_scrub *s, u32 cur) +{ + u64 ring_pa = 0; + u32 ring; + + if ((s32)(cur - s->fin_release) < 0 || + (cur == 0 && s->fin_release != 0)) { + if (s->ring_resets < 16) { + s->ring_resets++; + pr_info("sema-svc: scrub ring reset (fin regressed old=0x%x new=0x%x)\n", + s->fin_release, cur); + } + svc_scrub_ring_reset(s); + s->fin_release = cur; + return cur; /* no write this pass */ + } + if (!gsp_sema_scrub_ring) + return cur; + /* Match the ring read rate limit: no walk on a rate-limited pass. */ + if (time_before(jiffies, + s->ring_at + msecs_to_jiffies(SVC_SCRUB_RING_RATE_MS))) + return cur; + if (!svc_scrub_ring_page(vgpu, s, &ring_pa)) { + s->ring_page_ok = false; + pr_info_ratelimited("sema-svc: scrub ring ledger disabled (ring walk failed) h=0x%x va=0x%llx\n", + s->h_object, s->gpfifo_va); + return cur; + } + s->ring_page_ok = true; + s->ring_pa = ring_pa; + ring = svc_scrub_ring_cum(vgpu, s); + if (!s->ring_read_ok || !s->ring_valid) + return cur; /* unverified ring: no ledger */ + if (!ring || (s32)(ring - cur) <= 0) + return cur; /* no pushes, or forward-only */ + if (!s->ring_release_logged) { + s->ring_release_logged = true; + pr_info("sema-svc: scrub ring release fin_pa=0x%llx cum=%u (was 0x%x)\n", + s->fin_pa, ring, cur); + } + s->fin_release = ring; + return ring; +} + +/* * GR content-discovery epoch re-arm (FORENSICS-grv1 section 5 R1): a channel * registration means the guest (re)created channels - a fresh context attempt * with possibly fresh ring/push content - so an epoch that hit its pass cap @@ -1227,6 +2550,14 @@ static u32 svc_ring_pin_skips; * content-validated (entry0 must parse; a discovered base must carry this * channel's slot), so re-arming is safe and idempotent within a session. */ +/* + * SESSION 153-CONTINUED-76: the GR stale-ring heal timestamps live here so + * the channel-add path can force an immediate heal for a re-registered + * channel (see gsp_sema_service_channel_add). Declared before its use in + * the add path; the heal itself is defined with the ring-heal section. + */ +static unsigned long svc_gr_heal_at[32]; + static void svc_chan_rearm_rank(struct svc_channel *ch) { if (!ch || !ch->valid) @@ -1234,12 +2565,149 @@ static void svc_chan_rearm_rank(struct svc_channel *ch) if (ch->engine_type) { clear_bit(ch->ce_index, &svc_ring_done); clear_bit(ch->ce_index, &svc_pb_done); + clear_bit(ch->ce_index, &svc_ce_initrel_done); + svc_ce_initrel_until[ch->ce_index] = 0; + svc_ce_initrel_next[ch->ce_index] = 0; + svc_ce_initrel_val[ch->ce_index] = 0; } else { clear_bit(ch->gr_index, &svc_gr_ring_done); + /* + * r914: re-arm the ring-anchored window pairing too. The GL + * client re-registers with a NEW ring PA on every attempt, + * and with only the ring bit cleared the stale first pair + * (svc_pb_win_lookup returns the first slot match) kept + * answering the push reads with the previous attempt's + * pages - which the guest had since reused for kernel + * structures (measured r913: the push read resolved to a + * page of kernel pointers and every entry went unreadable). + */ + clear_bit(ch->gr_index, &svc_pb_gr_done); } } /* + * SESSION 153-CONTINUED-75: does the service hold a page directory for this + * VA-space handle? The channel-registration log uses this to name the + * discovery gap on the line itself (a nonzero channel hVASpace absent from + * svc.vas[] is exactly the "undiscovered VAS" condition the CONT-74 + * investigation looked for by hand). + */ +static bool svc_vaspace_known(u32 h_vaspace) +{ + unsigned long flags; + bool found = false; + int i; + + if (!h_vaspace) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].h_vaspace == h_vaspace) { + found = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return found; +} + +/* + * CONT.407: install a DUP_OBJECT'd VA-space handle as an alias of the + * source handle's directory. The driver gives a second client the same + * page directory under a new handle (measured: GL dups VAS 0x10005 as + * 0xbfef0001 and allocates its GR channel with hVASpace=0xbfef0001). The + * alias copies the source's pdb/entries/ap/verified state verbatim, so the + * channel-scoped lookup (svc_cur_vas) answers with the client's own + * directory instead of falling through to the newest-first global walk + * (r1702/r1703: gpfifo 0x1203cfff0 re-resolved to the bogus 0x1bcfff0). + * Returns false when the source is not a registered VAS; the caller then + * leaves DUP_OBJECT as a pure handle echo. + */ +bool gsp_sema_service_vaspace_alias(u32 h_dst, u32 h_src) +{ + struct sema_vaspace_desc d; + struct svc_vaspace snap; + unsigned long flags; + bool found = false; + int i; + + if (!h_dst || !h_src || h_dst == h_src) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].h_vaspace == h_src) { + snap = svc.vas[i]; + found = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + if (!found) + return false; + memset(&d, 0, sizeof(d)); + d.h_vaspace = h_dst; + d.pdb = snap.pdb; + d.num_entries = snap.entries; + d.b_vidmem = snap.ap == SVC_APERTURE_FB; + d.verified = snap.verified; + gsp_sema_service_vaspace_add(&d); + pr_info("sema-svc: VAS alias 0x%x -> 0x%x (pdb=0x%llx entries=%u verified=%d)\n", + h_dst, h_src, snap.pdb, snap.entries, snap.verified ? 1 : 0); + return true; +} + +/* + * CONT.400: is this VA space backed by a directory proven against the + * client's own content-validated ring (or published by RM)? Only then may + * the parked-candidate promote skip its walk. A directory that exists but is + * only UNVERIFIED (bounded age-out) must still be upgraded by content. + */ +static bool svc_vaspace_verified(u32 h_vaspace) +{ + unsigned long flags; + bool found = false; + int i; + + if (!h_vaspace) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].verified && + svc.vas[i].h_vaspace == h_vaspace) { + found = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return found; +} + +/* + * r1776 (V11): is the stored directory for this handle already this pdb + * AND verified? The candidate-promote success path re-walks kept + * candidates every sweep (the keep>=2 rule below); re-registering an + * unchanged verified directory would clear svc.lat and log the + * "VA space ... page directory" line every 100 ms, while an unverified + * stored entry (age-out) must still be upgraded. + */ +static bool svc_vaspace_pdb_match(u32 h_vaspace, u64 pdb) +{ + unsigned long flags; + bool found = false; + int i; + + if (!h_vaspace || !pdb) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].verified && + svc.vas[i].h_vaspace == h_vaspace && + svc.vas[i].pdb == pdb) { + found = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return found; +} + +/* * Register a guest channel. Called from the GSP-RM emulation when the guest * driver allocates a channel (the allocation carries the GPFIFO GPU VA, its * entry count and the guest-physical USERD/instance blocks), and dropped when @@ -1249,11 +2717,32 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) { struct svc_channel *ch = NULL; unsigned long flags; + u64 prev_notify2d_va = 0; + u32 prev_notify2d_gen = 0; int i; if (!desc || !desc->h_object || !desc->gpfifo_va || - desc->gpfifo_entries < 2) + desc->gpfifo_entries < 2) { + /* + * SESSION 153-CONTINUED-15 (draw attribution): a refused + * channel registration used to be silent, so a real client + * channel with an empty gpfifo VA looked identical to a + * channel that was never allocated. Name every refusal. + */ + static u32 refuse_log; + + if (refuse_log < 32) { + refuse_log++; + pr_info("sema-svc: chan add REFUSED h=0x%x gpfifo=0x%llx ent=%u eng=%u userd=0x%llx/%u\n", + desc ? desc->h_object : 0, + desc ? desc->gpfifo_va : 0, + desc ? desc->gpfifo_entries : 0, + desc ? desc->engine_type : 0, + desc ? desc->userd_pa : 0, + desc ? desc->userd_as : 0); + } return; + } if (!desc->engine_type) { WRITE_ONCE(svc_gr_content_rearm, true); svc_cepin_log = 0; @@ -1281,6 +2770,28 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) } } spin_lock_irqsave(&svc.chan_lock, flags); + /* + * r1652: the GL client's kick channel (0xbfef0002) is re-registered + * while the client keeps the class2d notification target it armed on + * the previous instance: its short kick push carries NOTIFY 0x10c + * without SET_NOTIFY_A/B. The old slot still holds the target after + * gsp_sema_service_channel_del() (valid=false only), but this add's + * memset() would erase it, leaving the short NOTIFY with no address: + * svc_notify2d_write() then drops the completion, no channel event is + * posted, and the client spins in its kick/fence wait. Carry the + * target across a same-session, same-handle re-registration only; a + * different session (or a fresh SET_NOTIFY_A/B) still wins. + */ + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].vgpu != desc->vgpu || + svc.chan[i].h_object != desc->h_object) + continue; + if (svc.chan[i].notify2d_va) { + prev_notify2d_va = svc.chan[i].notify2d_va; + prev_notify2d_gen = svc.chan[i].session_gen; + break; + } + } for (i = 0; i < SVC_MAX_CHANNELS; i++) { if (svc.chan[i].valid && svc.chan[i].vgpu == desc->vgpu && svc.chan[i].h_object == desc->h_object) { @@ -1355,6 +2866,19 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) ch->counter_gen = keep_counter_gen; } /* + * r1652: restore the class2d notification target saved before the + * memset. A short kick (NOTIFY only) then still has the address the + * guest programmed on the previous incarnation of this handle, and + * svc_notify2d_write() can author the completion again. The session + * generation check keeps a dead session's VA out of a new client; + * any SET_NOTIFY_A/B in the new incarnation overwrites it. + */ + if (prev_notify2d_va && prev_notify2d_gen == svc_sess_gen) { + ch->notify2d_va = prev_notify2d_va; + pr_info_ratelimited("sema-svc: channel 0x%x kept class2d notify va=0x%llx across re-registration\n", + ch->h_object, ch->notify2d_va); + } + /* * Rank among the live CE channels OF THIS GUEST. The RM hands * out object handles in creation order and the guest's UVM * allocates one push buffer, ring and tracking semaphore per @@ -1383,6 +2907,38 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) !svc.chan[k].engine_type && svc.chan[k].h_object < desc->h_object) rank++; + /* + * r1774 (V9): the count above is creation order among + * LIVE handles, so a channel whose handle sorts after a + * long-lived one collides with it (r1773 c31: the + * direct-GL GR channel 0xbeef0100 and the live kick + * channel 0xbfef0002 both got rank 2). The rank keys + * the one-shot ring-done bits and the autopin/scan + * gates, so the collision silently disabled the + * content-scan backup for the GL ring. If the counted + * rank is already held by a live GR channel, take the + * smallest unused rank instead (identical result in + * the normal creation-order case). + */ + { + u32 used = 0; + u8 r2; + + for (k = 0; k < SVC_MAX_CHANNELS; k++) + if (svc.chan[k].valid && + svc.chan[k].vgpu == desc->vgpu && + !svc.chan[k].engine_type && + svc.chan[k].gr_index < 32) + used |= 1u << + svc.chan[k].gr_index; + if (used & (1u << rank)) { + for (r2 = 0; r2 < 32; r2++) + if (!(used & (1u << r2))) + break; + if (r2 < 32) + rank = r2; + } + } ch->gr_index = rank; } ch->valid = true; @@ -1391,10 +2947,34 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) ch->h_object = desc->h_object; ch->engine_type = desc->engine_type; ch->cid = desc->cid; + ch->h_vaspace = desc->h_vaspace; ch->gpfifo_va = desc->gpfifo_va; ch->gpfifo_entries = desc->gpfifo_entries; ch->userd_pa = desc->userd_pa; ch->userd_as = desc->userd_as ? desc->userd_as : SVC_APERTURE_SYS; + /* CONT.405: keep the wire block for BAR1-alias resolution. */ + ch->userd_wire = ch->userd_as == SVC_APERTURE_FB ? + desc->userd_pa : 0; + /* + * r831: a registered userd value that looks like a real guest + * PA (the RC watchdog's channel carries its own userd block) + * is the driver's own declaration and outranks the ring+0x2000 + * convention guess the ring auto-pin applies (that guess is + * right for the CUDA GR layout, where the registered value is + * pool-relative - observed 0x10000). Tiny registered values + * stay guesses. + */ + /* + * CONT.402: a declared FBMEM userdMem is the driver's own + * control-page location and must be trusted even though its FB + * offset is small (the direct-GL client declares fb 0x160000). + * The old rule treated every value < 0x10000000 as a guess and + * fell through to the RAM scan, which adopted an unrelated + * channel's page (0x114371000) and read a phantom GP_PUT. + */ + ch->userd_reg = (desc->userd_as == SVC_APERTURE_FB && + desc->userd_pa != 0) || + desc->userd_pa >= 0x10000000ULL; ch->instance_pa = desc->instance_pa; ch->notifier_pa = desc->notifier_pa; ch->notifier_as = desc->notifier_as ? desc->notifier_as @@ -1425,6 +3005,18 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * the previous session's one-shot state (second-client wedge). */ svc_chan_rearm_rank(ch); + /* + * SESSION 153-CONTINUED-76: a re-registered GR channel starts + * the stale-ring heal immediately instead of waiting out the + * previous instance's 1 s gate. The client churns its channels + * every ~12 s; a ring pin inherited from the dead session (the + * session flush may have been raced by this registration, see + * the add-vs-flush skip) can keep answering with the previous + * ring's pages for the new instance, and the ~40 passes the + * USERD/FB sweeps need must not also wait on the heal. + */ + if (!ch->engine_type && ch->gr_index < 32) + svc_gr_heal_at[ch->gr_index] = 0; svc.last_chan_reg = jiffies; /* * CeUtils scrubber channel: NUM_COPY_BLOCKS = 4096 GPFIFO @@ -1436,12 +3028,26 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) * channel; the stashed geometry survives, so the scrub path * does not depend on the channel-table identity at all. */ - if (desc->gpfifo_entries >= 4096 && desc->gpfifo_va && + /* + * m112b: the CeUtils scrubber is the ONLY exactly-4096-entry + * channel (NUM_COPY_BLOCKS). The old `>= 4096` test also + * matched every client channel (this driver version: 32768 + * entries, not the 1024 the comment below assumed), so the + * single scrub slot bound to whichever client channel + * registered last (observed r701: the slot held h=0xbeee0102 + * userd=0x8f400/entries=32768 and the pass authored put=0 for + * the wrong channel while the real scrubber's payload stayed + * zero - the vkAllocateMemory -2 gate). + */ + if (desc->gpfifo_entries == 4096 && desc->gpfifo_va && desc->userd_pa) { struct svc_scrub *s = svc_scrub_get(desc->vgpu, true); if (!s) goto scrub_done; /* table full: max guests */ + pr_info("sema-svc: scrub stash bind vgpu=%px slot=%d h=0x%x entries=%u\n", + (void *)desc->vgpu, (int)(s - svc_scrubs), + desc->h_object, desc->gpfifo_entries); /* Re-seed the counting on a fresh scrubber object or a * new client session (m176: the counter/submitted state * is per-session; a recycled scrubber handle after a @@ -1453,6 +3059,14 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) s->gpfifo_va != desc->gpfifo_va) { s->seeded = false; s->submitted = 0; + /* + * r1668: a re-registered scrubber may reuse the + * same GPU VA with a new backing page. Force the + * finish-word resolution to run again instead of + * authoring the previous allocation's page. + */ + s->pa_ok = false; + s->authored_gen = 0; } s->session_gen = svc_sess_gen; s->userd_ok = true; @@ -1466,12 +3080,34 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc) desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, s->userd_as); + } else if (desc->gpfifo_entries >= 4096 && + desc->gpfifo_entries != 32768) { + /* m112b: a non-32768 channel with >=4096 entries that + * fails the stash condition (missing VA or USERD PA) + * is a scrubber candidate the service cannot author + * for - log it so a silent no-authoring boot is + * diagnosable. */ + pr_info_ratelimited("sema-svc: scrubber-stash MISS h=0x%x entries=%u va=0x%llx userd_pa=0x%llx\n", + desc->h_object, desc->gpfifo_entries, + desc->gpfifo_va, desc->userd_pa); } scrub_done: - pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx\n", + /* + * SESSION 153-CONTINUED-75: name the channel's declared VA + * space and whether the service has a page directory for it. + * hVASpace==0 is the driver's "attach to the TSG address + * space" (nv_gpu_ops.c: "If zero then it will attach to the + * TSG address space"), which is why the GR channel's + * low-aperture ring/push VAs resolve through the TSG/client + * VAS published by its CE siblings, not through a handle of + * its own. A nonzero handle that never appears in svc.vas[] + * names the discovery gap directly. + */ + pr_info("sema-svc: channel 0x%x registered gpfifo=0x%llx entries=%u userd=0x%llx eng=%u cid=%u notif=0x%llx vas=0x%x vas_known=%d\n", desc->h_object, desc->gpfifo_va, desc->gpfifo_entries, desc->userd_pa, desc->engine_type, desc->cid, - desc->notifier_pa); + desc->notifier_pa, desc->h_vaspace, + svc_vaspace_known(desc->h_vaspace)); } else pr_warn_ratelimited("sema-svc: channel 0x%x dropped: channel table full (%d)\n", @@ -1500,6 +3136,8 @@ static void svc_opdef_drop(struct nv_vgpu_device *vgpu, static void svc_opdef_reset_all(void); static void svc_opdef_finish_ch(struct svc_channel *ch, u32 idx); static u32 svc_opdef_log; /* m176: forward decl for the session flush */ +/* r633: per-session synthetic shadow budget reset (defined with the fill). */ +static void svc_gr_synth_ep_reset(void); /* * One-cycle GAP-1/GAP-2 instrumentation budgets (agent-datapath plan C1-C10): * small, reset per session where they are service-thread state, so a later @@ -1510,9 +3148,31 @@ static u32 svc_opdef_clr_log; /* C3: opdef-clear caller witness */ static u32 svc_busy_defer_log; /* C8: GR busy-defer timeout witness */ static u32 svc_hold_fb_log; /* C7: hold failure/timeout fallback witness */ static u32 svc_cand_ovw_log; /* C9: candidate overwrite witness */ +static u32 svc_gr_push_dump; /* SESSION 127: bounded capture tail dump + * (sema_dbg only, 4 per boot) */ static u32 svc_inline_recs_log; /* C10: inline-recs digest witness */ static u32 svc_gr_await_log; /* r351: launch-held-for-trust witness */ static u32 svc_gr_retry_log; /* r351: retry-forward/proceed witness */ +static bool svc_hold_bypass(void); /* r563: stall self-heal forward decl */ + +/* + * r551: releases DELIVERED by the opdef hold-expiry paths (flush/sweep) are + * remembered by (channel, entry, va, payload) for a short window, so a + * re-decode of the parked entry does not withhold the (monotonic, + * forward-only) release again - see the helpers with the opdef machinery. + * Declared here because svc_session_flush() resets them per session. + */ +#define SVC_OPDEF_DONE_SLOTS 256 +#define SVC_OPDEF_DONE_TTL_MS 60000u +struct svc_opdef_done_rec { + u32 h_object; + u32 entry_idx; + u64 va; + u32 payload; + unsigned long j; +}; +static struct svc_opdef_done_rec svc_opdef_done[SVC_OPDEF_DONE_SLOTS]; +static u32 svc_opdef_done_next; void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) { @@ -1581,6 +3241,406 @@ void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object) * the page tables itself and hands their base address to the GPU). Store it so * the service can translate that space's VAs directly. */ +/* + * r496j: unverified page-directory candidates from COPY_SERVER_PDES. A + * kernel-RM internal VAS publishes its chain there, but registering every + * client's directory directly lets a stale kernel directory answer another + * client's VA in the walker's first-success order (the r459 stale-directory + * class; observed as a CUDA client hanging after DEVICE). Candidates are + * parked here and registered only after a walk of the client's own + * content-validated ring VA through the candidate reproduces that ring's PA + * (svc_gr_ram_ring_scan's latch calls svc_vas_candidates_verify). + */ +#define SVC_VAS_CAND_MAX 16 +static struct { + bool valid; + u32 h_vaspace; + u64 pdb; + u32 entries; + bool vidmem; + bool aged_out; /* CONT.401: registered-once guard for the age-out */ + unsigned long stamp; /* r537: park time for the bounded fallback */ +} svc_vas_cand[SVC_VAS_CAND_MAX]; + +static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, + u64 va, int tap, u64 *addr, int *ap, bool trace); + +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem) +{ + int i; + + if (!pdb || entries < 2 || entries > 4096) + return; + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (svc_vas_cand[i].valid && + svc_vas_cand[i].h_vaspace == h_vaspace && + svc_vas_cand[i].pdb == pdb) + return; /* already parked */ + if (!svc_vas_cand[i].valid) { + svc_vas_cand[i].valid = true; + svc_vas_cand[i].h_vaspace = h_vaspace; + svc_vas_cand[i].pdb = pdb; + svc_vas_cand[i].entries = entries; + svc_vas_cand[i].vidmem = vidmem; + svc_vas_cand[i].aged_out = false; /* CONT.401: recycled slot */ + svc_vas_cand[i].stamp = jiffies; + return; + } + } + /* + * r1774 (V10): the table is 16 slots and the direct-GL client rebuilds + * its page directory every few seconds (r1774 c1: 19 distinct pdbs for + * VAS 0xbeef0300 alone, 26 distinct parks overall). Without eviction + * the first 16 parks filled the table and every FRESH pdb - the one + * that maps the current ring - was silently dropped; once the + * registered directory went stale the promote walk had nothing left + * to try ("cand ... walk rc=-2", "NOT mapped by any registered VAS") + * and the client stalled. Evict the OLDEST candidate of the same VAS + * (its directory is the one being rebuilt); if the VAS has no slot + * (all 16 belong to other VASes), evict the globally oldest one. + */ + { + int victim = -1; + unsigned long oldest = 0; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (!svc_vas_cand[i].valid || + svc_vas_cand[i].h_vaspace != h_vaspace) + continue; + if (!oldest || + time_before(svc_vas_cand[i].stamp, oldest)) { + oldest = svc_vas_cand[i].stamp; + victim = i; + } + } + if (victim < 0) + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + if (!svc_vas_cand[i].valid) + continue; + if (!oldest || + time_before(svc_vas_cand[i].stamp, + oldest)) { + oldest = svc_vas_cand[i].stamp; + victim = i; + } + } + if (victim >= 0) { + svc_vas_cand[victim].valid = true; + svc_vas_cand[victim].h_vaspace = h_vaspace; + svc_vas_cand[victim].pdb = pdb; + svc_vas_cand[victim].entries = entries; + svc_vas_cand[victim].vidmem = vidmem; + svc_vas_cand[victim].aged_out = false; + svc_vas_cand[victim].stamp = jiffies; + } + } +} + +/* + * CONT.401: a client restart rebuilds its page directory under the SAME VAS + * handle (observed: X VAS 0x10005 pdb 0x12a82f000 -> 0x14b259000). The + * CONT.400 "never downgrade verified" guard then pins the previous session's + * directory forever: the pump serves stale ring/push pages, consumes zero + * entries, and the DDX pushbuffer NOP test times out ("Failed to initialize + * DMA" / "Failed to allocate push buffer"). Return true when a parked + * candidate names a different directory for this handle, i.e. the stored + * verified directory is stale. + */ +static bool svc_vaspace_stale_candidate(u32 h_vaspace) +{ + unsigned long flags; + u64 stored = 0; + bool stale = false; + int i; + + if (!h_vaspace) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].verified && + svc.vas[i].h_vaspace == h_vaspace) { + stored = svc.vas[i].pdb; + break; + } + if (stored) + for (i = 0; i < SVC_VAS_CAND_MAX; i++) + if (svc_vas_cand[i].valid && + svc_vas_cand[i].h_vaspace == h_vaspace && + svc_vas_cand[i].pdb != stored) { + stale = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return stale; +} + +/* CONT.401: drop the verified mark so a rebuilt directory can install. */ +static void svc_vaspace_demote(u32 h_vaspace) +{ + unsigned long flags; + int i; + + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].h_vaspace == h_vaspace) { + svc.vas[i].verified = false; + break; + } + memset(svc.lat, 0, sizeof(svc.lat)); + spin_unlock_irqrestore(&svc.vas_lock, flags); +} + +/* + * r537 bounded fallback: register a parked candidate that has waited for + * its content-verified ring walk longer than SVC_VAS_CAND_AGE_MS. The + * verification (svc_vas_candidates_verify) needs the client's ring to be + * found by the RAM content scan; on a boot where that scan never anchors + * the ring (observed: pd_found=0 and every launch operand walk rc=-2 for + * minutes, the client spinning with its data VAs unresolvable) the + * candidate stays parked forever and the client can never proceed. The + * candidate itself came from the client's own RM publish (its VAS + * handle), and the r459 newest-first walk order keeps an older stale + * directory from winning, so after the bounded wait the registration is + * the lesser evil; it is logged loudly for the record. + */ +#define SVC_VAS_CAND_AGE_MS 15000u +/* + * r705h: the candidate age-out deadline is runtime-tunable. The parked + * COPY_SERVER_PDES candidate for the GL/Vulkan client ages out (UNVERIFIED) + * at exactly the moment the client's channels first register, so the + * content-verified walk can never anchor it; raising the deadline keeps the + * candidate parked until a host tool supplies a verified ring pair via + * sema_pdb_discover. 0 disables the age-out entirely. + */ +static u32 svc_vas_cand_age_ms = SVC_VAS_CAND_AGE_MS; +module_param_named(sema_vas_cand_ms, svc_vas_cand_age_ms, uint, 0644); + +/* + * SESSION 127 C5C0 translator gate. When 1, the sweep additionally + * translates continuation entries into launch work (tail-aware method + * parse, inline-QMD capture, LOAD_INLINE_DATA materialization, C5B5 + * copy-out capture, candidate-owned release withholding). + * + * SESSION 130 re-test of the r727 "translator regressions" (translator on, + * utility sweep OFF, r746): victory_saxpy and victory_math now PASS with + * checksums identical to the gate-off baseline (saxpy + * csum=981440890.500000), so the saxpy/math half of the r727 regression + * is FIXED (the SESSION 129/130 sweep changes); victory_mail PASSES. The + * remaining xlate-on regression is victory_multikernel: its launch is + * captured and submitted to the physical channel but the forward times + * out repeatedly (`GR LAUNCH FAILED: timeout after 2000 ms + * (qmd_rel=0xc0ffee01 want=0x600df00d ...) h=0x5c000019 + * push=0x20043c394`, `hold active`), so the client hangs - the default + * stays 0 until that forward path is fixed. The UTILITY SWEEP + * (`sema_gr_util_scan`) is the OTHER regression source: at 24 it broke + * saxpy (`bad=1047527`) and hung math on r745, so it stays opt-in + * (default 0) until its channel reads prefer the direct walk over pins. + */ +static bool gsp_sema_gr_xlate; +module_param_named(sema_gr_xlate, gsp_sema_gr_xlate, bool, 0644); +MODULE_PARM_DESC(sema_gr_xlate, + "C5C0 translator: sweep continuation entries into launch work (default 0 = releases-only sweep)"); +/* + * SESSION 128 utility-channel inline-download materialization + * (sema_gr_util_scan, default 0 = off): the driver's non-GR upload + * channels (the MEM2MEM/A140 + 2D utilities, registered but with an + * unresolved USERD so the pump's put-bounded entry loop never decodes + * them) carry the engine-side code/data downloads: the + * GRCE/INLINE_TO_MEMORY operand group on the mem2mem subchannel (0x180.. + * 0x18c), LAUNCH_DMA (0x1b0) and a LOAD_INLINE_DATA (0x1b4) run whose + * payload the engine fetches from the next GPFIFO entry (the + * source-view call). Without this pass those destinations stay zero in + * the vidmem shadow - observed: the Vulkan client's code-heap/SASS + * upload (12 KB) and command-stream staging (256 B) never land, so the + * forwarded launch has no program content. Bounded: scan up to N + * entries per channel per pass, materialize once-only keyed by + * (push_va, content hash, dst, bytes) via the same + * svc_grce_inline_write the GR sweep uses. Requires sema_gr_xlate. + */ +static u32 gsp_sema_gr_util_scan; +module_param_named(sema_gr_util_scan, gsp_sema_gr_util_scan, uint, 0644); +MODULE_PARM_DESC(sema_gr_util_scan, + "Utility-channel i2m sweep: entries scanned per pass (default 0 = off; requires sema_gr_xlate)"); +MODULE_PARM_DESC(sema_vas_cand_ms, "Parked-VAS-candidate age-out (ms, default 15000; 0 = never)"); + +static void svc_vas_candidates_age_out(struct nv_vgpu_device *vgpu) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (!svc_vas_cand_age_ms || + time_before(jiffies, svc_vas_cand[i].stamp + + msecs_to_jiffies(svc_vas_cand_age_ms))) + continue; + /* + * CONT.401: register the fallback directory exactly once. + * Re-registering (and re-logging) on every pump pass flooded + * dmesg at ~50 Hz with sema_vas_cand_ms=1 and kept the pump + * busy, which starved the ring content scan. The candidate + * stays parked (valid=1) for the verified upgrade path. + */ + if (svc_vas_cand[i].aged_out) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + /* + * CONT.400: register the parked directory as UNVERIFIED so the + * client can make progress, but KEEP the candidate parked. The + * age-out previously cleared it, which meant that once the + * 1 ms timer fired the content-verified upgrade path + * (svc_chan_vas_candidate_promote) had nothing left to walk - + * exactly why the beef GR ring (VAS 0xbeef0300) stayed + * ring_pa=0 for whole boots. Bumping the stamp only re-arms the + * bounded wait; the candidate stays available for promotion. + */ + pr_warn("sema-svc: VAS 0x%x page directory registered UNVERIFIED after %u ms parked (pdb=0x%llx entries=%u; the content-verified ring walk never anchored this boot)\n", + d.h_vaspace, svc_vas_cand_age_ms, d.pdb, + d.num_entries); + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].aged_out = true; + svc_vas_cand[i].stamp = jiffies; + } +} + +/* + * r1777 (V12): parked-directory L0 sanity, the directory-incarnation gate. + * + * The direct-GL client rebuilds its VAS every few seconds (~19 pdbs per + * cycle parked under churning handles). A parked pdb whose page the guest + * kernel has since freed and reused reads kernel pointers (r1776: + * L0=0xffff8a0f8ea410a8) and its walk answers fabricated mappings - the + * rc=-6 "mapped but all-zero" ring page class came through exactly such a + * directory. Screen the L0 before walking: at least one plausible PDE + * (valid bit set, not a kernel-pointer pattern); an all-zero or + * kernel-pointer directory is skipped, so the freshest sane incarnation + * wins. An unreadable directory stays "unknown" and is walked as before - + * this gate can only reject positive evidence of a freed directory. + */ +static bool svc_pdb_l0_sane(struct nv_vgpu_device *vgpu, u64 pdb, int ap) +{ + u64 ent[64]; + bool any = false; + bool hostile = false; + int i; + + if (svc_read_ap(vgpu, pdb, ap, ent, sizeof(ent))) + return true; + for (i = 0; i < ARRAY_SIZE(ent); i++) { + if (!ent[i]) + continue; + if ((ent[i] >> 48) == 0xffffu) { + hostile = true; + continue; + } + if (ent[i] & 1u) + any = true; + } + /* + * Reject only on positive evidence of a freed/reused directory: + * kernel-pointer garbage with NO plausible PDE anywhere in the first + * 512 bytes. A directory that has valid PDEs passes even if stale + * entries linger in its unused slots (r1777: the strict + * reject-on-first-kernel-pointer variant skipped real directories). + */ + return any || !hostile; +} + +/* + * r1780 (V35): the registered verified directory can be FREED by the guest + * when it rebuilds its VAS; the page then reads all-zero (measured c36: + * pdb 0x11d67f000 was valid at t0 and zero seconds later). The beef family + * is exempt from the stale-candidate demote below to avoid churn, so a + * dead verified entry blocked every fresh parked candidate and the ring/ + * push walks regressed to rc=-2. Detect the all-zero page and demote it so + * the promotion pass can install the freshest content-verified directory. + */ +static bool svc_vas_registered_pdb(u32 h_vaspace, u64 *pdb, int *ap) +{ + unsigned long flags; + bool ok = false; + int i; + + if (!h_vaspace) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].verified && + svc.vas[i].h_vaspace == h_vaspace) { + *pdb = svc.vas[i].pdb; + *ap = svc.vas[i].ap; + ok = true; + break; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + return ok; +} + +static bool svc_pdb_reads_zero(struct nv_vgpu_device *vgpu, u64 pdb, int ap) +{ + u64 ent[64]; + int i; + + if (!pdb) + return true; + if (svc_read_ap(vgpu, pdb, ap, ent, sizeof(ent))) + return true; + for (i = 0; i < ARRAY_SIZE(ent); i++) + if (ent[i]) + return false; + return true; +} + +/* + * Verify each parked candidate against one content-validated translation + * (the ring VA -> PA the RAM scan just proved): walk the ring page through + * the candidate and register only the directory that reproduces it. Runs on + * the service thread from the ring-scan latch; the walk itself is cheap (a + * few table reads). + */ +static void svc_vas_candidates_verify(struct nv_vgpu_device *vgpu, u64 va, + u64 pa) +{ + int i; + + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = 0; + int ap = svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS; + struct sema_vaspace_desc d; + + if (!svc_vas_cand[i].valid) + continue; + if (gsp_sema_ce_ram && + !svc_pdb_l0_sane(vgpu, svc_vas_cand[i].pdb, ap)) + continue; /* r1777 (V12): freed/reused directory */ + if (svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, va & ~0xfffULL, + ap, &wpa, &wap, false) || + (wpa & ~0xfffULL) != (pa & ~0xfffULL)) + continue; + memset(&d, 0, sizeof(d)); + d.h_vaspace = svc_vas_cand[i].h_vaspace; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + d.verified = true; /* CONT.400: verified ring walk */ + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + pr_info("sema-svc: VAS 0x%x page directory registered after verified ring walk (pdb=0x%llx entries=%u ap=%d)\n", + d.h_vaspace, d.pdb, d.num_entries, ap); + break; + } +} + void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) { struct svc_vaspace *vs = NULL; @@ -1599,12 +3659,53 @@ void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc) if (!vs && !svc.vas[i].valid) vs = &svc.vas[i]; } + /* + * r459: the table is 8 slots and the guest driver does not always + * send a teardown for a client's VA space (observed: 23 distinct + * registrations, 0 drops in one boot). Once all slots hold a dead + * client's directory, a NEW client's registration found no slot and + * every walk of its VAs failed (copies rc=-2, launches reading + * stale pages - the 10th-client mid-boot degradation). Evict the + * least-recently-registered entry instead of refusing the new one; + * the old client is gone and its directory pointer is dead guest + * memory, so the eviction can only remove stale authority. + */ + if (!vs) { + unsigned long oldest = 0; + int oi = 0; + + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (!svc.vas[i].valid) + continue; + if (!oldest || time_before(svc.vas[i].stamp, oldest)) { + oldest = svc.vas[i].stamp; + oi = i; + } + } + vs = &svc.vas[oi]; + pr_info("sema-svc: VA space table full; evicting 0x%x (pdb=0x%llx) for 0x%x\n", + vs->h_vaspace, vs->pdb, desc->h_vaspace); + } if (vs) { + /* + * CONT.400: never downgrade a verified directory with an + * unverified one. The bounded age-out re-registers the last + * parked candidate every pass; if that candidate is a stale + * PDB and we let it overwrite the verified one, the ring + * translation regresses to zero after it was working. + */ + if (vs->valid && vs->verified && !desc->verified) { + vs->stamp = jiffies; + spin_unlock_irqrestore(&svc.vas_lock, flags); + return; + } vs->valid = true; + vs->verified = desc->verified; vs->h_vaspace = desc->h_vaspace; vs->pdb = desc->pdb; vs->entries = desc->num_entries; vs->ap = desc->b_vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + vs->stamp = jiffies; /* A new directory invalidates what we learned from the old one. */ memset(svc.lat, 0, sizeof(svc.lat)); } @@ -1733,7 +3834,7 @@ static bool svc_ap_ok(struct nv_vgpu_device *vgpu, u64 addr, int ap) { if (ap == SVC_APERTURE_FB) return vgpu->bars[1].vaddr && addr + 8 <= svc_fb_shadow_size(vgpu); - return addr + 8 <= SEMA_PA_MAX; + return addr + 8 <= svc_pa_max(); } static int svc_read_ent(struct nv_vgpu_device *vgpu, u64 table, u64 off, @@ -1908,8 +4009,171 @@ static int svc_gmmu_walk(struct nv_vgpu_device *vgpu, u64 pdb, u32 entries, } /* + * r1780 (V34): bounded FB/RAM page-directory scan for a content-verified + * pair. The direct-GL client's VAS directory is not in the parked + * COPY_SERVER_PDES candidates on every boot (the retired page reads zero), + * yet its page tables can live in the emulated FB aperture. Walk every + * 4 KB-aligned candidate root in the FB shadow (and, optionally, in guest + * RAM up to a caller bound) for `va` and accept the root that reproduces + * the caller's content-verified PA. On a hit register the directory under + * the caller's VAS handle so the ordinary push/VAS walk resumes. + * + * This is the deterministic replacement for guessing a PDB: the pair is + * content truth (a GPFIFO ring page or a push page the guest wrote), not a + * heuristic. Bounded and read-only; a wrong root costs one failed walk. + */ +int gsp_sema_service_pdb_scan(u64 va, u64 pa, u32 h_vas, u64 ram_lim) +{ + struct nv_vgpu_device *vgpu; + u64 off, lim; + int ap_in, hits = 0; + + if (!va || !pa || !h_vas) + return -EINVAL; + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + for (ap_in = SVC_APERTURE_FB; ap_in >= SVC_APERTURE_SYS; + ap_in--) { + if (ap_in == SVC_APERTURE_FB) { + if (!vgpu->bars[1].vaddr) + continue; + /* full per-type shadow (512 MiB), not the + * old 64 MiB cap: the client's vidmem + * allocations reach the top of it. */ + lim = svc_fb_shadow_size(vgpu); + } else { + if (!ram_lim) + continue; + lim = min_t(u64, ram_lim, svc_pa_max()); + } + for (off = 0; off + SEMA_PAGE_SIZE <= lim; + off += SEMA_PAGE_SIZE) { + u64 addr = 0; + int ap = ap_in; + + if (svc_gmmu_walk(vgpu, off, 4, va, ap_in, + &addr, &ap, false)) + continue; + if ((addr & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + { + struct sema_vaspace_desc d = { + .h_vaspace = h_vas, + .pdb = off, + .num_entries = 4, + .b_vidmem = + ap_in == SVC_APERTURE_FB, + .verified = true, + }; + + gsp_sema_service_vaspace_add(&d); + } + pr_info("sema-svc: pdb-scan HIT h=0x%x pdb=0x%llx entries=4 ap=%d for va=0x%llx -> pa=0x%llx (verified)\n", + h_vas, off, ap_in, va, addr); + hits++; + if (hits >= 4) + break; + } + if (hits >= 4) + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (!hits) + pr_info("sema-svc: pdb-scan no root maps va=0x%llx -> pa=0x%llx (h=0x%x, fb=%s ram=0x%llx)\n", + va, pa, h_vas, + "scanned", ram_lim); + return hits ? 0 : -ENOENT; +} +EXPORT_SYMBOL_GPL(gsp_sema_service_pdb_scan); + +/* r1780 (V34): direct VAS registration for a directory found out-of-band + * (e.g. by the pdb scan above or an offline walk). "h,pdb,entries,vidmem". */ +void gsp_sema_service_vas_force(u32 h_vas, u64 pdb, u32 entries, bool vid) +{ + struct sema_vaspace_desc d = { + .h_vaspace = h_vas, + .pdb = pdb, + .num_entries = entries ? entries : 4, + .b_vidmem = vid, + .verified = true, + }; + + gsp_sema_service_vaspace_add(&d); + pr_info("sema-svc: vas-force h=0x%x pdb=0x%llx entries=%u ap=%d\n", + h_vas, pdb, entries ? entries : 4, vid ? 2 : 1); +} +EXPORT_SYMBOL_GPL(gsp_sema_service_vas_force); + +/* * Page directories the driver published (SET_PAGE_DIRECTORY). */ +/* + * r1648: collect EVERY published VAS's translation of one VA. The + * first-success walk (svc_walk_vaspaces) can pick a stale/unverified + * directory that maps a UVM pool VA to the old (reused) page while the + * live directory maps it to the page the guest actually polls. For the + * forward-only pool counter the safe fix is to write the value through + * every distinct translation: a stale semaphore page absorbs a harmless + * duplicate, and whichever page the guest reads gets the value. Used only + * by the pool release/tracker writers, never by the general resolver. + */ +static int svc_walk_vaspaces_collect(struct nv_vgpu_device *vgpu, u64 va, + u64 *pas, int *aps, int max) +{ + unsigned long flags; + struct svc_vaspace local[SVC_MAX_VASPACES]; + int i, n = 0, count = 0; + + if (!pas || !aps || max <= 0) + return 0; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (svc.vas[i].valid) + local[n++] = svc.vas[i]; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + int j = i - 1; + + while (j >= 0 && time_before(local[j].stamp, k.stamp)) { + local[j + 1] = local[j]; + j--; + } + local[j + 1] = k; + } + for (i = 0; i < n && count < max; i++) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS, alt, j; + bool hit = false; + + if (va >> SVC_MMU_VA_BITS) + break; + alt = local[i].ap == SVC_APERTURE_FB ? SVC_APERTURE_SYS + : SVC_APERTURE_FB; + if (!svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, va, + local[i].ap, &wpa, &wap, false)) + hit = true; + else if (!svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, + va, alt, &wpa, &wap, false)) + hit = true; + if (!hit || !wpa) + continue; + for (j = 0; j < count; j++) + if ((pas[j] & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (wpa & ~(u64)(SEMA_PAGE_SIZE - 1))) + break; + if (j < count) + continue; + pas[count] = wpa; + aps[count] = wap; + count++; + } + return count; +} + static int svc_walk_vaspaces(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { @@ -1932,6 +4196,26 @@ static int svc_walk_vaspaces(struct nv_vgpu_device *vgpu, u64 va, } spin_unlock_irqrestore(&svc.vas_lock, flags); + /* + * r459: try the NEWEST registration first. A dead client's stale + * directory (the table outlives clients; only 8 slots) can still + * resolve a VA - through guest memory the driver has since reused - + * to a wrong page, and the first success wins. Ordering by the + * registration stamp makes the live client's directory the + * authority and only falls through to older ones when it cannot + * answer. + */ + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + int j = i - 1; + + while (j >= 0 && time_before(local[j].stamp, k.stamp)) { + local[j + 1] = local[j]; + j--; + } + local[j + 1] = k; + } + for (i = 0; i < n; i++) { int alt; @@ -2057,8 +4341,23 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) } if (n < SVC_PD_MIN_HITS) { n = 0; - if (svc.map_n < SVC_PD_SAMPLES) - return; /* no evidence yet: retry when pairs appear */ + /* + * r510: a LARGE map is a mixed set of VA spaces cached over + * the whole guest boot (the comment above already names the + * consequence: no single directory reproduces a mixed set), so + * scanning with its stride subsample burns the attempts budget + * on evidence that cannot succeed - and by the time the + * push-buffer pairs (the strongest samples) exist, the budget + * is spent (r508/r509 witness: 'not found (base=0x0 ap=ram + * cands=39621 best=0/8)' during the first client, eight RAM + * walks consumed, every later client's high argument pointers + * unresolvable - the terminal class). Only a SMALL map, which + * is one session's own pairs, still falls back; otherwise wait + * for the push-buffer pairs and keep the attempts. + */ + if (svc.map_n > (u32)(SVC_PD_SAMPLES * 2) || + svc.map_n < SVC_PD_SAMPLES) + return; /* no coherent evidence yet: retry later */ step = svc.map_n / SVC_PD_SAMPLES; for (i = 0; i < SVC_PD_SAMPLES; i++) { sva[i] = svc.map[i * step].gpu_page; @@ -2094,7 +4393,7 @@ static void svc_pd_discover(struct nv_vgpu_device *vgpu) * matters is the first one of the page. */ for (pass = 0; pass < 2 && !svc_pd_found; pass++) { int ap = pass == 0 ? SVC_APERTURE_SYS : SVC_APERTURE_FB; - u64 limit = pass == 0 ? SEMA_PA_MAX : svc_fb_shadow_size(vgpu); + u64 limit = pass == 0 ? svc_pa_max() : svc_fb_shadow_size(vgpu); u8 *fb = (u8 *)vgpu->bars[1].vaddr; u64 gpa; @@ -2194,8 +4493,8 @@ static void svc_pd_ring_discover(struct nv_vgpu_device *vgpu) buf = kmalloc(chunk, GFP_KERNEL); if (!buf) return; - for (gpa = 0; gpa < SEMA_PA_MAX && !svc_pd_r_found; gpa += chunk) { - u32 len = (u32)min_t(u64, chunk, SEMA_PA_MAX - gpa); + for (gpa = 0; gpa < svc_pa_max() && !svc_pd_r_found; gpa += chunk) { + u32 len = (u32)min_t(u64, chunk, svc_pa_max() - gpa); u64 off; if (svc_read(vgpu, gpa, buf, len)) @@ -2290,6 +4589,261 @@ static void svc_lat_store(u64 va, u64 addr, int ap) } /* + * SESSION 153-CONTINUED-67: invalidate a learned-cache entry so the next + * lookup of this page re-runs the source chain underneath it (discovered + * map, registered-VAS walk, discovered walk, targeted resolve) instead of + * reusing a cached answer. Used by the repeated all-zero read + * invalidation in svc_read_va_push_ex: every walk success re-stores the + * answer here (svc_lat_store after each svc_lookup_trusted sub-source), + * so a stale walk result otherwise shadows a fresh walk for the rest of + * the boot (r1127: the beef setup push kept answering the same zero page + * while the client's content was live elsewhere). + */ +static void svc_lat_drop(u64 va) +{ + struct svc_lat *l = &svc.lat[(va >> SEMA_PAGE_SHIFT) % SVC_LAT_MAX]; + + if (l->valid && + (l->va & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (va & ~(u64)(SEMA_PAGE_SIZE - 1))) + l->valid = false; +} + +/* + * r820: driver-declared kernel-buffer translations (PROMOTE_CTX). + * + * The guest RM announces the physical backing of a channel's context + * buffers through NV2080_CTRL_CMD_GPU_PROMOTE_CTX: every entry carries + * (gpuVirtAddr, gpuPhysAddr, size, physAttr, bufferId). These are + * KERNEL-owned buffers - the nvidia-push pushbuffer, the progress-tracker + * segment, the FECS/patch/access-map blocks - so the userspace pin + * scanner can never cover them (/proc/<pid>/pagemap has no kernel VAs) + * and the page-table walk can answer with a stale/wrong page. Observed + * r817 (Xorg GL init): the walk resolved the 64 KB pushbuffer VA + * 0x120010000 to pa=0x103fed000, a page reading all zeros, while the + * driver's own promote entry said pa=0x11e120000. The NOP sanity push of + * __nvPushTestPushBuffer then never decoded, GP_GET never advanced and + * the X server died with "Failed to initialize DMA". + * + * The registry is a small table keyed by the entry VA, newest wins: a + * client that re-promotes the same buffer - observed between two Xorg + * starts, the same channel object announced a NEW pa for the same VA - + * replaces the old record, so a stale backing is never preferred over + * the live statement. VA-only bookings (bInitialize=0 re-promotes of + * already-initialized buffers, zero PA/size) and bNonmapped entries + * (va=0) carry no backing and must not erase the last statement. + * + * Lookups run before the learned cache in svc_lookup_trusted: the wrong + * r817 walk answer had been cached in the learned table (svc_lat_store + * after every walk success), so a promote hit must both pre-empt and + * overwrite that entry; the hit re-stores the corrected page into the + * learned table so the direct svc_lat_lookup callers heal too. + */ +#define SVC_PROMO_MAX 256 +struct svc_promo_rec { + u64 va; + u64 pa; + u64 size; + unsigned long seq; + u32 phys_attr; + u16 buffer_id; + bool valid; +}; +static struct svc_promo_rec svc_promo[SVC_PROMO_MAX]; +static unsigned long svc_promo_seq; +static DEFINE_SPINLOCK(svc_promo_lock); +static u32 svc_promo_hit_log; /* bounded "answered from PROMOTE_CTX" witness */ + +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id) +{ + struct svc_promo_rec *rec = NULL, *free_rec = NULL, *oldest = NULL; + unsigned long flags; + int i; + + if (!va || !pa || !size) + return; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + if (svc_promo[i].valid && svc_promo[i].va == va) { + rec = &svc_promo[i]; + break; + } + if (!svc_promo[i].valid && !free_rec) + free_rec = &svc_promo[i]; + if (svc_promo[i].valid && + (!oldest || svc_promo[i].seq < oldest->seq)) + oldest = &svc_promo[i]; + } + if (!rec) + rec = free_rec ? free_rec : oldest; + if (!rec) { + spin_unlock_irqrestore(&svc_promo_lock, flags); + return; + } + rec->va = va; + rec->pa = pa; + rec->size = size; + rec->phys_attr = phys_attr; + rec->buffer_id = (u16)buffer_id; + rec->seq = ++svc_promo_seq; + rec->valid = true; + spin_unlock_irqrestore(&svc_promo_lock, flags); +} + +static int svc_promo_lookup(u64 va, u64 *addr, int *ap) +{ + unsigned long flags; + u64 best_seq = 0; + int rc = -ENOENT; + int i; + + spin_lock_irqsave(&svc_promo_lock, flags); + for (i = 0; i < SVC_PROMO_MAX; i++) { + const struct svc_promo_rec *rec = &svc_promo[i]; + + /* + * r823: answer ONLY the base page. The promoted (va, size) + * pair is a VA-space booking, not a contiguous mapping: the + * ctx MAIN buffer's span [0x120020000, 0x120153000) contains + * OTHER buffers' VAs (the channel GPFIFO at 0x120090000), and + * the range arithmetic answered such an interior VA with + * MAIN.pa + offset - a zero page (observed r823: the GPFIFO + * VA resolved to 0x11ce70000, the ring-entry validation read + * va=0x0 len=0 and the latch never fired). Only the page at + * the declared VA has a proven claim to the declared PA; the + * pushbuffer's NOP sits exactly there (nvPushHeader writes at + * the segment base). + */ + if (!rec->valid || va < rec->va || + va - rec->va >= SEMA_PAGE_SIZE) + continue; + if (best_seq && rec->seq <= best_seq) + continue; + best_seq = rec->seq; + *addr = rec->pa + (va - rec->va); + /* + * physAttr 0x4 was observed only on vidmem entries (pa in + * the emulated FB aperture, e.g. 0x1f4000000), 0x5/0x6 on + * system memory. Unknown values fall to SYS, whose read + * guard (pa < SEMA_PA_MAX) still rejects nonsense. + */ + *ap = (rec->phys_attr == 0x4) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + rc = 0; + } + spin_unlock_irqrestore(&svc_promo_lock, flags); + if (!rc && svc_promo_hit_log < 64) { + svc_promo_hit_log++; + pr_info("sema-promo: va=0x%llx -> pa=0x%llx ap=%d (driver PROMOTE_CTX statement)\n", + va, *addr, *ap); + } + return rc; +} + +/* + * SESSION 153-CONTINUED-95: per-(engine,bufferId) record of the guest + * driver's GR context-buffer promote statements, for the launch-time + * CTX-BIND receipt. The response-free sema promote table above is keyed by + * VA and answers translations; this one keeps the latest PROMOTE_CTX entry + * for the context buffers the mission asks about - MAIN (bufferId 0) and + * PATCH (bufferId 2) - including the bInitialize flag, so a launch can log + * whether the client's promoted GR context buffer is installed on the + * engine's replay channel (it is not: the channel's context is the + * server/own buffer the ladder built; this record proves which buffer the + * client believes is its context, with pa/va/size/init as sent). + */ +#define SVC_CTXP_MAX 8 +struct svc_ctxp_rec { + u32 engine; + u32 buffer_id; + u64 pa; + u64 va; + u64 size; + u32 phys_attr; + u32 b_init; + unsigned long seq; + bool valid; +}; +static struct svc_ctxp_rec svc_ctxp[SVC_CTXP_MAX]; +static unsigned long svc_ctxp_seq; +static DEFINE_SPINLOCK(svc_ctxp_lock); + +void gsp_sema_service_ctx_promote_note(u32 engine, u32 buffer_id, u64 pa, + u64 va, u64 size, u32 phys_attr, + u32 b_init) +{ + struct svc_ctxp_rec *rec = NULL, *free_rec = NULL, *oldest = NULL; + unsigned long flags; + int i; + + spin_lock_irqsave(&svc_ctxp_lock, flags); + for (i = 0; i < SVC_CTXP_MAX; i++) { + if (svc_ctxp[i].valid && + svc_ctxp[i].engine == engine && + svc_ctxp[i].buffer_id == buffer_id) { + rec = &svc_ctxp[i]; + break; + } + if (!svc_ctxp[i].valid && !free_rec) + free_rec = &svc_ctxp[i]; + if (svc_ctxp[i].valid && + (!oldest || svc_ctxp[i].seq < oldest->seq)) + oldest = &svc_ctxp[i]; + } + if (!rec) + rec = free_rec ? free_rec : oldest; + if (rec) { + rec->engine = engine; + rec->buffer_id = buffer_id; + rec->pa = pa; + rec->va = va; + rec->size = size; + rec->phys_attr = phys_attr; + rec->b_init = b_init; + rec->seq = ++svc_ctxp_seq; + rec->valid = true; + } + spin_unlock_irqrestore(&svc_ctxp_lock, flags); +} + +int gsp_sema_service_ctx_promote_get(u32 engine, u32 buffer_id, u64 *pa, + u64 *va, u64 *size, u32 *phys_attr, + u32 *b_init) +{ + unsigned long flags; + u64 best_seq = 0; + int rc = -ENOENT; + int i; + + spin_lock_irqsave(&svc_ctxp_lock, flags); + for (i = 0; i < SVC_CTXP_MAX; i++) { + const struct svc_ctxp_rec *rec = &svc_ctxp[i]; + + if (!rec->valid || rec->engine != engine || + rec->buffer_id != buffer_id) + continue; + if (best_seq && rec->seq <= best_seq) + continue; + best_seq = rec->seq; + if (pa) + *pa = rec->pa; + if (va) + *va = rec->va; + if (size) + *size = rec->size; + if (phys_attr) + *phys_attr = rec->phys_attr; + if (b_init) + *b_init = rec->b_init; + rc = 0; + } + spin_unlock_irqrestore(&svc_ctxp_lock, flags); + return rc; +} + +/* * VA shadow: page-granular backing for device VAs whose VA->PA translation * cannot be derived from the guest's page tables. * @@ -2316,6 +4870,16 @@ static void svc_lat_store(u64 va, u64 addr, int ap) struct svc_vas_rec { u64 va_page; /* 0 = empty */ u8 *data; + /* + * r635: the page holds bytes a client/host WRITE actually landed + * (an H2D shadow-lane write, an inline write, a bridge write, an + * engine writeback). Synthetic pages minted by the GR fill serve + * zeros and must never be mistaken for the client's own content: + * only real content anchors a span or arms the launch-level anchor + * (without this flag the fill would re-anchor on its own minted + * pages and mint without bound across retry passes). + */ + u8 content; }; static struct svc_vas_rec svc_vas[SVC_VAS_SLOTS]; @@ -2351,6 +4915,14 @@ static bool svc_vas_has(u64 va) return svc_vas_find(va) != NULL; } +/* r635: the page holds REAL written content (not a synthetic zero page). */ +static bool svc_vas_has_content(u64 va) +{ + struct svc_vas_rec *r = svc_vas_find(va); + + return r && r->content; +} + /* * A VA may receive a new shadow record only when it is engine-only device * memory: high in the client VA space, where the guest's own sysmem objects @@ -2358,8 +4930,23 @@ static bool svc_vas_has(u64 va) * 100 TiB). Explicit sources (GRCE inline writes, the bridge's output write) * shadow any VA because their target is device memory by construction. */ +bool gsp_sema_no_shadow; /* r626 A/B: disable the shadow lane */ +module_param_named(sema_no_shadow, gsp_sema_no_shadow, bool, 0644); +MODULE_PARM_DESC(sema_no_shadow, + "Disable the VA shadow lane (copy-dst and fill fallbacks) for an A/B boot; unresolved device VAs hold instead (default 0)"); + static bool svc_vas_can_shadow(u64 va) { + /* + * r626 A/B: with the knob set, every shadow fallback is refused and + * the caller takes its -ENOENT/hold path. The r613c/r616/r623c + * corruption class (nvidia channel structures overwritten with + * PTE-like 0x444../0x4456.. patterns, fatal panic in + * kchannelGetGfid_IMPL via nvGpuOpsStopChannel) follows the + * shadow-lane era; this knob isolates the lane as a variable. + */ + if (gsp_sema_no_shadow) + return false; return va >= SVC_VAS_FLOOR; } @@ -2409,6 +4996,7 @@ static int svc_vas_write(u64 va, const u8 *buf, size_t len) if (!r) return -ENOMEM; memcpy(r->data + (va & (SEMA_PAGE_SIZE - 1)), buf, chunk); + r->content = 1; /* r635: real bytes landed here */ svc.vas_writes += chunk; buf += chunk; va += chunk; @@ -2521,6 +5109,37 @@ static void pin_store_tab(struct svc_pin *tab, u32 slots, u32 probe_max, (*drops)++; } +/* + * SESSION 153-CONTINUED-64d: same store, but reports a probe-window + * exhaustion back to the caller so it can evict instead of losing the pin + * (see svc_ipin_store). Returns true when the slot was NOT stored. + */ +static bool pin_store_tab_nodrop(struct svc_pin *tab, u32 slots, + u32 probe_max, u32 *n, u64 *drops, u64 va, + u64 pa, int ap, u8 win) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = slots - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + u32 probe; + + for (probe = 0; probe < probe_max; probe++) { + struct svc_pin *p = &tab[(h + probe) & mask]; + + if (!p->va_page || p->va_page == page) { + if (!p->va_page) + (*n)++; + p->va_page = page; + p->pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + p->ap = ap; + p->win = win; + return false; + } + } + (*drops)++; + return true; +} + static int svc_pin_lookup(u64 va, u64 *pa, int *ap) { return pin_lookup_tab(svc.pin, SVC_PIN_SLOTS, SVC_PIN_PROBE, va, pa, ap, @@ -2552,6 +5171,36 @@ static void svc_pin_store_win(u64 va, u64 pa, int ap) } /* + * SESSION 154 (M-GL1.2): drop a validated pin whose content no longer + * parses as the page's kind of stream (a stale translation kept answering + * for a page the guest has since reused - r913: va 0x1203c0000 read + * `first8=00000001` through a validated pin while the real push bytes sat + * elsewhere). The next read then consults the window guess, the trusted + * walk and the content sources instead of the dead pin. + */ +static void svc_pin_drop(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = SVC_PIN_SLOTS - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + u32 probe; + + for (probe = 0; probe < SVC_PIN_PROBE; probe++) { + struct svc_pin *p = &svc.pin[(h + probe) & mask]; + + if (!p->va_page) + return; + if (p->va_page == page) + break; + } + if (probe == SVC_PIN_PROBE) + return; + memset(&svc.pin[(h + probe) & mask], 0, sizeof(struct svc_pin)); + if (svc.pin_n) + svc.pin_n--; +} + +/* * Injected (host-tool) translations. Kept separate from the discovered pin * table so a hand-verified VA->PA pair can never be shadowed, evicted or * silently dropped by the self-discovery stream: in the shared table a hot @@ -2561,6 +5210,7 @@ static void svc_pin_store_win(u64 va, u64 pa, int ap) * every push was "implausible"). */ static inline bool svc_pool_sem_va(u64 va); +static bool svc_kick_family(const struct svc_channel *ch); static void svc_pool_pa_forget(u64 va); /* @@ -2589,6 +5239,57 @@ struct svc_ipin_alt { static struct svc_ipin_alt svc_ipin_alt[SVC_IPIN_ALT_SLOTS]; static u64 svc_pushalt_hits; /* push reads served by a displaced pair */ static u32 svc_pushalt_log; +/* r707x: per-client push-read attempt witness (pump read site). */ +static u32 svc_rdw_log; +/* r820: bounded per-session USERD-scan decision witness (svc_chan_userd). + * GR channels only and capped: a hot CE channel re-running this every + * ~16 ms burned the whole budget before the channel under diagnosis ever + * appeared. Each scan reports its decision, a scan that found nothing + * reportable logs a sweep summary. */ +static u32 svc_userdp_log; +/* SESSION 153-CONTINUED-44 (r1026): bounded witness for zero-put USERD + * candidate skips (svc_chan_userd; the screen-pattern false block). */ +static u32 svc_zero_put_skip_log; +/* r829: bounded witness for the zero-entry holds (bad ring translations). */ +static u32 svc_zero_entry_log; +static u8 svc_read_last_src; +/* r715: PA the winning source of the last push read resolved to. */ +static u64 svc_read_last_pa; +/* + * r820: set when the last svc_read_va_push_ex chunk was answered by a + * driver-declared PROMOTE_CTX kernel buffer. The push decode uses it to + * accept the NOP sanity push: nvidia-push's __nvPushTestPushBuffer writes + * NVA16F_DMA_NOP (0x00000000) into a promoted kernel pushbuffer, so the + * whole small push is zeros - unparseable as a method stream, but exactly + * right from a translation the driver itself declared. + */ +static bool svc_read_last_promote; +/* r715: per-client push-decode branch witness (pump read site). */ +static u32 svc_pushdec_log; +/* r707d: bounded push-read failure witness (see svc_read_va_push_ex). */ +static u32 svc_pushfail_log; +/* r1405: bounded push-read ceiling-rejection witness (see svc_read_va_push_ex). */ +static u32 svc_pinrej_log; +/* + * SESSION 132: the util-sweep witnesses are per-CLIENT budgets (reset in + * svc_session_flush), not per-boot: on a multi-client boot the driver-init + * reads of an earlier client exhausted the boot budget and hid the later + * clients' rows (the same fix the r707x/r715 pump witnesses got). + */ +static u32 svc_util_dec_log; +static u32 svc_gr_util_log; +static struct { u64 va; u64 h; } svc_util_dec_keys[32]; +/* + * SESSION 143 (r803): per-channel sweep VISIBILITY. The r802 evidence could + * not say whether the Vulkan client's copy-class channels (0xbeef0101/2, + * eng 11/12) were swept at all: the per-pass log capped at 24 rows and the + * early return (no gpfifo/entries) prints nothing. These one-shot (per + * client, reset in svc_session_flush) keys name the first sweep of EVERY + * channel - begin or skip + reason - so "channel absent from the log" can + * never again be confused with "channel not in the table". + */ +static u32 svc_util_seen_keys[32]; +static u32 svc_util_seen_n; static int svc_ipin_alt_lookup(u64 va, u64 *pa, int *ap) { @@ -2655,8 +5356,32 @@ static void svc_ipin_store(u64 va, u64 pa, int ap) (old_pa & ~(u64)(SEMA_PAGE_SIZE - 1)) != (pa & ~(u64)(SEMA_PAGE_SIZE - 1))) svc_ipin_alt_store(va, old_pa, old_ap); - pin_store_tab(svc.ipin, SVC_IPIN_SLOTS, SVC_IPIN_PROBE, &svc.ipin_n, - &svc.ipin_drop, va, pa, ap, 0); + /* + * SESSION 153-CONTINUED-64d: the table SATURATES under client map + * churn (r1122: `ipins=32768/148777` - the old store DROPPED every + * pin past the 32768 slots, so the demand stream's setup-push pages + * could never land and the held read never flipped). Slots: 1<<15. + * When the probe window is exhausted, evict the home slot's + * occupant instead of dropping: the evicted pair is preserved in + * the alt table (where a reader may still fall back to it) and both + * the scanner's periodic re-assert and the demand resend cadences + * restore any evicted entry that is still needed. A dropped demand + * pin, by contrast, is unrecoverable within the hold window. + */ + if (pin_store_tab_nodrop(svc.ipin, SVC_IPIN_SLOTS, SVC_IPIN_PROBE, + &svc.ipin_n, &svc.ipin_drop, va, pa, ap, 0)) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 mask = SVC_IPIN_SLOTS - 1; + u32 h = (u32)((page >> SEMA_PAGE_SHIFT) * 2654435761u) & mask; + struct svc_pin *p = &svc.ipin[h]; + + if (p->va_page && p->va_page != page) + svc_ipin_alt_store(p->va_page, p->pa_page, p->ap); + p->va_page = page; + p->pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + p->ap = ap; + p->win = 0; + } /* * A fresh injection supersedes every older translation for this * page: drop the verified pool cache entry so a subsequent pool @@ -2786,8 +5511,8 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, return -ENOMEM; end = tr_cursor + SVC_TR_SLICE; - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; + if (end > svc_pa_max()) + end = svc_pa_max(); for (gpa = tr_cursor; gpa < end; gpa += SVC_TR_CHUNK) { u64 chunk = min_t(u64, SVC_TR_CHUNK, end - gpa); @@ -2816,7 +5541,7 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, if (cpu_va < direct_base) continue; pa = cpu_va - direct_base; - if (pa >= SEMA_PA_MAX || (pa & 0xfff)) + if (pa >= svc_pa_max() || (pa & 0xfff)) continue; *pa_out = pa | (gpu_va & (SEMA_PAGE_SIZE - 1)); @@ -2834,7 +5559,7 @@ static int svc_targeted_resolve(struct nv_vgpu_device *vgpu, u64 gpu_va, tr_cursor = end; kfree(buf); - if (tr_cursor >= SEMA_PA_MAX) { + if (tr_cursor >= svc_pa_max()) { pr_info("sema-svc: targeted resolve va=0x%llx: full RAM scan found nothing (scan #%llu)\n", page_va, tr_scans); svc_tr_note_failure(page_va); @@ -2870,30 +5595,106 @@ static bool svc_resolve_in_progress(u64 va) * VA above the window base, including sysmem-backed VAs, so callers that must * not act on a guess (push execution) use this variant. */ +/* + * r835: the VAS handle of the channel currently being serviced (0 = none). + * Channel-scoped VA resolution consults this VAS's page tables FIRST: the + * global pin/lat/map layers are VA-keyed and every client's RM window sits + * at the same VAs (0x1200xxxxx), so a lookup outside the owning client's + * page tables can answer ANOTHER client's page - the X server's ring + * resolved to the GL client's (zero) page and its pushes decoded empty, + * wedging X's readback wait exactly like the RC watchdog's ring starvation. + */ +static u32 svc_cur_vas; + +/* + * r1483: walk the page tables of one specific published VAS handle. The + * render-target seed uses this with the VAS recorded by + * svc_surf_prelude_capture - a VA-keyed pin from another client's VAS must + * never answer for a surface the captured bind does not belong to. + */ +static int svc_vas_lookup_h(struct nv_vgpu_device *vgpu, u32 h_vaspace, + u64 va, u64 *addr, int *ap) +{ + int i; + + if (!h_vaspace) + return -ENOENT; + for (i = 0; i < SVC_MAX_VASPACES; i++) { + u64 pa = 0; + int a = SVC_APERTURE_SYS; + + if (!svc.vas[i].valid || + svc.vas[i].h_vaspace != h_vaspace) + continue; + if (svc_gmmu_walk(vgpu, svc.vas[i].pdb, svc.vas[i].entries, + va & ~(u64)(SEMA_PAGE_SIZE - 1), + svc.vas[i].ap, &pa, &a, false)) + continue; + *addr = pa + (va & (SEMA_PAGE_SIZE - 1)); + *ap = a; + return 0; + } + return -ENOENT; +} + +static int svc_chanvas_lookup(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, + int *ap) +{ + return svc_vas_lookup_h(vgpu, READ_ONCE(svc_cur_vas), va, addr, ap); +} + +/* + * SESSION 153-CONTINUED-67: which sub-source of svc_lookup_trusted answered + * the last call (1=channel VAS 2=pin 3=learned 4=discovered map 5=VAS walk + * 6=discovered walk 7=targeted 8=promote 0=miss). Printed as `sub=` by the + * pushdec witness so a repeated-zero read can be attributed to the exact + * layer (the r1127 investigation could not tell a stale learned entry from + * a stale registered-VAS walk). + */ +static u8 svc_trust_src; + static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, int *ap) { u64 pa = 0; + svc_trust_src = 0; + /* + * r835: the decoded channel's own page tables answer first. When a + * channel is being serviced (svc_cur_vas set) its VAS is the only + * authoritative source for its VAs; the VA-keyed layers below may + * hold another client's pages for the same numeric VA. A failed + * walk falls through to the old chain unchanged. + */ + if (!svc_chanvas_lookup(vgpu, va, addr, ap)) { + svc_trust_src = 1; + return 0; + } if (!svc_pin_lookup_any(va, addr, ap)) { + svc_trust_src = 2; return 0; } - if (!svc_lat_lookup(va, addr, ap)) + if (!svc_lat_lookup(va, addr, ap)) { + svc_trust_src = 3; return 0; + } if (!svc_map_lookup(va, &pa)) { *addr = pa; *ap = SVC_APERTURE_SYS; svc_lat_store(va, pa, SVC_APERTURE_SYS); + svc_trust_src = 4; return 0; } if (!svc_walk_vaspaces(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 5; return 0; } if (!svc_walk_discovered(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 6; return 0; } if (!svc_pd_found) { @@ -2908,6 +5709,7 @@ static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, if (!svc_walk_discovered(vgpu, va & ~0xfffULL, addr, ap)) { *addr += va & 0xfff; svc_lat_store(va, *addr, *ap); + svc_trust_src = 6; return 0; } } @@ -2916,6 +5718,25 @@ static int svc_lookup_trusted(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, *addr = pa; *ap = SVC_APERTURE_SYS; svc_lat_store(va, pa, SVC_APERTURE_SYS); + svc_trust_src = 7; + return 0; + } + /* + * r826: driver-declared PROMOTE_CTX buffers are the LAST resort, not + * the first. The promoted (va, pa) pair names a BOOKING, and even its + * base page can belong to a different buffer than the one mapped at + * that VA (r823: the ctx-MAIN booking's span swallowed the GPFIFO VA; + * r826: the PATCH booking's base VA 0x120001000 hosts nvidia-push's + * separately mapped 4 KB progress semaphore - writing the release to + * the booking's pa left the driver's own polled page at zero and the + * X server failed at its 10 s wait while the release value sat in the + * wrong page). The page-table walks describe the real mappings and + * must answer first; the promote record is only a fallback for VAs + * the walks cannot resolve at all. + */ + if (!svc_promo_lookup(va, addr, ap)) { + svc_lat_store(va, *addr, *ap); + svc_trust_src = 8; return 0; } return -ENOENT; @@ -3209,8 +6030,12 @@ static inline bool svc_pool_pa_identity(u64 va, u64 pa) (va & ~(u64)(SEMA_PAGE_SIZE - 1)); } -static void svc_pool_boot_store(u64 va, u64 pa, int ap); /* m168 */ +static void svc_pool_boot_store(struct nv_vgpu_device *vgpu, u64 va, u64 pa, + int ap); /* m168 */ static bool svc_dst_guard_hold(u64 va); /* m169b */ +static bool svc_dst_guard_should_hold(u64 va); /* r609 */ +static void svc_dst_guard_age(void); /* r609 */ +static void svc_dst_guard_reset(void); /* r609 */ static bool svc_dst_guard_pending(u64 va); /* r231 */ static u32 svc_read_order_log; /* r231 budget */ static u32 svc_atrace_log; /* r234 A-TRACE budget */ @@ -3237,7 +6062,8 @@ static int svc_pool_pa_lookup(u64 va, u64 *pa, int *ap) return -ENOENT; } -static void svc_pool_pa_store(u64 va, u64 pa, int ap) +static void svc_pool_pa_store(struct nv_vgpu_device *vgpu, u64 va, u64 pa, + int ap) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u32 i; @@ -3246,7 +6072,7 @@ static void svc_pool_pa_store(u64 va, u64 pa, int ap) if (svc_pool_pa_identity(page, pa)) return; /* m168: every proven resolution also feeds the boot-scoped cache. */ - svc_pool_boot_store(va, pa, ap); + svc_pool_boot_store(vgpu, va, pa, ap); for (i = 0; i < svc_pool_pa_n; i++) { if (svc_pool_pa[i].va_page == page) { svc_pool_pa[i].pa_page = @@ -3295,6 +6121,26 @@ struct svc_pool_boot_pa { u64 va_page; u64 pa_page; int ap; + /* + * r539: secondary candidate. The pool VA can be answered by two + * read-back-proven sources that disagree (the injected pin - the + * guest kernel's own tracker->page->rm_mem chain - and the host + * GMMU walk of the client's page tables). Either can be the page + * the client actually polls: the r536 fix preferred the walk and + * the r538 init-stall boot showed the pin and the walk flapping + * every resolve while a client sat in uvm channel_pool_add -> + * uvm_push_end_and_wait. A release written only to the losing page + * strands the tracker. Keep BOTH pages and fan every pool release + * out to the secondary as well: the word is a forward-only + * monotonic counter, so a duplicate write is absorbed. The + * secondary is dropped when it stops being readable and ages out + * after SVC_POOL_DUAL_TTL_MS of not being re-proven by any source + * (a page the guest has since freed must not be written). + */ + u64 pa2_page; + int ap2; + unsigned long seen; /* last proof time of the primary */ + unsigned long seen2; /* last proof time of the secondary */ }; static struct svc_pool_boot_pa svc_pool_boot[SVC_POOL_PA_MAX]; @@ -3302,6 +6148,12 @@ static u32 svc_pool_boot_n; static u32 svc_pool_boot_next; static u32 svc_pool_boot_conflicts; static u32 svc_pool_boot_hits; +/* r539: fresh-secondary window; the flapping sources (the watcher's ~2 s + * pool re-inject, the per-resolve walk) re-prove both candidates well inside + * this, while a page the guest has since freed and reused stops being + * re-proven and ages out before any write can land on it (the r372 + * stale-page corruption rule). */ +#define SVC_POOL_DUAL_TTL_MS 10000u static int svc_pool_boot_lookup(u64 va, u64 *pa, int *ap) { @@ -3322,7 +6174,55 @@ static int svc_pool_boot_lookup(u64 va, u64 *pa, int *ap) return -ENOENT; } -static void svc_pool_boot_store(u64 va, u64 pa, int ap) +/* + * r1640: when a pool-page promotion replaces the primary candidate, a + * release already written to the old page is invisible on the new one. + * UVM's tracker polls whichever page its own mapping names, so backfill + * the new page from the old one at promotion time. Pool semaphore words + * are forward-only monotonic counters, so a per-word max-merge is safe. + * Reads both 4 KiB pages first; never writes on a read failure, and + * never merges a page into itself. + */ +static void svc_pool_page_merge(struct nv_vgpu_device *vgpu, u64 va, + u64 old_pa, int old_ap, u64 new_pa, int new_ap) +{ + u32 *old_buf; + u32 *new_buf; + u32 words = 0; + u32 i; + + if (!vgpu) + return; + if ((old_pa & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (new_pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + return; + old_buf = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + new_buf = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + if (!old_buf || !new_buf) + goto out; + if (svc_read_ap(vgpu, old_pa, svc_ap_of(old_ap), old_buf, + SEMA_PAGE_SIZE)) + goto out; + if (svc_read_ap(vgpu, new_pa, svc_ap_of(new_ap), new_buf, + SEMA_PAGE_SIZE)) + goto out; + for (i = 0; i < SEMA_PAGE_SIZE / sizeof(u32); i++) { + if (old_buf[i] <= new_buf[i]) + continue; + if (!svc_write_ap(vgpu, new_pa + i * sizeof(u32), + svc_ap_of(new_ap), &old_buf[i], + sizeof(old_buf[i]))) + words++; + } + pr_info_ratelimited("vgsp-sema: pool-merge va=0x%llx old_pa=0x%llx new_pa=0x%llx words=%u\n", + va, old_pa, new_pa, words); +out: + kfree(old_buf); + kfree(new_buf); +} + +static void svc_pool_boot_store(struct nv_vgpu_device *vgpu, u64 va, u64 pa, + int ap) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); @@ -3332,17 +6232,55 @@ static void svc_pool_boot_store(u64 va, u64 pa, int ap) return; for (i = 0; i < svc_pool_boot_n; i++) { if (svc_pool_boot[i].va_page == page) { - if (svc_pool_boot[i].pa_page != pa_page) { - /* The page moved (or a source lied): a - * fresh proof wins, but the conflict is on - * record. */ - svc_pool_boot_conflicts++; - pr_info("sema-svc: pool-boot conflict va=0x%llx cached=0x%llx new=0x%llx (replacing)\n", - page, svc_pool_boot[i].pa_page, - pa_page); - } - svc_pool_boot[i].pa_page = pa_page; - svc_pool_boot[i].ap = ap; + if (svc_pool_boot[i].pa_page == pa_page) { + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + return; + } + if (svc_pool_boot[i].pa2_page == pa_page) { + /* The previously-secondary page is the + * freshest proof: promote it, demote the + * old primary. No conflict line: both + * candidates stay live and both receive + * the fan-out. */ + u64 op = svc_pool_boot[i].pa_page; + int oap = svc_pool_boot[i].ap; + unsigned long oseen = svc_pool_boot[i].seen; + + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + svc_pool_boot[i].pa2_page = op; + svc_pool_boot[i].ap2 = oap; + svc_pool_boot[i].seen2 = oseen; + svc_pool_page_merge(vgpu, page, op, oap, + pa_page, ap); + return; + } + /* + * r539: a second, DIFFERENT page. Keep the old + * primary as the secondary candidate instead of + * discarding it - the r538 init-stall boot had the + * pin and the walk flapping forever, and the client + * polls whichever page its own mapping names. + */ + svc_pool_boot_conflicts++; + pr_info_ratelimited("sema-svc: pool-boot second candidate va=0x%llx primary=0x%llx -> 0x%llx (old primary kept as secondary)\n", + page, svc_pool_boot[i].pa_page, pa_page); + { + u64 op = svc_pool_boot[i].pa_page; + int oap = svc_pool_boot[i].ap; + + svc_pool_boot[i].pa2_page = + svc_pool_boot[i].pa_page; + svc_pool_boot[i].ap2 = svc_pool_boot[i].ap; + svc_pool_boot[i].seen2 = svc_pool_boot[i].seen; + svc_pool_boot[i].pa_page = pa_page; + svc_pool_boot[i].ap = ap; + svc_pool_boot[i].seen = jiffies; + svc_pool_page_merge(vgpu, page, op, oap, + pa_page, ap); + } return; } } @@ -3354,6 +6292,304 @@ static void svc_pool_boot_store(u64 va, u64 pa, int ap) svc_pool_boot[i].va_page = page; svc_pool_boot[i].pa_page = pa_page; svc_pool_boot[i].ap = ap; + svc_pool_boot[i].pa2_page = 0; + svc_pool_boot[i].ap2 = 0; + svc_pool_boot[i].seen = jiffies; + svc_pool_boot[i].seen2 = 0; +} + +/* + * r539: fan one applied pool release out to the secondary candidate page + * (see struct svc_pool_boot_pa). Forward-only and read-back proven, exactly + * like the primary write; a failure is counted, never propagated - the + * primary write already landed. A secondary that stopped being readable is + * dropped; one that was not re-proven by any source for + * SVC_POOL_DUAL_TTL_MS is left alone (the guest may have reused the page). + */ +static void svc_pool_boot_fanout(struct nv_vgpu_device *vgpu, u32 h, u64 va, + u32 val) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + u32 cur = 0; + u32 back = 0; + u32 i; + + for (i = 0; i < svc_pool_boot_n; i++) + if (svc_pool_boot[i].va_page == page) + break; + if (i == svc_pool_boot_n || !svc_pool_boot[i].pa2_page) + return; + if (!svc_pool_boot[i].seen2 || + time_after(jiffies, svc_pool_boot[i].seen2 + + msecs_to_jiffies(SVC_POOL_DUAL_TTL_MS))) + return; + pa = svc_pool_boot[i].pa2_page | (va & (SEMA_PAGE_SIZE - 1)); + ap = svc_pool_boot[i].ap2; + if (svc_pool_pa_identity(va, pa)) + return; + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &cur, sizeof(cur))) { + /* The page is gone: drop the stale secondary. */ + svc_pool_boot[i].pa2_page = 0; + return; + } + if (cur >= val) + return; /* forward-only: nothing to add */ + if (svc_write_ap(vgpu, pa, svc_ap_of(ap), &val, sizeof(val))) + return; + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &back, sizeof(back))) + return; + if (back != val) + return; + svc.pool_dual_applied++; + if (svc.pool_dual_log < 128) { + svc.pool_dual_log++; + pr_info("vgsp-sema: pool-fanout ch=0x%x va=0x%llx pa2=0x%llx cur=%u -> val=%u (secondary candidate received the release)\n", + h, va, pa, cur, val); + } +} + +/* + * r1648: write a pool release through EVERY published VAS's translation of + * the semaphore VA. The pool page moves between the RM memdesc, the + * self-discovered directory and the per-client VAS directories; the + * first-success walk can land on a stale page while the guest polls the + * live one (r1647: host wrote 0x2600000, the guest read 0x126502000). + * The counter is forward-only, so a duplicate on a stale (or reused) + * semaphore page is harmless; only write where the value is lower and the + * read-back proves the write. Diagnostics are bounded. + */ +static void svc_pool_write_all_vas(struct nv_vgpu_device *vgpu, u32 h, u64 va, + u32 val) +{ + u64 pas[SVC_MAX_VASPACES]; + int aps[SVC_MAX_VASPACES]; + u64 off = va & (u64)(SEMA_PAGE_SIZE - 1); + int n, i; + + n = svc_walk_vaspaces_collect(vgpu, va & ~(u64)(SEMA_PAGE_SIZE - 1), + pas, aps, SVC_MAX_VASPACES); + for (i = 0; i < n; i++) { + u64 a = (pas[i] & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + u32 cur = 0, back = 0; + + if (svc_pool_pa_identity(va, a)) + continue; + if (svc_read_ap(vgpu, a, svc_ap_of(aps[i]), &cur, sizeof(cur))) + continue; + if (cur >= val) + continue; + if (svc_write_ap(vgpu, a, svc_ap_of(aps[i]), &val, + sizeof(val))) + continue; + if (svc_read_ap(vgpu, a, svc_ap_of(aps[i]), &back, + sizeof(back)) || back != val) + continue; + if (svc.pool_vas_log < 64) { + svc.pool_vas_log++; + pr_info("vgsp-sema: pool-vas-wr ch=0x%x va=0x%llx pa=0x%llx val=%u (all-VAS fan-out)\n", + h, va, a, val); + } + } +} + +/* + * V14 (CONT.430): GLX shared-surface refCount sync. + * + * The GLX client's kickoff-suffix RELEASE_SEMAPHORE advances + * sharedSurface->gpuRefCount[sd].val in the shared surface; the X DDX's + * idle-channel poll checks the CPU mapping of the same page and waits for + * refCount == lastKickoffRefCount, warning after + * 5 s. The single-success VAS walk can name a stale incarnation of the + * client VA and write a page no process maps: r1780 measured the release + * target va=0x420a00800 resolving to pa=0x2493d800 (unmapped) while the + * DDX's page held refCount=8 / lkr=16. Fan the forward-only value out + * through EVERY published VAS translation (the r1648 pool pattern) and, + * when a candidate page carries a lastKickoffRefCount at least the + * decoded payload, author that lkr instead: the DDX stops waiting on + * refCount == lkr and lkr is the client's own last kickoff value (the + * client writes it in CPU memory before the kick; the GPU-side release + * we are emulating would have produced exactly it). + * + * Layout: GLSsharedSurface in the 570.144 DDX has gpuRefCount[] at +0x800 + * of the shared mapping and lastKickoffRefCount at +0x880 (NV_MAX_SUBDEVICES + * = 8, so 8 * 16 B = 0x80; DDX +0x10d237 polls gpuRefCount[j] and + * +0x10d23c reads 0x880(%rcx)). The release VA we decode is + * gpuMapping + 0x800 + 16*sd; lastKickoffRefCount is therefore at + * page_base + 0x880, not at release_va + 0x40. + * + * Forward-only + read-back proven; a stale page absorbing a duplicate is + * harmless. Gated by sema_gl_shsurf_allvas for A/B. + */ +static bool gsp_sema_gl_shsurf_allvas = true; +module_param_named(sema_gl_shsurf_allvas, gsp_sema_gl_shsurf_allvas, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shsurf_allvas, + "Sync the GLX shared-surface gpuRefCount through every published VAS translation (and to the client's lastKickoffRefCount when present); default 1"); + +static bool gsp_sema_gl_shsurf_dbg; +module_param_named(sema_gl_shsurf_dbg, gsp_sema_gl_shsurf_dbg, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shsurf_dbg, + "Log every candidate VAS translation for the GLX shared-surface refCount sync (bounded)"); + +/* + * r1780 (CONT.430g): the shared-surface deadlock. The GLX client writes + * lastKickoffRefCount during its GL init (before its first kick), while the + * X DDX is already polling gpuRefCount against that same value. If a previous + * client left gpuRefCount lower, the DDX waits for a release that the client + * will only emit after the very GLX request the DDX is blocking. Sync the + * known shared-surface VA to the client's lkr once per second (and on a + * manual kick) so the wait clears and the client can proceed to kick. + * The VA is the release target observed across boots; the strict lkr gate + * (high dword zero, small, cur < lkr) keeps it harmless if the VA maps + * something else or the client is not at the surface. + */ +static bool gsp_sema_gl_shsurf_watch = true; +module_param_named(sema_gl_shsurf_watch, gsp_sema_gl_shsurf_watch, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shsurf_watch, + "Periodically sync the known GLX shared-surface VA to the client's lastKickoffRefCount; default 1"); + +static bool gsp_sema_gl_shsurf_kick; +module_param_named(sema_gl_shsurf_kick, gsp_sema_gl_shsurf_kick, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shsurf_kick, + "Write 1 to run the shared-surface sync immediately on the service thread"); + +/* + * DANGER (r1780 c16): the strict signature scan can match a guest page-table + * page (the 1.4M "refCount/lkr" values reappear at the same offsets across + * VMs, and writing the lkr into it coincides with guest bad-pmd/soft-lockup + * storms in unrelated processes). The injected-pin author is therefore + * DISABLED by default; only enable it with a page proven to be the client's + * shared surface (e.g. the release target VA resolved through a verified + * VAS). The VAS-walk author remains on. + */ +static bool gsp_sema_gl_shsurf_pin; +module_param_named(sema_gl_shsurf_pin, gsp_sema_gl_shsurf_pin, + bool, 0644); +MODULE_PARM_DESC(sema_gl_shsurf_pin, + "DANGER: author the injected shared-surface pin page (signature scans can hit page tables); default 0"); + +#define SVC_GLSHSURF_VA 0x420a00800ULL + +/* + * r1780 (CONT.431b): direct sync of an operator-injected shared-surface page + * (sema_map_add). The VAS walk cannot always name the client incarnation, but + * a verified {known shared-surface VA -> page} pin is authoritative: read the + * page's lastKickoffRefCount and author it into gpuRefCount[0] forward-only. + * This is the live unblock for the DDX idle wait when the page has already + * been identified by the strict signature scan. + */ +static void svc_gl_shsurf_pin_sync(struct nv_vgpu_device *vgpu) +{ + u64 lkr = 0, pa = 0; + u32 cur = 0, want; + int ap = SVC_APERTURE_SYS; + static u32 log; + + /* + * svc_pin_lookup_any returns 0 on a hit (Linux convention); try the + * exact VA first, then the page-aligned form the commands use. + */ + if (!svc_pin_lookup_any(SVC_GLSHSURF_VA, &pa, &ap)) { + /* exact VA hit: use its translation */ + } else if (!svc_pin_lookup_any(SVC_GLSHSURF_VA & ~0xfffULL, + &pa, &ap)) { + /* page-aligned hit: use its translation */ + } else { + return; + } + if (svc_read_ap(vgpu, (pa & ~0xfffULL) | 0x880ULL, ap, + &lkr, sizeof(lkr))) + return; + if ((lkr >> 32) || !(lkr & 0xffffffffULL) || + (lkr & 0xffffffffULL) > 0x1000000ULL) + return; + want = (u32)lkr; + if (svc_read_ap(vgpu, (pa & ~0xfffULL) | 0x800ULL, ap, &cur, + sizeof(cur))) + return; + if (cur >= want) + return; + if (svc_write_ap(vgpu, (pa & ~0xfffULL) | 0x800ULL, ap, &want, + sizeof(want))) + return; + if (log < 64) { + log++; + pr_info("sema-gl: shsurf-pin-sync pa=0x%llx cur=%u lkr=0x%llx\n", + pa, cur, lkr); + } +} + +static bool svc_gl_shsurf_sync(struct nv_vgpu_device *vgpu, u32 h, u64 va, + u32 val) +{ + u64 pas[SVC_MAX_VASPACES]; + int aps[SVC_MAX_VASPACES]; + u64 off = va & (u64)(SEMA_PAGE_SIZE - 1); + static u32 log, dbg_log; + int n, i, wrote = 0; + + n = svc_walk_vaspaces_collect(vgpu, va & ~(u64)(SEMA_PAGE_SIZE - 1), + pas, aps, SVC_MAX_VASPACES); + for (i = 0; i < n; i++) { + u64 pbase = pas[i] & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 a = pbase | off; + u32 cur = 0, back = 0, want = val; + u64 lkr = 0; + bool have_lkr; + + if (svc_read_ap(vgpu, a, svc_ap_of(aps[i]), &cur, + sizeof(cur))) + continue; + /* + * GLSsharedSurface: gpuRefCount[8] at +0x800, then + * lastKickoffRefCount at +0x880 (64 B aligned). Only trust a + * plausible value: a small non-zero count at least the + * decoded payload and not more than the client could have + * kicked so far. A dense data page is unlikely to satisfy + * all three at once. + */ + have_lkr = !svc_read_ap(vgpu, pbase | 0x880ULL, + svc_ap_of(aps[i]), &lkr, + sizeof(lkr)) && + (lkr >> 32) == 0 && + (lkr & 0xffffffffULL) >= (u64)want && + (lkr & 0xffffffffULL) <= 0x1000000ULL; + /* + * r1780 (CONT.430e): strict author. Only a page that + * carries the client's own lastKickoffRefCount is the + * shared surface; never fall back to writing the raw + * payload into an unvalidated translation (that branch + * was the one that could hit a data page). The DDX + * asserts refCount <= lkr, so write exactly lkr and only + * forward. + */ + if (!have_lkr || cur >= (u32)lkr) + continue; + want = (u32)lkr; + if (gsp_sema_gl_shsurf_dbg && dbg_log < 32) { + dbg_log++; + pr_info("sema-gl: shsurf-cand h=0x%x va=0x%llx pa=0x%llx cur=%u lkr=0x%llx ok=%d want=%u\n", + h, va, a, cur, lkr, have_lkr ? 1 : 0, want); + } + if (svc_write_ap(vgpu, a, svc_ap_of(aps[i]), &want, + sizeof(want))) + continue; + if (svc_read_ap(vgpu, a, svc_ap_of(aps[i]), &back, + sizeof(back)) || back != want) + continue; + wrote++; + if (log < 64) { + log++; + pr_info("sema-gl: shsurf-sync h=0x%x va=0x%llx pa=0x%llx pay=%u lkr=0x%llx (all-VAS)\n", + h, va, a, want, lkr); + } + } + return wrote > 0; } static void svc_pool_boot_forget(u64 va) @@ -3399,16 +6635,67 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, * the injected page is readable before trusting it; a failed * proof falls through to the read-back-validated cache, the * structural walk and the targeted rm_mem record. + * + * r536: the read-back proof does NOT catch a moved pool whose + * OLD page is still readable - the write then lands on the old + * page while the client polls the new one and the tracker + * starves (observed: "pending release starved va=0x12100000x + * tries=64" for ~45 s after the pool moved, with the stale pin + * pa=0x120b30000 still readable; the fresh pin pa=0x130e9d000 + * arrived 45 s later). The client's OWN page tables are the + * GPU/UVM ground truth for where the tracker reads, so when a + * structural walk resolves a DIFFERENT page than the pin, the + * walk wins (both are read-back proven at the exact offset). */ u32 probe = 0; if (!svc_read_ap(vgpu, pa, svc_ap_of(a), &probe, sizeof(probe))) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + bool walk_hit = false; + + if (!svc_walk_discovered(vgpu, page, &wpa, &wap) || + !svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + walk_hit = true; + if (walk_hit && + (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + !svc_pool_pa_identity(va, wpa)) { + u32 wprobe = 0; + u64 exact = (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) | + (va & (SEMA_PAGE_SIZE - 1)); + + if (!svc_read_ap(vgpu, exact, svc_ap_of(wap), + &wprobe, sizeof(wprobe))) { + /* + * r539: record the PIN as the + * secondary candidate before the walk + * page becomes the primary. The walk + * is not infallible (the r538 boot: + * pin and walk flapped on every + * resolve and the client polled the + * pin's page); keeping both means the + * release fan-out reaches whichever + * page the client actually polls. + */ + svc_pool_boot_store(vgpu, va, pa, a); + svc_pool_pa_store(vgpu, va, exact, wap); + svc_tr_clear_failure(page); + if (src) + *src = "walk-over-pin"; + svc_pool_pa_note(h, va, exact, + "walk-over-pin"); + *addr = exact; + *ap = wap; + return 0; + } + } svc_tr_clear_failure(page); if (src) *src = "pin"; svc_pool_pa_note(h, va, pa, "pin"); - svc_pool_boot_store(va, pa, a); + svc_pool_boot_store(vgpu, va, pa, a); *addr = pa; *ap = a; return 0; @@ -3423,6 +6710,51 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, if (!svc_pool_boot_lookup(va, &pa, &a)) { u32 bprobe = 0; + /* + * r580: the boot cache survives session flushes, but a pool + * page MOVES between sessions (the client's new allocation) + * while the old page often stays readable - the read-back + * proof cannot tell. The client's own page tables are the + * ground truth for where ITS tracker reads, so try the + * structural walk BEFORE trusting the cache: a differing, + * read-proven walk page wins and the cached page stays as the + * fan-out's secondary candidate. Live evidence (r579 boot): + * releases were written to the stale cache pa=0x11e6d9000 + * while the current page was 0x134730000 (the pin) and the + * client's tracker starved; the pool-wr line said + * src=bootcache. + */ + { + u64 wpa2 = 0; + int wap2 = SVC_APERTURE_SYS; + bool walk_hit = false; + + if (!svc_walk_discovered(vgpu, page, &wpa2, &wap2) || + !svc_walk_vaspaces(vgpu, page, &wpa2, &wap2)) + walk_hit = true; + if (walk_hit && + (wpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)) && + !svc_pool_pa_identity(va, wpa2)) { + u64 exact = (wpa2 & ~(u64)(SEMA_PAGE_SIZE - 1)) | + (va & (SEMA_PAGE_SIZE - 1)); + + if (!svc_read_ap(vgpu, exact, svc_ap_of(wap2), + &bprobe, sizeof(bprobe))) { + svc_pool_boot_store(vgpu, va, pa, a); + svc_pool_pa_store(vgpu, va, exact, wap2); + svc_tr_clear_failure(page); + if (src) + *src = "walk-over-boot"; + svc_pool_pa_note(h, va, exact, + "walk-over-boot"); + *addr = exact; + *ap = wap2; + return 0; + } + } + } + if (!svc_read_ap(vgpu, pa, svc_ap_of(a), &bprobe, sizeof(bprobe))) { svc_tr_clear_failure(page); @@ -3468,7 +6800,7 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, if (!svc_pool_pa_identity(va, exact) && !svc_read_ap(vgpu, exact, SVC_APERTURE_SYS, &probe, sizeof(probe))) { - svc_pool_pa_store(va, exact, SVC_APERTURE_SYS); + svc_pool_pa_store(vgpu, va, exact, SVC_APERTURE_SYS); svc_tr_clear_failure(page); if (src) *src = "walk"; @@ -3509,7 +6841,7 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, if (!svc_pool_pa_identity(va, exact) && !svc_read_ap(vgpu, exact, svc_ap_of(wap), &probe, sizeof(probe))) { - svc_pool_pa_store(va, exact, wap); + svc_pool_pa_store(vgpu, va, exact, wap); svc_tr_clear_failure(page); if (src) *src = "vaspaces"; @@ -3524,7 +6856,7 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, if (!svc_pool_pa_identity(va, pa) && !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, &probe, sizeof(probe))) { - svc_pool_pa_store(va, pa, SVC_APERTURE_SYS); + svc_pool_pa_store(vgpu, va, pa, SVC_APERTURE_SYS); svc_tr_clear_failure(page); if (src) *src = "targeted"; @@ -3565,7 +6897,7 @@ static int svc_pool_pa_resolve(struct nv_vgpu_device *vgpu, u32 h, u64 va, } if (!svc_read_ap(vgpu, pa, svc_ap_of(*ap), &probe, sizeof(probe))) { - svc_pool_pa_store(va, pa, *ap); + svc_pool_pa_store(vgpu, va, pa, *ap); svc_tr_clear_failure(page); if (src) *src = "trusted"; @@ -3596,7 +6928,7 @@ static int svc_sem_read(struct nv_vgpu_device *vgpu, u64 va, u32 *val) * write depends on (this was the v17 bypass). */ if (!svc_targeted_resolve(vgpu, va, &addr)) { - svc_pool_pa_store(va, addr, SVC_APERTURE_SYS); + svc_pool_pa_store(vgpu, va, addr, SVC_APERTURE_SYS); if (!svc_read_ap(vgpu, addr, SVC_APERTURE_SYS, val, sizeof(*val))) @@ -3623,6 +6955,66 @@ static bool svc_wait_pend_release(u64 va); static bool svc_gr_rel_match(u64 va, u32 *payload); static bool svc_gr_rel_inflight(u64 va, u32 *payload); +/* + * r1645: UVM allocates one tracking semaphore per channel from the GPU + * semaphore pool, in channel-creation order: channel rank N owns pool slot + * N (slot 0 is rank 0's semaphore, not a reserved pool slot). The r1642 + * gdbstub observation ("push released map_base+0 while the tracker polled + * map_base+4") was rank 0 already satisfied and rank 1 waiting: writing + * rank+1 pre-satisfied the NEXT rank before its uvm_gpu_semaphore_alloc() + * zeroed the slot, so the value was lost and rank 1 stalled. r1645 live + * proof: the host's rank+1 write to slot 1 raced rank 1's allocation, the + * guest read 0, and a manual write of 1 to slot 1 (rank 1's own slot) + * cleared uvm_push_end_and_wait. A CE channel's release therefore belongs + * at its own rank slot; apply the same forward-only value there as a + * fallback when the decoded release address was masked to the page base. + */ +#define SVC_UVM_POOL_MAP_BASE 0x121000000ULL +static void svc_pool_tracker_apply(struct nv_vgpu_device *vgpu, u32 h, + u64 va, u32 val) +{ + u64 tva, addr = 0; + const char *src = NULL; + int ap = 0; + u32 cur = 0, back = 0; + int i; + + if (va < SVC_UVM_POOL_MAP_BASE || + va >= SVC_UVM_POOL_MAP_BASE + SEMA_PAGE_SIZE) + return; + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid && svc.chan[i].h_object == h && + svc.chan[i].engine_type) + break; + if (i == SVC_MAX_CHANNELS) + return; + tva = SVC_UVM_POOL_MAP_BASE + 4ULL * (u64)svc.chan[i].ce_index; + if (tva == va) + return; + if (svc_pool_pa_resolve(vgpu, h, tva, &addr, &ap, &src)) + return; + if (svc_read_ap(vgpu, addr, svc_ap_of(ap), &cur, sizeof(cur))) + return; + if (cur >= val) + return; + if (svc_write_ap(vgpu, addr, svc_ap_of(ap), &val, sizeof(val))) + return; + if (svc_read_ap(vgpu, addr, svc_ap_of(ap), &back, sizeof(back)) || + back != val) + return; + /* The pool page can flap between the RM memdesc and the VAS walk + * (r1645: the guest polled the walk page while the boot cache named + * a stale, reused page). Fan the tracker value out to the secondary + * candidate as well, exactly like the decoded release path. */ + svc_pool_boot_fanout(vgpu, h, tva, val); + svc_pool_write_all_vas(vgpu, h, tva, val); + if (svc.pool_tracker_log < 64) { + svc.pool_tracker_log++; + pr_info("sema-svc: pool-tracker-wr ch=0x%x rank=%u va=0x%llx pa=0x%llx val=%u\n", + h, svc.chan[i].ce_index, tva, addr, val); + } +} + static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) { u64 addr = 0, old_addr; @@ -3634,6 +7026,20 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) int wap, rc; bool verified = false; bool rb_ok = false; + static u32 sem_sw_dev_log; /* r619: non-pool device-VA write budget */ + static u32 sem_sw_low_log; /* r1656: low GR release-VA write budget */ + + /* + * V14 (CONT.430): the GLX shared-surface refCount word. The decoded + * release target is <shared gpu mapping>+0x800 in the observed + * boots; the first-success walk may name a stale incarnation, so + * author the value through every published translation (and to the + * client's own lastKickoffRefCount when the page carries one). + * Forward-only, read-back proven, bounded logging. + */ + if (gsp_sema_gl_shsurf_allvas && !is_pool && + va >= 0x200000000ULL && (va & 0xfffULL) == 0x800ULL) + (void)svc_gl_shsurf_sync(vgpu, h, va, val); /* * Pool semaphores resolve pin -> verified cache -> targeted rm_mem @@ -3693,10 +7099,49 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) * sysmem semaphore range (0x792..), with the resolver provenance and * the read-back, so the second session's translations can be * compared with the first session's for the same VA. Bounded. + * + * r619 diag: the same proof for NON-pool device-VA writes (the VAS + * window >= 0x200000000; e.g. 0x20480fef0 push releases, 0x204827f7c + * QMD releases). The r618 hung math client read 0 at 0x20480fef0 + * while the host's deferred releases logged `cur` 97..105 on its own + * target - a write-target divergence that was invisible because this + * range never printed `pa`. r620: the first cut used the 1 TiB + * SVC_VAS_FLOOR and excluded this very window (no lines ever); + * bounded per boot (512). */ - if (va >= 0x79240000000ULL && va < 0x79260000000ULL && - svc.sem_sw_log < 8192) { - svc.sem_sw_log++; + if (((va >= 0x79240000000ULL && va < 0x79260000000ULL && + svc.sem_sw_log < 8192) || + (va >= 0x200000000ULL && !is_pool && sem_sw_dev_log < 4096) || + /* + * SESSION 153-CONT-7 (GL client attribution): the X/GL lanes' + * semaphore window. The X server's progress-tracker release + * (va=0x120001000, kernel progress semaphore) and the GL + * channels' low-aperture objects sit below 0x200000000, so + * their write attempts were invisible (no sema-sw line) while + * the client's polled word stayed zero - the r939 boot left + * no trace of WHERE such a write landed. Bounded like the + * other windows so a hot loop cannot flood dmesg. + */ + (va >= 0x120000000ULL && va < 0x120600000ULL && + svc.sem_sw_xg_log < 2048) || + /* + * r1656: the direct-GL GR/GRCE release arena (0x04000000.. + * 0x0fffffff). The GR init push's SET_REPORT_SEMAPHORE + * release (e.g. va=0x4238000 pay=1) lands here, outside every + * window above, so a successful (or failed) write was + * invisible - the "decoded but never applied" class. Bounded + * so a hot release loop cannot flood dmesg. + */ + (va >= SVC_LOW_REL_VA_LO && va < SEMA_VA_LO && + sem_sw_low_log < 512))) { + if (va >= 0x79240000000ULL && va < 0x79260000000ULL) + svc.sem_sw_log++; + else if (va >= 0x120000000ULL && va < 0x120600000ULL) + svc.sem_sw_xg_log++; + else if (va >= SVC_LOW_REL_VA_LO && va < SEMA_VA_LO) + sem_sw_low_log++; + else + sem_sw_dev_log++; pr_info("sema-sw: h=0x%x va=0x%llx pa=0x%llx ap=%d src=%s val=0x%x rc=%d rb_ok=%d back=0x%x pool=%d\n", h, va, addr, wap, src, val, rc, rb_ok, back, is_pool); } @@ -3704,6 +7149,19 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) if (is_pool) { svc_pool_wr_note(h, va, addr, src, cur, val, "ok"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); + svc_pool_write_all_vas(vgpu, h, va, val); + svc_pool_tracker_apply(vgpu, h, va, val); + } else if (va >= 0x4000000ULL && va < 0x4300000ULL) { + /* + * r1657: direct-GL kernel-VAS semaphores. The + * general resolver can name a stale page for these + * (r1657 beef0100 rel 0x4238000 landed at the + * fallback PA while the client polls the beef0300 + * walk page). Fan the forward-only completion out + * through every published VAS translation too. + */ + svc_pool_write_all_vas(vgpu, h, va, val); } return 0; } @@ -3726,10 +7184,12 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) !svc_read_ap(vgpu, addr, SVC_APERTURE_SYS, &back, sizeof(back)) && back == val) { if (is_pool) { - svc_pool_pa_store(va, addr, SVC_APERTURE_SYS); + svc_pool_pa_store(vgpu, va, addr, SVC_APERTURE_SYS); svc_pool_wr_note(h, va, addr, "targeted", cur, val, "reresolve"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); + svc_pool_write_all_vas(vgpu, h, va, val); } svc.sem_write_reresolve++; pr_info_ratelimited("vgsp-sema: sem-write re-resolve va=0x%llx pa=0x%llx old=0x%llx new=0x%x\n", @@ -3742,6 +7202,9 @@ static int svc_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, u32 val) svc_pool_wr_note(h, va, old_addr, src, cur, val, "advisory"); svc_pool_alt_apply(vgpu, h, va, val); + svc_pool_boot_fanout(vgpu, h, va, val); + svc_pool_write_all_vas(vgpu, h, va, val); + svc_pool_tracker_apply(vgpu, h, va, val); } return 0; /* the original write landed; read-back is advisory */ } @@ -3778,76 +7241,92 @@ static void svc_scan_pass(struct nv_vgpu_device *vgpu, u8 *buf, u64 gpa; u32 nonz = 0; - for (gpa = 0; gpa < SEMA_PA_MAX; gpa += SEMA_PAGE_SIZE) { - u32 off; + /* + * r534: read guest RAM in SVC_SCAN_CHUNK (64 KB) blocks instead of + * one 4 KB page per vfio_dma_rw call. The full 6 GB scan is the + * rebuild's whole cost (1.5 M guest reads ~= 10-11 s per rebuild, + * once per 10 s: the pump got one pass per rebuild and the second + * client's resolutions could wait minutes). A 64 KB read amortizes + * the iommu walk + pin over 16 pages, cutting the scan to well under + * a second with identical pair/base detection (the inner page loop + * is byte-for-byte the old per-page body). + */ + for (gpa = 0; gpa < svc_pa_max(); gpa += SVC_SCAN_CHUNK) { + u32 po; if (READ_ONCE(svc_sess_flush)) break; /* ISSUE-0002: a flush is waiting on this scan */ - if (svc_read(vgpu, gpa, buf, SEMA_PAGE_SIZE)) + if (svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) continue; - for (off = 0; off + 16 <= SEMA_PAGE_SIZE; off += 8) { - u64 a, b; + for (po = 0; po < SVC_SCAN_CHUNK; po += SEMA_PAGE_SIZE) { + u8 *page = buf + po; + u64 pg = gpa + po; + u32 off; - memcpy(&a, buf + off, 8); - memcpy(&b, buf + off + 8, 8); + for (off = 0; off + 16 <= SEMA_PAGE_SIZE; off += 8) { + u64 a, b; - /* (a) self-pointer -> base candidate */ - if (a >= KVA_LO && a < KVA_HI && - (a & 0xfff) == ((gpa + off) & 0xfff)) { - u64 cb = a - gpa - off; - u32 i; + memcpy(&a, page + off, 8); + memcpy(&b, page + off + 8, 8); - if ((cb & 0x3fffffffULL) == 0) { - for (i = 0; i < *ncand; i++) { - if (cands[i].base == cb) { - cands[i].count++; - break; + /* (a) self-pointer -> base candidate */ + if (a >= KVA_LO && a < KVA_HI && + (a & 0xfff) == ((pg + off) & 0xfff)) { + u64 cb = a - pg - off; + u32 i; + + if ((cb & 0x3fffffffULL) == 0) { + for (i = 0; i < *ncand; i++) { + if (cands[i].base == cb) { + cands[i].count++; + break; + } + } + if (i == *ncand && *ncand < 8) { + cands[*ncand].base = cb; + cands[*ncand].count = 1; + (*ncand)++; } - } - if (i == *ncand && *ncand < 8) { - cands[*ncand].base = cb; - cands[*ncand].count = 1; - (*ncand)++; } } - } - /* (b) {cpu_va, gpu_va} pair (page aligned: real mappings) */ - if (a >= KVA_LO && a < KVA_HI && - b >= SEMA_VA_LO && b < SEMA_VA_HI && - (a & 0xfff) == 0 && (b & 0xfff) == 0 && - *npairs < PAIR_MAX) { - u8 ok = 0; + /* (b) {cpu_va, gpu_va} pair (page aligned: real mappings) */ + if (a >= KVA_LO && a < KVA_HI && + b >= SEMA_VA_LO && b < SEMA_VA_HI && + (a & 0xfff) == 0 && (b & 0xfff) == 0 && + *npairs < PAIR_MAX) { + u8 ok = 0; - /* - * UVM's uvm_rm_mem_t stores vas[UVM_ID_CPU] - * (offset 48) directly before vas[UVM_ID_GPU0] - * (offset 56), so a pair taken from a live rm_mem - * is preceded by the header {u32 type; bitmap - * mapped_on}. Validate it when it fits in the same - * page: freed allocations and unrelated tables - * that happen to hold KVA/VA pairs do not match, - * and a stale entry for the same VA must not - * shadow the live one. - */ - if (off >= 48) { - u32 type; - u64 mapped_on; - - memcpy(&type, buf + off - 48, 4); - memcpy(&mapped_on, buf + off - 40, 8); - if ((type == 0 || type == 1) && - (mapped_on & 0x3) == 0x3) - ok = 1; + /* + * UVM's uvm_rm_mem_t stores vas[UVM_ID_CPU] + * (offset 48) directly before vas[UVM_ID_GPU0] + * (offset 56), so a pair taken from a live rm_mem + * is preceded by the header {u32 type; bitmap + * mapped_on}. Validate it when it fits in the same + * page: freed allocations and unrelated tables + * that happen to hold KVA/VA pairs do not match, + * and a stale entry for the same VA must not + * shadow the live one. + */ + if (off >= 48) { + u32 type; + u64 mapped_on; + + memcpy(&type, page + off - 48, 4); + memcpy(&mapped_on, page + off - 40, 8); + if ((type == 0 || type == 1) && + (mapped_on & 0x3) == 0x3) + ok = 1; + } + pairs[*npairs].cpu_va = a; + pairs[*npairs].gpu_va = b; + pairs[*npairs].valid_hdr = ok; + (*npairs)++; } - pairs[*npairs].cpu_va = a; - pairs[*npairs].gpu_va = b; - pairs[*npairs].valid_hdr = ok; - (*npairs)++; } + nonz++; } - nonz++; } } @@ -3896,7 +7375,7 @@ static void svc_add_pair_merge(u64 base, const struct raw_pair *p, u32 sorted_n) if (p->cpu_va < base) return; pa = p->cpu_va - base; - if (pa >= SEMA_PA_MAX || (pa & 0xfff)) + if (pa >= svc_pa_max() || (pa & 0xfff)) return; if (p->gpu_va < SEMA_VA_LO || p->gpu_va >= SEMA_VA_HI) return; @@ -4069,10 +7548,47 @@ struct svc_wait_op { }; static struct svc_wait_op svc_waits[SVC_WAIT_MAX]; static u32 svc_wait_n; /* acquires captured this entry */ +/* + * r484 THE STAGING-DRAIN COMPLETION EDGE: when the acquire gate holds an + * entry, the op can still have executed on an earlier decode whose gate did + * not trigger (svc_wait_n was 0 on that pass), and the once-only done + * bitmap then suppresses it on the first pass whose gate finally passes - + * the pass at which the dependency just became installed and the op's + * inputs changed. The release then applies and the guest drains a staging + * buffer holding the PRE-dependency bytes (streams_events s=2: the D2H + * copy read dc2 before the vadd_into wrote it, ran early on gate-less + * passes, was once-only-skipped on the passing pass, and the client + * drained a2+b2 instead of a2+b2+c1). Remember the last held (channel, + * entry) and force exactly ONE re-run of its ops on the first passing + * pass. + */ +static u32 svc_wait_hold_h; +static u32 svc_wait_hold_idx = 0xffffffffu; +static bool svc_wait_repass; static u32 svc_wait_ovf; /* acquires dropped past the array */ static u64 svc_wait_held; static u64 svc_wait_pass; static u64 svc_wait_giveup; +static u64 svc_wait_extended; /* r613: give-up deferred while a + * GR forward is owed */ +static u32 svc_wait_ext_log; +/* + * r613: episode extension bound while a GR forward is owed. Comfortably + * above wait-hold (default 4000 ms) + the absolute launch defer cap + * (SVC_GR_DEFER_CAP_MS, 15000 ms) + margin, and far below every client-side + * timeout (>= 60 s). + * + * r640: raised 25 s -> 45 s. The r639 streams2 transcript measured the + * readback's episode give up at waited_ms=28378 while the launch whose + * release it waits for only forwarded (GR-STALL-backend, software release + * authored) at 419322.99 - ~9 s AFTER the give-up - because the launch's + * own path paid the watermark defer (15 s cap) + arg-partial defer (8 s) + + * backend poll before it moved. The readback then ran out of order and the + * client read zeros (bad=65534). 45 s still leaves >15 s of headroom under + * the clients' >= 60 s timeouts, and only an episode with an unoffered + * candidate ever reaches it. + */ +#define SVC_WAIT_EXT_CAP_MS 45000u static u32 svc_wait_log; /* per-session held-row print budget */ static u32 svc_wait_pass_log; /* per-session resolved-row budget */ static u32 svc_remapconst_log; /* r343 FILL: SET_REMAP_CONST_A rows */ @@ -4095,6 +7611,10 @@ static bool svc_wait_rel_have; struct svc_wait_hold { bool active; u32 h; + u32 idx; /* r638: GPFIFO entry index this hold owns + * (0xffffffff when unset); the CE stream-order + * watermark uses it to detect a mutual citation + * (a held consumer entry, not a producer backlog) */ unsigned long since; }; #define SVC_WAIT_HOLD_MAX 16 @@ -4139,7 +7659,8 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) return; if (!buf) { - buf = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + /* r534: sized for svc_scan_pass's 64 KB chunked guest reads. */ + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); if (!buf) return; } @@ -4219,17 +7740,19 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) svc.false_trans, svc.gpfifo_skipped, svc.gpfifo_rescued, svc.releases_faked, svc.releases_recovered, svc.counter_rel_applied); - pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu gr_nocap=%llu/%llu gr_hold_exp=%llu gr_swrel=%llu gr_wedge=%llu pend_ovf=%llu\n", + pr_info("sema-svc: drain held=%llu starved=%llu forwarded=%llu faked=%llu rec=%llu ops_once=%llu reident=%llu opdef=%llu/%llu/%llu gr_nocap=%llu/%llu gr_owed=%llu gr_hold_exp=%llu gr_swrel=%llu gr_wedge=%llu pend_ovf=%llu\n", svc.gpfifo_held, svc.gpfifo_starved, svc.releases_forwarded, svc.releases_faked, svc.releases_recovered, svc.ops_once_skipped, svc.ops_reident, svc.opdef_held, svc.opdef_applied, svc.opdef_expired, svc.gr_nocap_held, svc.gr_nocap_consumed, + svc.gr_owed_resolved, svc.fwd_gr_hold_exp, svc.fwd_gr_stall_swrel, svc.gr_stale_wedge, svc_pending_ovf); - pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu altw=%llu residue=%llu wait=%llu/%llu giveup=%llu ovf=%u\n", + pr_info("sema-svc: counter pub_calls=%llu gr_rel=%llu reresolve=%llu pushpg_fb=%llu kbpg=%llu altw=%llu residue=%llu wait=%llu/%llu giveup=%llu ovf=%u\n", svc.counter_pub_calls, svc.gr_rel_applied, svc.sem_write_reresolve, svc.pushpg_fallback, + svc.kick_badpg, svc.pool_alt_applied, svc.pool_residue, svc_wait_held, svc_wait_pass, svc_wait_giveup, svc_wait_ovf); @@ -4250,12 +7773,13 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) * content" window. n approaching SVC_LEDGER_MAX or evict > 0 in a * client's window is the tell. */ - pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu pinwins=%llu reloc_div=%llu rewrites=%llu pinrep=%llu alt_skip=%llu\n", + pr_info("sema-svc: ledger n=%u/%u stores=%llu evict=%llu resets=%llu diverges=%llu pinwins=%llu reloc_div=%llu rewrites=%llu pinrep=%llu alt_skip=%llu dual=%llu/%u\n", svc_ledger_n, SVC_LEDGER_MAX, svc_ledger_stores, svc_ledger_evictions, svc_ledger_evict_resets, svc_ledger_diverges, svc_ledger_pinwins, svc_ledger_reloc_diverges, svc_ledger_rewrites, - svc_ledger_pin_repairs, svc_pool_alt_skipped); + svc_ledger_pin_repairs, svc_pool_alt_skipped, + svc.pool_dual_applied, svc_pool_boot_conflicts); /* * P6b wave 1: guest-forwarding summary. Printed whenever any counter * moved (not only on requests) so a run that captured no candidate is @@ -4317,6 +7841,10 @@ static void svc_rebuild(struct nv_vgpu_device *vgpu) #define CE_M_LINE_COUNT 0x41c #define CE_M_PITCH_IN 0x410 #define CE_M_PITCH_OUT 0x414 +#define CE_M_SRC_ORIGIN_X 0x744 +#define CE_M_SRC_ORIGIN_Y 0x748 +#define CE_M_DST_ORIGIN_X 0x74c +#define CE_M_DST_ORIGIN_Y 0x750 #define CE_M_SET_REMAP_CONST_A 0x700 #define CE_M_SET_REMAP_CONST_B 0x704 #define CE_M_SET_REMAP_COMPONENTS 0x708 @@ -4424,6 +7952,18 @@ struct svc_push_state { u32 pitch_in, pitch_out; u32 have_pitch_in, have_pitch_out; u32 line_count; + /* + * SESSION 153 CONT.25: C5B5 SRC_ORIGIN_X/Y (0x744/0x748) and + * DST_ORIGIN_X/Y (0x74c/0x750). The X DDX's read/present paths + * position each strip with SRC_ORIGIN (x in bytes, y in rows for + * compSizeFactor == Bpp == 4; class C5B5 has no null remapper) + * instead of folding it into OFFSET_IN. The linear shadow holds + * the pixels linearly, so the effective operand becomes + * base + origin_y * pitch + origin_x. + */ + u32 src_origin_x, src_origin_y; + u32 dst_origin_x, dst_origin_y; + u32 have_src_origin, have_dst_origin; u32 sem_va_hi, sem_va_lo, sem_payload; u32 have_sem; u32 have_remap; @@ -4447,6 +7987,66 @@ struct svc_push_state { u32 qmd[SVC_QMD_WORDS]; }; +/* + * SESSION 153 CONT.25: CE register state persists across pushes on the + * same channel/subchannel in hardware. The X DDX's read-image loop pushes + * the full setup once (SET_SRC_BLOCK_SIZE / OFFSET_IN / PITCH / + * LINE_LENGTH, a 112-byte push) and then per-strip DELTA pushes that only + * carry SRC_ORIGIN, SET_SEMAPHORE_A/PAYLOAD, OFFSET_OUT, (LINE_COUNT) and + * LAUNCH_DMA (a 52-byte push; raw witness: the delta launch decodes + * line=0/in=0/have_out=1/have_sem=1 on h=0x1000013). Decoding each push + * in isolation left line_length 0, so the transfer-shaped delta launch + * built NO op: every other 9-row strip never executed and the readback + * kept half the rows black (the visible even/odd banding). Remember the + * last full setup per channel and fill a delta launch's missing fields + * from it. + */ +struct svc_ce_saved { + u64 h_object; + u64 offset_in; + u32 pitch_in, pitch_out, line_length, line_count; + u32 src_phys_mode, dst_phys_mode; + u32 have_in, have_pitch_in, have_pitch_out; + u32 have_src_mode, have_dst_mode; +}; +#define SVC_CE_SAVED_N 8 +static struct svc_ce_saved svc_ce_saved[SVC_CE_SAVED_N]; +static u32 svc_ce_saved_next; +static u32 svc_cedelta_log; + +static struct svc_ce_saved *svc_ce_saved_slot(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_CE_SAVED_N; i++) + if (svc_ce_saved[i].h_object == h) + return &svc_ce_saved[i]; + return NULL; +} + +static void svc_ce_saved_store(const struct svc_push_state *st, u64 h) +{ + struct svc_ce_saved *s = svc_ce_saved_slot(h); + + if (!s) { + s = &svc_ce_saved[svc_ce_saved_next++ % SVC_CE_SAVED_N]; + memset(s, 0, sizeof(*s)); + s->h_object = h; + } + s->offset_in = st->offset_in; + s->pitch_in = st->pitch_in; + s->pitch_out = st->pitch_out; + s->line_length = st->line_length; + s->line_count = st->line_count; + s->src_phys_mode = st->src_phys_mode; + s->dst_phys_mode = st->dst_phys_mode; + s->have_in = st->have_in; + s->have_pitch_in = st->have_pitch_in; + s->have_pitch_out = st->have_pitch_out; + s->have_src_mode = st->have_src_mode; + s->have_dst_mode = st->have_dst_mode; +} + /* Host CUDA bridge (gsp_cuda_bridge.c): launch transport + service hook * (prototypes in gsp_cuda_bridge.h; helpers exported to it below). */ @@ -4493,6 +8093,22 @@ static int svc_op_translate(struct nv_vgpu_device *vgpu, u64 va, u64 *addr, svc.op_tr_src = 1; return 0; } + /* + * SESSION 153-CONTINUED-44 (r1027): the CONT-43 walk-first for + * below-floor CE operands is REVERTED. With it live, the moment the + * client's channels actually ran (submissions 33646 via the slot + * hold + zero-put-skip), the guest kernel oopsed repeatedly + * (`clear_page_erms` page fault from `rm_ioctl [nvidia]`, then + * `_mapInsertBase` general protection, Xorg exited) - i.e. the + * walk-first resolutions for below-floor COPY operands clobbered + * guest kernel memory, the r365/r369 corruption class the fill + * guard exists for. The DDX screen-repaint poison that CONT-43 + * fixed is a FILL class and stays fixed by svc_fill_floor_walk in + * svc_fill_1d; the copy path returns to the historical order + * (pins -> scanned map -> walk -> discovered -> targeted scan), + * which is the order every pre-r1025 boot (including the working + * r1024) ran with. + */ if (!svc_map_lookup(va, &pa)) { *addr = pa; *aperture = CE_APERTURE_SYS; @@ -4857,6 +8473,37 @@ static int svc_fill(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, done_out); } +/* + * SESSION 153-CONT-43: resolve a below-floor fill destination through the + * guest's published page directories before the pin/ledger/map chain. + * + * Below SVC_VAS_FLOOR the driver maps its ABI/MMIO window (the X server's + * screen VA 0x120400000 family, the client pushbuffer region, ...). The + * content-scanned map and the landing ledger are heuristics built for the + * above-floor client VAs; for these low VA numbers a page-table page + * discovered by the scanner can hold a false identity pair (va == pa), and + * the LOWFILL guard then refuses every fill that the pair shadows - the + * observed X server repaint death (`LOWFILL-REFUSED va=0x120458c00 + * pa=0x120458c00`, fill done=52800 rc=-2, replayed ~15 Hz forever). The + * guest's page tables are the truth for this region: resolve through them + * first. Returns true when a translation was applied. + */ +static bool svc_fill_floor_walk(struct nv_vgpu_device *vgpu, u64 va0, + u64 *addr, u32 *aperture) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + + if (va0 >= SVC_VAS_FLOOR) + return false; + if (svc_walk_vaspaces(vgpu, va0 & ~0xfffULL, &wpa, &wap)) + return false; + *addr = wpa + (va0 & 0xfff); + *aperture = (wap == SVC_APERTURE_FB) ? CE_APERTURE_VID + : CE_APERTURE_SYS; + return true; +} + static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 elem, u32 count, u32 phys, u32 aperture, u64 *done_out) { @@ -4889,7 +8536,55 @@ static int svc_fill_1d(struct nv_vgpu_device *vgpu, u64 addr, u64 value, u32 ele u32 aper; u32 prov = 0; /* r384: 1 pin, 2 ledger, 3 walk/scan */ - if (svc_vas_has(addr)) { + /* + * r584: a VID-aperture fill whose destination is a vidmem + * WINDOW VA (the guest's 0x2xxxxxxxx family) is offset- + * mapped by construction: VA - SVC_VIDMEM_VA_BASE is the + * FB offset into the emulated BAR1 shadow (the svc_lookup + * window convention; see also the 0x200080000 rationale at + * SVC_VIDMEM_VA_BASE). The generic chain below (pin -> + * ledger -> translate) answered SYS for the r583 boot's CE + * channel-setup fill to 0x204820000 (a stale landing), the + * r369 below-floor guard refused it, the fill never ran, + * and its DSTPEND note stalled GR launches to the fail- + * closed cap. Window VAs resolve structurally - no guest + * page table can express an "engineering window" offset. + */ + if (aperture == CE_APERTURE_VID && + va0 >= SVC_VIDMEM_VA_BASE && + va0 - SVC_VIDMEM_VA_BASE + total <= + svc_fb_shadow_size(vgpu)) { + addr = va0 - SVC_VIDMEM_VA_BASE; + /* aperture stays CE_APERTURE_VID: the whole-range + * contiguity block below writes it into the shadow */ + prov = 4; /* structural window convention */ + if (svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill-window va=0x%llx -> fb offset 0x%llx (vidmem window convention) elem=%u count=%u value=0x%llx\n", + va0, addr, elem, count, value); + } + } else if (svc_fill_floor_walk(vgpu, va0, &addr, &aperture)) { + /* + * SESSION 153-CONT-43: below-floor VA (the driver's + * ABI/MMIO mapping window). The guest's published + * page directory is the truth there; a pin or a stale + * landing-table entry can hold a false identity pair + * (va==pa), observed live as the X server's screen + * repaint dying ~44 rows in (`LOWFILL-REFUSED + * va=0x120458c00 pa=0x120458c00` repeating at + * ~15 Hz, the screen frozen on a stale fill + * artifact). The walk resolves the same VA through + * the guest's 2 MB big page to vidmem + * (pa=0x200000+off), which is what the driver meant. + */ + prov = 6; + if (svc_lowfill_log < 64) { + svc_lowfill_log++; + pr_info("sema-lowfill-walk va=0x%llx -> pa=0x%llx ap=%u elem=%u count=%u value=0x%llx\n", + va0, addr, aperture, elem, count, + value); + } + } else if (svc_vas_has(addr)) { shadow = true; } else if (!svc_pin_lookup_any(addr, &pa, &aper)) { u64 lpa = 0; @@ -5242,7 +8937,13 @@ static struct svc_fwd_cand svc_fwd_cand; */ #define SVC_GR_QMD_WORDS 64 #define SVC_GR_INLINE_MAX 8 -#define SVC_GR_INLINE_BYTES 64 +/* + * SESSION 153-CONTINUED-88: 256 -> 2048 (static_asserts below tie it to + * GSP_BRIDGE_INLINE_BYTES / GSP_PHYS_GR_INLINE_BYTES). The GL window's real + * program upload payloads are 1536 B / 1408 B; the 256 B cap truncated them + * to their first 256 bytes, so the engine never saw the full image. + */ +#define SVC_GR_INLINE_BYTES 2048 #define SVC_GR_POINTER_MAX 8 #define SVC_GR_PAGES_MAX 16 /* @@ -5263,7 +8964,35 @@ static struct svc_fwd_cand svc_fwd_cand; * so the hand-off array is never the binding limit. */ #define SVC_GR_DATA_MAX 32768 -#define SVC_GR_PUSH_WORDS 256 +/* + * r929: 256 -> 512 dwords (2 KB) so a whole GL 3D section push (the X + * server's one-shot stream is 1524 B = 381 dwords) survives in raw[] - the + * 3D stream replay needs the window verbatim, not just its tail. Must equal + * GSP_PHYS_GR_RAW_WORDS (static_assert below). + * + * SESSION 153-CONTINUED-45: 512 -> 2048 dwords (8 KB). The DIRECT GL + * client (glcore, channel family 0xbeef) emits its per-frame render pushes + * at 5016-7144 B (1254-1786 dwords) with the draw state (program binds, + * vertex-array bases, constant uploads) spread across the whole push; the + * 512-dword tail kept only the release block, so the section capture could + * never satisfy `len == push_n*4` and the verbatim 3D replay never marked + * (CONT-44: zero src=stream captures on the client channel). The window + * slice still has to fit the backend's method block; only the raw[] + * retention is widened here. + * + * SESSION 154-1: 2048 -> 4096 dwords (16 KB). The direct client's + * occasional full-frame push is 12288 B (3072 dwords; r1483 + * `section open (draw) push=0x4310110 len=12288`, window 2115 words at + * off 0xef4). With the 2048-word tail the window start fell before the + * retained tail, `len == push_n*4` failed and the whole capture was + * dropped: `3D mark failed ... raw_ok=0 fit_ok=0 (rawlen=3072 + * pushn=2048 len=12288)`. The launch then replayed the previous + * window's fallback words and the client's real draw never reached the + * engine. 4096 words carries the whole 12288-B push so the existing + * method-aligned svc_gl_stream_chunks() split (<=448-word chunks, six + * per frame) can run. Must stay equal to GSP_PHYS_GR_RAW_WORDS. + */ +#define SVC_GR_PUSH_WORDS 4096 #define SVC_GR_QMD_WORD_NONE 0xffffffffu /* svc_gr_cand.rel_flags bits (decoded from qmd[24] and qmd[4]). */ #define SVC_GR_REL_REDUCTION 0x1u @@ -5290,6 +9019,13 @@ struct svc_gr_data { u64 pa; /* guest PA, page aligned (0 = unresolved) */ u64 reloc_va; /* m102: server-managed VA the page maps to */ u8 ap; /* SVC_APERTURE_* of pa */ + /* + * r632: the page is served from the VA shadow lane instead of a guest + * page (no resolver and no landing answered for it). The submit build + * hands the shadow page's host address to the bounce machinery - see + * the fill's shadow-serve branch and the r632 note there. + */ + u8 shadow; }; /* @@ -5306,6 +9042,9 @@ struct svc_gr_data_win { u64 reloc_base; /* relocated window base */ }; +/* CONT-87: max program binds carried from a captured window (mirrors + * SVC_GL_PROG_MAX; defined here because the candidate struct precedes it). */ +#define SVC_GR_CAND_PROG_BINDS 8 struct svc_gr_cand_rec { bool valid; bool offered; /* consumed by a request or a fallback */ @@ -5325,6 +9064,35 @@ struct svc_gr_cand_rec { u32 push_n; /* raw tail words kept in raw[] */ u32 push_qmd_hdr; /* raw[] offset of the 0x318 run header */ u32 push_qmd_end; /* raw[] offset just past that run */ + /* + * r929 (GL 3D path): this capture is a real C597 method stream (the + * section carried its own operands, op_src=stream) whose whole push + * fits in raw[]. The backend replays raw[stream_off..stream_off+ + * stream_n) verbatim as a 3D method window (SET_OBJECT TURING_A + + * the engine's own uploads/binds) instead of running the synthesized + * compute QMD. The window is sliced from the section's C597 bind. + */ + bool stream_3d; + u32 stream_off; /* dword offset of the window inside raw[] */ + u32 stream_n; /* dwords of the replayable window */ + u32 stream_rel; /* stream-relative dword offset of this + * window (0 = the section's first chunk); the + * backend uses it to detect a dropped earlier + * chunk of the same push */ + /* + * SESSION 153-CONTINUED-87: every SET_PIPELINE_PROGRAM_ADDRESS pair + * the captured window itself binds (VA + pipeline slot), copied from + * svc_gl at capture. The launch builder re-emits them AHEAD of the + * window: the captured per-frame window carries draws (BEGIN 0x1618) + * BEFORE its own program binds (measured: draw at window word 93, + * j=1 pair at word 184), on the real hardware the bind already + * happened in an earlier push - on the replay channel's fresh + * context the early draws would validate an UNBOUND stage and latch + * the Shader Program Header fault (the r1277+ 9/11 wall). + */ + u32 n_prog_binds; + u64 prog_bind_va[SVC_GR_CAND_PROG_BINDS]; + u32 prog_bind_slot[SVC_GR_CAND_PROG_BINDS]; u32 raw[SVC_GR_PUSH_WORDS]; u32 qmd[SVC_GR_QMD_WORDS]; u64 qmd_dev_va; @@ -5342,6 +9110,7 @@ struct svc_gr_cand_rec { u32 pointer_word[SVC_GR_POINTER_MAX]; /* QMD word of the pair */ }; + /* * fail4: the capture was a single slot; two concurrent launches (streams2's * vadd pair on two channels) overwrote each other's candidate while the @@ -5350,11 +9119,40 @@ struct svc_gr_cand_rec { * side by side in a ring; the accessor macro addresses the current slot, * so every existing use keeps working on the selected candidate, and * capture/consume select the slot instead of clobbering it. + * + * SESSION 153-CONTINUED-45: 4 -> 8 slots. The DIRECT GL client's draw + * sections are 1470-dword windows split into up to four <=448-dword chunks + * (see svc_gl_stream_chunks); with four slots a single big push could + * evict its own later chunks (or the X channel's pending candidate) before + * the pump drained them. */ -#define SVC_GR_CAND_RING 4 +#define SVC_GR_CAND_RING 8 static struct svc_gr_cand_rec svc_gr_cand_ring[SVC_GR_CAND_RING]; static u32 svc_gr_cand_pos; #define svc_gr_cand (svc_gr_cand_ring[svc_gr_cand_pos]) + +/* + * r613: is a GR forward owed for this device - a captured candidate that has + * not been offered (submitted or refused) yet? The wait gate uses this to + * distinguish "the acquire's release can never be installed by firmware + * alone" from "the release is pending behind OUR OWN launch deferral": the + * r612 streams2 capture gave up a D2H readback's acquire (0x20480ffe0 pay=4, + * cur=2) at waited_ms=10333 while the vadd launches that author that + * semaphore sat deferred by the CE watermark for the full 15 s cap - the + * readback then ran BEFORE the kernels and the client read zeros + * (bad=65534). While a forward is owed, the episode extends (bounded by + * wait-hold + defer cap + margin) instead of failing open. + */ +static bool svc_gr_forward_pending(void) +{ + u32 i; + + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + !svc_gr_cand_ring[i].offered) + return true; + return false; +} /* * r383: push-VA history of recent captures, beyond the (four-slot) * candidate ring's lifetime. A replayed decode of a complete QMD asks @@ -5448,8 +9246,8 @@ static_assert(SVC_GR_PUSH_WORDS == GSP_PHYS_GR_RAW_WORDS, "the sema raw push cap must match the backend ABI"); static_assert(SVC_GR_INLINE_MAX <= GSP_PHYS_GR_MAX_INLINE, "the sema GR inline capture must fit the backend ABI"); -static_assert(sizeof(struct svc_gr_inline) == 80 && - offsetof(struct svc_gr_inline, data) == 12, +static_assert(offsetof(struct svc_gr_inline, data) == 12 && + sizeof(struct svc_gr_inline) >= 12 + SVC_GR_INLINE_BYTES, "the sema GR inline record must stay layout-compatible with the backend"); static_assert(sizeof(struct svc_gr_inline) == sizeof(struct gsp_bridge_inline) && offsetof(struct svc_gr_inline, dst_va) == @@ -5474,6 +9272,37 @@ static_assert(sizeof(struct svc_gr_page) == 24, "the sema GR page descriptor must stay {u64,u64,u8,u32} = 24 bytes"); static u64 svc_push_hash(const u8 *p, u32 len); +/* SESSION 149 R15: exactly-once guard for the hash-verified replay capture. */ +static bool svc_gr_cand_push_seen(u64 push_va, u64 hash); +/* SESSION 149 R16 (M3): trusted-walk VA read, defined below the decode. */ +static int svc_read_va_trusted(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len); +/* r1475: general VA read (trusted sources + vidmem window), defined below. */ +static int svc_read_va(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len); +/* + * SESSION 149 R17 (M4): the section's memory-writing upload commands are + * applied through the shared inline-write path (defined below the section + * code) with its (push, hash, dst, bytes) once-only guard. + */ +static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, + const u8 *data, u32 bytes, u32 ch_h); +static bool svc_inline_once_seen(u64 push_va, u64 hash, u64 dst, u32 bytes); +static void svc_inline_once_store(u64 push_va, u64 hash, u64 dst, u32 bytes); + +/* + * SESSION 153-CONTINUED-88: newest captured inline-write payload landing + * exactly at `dst_va` with at least `min_len` bytes (defined next to the + * destination ring below). Used by svc_gl_flush to deliver the window's REAL + * program uploads (the driver's own SPH+SASS images) to the engine-visible + * page instead of the minimal authored blob. + */ +static bool svc_grce_dst_newest(u64 dst_va, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz); +/* r1476: newest captured upload whose first word is the given SPH magic + * (used when a window's own bind-VA upload was never captured). */ +static bool svc_grce_dst_sph(u32 sph_w0, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz); /* QMD field decode: the exact shape the host CUDA bridge uses. */ static void svc_gr_qmd_decode(struct svc_gr_cand_rec *c) @@ -5592,10 +9421,11 @@ static void svc_gr_cand_capture(struct svc_channel *ch, svc_gr_cand.param_size, svc_gr_cand.rel_va, svc_gr_cand.rel_payload, svc_gr_cand.rel_flags, svc_gr_cand.n_pointer, svc_gr_cand.n_inline); - if (push && push_len >= 16) { + if (gsp_sema_dbg && svc_gr_push_dump < 4 && push && push_len >= 16) { u32 n = push_len / 4; u32 i, j; + svc_gr_push_dump++; if (n > 256) n = 256; for (i = 0; i < n; i += 16) { @@ -5612,6 +9442,3449 @@ static void svc_gr_cand_capture(struct svc_channel *ch, } /* + * SESSION 149 R14 (sema_gl_xlate, default 0 = off): GL/3D direct method + * stream -> synthesized GR candidate recognizer. + * + * The GL channel's large per-frame pushes bind the TURING_A 3D class + * (0xC597) on the channel subchannel and carry no CUDA QMD, so the + * inline-QMD capture above never fires for them (the victim-section + * classification of the r882 stream, push VA 0x120010004 / 1524 B: + * SET_OBJECT C597, SET_TEX_HEADER_POOL_A -> 0x1214fa220/0x10, 20x + * LINE_LENGTH_IN(0x180)+LAUNCH_DMA(0x1b0) inline uploads to + * 0x1214f8f00+0x100k, SET_PIPELINE_PROGRAM_ADDRESS_A/B(0x2014/0x2018 + * + j*64) -> 0x121201b00, BIND_GROUP_CONSTANT_BUFFER(0)). This section + * collects the OBSERVED VA operands (nothing is invented) and synthesizes + * the 64-word QMD those fields would carry; the candidate then flows + * through the unchanged svc_gr_request / svc_fwd_gr_work / + * gsp_phys_backend_guest_gr_submit machinery. + * + * Strictly additive: gated by sema_gl_xlate and inert for every stream + * without the C597 class bind (0xC5B5 copy-class and 902D 2D binds never + * open a section). Class numbers from the public class headers + * (clc597.h: TURING_A 0xC597). + */ +static bool gsp_sema_gl_xlate; +module_param_named(sema_gl_xlate, gsp_sema_gl_xlate, bool, 0644); +MODULE_PARM_DESC(sema_gl_xlate, + "GL/3D (C597) direct stream -> synthesized GR candidate (default 0 = off)"); + +/* + * r1475: fixed-CB page copy into the forced MME seed page. + * + * The client's VS reads its transform constants from the fixed constant + * buffer whose base is captured by SET_MME_SHADOW_SCRATCH(0x18) (method + * 0x3460) as (VA >> 8). The GR launch builder forces the MME seed to a + * mapped page (sema_gl_mme_seed_val) so the engine always has a base, but + * that page holds screen/stale data, not the client's uniforms. When this + * is set, the service copies the client's real fixed-CB page into that + * forced seed page at decode time, before the launch, so the VS reads the + * client's constants; the MME's later constant writes (offsets + * 0x20/0x30/0x1480/0x1e0/0xc8) overwrite only their own offsets and the + * VS's 0x80.. region survives. 0 = off. + */ +static unsigned long long sema_gl_cb_copy_to; +module_param_named(sema_gl_cb_copy_to, sema_gl_cb_copy_to, ullong, 0644); +MODULE_PARM_DESC(sema_gl_cb_copy_to, + "VA of the forced MME seed page to receive the client's fixed-CB page (0 = off)"); + +/* + * r1476: explicit fixed-CB source VA. The client's 0x3460 write is not always + * decoded before the first draw (the GL client's channel init can land after + * the first replayed window), so allow the caller to name the CB base + * directly (observed 0x0cd70000 for the glxgears beef channel). 0 = use the + * captured scratch18 value. + */ +static unsigned long long sema_gl_cb_src_va; +module_param_named(sema_gl_cb_src_va, sema_gl_cb_src_va, ullong, 0644); +MODULE_PARM_DESC(sema_gl_cb_src_va, + "explicit fixed-CB source VA for sema_gl_cb_copy_to (0 = captured scratch18)"); + +/* + * SESSION 153-CONTINUED-4: native publish-counter advance (pace-wall fix). + * + * The GLX client's swap-completion ladder waits for the u32 at +0x8 of the + * driver-shared page whose struct magic is 0x0e836365 at +0x4 (the page is + * CPU-mapped from the X driver's memfd; r149-R3 watchpoint forensics named + * it "the publish counter our emulation never advances"). In a real stack + * the GPU completion of the client's kick channel (h=0xbfef00xx family) + * advances it to 3; ours never did, so every wait fell through the client's + * multi-second give-up ladder (140 FPS -> 0.045 FPS). A guest daemon write + * proved the value: u32 3 -> "702 frames in 5.0 seconds = 140.398 FPS". + * + * Native replacement: when the pump RETIRES a push of a kick-family channel + * (svc_replay_note consume path), write 3 to the page. The page moves in + * guest RAM per boot, so it is discovered once by content: the full 8 GiB + * RAM scan shows the magic page-aligned 16-byte signature + * (magic 0x0e836365 at +0x4, counter <= 3 at +0x8) matches exactly ONE page + * (verified unique on the r934 image before implementating). Discovery walks + * guest RAM descending (the driver's allocations live high), is rate-limited + * and attempt-capped, and every write is read back; a mismatch drops the + * discovered page so a stale/foreign target can never persist. Guarded by + * sema_pub_adv (default on; the family guard makes it inert for CUDA lanes) + * and fully logged. + */ +static bool gsp_sema_pub_adv = true; +module_param_named(sema_pub_adv, gsp_sema_pub_adv, bool, 0644); +MODULE_PARM_DESC(sema_pub_adv, + "Advance the GLX publish counter (u32 3) when a kick-channel push retires (default 1)"); +static u64 gsp_sema_pub_pa; +module_param_named(sema_pub_pa, gsp_sema_pub_pa, ullong, 0644); +MODULE_PARM_DESC(sema_pub_pa, + "Force the publish page PA for sema_pub_adv (0 = content-discover per boot)"); + +/* + * SESSION 153-CONTINUED-36: the GL shared-state keeper (automates the + * slot-word handshake that gsp-phys-shared-bit0 previously did by hand). + * + * Every direct GLX client receives a shared page from the X driver whose + * word at +8 is a state field; glcore's device init reads it and SKIPS its + * own class-0xc46f channel creation while bit 0 is set. The emulated X + * driver publishes 3 (bits 0+1), so direct clients never allocate a + * channel and never render (witness: no c46f alloc in the RM log, glClear + * a no-op, err@ReadPixels=0x502). Clearing bit 0 while the client inits + * lets the channel alloc proceed (r999-r1002 by hand, r1012 by the + * debugfs write). The page moves per boot and is named here by guest PA; + * the keeper clears bit 0 on every service pass (up to ~1 kHz while the + * pump is active) so the client cannot re-publish 3 fast enough to block + * a later client's init. The PA is discovered by content (cookie word + * 0xc0b43fec at +4 / word5 0xcaf00006) with the host helper slot_hold.py + * or by hand, then written to this parameter; witness is bounded. + */ +static u64 gsp_sema_shared_hold_pa; +module_param_named(sema_shared_hold_pa, gsp_sema_shared_hold_pa, ullong, 0644); +MODULE_PARM_DESC(sema_shared_hold_pa, + "Guest PA of the GL shared-state page; clear bit 0 of its +8 word every pass (0 = off)"); +static u32 svc_shared_hold_log; + +static void svc_shared_hold(struct nv_vgpu_device *vgpu) +{ + u32 w[2]; + + if (!gsp_sema_shared_hold_pa) + return; + if (svc_read(vgpu, gsp_sema_shared_hold_pa + 8, w, sizeof(w))) + return; + if (!(w[0] & 1u)) + return; + w[0] &= ~1u; + if (svc_write(vgpu, gsp_sema_shared_hold_pa + 8, w, 4)) + return; + svc_shared_hold_log++; + if (svc_shared_hold_log <= 16 || + (svc_shared_hold_log % 4096) == 0) + pr_info("sema-shared: held pa=0x%llx word -> 0x%x (n=%u)\n", + gsp_sema_shared_hold_pa, w[0], svc_shared_hold_log); +} + +/* + * SESSION 149 R17 (M4): one recorded section upload command. The 3D-class + * i2m shape is a 0x180 (LINE_LENGTH_IN) run naming the destination followed + * by a 0x1b0 (LAUNCH_DMA + LOAD_INLINE_DATA) run carrying the payload. + */ +#define SVC_GL_UPLOAD_MAX 24 +struct svc_gl_up { + u64 dst_va; /* from the 0x180 run's OFFSET_OUT_UPPER|OFFSET_OUT */ + u32 len; /* payload bytes copied (payload = run data minus cfg) */ + u8 data[SVC_GR_INLINE_BYTES]; +}; + +#define SVC_GL_PROG_MAX 8 +struct svc_gl_sec { + bool seen; + u32 subch; + u32 start_off; /* byte offset of the C597 SET_OBJECT that opened the + * section - the 3D replay window starts here (r929: + * the prelude before it carries the channel's own + * host-sem/2D ops whose VAs are not mapped in the + * replay channel and FAULT the method walk) */ + u64 prog_va; /* SET_PIPELINE_PROGRAM_ADDRESS_A/B(j) upper|lower */ + /* + * SESSION 153-CONTINUED-11: every SET_PIPELINE_PROGRAM_ADDRESS pair + * the window binds (the engine validates the SPH at EACH bound + * program VA when a draw issues - r944/r946 faults on the draw's + * program pages), with the pipeline slot's shader type collected + * from SET_PIPELINE_SHADER(j) (0x2000+j*0x40, TYPE bits 7:4; 0xff = + * unknown). The per-VA inline records author the program bytes the + * driver's upload never delivered. + */ + u64 progs[SVC_GL_PROG_MAX]; + u8 prog_slot[SVC_GL_PROG_MAX]; + u32 n_progs; + u64 param_va; /* SET_TEX_HEADER_POOL_A/B/C upper|lower */ + u32 param_size; /* observed pool value word (0x10 on the r882 stream) */ + u64 rel_va; /* last VALID release decoded on the channel */ + u32 rel_payload; + u64 ptr[SVC_GR_POINTER_MAX]; + u32 n_ptr; + /* + * SESSION 149 R17 (M4): the section's memory-writing upload commands + * collected in order. On real hardware the 3D engine executes these + * i2m copies (constants, tex pools, ...); in this emulation no engine + * ran them, so every destination stayed zero. svc_gl_flush applies + * the recorded bytes through svc_grce_inline_write (FB aperture -> + * vidmem shadow, sysmem -> guest RAM, unresolved -> VA shadow). + */ + u64 up_dst; /* destination of the last 0x180 run */ + u32 up_bytes; /* its length word (stream); witness only */ + u32 n_up; + struct svc_gl_up up[SVC_GL_UPLOAD_MAX]; + /* + * SESSION 149 R15 (M2-S1): bytes of the first LAUNCH_DMA / + * LOAD_INLINE_DATA run observed inside the section (the observed + * upload payload). Attached to the synthesized candidate as an + * inline record at the program VA so the backend's window-0 shadow + * copy has a non-empty source even when the guest program page + * resolves empty (the GL program page is not among the section's + * own upload destinations). + */ + u8 blob[SVC_GR_INLINE_BYTES]; + u32 blob_len; + /* + * SESSION 153-CONTINUED-18: the kick notification this push carried + * (class2d SET_NOTIFY_A/B + NOTIFY on subchannel 3, decoded after the + * C597 bind that opened this section). Executed at flush time by + * svc_kick_notify_exec when both the target and the NOTIFY are in + * this same push (the short variant carries the NOTIFY without the + * arm pair; it never executes and stays witness-only). + */ + u64 kick_notify_va; + u32 kick_notify_val; + bool kick_notify_seen; +}; +static struct svc_gl_sec svc_gl; /* service thread only */ +/* + * SESSION 153-CONTINUED-11: the channel's sticky per-slot shader types + * (SET_PIPELINE_SHADER(j) TYPE field). Sticky like the C597 bind: the init + * push binds the programs once; later draw pushes reuse the state. 0xff = + * never seen. File-static so a section reset cannot wipe it. + */ +static u8 svc_gl_shader_type[32] = { + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff }; +/* + * SESSION 153-CONTINUED-10 (task 1): the C597 class ATTACHMENT. + * + * A SET_OBJECT bind is sticky on real hardware: the class stays bound to + * the channel subchannel across pushes until another bind replaces it. The + * GLX server's C597 SET_OBJECT arrives once (the driver's init push at + * 0x120010004); every later per-frame CONTENT push (588/644/756 B, ~5/s) + * carries real C597 draws (NVC597_BEGIN 0x1618 + vertex streams + colour + * target + LOAD_CONSTANT_BUFFER payloads) with NO per-push bind, so the + * SET_OBJECT-only recognizer below never opens a section for them and the + * 3D replay lane never sees a draw. Remember (h_object, subch) when a C597 + * bind is seen - including catches of the init push - and treat any later + * method run on that pair, on engine_type==0, as C597 class space. + */ +static u64 svc_gl_att_h; /* channel whose subch carries the C597 bind */ +static u32 svc_gl_att_subch; +static u32 svc_gl_draw_log; /* bounded draw-section-open witness */ +/* + * SESSION 153-CONTINUED-13: the C597 attachment is per-channel. The single + * (h_object, subch) slot was overwritten by the RM channel's periodic + * C597-bind pushes (h=0xbfef0002, observed every ~7 s on this boot), after + * which the GLX client's draw pushes on 0x1000013 failed the BEGIN + * recogniser's `svc_gl_att_h == ch->h_object` test and no draw section was + * ever opened (a boot-level, order-dependent race: the r954 boot decoded + * no such pushes). A small table (first match wins, dedup on note) keeps + * every channel's bind sticky independently, so the draw channel's + * attachment cannot be stolen. + */ +#define SVC_GL_ATT_MAX 8 +static u64 svc_gl_att_tab_h[SVC_GL_ATT_MAX]; +static u8 svc_gl_att_tab_sub[SVC_GL_ATT_MAX]; + +static void svc_gl_att_note(u64 h, u32 subch) +{ + u32 i, free = SVC_GL_ATT_MAX; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) { + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) { + /* refresh LRU position: move to slot 0 */ + if (i) { + memmove(&svc_gl_att_tab_h[1], + &svc_gl_att_tab_h[0], + (size_t)i * sizeof(u64)); + memmove(&svc_gl_att_tab_sub[1], + &svc_gl_att_tab_sub[0], + (size_t)i); + svc_gl_att_tab_h[0] = h; + svc_gl_att_tab_sub[0] = (u8)subch; + } + return; + } + if (!svc_gl_att_tab_h[i] && free == SVC_GL_ATT_MAX) + free = i; + } + if (free == SVC_GL_ATT_MAX) { + /* table full: evict the oldest (last) slot */ + free = SVC_GL_ATT_MAX - 1; + } + memmove(&svc_gl_att_tab_h[1], &svc_gl_att_tab_h[0], + (size_t)free * sizeof(u64)); + memmove(&svc_gl_att_tab_sub[1], &svc_gl_att_tab_sub[0], + (size_t)free); + svc_gl_att_tab_h[0] = h; + svc_gl_att_tab_sub[0] = (u8)subch; + /* keep the legacy single slot pointing at the newest note (logging) */ + svc_gl_att_h = h; + svc_gl_att_subch = subch; +} + +static bool svc_gl_att_has(u64 h, u32 subch) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) + return true; + return false; +} + +static u32 svc_gl_att_subch_of(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h) + return svc_gl_att_tab_sub[i]; + return 0xffu; +} + +static void svc_gl_att_clear(u64 h, u32 subch) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h && + svc_gl_att_tab_sub[i] == (u8)subch) { + u32 n = SVC_GL_ATT_MAX - 1 - i; + + if (n) + memmove(&svc_gl_att_tab_h[i], + &svc_gl_att_tab_h[i + 1], + (size_t)n * sizeof(u64)); + if (n) + memmove(&svc_gl_att_tab_sub[i], + &svc_gl_att_tab_sub[i + 1], + (size_t)n); + svc_gl_att_tab_h[SVC_GL_ATT_MAX - 1] = 0; + svc_gl_att_tab_sub[SVC_GL_ATT_MAX - 1] = 0; + if (svc_gl_att_h == h) + svc_gl_att_h = svc_gl_att_tab_h[0]; + return; + } +} +/* + * CONT.411: THE SECTION BELONGS TO THE ENGINE THE RUN ADDRESSES. + * + * The driver's channel layout fixes one subchannel per engine + * (cla06fsubch.h / NVA06F_SUBCHANNEL_*): + * + * 0 = 3D, 1 = COMPUTE, 2 = I2M, 3 = 2D, 4 = COPY_ENGINE + * + * A SET_OBJECT binds a class object to one of those subchannels and the + * methods that class executes arrive on the SAME subchannel. The section + * recognisers below used to key every check on the subchannel of the + * SET_OBJECT bind, which is wrong for a channel that interleaves engines + * in one push: the observed kick channel (h=0xbfef0002) carries the C597 + * class bind on the 3D subchannel 0 (`op=1 sub=0 addr=0x0 v0=0x0000c597`) + * and its method runs on the 2D subchannel 3 in the same 64 B push, so + * `svc_gl_att_has(h, subch)` rejected the very run that carries the + * window and every draw section stayed open-but-never-captured. + * + * Rule implemented here: attachment stays keyed by (channel, bind + * subchannel) - it is what the SET_OBJECT proved - but a run is accepted + * into the section when its own subchannel is either the bind subchannel + * or the engine subchannel the channel's runs are actually using + * (recorded per channel below). The recorded run subchannel is always a + * valid engine subchannel; nothing outside 0..4 is ever adopted. + */ +#define SVC_SUBCH_3D 0 +#define SVC_SUBCH_COMPUTE 1 +#define SVC_SUBCH_I2M 2 +#define SVC_SUBCH_2D 3 +#define SVC_SUBCH_CE 4 + +static u64 svc_gl_att_run_h[SVC_GL_ATT_MAX]; +static u8 svc_gl_att_run_sub[SVC_GL_ATT_MAX]; + +static void svc_gl_att_run_note(u64 h, u32 subch) +{ + u32 i, free = SVC_GL_ATT_MAX; + + if (!h || subch > SVC_SUBCH_CE) + return; + for (i = 0; i < SVC_GL_ATT_MAX; i++) { + if (svc_gl_att_run_h[i] == h) { + svc_gl_att_run_sub[i] = (u8)subch; + return; + } + if (!svc_gl_att_run_h[i] && free == SVC_GL_ATT_MAX) + free = i; + } + if (free == SVC_GL_ATT_MAX) + free = SVC_GL_ATT_MAX - 1; /* evict oldest */ + if (free) { + memmove(&svc_gl_att_run_h[1], &svc_gl_att_run_h[0], + (size_t)free * sizeof(u64)); + memmove(&svc_gl_att_run_sub[1], &svc_gl_att_run_sub[0], + (size_t)free); + } + svc_gl_att_run_h[0] = h; + svc_gl_att_run_sub[0] = (u8)subch; +} + +static u32 svc_gl_att_run_subch_of(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_run_h[i] == h) + return svc_gl_att_run_sub[i]; + return 0xffu; +} + +static bool svc_gl_att_has_any(u64 h) +{ + u32 i; + + for (i = 0; i < SVC_GL_ATT_MAX; i++) + if (svc_gl_att_tab_h[i] == h) + return true; + return false; +} + +/* + * Accept a run's subchannel for section recognition: the bind's own + * subchannel always, or the engine subchannel the channel's runs use + * (only when the channel is attached at all and the run subchannel is a + * valid engine subchannel). + */ +static bool svc_gl_att_accept_run(u64 h, u32 subch) +{ + if (svc_gl_att_has(h, subch)) + return true; + if (!svc_gl_att_has_any(h) || subch > SVC_SUBCH_CE) + return false; + return subch == svc_gl_att_run_subch_of(h); +} + +/* + * SESSION 153 (G-B gate: the client's own C597 draws must open sections). + * The replay guard at svc_gl_flush consults svc_gr_cand_push_seen(), which + * returns true when the push VA sits in the GR candidate ring - even when + * no GL section was ever captured for that push. The GL client's draw + * pushes are admitted as GR candidates during driver init (they are read + * as ordinary GR pushes first), so every later replay of them is rejected + * as "replay already captured" and no GL section ever opens. Track GL + * section captures in their own table: a replay only skips when THIS table + * holds the exact (push_va, content hash) pair. The hash keeps a later, + * different push at the same VA (a frame loop: same slot, new content) + * capturable, matching the r383 reasoning for the GR seen table. + */ +#define SVC_GL_CAP_HIST 256 +static struct { + u64 push_va; + u64 hash; +} svc_gl_cap_hist[SVC_GL_CAP_HIST]; +static u32 svc_gl_cap_next; +static u32 svc_gl_cap_seen_log; + +static bool svc_gl_cap_seen(u64 push_va, u64 hash) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GL_CAP_HIST; i++) + if (svc_gl_cap_hist[i].push_va == push_va && + svc_gl_cap_hist[i].hash == hash) + return true; + return false; +} + +static void svc_gl_cap_store(u64 push_va, u64 hash) +{ + u32 i = svc_gl_cap_next++ % SVC_GL_CAP_HIST; + + svc_gl_cap_hist[i].push_va = push_va; + svc_gl_cap_hist[i].hash = hash; +} +/* + * SESSION 153 CONT.6 (task 1): the per-frame GL section-arrival ledger. + * svc_gl_flush is the ONLY point where a C597 section's whole push buffer + * (content included) reaches the GL machinery, so every arrival is + * recorded here: (push_va, content hash, arrival count). The question the + * ledger answers deterministically is whether a NEW (different-content) + * section ever arrives after the driver's init - the wall evidence says + * the X server's 1524 B section is byte-identical every capture ("replay + * already captured" by content hash), but that evidence only covered the + * budgeted replay/skip logs. Now: first arrival, every content change + * (with the first words and the bound-site words), and a periodic + * liveness row per push VA. A hash never seen before on ANY push VA is + * NEW-CONTENT - the moment a genuinely new section (a draw stream) + * appears, it is on record with its words. + */ +#define SVC_GL_ARRIVAL_MAX 64 +static struct { + u64 push_va; + u64 hash; + u32 arrivals; +} svc_gl_arrivals[SVC_GL_ARRIVAL_MAX]; +static u32 svc_gl_arrival_n; +static u64 svc_gl_arrival_total; +static u32 svc_gl_arrival_log; /* bounded new/changed/periodic rows */ + +static void svc_gl_arrival_note(u64 push_va, const u8 *buf, u32 len) +{ + u64 ph = svc_push_hash(buf, len); + u32 i, slot = SVC_GL_ARRIVAL_MAX; + bool known_hash = false; + + for (i = 0; i < svc_gl_arrival_n; i++) { + if (svc_gl_arrivals[i].push_va == push_va) { + slot = i; + break; + } + if (svc_gl_arrivals[i].hash == ph) + known_hash = true; + } + svc_gl_arrival_total++; + if (slot < SVC_GL_ARRIVAL_MAX) { + bool changed = svc_gl_arrivals[slot].hash != ph; + + svc_gl_arrivals[slot].arrivals++; + if (!changed && + svc_gl_arrivals[slot].arrivals % 1024 != 0) + return; + if (svc_gl_arrival_log < 512) { + svc_gl_arrival_log++; + pr_info("sema-glass: %s h=0x%x push=0x%llx len=%u hash=0x%llx n=%u total=%llu first=%08x %08x %08x %08x\n", + changed ? "CONTENT-CHANGED" : "arrival", + 0, push_va, len, ph, + svc_gl_arrivals[slot].arrivals, + svc_gl_arrival_total, + len >= 4 ? get_unaligned_le32(buf) : 0, + len >= 8 ? get_unaligned_le32(buf + 4) : 0, + len >= 12 ? get_unaligned_le32(buf + 8) : 0, + len >= 16 ? get_unaligned_le32(buf + 12) : 0); + } + svc_gl_arrivals[slot].hash = ph; + return; + } + if (svc_gl_arrival_n < SVC_GL_ARRIVAL_MAX) { + slot = svc_gl_arrival_n++; + svc_gl_arrivals[slot].push_va = push_va; + svc_gl_arrivals[slot].hash = ph; + svc_gl_arrivals[slot].arrivals = 1; + if (svc_gl_arrival_log < 512) { + svc_gl_arrival_log++; + pr_info("sema-glass: %s h=0x%x push=0x%llx len=%u hash=0x%llx n=1 total=%llu first=%08x %08x %08x %08x\n", + known_hash ? "KNOWN-CONTENT-OTHER-VA" : "NEW-CONTENT", + 0, push_va, len, ph, svc_gl_arrival_total, + len >= 4 ? get_unaligned_le32(buf) : 0, + len >= 8 ? get_unaligned_le32(buf + 4) : 0, + len >= 12 ? get_unaligned_le32(buf + 8) : 0, + len >= 16 ? get_unaligned_le32(buf + 12) : 0); + } + } +} +static u32 svc_gl_rep_log; /* bounded replay-already-captured witness */ +static u32 svc_gl_prog_log; /* R16 bounded program-read witness */ +static u32 svc_gl_up_log; /* R17 bounded section-upload-apply witness */ +/* + * SESSION 149 R17 (M4): explicit program bytes for the synthesized GL + * launch (hex string, <= GSP_BRIDGE_INLINE_BYTES). The GL program page + * (e.g. 0x121201b00) is empty in every buffer this emulation can see: no + * memory-writing command in any readable push targets it and the + * context-init upload never reached a decodable ring. When set, these + * bytes are attached as the program record (after a trusted VA read, so a + * real upload still wins). Development use only: a minimal real sm_75 + * SASS program (EXIT;NOP;...) proves the engine executes OUR bytes. + */ +static char *sema_gl_prog_hex; +module_param_named(sema_gl_prog_hex, sema_gl_prog_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_prog_hex, + "explicit program bytes (hex) for the synthesized GL launch (<=256 B)"); +/* + * SESSION 153-CONTINUED-11: per-stage program images. A Turing graphics + * program is [SPH V4 (128 B)][SASS]; the SPH tells the engine the stage + * (VTG vs PS) and the I/O mapping. The engine validates the SPH of every + * bound pipeline program at draw time, so every program VA the window + * binds needs bytes. sema_gl_vs_hex feeds VTG-typed slots, sema_gl_ps_hex + * pixel-typed slots, and sema_gl_prog_hex is the generic fallback (also + * used for the single-record compute path). All three re-parse whenever + * their string changes, so a running module can be iterated against the + * "Shader Program Header N Error" fault oracle without a reload. + */ +static char *sema_gl_vs_hex; +module_param_named(sema_gl_vs_hex, sema_gl_vs_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_vs_hex, + "explicit VTG program image (SPH+SASS hex, <=256 B) for vertex slots"); +static char *sema_gl_ps_hex; +module_param_named(sema_gl_ps_hex, sema_gl_ps_hex, charp, 0644); +MODULE_PARM_DESC(sema_gl_ps_hex, + "explicit PS program image (SPH+SASS hex, <=256 B) for pixel slots"); +/* + * r1492: deliver the known-good VTG image when the captured upload is + * header-only (page-splice corruption). Default 1; live A/B. + */ +static bool sema_gl_vs_img = true; +module_param_named(sema_gl_vs_img, sema_gl_vs_img, bool, 0644); +MODULE_PARM_DESC(sema_gl_vs_img, + "Deliver the built-in known-good VTG image (SPH+SASS) when the captured VS upload has no SASS body (default 1)"); +/* + * r1500: deliver the known-good pixel image. The active indexed draw + * binds a PIXEL stage (j=5) but no per-window upload of that program was + * ever captured (the 0x2154 pair is absent from the sliced window), so + * the PS slot page stayed empty and the fragment stage had no code. The + * client's real 384 B PS image (SPH V4 000a1482 + SASS) was recovered + * from its init-push page; deliver it at the PS bind VA exactly like the + * VTG image. Default 1; A/B via sema_gl_ps_img=0, overridden by + * sema_gl_ps_hex. + */ +static bool sema_gl_ps_img = true; +module_param_named(sema_gl_ps_img, sema_gl_ps_img, bool, 0644); +MODULE_PARM_DESC(sema_gl_ps_img, + "Deliver the built-in known-good PS image (SPH+SASS) when no captured PS upload/program exists (default 1)"); +static u8 svc_gl_prog[GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_prog_len; +static u8 svc_gl_vs[GSP_BRIDGE_INLINE_BYTES]; +/* + * r1492: known-good VTG program image (SPH V4T1 020a0481 + SASS), the + * client's own fixed-function vertex shader recovered from its init-push + * page. r1491 proved the runtime capture of this 1.4-1.5 KB upload can be + * spliced at a page boundary: the delivered bytes were the SPH header + * plus ZERO SASS, so the VS executed no code and the draw stored nothing. + * The captured image must pass svc_gl_img_code_ok(); when it does not, + * this image is delivered at the bind VA instead. A/B via + * sema_gl_vs_img=0 (captured image only) and overridden by sema_gl_vs_hex. + * The bytes are the client's compiled program (no private provenance). + */ +static const u8 svc_gl_vs_img[1408] = { + + 0x81, 0x04, 0x0a, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0f, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0xf0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x21, 0x73, 0x04, 0xff, 0xff, 0xa0, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x22, 0x0e, 0x00, + 0x02, 0x7a, 0x08, 0x00, 0x00, 0xf0, 0x01, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x02, 0x7a, 0x0c, 0x00, 0x00, 0xa8, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x02, 0x7a, 0x0b, 0x00, 0x00, 0xe0, 0x01, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x02, 0x7a, 0x0d, 0x00, 0x00, 0xa9, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x02, 0x7a, 0x0a, 0x00, 0x00, 0xe1, 0x01, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x76, 0x0c, 0x0c, 0x00, 0xec, 0x01, 0x00, 0x0b, 0x00, 0x01, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x02, 0x7a, 0x0f, 0x00, 0x00, 0xaa, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xc6, 0x0f, 0x00, + 0x23, 0x76, 0x0d, 0x0d, 0x00, 0xed, 0x01, 0x00, 0x0a, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x20, 0x7a, 0x00, 0x04, 0x00, 0x30, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x1f, 0x04, + 0x20, 0x7a, 0x01, 0x04, 0x00, 0x34, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x00, 0x05, 0x00, 0x31, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x20, 0x7a, 0x02, 0x04, 0x00, 0x38, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x01, 0x05, 0x00, 0x35, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x23, 0x7a, 0x00, 0x06, 0x00, 0x32, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x02, 0x05, 0x00, 0x39, 0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x01, 0x06, 0x00, 0x36, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x20, 0x72, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x02, 0x06, 0x00, 0x3a, 0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x72, 0x03, 0x01, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x00, 0xc8, 0x0f, 0x00, + 0x23, 0x72, 0x05, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x00, 0xca, 0x0f, 0x00, + 0x0b, 0x72, 0x00, 0x05, 0xff, 0x00, 0x00, 0x00, 0x00, 0x40, 0xf1, 0x03, 0x00, 0xd8, 0x0f, 0x00, + 0x08, 0x03, 0x05, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x22, 0x0e, 0x00, + 0x0b, 0x7a, 0x00, 0xff, 0x00, 0xf0, 0x01, 0x00, 0x00, 0xd0, 0xf1, 0x03, 0x00, 0xe2, 0x0f, 0x00, + 0x20, 0x72, 0x04, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x1f, 0x08, + 0x20, 0x72, 0x06, 0x01, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x08, + 0x20, 0x7a, 0x01, 0x04, 0x00, 0x14, 0x02, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x20, 0x72, 0x07, 0x02, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x09, 0x78, 0x05, 0x08, 0x00, 0x00, 0x00, 0xc3, 0x00, 0x00, 0x81, 0x07, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x7a, 0x00, 0x06, 0x00, 0x15, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xc6, 0x0f, 0x00, + 0x09, 0x78, 0x08, 0x05, 0x00, 0x00, 0x00, 0x43, 0x00, 0x00, 0x81, 0x03, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x7a, 0x00, 0x07, 0x00, 0x16, 0x02, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x20, 0x7a, 0x05, 0x04, 0x00, 0x08, 0x02, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x02, 0x7a, 0x04, 0x00, 0x00, 0xe2, 0x01, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x09, 0x72, 0x09, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x81, 0x07, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x7a, 0x06, 0x06, 0x00, 0x09, 0x02, 0x00, 0x05, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x21, 0x73, 0x00, 0xff, 0xff, 0x80, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x22, 0x0e, 0x00, + 0x23, 0x76, 0x0f, 0x0f, 0x00, 0xee, 0x01, 0x00, 0x04, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x08, 0x73, 0x09, 0x00, 0x09, 0x00, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x62, 0x0e, 0x00, + 0x23, 0x7a, 0x0a, 0x07, 0x00, 0x0a, 0x02, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0xca, 0x0f, 0x00, + 0x09, 0x72, 0x0a, 0xff, 0x0a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x81, 0x07, 0x00, 0xc8, 0x0f, 0x00, + 0x0b, 0x72, 0x00, 0x0a, 0xff, 0x00, 0x00, 0x00, 0x00, 0x40, 0xf3, 0x03, 0x00, 0xe2, 0x0f, 0x00, + 0x20, 0x72, 0x0b, 0x08, 0x09, 0x00, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x2f, 0x00, + 0x20, 0x7a, 0x04, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xc8, 0x1f, 0x00, + 0x08, 0x73, 0x0b, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x22, 0x0e, 0x00, + 0x20, 0x7a, 0x06, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x20, 0x7a, 0x08, 0x00, 0x00, 0x28, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x20, 0x7a, 0x00, 0x00, 0x00, 0x2c, 0x00, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x05, 0x01, 0x00, 0x21, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x07, 0x01, 0x00, 0x25, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x09, 0x01, 0x00, 0x29, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x23, 0x7a, 0x01, 0x01, 0x00, 0x2d, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x00, 0x02, 0x00, 0x22, 0x00, 0x00, 0x05, 0x00, 0x01, 0x00, 0x00, 0xe2, 0x0f, 0x04, + 0x08, 0x78, 0x0b, 0x0b, 0x00, 0x00, 0x80, 0x3f, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe2, 0x1f, 0x00, + 0x23, 0x7a, 0x0e, 0x02, 0x00, 0x2e, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x01, 0x0a, 0x00, 0x1c, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x08, 0x72, 0x0b, 0x0b, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x7a, 0x04, 0x02, 0x00, 0x26, 0x00, 0x00, 0x07, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x06, 0x02, 0x00, 0x2a, 0x00, 0x00, 0x09, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x23, 0x7a, 0x02, 0x0a, 0x00, 0x1d, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x0a, 0x0a, 0x00, 0x1e, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x07, 0x01, 0x00, 0xe4, 0x01, 0x00, 0x0c, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x01, 0x03, 0x00, 0x27, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x04, 0x0b, 0x00, 0x20, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x05, 0x0b, 0x00, 0x21, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x23, 0x7a, 0x08, 0x02, 0x00, 0xe5, 0x01, 0x00, 0x0d, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x0a, 0x0a, 0x00, 0xe6, 0x01, 0x00, 0x0f, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x0b, 0x0b, 0x00, 0x22, 0x02, 0x00, 0xff, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x02, 0x03, 0x00, 0x2b, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x04, + 0x23, 0x7a, 0x04, 0x04, 0x00, 0xe8, 0x01, 0x00, 0x07, 0x00, 0x01, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x02, 0x7a, 0x07, 0x00, 0x00, 0xe7, 0x01, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x23, 0x7a, 0x00, 0x03, 0x00, 0x23, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xc4, 0x0f, 0x00, + 0x23, 0x7a, 0x0e, 0x03, 0x00, 0x2f, 0x00, 0x00, 0x0e, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x05, 0x05, 0x00, 0xe9, 0x01, 0x00, 0x08, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x23, 0x7a, 0x06, 0x0b, 0x00, 0xea, 0x01, 0x00, 0x0a, 0x00, 0x01, 0x00, 0x00, 0xe4, 0x0f, 0x00, + 0x22, 0x73, 0x00, 0xff, 0x00, 0x70, 0x00, 0x00, 0xff, 0x08, 0x00, 0x00, 0x00, 0xe8, 0x0f, 0x00, + 0x22, 0x73, 0x00, 0xff, 0x0e, 0x7c, 0x00, 0x00, 0xff, 0x00, 0x00, 0x00, 0x00, 0xe8, 0x0f, 0x00, + 0x22, 0x73, 0x00, 0xff, 0x04, 0x80, 0x02, 0x00, 0xff, 0x0c, 0x00, 0x00, 0x00, 0xe2, 0x0f, 0x00, + 0x4d, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x03, 0x00, 0xea, 0x0f, 0x00, + 0x47, 0x79, 0x00, 0x00, 0xf0, 0xff, 0xff, 0xff, 0xff, 0xff, 0x83, 0x03, 0x00, 0xc0, 0x0f, 0x00, +}; + +/* + * r1500: known-good pixel program image (SPH V4 000a1482 + SASS), the + * client's own compiled fragment shader recovered from its init-push + * page. The active indexed draw binds type-5 (PIXEL) at the program + * family base (j=5) but the sliced window carries no 0x2154 pair and no + * PS upload, so that page stayed empty and the fragment stage had no + * code. Delivered at the PS bind VA when no captured PS exists, analogous + * to svc_gl_vs_img for VTG. A/B via sema_gl_ps_img=0, overridden by + * sema_gl_ps_hex. The bytes are the client's compiled program. + */ +static const u8 svc_gl_ps_img[384] = { + + 0x82, 0x14, 0x0a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xaa, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x26, 0x73, 0x04, 0x00, 0xff, 0x00, 0x00, 0x00, 0x1f, 0x00, 0x0e, 0x00, 0x00, 0x02, 0x0e, 0x04, + 0x26, 0x73, 0x00, 0x00, 0xff, 0x00, 0x00, 0x00, 0xa0, 0x80, 0x00, 0x00, 0x00, 0x62, 0x0e, 0x00, + 0x26, 0x73, 0x01, 0x00, 0xff, 0x00, 0x00, 0x00, 0xa1, 0x80, 0x02, 0x00, 0x00, 0xa2, 0x0e, 0x00, + 0x26, 0x73, 0x02, 0x00, 0xff, 0x00, 0x00, 0x00, 0xa2, 0x80, 0x04, 0x00, 0x00, 0xe2, 0x0e, 0x00, + 0x26, 0x73, 0x03, 0x00, 0xff, 0x00, 0x00, 0x00, 0xa3, 0x80, 0x06, 0x00, 0x00, 0x22, 0x0f, 0x00, + 0x08, 0x73, 0x04, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x64, 0x1e, 0x00, + 0x20, 0x82, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xe4, 0x2f, 0x08, + 0x20, 0x92, 0x01, 0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xe4, 0x4f, 0x08, + 0x20, 0xa2, 0x02, 0x02, 0x04, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc4, 0x8f, 0x00, + 0x20, 0xb2, 0x03, 0x03, 0x04, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xe2, 0x0f, 0x01, + 0x4d, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x03, 0x00, 0xee, 0x0f, 0x00, + 0x47, 0x79, 0x00, 0x00, 0xf0, 0xff, 0xff, 0xff, 0xff, 0xff, 0x83, 0x03, 0x00, 0xc0, 0x0f, 0x00, + 0x18, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x0f, 0x00, + 0x18, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x0f, 0x00, + 0x18, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x0f, 0x00, + 0x18, 0x79, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x0f, 0x00, +}; + +/* + * r1481: shader-image cache filled from any push chunk that starts with a + * real SPH magic. The client's init push carries its VS/PS images as raw + * continuation chunks (first word 020a0481 / 000a1482) whose method + * headers sit in a zero first page; the normal upload decoder calls them + * implausible (unreadable=1), so the ring may never hold them. Cache the + * bytes here and let the program-record fallback author them at the bind + * VA (VS cache index 1, PS index 0). + */ +static u8 svc_gl_sph_cache[2][GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_sph_cache_len[2]; +static u32 svc_gl_vs_len; +static u8 svc_gl_ps[GSP_BRIDGE_INLINE_BYTES]; +static u32 svc_gl_ps_len; +static char svc_gl_prog_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; +static char svc_gl_vs_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; +static char svc_gl_ps_seen[GSP_BRIDGE_INLINE_BYTES * 2 + 1]; + +static void svc_gl_hex_parse(const char *h, const char *tag, u8 *dst, + u32 *dst_len, char *seen) +{ + u32 n, i; + + if (!strcmp(seen, h ? h : "")) + return; /* unchanged since the last parse */ + strscpy(seen, h ? h : "", GSP_BRIDGE_INLINE_BYTES * 2 + 1); + *dst_len = 0; + if (!h || !h[0]) + return; + n = (u32)strlen(h); + if (n % 2 || n / 2 > GSP_BRIDGE_INLINE_BYTES) { + pr_warn("sema-gl: %s param ignored (bad length %u)\n", + tag, n); + return; + } + for (i = 0; i < n; i += 2) { + unsigned int b; + + if (sscanf(h + i, "%2x", &b) != 1) { + pr_warn("sema-gl: %s param bad at %u\n", tag, i); + return; + } + dst[i / 2] = (u8)b; + } + *dst_len = n / 2; + pr_info("sema-gl: %s param: %u bytes stored\n", tag, *dst_len); +} + +static void svc_gl_prog_apply_param(void) +{ + svc_gl_hex_parse(sema_gl_prog_hex, "prog-hex", svc_gl_prog, + &svc_gl_prog_len, svc_gl_prog_seen); + svc_gl_hex_parse(sema_gl_vs_hex, "vs-hex", svc_gl_vs, + &svc_gl_vs_len, svc_gl_vs_seen); + svc_gl_hex_parse(sema_gl_ps_hex, "ps-hex", svc_gl_ps, + &svc_gl_ps_len, svc_gl_ps_seen); + if (READ_ONCE(sema_gl_vs_img) && !svc_gl_vs_len) { + memcpy(svc_gl_vs, svc_gl_vs_img, sizeof(svc_gl_vs_img)); + svc_gl_vs_len = (u32)sizeof(svc_gl_vs_img); + } + if (READ_ONCE(sema_gl_ps_img) && !svc_gl_ps_len) { + memcpy(svc_gl_ps, svc_gl_ps_img, sizeof(svc_gl_ps_img)); + svc_gl_ps_len = (u32)sizeof(svc_gl_ps_img); + } +} + +/* + * r1492: is a captured program image complete enough to deliver? + * + * The client's 0x180/0x1b0 upload runs carry a 128 B SPH followed by SASS. + * The push decoder reads a long entry page by page; when the image payload + * straddles a page whose continuation the ring mapping does not describe, + * the captured bytes are the SPH (in the first page) plus ZERO for the + * SASS tail. r1491 proved that shape on the beef VTG page: the shadow + * receipt read `020a0481 ...` at +0x900 and `code4=00000000` at +0x980, + * while the same page's PS and TESS images had real SASS. Delivering such + * an image overwrites the real program page with a header that has no + * code, so the VS executes NOPs and no vertex reaches the raster. + * + * Gate: SPH magic (SPH type 1=VTG or 2=PS in bits 4:0) and at least one + * nonzero word in the code region [0x80, min(len, 0x200)). A header-only + * image is rejected and the caller falls through to the known-good image + * path (sema_gl_vs_hex / ps_hex or the cached image). + */ +static bool svc_gl_img_code_ok(const u8 *d, u32 n) +{ + u32 w0, i; + + if (!d || n < 0x90u) + return false; + w0 = get_unaligned_le32(d); + if ((w0 & 0x1fu) != 1u && (w0 & 0x1fu) != 2u) + return false; + for (i = 0x80u; i + 4u <= n && i < 0x200u; i += 4u) + if (get_unaligned_le32(d + i)) + return true; + return false; +} + +/* + * SESSION 153-CONTINUED-87: the SPH VERSION of the live program images. + * The launch builder emits SET_SPH_VERSION from this value so the + * window's accepted version range always covers the images the engine + * is about to fetch. The client's own init push carries 4|(4<<16), but + * the r950 gate proved the V3 image executes; a 4/4 window over a V3 + * SPH rejects every fetched header (SPH9|SPH11|SPH18, r1273-r1275) - + * the r950-era acceptance ran with no version window at all. Returns + * the max VERSION field of the loaded blobs (0 = none). + */ +int gsp_sema_gl_sph_blob_version(void) +{ + u32 v = 0, ver; + + if (svc_gl_vs_len >= 4) { + ver = (get_unaligned_le32(svc_gl_vs) >> 5) & 0x1fu; + if (ver > v) + v = ver; + } + if (svc_gl_ps_len >= 4) { + ver = (get_unaligned_le32(svc_gl_ps) >> 5) & 0x1fu; + if (ver > v) + v = ver; + } + return (int)v; +} + +/* + * SESSION 153 (GL client draw sections): the GL client's own per-frame + * pushes bind TURING_A (0xC597) exactly like the X server's one-shot + * section, but carry no program/param/release operands at all (measured: + * `0001fff0 20010000 0000c597 0001fff0 20016043 00000000 20016040 + * 00000000` on h=0xbfef0002 - the C597 SET_OBJECT plus 2D-class + * NoOp/Notify only). The old flush guard required the full operand trio + * and silently dropped every client section, so no client capture ever + * appeared. Two additive sources close that gap: + * + * 1. a per-channel operand HOLD: the last valid prog/param/rel operands + * seen in a section on this channel are remembered and adopted by a + * later draw section on the same channel (still marked src=hold); + * 2. a dev fallback (gsp_sema_gl_draw_synth): when the client + * never supplied operands at all, the draw section captures with the + * operand values the X server's one-shot GL section supplied in the + * same boot class (r891/r923), marked src=draw-synth. + * + * r929 (Path A): the dev fallback is now OFF by default. The stream + * captures (real operands, src=stream) are the primary 3D path and replay + * verbatim; a draw-synth capture synthesizes a compute QMD from fake/dev + * operands, and on this boot class its launches never completed + * (qmd_rel=0x600df00d / ch_rel miss -> SYNTH-FAULT + GR-COMPLETE-SW every + * ~19 s per kick, GZDoom wedged). Set sema_gl_draw_synth=1 to A/B the old + * fallback. + */ +#define SVC_GL_HOLD_MAX 8 +struct svc_gl_hold { + u32 h; + u64 prog_va; + u64 param_va; + u32 param_size; + u64 rel_va; + u32 rel_payload; +}; +static struct svc_gl_hold svc_gl_hold_tab[SVC_GL_HOLD_MAX]; +static u32 svc_gl_open_log; /* bounded section-open witness */ +static u32 svc_gl_skip_log; /* bounded flush-skip witness */ +static u32 svc_gl_np_log; /* CONT.45 bounded no-param stream witness */ +static u32 svc_gl_mark_log; /* CONT.45 bounded 3D-mark-failure witness */ +static u32 svc_gl_draw_args_log; /* r1500 bounded draw-args witness */ +/* + * SESSION 153-CONTINUED-18 (task 1): the kick NOTIFY executor. + * + * The GL client's per-frame kick push on the kick-family channel + * (h=0xbfef0002) is [SET_OBJECT(C597)][SET_NOTIFY_A {1, 0x2000e0b0}] + * [NOTIFY 0x10C][NO_OP]: the class2d notification write at VA + * 0x12000e0b0 (the shared .nvidia_drv member, page offset 0xB0; its u16 + * status at +0xE). On real hardware the 2D engine writes a completed + * NvNotification there (nonzero timestamp at +0x0, status 0 at +0xE) - + * that write is the client's per-frame completion edge; the lane used to + * log "flush skip" and drop it, leaving the member all-zero forever. This + * executor authors the NvNotification through the same trusted VA walk + * the release path uses (svc_lookup_trusted + svc_write_ap) and verifies + * by readback. sema_kick_notify=0 disables (A/B); the witness line + * "sema-kick: exec ..." is bounded (first 48 + every 512th). + */ +static bool gsp_sema_kick_notify = true; +module_param_named(sema_kick_notify, gsp_sema_kick_notify, bool, 0644); +MODULE_PARM_DESC(sema_kick_notify, + "Execute the GL client's per-frame kick NOTIFY (author the NvNotification at the SET_NOTIFY target VA; default 1, session 153 CONT.18)"); +static u32 svc_kick_log; /* bounded kick-notify witness */ +static u64 svc_kick_count; /* executor invocations */ +/* + * r1772 (IDLE-DMA-RCA F2, from the r1771 c18 stall): an ADOPTED + * kick-family USERD latch is not yet CONFIRMED (both adoption paths + * seed confirmed=false), so the CONT.407 put=0 keep below did not + * cover it: c18 showed "kick-family ADOPT on PUT step (put 2 -> 3)" + * at [42774.3] then "speculative latch REJECTED (put=0 latch=3)" 7 s + * later, and the kick lane authored nothing for the next 78 s. The + * ADOPT itself required a ring-entry-validated, positive PUT step + * (the same evidence the confirmed path relies on), so an unconfirmed + * latch is kept across a transient put=0 too; the keeps<64 bound + * retains the churn protection. sema_kick_latch_keep=0 restores the + * pre-r1772 condition for A/B. + */ +static bool gsp_sema_kick_latch_keep = true; +module_param_named(sema_kick_latch_keep, gsp_sema_kick_latch_keep, bool, 0644); +MODULE_PARM_DESC(sema_kick_latch_keep, + "Keep an adopted-but-unconfirmed kick-family USERD latch across a transient put=0 (CONT.407 extension; default 1)"); +/* + * r982: the notify page a pending-marker hit proved, per knowledge the + * exec picked up. Kept in the service's own scope (not the channel + * snapshot): the kick channel is the only consumer and the entry is + * re-validated by a read before every use, so a stale PA can only cost + * one failed probe, never a write into freed memory. + */ +static struct { + u64 h; /* channel handle that proved the page */ + u64 pa; /* PA of the marker-proven notify page */ +} svc_kick_pa_cache[4]; + +static void svc_kick_pa_cache_store(u64 h, u64 pa) +{ + u32 i; + + for (i = 0; i < ARRAY_SIZE(svc_kick_pa_cache); i++) { + if (svc_kick_pa_cache[i].h == h) { + svc_kick_pa_cache[i].pa = pa; + return; + } + } + for (i = 0; i < ARRAY_SIZE(svc_kick_pa_cache); i++) { + if (!svc_kick_pa_cache[i].h) { + svc_kick_pa_cache[i].h = h; + svc_kick_pa_cache[i].pa = pa; + return; + } + } + svc_kick_pa_cache[0].h = h; + svc_kick_pa_cache[0].pa = pa; +} + +/* + * CONT.411 (task 3): the release/counter completion slot the GL kick + * channel polls. The kick executor already authors the NvNotification + * (timestamp + status) into the client-visible page it resolves; record + * the slot it proved (handle, VA, PA, aperture, last value) so a later + * NOTIFY that arrives without its own SET_NOTIFY target can reuse the + * proven slot instead of falling back to "no target" - the client's poll + * then always sees a written completion for this channel. + */ +static struct { + u64 h; /* channel handle */ + u64 va; /* completion VA the client polls */ + u64 pa; /* PA the executor proved writable */ + u32 ap; /* aperture of pa */ + u32 val; /* last value authored (low word of the stamp) */ + u64 n; /* registrations */ +} svc_compl_slot[4]; +static u32 svc_compl_log; + +static void svc_compl_slot_store(struct svc_channel *ch, u64 va, u64 pa, + u32 ap, u32 val) +{ + u32 i, free = ARRAY_SIZE(svc_compl_slot); + + if (!ch || !ch->h_object || !va || ap != SVC_APERTURE_SYS) + return; + for (i = 0; i < ARRAY_SIZE(svc_compl_slot); i++) { + if (svc_compl_slot[i].h == ch->h_object) { + free = i; + break; + } + if (!svc_compl_slot[i].h && free == ARRAY_SIZE(svc_compl_slot)) + free = i; + } + if (free == ARRAY_SIZE(svc_compl_slot)) + free = 0; /* evict oldest */ + svc_compl_slot[free].h = ch->h_object; + svc_compl_slot[free].va = va; + svc_compl_slot[free].pa = pa; + svc_compl_slot[free].ap = ap; + svc_compl_slot[free].val = val; + svc_compl_slot[free].n++; + if (gsp_sema_dbg && (svc_compl_log < 64 || + !(svc_compl_slot[free].n & 127))) { + svc_compl_log++; + pr_info("sema-compl: h=0x%x slot registered va=0x%llx pa=0x%llx ap=%u val=%08x n=%llu (client poll target)\n", + ch->h_object, va, pa, ap, val, + (unsigned long long)svc_compl_slot[free].n); + } +} + +static u64 svc_compl_slot_va_of(u64 h) +{ + u32 i; + + for (i = 0; i < ARRAY_SIZE(svc_compl_slot); i++) + if (svc_compl_slot[i].h == h) + return svc_compl_slot[i].va; + return 0; +} + +static bool gsp_sema_gl_draw_synth; +module_param_named(sema_gl_draw_synth, gsp_sema_gl_draw_synth, bool, 0644); +MODULE_PARM_DESC(sema_gl_draw_synth, + "Capture a draw-shaped C597 section with the dev fallback operands when the client supplied none (default 0 since r929; witness marks src=draw-synth)"); + +/* + * Apply a decoded GL release forward-only when the draw section is not + * captured. The GLX client's kickoff suffix releases the shared-surface + * refCount (sharedSurface+0x800+0x10*sd) that the X DDX's nvClipIdleChannels + * polls; if the draw section never reaches a capturable flush the release + * stayed in svc_gl/svc_gl_hold_tab and the DDX sat out its 5 s timeout + * ("Wait for channel idle timed out"). The release is the only edge the + * client needs here (no host-side draw work to order against), so applying + * it when the section is skipped is safe; forward-only + jump-bound guards + * make a duplicate/stale apply harmless. + */ +static bool gsp_sema_gl_hold_apply; +module_param_named(sema_gl_hold_apply, gsp_sema_gl_hold_apply, bool, 0644); +MODULE_PARM_DESC(sema_gl_hold_apply, + "Apply a decoded GL release forward-only when the draw section is not captured (default 0)"); +#define SVC_GL_SYNTH_PROG_VA 0x121201b00ULL +#define SVC_GL_SYNTH_PARAM_VA 0x1214fa220ULL +#define SVC_GL_SYNTH_PARAM_SIZE 0x10u +#define SVC_GL_SYNTH_REL_VA 0x120001000ULL +#define SVC_GL_SYNTH_REL_PAY 1u + +static void svc_gl_hold_store(u32 h, u64 prog, u64 param, u32 psize, + u64 rel, u32 rpay) +{ + u32 i, free = SVC_GL_HOLD_MAX; + + /* + * SESSION 153-CONTINUED-13: rel == 0 is admitted (a section that has + * real operands but flushed before the channel's release was decoded + * stashes them here); a later call with rel != 0 completes the entry + * and a rel == 0 call must NEVER wipe a stored release. + */ + if (!h || !prog || !param) + return; + for (i = 0; i < SVC_GL_HOLD_MAX; i++) { + if (svc_gl_hold_tab[i].h == h) { + free = i; + break; + } + if (!svc_gl_hold_tab[i].h && free == SVC_GL_HOLD_MAX) + free = i; + } + if (free == SVC_GL_HOLD_MAX) + free = h % SVC_GL_HOLD_MAX; + svc_gl_hold_tab[free].h = h; + svc_gl_hold_tab[free].prog_va = prog; + svc_gl_hold_tab[free].param_va = param; + svc_gl_hold_tab[free].param_size = psize; + if (rel) { + svc_gl_hold_tab[free].rel_va = rel; + svc_gl_hold_tab[free].rel_payload = rpay; + } +} + +static bool svc_gl_hold_load(u32 h, u64 *prog, u64 *param, u32 *psize, + u64 *rel, u32 *rpay) +{ + u32 i; + + for (i = 0; i < SVC_GL_HOLD_MAX; i++) { + if (svc_gl_hold_tab[i].h != h || !h) + continue; + if (prog) + *prog = svc_gl_hold_tab[i].prog_va; + if (param) + *param = svc_gl_hold_tab[i].param_va; + if (psize) + *psize = svc_gl_hold_tab[i].param_size; + if (rel) + *rel = svc_gl_hold_tab[i].rel_va; + if (rpay) + *rpay = svc_gl_hold_tab[i].rel_payload; + return true; + } + return false; +} + +/* + * SESSION 149 R17 (M4): standalone GL-class upload tracker. A context-init + * upload push (the one-shot program upload) is NOT required to sit inside a + * C597 section, but it carries the same subchannel-0 0x180/0x1b0 i2m shape. + * The tracker pairs the last 0x180 destination with each following 0x1b0 + * payload and applies it on sight (sema_gl_xlate, eng==0, trusted/allow, + * once-only) so the destination (e.g. the program page) gets its content + * even when the push carries no class bind this pass recognises. + */ +static u64 svc_gl_spot_dst; /* last 0x180 destination (per pass) */ +static u32 svc_gl_spot_bytes; /* its length word (witness) */ +static u32 svc_gl_lid_log; /* bounded standalone-apply witness */ + +static void svc_gl_add_ptr(u64 va) +{ + u32 k; + + if (!va) + return; + /* + * SESSION 153-CONTINUED-48: dedup by 4 KB page, not exact VA. + * The QMD pointer region holds only seven pairs (words 34..47, + * SVC_GR_POINTER_MAX - 1); a window that binds several different + * offsets of one buffer (r1058 chunk1: five index-buffer VAs + * across three pages) used to consume one slot per exact VA and + * hit the cap, so no later operand page (the raw CALL_MME_MACRO + * payload VAs 0x43545f0/0xb94f700) could join the set and the + * shadow relocation left them raw - the raw operand the engine + * faulted on (grfifo=0xbadf1201). One pointer per page carries + * the same page set with slots left for the remaining operand + * families; the first VA seen for a page is kept so the value + * stays a real client VA (the fill resolves pages, and the + * relocation matches by page). + */ + for (k = 0; k < svc_gl.n_ptr; k++) + if (svc_gl.ptr[k] == va || + (svc_gl.ptr[k] & ~(u64)0xfffu) == (va & ~(u64)0xfffu)) + return; + if (svc_gl.n_ptr < SVC_GR_POINTER_MAX - 1) + svc_gl.ptr[svc_gl.n_ptr++] = va; +} + +/* + * Collect one observed VA operand from the open C597 section. Every case + * is transcribed from the decoded r882 stream words; the upper-word masks + * follow the class definitions (ADDRESS_UPPER 7:0 / lower 31:0). + */ +static void svc_gl_field(u32 addr, u32 count, const u8 *data, u32 dlen) +{ + u32 v[4] = { 0, 0, 0, 0 }; + u32 n = count > 4 ? 4 : count, k; + + for (k = 0; k < n; k++) { + if (4 * (k + 1) > dlen) + break; + memcpy(&v[k], data + 4 * k, 4); + } + /* + * SESSION 153-CONTINUED-11: SET_PIPELINE_SHADER(j) = 0x2000+j*0x40, + * [ENABLE(0) | TYPE(7:4)]. Collected per slot so a bound program VA + * can be typed (VTG vs PS) for the program-bytes delivery. + */ + /* + * SESSION 153-CONTINUED-48: NVC597_SET_REPORT_SEMAPHORE_A..D = 0x1b00, + * observed as 4-word runs [A=upper(7:0), B=lower(31:0), C=payload, + * D=flags] at the end of every glcore draw window (payload = the + * client's frame counter, e.g. 0x4f9f6; VA e.g. 0x4181bb0/0xc91e6c0). + * These are the engine's own completion writes; with the VA raw the + * replayed engine writes a client VA the GPC engines cannot reach - + * the same VIRT_WRITE fault class the 0x10c NOTIFY drop exists for + * (r1061: the kept [0,0x2000) window that still faults is exactly + * the one carrying these raw 0x1b00 VAs). Collect the semaphore page + * so the window relocation can rewrite the pair to a shadow VA: the + * write then lands in the SAME HPA and the client's wait sees the + * real hardware completion. + */ + if (addr == 0x1b00) { + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xffu) << 32) | + (u64)(v[1] & 0xfffffffcu)); + return; + } + if (addr >= 0x2000 && addr < 0x2800 && (addr & 0x3f) == 0) { + u32 j = (addr - 0x2000) / 0x40; + + if (j < 32 && count >= 1) + svc_gl_shader_type[j] = (u8)((v[0] >> 4) & 0xf); + return; + } + /* + * SESSION 153-CONTINUED-48: CALL_MME_MACRO(j)=0x3800+j*8 / + * CALL_MME_DATA(j)=0x3804+j*8 runs carry raw 32-bit client VAs as + * macro arguments (r1058 chunk1: the macro(2) run payloads + * 0x44ff800/0x44ff000 relocated only because the page set already + * held 0x44ff000 via the index bind; the sibling payloads + * 0x43545f0/0x4353f60/0xb94f700/0xb941578 stayed raw because their + * pages missed the capped pointer list - the raw operand class the + * GR faulted on). Collect every VA-shaped payload word so the page + * set - and therefore the 3D-window shadow relocation - covers the + * macro arguments too. This must run before the 0x2014..0x3fff + * program-pair recognizer below, whose (addr & 0x3f) == 0x14 shape + * would otherwise swallow a CALL_MME_DATA(2)=0x3814 run. Shape + * gates: 4-byte aligned; >= 64 KB excludes the method/flag scalars + * (0x0c/0x53/0x338); < 512 MB excludes the float constants the + * macro(57) runs carry (1.0f=0x3f800000) while keeping every + * observed client operand VA (max 0xb94f700). + */ + if (addr >= 0x3800 && addr < 0x3a00) { + for (k = 0; k < count; k++) { + u32 w; + + if (4 * (k + 1) > dlen) + break; + memcpy(&w, data + 4 * k, 4); + if (w >= 0x10000u && w < 0x20000000u && + !(w & 3u)) + svc_gl_add_ptr((u64)w); + } + return; + } + /* SET_PIPELINE_PROGRAM_ADDRESS_A/B(j), A=upper(7:0), B=lower(31:0). + * Observed: 0x20020815 (j=1) [00000001 21201b00] -> 0x121201b00. + * SESSION 153-CONTINUED-11: EVERY pair is recorded (the engine + * validates each bound program's SPH at draw time) and the program + * page joins the candidate's page set so the per-VA inline records + * can land there. + */ + if (addr >= 0x2014 && addr < 0x4000 && (addr & 0x3f) == 0x14 && + count >= 2) { + u64 pva = ((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc); + u32 j = (addr - 0x2014) / 0x40, k; + + if (!svc_gl.prog_va) + svc_gl.prog_va = pva; + if (pva && j < 32) { + bool dup = false; + + for (k = 0; k < svc_gl.n_progs; k++) + if (svc_gl.progs[k] == pva) + dup = true; + if (!dup && svc_gl.n_progs < SVC_GL_PROG_MAX) { + svc_gl.progs[svc_gl.n_progs] = pva; + svc_gl.prog_slot[svc_gl.n_progs] = (u8)j; + svc_gl.n_progs++; + } + svc_gl_add_ptr(pva); + } + return; + } + switch (addr) { + case 0x1574: /* SET_TEX_HEADER_POOL_A/B/C: upper, lower, index. + * Observed: [00000001 214fa220 00000010]. + */ + if (count >= 3 && !svc_gl.param_va) { + svc_gl.param_va = ((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc); + svc_gl.param_size = v[2]; /* observed 0x10 */ + } + return; + case 0x155c: /* SET_TEX_SAMPLER_POOL_A/B/C: [1 214fa200 0] */ + if (count >= 3) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x0f84: /* SET_VERTEX_STREAM_SUBSTITUTE_A/B: [1 21200000] */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x0180: /* LINE_LENGTH_IN run: VALUE, LINE_COUNT, + * OFFSET_OUT_UPPER, OFFSET_OUT, PITCH_OUT. + * Observed: [0000000c 1 1 214f8f00 0000000c]. + */ + if (count >= 4) + svc_gl_add_ptr(((u64)(v[2] & 0xff) << 32) | + (u64)(v[3] & 0xfffffffc)); + /* R17: this run names the destination the next 0x1b0 + * LAUNCH_DMA/LOAD_INLINE_DATA run writes to. */ + if (count >= 5) { + svc_gl.up_dst = ((u64)(v[2] & 0xff) << 32) | + (u64)(v[3] & 0xfffffffc); + svc_gl.up_bytes = v[0]; + } + return; + case 0x01b0: /* LAUNCH_DMA + LOAD_INLINE_DATA (op3/op5): the run's + * data words ARE the observed upload payload. Keep the + * FIRST run's bytes for the M2-S1 inline attachment. + * Observed: a004006c -> [00000041 00000670 00000690 + * 000006b0] = LAUNCH_DMA cfg + 3 LID words = 12 B. + * SESSION 149 R16 (M3): keep only the LID payload (the + * cfg word is not code) and bound the copy by the run's + * OWN count - dlen reaches the end of the push, so the + * R15 copy over-read the run (blob=64 = 12 real bytes + + * 52 B of the following method words). These bytes are + * only the fallback: svc_gl_flush reads the real + * program bytes from prog_va via the trusted walk. + */ + if (!svc_gl.blob_len && count > 1 && dlen > 4) { + u32 n = (count - 1) * 4; + + if (n > dlen - 4) + n = dlen - 4; + if (n > sizeof(svc_gl.blob)) + n = (u32)sizeof(svc_gl.blob); + if (n) { + memcpy(svc_gl.blob, data + 4, n); + svc_gl.blob_len = n; + } + } + /* + * R17: record the run's payload at the destination the + * preceding 0x180 run named (the payload starts after the + * LAUNCH_DMA config word - same split as the blob above). + */ + if (svc_gl.up_dst && count > 1 && dlen > 4 && + svc_gl.n_up < SVC_GL_UPLOAD_MAX) { + struct svc_gl_up *u = &svc_gl.up[svc_gl.n_up]; + u32 n = (count - 1) * 4; + + if (n > dlen - 4) + n = dlen - 4; + if (n > sizeof(u->data)) + n = (u32)sizeof(u->data); + if (n) { + u->dst_va = svc_gl.up_dst; + u->len = n; + memcpy(u->data, data + 4, n); + svc_gl.n_up++; + } + } + return; + case 0x17c8: /* SET_INDEX_BUFFER_A/B: A=ADDRESS_UPPER(7:0), + * B=ADDRESS_LOWER(31:0). SESSION 153-CONTINUED-46: + * the indexed draws (glxgears' glDrawElements) fetch + * their indices from this VA; r1039 evidence: the + * client's 3D window binds it + * ("200205f2 ... data=00000000 0450fcc8" -> VA + * 0x0450fcc8) but the candidate page set held only the + * program page (n_pages=1) while X's retiring windows + * carry 3-6 pages - the index page is never mapped in + * the replay channel. Collect it like every other + * address operand. + */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + case 0x1c20: /* SET_VERTEX_STREAM_A_FORMAT/BASE(j) for j >= 2: + * same [fmt, BASE_UPPER, BASE_LOWER] shape as 0x1c00/ + * 0x1c10; the old switch only matched j=0/1. + */ + case 0x1c30: + case 0x1c40: + case 0x1c50: + case 0x1c60: + case 0x1c70: + case 0x1c80: + case 0x1c90: + case 0x1ca0: + case 0x1cb0: + case 0x1cc0: + case 0x1cd0: + case 0x1ce0: + case 0x1cf0: + case 0x1c00: /* SET_VERTEX_STREAM_A_FORMAT(j) + j*0x10: the run + * carries [fmt, BASE_UPPER, BASE_LOWER] - the vertex + * stream the draw reads positions/colours from. + * Observed 0x1c00 [00001000 00000001 2150b790] and + * 0x1c10 [00001008 00000001 215257c0] (SESSION + * 153-CONTINUED-10 draw-push decode). + */ + case 0x1c10: + if (count >= 3) + svc_gl_add_ptr(((u64)(v[1] & 0xff) << 32) | + (u64)(v[2] & 0xfffffffc)); + return; + case 0x2380: /* SET_CONSTANT_BUFFER_SELECTOR_A/B: [size, upper, + * lower]. Observed [00010000 00000001 2150a500]. + */ + if (count >= 3) + svc_gl_add_ptr(((u64)(v[1] & 0xff) << 32) | + (u64)(v[2] & 0xfffffffc)); + return; + case 0x0800: /* SET_COLOR_TARGET_A/B(j=0): [OFFSET_UPPER, + * OFFSET_LOWER, w, h, fmt, ...] - the surface the + * draw rasterizes into. Observed [00000001 222fa200 + * 00000040 00000004 000000f7 00000000]. + */ + if (count >= 2) + svc_gl_add_ptr(((u64)(v[0] & 0xff) << 32) | + (u64)(v[1] & 0xfffffffc)); + return; + default: + return; + } +} + +/* + * SESSION 153-CONTINUED-10: re-walk [start, end) of the push and collect + * the C597 operand fields of every method run on the section's subchannel. + * Used when a draw section opens at its BEGIN marker (0x1618): the state + * methods that precede the marker (vertex stream binds, colour target, + * constant buffer binds) were decoded before the section existed, so their + * VAs must be re-collected or the candidate's page set starves and the + * replay's shadow relocation leaves them pointing at unreachable guest + * VAs (the r929 v2 FAULT_PDE class). Method shapes mirror the main + * decoder: op 1/3/5 carry 1+count dwords, everything else is header-only. + */ +static void svc_gl_scan_window(u32 start, u32 end, const u8 *buf, u32 len, + u32 att_subch) +{ + u32 o = start; + + while (o + 4 <= end && o + 4 <= len) { + u32 hdr, op, count, subch, addr; + + memcpy(&hdr, buf + o, 4); + op = (hdr >> 29) & 0x7; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + if ((op == 1 || op == 3 || op == 5) && count && + o + 4 + count * 4 <= len) { + if (subch == att_subch) + svc_gl_field(addr, count, buf + o + 4, + len - o - 4); + o += 4 + count * 4; + continue; + } + o += 4; + } +} + +/* + * SESSION 153-CONTINUED-18 (task 1): execute the GL client's per-frame + * kick NOTIFY (see the sema_kick_notify comment above). The GPU's write + * is a completed NvNotification: nonzero timestamp at +0x0 (ns since + * epoch), info32 at +0x8 (the NOTIFY payload), info16 at +0xC, status 0 + * at +0xE. Resolution is the release path's trusted walk + * (svc_lookup_trusted -> svc_write_ap), then readback verification. + */ +static bool svc_kick_family(const struct svc_channel *ch); +static u64 svc_push_pa_ceiling(void); +static bool svc_kick_notify_exec(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 va, u32 val, + u64 push_va) +{ + u64 pa = 0, ts = ktime_get_real_ns(); + int ap = SVC_APERTURE_SYS; + u8 old[16], nw[16], rb[16]; + bool ok = false, changed = false; + u32 o0 = 0, n0 = 0; + int rc = -ENOENT; + const char *rsrc = "trusted"; + u32 kc = 0; + + memset(nw, 0, sizeof(nw)); + memcpy(nw, &ts, sizeof(ts)); + memcpy(nw + 8, &val, sizeof(val)); + if (!svc_lookup_trusted(vgpu, va, &pa, &ap)) { + int r1, r2; + + /* + * SESSION 153-CONTINUED-20: prefer the client-visible sysmem + * mapping. The target is the completion word the client's + * flush loop polls through its own memfd; r974/r975 measured + * the serviced-channel VAS walk answering this VA with the FB + * shadow (ap=2, pa=0x20e0b0) - a write no client can observe + * (r973's SYS answer, pa=0x12ae340b0, was the readable one, + * and the ch=NULL dev probe on r975 re-resolved the same VA + * to SYS pa=0x104b640b0 = the kick ring page + 0x1000 + 0xb0, + * exactly the r973 ring-adjacent layout). When the biased + * chain lands in FB, re-resolve with the serviced-channel VAS + * bias cleared and take the sysmem answer when it exists. + */ + if (svc_ap_of(ap) == SVC_APERTURE_FB) { + u64 pa2 = 0; + int ap2 = SVC_APERTURE_FB; + u32 save = svc_cur_vas; + + svc_cur_vas = 0; + if (!svc_lookup_trusted(vgpu, va, &pa2, &ap2) && + svc_ap_of(ap2) == SVC_APERTURE_SYS) { + pa = pa2; + ap = ap2; + rsrc = "trusted-u"; + } + svc_cur_vas = save; + } + /* + * r978: r977's exec fires (27 calls) but authors the word in + * the vidmem shadow (ap=2 pa=0x20e0b0) that no client reads, + * while r973 - the boot where the kick consumption visibly + * advanced the client - wrote the RING-ADJACENT sysmem slot + * (exec pa=0x12ae340b0 over ring pa=0x12ae33000 = ring + + * 0x1000 + 0xb0; the r975 probe measured the identical rule, + * pa=0x104b640b0 over ring 0x104b63000). Both walk answers + * miss that page this boot, so before falling back to the + * shadow: (a) the general lookup (the discovered rm_mem map - + * r979's sema_probe proved it resolves this VA to SYS after + * the map rebuild, pa=0x1184a40b0), (b) a driver-declared pin + * for this VA (the guest pin stream is the client's own + * statement of the mapping), then (c) for a kick-family + * channel the ring-relative completion slot, computed from + * the live ring pin each call. + */ + if (svc_ap_of(ap) != SVC_APERTURE_SYS) { + u64 mpa = 0; + int map_ap = SVC_APERTURE_SYS; + + if (!svc_lookup(vgpu, va, &mpa, &map_ap) && + svc_ap_of(map_ap) == SVC_APERTURE_SYS) { + pa = mpa; + ap = map_ap; + rsrc = "map"; + } + } + if (svc_ap_of(ap) != SVC_APERTURE_SYS) { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(va, &ipa, &iap) && + svc_ap_of(iap) == SVC_APERTURE_SYS) { + pa = ipa; + ap = iap; + rsrc = "ipin"; + } + } + if (svc_ap_of(ap) != SVC_APERTURE_SYS && ch && + svc_kick_family(ch)) { + u64 ring_pa = 0; + int ring_ap = SVC_APERTURE_SYS; + + if (!svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, + &ring_ap) && + svc_ap_of(ring_ap) == SVC_APERTURE_SYS) { + u64 cand = (ring_pa & + ~(u64)(SEMA_PAGE_SIZE - 1)) + + SEMA_PAGE_SIZE + + (va & (SEMA_PAGE_SIZE - 1)); + + if (cand < svc_push_pa_ceiling()) { + pa = cand; + ap = SVC_APERTURE_SYS; + rsrc = "ring+10b0"; + } + } + } + /* + * r981: the client's pending notification marker. r979/r980 + * measured the REAL slot's +0xC u32 reading 0x80000000 + * while the client waited (and 0 once completed; the write + * below clears it) - on the page the discovered map names, + * ring + 0x20b0 on both boots, one page ABOVE the r973-era + * ring + 0x10b0 heuristic the exec had been writing + * (0x1184a40b0 vs 0x1184a30b0 on r979; 0x12e07a0b0 vs + * 0x12e0790b0 on r980 - the completion landed on the wrong + * page both boots). When the chosen candidate does not show + * the marker, probe the ring-relative alternatives and use + * the one that does; the readback below still verifies the + * write. A slot already completed (marker 0) keeps the + * resolved candidate unchanged. + * + * r982: remember the page a pending-marker hit named, per + * handle. r981's boot: exec n=1 found the marker page + * (pa=0x129c1b0b0, ring+0x20b0) and completed it, but the + * marker is 0 afterwards so every later exec fell back to + * ring+0x10b0 (0x129c1a0b0 - wrong page, 33 further writes) + * while the map/trusted stages were still catching up. The + * cached page is retried at the top of the next call (read + * first, used only while the read succeeds), so all later + * completions stay on the page the marker proved. + */ + if (ch && svc_kick_family(ch)) { + /* Retry the page a marker hit proved, if still readable. */ + for (kc = 0; kc < 4; kc++) { + u32 probe_buf = 0; + + if (svc_kick_pa_cache[kc].h != ch->h_object || + !svc_kick_pa_cache[kc].pa) + continue; + if (!svc_read_ap(vgpu, svc_kick_pa_cache[kc].pa, + SVC_APERTURE_SYS, &probe_buf, + sizeof(probe_buf)) && + probe_buf != 0xffffffffu) { + pa = svc_kick_pa_cache[kc].pa; + ap = SVC_APERTURE_SYS; + rsrc = "kickcache"; + } + break; + } + } + if (ch && svc_kick_family(ch)) { + u32 pend = 0; + + if (!svc_read_ap(vgpu, pa + 0xc, svc_ap_of(ap), + &pend, sizeof(pend)) && + pend != 0x80000000u) { + u64 ring_pa = 0; + int ring_ap = SVC_APERTURE_SYS; + + if (!svc_pin_lookup_any(ch->gpfifo_va, + &ring_pa, &ring_ap) && + svc_ap_of(ring_ap) == SVC_APERTURE_SYS) { + u64 base = ring_pa & + ~(u64)(SEMA_PAGE_SIZE - 1); + u32 off = (u32)(va & (SEMA_PAGE_SIZE - 1)); + u64 cands[2]; + int i; + + cands[0] = base + + 2 * SEMA_PAGE_SIZE + off; + cands[1] = base + + SEMA_PAGE_SIZE + off; + for (i = 0; i < 2; i++) { + u32 m = 0; + + if (cands[i] >= + svc_push_pa_ceiling()) + continue; + if (!svc_read_ap(vgpu, + cands[i] + 0xc, + SVC_APERTURE_SYS, + &m, sizeof(m)) && + m == 0x80000000u) { + pa = cands[i]; + ap = SVC_APERTURE_SYS; + rsrc = i ? + "ring+10b0p" : + "ring+20b0p"; + svc_kick_pa_cache_store( + ch->h_object, + cands[i]); + break; + } + } + } + } else if (pend == 0x80000000u && + svc_ap_of(ap) == SVC_APERTURE_SYS) { + /* + * The chosen candidate IS the marker page: + * prove it for the cache so later execs keep + * writing the same page (r981: after the + * single marker hit the fallback wrote two + * wrong ring-relative pages for 33 further + * execs). Only a SYS answer that showed the + * marker is cached; a zero read (already + * completed) caches nothing - unproven. + */ + svc_kick_pa_cache_store(ch->h_object, pa); + } + } + /* + * CONT.412: the channel's registered notifier page is the + * page the guest's submit poll actually watches (the same + * page svc_notify2d_write zeroes). Prefer it over the + * trusted-walk answer, which can resolve the notification + * VA to a different sysmem page (measured: + * notifier_pa+0xb0=0x1288780b0 vs trusted pa=0x124d610b0) + * and leave the poll word unwritten forever. + */ + if (ch && svc_kick_family(ch) && ch->notifier_pa) { + pa = ch->notifier_pa + + (va & (SEMA_PAGE_SIZE - 1)); + ap = ch->notifier_as; + rsrc = "chan-notif"; + } + r1 = svc_read_ap(vgpu, pa, svc_ap_of(ap), old, sizeof(old)); + r1 = svc_read_ap(vgpu, pa, svc_ap_of(ap), old, sizeof(old)); + r2 = svc_write_ap(vgpu, pa, svc_ap_of(ap), nw, sizeof(nw)); + if (!r1 && !r2 && + !svc_read_ap(vgpu, pa, svc_ap_of(ap), rb, sizeof(rb))) { + rc = 0; + ok = true; + changed = memcmp(old, rb, sizeof(old)) != 0; + memcpy(&o0, old, 4); + memcpy(&n0, rb, 4); + } else { + /* CONT.18 probe: name the failing step. */ + rc = r1 ? (r1 < 0 ? r1 : -100) : + (r2 ? (r2 < 0 ? r2 : -101) : -102); + } + } + svc_kick_count++; + /* + * CONT.411 (task 3): register the proven completion slot for this + * channel. Only a successful SYS write is registered - that is the + * page the client's own poll can observe (the FB shadow write was + * the r977 dead end). The slot is reused by the flush's short-kick + * fallback when a later NOTIFY arrives with no target of its own. + */ + if (ok && ch) + svc_compl_slot_store(ch, va, pa, ap, n0); + if (gsp_sema_dbg && (svc_kick_log < 48 || !(svc_kick_count & 127))) { + svc_kick_log++; + pr_info("sema-kick: exec h=0x%x push=0x%llx va=0x%llx pa=0x%llx ap=%d src=%s val=0x%x ts=0x%llx old=%08x new=%08x changed=%d n=%llu rc=%d\n", + ch ? ch->h_object : 0, push_va, va, pa, ap, rsrc, val, + (unsigned long long)ts, o0, n0, changed ? 1 : 0, + (unsigned long long)svc_kick_count, rc); + } + /* + * SESSION 153-CONTINUED-19: the completed NOTIFY is the submitting + * client's completion edge - post the channel event so its OS event + * fires (r934's lesson on the notify2d path: without the event the + * GL client's kick submit wait never completes and the app retries + * on a ~16-22 s timer; r972 measured exactly that cadence with the + * event missing here). Only for a real channel (the dev probe + * passes ch=NULL). + */ + if (ok && ch) + gsp_rm_post_channel_event(vgpu, ch->h_object); + return ok; +} + +/* + * SESSION 153-CONTINUED-18 (task 1, dev probe): run the kick-notify + * executor on an explicit VA against the service's own device. The + * execution MUST run in a kernel-thread context: vfio_iommu_type1's + * dma_rw only copies when current->mm == dma->mm or the caller is a + * kthread (kthread_use_mm); a sysfs write from a shell process has + * current->mm != QEMU's mm and gets -EFAULT (measured rc=-14). The + * workqueue item below is a kthread - the same context class as the + * sema service thread that runs the real kick lane. + * echo 0x12000e0b0 > .../parameters/sema_kick_probe + */ +static u64 svc_kick_probe_va; +static void svc_kick_probe_work_fn(struct work_struct *w); +static DECLARE_WORK(svc_kick_probe_work, svc_kick_probe_work_fn); + +static void svc_kick_probe_work_fn(struct work_struct *w) +{ + struct nv_vgpu_device *vgpu = NULL; + unsigned long flags; + int i; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu) { + vgpu = svc.chan[i].vgpu; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (vgpu) + svc_kick_notify_exec(vgpu, NULL, svc_kick_probe_va, 0, 0); +} + +static int nv_vgpu_sema_kick_probe_set(const char *val, + const struct kernel_param *kp) +{ + if (!val || kstrtoull(val, 0, &svc_kick_probe_va)) + return -EINVAL; + if (svc_kick_probe_va) + schedule_work(&svc_kick_probe_work); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_kick_probe_ops = { + .set = nv_vgpu_sema_kick_probe_set, +}; + +module_param_cb(sema_kick_probe, &nv_vgpu_sema_kick_probe_ops, NULL, 0200); +MODULE_PARM_DESC(sema_kick_probe, "Run the kick-notify executor on a VA (dev probe; session 153 CONT.18)"); + +/* + * SESSION 153-CONTINUED-44 (r1026): host-driven present blit. + * + * The client's rendered frame provably exists in guest RAM (its readback + * staging) while the X server scans the emulated screen surface in vidmem + * (base 0x200000, pitch 5120 for the 1280x720 4-bpp screen), so the window + * shows only the DDX's cleared fill artifact. This is a bounded, explicit + * present emulation: the host names the guest-PA staging of the CLIENT'S + * OWN GPU-rendered frame, and the kernel blits those rows into the scanned + * surface - never synthesizing pixels. The host drives it with: + * + * echo <src_gpa>,<src_pitch>,<w>,<h>[,<dst_off>,<dst_pitch>] \ + * > .../parameters/sema_screen_blit + * + * (all values 0x-hex or decimal; src_pitch and w/h in bytes/pixels, bpp=4; + * defaults dst_off=0x200000, dst_pitch=5120). Bounded: w,h <= 4096, + * pitch <= 65536, total <= 64 MiB, one row buffer. Runs on a workqueue + * (vfio_dma_rw for the guest read, bars[1] shadow for the vidmem write), + * logs one line with the byte/row counts. + */ +#define SVC_SCREEN_BLIT_DEF_OFF 0x200000ULL +#define SVC_SCREEN_BLIT_DEF_PITCH 5120u +static u64 svc_screen_blit_gpa; +static u32 svc_screen_blit_spitch, svc_screen_blit_w, svc_screen_blit_h; +static u64 svc_screen_blit_doff; +static u32 svc_screen_blit_dpitch; +static void svc_screen_blit_work_fn(struct work_struct *w); +static DECLARE_WORK(svc_screen_blit_work, svc_screen_blit_work_fn); + +static void svc_screen_blit_work_fn(struct work_struct *w) +{ + struct nv_vgpu_device *vgpu = NULL; + unsigned long flags; + u8 *row, *base; + size_t size; + u32 y, w4 = svc_screen_blit_w * 4; + u64 bytes = 0; + int i, rc = 0; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu) { + vgpu = svc.chan[i].vgpu; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!vgpu) { + pr_info("sema-screen-blit: no vgpu (no channel registered yet)\n"); + return; + } + base = vgpu->bars[1].vaddr; + size = svc_fb_shadow_size(vgpu); + if (!base || svc_screen_blit_doff + (u64)svc_screen_blit_dpitch * + svc_screen_blit_h > size) { + pr_info("sema-screen-blit: bounds (off=0x%llx pitch=%u h=%u size=%zu)\n", + svc_screen_blit_doff, svc_screen_blit_dpitch, + svc_screen_blit_h, size); + return; + } + row = kmalloc(svc_screen_blit_spitch, GFP_KERNEL); + if (!row) { + pr_info("sema-screen-blit: no memory for row (pitch=%u)\n", + svc_screen_blit_spitch); + return; + } + for (y = 0; y < svc_screen_blit_h; y++) { + rc = svc_read(vgpu, svc_screen_blit_gpa + (u64)y * + svc_screen_blit_spitch, row, + svc_screen_blit_spitch); + if (rc) + break; + memcpy(base + svc_screen_blit_doff + (u64)y * + svc_screen_blit_dpitch, row, w4); + bytes += w4; + } + kfree(row); + pr_info("sema-screen-blit: src=0x%llx spitch=%u w=%u h=%u -> off=0x%llx dpitch=%u rows_done=%u bytes=%llu rc=%d\n", + svc_screen_blit_gpa, svc_screen_blit_spitch, + svc_screen_blit_w, svc_screen_blit_h, + svc_screen_blit_doff, svc_screen_blit_dpitch, y, bytes, rc); +} + +static int nv_vgpu_sema_screen_blit_set(const char *val, + const struct kernel_param *kp) +{ + char buf[128]; + int nf = 0; + u64 v[6] = {0}; + + if (!val || strlen(val) >= sizeof(buf)) + return -EINVAL; + strscpy(buf, val, sizeof(buf)); + { + char *p = buf; + + while (nf < 6 && p && *p) { + char *c = strchr(p, ','); + + if (c) + *c++ = '\0'; + if (kstrtoull(p, 0, &v[nf])) + return -EINVAL; + nf++; + p = c; + } + } + if (nf < 4) + return -EINVAL; + svc_screen_blit_gpa = v[0]; + svc_screen_blit_spitch = (u32)v[1]; + svc_screen_blit_w = (u32)v[2]; + svc_screen_blit_h = (u32)v[3]; + svc_screen_blit_doff = nf > 4 ? v[4] : SVC_SCREEN_BLIT_DEF_OFF; + svc_screen_blit_dpitch = nf > 5 ? (u32)v[5] : + SVC_SCREEN_BLIT_DEF_PITCH; + if (!svc_screen_blit_gpa || !svc_screen_blit_w || + !svc_screen_blit_h) + return -EINVAL; + if (svc_screen_blit_w > 4096 || svc_screen_blit_h > 4096 || + svc_screen_blit_spitch > 0x10000 || + svc_screen_blit_spitch < svc_screen_blit_w * 4 || + svc_screen_blit_dpitch > 0x10000 || + svc_screen_blit_dpitch < svc_screen_blit_w * 4 || + (u64)svc_screen_blit_spitch * svc_screen_blit_h > (64u << 20)) + return -EINVAL; + schedule_work(&svc_screen_blit_work); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_screen_blit_ops = { + .set = nv_vgpu_sema_screen_blit_set, +}; + +module_param_cb(sema_screen_blit, &nv_vgpu_sema_screen_blit_ops, NULL, 0200); +MODULE_PARM_DESC(sema_screen_blit, + "Blit a guest-RAM frame into the scanned vidmem surface: src_gpa,src_pitch,w,h[,dst_off,dst_pitch] (session 153 CONT.44, dev present emulation)"); + + +/* + * Close the open section: synthesize the QMD from the collected fields and + * capture it as the forwarding candidate. The QMD word layout mirrors + * svc_gr_qmd_decode() (the CUDA bridge contract): 48/49 program, 32/33 + * parameter + size, 23..25 release, 34..47 pointer pairs, qmd[4] b10 + * REL_ENABLE. Every VA is copied from a decoded operand; qmd_dev_va stays 0 + * (synthesized - no QMD buffer exists in guest memory) so the backend uses + * its qmdwin/scratch path. + */ + +/* + * SESSION 153-CONTINUED-45 (task 1): big-window verbatim replay. + * + * The direct GL client (glcore, channel family 0xbeef) renders through + * 5016-7144 B pushes whose C597 section window is 1253-1470 dwords - far + * beyond the backend's 448-dword method-block window (any window slice goes + * verbatim through the module pushbuffer page). The engine executes 3D + * methods from ANY number of sequential ring entries with channel state + * persisting across them (that is how real drivers split work), and the + * forward pump already forwards ONE candidate per pass FIFO per channel + * (svc_gr_request picks THIS channel's oldest unconsumed candidate). So the + * window is split into method-aligned chunks of <=448 dwords, each enqueued + * as its own candidate in ring order; successive passes replay them in + * order and the physical channel accumulates the section's state, exactly + * as if the one big window had executed. + * + * Boundary rule: a chunk ends at the last METHOD boundary that fits the + * budget - methods are never split across chunks (a run's data words would + * otherwise be orphaned). The QMD/rel/inline records are cloned from the + * section's base candidate; only stream_off/stream_n differ, so every + * chunk launch carries the same release identity and program images. + */ +#define SVC_GL_STREAM_CHUNK_MAX 6 +/* + * Words per chunk: 384 is the largest window the physical engine is + * PROVEN to retire (r997: `3D stream marked h=0x1000013 + * push=0x120010004 window=384 words` -> `GUEST GR LAUNCH OK`), + * while the client's 394-word first window schedule-stalled its + * replay (r1030: grfifo frozen at 1, fecs idle, timeout + software + * release). Stay under the proven bound. + * SESSION 153-CONTINUED-45: runtime-settable for the bisect - the + * /sys/module/vgsp/parameters/sema_gl_chunk_words knob overrides it so a + * live client's window can be cut to 64/128/256 words to find the first + * length (or starting method) that schedules and retires. + */ +static unsigned svc_gl_chunk_words_param; +module_param_named(sema_gl_chunk_words, svc_gl_chunk_words_param, uint, 0644); +MODULE_PARM_DESC(sema_gl_chunk_words, + "3D stream chunk size in words for the chunked verbatim replay (0 = built-in 384)"); +#define SVC_GL_STREAM_CHUNK_WORDS 384u + +static u32 svc_gl_chunk_boundary(const u8 *buf, u32 len, u32 start, + u32 at_off, u32 max_words) +{ + u32 off = start; + u32 limit = start + max_words * 4u; + u32 last = start; + + while (off + 4 <= at_off && off + 4 <= len) { + u32 hdr, op, cnt, nxt; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + if (op == 4 || op == 7 || !cnt) + nxt = off + 4; + else if ((op == 1 || op == 3 || op == 5) && cnt <= 0xfff && + off + 4 + cnt * 4u <= len && + off + 4 + cnt * 4u <= at_off) + nxt = off + 4 + cnt * 4u; + else if ((op == 1 || op == 3 || op == 5) && cnt <= 0xfff && + off + 4 + cnt * 4u > at_off) + /* + * SESSION 153-CONTINUED-45: a run whose data the + * window edge would cut MUST NOT be a boundary - + * ending a chunk on its header alone makes the + * PBDMA fault mid-method (r1031 PB-FAULT-DUMP P0 + * hdr=20164010 cnt=22, its 22 data words absent) + * and the whole replay dies silently. Stop before + * it; the orphaned head is left out of the replay. + */ + break; + else + nxt = off + 4; /* resync word, as the decoders */ + if (nxt > limit) + break; + last = nxt; + off = nxt; + } + if (last == start) { + /* One method alone overruns the budget: take what fits so the + * window makes progress instead of wedging. */ + last = start + max_words * 4u; + if (last > at_off) + last = at_off; + if (last <= start) + last = start + 4u <= at_off ? start + 4u : at_off; + } + return last; +} + +static void svc_gl_stream_chunks(struct svc_channel *ch, const u8 *buf, + u32 len, u32 at_off) +{ + u32 cur = svc_gl.start_off; + u32 n = 0, pending = 0, ri; + + if (!svc_gr_cand.valid) + return; + /* + * Backpressure: one frame's window is self-contained (it re-emits the + * whole state), so while the previous frame's chunks are still being + * replayed there is no reason to queue another window - it would only + * evict in-flight chunks and execute a torn state sequence. Count + * this channel's unconsumed candidates (including the base just + * captured); at >= 8 (a full frame's 4 chunks plus slack) the new + * capture is dropped whole: the base candidate is invalidated so the + * pass-end request cannot forward it down the compute path either. + */ + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].h_object == ch->h_object) + pending++; + if (pending >= SVC_GR_CAND_RING) { + if (gsp_sema_dbg && svc_gl_mark_log < 64) { + svc_gl_mark_log++; + pr_info("sema-gl: stream chunks backpressure h=0x%x push=0x%llx pending=%u -> capture dropped\n", + ch->h_object, svc_gr_cand.push_va, pending); + } + svc_gr_cand.valid = false; + return; + } + while (cur + 8u <= at_off && n < SVC_GL_STREAM_CHUNK_MAX) { + u32 cw = svc_gl_chunk_words_param ? + svc_gl_chunk_words_param : SVC_GL_STREAM_CHUNK_WORDS; + u32 end; + + if (cw > 448u) + cw = 448u; + end = svc_gl_chunk_boundary(buf, len, cur, at_off, cw); + struct svc_gr_cand_rec *c = &svc_gr_cand; + + if (n) { + u32 ri, found = SVC_GR_CAND_RING; + + for (ri = 1; ri < SVC_GR_CAND_RING; ri++) { + u32 rp = (svc_gr_cand_pos + ri) % + SVC_GR_CAND_RING; + + if (!svc_gr_cand_ring[rp].valid || + svc_gr_cand_ring[rp].offered) { + found = rp; + break; + } + } + if (found == SVC_GR_CAND_RING) { + pr_info("sema-gl: stream chunks: no free ring slot after %u chunks h=0x%x push=0x%llx (window truncated)\n", + n, ch->h_object, svc_gr_cand.push_va); + break; + } + svc_gr_cand_ring[found] = svc_gr_cand; + c = &svc_gr_cand_ring[found]; + } + c->valid = true; + c->offered = false; + c->stream_3d = true; + c->stream_off = cur / 4u; + c->stream_n = (end - cur) / 4u; + /* + * Stream-relative offset for the backend's dropped-chunk + * detection. cur walks from svc_gl.start_off, so the first + * chunk of the section is 0 and each later chunk is exactly + * the number of words already sliced off. + */ + c->stream_rel = (cur - svc_gl.start_off) / 4u; + n++; + cur = end; + } + if (n) { + pr_info("sema-gl: 3D stream chunked h=0x%x push=0x%llx window=%u words -> %u chunk(s) (verbatim replay in ring order; method-aligned)\n", + ch->h_object, svc_gr_cand.push_va, + (at_off - svc_gl.start_off) / 4u, n); + } + /* + * SESSION 153-CONTINUED-46: the first chunk's full method stream. + * Every beef chunk launch stalls the physical front-end (grfifo + * frozen, fecs_cur=0, PBDMA stopped mid-block at a + * SET_VERTEX_ATTRIBUTE_A run while X windows retire) and the window + * tail (`3DSTREAM ... tail=`) alone cannot name the stalling method. + * Dump the first chunk's words (up to 96) as operand lines so the + * exact method list is on record; bounded per boot. + */ + { + static u32 chunkdump_log; + + if (chunkdump_log < 8 && svc_gl.start_off + 8u <= at_off) { + u32 cur2 = svc_gl.start_off, end2, k2, wcnt; + u32 cw2 = svc_gl_chunk_words_param ? + svc_gl_chunk_words_param : + SVC_GL_STREAM_CHUNK_WORDS; + + chunkdump_log++; + if (cw2 > 448u) + cw2 = 448u; + end2 = svc_gl_chunk_boundary(buf, len, cur2, at_off, + cw2); + wcnt = (end2 - cur2) / 4u; + if (wcnt > 96u) + wcnt = 96u; + pr_info("sema-gl: CHUNK1 h=0x%x push=0x%llx off=0x%x words=%u (dump up to 96)\n", + ch->h_object, svc_gr_cand.push_va, cur2, + (end2 - cur2) / 4u); + for (k2 = 0; k2 + 4 <= wcnt; k2 += 4) + pr_info("sema-gl: CHUNK1[%02u] %08x %08x %08x %08x\n", + k2, + get_unaligned_le32(buf + cur2 + 4u * k2), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 1)), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 2)), + get_unaligned_le32(buf + cur2 + 4u * (k2 + 3))); + for (; k2 < wcnt; k2++) + pr_info("sema-gl: CHUNK1[%02u] %08x\n", k2, + get_unaligned_le32(buf + cur2 + 4u * k2)); + } + } +} + +/* + * r1512: draw-coverage receipt (defined after the fixed-CB copy, which owns + * the persisted real-CB page this receipt reports). Forward-declared here + * because svc_gl_flush is its only caller. + */ +static void svc_gl_cover_log(struct nv_vgpu_device *vgpu, const u8 *buf, + u32 len, u32 start_off, u32 at_off, + u32 h_object, u64 push_va, u32 topo, u32 isz, + u32 first, u32 count, u64 iva, u32 stride, + u32 voff, const u32 *cand, u32 ncand); + +/* + * CONT.411 (task 2): the GL kick channel's flush witness. + * + * The 0xbfef0002 section arrives as a 32/64 B push and its flush was + * invisible: the kick-exec branch exits before the "flush skip" witness, + * so the log could not say whether the flush ran, was rejected by the + * trusted/seen guard, or exited at the operand checks. Witness every exit + * for this channel with the full operand set so the next cycle can name + * the exact branch without a rebuild. Bounded (128 lines) because this + * channel flushes on every pass. + */ +#define SVC_GL_WATCH_H 0xbfef0002ULL +static u32 svc_gl_watch_log; + +static void svc_gl_watch_note(const struct svc_channel *ch, const char *why, + u64 push_va, u32 len, u32 at_off, bool trusted) +{ + if (!gsp_sema_dbg || !ch || ch->h_object != SVC_GL_WATCH_H) + return; + if (svc_gl_watch_log >= 128) + return; + svc_gl_watch_log++; + pr_info("sema-gl-w: %s h=0x%x push=0x%llx len=%u seen=%d trusted=%d prog=0x%llx param=0x%llx rel=0x%llx start=0x%x at=0x%x n_ptr=%u blob=%u subch=%u\n", + why, ch->h_object, push_va, len, svc_gl.seen ? 1 : 0, + trusted ? 1 : 0, svc_gl.prog_va, svc_gl.param_va, + svc_gl.rel_va, svc_gl.start_off, at_off, svc_gl.n_ptr, + svc_gl.blob_len, svc_gl.subch); +} + +static void svc_gl_flush(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, + const struct svc_push_state *st, const u8 *buf, + u32 len, u64 push_va, bool trusted, bool replay, + u32 at_off) +{ + struct svc_push_state s2; + /* + * SESSION 153-CONTINUED-88: static (was stack). With the inline + * record grown to 2048 B the 8-record array is ~16 KB - too much for + * a kernel stack; this function runs on the single service thread + * (same reason as the static recs[] in svc_gr_forward and the ops + * array in svc_decode_stream), so a static array is safe. + */ + static struct gsp_bridge_inline prs[SVC_GL_PROG_MAX]; + u32 n_recs = 0, k; + /* + * SESSION 149 R16 (M3): the inline attachment's source. Priority: + * the REAL program bytes read from prog_va through the trusted walk + * (the GL program page), else the observed LID payload bounded to its + * own run. r888's blob=64 over-read the 12 B LID run. + */ + u8 blob[SVC_GR_INLINE_BYTES]; + u32 blob_n = 0; + int blob_rc = -1; + bool blob_trusted = false; + const char *blob_src = "none"; + const char *op_src = "stream"; + u64 kick_va; + + /* + * SESSION 153 CONT.6 (task 1): record every section arrival (any + * trusted or replay state, before any early exit) so a content + * change after init cannot hide behind the bounded capture logs. + */ + if (gsp_sema_dbg) + svc_gl_arrival_note(push_va, buf, len); + /* CONT.411: state at every flush entry for the kick channel. */ + svc_gl_watch_note(ch, "entry", push_va, len, at_off, trusted); + svc_gl_prog_apply_param(); + if (!svc_gl.seen || !trusted) { + svc_gl_watch_note(ch, !svc_gl.seen ? "early-noseen" : + "early-untrusted", push_va, len, at_off, + trusted); + goto out; + } + { + u64 hp = 0, hm = 0, hr = 0; + u32 hms = 0, hrp = 0; + bool held = false; + + if (!svc_gl.prog_va || !svc_gl.param_va || !svc_gl.rel_va) { + held = svc_gl_hold_load(ch->h_object, &hp, &hm, &hms, + &hr, &hrp); + if (!svc_gl.prog_va && held) { + svc_gl.prog_va = hp; + op_src = "hold"; + } + if (!svc_gl.param_va && held) { + svc_gl.param_va = hm; + svc_gl.param_size = hms; + op_src = "hold"; + } + if (!svc_gl.rel_va && held) { + svc_gl.rel_va = hr; + svc_gl.rel_payload = hrp; + op_src = "hold"; + } + if ((!svc_gl.prog_va || !svc_gl.param_va) && + gsp_sema_gl_draw_synth && !ch->engine_type) { + if (!svc_gl.prog_va) + svc_gl.prog_va = SVC_GL_SYNTH_PROG_VA; + if (!svc_gl.param_va) { + svc_gl.param_va = SVC_GL_SYNTH_PARAM_VA; + svc_gl.param_size = + SVC_GL_SYNTH_PARAM_SIZE; + } + op_src = "draw-synth"; + } + if (!svc_gl.rel_va && gsp_sema_gl_draw_synth && + !ch->engine_type) { + svc_gl.rel_va = SVC_GL_SYNTH_REL_VA; + svc_gl.rel_payload = SVC_GL_SYNTH_REL_PAY; + op_src = "draw-synth"; + } + if (!svc_gl.prog_va || !svc_gl.param_va || + !svc_gl.rel_va) { + /* + * SESSION 153-CONTINUED-45 (task 1): the + * DIRECT GL client's (glcore) per-frame draw + * sections bind a program (SET_PIPELINE_ + * PROGRAM_ADDRESS) and carry the bracketing + * release, but no tex-header pool - the + * sticky pool was bound in an earlier push. + * The verbatim 3D replay path never reads + * param_va (the synthesized compute fields + * are what stream_3d bypasses), so a + * prog+rel section whose window fits the + * replay method block is capturable as a + * real stream window with param=0. Bounded + * to the GL stream lane and the fit so the + * historical skip still guards everything + * else; CONT-44 evidence: every client draw + * flush-skipped here with prog/rel real + * (`flush skip h=0xbeef0100 ... prog=0x4390900 + * param=0x0 rel=0x519c0a0`). + */ + bool stream_noparam = false; + + /* + * SESSION 153-CONTINUED-45: the push page is + * cyclically zeroed/rewritten by the client + * (watched live: real -> ZERO -> new content + * on a ~6 s cadence); a capture landing in the + * zero window decodes garbage (prog found + * deeper, window head all zeros) and its + * replay makes the PBDMA fault on the zero + * lead (r1031 PB-FAULT-DUMP P0). Only capture + * when the window head is a real word - a + * later pass/entry on the filled page captures + * the frame. + * + * SESSION 153-CONTINUED-67: the w2/r1127 + * zero-head refusals were NOT mid-write + * windows that later filled: the same word at + * the same offset read zero across every pass + * of the boot (64 refusals, census content + * stable at 0x1f8..0x614), because the window + * start had been armed on a zero word (a + * no-op INC(0) slot) in the gap after the last + * foreign-subchannel run. A zero word is not a + * method: advance past the leading zero words + * to the first real word and capture the + * window from there. Only a window that is + * ENTIRELY zero is still refused (the r1031 + * protection). + */ + if (at_off > svc_gl.start_off && + svc_gl.start_off + 4u <= len) { + u32 w0h; + + memcpy(&w0h, buf + svc_gl.start_off, 4); + if (!w0h) { + u32 adv = svc_gl.start_off; + u32 lim = min_t(u32, at_off, + len); + + while (adv + 4u <= lim) { + memcpy(&w0h, + buf + adv, 4); + if (w0h) + break; + adv += 4u; + } + if (!w0h) { + if (gsp_sema_dbg && + svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: zero-head window h=0x%x push=0x%llx start=0x%x win=%u words (whole window zero; not captured)\n", + ch->h_object, + push_va, + svc_gl.start_off, + (at_off - svc_gl.start_off) / 4u); + } + goto out; + } + if (svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: zero-lead window h=0x%x push=0x%llx start=0x%x->0x%x win=%u words (skipped zero no-ops; capturing)\n", + ch->h_object, + push_va, + svc_gl.start_off, + adv, + (at_off - adv) / 4u); + } + svc_gl.start_off = adv; + } + } + /* + * SESSION 153-CONTINUED-46: the admission + * bound uses the BUILT-IN 384-word chunk + * budget, never the sema_gl_chunk_words + * bisect knob. r1033 evidence: with the knob + * at 64 the bound shrank to 6*64=384 words + * and the client's real 1253-word windows + * were rejected outright ("flush skip ... + * win=1253 words (no operands, no hold)") - + * the knob is meant to cut the replayed + * chunks smaller, not to stop the capture. + * With the full bound every real window is + * admitted and chunked (the knob then + * truncates the replayed words, logged). + */ + if (svc_gl.prog_va && svc_gl.rel_va && + !svc_gl.param_va && + gsp_sema_gl_xlate && !ch->engine_type && + op_src[0] != 'd' && + at_off > svc_gl.start_off && + (at_off - svc_gl.start_off) / 4u <= + SVC_GL_STREAM_CHUNK_MAX * + SVC_GL_STREAM_CHUNK_WORDS) + stream_noparam = true; + if (!stream_noparam) { + /* + * SESSION 153-CONTINUED-13: the release may be + * decoded AFTER this section in ring order + * (the r955 init: the 24 B release pushes sit + * behind the 1524 B init push in the ring, so + * every replay pass of the section flushed + * before svc_gl.rel_va was known). When the + * operands are real but only the release is + * missing, stash them for the channel so the + * next section/pass (by then the constant + * release stream has been decoded) captures + * with src=hold instead of losing the init + * capture - without which the guest's GLX + * init completion degrades and the client + * never emits its draw pushes at all. + */ + if (svc_gl.prog_va && svc_gl.param_va && + !svc_gl.rel_va) + svc_gl_hold_store(ch->h_object, + svc_gl.prog_va, + svc_gl.param_va, + svc_gl.param_size, + 0, 0); + /* + * SESSION 153-CONTINUED-18 (task 1): replace + * the old "flush skip" drop with NOTIFY + * EXECUTION when this push carried the kick + * member (SET_NOTIFY_A/B + NOTIFY in the same + * push). The section has no draw operands - + * it is the client's completion kick - so the + * GPU's notification write is the only work + * it asks for. + */ + /* + * r1652: the short kick form carries only the class2d + * NOTIFY (0x10c), relying on the notification address + * already programmed in the channel context. Use the + * per-channel target when this section did not re-arm + * it; the carried value comes from the previous + * incarnation of the same handle (see + * gsp_sema_service_channel_add). Without the carried + * target the executor never runs and the GL client's + * kick wait spins. + */ + kick_va = svc_gl.kick_notify_va; + if (!kick_va && svc_kick_family(ch)) + kick_va = ch->notify2d_va; + /* CONT.411 (task 3): a NOTIFY with no per-push + * target falls back to the completion slot this + * channel proved earlier, so the poll target is + * never left unwritten just because SET_NOTIFY + * was carried by an earlier push. */ + if (!kick_va && svc_gl.kick_notify_seen) + kick_va = svc_compl_slot_va_of( + ch->h_object); + /* CONT.411 (task 3): name the completion slot + * this section resolves to before the exec so + * the witness distinguishes "no slot" from + * "slot write failed". */ + svc_gl_watch_note(ch, kick_va ? + "kick-try" : "kick-noslot", + push_va, len, at_off, + trusted); + if (gsp_sema_kick_notify && + svc_gl.kick_notify_seen && kick_va && + svc_kick_notify_exec(vgpu, ch, kick_va, + svc_gl.kick_notify_val, + push_va)) { + /* authored once; a replay pass must not + * re-author an unchanged member. */ + svc_gl.kick_notify_seen = false; + svc_gl.kick_notify_va = 0; + svc_gl_watch_note(ch, "kick-done", + push_va, len, at_off, + trusted); + goto out; + } + if (gsp_sema_dbg && svc_gl_skip_log < 2048) { + svc_gl_skip_log++; + pr_info("sema-gl: flush skip h=0x%x push=0x%llx prog=0x%llx param=0x%llx rel=0x%llx win=%u words (no operands, no hold)\n", + ch->h_object, push_va, + svc_gl.prog_va, svc_gl.param_va, + svc_gl.rel_va, + at_off > svc_gl.start_off ? + (at_off - svc_gl.start_off) / 4u : 0u); + } + /* + * CONT.412: an empty/no-op submit must still + * complete. If the 2D NOTIFY path did not + * author the channel's kick completion (no + * NOTIFY seen in this push, or the target was + * dropped earlier), author it from the proven + * completion slot so the client's submit wait + * cannot spin forever on a submit that carried + * no draw operands. + */ + if (trusted && svc_kick_family(ch)) { + u64 cva = svc_compl_slot_va_of( + ch->h_object); + + if (!cva) + cva = ch->notify2d_va; + if (cva && svc_kick_notify_exec( + vgpu, ch, cva, 0, + push_va)) + svc_gl_watch_note(ch, + "empty-submit-complete", + push_va, len, at_off, + trusted); + else if (ch->notifier_pa) { + u8 nw[16]; + u64 ts = ktime_get_real_ns(); + + memset(nw, 0, sizeof(nw)); + memcpy(nw, &ts, sizeof(ts)); + if (!svc_write_ap(vgpu, + ch->notifier_pa + + 0x0b0, + ch->notifier_as, + nw, sizeof(nw))) + svc_gl_watch_note(ch, + "empty-submit-complete", + push_va, len, + at_off, + trusted); + } + } + svc_gl_watch_note(ch, "skip", push_va, len, + at_off, trusted); + goto out; + } else if (gsp_sema_dbg && svc_gl_np_log < 64) { + svc_gl_np_log++; + pr_info("sema-gl: stream capture no-param h=0x%x push=0x%llx prog=0x%llx rel=0x%llx win=%u words (real draw operands; pool param bound elsewhere)\n", + ch->h_object, push_va, + svc_gl.prog_va, svc_gl.rel_va, + (at_off - svc_gl.start_off) / 4u); + } + } + } + } + /* + * SESSION 149 R15 (M1.1a): admit a rewind replay when the same slot + * re-read valid bytes whose content hash matches the recorded pass + * (the pump sets cap_trust only for such a hash-verified replay; see + * the trust admission at the decode call site). The exactly-once guard + * for GL capture is GL-specific (SESSION 153): svc_gr_cand_push_seen() + * also matches the GR candidate ring and the session seen table, and + * the GL client's C597 draw pushes are admitted as GR candidates + * during driver init - so using it here suppressed every later GL + * section capture on those push VAs ("replay already captured" for + * every client draw). svc_gl_cap_seen() only matches pushes whose GL + * section was actually captured, with the content hash so a later + * different push in the same slot (a frame loop) stays capturable. + */ + if (replay && svc_gl_cap_seen(push_va, svc_push_hash(buf, len))) { + if (svc_gl_rep_log < 16) { + svc_gl_rep_log++; + pr_info("sema-gl: replay already captured h=0x%x push=0x%llx (no re-capture)\n", + ch->h_object, push_va); + } + goto out; + } + s2 = *st; + memset(s2.qmd, 0, sizeof(s2.qmd)); + s2.qmd[48] = (u32)svc_gl.prog_va; + s2.qmd[49] = ((u32)(svc_gl.prog_va >> 32)) & 0x1ffff; + s2.qmd[32] = (u32)svc_gl.param_va; + s2.qmd[33] = (((u32)(svc_gl.param_va >> 32)) & 0x1ffff) | + ((svc_gl.param_size >> 4) << 19); + s2.qmd[23] = (u32)svc_gl.rel_va; + s2.qmd[24] = ((u32)(svc_gl.rel_va >> 32)) & 0xff; + s2.qmd[25] = svc_gl.rel_payload; + s2.qmd[4] |= 1u << 10; /* REL_ENABLE */ + /* + * qmd[24] b31 (REL_ONE_WORD) stays clear on the first launch: the + * backend then redirects release0 to the module semaphore and the + * sema service authors the guest word (set it once rel_hpa resolving + * is wanted). qmd[24] b26 (REDUCTION) likewise stays clear. + */ + /* + * SESSION 149 R16 (M3): the scheduling fields the r888 engine + * refused to schedule without. Word/field offsets from the public + * QMDV02_03 class header (nvidia/inc/class/clc5c0qmd.h; the booter's + * own 48/49 program pair matches its QMDV02_03 offsets, so the + * rest of the layout is the same class): + * w12 CTA_RASTER_WIDTH MW(415:384) + * w13 CTA_RASTER_HEIGHT MW(431:416) + * w14 CTA_RASTER_DEPTH MW(463:448) + * w18 QMD_VERSION MW(579:576), QMD_MAJOR_VERSION MW(583:580), + * CTA_THREAD_DIMENSION0 MW(607:592) -> w18 hi word + * w19 CTA_THREAD_DIMENSION1 MW(623:608), + * CTA_THREAD_DIMENSION2 MW(639:624) + * w20 REGISTER_COUNT_V MW(656:648) <- SKEDCHECK06 field + * w29 BARRIER_COUNT MW(959:955) + * w31 SASS_VERSION MW(1023:1016) + * SKEDCHECK16 fired on CTA_THREAD_DIMENSION0==0 and SKEDCHECK06 on + * REGISTER_COUNT_V==0. The register count is the stream's own value: + * REGISTER_COUNT(1) = 0x1a = 26 (method 0x200c + 1*0x40 = 0x204c, + * words [0x1a, 0]) - the same number the driver's C5C0 path writes + * ("a NULL QMD will result in SKEDCHECK06_REGISTER_COUNT", cui + * turing_launch.c:781). CTA dims: the 0xC597 stream carries no grid / + * CTA operands we can locate (the 3D-class runs are format/viewport/ + * shader config; the C597 SHADER_CONFIG "CTA raster" words are not in + * this push), so a minimal 1x1x1 grid of 1x1x1 CTAs is used and + * documented as the choice: the checks only reject ZERO dims and a + * zero total-thread count, both of which a 1x1x1 launch satisfies. + * SHARED_MEMORY_SIZE (w17) stays 0 = "no shared memory" (valid and + * NOT among the failed checks); the m72 local-memory prelude already + * covers SKEDCHECK05. The driver's cache-control block (w5 top bits) + * is set as every captured guest QMD carries it; the backend turns it + * into its icache-invalidate bit for the shadow line. + */ + s2.qmd[12] = 1; /* CTA raster width (CTAs) */ + s2.qmd[13] = 1; /* CTA raster height (CTAs) */ + s2.qmd[14] = 1; /* CTA raster depth (CTAs) */ + s2.qmd[18] = 0x23u | (1u << 16); /* QMD 2.3, CTA dim0 = 1 */ + s2.qmd[19] = 1u | (1u << 16); /* CTA dim1 = 1, dim2 = 1 */ + s2.qmd[20] = 0x1au << 8; /* REGISTER_COUNT_V = 26 */ + s2.qmd[29] |= 1u << 27; /* BARRIER_COUNT = 1 */ + s2.qmd[31] |= 0x75u << 24; /* SASS_VERSION 7.5 (sm_75) */ + s2.qmd[5] |= 0x3c000000u; /* driver cache-control block */ + for (k = 0; k < svc_gl.n_ptr && k < SVC_GR_POINTER_MAX - 1; k++) { + s2.qmd[34 + 2 * k] = (u32)svc_gl.ptr[k]; + s2.qmd[35 + 2 * k] = ((u32)(svc_gl.ptr[k] >> 32)) & 0x1ffff; + } + s2.qmd_dev_va = 0; + s2.qmd_hdr = 0; + s2.qmd_end = 0; + s2.qmd_have = 0; + /* + * SESSION 149 R15 (M2-S1) + R16 (M3): attach program-page content as + * an inline record at the synthesized program VA. The GL program page + * (0x121201b00) is NOT among the section's own upload destinations, + * so without this the backend's w48 code-source screen sees an empty + * page, withdraws to the guest VA and the launch dies at FAULT_PDE; + * with it the proven win0 shadow copy + w48/49 rewrite line runs. + * + * R16 source order: the REAL program bytes read from prog_va through + * the trusted walk (the read machinery translates GL VAs now; src=4 + * answered this channel's ring/pushes in r888), else the observed + * LID payload bounded to its own run (12/24/28 B - "materialize + * exactly that"). A trusted read that lands on an all-zero page is + * treated as "not real" and falls through to the LID bytes. + */ + /* + * SESSION 149 R17 (M4): apply the section's memory-writing upload + * commands BEFORE reading the program page, so an upload that + * targets the program VA is visible to the read below (the section's + * engine commands never ran: every destination - constants, tex + * pools, program - stayed zero). The shared inline-write path + * resolves each destination (FB aperture -> emulated vidmem shadow, + * sysmem -> guest RAM, unresolved -> VA shadow) and the once-only + * guard is keyed by (push, content hash, dst, bytes) so a + * replay/rescue pass cannot double-apply. + */ + { + u64 ph = svc_push_hash(buf, len); + u32 ap_n = 0, ap_b = 0; + + for (k = 0; k < svc_gl.n_up && k < SVC_GL_UPLOAD_MAX; k++) { + struct svc_gl_up *u = &svc_gl.up[k]; + + if (svc_inline_once_seen(push_va, ph, u->dst_va, + u->len)) + continue; + svc_inline_once_store(push_va, ph, u->dst_va, u->len); + if (!svc_grce_inline_write(vgpu, u->dst_va, u->data, + u->len, ch->h_object)) { + ap_n++; + ap_b += u->len; + } + } + if (gsp_sema_dbg && svc_gl_up_log < 8) { + svc_gl_up_log++; + pr_info("sema-gl: uploads applied=%u/%u bytes=%u h=0x%x push=0x%llx\n", + ap_n, svc_gl.n_up, ap_b, ch->h_object, + push_va); + } + } + memset(blob, 0, sizeof(blob)); + /* + * SESSION 153-CONTINUED-10: the explicit operator program override + * (sema_gl_prog_hex) wins over the trusted page read. The GLX + * server's per-frame CONTENT pushes bind their own program slots + * (observed 0x121201f00 / 0x121215280) whose pages hold unrelated + * nonzero bytes at that offset; the R16 trusted-read priority + * accepts any nonzero 16 bytes as "the real program" and attached + * garbage, and the engine rejected the draw with "Graphics + * Exception: Shader Program Header 11/18 Error" (r944 fault + * evidence, push 0x120014578). An unset override keeps the + * historical read priority unchanged. + */ + if (svc_gl_prog_len) { + blob_n = svc_gl_prog_len; + memcpy(blob, svc_gl_prog, blob_n); + blob_src = "prog-hex"; + } else { + blob_rc = svc_read_va_trusted(vgpu, svc_gl.prog_va, blob, + sizeof(blob)); + if (!blob_rc) { + u32 z = 0; + + for (k = 0; k < 16; k++) + z |= blob[k]; + if (z) + blob_trusted = true; + } + if (blob_trusted) { + blob_n = sizeof(blob); + blob_src = "trusted-prog"; + } else if (svc_gl.blob_len) { + blob_n = svc_gl.blob_len; + memcpy(blob, svc_gl.blob, blob_n); + blob_src = "lid-fallback"; + } + } + /* + * r929 (Path A): a stream-derived capture is a 3D method window - the + * backend replays the raw push verbatim and the engine runs the real + * uploads/binds, so the synthesized compute attachments (the inline + * program record, and with it the prog-hex fallback) do not apply. + * Draw-synth/hold captures keep the old compute path unchanged. + * + * SESSION 153-CONTINUED-11: exception - the GL program-bytes + * delivery. The engine validates the SPH of EVERY pipeline program + * the channel state binds when a draw issues, and the driver's + * upload of those bytes never ran (the program pages read empty). + * For a window that binds program slots (the 0x2014/0x2018 pairs + * collected above) attach one inline record per program VA with the + * stage-typed image (vs-hex for VTG slots, ps-hex for pixel slots, + * prog-hex fallback) - this authors the missing upload. Windows + * without program pairs keep the historical single-record path. + */ + if (svc_gl.n_progs) { + u32 pi; + + for (pi = 0; pi < svc_gl.n_progs && n_recs < GSP_BRIDGE_INLINE_MAX; + pi++) { + const u8 *b = NULL; + u32 bl = 0; + u8 t = svc_gl_shader_type[svc_gl.prog_slot[pi] < 32 ? + svc_gl.prog_slot[pi] : 0]; + /* + * SESSION 153-CONTINUED-88: deliver the REAL captured + * upload payload to the engine-visible page. The + * window's own program uploads (the client's 0x180 + * LINE_LENGTH_IN + 0x1b0 LOAD_INLINE_DATA runs - + * observed 1536 B at 0x4350900 and 1408 B at + * 0x4350f00, w0=0x020a0481 = the driver's real SPH + * V4/VTG image) were captured at decode time into the + * destination ring but landed only in the BAR1 shadow; + * the engine validates the SPH of every bound program + * at the FILL-resolved page. Attach the real payload + * as an inline record at its own destination VA - the + * backend authors it into the engine-visible page on + * the stream3d path - and let it win over the minimal + * authored image below. Exact-VA match: the bind VA + * IS the upload destination. + */ + { + u32 rlen = 0; + u32 ri0 = n_recs; + + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + /* + * r1476: when this window's own bind-VA + * upload was never captured (the client's + * per-window 0x1b0 pair can be lost), fall + * back to the newest captured VTG SPH image + * (020a0481) and deliver it at the bind VA. + * The init push's real VS image is in the + * ring from the standalone 0x180/0x1b0 arm; + * without this a window whose VTG page is + * empty executes an empty VS (r1477: + * 0x41c0900 all zero, no pixels). + */ + { + bool got = svc_grce_dst_newest( + svc_gl.progs[pi], 128u, &rlen, + prs[n_recs].data, + (u32)sizeof(prs[n_recs].data)); + + /* + * r1492: a captured image whose + * SASS body is zero is the + * page-splice corruption, not a + * valid program; reject it so the + * authored image below (the real + * VS/PS bytes) can land instead. + */ + if (got && !svc_gl_img_code_ok( + prs[n_recs].data, + rlen)) { + if (gsp_sema_dbg && + svc_gl_prog_log < 64) { + svc_gl_prog_log++; + pr_info("sema-gl: real-upload REJECTED va=0x%llx len=%u first=%08x code4=%08x (SPH only; captured SASS body zero)\n", + svc_gl.progs[pi], + rlen, + get_unaligned_le32( + prs[n_recs].data), + rlen > 0x80u ? + get_unaligned_le32( + prs[n_recs].data + + 0x80u) : + 0u); + } + got = false; + rlen = 0; + } + if (!got && t != 5u && t != 0xffu) { + got = svc_grce_dst_sph( + 0x020a0481u, 128u, + &rlen, + prs[n_recs].data, + (u32)sizeof(prs[n_recs].data)); + if (got && !svc_gl_img_code_ok( + prs[n_recs].data, + rlen)) + got = false; + if (!got && + svc_gl_sph_cache_len[1]) { + rlen = min_t(u32, + svc_gl_sph_cache_len[1], + (u32)sizeof(prs[n_recs].data)); + memcpy(prs[n_recs].data, + svc_gl_sph_cache[1], + rlen); + got = svc_gl_img_code_ok( + prs[n_recs].data, + rlen); + } + } + if (got) { + u64 d = svc_gl.progs[pi]; + u32 head = 4096u - (u32)(d & 0xfff); + + if (rlen <= head) { + prs[n_recs].dst_va = d; + prs[n_recs].len = rlen; + n_recs++; + } else { + /* + * The payload crosses its + * 4K page (observed: the + * 1408 B image at + * 0x4350f00). The backend + * applies a record only + * within one page, so + * split: page remainder + + * tail at the next page's + * VA (skipped by the + * backend when that page + * is not in the request - + * the SPH head still + * lands). + */ + prs[n_recs].dst_va = d; + prs[n_recs].len = head; + n_recs++; + if (n_recs < + GSP_BRIDGE_INLINE_MAX) { + memset(&prs[n_recs], 0, + sizeof(prs[n_recs])); + prs[n_recs].dst_va = + (d & ~(u64)0xfff) + + 0x1000; + prs[n_recs].len = + rlen - head; + memcpy(prs[n_recs].data, + prs[n_recs - 1].data + head, + rlen - head); + n_recs++; + } + /* + * SESSION 153-CONTINUED-88: + * the tail record is applied + * only when its page is in the + * launch's page set. Add the + * tail page as a candidate + * operand page (the engine + * will fetch the program's + * second half from it), so + * the request carries it and + * the split delivery is + * complete instead of + * head-only. + */ + svc_gl_add_ptr((d & ~(u64)0xfff) + + 0x1000); + } + if (gsp_sema_dbg && + svc_gl_prog_log < 64) { + svc_gl_prog_log++; + pr_info("sema-gl: real-upload record va=0x%llx len=%u first=%08x (engine-visible delivery)\n", + d, rlen, + get_unaligned_le32(prs[ri0].data)); + } + continue; + } + } + } + /* + * SESSION 153-CONTINUED-87: the REAL program upload + * wins. When the window carries its own upload of a + * program-sized payload (>= 128 B) to this same 4K + * page, the minimal authored image must NOT overwrite + * the driver's bytes at the engine-visible page (the + * upload now lands there through the pin-aware + * svc_grce_inline_write). Keep an explicit image only + * where no real upload exists (the synthesized stage + * binds below). + */ + { + u32 uu; + bool real_up = false; + + for (uu = 0; uu < svc_gl.n_up; uu++) { + if (svc_gl.up[uu].len < 128u) + continue; + if ((svc_gl.up[uu].dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1)) != + (svc_gl.progs[pi] & + ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + /* + * r1492: same page, but a + * header-only payload is the + * page-splice corruption - do not + * let it suppress the authored + * real image. + */ + if (!svc_gl_img_code_ok( + svc_gl.up[uu].data, + svc_gl.up[uu].len)) { + if (gsp_sema_dbg && + svc_gl_prog_log < 64) { + svc_gl_prog_log++; + pr_info("sema-gl: real-upload page va=0x%llx len=%u code4=%08x rejected (authored image used)\n", + svc_gl.up[uu].dst_va, + svc_gl.up[uu].len, + svc_gl.up[uu].len > + 0x80u ? + get_unaligned_le32( + svc_gl.up[uu].data + + 0x80u) : + 0u); + } + continue; + } + real_up = true; + break; + } + if (real_up) { + if (gsp_sema_dbg && + svc_gl_prog_log < 64) + pr_info("sema-gl: real-upload wins va=0x%llx (authored image skipped)\n", + svc_gl.progs[pi]); + continue; + } + } + + if (t == 5 && svc_gl_ps_len) { + b = svc_gl_ps; + bl = svc_gl_ps_len; + } else if (t != 5 && t != 0xff && svc_gl_vs_len) { + b = svc_gl_vs; + bl = svc_gl_vs_len; + } else if (svc_gl_prog_len) { + b = svc_gl_prog; + bl = svc_gl_prog_len; + } else if (blob_n) { + b = blob; + bl = blob_n; + } + if (!b || !bl) + continue; + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + prs[n_recs].dst_va = svc_gl.progs[pi]; + prs[n_recs].len = bl; + memcpy(prs[n_recs].data, b, bl); + n_recs++; + } + /* + * SESSION 153-CONTINUED-87: author the program images for the + * client's missing stage binds. The replayed window binds only + * the j=1 (VTG) pair; the booter emits the client's own + * TESS_CONTROL (j=2, family+0x200) and PIXEL (j=5, family+0x0) + * binds ahead of the window (sema_gl_sph_bind), so their SPH + * bytes must exist at those VAs too - otherwise the DS fetches + * an empty/foreign page and still latches its header error. + * Observed family layout (guest-RAM scrape of the init push + * and the window itself): 0x4400000 j=5, 0x4400200 j=2, + * 0x4400300/0x4400900 j=1 - all in the window prog VA's own + * 4K page. The TESS image is the VTG image with the SPH + * SHADER_TYPE field (bits 13:10) re-pointed to 2, exactly + * like the driver's own placeholder bind. Beef client + * channels only, and only when the window does not already + * bind that exact VA. + */ + if ((gsp_phys_rmcall_gr_sph_bind_value() || + gsp_phys_rmcall_gr_sph_gs_img_value()) && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + svc_gl.progs[0]) { + u64 fam = svc_gl.progs[0] & + ~(u64)(SEMA_PAGE_SIZE - 1); + u32 k; + + /* + * CONT 25: author exactly the slots the booter emits. + * The booter emits j5 (PS) on mask 0x20 and j2 (TCS) + * on 0x04 when the all-slots bool is off; gate each + * image on the same condition instead of the bool + * alone (the r1575 zero-PS re-capture). + */ + for (k = 0; k < 2u && n_recs < SVC_GL_PROG_MAX; k++) { + u32 slot_bit = k ? 0x04u : 0x20u; + u64 dst = fam + (k ? 0x200u : 0x0u); + const u8 *b = k ? svc_gl_vs : svc_gl_ps; + u32 bl = k ? svc_gl_vs_len : svc_gl_ps_len; + bool dup = false; + u32 pi3; + + if (!(gsp_phys_rmcall_gr_sph_bind_value() || + (gsp_phys_rmcall_gr_sph_bind_mask_value() & + slot_bit))) + continue; + if (!b || bl < 4u) + continue; + for (pi3 = 0; pi3 < svc_gl.n_progs; pi3++) + if (svc_gl.progs[pi3] == dst) + dup = true; + if (dup) + continue; + memset(&prs[n_recs], 0, sizeof(prs[n_recs])); + prs[n_recs].dst_va = dst; + prs[n_recs].len = bl; + memcpy(prs[n_recs].data, b, bl); + if (k) { + u32 w0 = get_unaligned_le32( + prs[n_recs].data); + + w0 = (w0 & ~(0xfu << 10)) | + (2u << 10); /* TESS_CONTROL */ + put_unaligned_le32(w0, prs[n_recs].data); + } + n_recs++; + if (gsp_sema_dbg && svc_gl_prog_log < 64) { + pr_info("sema-gl: stage-image dst=0x%llx slot=%u len=%u first=%08x\n", + dst, k ? 2u : 5u, bl, + get_unaligned_le32(prs[n_recs - 1].data)); + } + } + /* + * SESSION 155 CONT 11: the synthesized header-only + * GEOMETRY SPH at family+0x780. The client never + * emits a geometry program, but the DS validates + * the GS stage anyway (SPH18 = GS check2 attribute + * map) even when the stage is disabled. The booter + * binds this page to stage 4 with ENABLE=0, so the + * engine does not execute it. SPH V4T1 header with + * SHADER_TYPE=GEOMETRY (4), SASS_VERSION=5 and + * pass-through maps (IMAP = the VTG OMAP = + * POSITION+COLOR; OMAP = the PS IMAP superset + * POSITION+COLOR). 128 bytes, same 4K family page + * as the bound VTG/TCS/PS images. + */ + if (gsp_phys_rmcall_gr_sph_gs_img_value() && + n_recs < SVC_GL_PROG_MAX) { + u32 gs_words[32] = { + 0x000a1081u, 0x00000000u, + 0x01000000u, 0x07000000u, + 0x00000003u, 0xf0000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x0000000fu, 0x00000000u, + 0x00000000u, 0x0000f000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x000f0000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + 0x00000000u, 0x00000000u, + }; + u32 gs_maps = + gsp_phys_rmcall_gr_sph_gs_maps_value(); + u64 dst = fam + + (gsp_phys_rmcall_gr_sph_gs_addr_value() ? + 0xe80u : 0x780u); + bool dup = false; + u32 pi4; + + /* + * SESSION 155 CONT 12: map A/B (see the + * sema_gl_sph_gs_maps comment). 0 keeps the + * pass-through header above. + */ + if (gs_maps == 1u) { + gs_words[5] = 0u; + gs_words[10] = 0u; + gs_words[13] = 0u; + gs_words[17] = 0u; + } else if (gs_maps == 2u) { + gs_words[5] = 0x80000000u; + gs_words[10] = 0u; + gs_words[13] = 0x0000f000u; + gs_words[17] = 0u; + } else if (gs_maps == 3u) { + gs_words[17] = 0u; + } else if (gs_maps == 4u) { + /* IMAP = previous OMAP; OMAP = + * PS IMAP exactly (POSITION.W). */ + gs_words[5] = 0xf0000000u; + gs_words[10] = 0x0000000fu; + gs_words[13] = 0x00008000u; + gs_words[17] = 0u; + } else if (gs_maps == 5u) { + /* mesh-dummy IMAP, OMAP POSITION.W. */ + gs_words[5] = 0u; + gs_words[10] = 0u; + gs_words[13] = 0x00008000u; + gs_words[17] = 0u; + } + /* SESSION 155 CONT 13: mesh-dummy FASTGS + * form (bit24 ENABLE_FAST_G_S). */ + if (gsp_phys_rmcall_gr_sph_gs_mode_value() & 1u) + gs_words[0] |= (1u << 24); + + for (pi4 = 0; pi4 < svc_gl.n_progs; pi4++) + if (svc_gl.progs[pi4] == dst) + dup = true; + if (!dup) { + memset(&prs[n_recs], 0, + sizeof(prs[n_recs])); + prs[n_recs].dst_va = dst; + prs[n_recs].len = 128u; + memcpy(prs[n_recs].data, gs_words, + sizeof(gs_words)); + n_recs++; + if (gsp_sema_dbg && + svc_gl_prog_log < 64) + pr_info("sema-gl: stage-image dst=0x%llx slot=4 len=128 maps=%u first=%08x w5=%08x w13=%08x\n", + dst, gs_maps, gs_words[0], + gs_words[5], gs_words[13]); + } + } + } + if (n_recs && gsp_sema_dbg && svc_gl_prog_log < 64) { + u32 pi2; + + svc_gl_prog_log++; + pr_info("sema-gl: prog-images h=0x%x push=0x%llx recs=%u n_progs=%u vs=%u ps=%u gen=%u\n", + ch->h_object, push_va, n_recs, + svc_gl.n_progs, + svc_gl_vs_len, svc_gl_ps_len, svc_gl_prog_len); + for (pi2 = 0; pi2 < svc_gl.n_progs && pi2 < 8; + pi2++) + pr_info("sema-gl: prog-slot[%u] va=0x%llx slot=%u type=%02x\n", + pi2, svc_gl.progs[pi2], + svc_gl.prog_slot[pi2], + svc_gl_shader_type[svc_gl.prog_slot[pi2] < 32 ? + svc_gl.prog_slot[pi2] : 0]); + } + } else if (blob_n && op_src[0] != 's') { + memset(&prs[0], 0, sizeof(prs[0])); + prs[0].dst_va = svc_gl.prog_va; + prs[0].len = blob_n; + memcpy(prs[0].data, blob, blob_n); + n_recs = 1; + } + /* + * SESSION 153: record this GL section capture in the GL-specific + * exactly-once table so only an actual capture suppresses later + * replays of the same (push_va, content hash). + */ + svc_gl_cap_store(push_va, svc_push_hash(buf, len)); + pr_info("sema-gl: section captured h=0x%x push=0x%llx len=%u off=0x%x prog=0x%llx param=0x%llx/0x%x rel=0x%llx/%u ptr=%u blob=%u replay=%u src=%s nprog=%u recs=%u\n", + ch->h_object, push_va, len, at_off, svc_gl.prog_va, + svc_gl.param_va, svc_gl.param_size, svc_gl.rel_va, + svc_gl.rel_payload, svc_gl.n_ptr, svc_gl.blob_len, + replay ? 1u : 0u, op_src, svc_gl.n_progs, n_recs); + /* + * SESSION 153: remember real operands for this channel so a later + * draw-only section on the same channel can still capture. A + * draw-synth capture carries no channel truth and is never stored. + */ + if (op_src[0] != 'd') + svc_gl_hold_store(ch->h_object, svc_gl.prog_va, + svc_gl.param_va, svc_gl.param_size, + svc_gl.rel_va, svc_gl.rel_payload); + /* R16 bounded witness: what the synthesized QMD now schedules and + * where the code bytes came from. */ + if (gsp_sema_dbg && svc_gl_prog_log < 8) { + svc_gl_prog_log++; + pr_info("sema-gl: M3 sched w12=%u w13=%u w14=%u w18=0x%08x w19=0x%08x w20=0x%08x w29=0x%08x w31=0x%08x w5=0x%08x\n", + s2.qmd[12], s2.qmd[13], s2.qmd[14], s2.qmd[18], + s2.qmd[19], s2.qmd[20], s2.qmd[29], s2.qmd[31], + s2.qmd[5]); + pr_info("sema-gl: M3 code src=%s read_rc=%d len=%u lid=%u first=%08x %08x %08x %08x\n", + blob_src, + blob_rc, blob_n, svc_gl.blob_len, + get_unaligned_le32(blob), + get_unaligned_le32(blob + 4), + get_unaligned_le32(blob + 8), + get_unaligned_le32(blob + 12)); + } + svc_gr_cand_capture(ch, &s2, n_recs ? prs : NULL, n_recs, buf, len, + push_va); + /* + * CONT-87: carry the window's own program binds so the launch + * builder can re-emit them ahead of the window (the window's early + * draws precede its binds; see svc_gr_cand_rec). + */ + { + u32 pb; + + svc_gr_cand.n_prog_binds = 0; + for (pb = 0; pb < svc_gl.n_progs && + pb < SVC_GR_CAND_PROG_BINDS; pb++) { + if (!svc_gl.progs[pb]) + continue; + svc_gr_cand.prog_bind_va[svc_gr_cand.n_prog_binds] = + svc_gl.progs[pb]; + svc_gr_cand.prog_bind_slot[svc_gr_cand.n_prog_binds] = + svc_gl.prog_slot[pb]; + svc_gr_cand.n_prog_binds++; + } + } + /* + * r929 (Path A): a stream-derived capture (op_src=stream: the section + * carried its own operands) is a raw 3D method window. Mark it for + * verbatim replay - but only when the WHOLE push survived in raw[] + * (len == push_n*4): a truncated window would feed the engine a + * half-method stream. The window is SLICED from the section's own + * C597 SET_OBJECT (svc_gl.start_off) through the section close: the + * prelude before the bind carries the X server channel's own + * host-sem + 2D/CE ops, whose semaphore VAs belong to that client's + * VM and are unmapped in the replay channel (r929 v1: the first + * host-sem write faulted FAULT_PDE @ 0x1_2000b000 and the method + * walk died before any 3D method). Every VA the sliced window + * references is covered by the candidate's own mapped pages. The + * 448-word cap leaves headroom for the backend's 512-word method + * block (2 bind words + the window + the 8-word release tail). + * Draw-synth/hold captures are never marked. + * + * SESSION 153-CONTINUED-10: op_src=hold is included. The GLX + * server's per-frame CONTENT pushes carry the draw window but no + * program/param operands (the class bind is sticky; no per-push + * SET_OBJECT), so their sections capture with the channel's real + * operands adopted from the hold table (filled by the init + * capture). The window itself is still the REAL push bytes sliced + * from its own first C597 state method to its END - exactly the + * raw 3D method window the r929 replay lane wants; the synthesized + * compute fields are what stream_3d bypasses. + */ + if ((op_src[0] == 's' || op_src[0] == 'h') && svc_gr_cand.valid && + len == svc_gr_cand.push_n * 4 && + svc_gl.start_off + 8u <= at_off && + (at_off - svc_gl.start_off) / 4u <= 448u) { + svc_gr_cand.stream_3d = true; + svc_gr_cand.stream_off = svc_gl.start_off / 4u; + svc_gr_cand.stream_n = (at_off - svc_gl.start_off) / 4u; + pr_info("sema-gl: 3D stream marked h=0x%x push=0x%llx window=%u words @off=0x%x (verbatim replay from the C597 bind)\n", + ch->h_object, push_va, svc_gr_cand.stream_n, + svc_gl.start_off); + } else if ((op_src[0] == 's' || op_src[0] == 'h') && + svc_gr_cand.valid) { + /* + * SESSION 153-CONTINUED-45: a REAL-operand section whose + * window cannot ride the single 448-dword block - split it + * into method-aligned chunks, each a verbatim replay in ring + * order (channel state persists across launches). The raw[] + * window must still hold the WHOLE push (the slices index + * into it), which is why the per-record raw bound was + * widened; if the raw geometry does not hold, the bounded + * diagnostic below names the failing condition (the next + * bound to size). + */ + if (len == svc_gr_cand.push_n * 4 && + svc_gl.start_off + 8u <= at_off) { + svc_gl_stream_chunks(ch, buf, len, at_off); + } else if (gsp_sema_dbg && svc_gl_mark_log < 64) { + svc_gl_mark_log++; + pr_info("sema-gl: 3D mark failed h=0x%x push=0x%llx win_start=0x%x at=0x%x win=%u words raw_ok=%d fit_ok=%d (rawlen=%u pushn=%u len=%u)\n", + ch->h_object, push_va, svc_gl.start_off, + at_off, + at_off > svc_gl.start_off ? + (at_off - svc_gl.start_off) / 4u : 0u, + len == svc_gr_cand.push_n * 4, + svc_gl.start_off + 8u <= at_off && + (at_off - svc_gl.start_off) / 4u <= 448u, + len / 4u, svc_gr_cand.push_n, len); + } + } + /* + * r1500: draw-args receipt. The remaining wall is "is the captured + * window a real primitive with a real count?" Decode the window's + * own BEGIN immediate (topology), SET_INDEX_BUFFER_A/B (0x17c8/cc) + * and the E/F + DRAW_INDEX_BUFFER triple (0x17d8/dc/e0: index size, + * first, count) with the same 1+cnt run grammar the replay uses. + * This is read-only and names the exact values the ENGINE will see; + * an index size of 1 means the buffer is read as 16-bit indices. + */ + if ((op_src[0] == 's' || op_src[0] == 'h') && + svc_gl_draw_args_log < 64 && svc_gl.start_off < at_off && + at_off <= len) { + u32 o = svc_gl.start_off; + u32 topo = 0xffffffffu, isz = 0xffffffffu; + u32 first = 0, count = 0; + u64 iva = 0; + u32 stride = 0, voff = 0; + u32 cand[4] = { 0, 0, 0, 0 }, ncand = 0; + + while (o + 4 <= at_off) { + u32 h, op, cnt, ad, w0 = 0, w1 = 0, w2 = 0; + u32 span = 1; + + memcpy(&h, buf + o, 4); + op = (h >> 29) & 7u; + cnt = (h >> 16) & 0xfffu; + ad = (h & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt && + o + 4u + cnt * 4u <= at_off) + span = 1u + cnt; + if (span > 1u) { + memcpy(&w0, buf + o + 4, 4); + if (span > 2u) + memcpy(&w1, buf + o + 8, 4); + if (span > 3u) + memcpy(&w2, buf + o + 12, 4); + } + if (op == 4u && ad == 0x1618u && + topo == 0xffffffffu) + topo = cnt; + if (op == 1u && ad == 0x17c8u && cnt >= 2u) + iva = ((u64)(w0 & 0xffu) << 32) | + (u64)(w1 & 0xfffffffcu); + if ((op == 1u || op == 3u || op == 5u) && + ad == 0x17d8u && cnt >= 3u) { + isz = w0; + first = w1; + count = w2; + } + /* + * r1512: vertex-stream shape for the coverage + * receipt. SET_VERTEX_STREAM_A_FORMAT (0x1c00) + * carries the stream-0 stride in bits 11:0; + * SET_VERTEX_ATTRIBUTE_A(0) (0x1160) carries the + * position attribute's byte offset in bits 20:7. + */ + if (op == 1u && ad == 0x1c00u && cnt >= 1u) + stride = w0 & 0xfffu; + if (op == 1u && ad == 0x1160u && cnt >= 1u) + voff = (w0 >> 7) & 0x3fffu; + /* + * The MME setup macro receives the stream base VAs + * as CALL_MME_DATA (0x3810) parameters; keep the + * plausible address words so the receipt can read + * the vertex page the macro will bind. + */ + if (op == 5u && ad == 0x3810u && cnt >= 3u) { + u32 k; + + for (k = 2; k < cnt && ncand < 4u; k++) { + u32 w = get_unaligned_le32( + buf + o + 4u + 4u * k); + + if (w >= 0x100000u && + w < 0x1000000000ULL && + !(w & 3u)) + cand[ncand++] = w; + } + } + o += span * 4u; + } + if (topo != 0xffffffffu || iva || isz != 0xffffffffu) { + svc_gl_draw_args_log++; + pr_info("sema-gl: GL-DRAW h=0x%x push=0x%llx topo=%u isz=%u first=%u count=%u iva=0x%llx\n", + ch->h_object, push_va, topo, isz, first, + count, iva); + if (topo != 0xffffffffu && count >= 3u && iva) + svc_gl_cover_log(vgpu, buf, len, + svc_gl.start_off, at_off, + ch->h_object, push_va, + topo, isz, first, count, iva, + stride, voff, cand, ncand); + } + } +out: + /* + * Apply the decoded GL release forward-only, whether the section was + * captured or skipped. The GLX client's kickoff suffix releases the + * shared-surface refCount that the X DDX's nvClipIdleChannels polls; + * when the forwarded draw does not complete (or the section is not + * capturable) the release stayed held and the DDX sat out its 5 s + * timeout. The release is the only edge the client needs; the + * forward-only + jump-bound guards make a duplicate/stale apply + * harmless. Gated by sema_gl_hold_apply for A/B. + */ + if (gsp_sema_gl_hold_apply && svc_gl.rel_va && svc_gl.rel_payload) { + u32 cur = 0; + + if (svc_sem_read(vgpu, svc_gl.rel_va, &cur)) { + svc_pending_add(svc_gl.rel_va, svc_gl.rel_payload, 0); + } else if (svc_gl.rel_payload > cur && + svc_gl.rel_payload - cur <= 0x100000) { + if (!svc_sem_write(vgpu, 0, svc_gl.rel_va, + svc_gl.rel_payload)) + pr_info("sema-gl: hold applied va=0x%llx pay=%u (cur=%u)\n", + svc_gl.rel_va, svc_gl.rel_payload, + cur); + } + } + /* + * SESSION 149 R15 (M1.1a): preserve the last valid release across + * the reset - it belongs to the channel (decoded from the bracketing + * 24 B release pushes), not to this section; wiping it here and at + * the section open starved the flush guard of rel_va on the trusted + * first pass (the r883/r886 one-shot decode). + */ + { + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + } +} + +/* * Consume the current GR candidate without forwarding it: hand the launch to * the host CUDA bridge when that path is enabled, then clear the candidate. * Exactly-once via `offered`. Called on the service thread (capture/refusal) @@ -5647,22 +12920,59 @@ static void svc_gr_bridge_fallback(struct nv_vgpu_device *vgpu) * Both still require the guest-memory path that makes any hardware * submission possible. Service-thread only, like the captures. */ -static bool svc_fwd_defer_arm(void) +static bool svc_fwd_defer_arm(u64 push_va) { if (!gsp_phys_guest_mem_enabled()) return false; if (gsp_phys_gr_forward_enabled()) { u32 ri; + /* + * r1656: the GR candidate ring survives across decode passes, + * so a candidate captured for ANOTHER launch must not withhold + * this push's release. r824 added this ownership test to the + * C56F release site only; the SET_REPORT_SEMAPHORE and + * LAUNCH_DMA sites kept the global test, and an unrelated + * lingering candidate parked a GR init release that was then + * discarded at the next pass reset (decoded, never applied). + * Only the launch's own push may defer; anything else applies + * inline. + */ for (ri = 0; ri < SVC_GR_CAND_RING; ri++) if (svc_gr_cand_ring[ri].valid && - !svc_gr_cand_ring[ri].offered) + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].push_va == push_va) return true; } return svc_fwd_cand.valid && gsp_phys_work_defer_release_enabled(); } +/* + * r824: the release deferral must be owned by a candidate of THIS push. + * svc_fwd_defer_arm() answers globally (any unoffered candidate of the + * boot's ring); an unrelated lingering candidate then parked the release + * of a push that has no forwarding at all. Observed: the X/GL DMA test's + * progress-tracker release (the kernel's progress semaphore) was recorded + * into svc_fwd_rel and dropped at the next pass's reset, every pass, so + * the semaphore never advanced and Xorg failed at its 10 s wait - while + * every other stage of the chain (latch, NOP acceptance, decode) had + * just started working. A release whose push owns no candidate cannot + * wait for a forwarded copy that will never exist; it applies inline. + */ +static bool svc_fwd_cand_owns_push(u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].offered && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + /* * P6b wave-1.5 deferred guest releases: every release group the current @@ -5708,6 +13018,106 @@ static u8 svc_fwd_rel_n; static bool svc_fwd_defer_armed; /* + * r633: PARKED push-level release records for a GR launch whose entry was + * HELD before it could be forwarded. The records above are per-decode state + * (reset at the top of every svc_decode_stream); the two GR hold paths + * (busy-defer hold, m150 data-stalled hold) break out of the entry loop + * BEFORE the release block runs, so the entry's records would be lost at + * the next decode - and the eventual forward (queued from the capture ring + * in a later pass, which does not re-decode the entry) would engage the + * release hold with whatever records that pass happens to carry (often + * none, or another entry's), leaving the push's tracking release unapplied + * and the client spinning in cudaDeviceSynchronize forever (the r632 mail + * MALLOC hang: push release 0x20480fff0 pay=112 never landed while the + * client's word stayed at 111). Park keyed by (vgpu, channel, push VA); + * restored into svc_fwd_rel[] by svc_gr_request when the launch is finally + * queued, so the pump's hold-engage holds exactly the queued push's + * records; delivered inline by the paths that consume a capture without a + * forward. Service-thread only. + */ +#define SVC_GRPARK_SLOTS 8 +static struct svc_grpark { + struct nv_vgpu_device *vgpu; + u32 h_object; + u64 push_va; + u8 n; + struct svc_fwd_rel rel[SVC_FWD_REL_MAX]; +} svc_grpark[SVC_GRPARK_SLOTS]; +static u32 svc_grpark_pos; + +static void svc_grpark_put(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va, const struct svc_fwd_rel *rel, u8 n) +{ + u32 i, slot = ~0u; + + if (!vgpu || !push_va || !rel || !n) + return; + if (n > SVC_FWD_REL_MAX) + n = SVC_FWD_REL_MAX; + for (i = 0; i < SVC_GRPARK_SLOTS; i++) { + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + slot = i; + break; + } + if (slot == ~0u && !svc_grpark[i].push_va) + slot = i; + } + if (slot == ~0u) + slot = svc_grpark_pos++ % SVC_GRPARK_SLOTS; + svc_grpark[slot].vgpu = vgpu; + svc_grpark[slot].h_object = h_object; + svc_grpark[slot].push_va = push_va; + svc_grpark[slot].n = n; + memcpy(svc_grpark[slot].rel, rel, + sizeof(svc_grpark[slot].rel)); +} + +static bool svc_grpark_get(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va, struct svc_fwd_rel *out, u8 *n) +{ + u32 i; + + for (i = 0; i < SVC_GRPARK_SLOTS; i++) { + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + if (out) + memcpy(out, svc_grpark[i].rel, + sizeof(svc_grpark[i].rel)); + if (n) + *n = svc_grpark[i].n; + return true; + } + } + return false; +} + +static void svc_grpark_drop(struct nv_vgpu_device *vgpu, u32 h_object, + u64 push_va) +{ + u32 i; + + for (i = 0; i < SVC_GRPARK_SLOTS; i++) + if (svc_grpark[i].vgpu == vgpu && + svc_grpark[i].h_object == h_object && + svc_grpark[i].push_va == push_va) { + memset(&svc_grpark[i], 0, sizeof(svc_grpark[i])); + return; + } +} + +static void svc_grpark_reset(void) +{ + memset(svc_grpark, 0, sizeof(svc_grpark)); + svc_grpark_pos = 0; +} + +/* r633: bounded evidence that a parked set was restored/delivered. */ +static u32 svc_grpark_log; + +/* * P6b wave 2: positive sentinel returned by svc_copy() when the captured * forwarding candidate's software copy is deliberately withheld (the * hardware-primary path will have the real engine author the data). @@ -5997,14 +13407,18 @@ static void svc_copy_snap_free(u64 dst, u64 src, size_t total, } /* - * r444: drop every snapshot still owned by one entry. A snapshot that was - * not consumed by the entry's own copy execution (the copy ran via the - * forwarded hardware path, or a post-execution re-decode re-captured the - * unchanged staging) has no future consumer once the entry's release is - * delivered - leaving it behind let a LATER entry whose copy shares the - * (dst,src,total) triple execute from the stale bytes. + * r456/r457: the snapshot lifecycle is capture-once -> free on the copy's + * own successful consumption (svc_copy) -> free on ring-slot reuse with a + * new push hash (svc_copy_snap_capture) -> free when a re-decode skips an + * already-done op (svc_run_ops) -> TTL. The old per-entry owner free was + * removed from every opdef_clear caller: an entry whose release was + * delivered can still be REPLAYED (the pin-batch rewind) with its copy + * still owed, and freeing at that point let the replay re-capture the + * driver's refilled ping-pong staging (the r457 mode-C b1 H2D wrote b2's + * bytes into db1). A stale snapshot can only ever serve its own + * (ch, idx, hash) owner, so the r444 cross-entry poisoning stays closed. */ -static void svc_copy_snap_free_owner(u32 owner_ch, u32 owner_idx) +static void __maybe_unused svc_copy_snap_free_owner(u32 owner_ch, u32 owner_idx) { u32 i; @@ -6032,6 +13446,24 @@ static void svc_copy_snap_capture(struct nv_vgpu_device *vgpu, u64 dst, if (svc_copy_snap_lookup(dst, src, total, owner_ch, owner_idx, owner_hash)) return; /* already captured */ + /* + * r456: evict any snapshot left by the SAME ring slot under a + * DIFFERENT push hash (the slot was reused by a fresh push). The + * surviving-across-retries snapshot (svc_opdef_clear APPLY no + * longer frees it) must never be readable by the next occupant of + * the slot; the pass hash changes whenever the guest rewrites the + * push, so a same-slot different-hash snapshot is by definition + * stale for the new push. + */ + for (i = 0; i < SVC_COPY_SNAP_N; i++) { + if (svc_copy_snaps[i].buf && + svc_copy_snaps[i].owner_ch == owner_ch && + svc_copy_snaps[i].owner_idx == owner_idx && + svc_copy_snaps[i].owner_hash != owner_hash) { + kfree(svc_copy_snaps[i].buf); + svc_copy_snaps[i].buf = NULL; + } + } for (i = 0; i < SVC_COPY_SNAP_N; i++) { if (!svc_copy_snaps[i].buf) { s = &svc_copy_snaps[i]; @@ -6094,6 +13526,20 @@ static int svc_copy(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t total, u64 *done_out, u32 rows, u32 dpitch, u32 spitch, u32 owner_ch, u32 owner_idx, u64 owner_hash) { + /* + * fail4: name every copy the executor is asked to run (pre- + * resolution), so a copy that fails or is held before any witness + * still has an entry record. SESSION 132: one record per COPY OP, + * not per row - a 480-row frame copy logged 480 rows and exhausted + * the per-session budget (128) inside the first op, hiding every + * later copy (including any SASS upload). + */ + if (gsp_sema_dbg && svc.copyent_log < 128) { + svc.copyent_log++; + pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu rows=%u dpitch=0x%x spitch=0x%x dphys=%u dap=%u sphys=%u sap=%u\n", + svc.cur_ch, dst, src, total, rows, dpitch, spitch, + dst_phys, dst_aperture, src_phys, src_aperture); + } if (rows > 1 && dpitch) { u64 done = 0; int rc = 0; @@ -6135,6 +13581,7 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot bool audit = total >= SVC_COPY_AUDIT_MIN; u8 *snap = NULL; static u64 bigcopy_log; + static u32 shadow_dst_log; /* r614 diag: shadow-lane dst witness */ bool trace64 = (total == 65536u); const char *dir = "?"; @@ -6142,17 +13589,6 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot *done_out = 0; /* 0 until the first byte lands */ if (!total || total > (256ULL << 20)) return -EINVAL; - /* - * fail4: name every copy the executor is asked to run (pre- - * resolution), so a copy that fails or is held before any witness - * still has an entry record. - */ - if (gsp_sema_dbg && svc.copyent_log < 128) { - svc.copyent_log++; - pr_info("sema-copyent: h=0x%x dst=0x%llx src=0x%llx total=%zu dphys=%u dap=%u sphys=%u sap=%u\n", - svc.cur_ch, dst, src, total, dst_phys, dst_aperture, - src_phys, src_aperture); - } snap = svc_copy_snap_lookup(dst_orig, src_orig, total, owner_ch, owner_idx, owner_hash); @@ -6251,6 +13687,26 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot } else { dst_shadow = true; svc.vas_shadowed++; + /* + * r614 diag: the r613b red boot's matmul + * wedge began here - the H2D into the + * client's cudaMalloc'd buffer found no + * ledger/pin/walk answer, took the shadow + * silently, and the later GR fill could not + * consume the shadowed bytes (probe=none for + * every arg base; the launch held 15 s, then + * the degenerate refusal consumed it and the + * client spun). Name the first shadow-lane + * device-dst copies of a boot (bounded), so + * the next red boot is greppable where it + * starts instead of only at the launch. + */ + if (shadow_dst_log < 64 && + dst_orig >= SVC_VAS_FLOOR) { + shadow_dst_log++; + pr_info("sema-shadow-dst: h=0x%x dst=0x%llx total=%zu (device VA with no ledger/pin/walk answer; copy runs into the shadow, a GR fill cannot consume it)\n", + owner_ch, dst_orig, total); + } } } else { dst = pa; @@ -6277,7 +13733,7 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot if (dst_virt && dst_aperture == CE_APERTURE_VID && dst_prov >= 3 && dst_orig >= SVC_VAS_FLOOR && gsp_sema_copy_dst_pin_guard && - svc_dst_guard_hold(dst_orig)) { + svc_dst_guard_should_hold(dst_orig)) { svc.copy_dst_hold++; if (svc.copy_dst_hold <= 16) pr_info("sema-svc: device-dst copy held: dst=0x%llx prov=%u (no pin yet; stale-table guard)\n", @@ -7084,6 +14540,22 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot * kept the stale pre-pin pages while the copy landed on the pinned * page (streams2 r254: the b1 rec range carried the a2 pages and * the vadd read a2's pattern as its second input). + * + * r627: NEVER overwrite a per-page landing the loop above already + * stored. That loop stores each page's true resolved PA; this + * pass's `dst + po` is a contiguity guess valid only while the VA + * range maps contiguously. For a scattered UVM host buffer (r623c + * saxpy: page 0 at pa 0x115781000, page 0x7f000 at pa 0x12e700000) + * the guess clobbered the true entries - + * `sema-ledger: rewrite va=0x70e64e77f000 old=0x12e700000 + * new=0x115800000` - and the NEXT copy of the same VA then + * followed the poisoned ledger: 504x `sema-ledger: DIVERGE write + * va=0x70e64e7xxxxx prev=0x1158xxxxx cur=0x12e7xxxxx`; the + * non-pin branch keeps the ledger, so ~2 MiB of client c-data + * landed in guest pages the kernel had since re-used and the + * guest died in kchannelGetGfid_IMPL (r623c panic). The loop's + * per-page entries are the authority; this pass only seeds pages + * the loop did not reach. */ if (!dst_phys && dst_virt && dst_orig >= SVC_VAS_FLOOR && total >= (64u << 10)) { @@ -7091,8 +14563,14 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot SVC_APERTURE_FB : SVC_APERTURE_SYS; size_t po; - for (po = 0; po < total; po += SEMA_PAGE_SIZE) + for (po = 0; po < total; po += SEMA_PAGE_SIZE) { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup(dst_orig + po, &lpa, &lap)) + continue; svc_ledger_store(dst_orig + po, dst + po, cap); + } } if (snap) svc_copy_snap_free(dst_orig, src_orig, total, owner_ch, @@ -7108,11 +14586,95 @@ static int svc_copy_1d(struct nv_vgpu_device *vgpu, u64 dst, u64 src, size_t tot * VA through the trusted sources (pins, discovered map, published page * directories, targeted rm_mem resolve), pin the page and write the bytes. */ +/* + * SESSION 154 r1506: the VAS of the channel that authored an inline + * write. The GR channel declares hVASpace=0 ("attach to the TSG address + * space") while its CE siblings (same object group 0xbeef01xx) declare the + * TSG's VAS, so a zero handle resolves through the group. A client write + * must never be translated through another client's VAS: the VA-keyed + * cross-VAS walk is how the GL client's fixed-CB/shader uploads at + * 0x120700000/0x120400000 landed in the X server's screen (screen pages + * 0x500000/0x200000 received the CB matrix and the shader image, while + * the client's own VAS maps the same VAs to its CB page 0x5cf000 and to + * its SYS buffer). Returns 0 when no VAS is known for the channel. + */ +static u32 svc_channel_owner_vas(struct nv_vgpu_device *vgpu, u32 h_object) +{ + unsigned long flags; + u32 i, group, vas = 0; + + if (!h_object) + return 0; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (!svc.chan[i].valid || svc.chan[i].vgpu != vgpu || + svc.chan[i].h_object != h_object) + continue; + if (svc.chan[i].h_vaspace) { + vas = svc.chan[i].h_vaspace; + break; + } + } + if (!vas) { + group = h_object & ~0xffu; + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (!svc.chan[i].valid || svc.chan[i].vgpu != vgpu) + continue; + if ((svc.chan[i].h_object & ~0xffu) == group && + svc.chan[i].h_vaspace) { + vas = svc.chan[i].h_vaspace; + break; + } + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return vas; +} + +/* + * SESSION 153-CONTINUED-88: write an inline payload into one landing page + * (page-aligned PA, aperture) at the destination VA's in-page offset. Shared + * by the primary landing write and the engine-visible pin dual-write. The + * write is clamped to the page end: a crossing payload's tail belongs to the + * NEXT page's own landing, not to whatever follows this page in the aperture. + */ +static int svc_grce_land_write(struct nv_vgpu_device *vgpu, u64 pa_page, + int ap, u64 dst_va, const u8 *data, u32 bytes) +{ + u64 pa = pa_page + (dst_va & 0xfff); + u32 room = 4096u - (u32)(dst_va & 0xfff); + + if (bytes > room) + bytes = room; + if (!bytes) + return 0; + if (ap == SVC_APERTURE_FB) { + u8 *base = vgpu->bars[1].vaddr; + + if (!base || pa + bytes > svc_fb_shadow_size(vgpu)) + return -EINVAL; + memcpy(base + pa, data, bytes); + return 0; + } + { + u32 o; + + for (o = 0; o < bytes; o += 4096) { + u32 chunk = bytes - o > 4096 ? 4096 : bytes - o; + + if (svc_write(vgpu, pa + o, (void *)(data + o), chunk)) + return -EIO; + } + } + return 0; +} + static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, const u8 *data, u32 bytes, u32 ch_h) { u64 pa = 0; int ap = SVC_APERTURE_SYS; + bool owner_known = false; if (!bytes || bytes > (1u << 20)) return -EINVAL; @@ -7130,6 +14692,75 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, return 0; } /* + * r1506 VA ownership (see svc_channel_owner_vas): resolve the + * destination through the authoring channel's own page tables first. + * The generic chain below walks EVERY published VAS, and every + * client's RM window sits at the same 0x1200xxxxx VAs: the GL + * client's fixed-CB write (dst 0x120700000) resolved through the X + * server's VAS to screen page 0x500000 and its shader upload + * (0x120400000) likewise reached the screen. The client's own VAS + * maps those VAs to its own pages; when its VAS knows the channel + * but not the page, the page is client-private and the cross-VAS + * walk is refused (write to the VA shadow instead). + */ + { + /* + * Scope only the beef GL client (0xbeef0xxx). Other channels + * (the X server 0x1000013, the bring-up channels) keep the + * legacy chain: their VAS handle may be a DUP alias that the + * svc.vas table does not name, and scoping them would strand + * their page-table/upload writes in the VA shadow. + */ + u32 ovas = ((ch_h & 0xfffff000u) == 0xbeef0000u) ? + svc_channel_owner_vas(vgpu, ch_h) : 0; + u64 vpa = 0; + int vap = SVC_APERTURE_SYS; + + if (ovas) { + owner_known = true; + if (!svc_vas_lookup_h(vgpu, ovas, + dst_va & ~0xfffULL, &vpa, &vap)) { + pa = vpa & ~0xfffULL; + ap = vap; + svc_pin_store(dst_va & ~0xfffULL, pa, ap); + /* + * The engine fetches a program page through + * its relocated shadow/ledger landing while the + * client owns the SYS page; keep both current. + */ + if (ap == SVC_APERTURE_SYS) { + u64 lpa = 0; + u32 lap = 0; + + if (!svc_ledger_lookup( + dst_va & ~0xfffULL, &lpa, + &lap) && + lap == SVC_APERTURE_FB && + (lpa & ~0xfffULL) != + (pa & ~0xfffULL)) + (void)svc_grce_land_write( + vgpu, + lpa & ~0xfffULL, lap, + dst_va, data, bytes); + } + if (gsp_sema_dbg && svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: owner-vas h=0x%x vas=0x%x dst=0x%llx -> pa=0x%llx ap=%s bytes=%u (client-owned page; cross-VAS walk skipped)\n", + ch_h, ovas, dst_va, pa, + ap == SVC_APERTURE_FB ? + "vidmem" : "ram", + bytes); + } + goto write_landing; + } + if (gsp_sema_dbg && svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: owner-vas h=0x%x vas=0x%x dst=0x%llx unmapped; cross-VAS walk refused (client-private page)\n", + ch_h, ovas, dst_va); + } + } + } + /* * m180 unification: a page with a known landing (previous H2D/fill or * engine write-back) is authoritative - write the inline data there * so the engine fill / D2H readers that follow the table see it. @@ -7138,15 +14769,106 @@ static int svc_grce_inline_write(struct nv_vgpu_device *vgpu, u64 dst_va, u64 lpa = 0; u32 lap = 0; - if (!svc_ledger_lookup(dst_va, &lpa, &lap)) { + /* + * r1506: the ledger is VA-keyed like the cross-VAS walk; a + * channel-owned VA whose own VAS does not map it must not + * follow another tenant's landing (the screen pages were + * recorded here by the X fill). + */ + if (!owner_known && !svc_ledger_lookup(dst_va, &lpa, &lap)) { pa = lpa & ~0xfffULL; ap = (lap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : SVC_APERTURE_SYS; + /* + * SESSION 153-CONTINUED-88: dual-write to the + * engine-visible page. The launch fill resolves a + * page through the LIVE PINS first ("pin wins", see + * svc_gr_page_note) and the engine reads exactly + * that page - a ledger landing in the BAR1 shadow + * (ap=vidmem) is NOT what the engine fetches. The + * observed case: the real program uploads (1536 B at + * 0x4350900, 1408 B at 0x4350f00) landed at pa + * 0x1b0900/0x1b0f00 (vidmem) while the engine read + * the pinned sysmem page (0x529ff4000). When an + * injected pin names a different page, author the + * bytes there too (the ledger landing stays written + * for its existing consumers). + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_va & ~0xfffULL, + &ipa, &iap) && + ipa && (ipa & ~0xfffULL) != + (pa & ~0xfffULL)) { + if (!svc_grce_land_write(vgpu, + ipa & ~0xfffULL, + iap, dst_va, + data, bytes) && + gsp_sema_dbg && + svc.grce_log < 64) { + svc.grce_log++; + pr_info("sema-grce: inline dual-write dst=0x%llx pa=0x%llx ap=%s bytes=%u (engine-visible pin) h=0x%x\n", + dst_va, + ipa & ~0xfffULL, + iap == SVC_APERTURE_FB ? + "vidmem" : "ram", + bytes, ch_h); + } + } + } svc_pin_store(dst_va & ~0xfffULL, pa, ap); goto write_landing; } } - if (svc_lookup_trusted(vgpu, dst_va & ~0xfffULL, &pa, &ap)) { + /* + * SESSION 153-CONTINUED-87: the launch fill resolves a page through + * the LIVE PINS first ("pin wins", see svc_gr_page_note) and the + * engine reads exactly that page. An upload whose trusted landing is + * a VIDMEM aperture page must also reach the pinned engine-visible + * page, or the engine validates stale content: the real program + * uploads (1536 B at 0x4350900, 1408 B at 0x4350f00) currently land + * only in the BAR1 shadow while the fill hands the engine the pinned + * sysmem page. Write to the pin when it names a different page. + */ + { + u64 ipa = 0; + int iap = SVC_APERTURE_SYS; + + if (!svc_ipin_lookup(dst_va & ~0xfffULL, &ipa, &iap) && + iap != SVC_APERTURE_FB && ipa) { + pa = ipa & ~0xfffULL; + ap = SVC_APERTURE_SYS; + goto write_landing; + } + } + /* + * r1506: an owner-known VA whose own page tables did not answer and + * whose injected pin did not either is client-private. Keep the bytes + * in the VA shadow (the client's own subsequent reads consult it) + * and never translate through another tenant's VAS. + */ + if (owner_known) { + if (!svc_vas_write(dst_va, data, bytes)) { + svc.vas_shadowed++; + if (svc.grce_log < 16) { + svc.grce_log++; + pr_info("sema-grce: inline write dst=0x%llx bytes=%u shadowed (client-private; no cross-VAS walk) h=0x%x\n", + dst_va, bytes, ch_h); + } + svc.memsets_done++; + return 0; + } + if (gsp_sema_dbg && svc.grce_log < 16) { + svc.grce_log++; + pr_info("sema-grce: inline write dst=0x%llx bytes=%u UNRESOLVED (client-private) h=0x%x\n", + dst_va, bytes, ch_h); + } + return -ENOENT; + } + if (!owner_known && + svc_lookup_trusted(vgpu, dst_va & ~0xfffULL, &pa, &ap)) { if (!svc_vas_write(dst_va, data, bytes)) { svc.vas_shadowed++; if (svc.grce_log < 16) { @@ -7274,6 +14996,30 @@ static bool svc_wait_satisfied(u32 cur, u8 op, u32 need) } } +/* + * r638: is (h, idx) currently owned by the bounded wait gate? The CE + * stream-order watermark consults this to tell a producer backlog (a CE + * copy the client ordered before the launch - the reason the watermark + * exists) from a consumer-side wait the forward itself will unblock (the + * mutual citation measured on r612/r636: the deferred launch's release is + * the only value that can satisfy the held copy's acquire, and the copy's + * unconsumed entry is the only reason the launch is deferred). + */ +static bool svc_wait_held_entry(u32 h, u32 idx) +{ + int i; + + if (idx == 0xffffffffu) + return false; + for (i = 0; i < SVC_WAIT_HOLD_MAX; i++) { + const struct svc_wait_hold *e = &svc_wait_holds[i]; + + if (e->active && e->h == h && e->idx == idx) + return true; + } + return false; +} + /* r34x: the per-channel first-hold clock for the bounded wait. */ static struct svc_wait_hold *svc_wait_hold_get(u32 h, bool create) { @@ -7293,10 +15039,218 @@ static struct svc_wait_hold *svc_wait_hold_get(u32 h, bool create) memset(free_slot, 0, sizeof(*free_slot)); free_slot->active = true; free_slot->h = h; + free_slot->idx = 0xffffffffu; /* r638: unset until a hold lands */ free_slot->since = jiffies; return free_slot; } +/* + * r506w/r507a: bounded per-VA failure latch for virtual ops. A virtual op + * that keeps failing -ENOENT (an unresolvable VA) re-runs on every decode + * pass and sets svc.op_tr_pending each time, so its push holds for the full + * SVC_OPDEF_HOLD_MS opdef bound - a client whose stream carries such an op + * (r505/r506 witness: the RM's channel-progress zeroing at 0x2000c0xxx + * after the first client, dst never translated) ground its whole stream + * at ~10 s per push ("hang after DEVICE" for matmul/multikernel). + * + * The window must still cover the legitimate pin-late retry (observed + * pin latency 0.5-2 s: the H2D copy's operands land a moment after the + * push, r507 matmul's 1 MB UVA copy at 0x7136... needed >8 fast passes), + * so the latch is TIME-based: the FIRST failure of a VA arms a + * SVC_OPDEF_HOLD_MS window during which the op holds its entry exactly as + * before; after a full bound with no success the op still attempts the + * write but no longer forces the hold - the entry consumes promptly and + * its withheld release applies, exactly the outcome the expired bound + * already sanctioned (op stays skipped, release delivered), minus the + * 10 s tax per subsequent pass. A success clears the record (a later + * failure gets a fresh window). Service-thread only, like every other op + * path. + */ +#define SVC_OPFAIL_MAX 32 +/* + * r507a: the op-failure window rides the opdef hold bound (SVC_OPDEF_HOLD_MS + * defined further down). Keep the literal here so the latch (used by + * svc_run_ops) precedes that define. + * + * r549: 10000 -> 3000. The window is the SECOND wait an unresolvable op + * costs: fail #1 arms this record, the entry is held for the opdef bound, + * the retry's fail #2 latches the op out of the hold. r548 evidence: a + * client whose init sequence carries ~65 device-dst ops with destinations + * no resolver covers walked one entry per 10 s (OPDEF-EXPIRED every + * 10.5 s, idx 7 -> 18) - the client spun at 100 % CPU for 10+ minutes and + * the battery timed out. Ops that DO resolve show 0.5-2 s pin latency, so + * 3 s keeps the m163 protection and cuts the dead-op cost to ~3 s per + * distinct destination. + */ +#define SVC_OPFAIL_WINDOW_MS 3000u +static struct { + u64 va; + unsigned long since; +} svc_opfail[SVC_OPFAIL_MAX]; +static u32 svc_opfail_log; + +static void svc_opfail_clear(u64 va) +{ + u32 i; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) + if (svc_opfail[i].va == va) + svc_opfail[i].va = 0; +} + +/* + * r541: device-destination copies that decoded but have not executed yet. + * + * smiley live evidence (r539): passes 2 and 4 verified a + (previous b) - + * the client's cudaMemcpy(b) completed (its release landed) while the H2D + * op was still held, the client submitted the launch, the GR forward read + * the shadow page BEFORE the H2D executed (b's copy ran at 294026.818, + * launch 2's engine read at ~294026.3), and the engine computed with the + * PREVIOUS pass's b (bad=65024). The same shape is the streams2 / + * streams_events readback lottery: a device page whose writer has not run. + * + * The guard: when a device-dst copy (or fill) op fails to execute this + * pass, its destination VA page is recorded here with a stamp. A GR launch + * whose data pages overlap a fresh record is NOT forwarded (the pump's + * bounded busy-defer retries it; the record clears when the op finally + * executes or the SVC_DSTPEND_MS window expires, at which point the + * historical forward stands). Keyed by VA page: the page identity is what + * the launch operand and the copy share, and it works even when neither + * side resolves. + */ +#define SVC_DSTPEND_MAX 64 +#define SVC_DSTPEND_MS 10000u +static struct { + u64 va_page; + unsigned long since; +} svc_dstpend[SVC_DSTPEND_MAX]; +static u32 svc_dstpend_n; +static u32 svc_dstpend_log; + +static void svc_dstpend_note(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + if (!page) + return; + for (i = 0; i < svc_dstpend_n; i++) { + if (svc_dstpend[i].va_page == page) { + /* + * r553: the window is ABSOLUTE from the first failed + * attempt for this page, never re-freshened by the + * retries. The retried decode of a copy that cannot + * resolve re-notes its destination on every pass; with + * a sliding stamp the record never expired and the r541 + * launch guard deferred the launch forever - r551 soak: + * victory_strided repeated "GR busy-defer timeout -> + * hold 8000 ms" every ~4 s for 427 s with the launch + * never forwarded (the copies' -ENOENT retries kept the + * note fresh). A record older than the window is + * refreshed only when a NEW failure episode starts. + */ + if (time_before(jiffies, svc_dstpend[i].since + + msecs_to_jiffies(SVC_DSTPEND_MS))) + return; + svc_dstpend[i].since = jiffies; + return; + } + } + if (svc_dstpend_n < SVC_DSTPEND_MAX) + i = svc_dstpend_n++; + else + i = (u32)(jiffies % SVC_DSTPEND_MAX); + svc_dstpend[i].va_page = page; + svc_dstpend[i].since = jiffies; +} + +/* + * r566: note the destinations of every COLLECTED device-dst copy that did + * not execute this pass, whenever the entry is being held (op_tr_pending). + * The r541 launch guard could only see copies that FAILED with -ENOENT + * (svc_dstpend_note at the failure site); the r568 mail failure is the + * missing case: the b H2D's COPY op was collected but never attempted while + * its entry was held for another operand, the launch's copy-in ran 0.6 s + * before the copy executed, and the engine computed c = a + 0 (the + * readback page held a's values). With the destination noted, the launch + * guard defers until the copy runs (its success path clears the record), + * bounded by the same dstpend TTL + busy-defer machinery as every other + * guard. + */ +static void svc_dstpend_note_ops(struct svc_memset_op *ops, u32 n, u32 base, + u64 *ops_done) +{ + u32 i; + + for (i = 0; i < n; i++) { + struct svc_memset_op *m = &ops[i]; + u32 seq = base + i; + + if (m->type != 1 || m->phys || m->dst < SVC_VAS_FLOOR) + continue; + if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) + continue; /* executed (this pass or earlier) */ + svc_dstpend_note(m->dst); + } +} + +static void svc_dstpend_clear(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_dstpend_n; i++) + if (svc_dstpend[i].va_page == page) + svc_dstpend[i].va_page = 0; +} + +static bool svc_dstpend_hit(u64 va) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_dstpend_n; i++) { + if (svc_dstpend[i].va_page != page) + continue; + if (time_after_eq(jiffies, svc_dstpend[i].since + + msecs_to_jiffies(SVC_DSTPEND_MS))) { + svc_dstpend[i].va_page = 0; + return false; + } + return true; + } + return false; +} + +static bool svc_opfail_latched(u64 va) +{ + u32 i; + u32 free_i = SVC_OPFAIL_MAX; + + for (i = 0; i < SVC_OPFAIL_MAX; i++) { + if (svc_opfail[i].va == va) { + if (time_after_eq(jiffies, + svc_opfail[i].since + + msecs_to_jiffies(SVC_OPFAIL_WINDOW_MS))) { + if (svc_opfail_log < 32) { + svc_opfail_log++; + pr_info("sema-svc: OPFAIL-LATCH va=0x%llx (>=%u ms of failures; op no longer holds its entry)\n", + va, SVC_OPFAIL_WINDOW_MS); + } + return true; + } + return false; + } + if (!svc_opfail[i].va && free_i == SVC_OPFAIL_MAX) + free_i = i; + } + if (free_i < SVC_OPFAIL_MAX) { + svc_opfail[free_i].va = va; + svc_opfail[free_i].since = jiffies; + } + return false; +} + static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, u32 n_ops, u32 ch_h, u64 *ops_done, u32 ops_base, u32 entry_idx, u64 pass_hash) @@ -7305,6 +15259,8 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, static u64 bigop_log; svc.cur_ch = ch_h; + /* r484: one-shot per call; set only by the hold->pass transition. */ + svc_wait_repass = false; /* * m163: apply every queued scanner injection before resolving this * pass's copy operands. Without this, a CE copy decoded in the window @@ -7345,7 +15301,6 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, struct svc_wait_hold *gh; bool hold = false; u32 wi; - /* * Kill switch: a wait that never satisfies (a counter our * emulation does not advance the way the driver expects) @@ -7398,26 +15353,57 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, pend = svc_wait_pend_release(w->va); if (waited_ms >= gsp_sema_wait_hold_ms) { /* - * Bounded episode expired: fail open for the - * WHOLE entry (one row, then the ops run). A - * per-wait continue here would ping-pong - * with a second unsatisfied wait: the first - * gave up and reset the shared clock before - * the second was judged, and the pair would - * hold forever 100 ms at a time. + * r613: a deferred GR launch extends the + * episode. The release this acquire waits for + * may be authored by a launch we are holding + * through our own bounded deferral (the r612 + * streams2 capture: the D2H readback's + * acquire on 0x20480ffe0 was given up at + * waited_ms=10333 while the vadd launches + * that release it sat deferred by the CE + * watermark for 15 s; the readback then ran + * before the kernels and the client read + * zeros). While a captured candidate is + * unoffered, "the value cannot be installed + * by firmware alone" is not yet true - keep + * holding, bounded by the defer cap plus + * margin. The client-side timeouts are all + * >= 60 s, so a <= 24 s episode cannot + * strand it. */ - svc_wait_giveup++; - if (svc_wait_log < 2048) { - svc_wait_log++; - pr_info("sema-wait: give-up ch=0x%x va=0x%llx op=%u pay=%u cur=%u pend=%u waited_ms=%u -> fail-open (cannot be satisfied by firmware alone; ops run)\n", - ch_h, w->va, w->op, w->payload, - cur, pend, waited_ms); + if (waited_ms < SVC_WAIT_EXT_CAP_MS && + svc_gr_forward_pending()) { + svc_wait_extended++; + if (svc_wait_ext_log < 32) { + svc_wait_ext_log++; + pr_info("sema-wait: extend ch=0x%x va=0x%llx op=%u pay=%u cur=%u waited_ms=%u -> hold (a GR forward is owed; the release is behind our own defer)\n", + ch_h, w->va, w->op, + w->payload, cur, + waited_ms); + } + } else { + /* + * Bounded episode expired: fail open for the + * WHOLE entry (one row, then the ops run). A + * per-wait continue here would ping-pong + * with a second unsatisfied wait: the first + * gave up and reset the shared clock before + * the second was judged, and the pair would + * hold forever 100 ms at a time. + */ + svc_wait_giveup++; + if (svc_wait_log < 2048) { + svc_wait_log++; + pr_info("sema-wait: give-up ch=0x%x va=0x%llx op=%u pay=%u cur=%u pend=%u waited_ms=%u -> fail-open (cannot be satisfied by firmware alone; ops run)\n", + ch_h, w->va, w->op, w->payload, + cur, pend, waited_ms); + } + if (gh) + gh->active = false; + hold = false; + svc_wait_pass++; + goto svc_wait_done; } - if (gh) - gh->active = false; - hold = false; - svc_wait_pass++; - goto svc_wait_done; } hold = true; if (svc_wait_log < 512) { @@ -7430,9 +15416,26 @@ static void svc_run_ops(struct nv_vgpu_device *vgpu, struct svc_memset_op *ops, } if (hold) { svc_wait_held++; + svc_wait_hold_h = ch_h; + svc_wait_hold_idx = entry_idx; + /* r638: record the held entry for the CE watermark's + * mutual-cycle check. */ + if (gh) + gh->idx = entry_idx; svc.op_tr_pending = 1; return; } + /* + * r484: the gate just PASSED for an entry it previously held - + * the dependency is installed now, so this pass's ops must + * re-run even if the once-only bitmap says they already did + * (they ran against the pre-dependency state). + */ + if (ch_h == svc_wait_hold_h && + entry_idx == svc_wait_hold_idx) { + svc_wait_repass = true; + svc_wait_hold_idx = 0xffffffffu; + } svc_wait_pass++; if (gh) gh->active = false; @@ -7450,8 +15453,21 @@ svc_wait_done: * ran; re-running a CE copy would read a staging buffer the * client has since refilled and overwrite finished data. */ - if (ops_done && seq < 64 && ((*ops_done >> seq) & 1)) { + if (!svc_wait_repass && ops_done && seq < 64 && + ((*ops_done >> seq) & 1)) { svc.ops_once_skipped++; + /* + * r456: this op already ran in an earlier pass. Any + * staging snapshot a later re-decode pass re-captured + * for it (the capture sites run on every decode) is + * a leak that the r444 APPLY-free used to sweep; with + * that free removed (r456 keeps the snapshot alive + * across retries) drop it here so it can never serve + * a re-push of the ring slot. + */ + if (m->type == 1 && m->bytes <= SVC_COPY_SNAP_MAX) + svc_copy_snap_free(m->dst, m->src, m->bytes, + ch_h, entry_idx, pass_hash); continue; } @@ -7527,11 +15543,28 @@ svc_wait_done: */ if (rc == SVC_COPY_FORWARDED) continue; - if (rc == 0) + if (rc == 0) { svc.memsets_done++; - else if (rc == -ENOENT) + /* r507a: a success re-arms the failure window. */ + svc_opfail_clear(m->dst); + /* r541: the device page's writer has run; lift the + * launch-ordering guard for it. */ + svc_dstpend_clear(m->dst); + } else if (rc == -ENOENT) { + /* + * r541: a copy/fill that cannot run yet leaves its + * target page stale; a GR launch must not read it + * (see svc_dstpend_note). Record every unexecuted + * target - the VA-page key keeps the guard narrow. + */ + svc_dstpend_note(m->dst); + if (svc_dstpend_log < 64) { + svc_dstpend_log++; + pr_info("sema-svc: DSTPEND note va=0x%llx type=%u ap=%u (device write unexecuted; GR launches touching this page hold)\n", + m->dst, m->type, m->aperture); + } svc.memsets_nomap++; - else + } else svc.memsets_fail++; /* * ISSUE-0002 (r71/r72): a failed LARGE op leaves work owed on @@ -7560,7 +15593,8 @@ svc_wait_done: if (rc != 0 && rc != SVC_COPY_FORWARDED && (m->bytes >= (256u << 10) || (m->type == 1 && m->dst >= SVC_VAS_FLOOR) || - (m->type == 0 && !m->phys && m->dst >= SVC_VAS_FLOOR))) + (m->type == 0 && !m->phys && m->dst >= SVC_VAS_FLOOR)) && + !svc_opfail_latched(m->dst)) svc.op_tr_pending = 1; if (gsp_sema_dbg && m->bytes == 256 && svc.sem256_log < 64) { svc.sem256_log++; @@ -7633,18 +15667,20 @@ static bool svc_stream_parse(const u8 *buf, u32 len) memcpy(&hdr, buf + off, 4); if (hdr == 0) { /* - * UVM's large page-table pushes begin with a - * zero-filled header region (observed 0x1004 - * bytes) before the first method, so zeros before - * the first method are leading padding; zeros - * after one are trailing padding and end the - * stream. + * NVC06F_DMA_NOP is 0x00000000: a zero word is + * NOP padding. UVM's large page-table pushes begin + * with a zero-filled header region (observed 0x1004 + * bytes) and the GR submit pushes carry zero gaps + * between method groups (observed r715: the 1432- + * and 17316-byte pushes pad between the context + * setup and the dispatch groups), so zeros are + * skipped anywhere in the stream, not only before + * the first method or at the tail. The exact-end + * requirement and the valid-header walk below are + * the false-positive arbiters. */ - if (!methods) { - off += 4; - continue; - } - break; /* trailing zero padding */ + off += 4; + continue; } op = (hdr >> 29) & 0x7; count = (hdr >> 16) & 0xfff; @@ -7664,16 +15700,27 @@ static bool svc_stream_parse(const u8 *buf, u32 len) return false; off += 4 + count * 4; break; + case 0: /* GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK */ + case 2: /* GRP2_USE_TERT */ case 4: /* IMMD_DATA_METHOD: header only */ - off += 4; - break; case 7: /* END_PB_SEGMENT: header only */ + /* + * r715: the class header defines op 0 as + * GRP0_USE_TERT (0) and op 2 as GRP2_USE_TERT - + * 4-byte header-only formats. The GR submit + * pushes emit 0x0001fff0 (SET_SUBDEVICE_MASK, + * value 0xfff) around their semaphore groups; the + * old parser rejected the first one on every big + * push, so the whole push read implausible and + * the entries were skipped without decode + * (session 124's consume-without-decode). + */ off += 4; break; default: return false; } - if (++methods > 1024) + if (++methods > 8192) return false; } while (off < len && buf[off] == 0) @@ -7693,6 +15740,88 @@ static bool svc_method_stream_plausible(const u8 *buf, u32 len) } /* + * r1780 (V36): FB-shadow content rescue for the direct-GL client's pushes. + * + * The GL client writes its pushbuffer through the BAR1 aperture, so the + * bytes land in the emulated FB shadow. Its VAS page directory is rebuilt + * every few seconds, so the push VA walk is dead exactly while the setup + * push (the 59556-byte shader/program upload) is being consumed. The + * linear-window guess is close but not exact (measured: push 0x4024004 read + * at 0x103187000 while the ring PA proved the allocation elsewhere), and + * the strict guest scanner cannot name the pages either. This rescue scans + * the shadow for a window of the demanded length that passes the same + * method-stream parse every trusted push passes, and pins the VA to the FB + * offset it found. Debug-gated (default off): a plausible window elsewhere + * in the shadow would execute the wrong methods. + */ +static bool gsp_sema_gr_fb_push; +module_param_named(sema_gr_fb_push, gsp_sema_gr_fb_push, bool, 0644); +MODULE_PARM_DESC(sema_gr_fb_push, + "Scan the FB shadow for an unreadable GR push of the same length and pin the VA to the found window (default 0)"); + +struct svc_fb_push_hit { + u32 len; + u32 n; + u64 off; +}; + +static bool svc_fb_push_rescue(struct nv_vgpu_device *vgpu, u64 va, u32 len, + u8 *out, u64 *off_out) +{ + static struct svc_fb_push_hit hits[8]; + u8 *base = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL), off; + static u32 log; + int i; + + if (!gsp_sema_gr_fb_push || !base || !len || len > 0x40000) + return false; + if (len > lim) + return false; + /* Reuse a previously proven window for this length first. */ + for (i = 0; i < ARRAY_SIZE(hits); i++) { + if (hits[i].n && hits[i].len == len && + hits[i].off + len <= lim && + svc_method_stream_plausible(base + hits[i].off, len)) { + memcpy(out, base + hits[i].off, len); + *off_out = hits[i].off; + return true; + } + } + /* Cheap first: the first 64 bytes must parse as a stream prefix. */ + for (off = 0; off + len <= lim; off += SEMA_PAGE_SIZE) { + u32 pre = min_t(u32, len, 64); + + if (!svc_method_stream_plausible(base + off, pre)) + continue; + if (!svc_method_stream_plausible(base + off, len)) + continue; + memcpy(out, base + off, len); + *off_out = off; + for (i = 0; i < ARRAY_SIZE(hits); i++) { + if (!hits[i].n || hits[i].len == len) { + hits[i].n = 1; + hits[i].len = len; + hits[i].off = off; + break; + } + } + if (log < 16) { + log++; + pr_info("sema-chan: FB-PUSH rescue h=0x%x va=0x%llx len=%u -> fb 0x%llx (plausible window; pinned for later reads)\n", + va, len, off); + } + return true; + } + if (log < 16) { + log++; + pr_info("sema-chan: FB-PUSH rescue h=0x%x va=0x%llx len=%u: no plausible window in 0x%llx shadow\n", + va, len, lim); + } + return false; +} + +/* * Decode one page as a method stream and execute the pushes it contains. * * Push buffers are written back to back and pushes can span page @@ -7718,6 +15847,8 @@ static u64 svc_rel_va_seen; * on every decoded release regardless of its target. */ static u32 svc_rel_count; +static u32 svc_fake_rel_log; /* r576b DIAG: sema-fake witness budget */ +static u32 svc_ce_foreign_log; /* r577: foreign-release refusal witness */ static u32 svc_dbg_streams; static u32 svc_chan_put_log; /* ISSUE-0002 (r55): per-rank put-snapshot throttle (ch is a local copy). */ @@ -7730,6 +15861,183 @@ static unsigned long svc_put_log_at[32]; */ static u32 svc_userd_dbg; static u32 svc_userd_dbg_ch[32]; +/* SESSION 134: per-CE-channel budget of the first-submission ring captures + * (see svc_ce_firstsub_capture) and the start of an unlatched CE channel's + * completion hold (see svc_ce_hold_ms). Session state; reset by + * svc_session_flush. */ +static u32 svc_ce_firstsub_n[32]; +static unsigned long svc_ce_hold_since[32]; +/* + * SESSION 134c: the FB offset each CE rank's first-submission capture latched + * (0 = none). The shadow scan is rank-blind on its own - it takes the first + * ring-shaped page in the shadow - and on r779 that latched rank 0's block + * (fb 0x80000, userd 0x90000) for rank 1 as well (both channels reported + * scan_hits=1 latch=1 off=0x80000). The rank-1 channel's first release was + * then decoded from rank 0's ring/page and stale-dropped (cur=1 already from + * rank 0's applied release), so the client's rank-1 tracker wait never + * completed and the client spun forever (victory_mail stall, three boots). + * The capture now prefers the rank's convention offset and refuses any + * offset another CE rank already latched. Session state; reset by + * svc_session_flush. + */ +static u64 svc_ce_firstsub_off[32]; +/* + * SESSION 144: FB ring-block ownership ACROSS engine classes. The GR + * autopin fallback scan (SESSION 133c) and the CE first-submission + * capture (SESSION 134c) each solved the same problem only within their + * own class; a live-but-foreign ring block passes every shape gate, so a + * channel whose own ring is not written yet latches another channel's + * block permanently and never reads its own entries. Observed on + * r805-r808: every CUDA GR rank's userd-shape scan latch resolved the CE + * rank-0 block (fb 0x80000, userd 0x90000; e.g. + * "auto-pinned GR ring h=0x5c000019 rank=0 ... -> fb 0x80000 (userd-shape + * scan ... entry0 va=0x120000000 len=40 put=35 get=35)" while the CE + * handle decoded the identical entry[35] raw words from pa=0x10350641c), + * so no GR launch was ever captured and victory_mail hung after DEVICE. + * A channel that cannot see its own block must stay unlatched (and retry + * next pass) rather than steal a foreign one. Keyed by the FB ring + * offset; claims are session state, reset by svc_session_flush. + */ +struct svc_fb_claim { + u32 h; /* owning channel handle */ + u64 ring_off; /* FB ring offset */ + u64 userd_off; /* FB USERD offset (0 = unknown) */ +}; +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL +static struct svc_fb_claim svc_fb_claim_tab[40]; +static u32 svc_fb_claim_n; + +static void svc_fb_claim_add(u32 h, u64 ring_off, u64 userd_off) +{ + u32 i; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == h) { + svc_fb_claim_tab[i].ring_off = ring_off; + svc_fb_claim_tab[i].userd_off = userd_off; + return; + } + } + if (svc_fb_claim_n < ARRAY_SIZE(svc_fb_claim_tab)) { + svc_fb_claim_tab[svc_fb_claim_n].h = h; + svc_fb_claim_tab[svc_fb_claim_n].ring_off = ring_off; + svc_fb_claim_tab[svc_fb_claim_n].userd_off = userd_off; + svc_fb_claim_n++; + } +} + +/* + * true = the candidate FB block (ring at ring_off, USERD at userd_off if + * known) is already claimed by a DIFFERENT channel, or is a live CE + * rank's convention block even before that rank latched, i.e. it must not + * be latched or stolen. Same-channel re-latches pass. + */ +static bool svc_fb_claim_foreign(struct nv_vgpu_device *vgpu, u32 self_h, + u64 ring_off, u64 userd_off) +{ + u32 i; + unsigned long flags; + int k; + + for (i = 0; i < svc_fb_claim_n; i++) { + if (svc_fb_claim_tab[i].h == self_h) + continue; + if (svc_fb_claim_tab[i].ring_off == ring_off) + return true; + if (userd_off && svc_fb_claim_tab[i].userd_off == userd_off) + return true; + } + /* + * SESSION 144b: reserve every live CE rank's convention block even + * before that rank has latched. r809: CE rank 1's block at fb + * 0xa0000 was still unclaimed when GR rank 0's userd-shape scan ran + * (rank 1 latched through the plain autopin path, not the + * first-submission capture) and the GR channel stole it again. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (k = 0; k < SVC_MAX_CHANNELS; k++) { + u64 conv; + + if (!svc.chan[k].valid || svc.chan[k].vgpu != vgpu || + !svc.chan[k].engine_type || + svc.chan[k].h_object == self_h || + svc.chan[k].ce_index >= 32) + continue; + conv = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * svc.chan[k].ce_index; + if (ring_off == conv || + (userd_off && userd_off == conv + 0x10000)) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + return true; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +static void svc_fb_claim_reset(void) +{ + svc_fb_claim_n = 0; + memset(svc_fb_claim_tab, 0, sizeof(svc_fb_claim_tab)); +} + +/* + * r1777 (V12): guest-RAM ring-page ownership, the RAM-side counterpart of + * the FB claim table above. A ring page found by the guest-RAM content scan + * (svc_gr_ram_ring_scan) or the low-aperture identity probe is recorded per + * channel so a sibling's scan cannot latch a page that is already another + * channel's ring: the direct-GL GR ring (0xbeef0100) and its CE siblings + * (0xbeef0101/0102) all queue low-aperture push entries, so shape alone + * cannot tell their ring pages apart (r1776: the CE sweep must not steal + * the GR page). One page per channel is enough - a channel's ring page is + * a single page. Reset per session by svc_session_flush like the FB table. + */ +static struct { + u32 h; + u64 pa; +} svc_ram_ring_owner[SVC_MAX_CHANNELS]; +static u32 svc_ram_owner_n; + +static void svc_ram_ring_owner_add(u32 h, u64 pa) +{ + u32 i; + + if (!pa) + return; + for (i = 0; i < svc_ram_owner_n; i++) { + if (svc_ram_ring_owner[i].h == h) { + svc_ram_ring_owner[i].pa = pa; + return; + } + } + if (svc_ram_owner_n < ARRAY_SIZE(svc_ram_ring_owner)) { + svc_ram_ring_owner[svc_ram_owner_n].h = h; + svc_ram_ring_owner[svc_ram_owner_n].pa = pa; + svc_ram_owner_n++; + } +} + +static void svc_ram_ring_owner_reset(void) +{ + svc_ram_owner_n = 0; + memset(svc_ram_ring_owner, 0, sizeof(svc_ram_ring_owner)); +} + +/* true = the page is already recorded as a DIFFERENT channel's RAM ring. */ +static bool svc_ram_ring_owner_foreign(u32 h, u64 pa) +{ + u64 page = pa & ~((u64)SEMA_PAGE_SIZE - 1); + u32 i; + + for (i = 0; i < svc_ram_owner_n; i++) + if (svc_ram_ring_owner[i].h != h && + (svc_ram_ring_owner[i].pa & ~((u64)SEMA_PAGE_SIZE - 1)) == + page) + return true; + return false; +} + /* * ISSUE-0002 (r47): deferred FB sweep for a CE channel whose layout USERD * probe fails at discovery. Session/reset-scoped, scheduled on the first @@ -7772,6 +16080,54 @@ struct svc_grce_rec { static struct svc_grce_rec svc_grce_dst_ring[SVC_GRCE_DST_MAX]; static u32 svc_grce_dst_ring_next; +/* + * SESSION 153-CONTINUED-88: newest captured inline-write payload landing + * exactly at `dst_va` with at least `min_len` bytes. Newest-first scan so a + * fresh upload always wins over a stale ring entry. Copies the payload into + * the caller's buffer (buf_sz bounds the copy) and returns true on a hit. + */ +static bool svc_grce_dst_newest(u64 dst_va, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz) +{ + u32 ri; + + for (ri = 0; ri < SVC_GRCE_DST_MAX; ri++) { + u32 idx = (svc_grce_dst_ring_next + SVC_GRCE_DST_MAX - 1u - + ri) % SVC_GRCE_DST_MAX; + const struct svc_grce_rec *r = &svc_grce_dst_ring[idx]; + + if (r->len < min_len || !r->dst_va) + continue; + if (r->dst_va != dst_va) + continue; + *len = min_t(u32, r->len, buf_sz); + memcpy(buf, r->data, *len); + return true; + } + return false; +} + +static bool svc_grce_dst_sph(u32 sph_w0, u32 min_len, u32 *len, u8 *buf, + u32 buf_sz) +{ + u32 ri; + + for (ri = 0; ri < SVC_GRCE_DST_MAX; ri++) { + u32 idx = (svc_grce_dst_ring_next + SVC_GRCE_DST_MAX - 1u - + ri) % SVC_GRCE_DST_MAX; + const struct svc_grce_rec *r = &svc_grce_dst_ring[idx]; + + if (r->len < min_len || !r->dst_va) + continue; + if (get_unaligned_le32(r->data) != sph_w0) + continue; + *len = min_t(u32, r->len, buf_sz); + memcpy(buf, r->data, *len); + return true; + } + return false; +} + static void svc_counter_learn(struct nv_vgpu_device *vgpu, u32 h, u64 sem_va); static void svc_pending_add(u64 va, u32 payload, u32 op); static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, @@ -8127,6 +16483,14 @@ static void svc_counter_flush_pending(struct nv_vgpu_device *vgpu, static struct { u64 va_page; u64 pa_page; + /* + * SESSION 153-CONTINUED-38: the VAS the page was derived for. The + * counter VA window (0x20400fXXX) is a fixed per-client layout, so + * two clients (the RC scrubber and the DDX) register the SAME VA and + * a VA-keyed cache/derive hands the second one the first one's page; + * that is the slot-ownership collision CONT-37 measured. 0 = any. + */ + u32 h_vas; } svc_cnt_der[SVC_CNT_DER_MAX]; static u32 svc_cnt_der_n; static u32 svc_cnt_der_log; @@ -8141,7 +16505,9 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) bool fresh = true; for (i = 0; i < svc_cnt_der_n; i++) { - if (svc_cnt_der[i].va_page == page) { + if (svc_cnt_der[i].va_page == page && + (!svc_cur_vas || !svc_cnt_der[i].h_vas || + svc_cnt_der[i].h_vas == svc_cur_vas)) { pa = svc_cnt_der[i].pa_page | off; fresh = false; break; @@ -8162,7 +16528,27 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) * it (r194: the counter page 0x20400f000 never resolved; its * walk hit probed unreadable and targeted was never reached). */ - if (!svc_walk_vaspaces(vgpu, page, &p, &a) && + /* + * SESSION 153-CONTINUED-38: the channel's OWN page tables + * answer first. The global walk sources can only name one + * page for the shared counter VA (0x20400fXXX is a fixed + * per-client layout), so a second client's slot derivation + * collided with the first client's page and its registration + * was rejected forever (CONT-37: "slot owned by another + * channel"). The per-channel walk (svc_cur_vas, set by the + * pump for the channel being serviced) names the page the + * client itself polls. + */ + if (!ok && svc_cur_vas && + !svc_chanvas_lookup(vgpu, page, &p, &a) && + a == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, + (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, + SVC_APERTURE_SYS, &probe, sizeof(probe))) { + ok = true; + src = "chanvas"; + } + if (!ok && !svc_walk_vaspaces(vgpu, page, &p, &a) && a == SVC_APERTURE_SYS && !svc_read_ap(vgpu, (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off, @@ -8208,6 +16594,7 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) svc_cnt_der[svc_cnt_der_n].va_page = page; svc_cnt_der[svc_cnt_der_n].pa_page = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_cnt_der[svc_cnt_der_n].h_vas = svc_cur_vas; svc_cnt_der_n++; } *pa_out = pa; @@ -8215,6 +16602,56 @@ static int svc_counter_derive(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out) } /* + * SESSION 153-CONTINUED-38: counter-slot disambiguation for a channel with + * no usable hVASpace (kernel channels register vas=0x0, so svc_cur_vas + * cannot answer). Two clients' channels can share a rank - the rank formula + * then names the SAME slot VA in each client's own address space - and the + * global walk sources answer with only one client's page, so the second + * client's autoreg collided with the first client's registered slot and was + * rejected forever (CONT-37: the GL client's beef0100 rank=2 vs the X + * client's bfef0002, both 0x2132acfd0). Walk EVERY registered VAS and + * return the first page that is readable AND different from avoid_pa: that + * is the second client's own page for the same VA. + */ +static int svc_counter_derive_not(struct nv_vgpu_device *vgpu, u64 va, + u64 avoid_pa, u64 *pa_out) +{ + unsigned long flags; + struct svc_vaspace local[SVC_MAX_VASPACES]; + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = va & (SEMA_PAGE_SIZE - 1); + int i, n = 0; + + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid) + local[n++] = svc.vas[i]; + spin_unlock_irqrestore(&svc.vas_lock, flags); + + for (i = 0; i < n; i++) { + u64 p = 0; + int a = SVC_APERTURE_SYS; + u32 probe; + + if (svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, + page, local[i].ap, &p, &a, false)) + continue; + if (a != SVC_APERTURE_SYS) + continue; + p = (p & ~(u64)(SEMA_PAGE_SIZE - 1)) | off; + if ((p & ~(u64)(SEMA_PAGE_SIZE - 1)) == + (avoid_pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + if (svc_read_ap(vgpu, p, SVC_APERTURE_SYS, &probe, + sizeof(probe))) + continue; + *pa_out = p; + return 0; + } + return -ENOENT; +} + +/* * Auto-register a GR channel's completion-counter slot from the observed * layout, without waiting for a readable push: the client zips its GR * channels in ascending handle order onto DESCENDING 16-byte slots, so the @@ -8329,6 +16766,47 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, break; } } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (collision) { + /* + * SESSION 153-CONTINUED-38: the slot VA is shared by + * rank across clients (each client's rank-r channel + * polls 0x20400fec0+(TOTAL-1-r)*0x10 in its OWN VAS), + * so a collision with another client's registered slot + * means the global derive answered with the OTHER + * client's page. Re-resolve through every registered + * VAS and keep the first readable page that differs - + * that page is this client's own slot. Only when no + * such page exists is the collision real and the + * registration must stay pending. + */ + u64 pa2 = 0; + + if (!svc_counter_derive_not(vgpu, va, pa, &pa2)) { + bool coll2 = false; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (j = 0; j < SVC_MAX_CHANNELS; j++) { + if (svc.chan[j].valid && + svc.chan[j].vgpu == vgpu && + svc.chan[j].h_object != + ch->h_object && + svc.chan[j].counter_pa == + (pa2 & ~0x3ULL)) { + coll2 = true; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!coll2) { + pr_info("sema-chan: counter auto h=0x%x rank=%u pa=0x%llx (own-VAS page; contender owner page 0x%llx kept)\n", + ch->h_object, ch->gr_index, + pa2 & ~0x3ULL, pa); + pa = pa2 & ~0x3ULL; + collision = false; + } + } + } if (!collision) { /* * Persist the registration into the authoritative @@ -8345,6 +16823,7 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, * hung. The authoritative store makes the * registration durable and the log one-shot. */ + spin_lock_irqsave(&svc.chan_lock, flags); ch->counter_pa = pa; for (j = 0; j < SVC_MAX_CHANNELS; j++) { if (svc.chan[j].valid && @@ -8354,8 +16833,8 @@ static void svc_counter_autoreg(struct nv_vgpu_device *vgpu, break; } } + spin_unlock_irqrestore(&svc.chan_lock, flags); } - spin_unlock_irqrestore(&svc.chan_lock, flags); if (collision) { pr_info_ratelimited("sema-chan: counter auto h=0x%x rank=%u pa=0x%llx rejected (slot owned by another channel; not installed)\n", ch->h_object, ch->gr_index, pa); @@ -8460,7 +16939,20 @@ static bool svc_rel_apply_906f(struct nv_vgpu_device *vgpu, op == HOST_SEM_OP_REDUCTION); return false; } - if (!svc_sem_read(vgpu, va, &cur) && payload > cur) { + if (svc_sem_read(vgpu, va, &cur)) { + /* + * r1656: a decoded release whose target page has no + * translation yet used to be dropped here with no write, no + * pending entry and no log - unlike svc_rel_apply_reportsem() + * and svc_rel_apply_launch(), which both queue the miss. Queue + * it so svc_apply_pending() retries once a pin or the client's + * VAS walk knows the page; a reduction stays additive. + */ + svc_pending_add(va, payload, + (op == HOST_SEM_OP_REDUCTION) ? 1u : 0u); + return false; + } + if (payload > cur) { svc.pushes_pending++; cur = (op == HOST_SEM_OP_REDUCTION) ? cur + payload : payload; if (!svc_sem_write(vgpu, ch->h_object, va, cur)) { @@ -8520,7 +17012,17 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, inc && inc_apply); return false; } - if (pending || inc_apply) { + /* + * r506w: generic absolute releases are forward-only by VALUE, whichever + * pass discovers the gap. The recorded pending bit came from the + * decode's read of the release word, and that read can be blinded by a + * stale/crossed landing (r506 witness: a withheld tracker release went + * to the flush with pending=0 while the fresh read still showed 0, and + * the release was then dropped by the counter fallback below - the + * client's tracker never advanced and its stream ground at the 10 s + * bound per push). Increments keep the one-shot claim gate (`inc_apply`). + */ + if (pending || inc_apply || (!inc && payload > cur)) { svc.pushes_pending++; if (inc) cur += payload; @@ -8542,10 +17044,20 @@ static bool svc_rel_apply_launch(struct nv_vgpu_device *vgpu, return true; } } - if (ch->engine_type == 0) - svc_counter_learn(vgpu, ch->h_object, sem_va); - svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, - inc && inc_apply); + /* + * r506w: the exact-slot counter fallback is for COUNTER VAs only. Its + * `release_pa ?: counter_pa` fallback writes (and forward-only-skips + * against) the channel's REGISTERED slot, which belongs to a different + * word - the r506 witness: releases to the tracker VA 0x20480fef0 were + * skipped against the polluted PUT counter (cur=96) and lost. For a + * non-counter VA the generic write above is the delivery path. + */ + if (svc_counter_va(sem_va)) { + if (ch->engine_type == 0) + svc_counter_learn(vgpu, ch->h_object, sem_va); + svc_counter_release_apply(vgpu, ch->h_object, sem_va, payload, + inc && inc_apply); + } /* * Stall-class fix (R2): when the decode said this absolute release is * genuinely pending and every write attempt above failed without @@ -8777,8 +17289,19 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } if (st->qmd_n > svc_gr_qmd_seen) svc_gr_qmd_seen = st->qmd_n; - if (st->qmd_have == 3) - st->qmd_dev_va = (((u64)st->qmd_hi << 32) | st->qmd_lo) << 8; + /* + * SESSION 127 C5C0 translator: the Vulkan driver writes + * SET_INLINE_QMD_ADDRESS_B only (the upper half stays absent for a + * <4G QMD; observed beec0100 idx23: `0x31c = 0x4b85f` then a 64-word + * 0x320 run). Accept the B-only form with an implicit zero upper + * half - the old A+B requirement left qmd_dev_va at 0 and the + * candidate logged with qmd=0x0 (the field is identity/logging + * only, but the 0x0 read confused every prior session's forensics). + */ + if (st->qmd_have & 2) + st->qmd_dev_va = + (((u64)((st->qmd_have & 1) ? st->qmd_hi : 0) << 32) | + st->qmd_lo) << 8; if (st->qmd_n == SVC_QMD_WORDS) { /* * Static: a 16-entry record array is ~1.2 KB and this is @@ -8800,15 +17323,85 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, svc_grce_dst_ring_next : SVC_GRCE_DST_MAX; if (have > GSP_BRIDGE_INLINE_MAX) have = GSP_BRIDGE_INLINE_MAX; - for (i = 0; i < have; i++) { - u32 idx = (svc_grce_dst_ring_next - 1 - i) % - SVC_GRCE_DST_MAX; - const struct svc_grce_rec *rec = &svc_grce_dst_ring[idx]; - - recs[n].dst_va = rec->dst_va; - recs[n].len = rec->len; - memcpy(recs[n].data, rec->data, rec->len); - n++; + /* + * SESSION 127 C5C0 translator: the rec ring is session-wide + * and mixes every client's LOAD_INLINE_DATA destinations. A + * stale rec whose page this QMD does not name made the fill + * refuse the whole launch (GR fill ENOENT + * va=0x80412000 qmd_word=NONE covered=1 - the Vulkan + * client's cancel page inherited from the driver-init + * phase). Keep only recs whose page is one of the pages the + * captured QMD names (the QMD buffer, the parameter bank, + * the program, the pointer pairs) - exactly the pages the + * backend's inline-served fill can use; keep everything when + * the QMD names none (fail-open). + */ + { + u64 keep[12]; + u32 nkeep = 0, k; + + if (st->qmd_dev_va) + keep[nkeep++] = st->qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[32] | + ((u64)(st->qmd[33] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + keep[nkeep++] = + ((u64)st->qmd[48] | + ((u64)(st->qmd[49] & 0x1ffff) << 32)) & + ~(u64)(SEMA_PAGE_SIZE - 1); + for (k = 0; k < SVC_GR_POINTER_MAX - 1 && nkeep < 12; k++) { + u32 w = 34 + 2 * k; + u64 pv = (u64)st->qmd[w] | + ((u64)(st->qmd[w + 1] & 0x1ffff) + << 32); + + if (pv) + keep[nkeep++] = pv & + ~(u64)(SEMA_PAGE_SIZE - 1); + } + for (i = 0; i < have; i++) { + u32 idx = (svc_grce_dst_ring_next - 1 - i) % + SVC_GRCE_DST_MAX; + const struct svc_grce_rec *rec = + &svc_grce_dst_ring[idx]; + u64 rpage = rec->dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1); + bool keepit = !nkeep; + u32 j; + + for (j = 0; j < nkeep; j++) + if (keep[j] == rpage) { + keepit = true; + break; + } + /* + * SESSION 127 (gated): a stale session-wide + * rec whose page resolves nowhere made the + * fill refuse the whole launch; keep it when + * it resolves (the CUDA inline-served + * pages), drop only unresolvable unnamed + * ones. With the translator off, the + * historical keep-all behaviour stands. + */ + if (!keepit && gsp_sema_gr_xlate) { + u64 rpa = 0; + u32 rap = SVC_APERTURE_SYS; + + if (svc_resolve_va(vgpu, rpage, &rpa, + &rap) && + svc_walk_vaspaces(vgpu, rpage, + &rpa, &rap)) + continue; + } else if (!keepit) { + continue; + } + recs[n].dst_va = rec->dst_va; + recs[n].len = rec->len; + memcpy(recs[n].data, rec->data, rec->len); + n++; + } } /* * C10 (GAP-2 A5): duplicate destinations in the newest-first @@ -8884,6 +17477,3702 @@ static void svc_qmd_accum(struct nv_vgpu_device *vgpu, struct svc_channel *ch, static u64 svc_pass_hash; /* r391f: this decode pass's push content hash */ +/* + * r481 RISK-REDUCTION diagnostic (default 0): dump the full decoded method + * stream of the first few large GR-class pushes (len >= 1024) so the launch + * push's acquire/release method groups (or their absence) are visible + * verbatim. The streams_events class evidence shows the st2 vadd's copy-in + * running ~115 ms BEFORE its stream's H2D soft copies; the ordering the + * driver relied on is only provable by reading the push's own methods. + * Bounded: gsp_sema_push_raw dumps, at most once per push VA. + * SESSION 153: gsp_sema_push_raw_small extends the same dump to the GL + * client's small pushes (16..511 B - the C597 draw/notify pushes are 32/64 B + * and were excluded by the >= 512 gate), with its own budget and a per + * (VA,len) once-only guard so a re-read of the same push shape does not + * flood. + */ +unsigned gsp_sema_push_raw; +unsigned gsp_sema_push_raw_small; +static u32 svc_mme_census_flag; +module_param_named(sema_push_raw_small, gsp_sema_push_raw_small, uint, 0644); +MODULE_PARM_DESC(sema_push_raw_small, + "Verbatim decode dump for small GR pushes (16..511 B), bounded count (default 0)"); +module_param_named(sema_mme_census, svc_mme_census_flag, uint, 0644); +MODULE_PARM_DESC(sema_mme_census, + "Census MME-family method ids (0x114..0x124, 0x234, 0x550..0x568, 0x3400..0x34fc, 0x3800..0x39fc) in beef pushes, bounded (default 0)"); +static u64 svc_pushraw_last_va; +static u32 svc_pushraw_last_len; +static u32 svc_pushraw_n; +static u32 svc_pushraw_small_n; +static u32 svc_pushraw_nz; + +/* + * SESSION 153-CONTINUED-52: MME-family method census for the beef (GL 3D + * client) channel. The replayed draw windows raise a real MME ILLEGAL_OPCODE + * (NV_PGRAPH_PRI_MME_HWW_ESR bit 2) - the MME executed a macro whose code it + * could not fetch. The macro calls (CALL_MME_MACRO/DATA 0x3800..0x39fc) carry + * macro code addresses, so the macro CONTEXT state - the MME Instruction/ + * Start-address RAM loads (0x114/0x118/0x11c/0x120), the shadow RAM control + * (0x124), MME version (0x234), the MME DMA window (0x550..0x568) and the + * shadow scratch (0x3400..0x34fc) - must have reached the ladder context too. + * If those setup methods never appear in ANY client push the MME can never + * have its macro library and the replay is doomed; if they do appear, the + * next question is which push forwards them. This census answers that: it + * scans every beef push for those method ids and logs a bounded one-line + * summary (first-hit offsets + first data word). Read-only; bounded. + */ +static u32 svc_mme_census_n; + +static void svc_mme_census(const u8 *buf, u32 len, u64 push_va, u32 h_object) +{ + u32 off = 0; + u32 nhit = 0; + char line[512]; + int lc = 0; + + if (!READ_ONCE(svc_mme_census_flag) || svc_mme_census_n >= 128) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4 <= len && nhit < 16) { + u32 hdr, op, cnt, addr; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if (op == 1u || op == 3u || op == 5u) + /* multi-word run: classify by its method id */ + ; + else if (op == 4u || op == 7u) + cnt = 0; + else + cnt = 0; + if ((addr >= 0x114u && addr <= 0x124u) || + addr == 0x234u || + (addr >= 0x550u && addr <= 0x568u) || + (addr >= 0x3400u && addr <= 0x34fcu) || + (addr >= 0x3800u && addr <= 0x39fcu)) { + u32 v0 = 0; + + if (off + 8 <= len) + memcpy(&v0, buf + off + 4, 4); + lc += scnprintf(line + lc, sizeof(line) - lc, + "%s0x%x@0x%x(%08x)", + nhit ? "," : "", addr, off, v0); + nhit++; + } + off += (op == 1u || op == 3u || op == 5u) && cnt ? + (1u + cnt) * 4u : 4u; + } + if (nhit) { + svc_mme_census_n++; + pr_info("sema-mme-census: h=0x%x va=0x%llx len=%u n=%u hits=[%s]\n", + h_object, push_va, len, nhit, line); + } +} + +/* + * SESSION 153-CONTINUED-101: MME fixed-constant-buffer scratch watch. + * + * mmeCodeOGL_C597_SETUP_FIXED_CONST (macro slot 0) is the only reader of + * MME shadow scratch 0x18/0xd18 (MME_SCRATCH_FIXED_CONSTANT_BUFFER = + * 0x10 + RM_MME_FIRST_USABLE_SHADOW_SCRATCH); its compiled C597 form DOES + * NOT include the bug2548370 branch, so it unconditionally emits + * SET_CONSTANT_BUFFER_SELECTOR_B/C = cbAddr[31:24]/[23:0]<<8 from that + * scratch. The value is written once per channel/context by the client's + * own gf100InitConstantBuffers: method 0x3460 (SET_MME_SHADOW_SCRATCH + * (0x18)) <- (u32)(fixed constant buffer VA >> 8). The replayed ladder + * context never receives that once-per-channel write, so the macro reads + * 0 and emits an empty CB selector; the draw then fetches constants from + * address 0. This watch records the client's own write (any channel, any + * push that decodes) and publishes the first value so the GR launch + * builder can seed scratch 0x18 ahead of the window (sema_gl_mme_seed). + * Value is stored exactly as pushed (client VA >> 8). Read-only. + */ +static u32 svc_scratch18_val; +static bool svc_scratch18_seen; +static u32 svc_scratch18_log; + +bool gsp_sema_mme_scratch18_get(u32 *val) +{ + if (svc_scratch18_seen && val) + *val = READ_ONCE(svc_scratch18_val); + return svc_scratch18_seen; +} + +/* + * SESSION 153-CONTINUED-115: companion capture for the bug2548370 slow-path + * selector. STATE 0xD19 / shadow scratch 0x19 (method 0x3464 = + * SET_MME_SHADOW_SCRATCH(0x19)) is the constant-buffer selector B/C + * (address>>8) the macro's in-band update emits from once the fast path is + * refused (d28 forced); when the replay context never receives the client's + * once-per-channel write the selector resolves to 0 and the consumer fetches + * constants from address 0 - the same failure mode documented for 0x18. The + * value is stored exactly as pushed; the GR launch builder can fall back to + * sema_mme_state_d19 when this never fires. + */ +static u32 svc_scratch19_val; +static bool svc_scratch19_seen; +static u32 svc_scratch19_log; + +bool gsp_sema_mme_scratch19_get(u32 *val) +{ + if (svc_scratch19_seen && val) + *val = READ_ONCE(svc_scratch19_val); + return svc_scratch19_seen; +} + +/* + * SESSION 153-CONTINUED-116: companion captures for the bug2548370 slow-path + * state family. STATE 0xD24 / shadow scratch 0x24 (method 0x3490 = + * SET_MME_SHADOW_SCRATCH(0x24)) is read by the macro body alongside the d19 + * selector; the replayed context never received the client's + * once-per-channel write, so the value resolves to whatever the replay left. + * The value is stored exactly as pushed (first write wins); the GR launch + * builder can fall back to sema_mme_state_d24 when this never fires. + */ +static u32 svc_scratch24_val; +static bool svc_scratch24_seen; +static u32 svc_scratch24_log; + +bool gsp_sema_mme_scratch24_get(u32 *val) +{ + if (svc_scratch24_seen && val) + *val = READ_ONCE(svc_scratch24_val); + return svc_scratch24_seen; +} + +/* + * SESSION 153-CONTINUED-116: companion of scratch 0x24 above - STATE 0xD25 / + * shadow scratch 0x25 (method 0x3494 = SET_MME_SHADOW_SCRATCH(0x25)), the + * next unseeded input on the same slow path. Same first-write-wins capture + * and builder fallback (sema_mme_state_d25). + */ +static u32 svc_scratch25_val; +static bool svc_scratch25_seen; +static u32 svc_scratch25_log; + +bool gsp_sema_mme_scratch25_get(u32 *val) +{ + if (svc_scratch25_seen && val) + *val = READ_ONCE(svc_scratch25_val); + return svc_scratch25_seen; +} + +static void svc_scratch18_watch(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0; + + if (svc_scratch18_seen && svc_scratch18_log >= 24u && + svc_scratch19_seen && svc_scratch19_log >= 24u && + svc_scratch24_seen && svc_scratch24_log >= 24u && + svc_scratch25_seen && svc_scratch25_log >= 24u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0xfffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt && + off + 4u + cnt * 4u <= len) { + if (addr == 0x3460u) { + u32 val; + + memcpy(&val, buf + off + 4, 4); + if (!svc_scratch18_seen) { + svc_scratch18_seen = true; + WRITE_ONCE(svc_scratch18_val, val); + } + if (svc_scratch18_log < 24u) { + svc_scratch18_log++; + pr_info("sema-scratch18: h=0x%x va=0x%llx off=0x%x op=%u cnt=%u val=0x%08x (SET_MME_SHADOW_SCRATCH(0x18); fixed-CB base>>8)\n", + h_object, push_va, off, op, cnt, + val); + } + off += 4u + cnt * 4u; + continue; + } + if (addr == 0x3464u) { + u32 val; + + memcpy(&val, buf + off + 4, 4); + if (!svc_scratch19_seen) { + svc_scratch19_seen = true; + WRITE_ONCE(svc_scratch19_val, val); + } + if (svc_scratch19_log < 24u) { + svc_scratch19_log++; + pr_info("sema-scratch19: h=0x%x va=0x%llx off=0x%x op=%u cnt=%u val=0x%08x (SET_MME_SHADOW_SCRATCH(0x19); CB selector B/C address>>8)\n", + h_object, push_va, off, op, cnt, + val); + } + off += 4u + cnt * 4u; + continue; + } + if (addr == 0x3490u) { + u32 val; + + memcpy(&val, buf + off + 4, 4); + if (!svc_scratch24_seen) { + svc_scratch24_seen = true; + WRITE_ONCE(svc_scratch24_val, val); + } + if (svc_scratch24_log < 24u) { + svc_scratch24_log++; + pr_info("sema-scratch24: h=0x%x va=0x%llx off=0x%x op=%u cnt=%u val=0x%08x (SET_MME_SHADOW_SCRATCH(0x24); slow-path state d24)\n", + h_object, push_va, off, op, cnt, + val); + } + off += 4u + cnt * 4u; + continue; + } + if (addr == 0x3494u) { + u32 val; + + memcpy(&val, buf + off + 4, 4); + if (!svc_scratch25_seen) { + svc_scratch25_seen = true; + WRITE_ONCE(svc_scratch25_val, val); + } + if (svc_scratch25_log < 24u) { + svc_scratch25_log++; + pr_info("sema-scratch25: h=0x%x va=0x%llx off=0x%x op=%u cnt=%u val=0x%08x (SET_MME_SHADOW_SCRATCH(0x25); slow-path state d25)\n", + h_object, push_va, off, op, cnt, + val); + } + off += 4u + cnt * 4u; + continue; + } + off += 4u + cnt * 4u; + continue; + } + if (op == 4u && addr == 0x3460u) { + u32 val = (hdr >> 16) & 0x1fffu; + + if (!svc_scratch18_seen) { + svc_scratch18_seen = true; + WRITE_ONCE(svc_scratch18_val, val); + } + if (svc_scratch18_log < 24u) { + svc_scratch18_log++; + pr_info("sema-scratch18: h=0x%x va=0x%llx off=0x%x op=4 immd val=0x%08x (SET_MME_SHADOW_SCRATCH(0x18))\n", + h_object, push_va, off, val); + } + } + if (op == 4u && addr == 0x3464u) { + u32 val = (hdr >> 16) & 0x1fffu; + + if (!svc_scratch19_seen) { + svc_scratch19_seen = true; + WRITE_ONCE(svc_scratch19_val, val); + } + if (svc_scratch19_log < 24u) { + svc_scratch19_log++; + pr_info("sema-scratch19: h=0x%x va=0x%llx off=0x%x op=4 immd val=0x%08x (SET_MME_SHADOW_SCRATCH(0x19); CB selector B/C address>>8)\n", + h_object, push_va, off, val); + } + } + if (op == 4u && addr == 0x3490u) { + u32 val = (hdr >> 16) & 0x1fffu; + + if (!svc_scratch24_seen) { + svc_scratch24_seen = true; + WRITE_ONCE(svc_scratch24_val, val); + } + if (svc_scratch24_log < 24u) { + svc_scratch24_log++; + pr_info("sema-scratch24: h=0x%x va=0x%llx off=0x%x op=4 immd val=0x%08x (SET_MME_SHADOW_SCRATCH(0x24))\n", + h_object, push_va, off, val); + } + } + if (op == 4u && addr == 0x3494u) { + u32 val = (hdr >> 16) & 0x1fffu; + + if (!svc_scratch25_seen) { + svc_scratch25_seen = true; + WRITE_ONCE(svc_scratch25_val, val); + } + if (svc_scratch25_log < 24u) { + svc_scratch25_log++; + pr_info("sema-scratch25: h=0x%x va=0x%llx off=0x%x op=4 immd val=0x%08x (SET_MME_SHADOW_SCRATCH(0x25))\n", + h_object, push_va, off, val); + } + } + off += 4u; + } +} + +/* + * r1475: copy the client's fixed-CB page into the forced MME seed page. + * + * The VS reads its transform constants from the fixed constant buffer whose + * base the client publishes through SET_MME_SHADOW_SCRATCH(0x18) (method + * 0x3460) as (VA >> 8). The GR launch builder forces the MME seed to a + * mapped page (sema_gl_mme_seed_val) so the engine always has a base, but + * that page holds screen/stale data, not the client's uniforms. Copy the + * client's real fixed-CB page into the forced seed page at decode time, + * before the launch, so the VS reads the client's constants; the MME's + * later constant writes (offsets 0x20/0x30/0x1480/0x1e0/0xc8) overwrite + * only their own offsets and the VS's 0x80.. region survives. + * + * Trigger: once per decode pass for a beef GL channel, gated by a content + * hash of the source page. The 0x3460 write itself is a once-per-channel + * event, so keying the copy on that write alone would copy stale init-time + * constants forever; the client rewrites the CB page every frame, and the + * hash gate picks that up while skipping an unchanged page (and any + * re-decode of the same push). The copy is idempotent and runs before + * svc_gl_flush's launch in the same pass. + */ +static u32 svc_gl_cb_copy_h; +static u32 svc_gl_cb_copy_val; +static u32 svc_gl_cb_copy_hash; +static u32 svc_gl_cb_copy_log; +static u32 svc_gl_cb_seed_log; +static u32 svc_gl_cover_n; +/* + * r1512: provenance of the page most recently written into the forced MME + * seed (what the VS's c[0x0] bank will contain). Read by the coverage + * receipt so a bad transform can be attributed to the copy source. + */ +static u64 svc_gl_cb_last_va; +static u64 svc_gl_cb_last_pa; +static s32 svc_gl_cb_last_ssrc; +/* + * r1494: the last fixed-CB page that resolved through a SYS (engine-page) + * source and passed the matrix gate. The client VA can move/reuse after + * the first frame (r1492 read back a method page with the same VA and + * r1493 fell back to identity), so a later resolution miss must not zero + * the seed page: reuse this page for the rest of the boot. Only a page + * that was never credibly resolved falls back to the synthetic identity. + */ +static u8 svc_gl_cb_good[SEMA_PAGE_SIZE]; +static bool svc_gl_cb_good_valid; + +static u32 svc_gl_cb_hash(const u8 *p, u32 n) +{ + u32 h = 2166136261u; + u32 i; + + for (i = 0; i < n; i++) { + h ^= p[i]; + h *= 16777619u; + } + return h; +} + +/* + * r1491: is the position block (c[0x0][0x80..0xbc], the 4x4 matrix the + * captured VS uses to turn the vertex into clip space) usable? A NaN/Inf + * word or an all-zero 16-word block means the source page is not the + * engine-written fixed constant buffer (or the matrix was never filled), + * and the draw would collapse every triangle regardless of the vertex + * fetch. Finite and not all-zero is the minimum credible shape. + */ +static bool svc_gl_cb_matrix_ok(const u8 *buf) +{ + u32 i, nz = 0, normal = 0; + + for (i = 0; i < 16u; i++) { + u32 w = get_unaligned_le32(buf + 0x80u + i * 4u); + u32 exp = (w >> 23) & 0xffu; + + if (exp == 0xffu) + return false; + if (!(w & 0x7fffffffu)) + continue; + nz++; + /* + * r1492: a method-stream page mis-resolved for the CB + * answers with headers/payload words whose "floats" are + * zero/denormal/tiny (0x2001xxxx -> exp 0x40). A real + * transform row has normal-scale components. Requiring four + * normal-magnitude words rejects the method pages that made + * r1492 accept a garbage matrix (`00000053 80000585 + * 20010d2a ... ok=1`). + */ + if (exp >= 0x70u && exp <= 0x8eu) + normal++; + } + return nz != 0u && normal >= 4u; +} + +/* + * r1513: strict projection gate. svc_gl_cb_matrix_ok only asks for finite, + * normal-magnitude words, and r1512 showed it accepting a page whose + * c[0x0][0x80..0xbc] block was a rotation/normal block (bottom row + * 0.1268 0.9922 0 1) - the VS then produced w~1 instead of the camera + * distance and every sampled vertex clipped. A transform used as + * gl_Position must leave X/Y independent of the eye-space Z/W row, i.e. + * row3.x and row3.y are zero for both the perspective (0,0,-1,0) and the + * orthographic (0,0,0,1) form, and at least one of row3.z/row3.w is + * non-zero. Keep the finite/normal checks and add that shape test. + */ +static bool svc_gl_cb_mvp_ok(const u8 *buf) +{ + u32 m12 = get_unaligned_le32(buf + 0x80u + 12u * 4u); + u32 m13 = get_unaligned_le32(buf + 0x80u + 13u * 4u); + u32 m14 = get_unaligned_le32(buf + 0x80u + 14u * 4u); + u32 m15 = get_unaligned_le32(buf + 0x80u + 15u * 4u); + u32 a12 = m12 & 0x7fffffffu, a13 = m13 & 0x7fffffffu; + + /* |row3.x|, |row3.y| < 0.02 (0x3ca3d70a); denormals and zero pass. */ + if (a12 > 0x3ca3d70au || a13 > 0x3ca3d70au) + return false; + if (!(m14 & 0x7fffffffu) && !(m15 & 0x7fffffffu)) + return false; + return svc_gl_cb_matrix_ok(buf); +} + +static void svc_gl_cb_dump_matrix(const u8 *buf, const char *src, u64 pa, + u32 ap, int ssrc) +{ + u32 i, w[16]; + + for (i = 0; i < 16u; i++) + w[i] = get_unaligned_le32(buf + 0x80u + i * 4u); + pr_info("vgsp-sema: sema-gl: cb-matrix src=%s pa=0x%llx ap=%u ssrc=%d m[0x80..0xbc]=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x ok=%u\n", + src, pa, ap, ssrc, w[0], w[1], w[2], w[3], w[4], w[5], + w[6], w[7], w[8], w[9], w[10], w[11], w[12], w[13], w[14], + w[15], svc_gl_cb_matrix_ok(buf)); + pr_info("vgsp-sema: sema-gl: cb-const src=%s c[0x2a0..0x2a8]=%08x %08x %08x c[0x780..0x7c8]=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x c[0x820..0x828]=%08x %08x %08x c[0x850..0x858]=%08x %08x %08x c[0x870..0x888]=%08x %08x %08x %08x %08x %08x %08x\n", + src, + get_unaligned_le32(buf + 0x2a0), get_unaligned_le32(buf + 0x2a4), + get_unaligned_le32(buf + 0x2a8), + get_unaligned_le32(buf + 0x780), get_unaligned_le32(buf + 0x784), + get_unaligned_le32(buf + 0x788), get_unaligned_le32(buf + 0x78c), + get_unaligned_le32(buf + 0x790), get_unaligned_le32(buf + 0x794), + get_unaligned_le32(buf + 0x798), get_unaligned_le32(buf + 0x79c), + get_unaligned_le32(buf + 0x7a0), get_unaligned_le32(buf + 0x7a4), + get_unaligned_le32(buf + 0x7a8), get_unaligned_le32(buf + 0x7c0), + get_unaligned_le32(buf + 0x7c4), get_unaligned_le32(buf + 0x7c8), + get_unaligned_le32(buf + 0x820), get_unaligned_le32(buf + 0x824), + get_unaligned_le32(buf + 0x828), + get_unaligned_le32(buf + 0x850), get_unaligned_le32(buf + 0x854), + get_unaligned_le32(buf + 0x858), + get_unaligned_le32(buf + 0x870), get_unaligned_le32(buf + 0x874), + get_unaligned_le32(buf + 0x878), + get_unaligned_le32(buf + 0x880), get_unaligned_le32(buf + 0x884), + get_unaligned_le32(buf + 0x888)); +} + +/* + * r1515: client fixed-CB upload capture. + * + * The client's fixed constant buffer is not a page the guest writes with a + * plain memory store: the driver publishes its base once per context + * (SET_MME_SHADOW_SCRATCH(0x18), method 0x3460, value = VA >> 8) and then + * streams the constants through the C597 constant-buffer methods: + * - 0x2380/0x2384/0x2388 SET_CONSTANT_BUFFER_SELECTOR_A/B/C + * (A size, B address[39:32], C address[31:0]); + * - 0x238c LOAD_CONSTANT_BUFFER_OFFSET (byte offset into the store); + * - 0x2390 LOAD_CONSTANT_BUFFER(i) data runs (NON_INC streams the + * constants from the current offset; INC/ONE_INC index them). + * r1512/r1514 measured the consequence of not applying them: the page the + * VS's c[0x0] bank resolves to holds screen/rotation data, the position + * block at 0x80 is not a projection, and 57/64 sampled gear vertices clip. + * + * This scanner runs on every decoded push of the beef GL channel (the + * 60216-byte setup push carries the 0x3460 write at off 0x464c and the + * constant stream after it). It maintains a 4 KB image of the first CB + * page (the VS reads c[0x0][0x80..0x888], all inside page 0), validates it + * with the strict projection gate, and only then marks it authoritative. + * svc_gl_cb_copy() prefers this image over any memory-page discovery and + * writes it into the forced MME seed page, so the replayed VS reads the + * client's real transform. The write is a pure emulated engine store (the + * client page is authored through the same inline-write path when it + * resolves). No FP in the kernel. + */ +static u8 svc_gl_cb_img[SEMA_PAGE_SIZE]; +static bool svc_gl_cb_img_valid; +static u64 svc_gl_cb_img_base; +static u64 svc_gl_cb_img_words; +static u64 svc_gl_cb_img_runs; +static u32 svc_gl_cb_off; +static u32 svc_gl_cb_sel_a; +static u64 svc_gl_cb_sel_bc; +static bool svc_gl_cb_sel_seen; +static u32 svc_gl_cb_scan_log; +static u32 svc_gl_cb_apply_log; +/* CB-DIAG (2026-11-10): bounded per-push capture summary. */ +static u32 svc_gl_cb_diag_log; +#define SVC_GL_CB_DIAG_MAX 48u + +#define SVC_GL_CB_IMG_LIMIT SEMA_PAGE_SIZE + +static void svc_gl_cb_img_reset(u64 base) +{ + memset(svc_gl_cb_img, 0, sizeof(svc_gl_cb_img)); + svc_gl_cb_img_valid = false; + svc_gl_cb_img_base = base; + svc_gl_cb_img_words = 0; + svc_gl_cb_img_runs = 0; + svc_gl_cb_off = 0; +} + +static void svc_gl_cb_upload_scan(struct nv_vgpu_device *vgpu, u32 h_object, + const u8 *buf, u32 len, u64 push_va) +{ + u32 off = 0, runs = 0, words = 0; + u64 base = 0; + bool dirty = false; + u32 i; + /* CB-DIAG counters */ + u32 nhdr = 0, nover = 0, nmatch = 0; + u32 n3800 = 0, n39c8 = 0, n39d0 = 0, n2380 = 0; + u32 n238c = 0, n2390 = 0, n3460 = 0; + u32 b3800_first = 0xffffffffu; + + (void)vgpu; + /* + * CB-DIAG (2026-11-10): the draws that actually replay ride the + * 0x01xxxxxx RM-channel family (e.g. 0x1000014), not only the beef + * GL channel; widen the scan so the fixed-CB uploads on the draw + * channel are captured too. + */ + if (((h_object & 0xfffff000u) != 0xbeef0000u && + (h_object & 0xff000000u) != 0x01000000u) || !buf || len < 16u) + return; + /* + * A new SET_MME_SHADOW_SCRATCH(0x18) names a new fixed-CB base: + * discard the previous image so a stale transform can never be + * applied to a new context. + */ + if (svc_scratch18_seen && READ_ONCE(svc_scratch18_val)) + base = (u64)READ_ONCE(svc_scratch18_val) << 8; + if (base && base != svc_gl_cb_img_base) + svc_gl_cb_img_reset(base); + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, w; + + memcpy(&hdr, buf + off, 4); + nhdr++; + if (!hdr) { + off += 4u; + continue; + } + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if (op == 0u || op == 2u || op == 7u) { + off += 4u; + continue; + } + if (op == 4u) { + /* IMMD: 13-bit payload; offset/selector only. */ + w = (hdr >> 16) & 0x1fffu; + if (addr == 0x3800u) { + n3800++; + if (b3800_first == 0xffffffffu) + b3800_first = w << 2; + } else if (addr == 0x238cu) { + n238c++; + } else if (addr == 0x2384u || addr == 0x2388u) { + n2380++; + } else if (addr == 0x3460u) { + n3460++; + } + if (addr == 0x3460u) { + base = (u64)w << 8; + if (base && base != svc_gl_cb_img_base) + svc_gl_cb_img_reset(base); + } else if (addr == 0x238cu) { + svc_gl_cb_off = w; + dirty = true; + } else if (addr == 0x2384u) { + svc_gl_cb_sel_bc = (svc_gl_cb_sel_bc & + ~0xffffffffULL) | + (u64)w; + svc_gl_cb_sel_seen = true; + } else if (addr == 0x2388u) { + svc_gl_cb_sel_bc = (svc_gl_cb_sel_bc & + 0xffffffffULL) | + ((u64)w << 32); + svc_gl_cb_sel_seen = true; + } else if (addr == 0x3800u) { + /* + * MME-wrapper form (570.144): the driver emits + * CALL_MME_MACRO(0) = SETUP_FIXED_CONST as an + * IMMD whose payload is the VPC dword offset + * (VPC_CONSTANT_DWORD(offs) = offs << 2); the + * macro converts it to the byte address the + * following LOAD_CONSTANT_BUFFER run writes at. + */ + svc_gl_cb_off = w << 2; + dirty = true; + } + off += 4u; + continue; + } + if (op != 1u && op != 3u && op != 5u) { + off += 4u; + continue; + } + if (!cnt || cnt > 0xfffu || + off + 4u + cnt * 4u > len) { + /* + * A run that overruns this push (or a data word that + * happens to decode as a header) is not the end of the + * stream: the client's setup push interleaves large + * inline payloads with method runs. Resync one word + * later, exactly like svc_scratch18_watch does, so the + * constant stream after the data block is still found. + */ + nover++; + off += 4u; + continue; + } + nmatch++; + if (addr == 0x238cu) + n238c++; + else if (addr >= 0x2380u && addr <= 0x238cu) + n2380++; + else if (addr == 0x3460u) + n3460++; + else if (addr == 0x39c8u) + n39c8++; + else if (addr == 0x39d0u) + n39d0++; + else if (addr >= 0x2390u && addr < 0x2390u + 0x1000u) + n2390++; + if (addr == 0x238cu) { + memcpy(&w, buf + off + 4u, 4); + svc_gl_cb_off = w; + dirty = true; + off += 4u + cnt * 4u; + continue; + } + if (addr >= 0x2380u && addr <= 0x238cu) { + u32 j; + + for (j = 0; j < cnt; j++) { + u32 m = (op == 3u) ? addr : addr + 4u * j; + + memcpy(&w, buf + off + 4u + 4u * j, 4); + switch (m) { + case 0x2380u: + svc_gl_cb_sel_a = w; + break; + case 0x2384u: + svc_gl_cb_sel_bc = + (svc_gl_cb_sel_bc & + ~0xffffffffULL) | + (u64)w; + svc_gl_cb_sel_seen = true; + break; + case 0x2388u: + svc_gl_cb_sel_bc = + (svc_gl_cb_sel_bc & + 0xffffffffULL) | + ((u64)w << 32); + svc_gl_cb_sel_seen = true; + break; + case 0x238cu: + svc_gl_cb_off = w; + dirty = true; + break; + default: + break; + } + } + } else if (addr == 0x3460u) { + memcpy(&w, buf + off + 4u, 4); + base = (u64)w << 8; + if (base && base != svc_gl_cb_img_base) + svc_gl_cb_img_reset(base); + } else if (addr == 0x39c8u) { + /* + * MME-wrapper form (570.144): the driver emits + * CALL_MME_MACRO(57) = + * mmeCodeOGL_C597_LOAD_CONSTANT_BUFFER_BUG2548370 + * with the argument stream inline in the run: + * [count][count dwords]. The macro writes the data + * words at the current fixed-CB offset and + * auto-increments, exactly like the native + * LOAD_CONSTANT_BUFFER(i) form. + */ + u32 n; + + memcpy(&n, buf + off + 4u, 4); + if (n > cnt - 1u) + n = cnt - 1u; + for (i = 0; i < n; i++) { + u64 doff = (u64)svc_gl_cb_off + + 4ULL * (u64)i; + + if (doff + 4ULL > SVC_GL_CB_IMG_LIMIT) + continue; + memcpy(&w, buf + off + 8u + 4u * i, 4); + memcpy(svc_gl_cb_img + doff, &w, 4); + words++; + dirty = true; + } + runs++; + svc_gl_cb_off += 4u * n; + } else if (addr == 0x39d0u) { + /* + * CALL_MME_MACRO(58) = + * mmeCodeOGL_C597_LOAD_CONSTANT_BUFFER_OFFSET: + * argument stream is [offsetBytes]. + */ + memcpy(&w, buf + off + 4u, 4); + svc_gl_cb_off = w; + dirty = true; + } else if (addr >= 0x2390u && addr < 0x2390u + 0x1000u) { + u32 idx = (addr - 0x2390u) / 4u; + + for (i = 0; i < cnt; i++) { + u64 doff; + + memcpy(&w, buf + off + 4u + 4u * i, 4); + /* + * NON_INC streams from the published offset; + * INC/ONE_INC index from the run's method. + */ + if (op == 3u) + doff = (u64)svc_gl_cb_off + + 4ULL * (u64)i; + else + doff = (u64)svc_gl_cb_off + + 4ULL * (u64)(idx + i); + if (doff + 4ULL > SVC_GL_CB_IMG_LIMIT) + continue; + memcpy(svc_gl_cb_img + doff, &w, 4); + words++; + dirty = true; + } + runs++; + if (op == 3u) + svc_gl_cb_off += 4u * cnt; + } + off += 4u + cnt * 4u; + } + /* + * CB-DIAG (2026-11-10): one bounded line per beef push so a live run + * shows whether the MVP upload reaches the image at byte 0x80. If + * n3800>0 but b3800_first>=0x1000, the client uploads its transform + * at a CB offset this 4 KB image cannot hold; if n39c8/n2390==0 the + * upload-bearing push was never decoded; if m80 is non-zero but + * mvp=0 the captured block is not the projection. + */ + if (svc_gl_cb_diag_log < SVC_GL_CB_DIAG_MAX) { + svc_gl_cb_diag_log++; + pr_info("vgsp-sema: sema-gl: CB-DIAG h=0x%x push=0x%llx len=%u hdr=%u run=%u over=%u match=%u[3800=%u 39c8=%u 39d0=%u 2380=%u 238c=%u 2390=%u 3460=%u] b3800first=0x%x words=%u off=0x%x base=0x%llx valid=%u mvp=%u m80=%08x %08x %08x %08x row3=%08x %08x %08x %08x\n", + h_object, push_va, len, nhdr, runs, nover, nmatch, + n3800, n39c8, n39d0, n2380, n238c, n2390, n3460, + b3800_first, words, svc_gl_cb_off, svc_gl_cb_img_base, + svc_gl_cb_img_valid ? 1u : 0u, + svc_gl_cb_mvp_ok(svc_gl_cb_img) ? 1u : 0u, + get_unaligned_le32(svc_gl_cb_img + 0x80), + get_unaligned_le32(svc_gl_cb_img + 0x84), + get_unaligned_le32(svc_gl_cb_img + 0x88), + get_unaligned_le32(svc_gl_cb_img + 0x8c), + get_unaligned_le32(svc_gl_cb_img + 0xb0), + get_unaligned_le32(svc_gl_cb_img + 0xb4), + get_unaligned_le32(svc_gl_cb_img + 0xb8), + get_unaligned_le32(svc_gl_cb_img + 0xbc)); + { + u32 nz = 0, q; + + for (q = 0; q < 0x100u; q++) + if (svc_gl_cb_img[q]) + nz++; + pr_info("vgsp-sema: sema-gl: CB-DIAG2 h=0x%x img[00..1c]=%08x %08x %08x %08x %08x %08x %08x %08x img[90..ac]=%08x %08x %08x %08x %08x %08x %08x %08x nz=0x%x\n", + h_object, + get_unaligned_le32(svc_gl_cb_img + 0x00), + get_unaligned_le32(svc_gl_cb_img + 0x04), + get_unaligned_le32(svc_gl_cb_img + 0x08), + get_unaligned_le32(svc_gl_cb_img + 0x0c), + get_unaligned_le32(svc_gl_cb_img + 0x10), + get_unaligned_le32(svc_gl_cb_img + 0x14), + get_unaligned_le32(svc_gl_cb_img + 0x18), + get_unaligned_le32(svc_gl_cb_img + 0x1c), + get_unaligned_le32(svc_gl_cb_img + 0x90), + get_unaligned_le32(svc_gl_cb_img + 0x94), + get_unaligned_le32(svc_gl_cb_img + 0x98), + get_unaligned_le32(svc_gl_cb_img + 0x9c), + get_unaligned_le32(svc_gl_cb_img + 0xa0), + get_unaligned_le32(svc_gl_cb_img + 0xa4), + get_unaligned_le32(svc_gl_cb_img + 0xa8), + get_unaligned_le32(svc_gl_cb_img + 0xac), + nz); + } + } + if (!dirty || !words) + return; + svc_gl_cb_img_words += words; + svc_gl_cb_img_runs += runs; + /* + * The strict projection gate is the validator: only a page whose + * c[0x0][0x80..0xbc] block is a projection transform becomes the + * authoritative CB source. A partial stream keeps accumulating. + */ + if (!svc_gl_cb_img_valid && svc_gl_cb_mvp_ok(svc_gl_cb_img)) { + svc_gl_cb_img_valid = true; + /* r1523: fresh draw-coverage receipts for the real-CB draws. */ + svc_gl_cover_n = 0; + pr_info("vgsp-sema: sema-gl: CB-UPLOAD h=0x%x push=0x%llx base=0x%llx sel=%u a=0x%08x bc=0x%llx off=0x%x runs=%llu words=%llu projection=ok first8=%08x %08x %08x %08x %08x %08x %08x %08x\n", + h_object, push_va, svc_gl_cb_img_base, + svc_gl_cb_sel_seen ? 1u : 0u, svc_gl_cb_sel_a, + svc_gl_cb_sel_bc, svc_gl_cb_off, + svc_gl_cb_img_runs, svc_gl_cb_img_words, + get_unaligned_le32(svc_gl_cb_img), + get_unaligned_le32(svc_gl_cb_img + 4), + get_unaligned_le32(svc_gl_cb_img + 8), + get_unaligned_le32(svc_gl_cb_img + 12), + get_unaligned_le32(svc_gl_cb_img + 16), + get_unaligned_le32(svc_gl_cb_img + 20), + get_unaligned_le32(svc_gl_cb_img + 24), + get_unaligned_le32(svc_gl_cb_img + 28)); + pr_info("vgsp-sema: sema-gl: CB-UPLOAD m80=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + get_unaligned_le32(svc_gl_cb_img + 0x80), + get_unaligned_le32(svc_gl_cb_img + 0x84), + get_unaligned_le32(svc_gl_cb_img + 0x88), + get_unaligned_le32(svc_gl_cb_img + 0x8c), + get_unaligned_le32(svc_gl_cb_img + 0x90), + get_unaligned_le32(svc_gl_cb_img + 0x94), + get_unaligned_le32(svc_gl_cb_img + 0x98), + get_unaligned_le32(svc_gl_cb_img + 0x9c), + get_unaligned_le32(svc_gl_cb_img + 0xa0), + get_unaligned_le32(svc_gl_cb_img + 0xa4), + get_unaligned_le32(svc_gl_cb_img + 0xa8), + get_unaligned_le32(svc_gl_cb_img + 0xac), + get_unaligned_le32(svc_gl_cb_img + 0xb0), + get_unaligned_le32(svc_gl_cb_img + 0xb4), + get_unaligned_le32(svc_gl_cb_img + 0xb8), + get_unaligned_le32(svc_gl_cb_img + 0xbc)); + } + if (svc_gl_cb_scan_log < 24u) { + svc_gl_cb_scan_log++; + pr_info("vgsp-sema: sema-gl: CB-UPLOAD-SCAN h=0x%x push=0x%llx base=0x%llx off=0x%x runs=%u words=%u valid=%u total_words=%llu\n", + h_object, push_va, svc_gl_cb_img_base, svc_gl_cb_off, + runs, words, svc_gl_cb_img_valid ? 1u : 0u, + svc_gl_cb_img_words); + } + if (svc_gl_cb_img_valid && sema_gl_cb_copy_to && + svc_gl_cb_apply_log < 24u) { + svc_gl_cb_apply_log++; + pr_info("vgsp-sema: sema-gl: CB-UPLOAD-APPLY h=0x%x base=0x%llx -> seed=0x%llx first8=%08x %08x %08x %08x %08x %08x %08x %08x (client CB image written by cb-copy)\n", + h_object, svc_gl_cb_img_base, sema_gl_cb_copy_to, + get_unaligned_le32(svc_gl_cb_img), + get_unaligned_le32(svc_gl_cb_img + 4), + get_unaligned_le32(svc_gl_cb_img + 8), + get_unaligned_le32(svc_gl_cb_img + 12), + get_unaligned_le32(svc_gl_cb_img + 16), + get_unaligned_le32(svc_gl_cb_img + 20), + get_unaligned_le32(svc_gl_cb_img + 24), + get_unaligned_le32(svc_gl_cb_img + 28)); + } +} + +/* + * r1505 (mission task 3): content-based fixed-CB discovery. The client's + * fixed-CB base VA is boot-dependent and the scratch18/0x0cd70000 chain can + * miss on a boot (r1504: 96/96 synthetic-id). The CB page is 4 KB whose + * c[0x0][0x80..0xbc] block holds the real transform rows, so read a candidate + * VA through the emitting channel's VAS first and accept only a page that + * passes svc_gl_cb_matrix_ok. Read-only, bounded, and called only after the + * whole historical chain failed. + */ +static bool svc_gl_cb_read_page(struct nv_vgpu_device *vgpu, u64 va, u8 *buf, + u64 *pa_out, u32 *ap_out) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0; + u32 ap = SVC_APERTURE_SYS; + bool resolved = false; + + if (!page) + return false; + if (!svc_chanvas_lookup(vgpu, page, &pa, &ap)) + resolved = true; + else if (!svc_walk_vaspaces(vgpu, page, &pa, &ap)) + resolved = true; + /* + * r1514: a failed resolution leaves pa at 0. Reading guest physical + * page 0 then succeeded and the zero/IVT page was accepted as the + * fixed CB (r1513: cbsrc=0x43e0000 pa=0x0 with a stale/synthetic + * identity). Only a real mapping may answer. + */ + if (!resolved || !pa) + return false; + pa &= ~(u64)(SEMA_PAGE_SIZE - 1); + if (svc_ap_of(ap) == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, buf, SEMA_PAGE_SIZE)) { + if (pa_out) + *pa_out = pa; + if (ap_out) + *ap_out = svc_ap_of(ap); + return true; + } + return false; +} + +static void svc_gl_cb_copy(struct nv_vgpu_device *vgpu, u32 h_object) +{ + /* Static: 4 KB is too much for the kernel stack; this runs on the + * single service thread (same reason as the static arrays in + * svc_gl_flush and svc_decode_stream). */ + static u8 buf[SEMA_PAGE_SIZE]; + u64 src_va = 0, pa = 0, hpa = 0; + u32 ap = 0, val, hash; + const char *src = "resolve"; + int rc, ssrc = 0; + + if (!sema_gl_cb_copy_to) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + /* + * r1515: the client's own constant-buffer upload is the highest + * authority. Once svc_gl_cb_upload_scan() has assembled the first CB + * page from the setup push's 0x2380/0x238c/0x2390 runs and the + * strict projection gate accepted it, deliver that image directly - + * no memory-page discovery can be more faithful than the bytes the + * client itself streamed into the store. + */ + if (svc_gl_cb_img_valid) { + memcpy(buf, svc_gl_cb_img, sizeof(svc_gl_cb_img)); + src_va = svc_gl_cb_img_base; + val = (u32)(src_va >> 8); + src = "const-upload"; + ssrc = 15; + goto have; + } + /* + * r1513: source selection by CONTENT, in authority order. r1512's + * coverage receipt measured the forced historical page 0x0cd70000 + * delivering a rotation block as the "MVP" (clip w ~= 1.1, bottom + * row 0.1268 0.9922 0 1; 57/64 sampled gear vertices clipped out). + * The client's own SET_MME_SHADOW_SCRATCH(0x18) is the authority for + * the fixed-CB base; on the validated boots its low values were + * 0x43e00/0x44500 (VA 0x43e0000/0x4450000), so try those observed + * client-CB VAs too. Every candidate must pass the strict projection + * gate; a page that only looks float-ish is no longer accepted. + */ + { + u64 cand[5]; + u32 ncand = 0, ci; + + if (svc_scratch18_seen && READ_ONCE(svc_scratch18_val)) + cand[ncand++] = (u64)READ_ONCE(svc_scratch18_val) << 8; + cand[ncand++] = 0x43e0000ULL; + cand[ncand++] = 0x4450000ULL; + if (sema_gl_cb_src_va) + cand[ncand++] = sema_gl_cb_src_va; + cand[ncand++] = 0x0cd70000ULL; + + for (ci = 0; ci < ncand; ci++) { + u64 cpa = 0; + u32 cap = 0; + + if (!svc_gl_cb_read_page(vgpu, cand[ci], buf, + &cpa, &cap) || + !cpa || !svc_gl_cb_mvp_ok(buf)) + continue; + src_va = cand[ci]; + pa = cpa; + ap = cap; + val = (u32)(src_va >> 8); + src = (ci == 0u && svc_scratch18_seen) ? + "scratch18" : "mvp-scan"; + ssrc = (ci == 0u && svc_scratch18_seen) ? 13 : 14; + goto have; + } + } + /* No strict-MVP page: keep the historical chain as the fallback. */ + if (sema_gl_cb_src_va) { + src_va = sema_gl_cb_src_va; + val = (u32)(src_va >> 8); + } else if (svc_scratch18_seen) { + val = READ_ONCE(svc_scratch18_val); + if (!val) + return; + src_va = (u64)val << 8; + } else { + src_va = 0x0cd70000ULL; + val = (u32)(src_va >> 8); + src = "captured-miss"; + } + /* + * r1491: the fixed constant buffer is ENGINE-written (the driver + * publishes its base through SET_MME_SHADOW_SCRATCH(FIXED_CONSTANT_ + * BUFFER), then VPC constant methods and the fixed-CB selector make + * the GPU fill it). Its content therefore lives in the page the GR + * engine reads: a guest-RAM GPA (SYS). The emulated BAR1 vidmem + * shadow only holds guest-CPU writes through the aperture, so + * reading a FB-resolved fixed CB there returns unrelated vidmem - + * r1490 read `09037223 00000004`, r1482 oscillated between a float + * row, method words and zeros. Resolve the page the way the engine + * does first: this channel's own VAS, then the GR relocation + * resolver (ledger + VAS walk + verified HPA). Only a SYS answer is + * content-valid; an FB answer falls through to the walk paths, and + * the FB shadow read is the last resort (and marks the matrix + * untrusted). + */ + if (!svc_chanvas_lookup(vgpu, src_va, &pa, &ap)) { + ssrc = 1; + src = "chanvas"; + if (ap == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, buf, + sizeof(buf))) + goto have; + } + /* + * r1492: walk every registered VAS before the GR ledger. The + * landing table (which gr_reloc_resolve prefers for VAs above + * SVC_VAS_FLOOR) is fed by our own launches and can answer with a + * re-used VA's page; the page tables are the mapping truth. + */ + if (!svc_walk_vaspaces(vgpu, + src_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &pa, &ap)) { + pa += src_va & (SEMA_PAGE_SIZE - 1u); + ssrc = 8; + src = "vaswalk"; + if (ap == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, buf, + sizeof(buf))) + goto have; + } + { + struct gsp_sema_gr_reloc r; + + if (!gsp_sema_gr_reloc_resolve(src_va, &r)) { + pa = r.pa; + ap = r.ap; + hpa = r.hpa; + ssrc = 2; + src = (r.ap == SVC_APERTURE_FB) ? "gr-fb" : "gr-sys"; + if (r.ap == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, buf, + sizeof(buf))) + goto have; + } + } + rc = svc_resolve_va(vgpu, src_va, &pa, &ap); + if (!rc && ap == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, buf, sizeof(buf))) { + ssrc = 3; + src = "resolve-sys"; + goto have; + } + if (!rc) { + /* An FB resolution is only valid for content the guest CPU + * writes through the aperture; use it but flag it. */ + if (!svc_read_ap(vgpu, pa, ap, buf, sizeof(buf))) { + ssrc = 4; + src = "resolve-fb"; + goto have; + } + } + /* Fallback: the trusted walk svc_gl_flush uses for the program page. */ + if (!svc_read_va_trusted(vgpu, src_va, buf, sizeof(buf))) { + ssrc = 5; + src = "trusted"; + goto have; + } + /* Fallback: the general lookup (adds the vidmem window convention). */ + if (!svc_read_va(vgpu, src_va, buf, sizeof(buf))) { + ssrc = 6; + src = "lookup"; + goto have; + } + /* + * r1505 (mission task 3): content-based discovery before the identity + * fallback. Two candidates are cheap and boot-independent: + * - the forced seed page itself (the fixed CB is ENGINE-written; an + * earlier launch or the client's own VPC writes may already have + * filled it with the real rows), and + * - this launch's own QMD pointer pages (the constant banks the + * engine fetches), accepted only when the 0x80 block passes the + * matrix gate. + * Only SYS reads are trusted, exactly like the chain above. + */ + { + static u8 cbscan[SEMA_PAGE_SIZE]; + u64 cpa = 0; + u32 cap = 0; + u32 ci; + + if (svc_gl_cb_read_page(vgpu, sema_gl_cb_copy_to, cbscan, + &cpa, &cap) && + svc_gl_cb_matrix_ok(cbscan)) { + memcpy(buf, cbscan, sizeof(cbscan)); + pa = cpa; + ap = cap; + src = "dst-seed"; + ssrc = 10; + goto have; + } + if (svc_gr_cand.valid && svc_gr_cand.h_object == h_object) { + for (ci = 0; ci < svc_gr_cand.n_pointer && ci < 8u; + ci++) { + if (!svc_gl_cb_read_page(vgpu, + svc_gr_cand.pointer_va[ci], + cbscan, &cpa, &cap) || + !svc_gl_cb_matrix_ok(cbscan)) + continue; + memcpy(buf, cbscan, sizeof(cbscan)); + pa = cpa; + ap = cap; + src_va = svc_gr_cand.pointer_va[ci]; + src = "ptr-scan"; + ssrc = 11; + goto have; + } + if (svc_gr_cand.param_va && + svc_gl_cb_read_page(vgpu, svc_gr_cand.param_va, + cbscan, &cpa, &cap) && + svc_gl_cb_matrix_ok(cbscan)) { + memcpy(buf, cbscan, sizeof(cbscan)); + pa = cpa; + ap = cap; + src_va = svc_gr_cand.param_va; + src = "param-scan"; + ssrc = 12; + goto have; + } + } + } + /* + * r1476: nothing resolved the client's fixed-CB VA. Rather than + * leave the VS reading screen data, author a synthetic identity MVP + * at the position block the VS reads (c[0x0][0x80..0xbc], one 4x4 + * row-major matrix). The vertices then pass through in model space; + * the viewport prelude maps NDC to the RT. This is a controlled + * positive control for the raster path: if pixels appear, the + * constant CONTENT is the only remaining unknown; if not, the + * raster/clip path is still wrong. + */ + memset(buf, 0, sizeof(buf)); + put_unaligned_le32(0x3f800000u, buf + 0x80); /* m00 */ + put_unaligned_le32(0x3f800000u, buf + 0x94); /* m11 */ + put_unaligned_le32(0x3f800000u, buf + 0xa8); /* m22 */ + put_unaligned_le32(0x3f800000u, buf + 0xbc); /* m33 */ + src = "synthetic-id"; + ssrc = 7; + goto have; +have: + /* + * r1491: a source whose 0x80 block is not a credible matrix would + * still collapse every vertex. Only a SYS (engine-page) read is + * trusted; an FB-shadow read is by construction not the engine's + * page, so it must not become the seed either. + * + * r1494: split "is this the real CB?" from "what do we write?". + * A validated SYS page becomes the boot's remembered real CB. When + * a later resolution misses (or answers with a method page), write + * that remembered page again instead of the identity fallback - + * the fallback may only run when no real CB was ever resolved this + * boot. Do not skip the write on an unchanged hash: the seed page + * lives in the client's screen allocation and the guest can have + * rewritten it between launches, so "same content as last time" is + * not the same as "page still holds it". + */ + if (ssrc != 4 && ssrc != 5 && ssrc != 6 && ssrc != 7 && + ssrc != 13 && ssrc != 14) { + /* + * r1513: a page that failed the strict projection gate must + * not reach the VS as the seed; only the content-discovered + * candidates (13/14) skip this re-check. + */ + if (!svc_gl_cb_mvp_ok(buf)) { + if (svc_gl_cb_good_valid) { + memcpy(buf, svc_gl_cb_good, + sizeof(svc_gl_cb_good)); + src = "cached-real"; + ssrc = 9; + } else { + memset(buf, 0, sizeof(buf)); + put_unaligned_le32(0x3f800000u, buf + 0x80); + put_unaligned_le32(0x3f800000u, buf + 0x94); + put_unaligned_le32(0x3f800000u, buf + 0xa8); + put_unaligned_le32(0x3f800000u, buf + 0xbc); + src = "synthetic-id"; + ssrc = 7; + } + goto cb_ready; + } + memcpy(svc_gl_cb_good, buf, sizeof(svc_gl_cb_good)); + svc_gl_cb_good_valid = true; + } else if (ssrc == 13 || ssrc == 14) { + memcpy(svc_gl_cb_good, buf, sizeof(svc_gl_cb_good)); + svc_gl_cb_good_valid = true; + } else if (svc_gl_cb_good_valid) { + memcpy(buf, svc_gl_cb_good, sizeof(svc_gl_cb_good)); + src = "cached-real"; + ssrc = 9; + } else { + memset(buf, 0, sizeof(buf)); + put_unaligned_le32(0x3f800000u, buf + 0x80); /* m00 */ + put_unaligned_le32(0x3f800000u, buf + 0x94); /* m11 */ + put_unaligned_le32(0x3f800000u, buf + 0xa8); /* m22 */ + put_unaligned_le32(0x3f800000u, buf + 0xbc); /* m33 */ + src = "synthetic-id"; + ssrc = 7; + } +cb_ready: + hash = svc_gl_cb_hash(buf, sizeof(buf)); + (void)hash; + rc = svc_grce_inline_write(vgpu, sema_gl_cb_copy_to, buf, sizeof(buf), + h_object); + /* + * r1521: deliver the assembled image to the seed page's real backing + * store as well. The forced seed page is an engine-visible page the + * service chooses (a page of the captured X screen), not a client + * allocation: the r1506 authoring-VAS rule correctly refuses a beef + * channel write to it, so svc_grce_inline_write() above only updates + * the VA shadow while the engine keeps reading the (black) screen + * page. Resolve the seed VA through the VAS/walk/trusted chain and + * write the shadow/FB page directly - the seed destination is a + * service-forced engine input, so the ownership rule does not apply. + */ + { + u64 dpa = 0; + int dap = SVC_APERTURE_SYS; + bool resolved = false; + + if (!svc_chanvas_lookup(vgpu, + sema_gl_cb_copy_to & ~0xfffULL, + &dpa, &dap)) + resolved = true; + else if (!svc_walk_vaspaces(vgpu, + sema_gl_cb_copy_to & ~0xfffULL, + &dpa, &dap)) + resolved = true; + else if (!svc_lookup_trusted(vgpu, + sema_gl_cb_copy_to & ~0xfffULL, + &dpa, &dap)) + resolved = true; + if (resolved && dpa) { + u64 d = (dpa & ~0xfffULL) + + (sema_gl_cb_copy_to & 0xfffULL); + + if (!svc_write_ap(vgpu, d, svc_ap_of(dap), buf, + sizeof(buf)) && + svc_gl_cb_seed_log < 12u) { + u32 m[4]; + + memcpy(m, buf + 0x80, sizeof(m)); + svc_gl_cb_seed_log++; + pr_info("vgsp-sema: sema-gl: CB-SEED h=0x%x seed=0x%llx dpa=0x%llx ap=%u ssrc=%d m80=%08x %08x %08x %08x (image written to the engine-visible seed page)\n", + h_object, sema_gl_cb_copy_to, d, + svc_ap_of(dap), ssrc, + m[0], m[1], m[2], m[3]); + } + } + } + if (svc_gl_cb_copy_log < 96u) { + u32 w[8]; + + memcpy(w, buf, sizeof(w)); + svc_gl_cb_copy_log++; + pr_info("vgsp-sema: sema-gl: cb-copy src=0x%llx dst=0x%llx pa=0x%llx ap=%u hpa=0x%llx ssrc=%d first8=%08x %08x %08x %08x %08x %08x %08x %08x src=%s write=%d cached=%d (fixed-CB page -> forced seed page)\n", + src_va, sema_gl_cb_copy_to, pa, ap, hpa, ssrc, + w[0], w[1], w[2], w[3], w[4], w[5], w[6], w[7], + src, rc, svc_gl_cb_good_valid ? 1 : 0); + if (svc_gl_cb_copy_log <= 12u) + svc_gl_cb_dump_matrix(buf, src, pa, ap, ssrc); + } + if (!rc) { + svc_gl_cb_copy_h = h_object; + svc_gl_cb_copy_val = val; + svc_gl_cb_copy_hash = hash; + svc_gl_cb_last_va = src_va; + svc_gl_cb_last_pa = pa; + svc_gl_cb_last_ssrc = ssrc; + } +} + +/* + * r1512: draw-coverage diagnostic. + * + * The remaining wall is "the engine executes the draw but no fragment page + * ever changes". This receipt measures the one quantity that separates the + * two candidate causes (clipped by the viewport vs a zero/garbage vertex + * output): it reports the draw's real inputs and the clip-space result + * computed with the exact matrix the replay delivers to the VS, plus the + * viewport state the replay injects. + * + * Inputs, all from the decoded window: + * - index VA (0x17c8) and count (0x17d8), index size; + * - stream-0 stride (0x1c00 bits 11:0) and position attribute offset + * (0x1160 bits 20:7); + * - the CALL_MME_DATA (0x3810) parameter VAs; the first one that reads + * back non-zero is the vertex-stream base the setup macro will bind. + * + * The VS this replays (1408 B Turing SASS) reads position from attribute + * a[0x80..0x8c] and writes clip space to a[0x70]=x, a[0x74]=y, a[0x78]=z, + * a[0x7c]=w with a row-major 4x4 at c[0x0][0x80..0xbc]: + * x' = X*C80 + Y*C84 + Z*C88 + W*C8c + * y' = X*C90 + Y*C94 + Z*C98 + W*C9c + * z' = X*Ca0 + Y*Ca4 + Z*Ca8 + W*Cac + * w' = X*Cb0 + Y*Cb4 + Z*Cb8 + W*Cbc + * so a correct MVP must produce w ~= camera distance (~40) and |x'|,|y'|, + * |z'| <= w' for on-screen gear vertices. + * + * The matrix is evaluated in Q16.16 integer math (the kernel cannot use + * floating point); the receipt prints milli-units and the raw words stay + * available in the same line for an exact userspace recomputation. + * Read-only, bounded (12 receipts per boot), one group of lines per draw. + */ +#define SVC_GL_COVER_IDX_MAX 64u + +static s64 svc_gl_f2q(u32 b) +{ + u32 e = (b >> 23) & 0xffu; + u32 m = (b & 0x7fffffu) | 0x800000u; + int sh = (int)e - 134; /* exp-127-23+16 */ + s64 v; + + if (e == 0u || e == 0xffu) + return 0; + if (sh >= 0) { + if (sh > 24) + return 0x7fffffffffffffffLL; + v = (s64)m << sh; + } else { + if (sh < -24) + return 0; + v = (s64)m >> (-sh); + } + return (b & 0x80000000u) ? -v : v; +} + +static s64 svc_gl_qmul(s64 a, s64 b) +{ + return (a * b) >> 16; +} + +static s64 svc_gl_qdiv(s64 a, s64 b) +{ + if (!b) + return 0; + return (a << 16) / b; +} + +static bool svc_gl_cover_resolve(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa, u32 *ap) +{ + if (!svc_lookup_trusted(vgpu, va, pa, ap)) + return true; + if (!svc_walk_vaspaces(vgpu, va & ~(u64)(SEMA_PAGE_SIZE - 1u), + pa, ap)) { + *pa += va & (SEMA_PAGE_SIZE - 1u); + return true; + } + return false; +} + +static void svc_gl_cover_idx_read(struct nv_vgpu_device *vgpu, u64 iva, + u32 isz, u32 *idx, u32 *nidx) +{ + u8 raw[SVC_GL_COVER_IDX_MAX * 4u]; + u32 esz = (isz == 0u) ? 1u : (isz == 2u) ? 4u : 2u; + u32 nraw = (u32)sizeof(raw); + u32 i; + + *nidx = 0; + if (svc_read_va_trusted(vgpu, iva, raw, nraw)) + return; + for (i = 0; i + esz <= nraw && *nidx < SVC_GL_COVER_IDX_MAX; i += esz) { + u32 v = 0; + + if (esz == 1u) + v = raw[i]; + else if (esz == 2u) + v = get_unaligned_le16(raw + i); + else + v = get_unaligned_le32(raw + i); + idx[(*nidx)++] = v; + } +} + +static void svc_gl_cover_log(struct nv_vgpu_device *vgpu, const u8 *buf, + u32 len, u32 start_off, u32 at_off, + u32 h_object, u64 push_va, u32 topo, u32 isz, + u32 first, u32 count, u64 iva, u32 stride, + u32 voff, const u32 *cand, u32 ncand) +{ + static u8 cbalt[SEMA_PAGE_SIZE]; + const u8 *cbp; + u32 cbm[16], cbfirst[8]; + u32 vbase = 0, vpa = 0, vap = 0, vbase_fb = 0; + u32 idx[SVC_GL_COVER_IDX_MAX]; + u32 nidx = 0, i, nsamp, nvert = 0, nwle0 = 0, ninside = 0; + u32 nclip = 0, nbad = 0; + s64 M[4][4]; + s64 minx = 0, maxx = 0, miny = 0, maxy = 0; + s64 minw = 0, maxw = 0, minnx = 0, maxnx = 0, minny = 0, maxny = 0; + s64 minsx = 0, maxsx = 0, minsy = 0, maxsy = 0; + s64 sxq = 0, syq = 0, oxq = 0, oyq = 0; + u32 cliph = 0, clipv = 0, minz = 0, maxz = 0; + int vpmode = 0; + u32 vpw[20] = { 0 }; + u32 vpn = 0; + u32 gr = 0, trap = 0, trapdata = 0, ds = 0; + void __iomem *bar0; + u32 k; + + /* + * r1523: the draw-coverage receipt is only meaningful once the fixed-CB + * image is the client's real page (the first receipts used to burn the + * 12/boot budget on the pre-CB synthetic draws). The counter re-arms + * when a new CB image is validated (svc_gl_cb_upload_scan). + */ + if (!svc_gl_cb_img_valid || svc_gl_cover_n >= 12u) + return; + svc_gl_cover_n++; + + if (!stride) + stride = 24u; + if (voff >= stride) + voff = 0; + + for (i = 0; i < ncand && !vbase; i++) { + u32 w[3]; + u64 pa = 0; + u32 ap = SVC_APERTURE_SYS; + + if (!svc_gl_cover_resolve(vgpu, cand[i], &pa, &ap)) + continue; + if (svc_read_ap(vgpu, pa, ap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + w, sizeof(w))) + continue; + if (!w[0] && !w[1] && !w[2]) + continue; + vbase = cand[i]; + vpa = (u32)pa; + vap = ap; + } + if (!vbase) { + u64 pa = 0; + u32 ap = SVC_APERTURE_SYS; + + vbase = 0x44ff000u; + vbase_fb = 1; + if (svc_gl_cover_resolve(vgpu, vbase, &pa, &ap)) { + vpa = (u32)pa; + vap = ap; + } + } + + svc_gl_cover_idx_read(vgpu, iva, isz, idx, &nidx); + + if (svc_gl_cb_good_valid) { + cbp = svc_gl_cb_good; + } else { + memset(cbalt, 0, sizeof(cbalt)); + put_unaligned_le32(0x3f800000u, cbalt + 0x80); + put_unaligned_le32(0x3f800000u, cbalt + 0x94); + put_unaligned_le32(0x3f800000u, cbalt + 0xa8); + put_unaligned_le32(0x3f800000u, cbalt + 0xbc); + cbp = cbalt; + } + for (k = 0; k < 8u; k++) + cbfirst[k] = get_unaligned_le32(cbp + 4u * k); + for (k = 0; k < 16u; k++) + cbm[k] = get_unaligned_le32(cbp + 0x80u + 4u * k); + for (i = 0; i < 4u; i++) { + u32 r; + + for (r = 0; r < 4u; r++) + M[i][r] = svc_gl_f2q( + get_unaligned_le32(cbp + 0x80u + + (i * 4u + r) * 4u)); + } + + gsp_phys_gr_vp_state_get(&vpmode, vpw, &vpn); + if (vpn >= 3u && vpw[0] == 0x20020280u) { + sxq = svc_gl_f2q(vpw[1]); + syq = svc_gl_f2q(vpw[2]); + if (vpn >= 8u) { + oxq = svc_gl_f2q(vpw[5]); + oyq = svc_gl_f2q(vpw[6]); + } + if (vpn >= 14u) { + cliph = vpw[9]; + clipv = vpw[10]; + minz = vpw[12]; + maxz = vpw[13]; + } + } else if (vpn >= 3u && vpw[0] == 0x20060280u) { + sxq = svc_gl_f2q(vpw[1]); + syq = svc_gl_f2q(vpw[2]); + if (vpn >= 7u) { + oxq = svc_gl_f2q(vpw[4]); + oyq = svc_gl_f2q(vpw[5]); + } + } + + bar0 = gsp_phys_gpu_bar0(); + if (bar0) { + gr = ioread32(bar0 + 0x00400700u); + trap = ioread32(bar0 + 0x00400704u); + trapdata = ioread32(bar0 + 0x00400708u); + ds = ioread32(bar0 + 0x00405840u); + } + + pr_info("sema-gl: GL-COVER h=0x%x push=0x%llx topo=%u isz=%u first=%u count=%u iva=0x%llx stride=%u voff=%u vbase=0x%x%s vpa=0x%x ap=%d nidx=%u cbsrc=0x%llx cbpa=0x%llx cbssrc=%d cbvalid=%u\n", + h_object, push_va, topo, isz, first, count, iva, stride, + voff, vbase, vbase_fb ? "(fallback)" : "", vpa, vap, nidx, + svc_gl_cb_last_va, svc_gl_cb_last_pa, svc_gl_cb_last_ssrc, + svc_gl_cb_good_valid ? 1u : 0u); + pr_info("sema-gl: GL-COVER vp mode=%d n=%u scale=(%lld,%lld)m offset=(%lld,%lld)m cliph=%08x clipv=%08x minz=%08x maxz=%08x\n", + vpmode, vpn, (sxq * 1000) >> 16, (syq * 1000) >> 16, + (oxq * 1000) >> 16, (oyq * 1000) >> 16, + cliph, clipv, minz, maxz); + pr_info("sema-gl: GL-COVER cb first8=%08x %08x %08x %08x %08x %08x %08x %08x\n", + cbfirst[0], cbfirst[1], cbfirst[2], cbfirst[3], + cbfirst[4], cbfirst[5], cbfirst[6], cbfirst[7]); + pr_info("sema-gl: GL-COVER cb m80=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + cbm[0], cbm[1], cbm[2], cbm[3], cbm[4], cbm[5], cbm[6], + cbm[7], cbm[8], cbm[9], cbm[10], cbm[11], cbm[12], + cbm[13], cbm[14], cbm[15]); + /* + * The client's own fixed-CB base (SET_MME_SHADOW_SCRATCH(0x18) -> + * VA>>8) is the authority for c[0x0]. Read it through the page + * tables so the receipt can compare it with the page the copy used. + */ + if (svc_scratch18_seen && READ_ONCE(svc_scratch18_val)) { + u64 sva = (u64)READ_ONCE(svc_scratch18_val) << 8; + u64 spa = 0; + u32 sap = SVC_APERTURE_SYS; + int src = 0; + + if (svc_gl_cover_resolve(vgpu, sva, &spa, &sap) && + !svc_read_ap(vgpu, spa & ~(u64)(SEMA_PAGE_SIZE - 1u), + sap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + cbalt, sizeof(cbalt))) { + u32 aw[8], am[16]; + + src = 1; + for (k = 0; k < 8u; k++) + aw[k] = get_unaligned_le32(cbalt + 4u * k); + for (k = 0; k < 16u; k++) + am[k] = get_unaligned_le32(cbalt + 0x80u + + 4u * k); + pr_info("sema-gl: GL-COVER cb-scratch src=%d va=0x%llx pa=0x%llx ap=%d first8=%08x %08x %08x %08x %08x %08x %08x %08x ok=%u\n", + src, sva, spa, sap, aw[0], aw[1], aw[2], + aw[3], aw[4], aw[5], aw[6], aw[7], + svc_gl_cb_matrix_ok(cbalt) ? 1u : 0u); + pr_info("sema-gl: GL-COVER cb-scratch m80=%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", + am[0], am[1], am[2], am[3], am[4], am[5], + am[6], am[7], am[8], am[9], am[10], am[11], + am[12], am[13], am[14], am[15]); + } else { + pr_info("sema-gl: GL-COVER cb-scratch unreadable va=0x%llx (read=%d)\n", + sva, src); + } + } else { + pr_info("sema-gl: GL-COVER cb-scratch not captured (0x3460 never decoded)\n"); + } + + nsamp = min_t(u32, nidx, SVC_GL_COVER_IDX_MAX); + for (i = 0; i < nsamp; i++) { + u64 va = (u64)vbase + (u64)idx[i] * stride + voff; + u32 p[3]; + s64 P[4], C[4]; + u32 cc = 0; + s64 nx, ny, sxp, syp; + u32 r; + + if (svc_read_va_trusted(vgpu, va, p, sizeof(p))) { + nbad++; + continue; + } + P[0] = svc_gl_f2q(p[0]); + P[1] = svc_gl_f2q(p[1]); + P[2] = svc_gl_f2q(p[2]); + P[3] = 65536; + for (r = 0; r < 4u; r++) + C[r] = svc_gl_qmul(M[r][0], P[0]) + + svc_gl_qmul(M[r][1], P[1]) + + svc_gl_qmul(M[r][2], P[2]) + + svc_gl_qmul(M[r][3], P[3]); + if (C[3] <= 0) { + cc |= 0x40u; + nwle0++; + } else { + if (C[0] < -C[3]) + cc |= 0x01u; + if (C[0] > C[3]) + cc |= 0x02u; + if (C[1] < -C[3]) + cc |= 0x04u; + if (C[1] > C[3]) + cc |= 0x08u; + if (C[2] < -C[3]) + cc |= 0x10u; + if (C[2] > C[3]) + cc |= 0x20u; + } + nx = svc_gl_qdiv(C[0], C[3]); + ny = svc_gl_qdiv(C[1], C[3]); + sxp = svc_gl_qmul(sxq, nx) + oxq; + syp = svc_gl_qmul(syq, ny) + oyq; + if (cc == 0u) + ninside++; + else + nclip++; + if (!nvert) { + minx = maxx = C[0]; + miny = maxy = C[1]; + minw = maxw = C[3]; + minnx = maxnx = nx; + minny = maxny = ny; + minsx = maxsx = sxp; + minsy = maxsy = syp; + } else { + minx = min(minx, C[0]); + maxx = max(maxx, C[0]); + miny = min(miny, C[1]); + maxy = max(maxy, C[1]); + minw = min(minw, C[3]); + maxw = max(maxw, C[3]); + minnx = min(minnx, nx); + maxnx = max(maxnx, nx); + minny = min(minny, ny); + maxny = max(maxny, ny); + minsx = min(minsx, sxp); + maxsx = max(maxsx, sxp); + minsy = min(minsy, syp); + maxsy = max(maxsy, syp); + } + nvert++; + if (nvert <= 4u) + pr_info("sema-gl: GL-COVER v%u idx=%u pos=(%lld,%lld,%lld)m clip=(%lld,%lld,%lld,%lld)m cc=0x%02x ndc=(%lld,%lld)m scr=(%lld,%lld)m\n", + nvert, idx[i], + (P[0] * 1000) >> 16, (P[1] * 1000) >> 16, + (P[2] * 1000) >> 16, + (C[0] * 1000) >> 16, (C[1] * 1000) >> 16, + (C[2] * 1000) >> 16, (C[3] * 1000) >> 16, + cc, (nx * 1000) >> 16, (ny * 1000) >> 16, + (sxp * 1000) >> 16, (syp * 1000) >> 16); + } + pr_info("sema-gl: GL-COVER sum samples=%u verts=%u bad=%u wle0=%u inside=%u clipped=%u ndcx=[%lld,%lld]m ndcy=[%lld,%lld]m scrx=[%lld,%lld]m scry=[%lld,%lld]m clip=(%lld,%lld,%lld,%lld)m minw=%lldm gr=%08x trap=%08x/%08x ds=%08x\n", + nsamp, nvert, nbad, nwle0, ninside, nclip, + (minnx * 1000) >> 16, (maxnx * 1000) >> 16, + (minny * 1000) >> 16, (maxny * 1000) >> 16, + (minsx * 1000) >> 16, (maxsx * 1000) >> 16, + (minsy * 1000) >> 16, (maxsy * 1000) >> 16, + (minx * 1000) >> 16, (maxx * 1000) >> 16, + (miny * 1000) >> 16, (maxy * 1000) >> 16, + nvert ? ((minw * 1000) >> 16) : 0, + gr, trap, trapdata, ds); + (void)first; + (void)len; + (void)start_off; + (void)at_off; + (void)buf; +} + +/* + * SESSION 153-CONTINUED-94: client init-push capture + local-memory census. + * + * CONT-93's "single most concrete next fix" was that the GL driver's + * download-state LOCAL MEMORY push (SET_SHADER_LOCAL_MEMORY_A/B/C/D, + * 0x790-0x79c) is the last channel-init class missing from the replayed + * context. The claim is directly decidable at this decode site: every + * client push that reaches the service is walked once, so any INC run at + * 0x790..0x79c is counted and logged with its values (bounded). The + * guest-RAM scrape alone cannot settle it: the client's pushbuffer is + * transient (the staging copies observed at init read back all-zero + * minutes later), so the first large beef push of a boot is preserved + * host-side and dumped through the gsp-phys-init-push debugfs file. + * Read-only instrumentation; beef channels only. + */ +#define SVC_INIT_CAP_BYTES (64u * 1024u) +static u8 svc_init_cap[SVC_INIT_CAP_BYTES]; +static u32 svc_init_cap_len; +static u32 svc_init_cap_h; +static u64 svc_init_cap_va; +static bool svc_init_cap_done; +static u32 svc_lmem_hits; +static u32 svc_lmem_seen; + +static void svc_mme_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object); +static void svc_attr_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object); +static void svc_init_push_scan(const u8 *buf, u32 len, u64 push_va, u32 h_object) +{ + u32 off = 0; + + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, m; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + m = (hdr & 0x1fffu) << 2; + if (m >= 0x790u && m <= 0x79cu && + (op == 1u || op == 3u || op == 5u) && + cnt >= 1u && cnt <= 8u && off + 4u + cnt * 4u <= len) { + svc_lmem_seen++; + if (svc_lmem_hits < 16u) { + u32 k, v[4] = {0, 0, 0, 0}; + + for (k = 0; k < cnt && k < 4u; k++) + memcpy(&v[k], buf + off + 4u + k * 4u, 4); + svc_lmem_hits++; + pr_info("vgsp-sema: INIT-LMEM h=0x%x va=0x%llx m=0x%x cnt=%u vals=%08x,%08x,%08x,%08x\n", + h_object, push_va, m, cnt, + v[0], v[1], v[2], v[3]); + } + } + if ((op == 1u || op == 3u || op == 5u) && cnt && cnt <= 0xfffu) + off += 4u + cnt * 4u; + else + off += 4u; + } + if (!svc_init_cap_done && len >= 4096u && + memchr_inv(buf, 0, min_t(u32, len, 64u))) { + svc_init_cap_len = min_t(u32, len, SVC_INIT_CAP_BYTES); + memcpy(svc_init_cap, buf, svc_init_cap_len); + svc_init_cap_h = h_object; + svc_init_cap_va = push_va; + svc_init_cap_done = true; + pr_info("vgsp-sema: INIT-CAP h=0x%x va=0x%llx len=%u lmem_runs=%u (gsp-phys-init-push)\n", + h_object, push_va, svc_init_cap_len, svc_lmem_seen); + /* + * r1504 (mission task 2): the client's channel-init push is + * the one push that is guaranteed to carry the MME program + * upload (LOAD_MME_INSTRUCTION_RAM 0x114/0x118, start + * addresses 0x11c/0x120, shadow control 0x124) and the + * pipeline/attr state. Synthesizing the prelude from this + * preserved image makes the macro program resident on every + * boot instead of waiting for a later setup push that may + * never be decoded (r1503: zero prelude captures -> every + * draw launch deferred forever). The capture is + * signature-deduped, so the normal per-push path is + * unaffected. This is exactly the same bytes the VS/PS image + * recovery already trusts. + */ + svc_mme_prelude_capture(svc_init_cap, svc_init_cap_len, + svc_init_cap_va, svc_init_cap_h); + svc_attr_prelude_capture(svc_init_cap, svc_init_cap_len, + svc_init_cap_va, svc_init_cap_h); + { + u32 mn = 0, mg = 0, an = 0, ag = 0; + + (void)gsp_sema_mme_prelude_get(&mn, &mg); + (void)gsp_sema_attr_prelude_get(&an, &ag); + pr_info("vgsp-sema: INIT-CAP-PRELUDE h=0x%x mme_words=%u gen=%u attr_words=%u gen=%u (boot-independent MME residency)\n", + h_object, mn, mg, an, ag); + } + } +} + +const u8 *gsp_sema_init_cap_get(u32 *len, u32 *h, u64 *va, u32 *lmem_seen) +{ + *len = svc_init_cap_len; + *h = svc_init_cap_h; + *va = svc_init_cap_va; + *lmem_seen = svc_lmem_seen; + return svc_init_cap; +} + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude buffer. + * + * The draw-window replay raises a real MME ILLEGAL_OPCODE + * (NV_PGRAPH_PRI_MME_HWW_ESR bit 2) on CALL_MME_MACRO(2)=0x3810 with + * mme_pc=0/mme_ir=1: the MME executed a macro call with no macro program + * loaded. The client DOES upload the program (the census records the + * 0x11c/0x114 pointer pairs and the 0x124 shadow-control runs), but the + * replay only ever forwards draw-window slices, so the ladder context + * never sees the setup. This buffer captures the setup runs verbatim - + * every method run whose id is in the C597 MME setup families: + * + * 0x114 LOAD_MME_INSTRUCTION_RAM_POINTER + * 0x118 LOAD_MME_INSTRUCTION_RAM (the macro code words) + * 0x11c LOAD_MME_START_ADDRESS_RAM_POINTER + * 0x120 LOAD_MME_START_ADDRESS_RAM (per-macro entry pc) + * 0x124 SET_MME_SHADOW_RAM_CONTROL (METHOD_TRACK[_WITH_FILTER]) + * 0x234 SET_MME_VERSION + * 0x550..0x568 MME mem address / data RAM / DMA / reduction + * 0x3400..0x34fc SET_MME_SHADOW_SCRATCH + * + * - in first-encounter order, so the booter can preplay them ahead of the + * first 3D window replay of a capture generation (see + * gsp_sema_mme_prelude_get and the gsp_phys_rmcall_gr_mme_prelude knob). + * Identical runs (a re-upload of the same program) are skipped by an + * exact-run signature table, so repeated setup pushes do not grow the + * buffer; genuinely new content appends and bumps the generation so the + * next window re-preplays. The writer is the single service thread and + * appends strictly in order; the generation only increments after a + * capture that appended, giving the booter a seqcheck against torn views. + * + * SESSION 153-CONTINUED-68: the capture can DROP runs once the buffer is + * full ("buffer full=N words=1023" on r1126) - e.g. a second GL program + * whose content never entered the replay - while the SA table (recorded + * unbounded, last-wins) still points slots at those never-replayed IR + * addresses; a call then fetches macro data that is not there (Graphics + * Exception: MISSING_MACRO_DATA). The coverage bitmap + * (svc_mme_ir_cov, instruction granularity) records exactly which IR + * instructions the REPLAY holds so the launch builder can stub every + * called slot whose entry is outside that set (see + * gsp_sema_mme_ir_covered). SESSION 153-CONTINUED-77: the buffer cap + * rose 1024 -> 2048 -> 3584 words (with the launch method block raised + * to 4096 words / four pushbuffer pages) because the fully captured + * client prelude exceeded each cap and was silently truncated; + * truncation leaves called slots outside the coverage set and keeps + * them stubbed. 3584 keeps the worst-case block (cap + 448-word chunk + * + init + SA stubs) inside 4096. + */ +#define SVC_MME_PRELUDE_WORDS 3584u +#define SVC_MME_PRELUDE_SIGS 2048u +static u32 svc_mme_prelude_buf[SVC_MME_PRELUDE_WORDS]; +static u32 svc_mme_prelude_n; +static u32 svc_mme_prelude_gen; +static u64 svc_mme_prelude_sigs[SVC_MME_PRELUDE_SIGS]; +static u32 svc_mme_prelude_sig_n; +static u32 svc_mme_prelude_dump_n; +static u32 svc_mme_prelude_flag = 1; +/* + * SESSION 153-CONTINUED-57: macro-program payload words seen by the + * capture (sum of the 0x114 LOAD_MME_INSTRUCTION_RAM run counts). The + * r1094 failure mode was a launch emitted with the capture still at 9 + * words (scratch only, zero code words): the MME executed macro(2) with + * pc=0 -> ILLEGAL_OPCODE -> poison 0.6 s before the client's real setup + * push became readable. gsp_sema_mme_prelude_mature() lets the forward + * path gate the first beef launch on the program actually being present. + */ +static u32 svc_mme_prelude_code_words; +module_param_named(sema_mme_prelude_cap, svc_mme_prelude_flag, uint, 0644); +MODULE_PARM_DESC(sema_mme_prelude_cap, + "Capture the client's MME context-setup method runs (0x114..0x124, 0x234, 0x550..0x568, 0x3400..0x34fc) from trusted beef pushes for the GR launch builder's one-time preplay (default 1)"); + +/* + * SESSION 153-CONTINUED-63: the MME START-ADDRESS RAM table (macro slot -> + * IR instruction address), captured unbounded by the 512-word prelude + * buffer. The r1116 0x238c wall proved the replay's SA table was + * incomplete (the capture buffer filled with the first 11 macros' code and + * 28 further setup runs - the remaining slots - never entered the prelude): + * the window's CALL_MME_MACRO(0x22) then read SA[0x22] = 0 on the fresh + * context, the MME jumped to IR 0 (the slot-0 macro, simulating with the + * caller's args emits LOAD_CONSTANT_BUFFER_OFFSET <- 0x20020; verified + * with the offline MME64 simulator), and the class error poisoned the + * lane. The builder uses this table to complete the SA RAM it replays and + * to stub every called-but-unknown slot with a harmless end-noop. + */ +static u64 svc_mme_sa_present; +static u32 svc_mme_sa_entry[64]; +static u32 svc_mme_sa_log; +/* + * SESSION 153-CONTINUED-70: last LOAD_MME_START_ADDRESS_RAM_POINTER (0x11c) + * value; the data port (0x120) writes entries for consecutive slots from + * here (see the 0x120 case in svc_mme_prelude_capture). + */ +static u32 svc_mme_sa_ptr; + +/* + * SESSION 153-CONTINUED-68: furthest IR instruction any CAPTURED 0x114 + * upload covers (pc + ceil(code words / 3); the client's upload addresses + * are instruction units - slot N's pc equals slot N-1's pc + its code + * instructions, verified across the r1099/r1116 captures). The launch + * builder's MME stub must land BEYOND the replayed program: a fixed stub + * PC inside a captured macro's body would clobber that macro's code (the + * window calls covered slots too). Recorded only for runs that entered the + * replay buffer, so the value always describes the replayed IR extent. + */ +static u32 svc_mme_ir_end; + +u32 gsp_sema_mme_ir_end_get(void) +{ + return READ_ONCE(svc_mme_ir_end); +} + +/* + * SESSION 153-CONTINUED-70: IR code words captured so far (the maturity + * gate's measure; the MME-PRE-DUMP prints it for the offline sim). + */ +u32 gsp_sema_mme_code_words_get(void) +{ + return READ_ONCE(svc_mme_prelude_code_words); +} + +/* + * SESSION 153-CONTINUED-68: instruction-granular coverage of the IR the + * REPLAY buffer actually holds (marked for every 0x114 run that entered + * svc_mme_prelude_buf). Values >= SVC_MME_IR_BITS are treated as covered + * "elsewhere" so an overflow cannot force every slot to be stubbed. + */ +#define SVC_MME_IR_BITS 8192u +static u8 svc_mme_ir_cov[SVC_MME_IR_BITS / 8u]; + +bool gsp_sema_mme_ir_covered(u32 pc) +{ + if (pc >= SVC_MME_IR_BITS) + return true; + return !!(svc_mme_ir_cov[pc >> 3] & (1u << (pc & 7u))); +} + +u64 gsp_sema_mme_sa_get(u32 *entries, u32 n) +{ + u32 i; + + if (entries) + for (i = 0; i < n && i < 64u; i++) + entries[i] = svc_mme_sa_entry[i]; + return READ_ONCE(svc_mme_sa_present); +} + +static void svc_mme_sa_record(u32 slot, u32 entry) +{ + if (slot >= 64u) + return; + if (!(svc_mme_sa_present & (1ull << slot)) && svc_mme_sa_log < 64) { + svc_mme_sa_log++; + pr_info("sema-mme-sa: slot[%u] entry=0x%x\n", slot, entry); + } + svc_mme_sa_entry[slot] = entry; + svc_mme_sa_present |= 1ull << slot; +} + +static bool svc_mme_setup_run_addr(u32 addr) +{ + return (addr >= 0x114u && addr <= 0x124u) || addr == 0x234u || + (addr >= 0x550u && addr <= 0x568u) || + (addr >= 0x3400u && addr <= 0x34fcu); +} + +static u64 svc_mme_run_sig(const u8 *p, u32 span, u32 hdr) +{ + u64 h = 1469598103934665603ULL ^ hdr; + u32 i; + + for (i = 1u; i < span; i++) { + u32 v; + + memcpy(&v, p + 4u * i, 4); + h ^= v; + h *= 1099511628211ULL; + } + return h; +} + +const u32 *gsp_sema_mme_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_mme_prelude_n); + if (gen) + *gen = READ_ONCE(svc_mme_prelude_gen); + return svc_mme_prelude_buf; +} + +static void svc_mme_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, added = 0, dups = 0, full = 0; + u32 words0 = svc_mme_prelude_n; + /* + * SESSION 153-CONTINUED-84: which 4K pages of the push delivered at + * least one setup run. r1240's carrier read decoded only pages 0 of + * 0x4001004 (15 slots, 933 code words) while r1236's identical + * stream decoded all 16 (55 slots, 2854 code words) - the bitmap + * names the truncation page in one line. + */ + u64 page_bits = 0; + + if (!READ_ONCE(svc_mme_prelude_flag)) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, i; + u64 sig; + bool dup = false; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + /* + * SESSION 153-CONTINUED-63: record the START-ADDRESS RAM + * pairs independent of the prelude buffer's 512-word bound + * (see svc_mme_sa_present). The client's LoadMmeStartAddress + * Ram pair run is INC 0x11c with [slot, IR-address] pairs; a + * second run for the same slots overwrites (last wins), the + * same order the engine would see. + */ + if (addr == 0x11cu && cnt >= 2u && span > 1u) { + u32 k; + + for (k = 0; k + 1u < cnt; k += 2u) { + u32 sv, ev; + + memcpy(&sv, buf + off + 4u + 4u * k, 4); + memcpy(&ev, buf + off + 8u + 4u * k, 4); + svc_mme_sa_record(sv, ev); + /* the pointer follows the pair (see the + * 0x120 data-port case below). */ + svc_mme_sa_ptr = sv + 1u; + } + } else if (addr == 0x11cu && cnt == 1u && span > 1u) { + /* pointer-only write (LOAD_MME_START_ADDRESS_RAM_ + * POINTER <- slot) */ + u32 sv; + + memcpy(&sv, buf + off + 4u, 4); + svc_mme_sa_ptr = sv; + } + /* + * SESSION 153-CONTINUED-70: SA data-port form. The pair form + * above only catches [slot,entry] runs AT 0x11c; a client that + * seeds the pointer once and then streams entries through + * LOAD_MME_START_ADDRESS_RAM (0x120) left every slot beyond + * the seeded range unrecorded, so the builder stubbed macros + * the client had really uploaded (r1143/r1144: only slots + * 0..12 known while the window calls 34/39/40/48/51/57). The + * data port takes consecutive slots from the last 0x11c + * pointer value; whether the hardware auto-increments is not + * externally documented, so the host-side table is treated as + * a hint - the builder's keep/stub decision still A/Bs one + * slot at a time and the engine only ever sees the client's + * own runs. + */ + if (addr == 0x120u && cnt && span > 1u) { + u32 k; + + for (k = 0; k < cnt; k++) { + u32 ev; + + memcpy(&ev, buf + off + 4u + 4u * k, 4); + svc_mme_sa_record(svc_mme_sa_ptr + k, ev); + } + svc_mme_sa_ptr += cnt; + } + if (!svc_mme_setup_run_addr(addr)) { + off += span * 4u; + continue; + } + page_bits |= 1ull << ((off >> 12) & 63u); + sig = svc_mme_run_sig(buf + off, span, hdr); + for (i = 0; i < svc_mme_prelude_sig_n; i++) { + if (svc_mme_prelude_sigs[i] == sig) { + dup = true; + break; + } + } + if (dup) { + dups++; + off += span * 4u; + continue; + } + if (svc_mme_prelude_n + span > SVC_MME_PRELUDE_WORDS || + svc_mme_prelude_sig_n >= SVC_MME_PRELUDE_SIGS) { + full++; + off += span * 4u; + continue; + } + memcpy(&svc_mme_prelude_buf[svc_mme_prelude_n], buf + off, + span * 4u); + /* + * SESSION 153-CONTINUED-57: count the macro-program payload - + * the 0x114 instruction-RAM uploads are the only runs that + * make the prelude functionally complete; the maturity gate + * keys on them. + * + * SESSION 153-CONTINUED-68: also track the furthest covered + * IR instruction (see svc_mme_ir_end) so the builder's stub + * lands beyond the replayed program. + */ + if (addr == 0x114u && span > 1u) { + u32 pc = 0, code_words = span - 2u, ir_end, k; + + if (span >= 2u) + memcpy(&pc, buf + off + 4u, 4); + ir_end = pc + (code_words + 2u) / 3u; + if (ir_end > READ_ONCE(svc_mme_ir_end)) + WRITE_ONCE(svc_mme_ir_end, ir_end); + /* + * SESSION 153-CONTINUED-68: mark the instructions this + * replayed run covers (see svc_mme_ir_cov). + */ + for (k = 0; k < (code_words + 2u) / 3u; k++) { + u32 a = pc + k; + + if (a < SVC_MME_IR_BITS) + svc_mme_ir_cov[a >> 3] |= 1u << (a & 7u); + } + svc_mme_prelude_code_words += span - 1u; + } + svc_mme_prelude_n += span; + svc_mme_prelude_sigs[svc_mme_prelude_sig_n++] = sig; + added++; + off += span * 4u; + } + if (added) { + WRITE_ONCE(svc_mme_prelude_gen, svc_mme_prelude_gen + 1); + if (svc_mme_prelude_dump_n < 4) { + u32 k; + + svc_mme_prelude_dump_n++; + pr_info("sema-mme-prelude: capture h=0x%x va=0x%llx len=%u added=%u words=%u gen=%u (was %u) pages=0x%llx\n", + h_object, push_va, len, added, + svc_mme_prelude_n, svc_mme_prelude_gen, + words0, (unsigned long long)page_bits); + for (k = 0; k + 4u <= svc_mme_prelude_n && + k < 512u; k += 4u) + pr_info("sema-mme-prelude[%03u] %08x %08x %08x %08x\n", + k, svc_mme_prelude_buf[k], + svc_mme_prelude_buf[k + 1], + svc_mme_prelude_buf[k + 2], + svc_mme_prelude_buf[k + 3]); + } + if (full) + pr_info("sema-mme-prelude: buffer full=%u words=%u dup=%u\n", + full, svc_mme_prelude_n, dups); + } else if (dups && svc_mme_prelude_dump_n < 8) { + svc_mme_prelude_dump_n++; + pr_info("sema-mme-prelude: re-upload all-duplicate h=0x%x va=0x%llx dup=%u words=%u\n", + h_object, push_va, dups, svc_mme_prelude_n); + } +} + +/* + * SESSION 153-CONTINUED-57: is the captured MME setup complete enough to + * launch against? The r1091 (works) / r1094 (poisons) A/B: a mature capture + * holds the client's macro program (>= 64 code words = at least four + * 0x114 uploads); the premature r1094 capture held 9 scratch-only words + * and the launch raised MME ILLEGAL_OPCODE within milliseconds. The gate + * in svc_gr_request defers beef forwards until this reads true (bounded by + * sema_gr_prelude_gate_ms so a client that never uploads cannot livelock + * the forward path). + */ +bool gsp_sema_mme_prelude_mature(void) +{ + return READ_ONCE(svc_mme_prelude_code_words) >= 64u; +} + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude. + * + * The full-MME-prelude boot (r1091) cleared the MME ILLEGAL_OPCODE but the + * first beef window then faulted the draw shader: NV_PGRAPH_PRI_DS_HWW_ESR + * (0x405840) latched 0xa2040a00 = SPH9|SPH11|SPH18 pending with + * M_PIPE_STAGE = VSB|PS, i.e. draws launched against shader-program-header + * slots whose SET_PIPELINE_PROGRAM_ADDRESS binds live in windows that were + * NEVER replayed (mid-stream state cut), and the same window then raised a + * class error at SET_VERTEX_ATTRIBUTE_A(2) (0x1168, INVALID_ENUM) - the + * window rebinds SOME attribute/pipeline state (byte-faithful, verified) + * but the surrounding record the client set earlier is missing. This + * buffer captures every trusted beef push's method runs in the C597 + * attribute/pipeline families, first-encounter order: + * + * 0x1100..0x11fc attribute formats, skip masks, vertex attribute A/B, + * anti-alias sample positions + * 0x2000..0x205c SET_PIPELINE_SHADER/REGISTER_COUNT/PROGRAM_ADDRESS/ + * BINDING - the per-stage pipeline record + * + * The launch builder preplays the captured runs (run-aligned, relocation + * applied exactly like the window itself) ahead of the first 3D window + * replay of a capture generation. The SET_PIPELINE_PROGRAM_ADDRESS_A/B + * operand VAs the runs bind are also recorded (svc_attr_prelude_ptr) so + * svc_gr_req_fill can put their pages into the launch page set - without + * them the shadow relocation MISSes the pair, the emitted address stays a + * raw client VA, and the DS cannot fetch the SPH through it. Identical + * re-uploads are deduped by an exact-run signature table (same scheme as + * the MME prelude). + */ +#define SVC_ATTR_PRELUDE_WORDS 512u +#define SVC_ATTR_PRELUDE_SIGS 256u +#define SVC_ATTR_PRELUDE_PTRS 8u +static u32 svc_attr_prelude_buf[SVC_ATTR_PRELUDE_WORDS]; +static u32 svc_attr_prelude_n; +static u32 svc_attr_prelude_gen; +static u64 svc_attr_prelude_sigs[SVC_ATTR_PRELUDE_SIGS]; +static u32 svc_attr_prelude_sig_n; +static u32 svc_attr_prelude_dump_n; +static u32 svc_attr_prelude_flag = 1; +static u64 svc_attr_prelude_ptr[SVC_ATTR_PRELUDE_PTRS]; +static u32 svc_attr_prelude_ptr_n; +module_param_named(sema_attr_prelude_cap, svc_attr_prelude_flag, uint, 0644); +MODULE_PARM_DESC(sema_attr_prelude_cap, + "Capture the client's attr/pipeline method runs (0x1100..0x11fc, 0x2000..0x21fc) from trusted beef pushes for the GR launch builder's preplay ahead of the first 3D window (default 1)"); + +static bool svc_attr_setup_run_addr(u32 addr) +{ + /* + * SESSION 153-CONTINUED-86: the pipeline family extends across ALL + * eight pipeline records (SET_PIPELINE_SHADER/BINDING/PROGRAM_ADDRESS + * for j=0..7 = 0x2000..0x21fc), not just j=0..1. The r1273 scrape of + * the client's own push image proved the guest driver binds stage 2 + * (TESS_CONTROL, VA 0x4400200) and stage 5 (PIXEL, VA 0x4400000) with + * the same pair runs; the old 0x2060 cut-off dropped those binds, the + * replay channel's stages 2/5 kept stale/zero program addresses, and + * the draw's SPH validation failed for the stages it could not fetch + * ("Shader Program Header 9/11/18 Error", DS_HWW_ESR 0xa2040a00). + */ + return (addr >= 0x1100u && addr < 0x1200u) || + (addr >= 0x2000u && addr < 0x2200u); +} + +const u32 *gsp_sema_attr_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_attr_prelude_n); + if (gen) + *gen = READ_ONCE(svc_attr_prelude_gen); + return svc_attr_prelude_buf; +} + +const u64 *gsp_sema_attr_prelude_ptrs(u32 *n) +{ + if (n) + *n = READ_ONCE(svc_attr_prelude_ptr_n); + return svc_attr_prelude_ptr; +} + +static void svc_attr_prelude_note_ptr(u64 va) +{ + u32 i; + + if (!va || svc_attr_prelude_ptr_n >= SVC_ATTR_PRELUDE_PTRS) + return; + for (i = 0; i < svc_attr_prelude_ptr_n; i++) + if (svc_attr_prelude_ptr[i] == va || + (svc_attr_prelude_ptr[i] & ~(u64)0xfffu) == + (va & ~(u64)0xfffu)) + return; + svc_attr_prelude_ptr[svc_attr_prelude_ptr_n++] = va; + /* + * SESSION 153-CONTINUED-59: demand the program page NOW. The r1104 + * first beat window MISSed its 0x2054 pair operand (va=0x4350900) + * because the page did not resolve at the launch fill; the engine + * read it raw -> MMU Fault HUBCLIENT_PD @0x4350000 -> the guest + * driver reset the channel and the lane died for the boot. Recording + * the page in the pin-demand set here (at capture time, often + * minutes before the first launch) lets pin_gr_stream demand it in + * the guest so the launch fill's note resolves. See also the + * sema_attr_settle_ms launch gate. + */ + svc_setup_demand_record(va & ~(u64)0xfffu); +} + +static void svc_attr_prelude_capture(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, added = 0, dups = 0, full = 0; + u32 words0 = svc_attr_prelude_n; + + if (!READ_ONCE(svc_attr_prelude_flag)) + return; + if ((h_object & 0xfffff000u) != 0xbeef0000u) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, i; + u64 sig; + bool dup = false; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + if (!svc_attr_setup_run_addr(addr)) { + off += span * 4u; + continue; + } + /* + * SET_PIPELINE_PROGRAM_ADDRESS_A/B(j) pair [upper(7:0), + * lower(31:0)]: remember the program VA so its page can join + * the launch page set (see the header comment). + */ + if (addr >= 0x2014u && addr < 0x2200u && + (addr & 0x3fu) == 0x14u && span >= 3u) { + u32 v0 = 0, v1 = 0; + + memcpy(&v0, buf + off + 4, 4); + memcpy(&v1, buf + off + 8, 4); + svc_attr_prelude_note_ptr(((u64)(v0 & 0xffu) << 32) | + (u64)(v1 & 0xfffffffcu)); + } + sig = svc_mme_run_sig(buf + off, span, hdr); + for (i = 0; i < svc_attr_prelude_sig_n; i++) { + if (svc_attr_prelude_sigs[i] == sig) { + dup = true; + break; + } + } + if (dup) { + dups++; + off += span * 4u; + continue; + } + if (svc_attr_prelude_n + span > SVC_ATTR_PRELUDE_WORDS || + svc_attr_prelude_sig_n >= SVC_ATTR_PRELUDE_SIGS) { + full++; + off += span * 4u; + continue; + } + memcpy(&svc_attr_prelude_buf[svc_attr_prelude_n], buf + off, + span * 4u); + svc_attr_prelude_n += span; + svc_attr_prelude_sigs[svc_attr_prelude_sig_n++] = sig; + added++; + off += span * 4u; + } + if (added) { + WRITE_ONCE(svc_attr_prelude_gen, svc_attr_prelude_gen + 1); + if (svc_attr_prelude_dump_n < 4) { + u32 k; + + svc_attr_prelude_dump_n++; + pr_info("sema-attr-prelude: capture h=0x%x va=0x%llx len=%u added=%u words=%u gen=%u ptr_n=%u (was %u)\n", + h_object, push_va, len, added, + svc_attr_prelude_n, svc_attr_prelude_gen, + svc_attr_prelude_ptr_n, words0); + for (k = 0; k + 4u <= svc_attr_prelude_n && + k < 512u; k += 4u) + pr_info("sema-attr-prelude[%03u] %08x %08x %08x %08x\n", + k, svc_attr_prelude_buf[k], + svc_attr_prelude_buf[k + 1], + svc_attr_prelude_buf[k + 2], + svc_attr_prelude_buf[k + 3]); + } + if (full) + pr_info("sema-attr-prelude: buffer full=%u words=%u dup=%u\n", + full, svc_attr_prelude_n, dups); + } +} + +/* + * SESSION 153-CONTINUED-62: surface (render-target) setup capture. The + * replayed beef 3D windows are mid-stream cuts of the client's per-frame + * pushes: the C597 colour-target bind (INC 0x800+j*0x40, [upper, lower, + * w, h, fmt, blksz]) is sticky context state the client (and the X + * server) writes once per context/window, so the draw windows themselves + * carry no render target and the replayed draw rasterizes into an + * unbound target - every launch retires with 0/N pages changed while + * every other witness (index buffers, programs, macros) is in place. + * The observed carrier is the X server's 1280x720 X-screen bind (VA + * 0x120400000, fmt 0xe6, 900 pages) on its own channel, so this capture + * is channel-agnostic (any trusted push) and keeps the LAST surface run + * whose shape the booter's alias scan can use. The launch builder + * replays it - aliased/relocated exactly like the window - ahead of any + * window that carries no usable render-target bind of its own; see + * gsp_phys_gr_surf_pre in gsp_phys_booter.c. A/B via sema_surf_prelude_cap. + */ +#define SVC_SURF_PRE_WORDS 16u +static u32 svc_surf_pre_buf[SVC_SURF_PRE_WORDS]; +static u32 svc_surf_pre_n; +static u32 svc_surf_pre_gen; +static u64 svc_surf_pre_sig; +static u32 svc_surf_pre_log; +static u32 svc_surf_pre_flag = 1; +/* + * SESSION 153-CONTINUED-63: the captured colour target's geometry, for the + * fill's render-target tracking (see svc_gr_data_fill and + * gsp_sema_gr_surf_recs): the fill seeds the surface base VA as a tracked + * data candidate so the engine's stores into the target become a retire + * witness and the copy-out ships them back to the guest's own pages. + */ +static u64 svc_surf_pre_va; +static u32 svc_surf_pre_npages; +/* + * SESSION 154 r1483: the VAS the captured colour-target bind came from. + * The pin/learned/map tables are keyed by VA ALONE, and every client's RM + * window sits at the same VAs (0x1200xxxxx): at r1482 the beef client held + * an injected pin for VA 0x120400000 (its own SYS allocation, guest PA + * 0x120400000) while the X server's screen surface of the SAME VA is a + * 900-page FB allocation. The fill seeded the captured surface VA and the + * global pin answered first, so the render-target recs became 10 pages of + * the beef client's buffer (the SASS-like content in the A-TRACE dump) + * instead of the screen the bind actually names. Resolve the captured + * surface through the page tables of the VAS that emitted the bind (the + * same channel-scoped rule as svc_chanvas_lookup, r835). Default 1; + * sema_gl_surf_vas=0 restores the legacy pin-first order for A/B. + */ +static u32 svc_surf_pre_hvas; +static u32 svc_surf_vas_first = 1; +module_param_named(sema_surf_prelude_cap, svc_surf_pre_flag, uint, 0644); +MODULE_PARM_DESC(sema_surf_prelude_cap, + "Capture the last C597 colour-target bind (INC 0x800+j*0x40, fmt 0xe6) from trusted pushes so the GR launch builder can replay it ahead of 3D windows that carry no render target (default 1)"); +module_param_named(sema_gl_surf_vas, svc_surf_vas_first, uint, 0644); +MODULE_PARM_DESC(sema_gl_surf_vas, + "Resolve the captured colour-target (render-target) pages through the page tables of the VAS that emitted the bind, before the global VA-keyed pin chain (default 1; 0 = legacy pin-first A/B)"); + +const u32 *gsp_sema_surf_prelude_get(u32 *n, u32 *gen) +{ + if (n) + *n = READ_ONCE(svc_surf_pre_n); + if (gen) + *gen = READ_ONCE(svc_surf_pre_gen); + return svc_surf_pre_buf; +} + +bool gsp_sema_surf_va_get(u64 *va, u32 *npages) +{ + u32 n = READ_ONCE(svc_surf_pre_npages); + + if (!n) + return false; + if (va) + *va = svc_surf_pre_va; + if (npages) + *npages = n; + return true; +} + +/* + * SESSION 154 r1506: surface OWNER receipt + span-coverage retarget. + * + * The captured colour-target bind names a VA (the X screen 0x120400000, + * 1280x720, 900 pages) that EVERY client also uses for its own RM window + * (the GL client's 10-page SYS buffer, the fixed-CB page at 0x120700000). + * The page tables are the only authority for which physical pages the bind + * means, and only a VAS that maps the WHOLE span is that surface's owner. + * The emitting channel's VAS is tried first; if it does not cover the + * first/middle/last page of the span, the first published VAS that does is + * chosen instead. The receipt names the emitting channel (owner class), + * every VAS's first-page answer, and the chosen VAS/PA so the collision is + * on the record instead of being inferred. + */ +static const char *svc_surf_owner_name(u32 h) +{ + if ((h & 0xfffff000u) == 0xbeef0000u) + return "gl"; + if (h == 0x1000013u || (h & 0xffff0000u) == 0xbfef0000u) + return "x"; + return "ch"; +} + +static u32 svc_surf_owner_vas(struct nv_vgpu_device *vgpu, u32 emit_vas, + u64 va, u64 npages, u32 emit_h, u64 push_va) +{ + static u32 svc_surf_alt_log; + static u32 svc_surf_own_log; + u64 mid_va = va + (((npages / 2u) ? (npages / 2u) : 1u) << 12); + u64 last_va = va + ((npages - 1u) << 12); + u32 chosen = emit_vas; + bool emit_ok = false; + u32 i, best_cov = 0; + + for (i = 0; i < SVC_MAX_VASPACES; i++) { + u64 p0 = 0, pm = 0, pl = 0; + int a0 = SVC_APERTURE_SYS, am = SVC_APERTURE_SYS; + int al = SVC_APERTURE_SYS; + u32 cov = 0; + bool valid = svc.vas[i].valid; + + if (!valid) + continue; + if (!svc_vas_lookup_h(vgpu, svc.vas[i].h_vaspace, va, &p0, &a0)) + cov++; + if (!svc_vas_lookup_h(vgpu, svc.vas[i].h_vaspace, mid_va, &pm, + &am)) + cov++; + if (!svc_vas_lookup_h(vgpu, svc.vas[i].h_vaspace, last_va, &pl, + &al)) + cov++; + if (svc_surf_alt_log < 12u) { + svc_surf_alt_log++; + pr_info("sema-surf-owner-alt: vas=0x%x cov=%u/3 first_pa=0x%llx ap=%d mid_pa=0x%llx last_pa=0x%llx\n", + svc.vas[i].h_vaspace, cov, p0, a0, pm, pl); + } + if (svc.vas[i].h_vaspace == emit_vas && cov == 3u) + emit_ok = true; + if (cov == 3u && !best_cov) { + best_cov = cov; + if (!emit_ok) + chosen = svc.vas[i].h_vaspace; + } + } + if (!emit_ok && best_cov == 3u && svc_surf_own_log < 8u) { + pr_info("sema-surf-owner: RETARGET emit_vas=0x%x -> vas=0x%x (emitting VAS does not cover the %llu-page span)\n", + emit_vas, chosen, npages); + } + if (svc_surf_own_log < 8u) { + u64 p0 = 0, pl = 0; + int a0 = SVC_APERTURE_SYS, al = SVC_APERTURE_SYS; + + if (chosen) + (void)svc_vas_lookup_h(vgpu, chosen, va, &p0, &a0); + if (chosen) + (void)svc_vas_lookup_h(vgpu, chosen, last_va, &pl, &al); + svc_surf_own_log++; + pr_info("sema-surf-owner: owner=%s h=0x%x push=0x%llx va=0x%llx pages=%llu emit_vas=0x%x chosen_vas=0x%x first_pa=0x%llx ap=%d last_pa=0x%llx ap=%d\n", + svc_surf_owner_name(emit_h), emit_h, push_va, va, + npages, emit_vas, chosen, p0, a0, pl, al); + } + return chosen; +} + +/* + * SESSION 154 r1510: screen/code-heap overlap quarantine. + * + * The guest driver's shader/code-heap upload (BAR1 aperture writes that the + * mmap'd shadow does not trap) can land on the captured X screen's first + * pages (r1508: vidmem 0x200000 held a program image while the X screen VA + * 0x120400000 maps there), so the xwd is a picture of code instead of the + * render target. The screen allocation is the legitimate owner of those + * pages; the code heap is an overlapping guest allocation whose offset the + * emulator does not control. + * + * At the surface capture (the X server's colour-target bind, normally + * before the client uploads) take a page snapshot of the captured surface. + * On later launches, if the surface's first page currently carries the + * client code-heap/SPH signature, restore the whole surface from that + * snapshot (once per pollution episode; the baseline stays armed so a later + * client context's upload is caught too). The restore is content-gated + * (only a page whose +0x700 word is a SPH magic or whose first word is the + * observed code-heap header fires) and happens before the engine runs, so + * raster stores that follow are preserved and the scanout stops showing + * code bytes. + */ +#define SVC_SURF_CLEAN_MAX 1024u +static struct nv_vgpu_device *svc_surf_clean_vgpu; +static u8 *svc_surf_clean_buf; +static u64 svc_surf_clean_va; +static u32 svc_surf_clean_npages; +static u32 svc_surf_clean_hvas; +static bool svc_surf_clean_armed; +static u32 svc_surf_clean_log; +static u32 svc_surf_clean_restore_log; +static u8 svc_surf_clean_page[SEMA_PAGE_SIZE]; +static DEFINE_MUTEX(svc_surf_clean_mtx); + +static bool svc_surf_code_sig(const u8 *p) +{ + u32 w0 = 0, w700 = 0; + + memcpy(&w0, p, 4); + memcpy(&w700, p + 0x700, 4); + return w0 == 0x0400780cu || w700 == 0x000a1482u || + w700 == 0x020a0481u || w700 == 0x020a0881u; +} + +static void svc_surf_clean_snapshot(struct nv_vgpu_device *vgpu, u64 va, + u32 npages, u32 hvas) +{ + u32 k; + + if (!vgpu || !va || npages < 2u || npages > SVC_SURF_CLEAN_MAX) + return; + mutex_lock(&svc_surf_clean_mtx); + if (svc_surf_clean_armed && va == svc_surf_clean_va && + npages == svc_surf_clean_npages) { + mutex_unlock(&svc_surf_clean_mtx); + return; + } + if (!svc_surf_clean_buf) { + svc_surf_clean_buf = vzalloc((size_t)SVC_SURF_CLEAN_MAX * + SEMA_PAGE_SIZE); + if (!svc_surf_clean_buf) { + mutex_unlock(&svc_surf_clean_mtx); + return; + } + } + for (k = 0; k < npages; k++) { + u64 page = va + ((u64)k << 12); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + u8 *dst = svc_surf_clean_buf + (size_t)k * SEMA_PAGE_SIZE; + + if (svc_vas_lookup_h(vgpu, hvas, page, &pa, &ap) || + svc_read_ap(vgpu, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + svc_ap_of(ap), dst, SEMA_PAGE_SIZE)) + memset(dst, 0, SEMA_PAGE_SIZE); + } + if (svc_surf_code_sig(svc_surf_clean_buf)) { + /* + * Captured after the overlap (the client initialised before + * the X bind this boot): the legitimate screen state for + * this harness is the X opaque-black clear (the root window + * is black and the earlier clean boot's page 0 read + * 0xff000000). Use that as the baseline for every page so + * the quarantine can still give the render target back to + * the engine. + */ + u32 *w = (u32 *)svc_surf_clean_buf; + size_t n = (size_t)npages * SEMA_PAGE_SIZE / 4u; + size_t i; + + for (i = 0; i < n; i++) + w[i] = 0xff000000u; + if (svc_surf_clean_log < 8u) { + svc_surf_clean_log++; + pr_info("SURF-CLEAN-SNAP va=0x%llx pages=%u hvas=0x%x poisoned -> black baseline armed\n", + va, npages, hvas); + } + } else if (svc_surf_clean_log < 8u) { + u32 w0 = 0; + + memcpy(&w0, svc_surf_clean_buf, 4); + svc_surf_clean_log++; + pr_info("SURF-CLEAN-SNAP va=0x%llx pages=%u hvas=0x%x first0=%08x (clean baseline armed)\n", + va, npages, hvas, w0); + } + svc_surf_clean_va = va; + svc_surf_clean_npages = npages; + svc_surf_clean_hvas = hvas; + svc_surf_clean_vgpu = vgpu; + svc_surf_clean_armed = true; + mutex_unlock(&svc_surf_clean_mtx); +} + +int gsp_sema_surf_clean_restore(u64 va, u64 npages) +{ + struct nv_vgpu_device *vgpu; + u32 k, changed = 0; + u64 page = 0; + int ap = SVC_APERTURE_SYS; + u8 *cur = svc_surf_clean_page; + u8 before[8] = { 0 }; + + mutex_lock(&svc_surf_clean_mtx); + vgpu = svc_surf_clean_vgpu; + if (!svc_surf_clean_armed || !svc_surf_clean_buf || !vgpu || + va != svc_surf_clean_va || npages != svc_surf_clean_npages) { + mutex_unlock(&svc_surf_clean_mtx); + return 0; + } + if (svc_vas_lookup_h(vgpu, svc_surf_clean_hvas, va, &page, &ap) || + svc_read_ap(vgpu, page & ~(u64)(SEMA_PAGE_SIZE - 1), + svc_ap_of(ap), cur, SEMA_PAGE_SIZE)) { + mutex_unlock(&svc_surf_clean_mtx); + return 0; + } + if (!svc_surf_code_sig(cur)) { + if (svc_surf_clean_log < 8u) { + svc_surf_clean_log++; + pr_info("SURF-CLEAN-IDLE va=0x%llx pages=%llu page0_first=%08x (no code signature; baseline kept)\n", + va, npages, *(const u32 *)cur); + } + mutex_unlock(&svc_surf_clean_mtx); + return 0; + } + memcpy(before, cur, sizeof(before)); + for (k = 0; k < npages; k++) { + u64 p = va + ((u64)k << 12); + u64 pa = 0; + int a = SVC_APERTURE_SYS; + + if (svc_vas_lookup_h(vgpu, svc_surf_clean_hvas, p, &pa, &a)) { + a = SVC_APERTURE_SYS; + if (svc_walk_vaspaces(vgpu, p, &pa, &a)) + continue; + } + if (svc_write_ap(vgpu, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + svc_ap_of(a), + svc_surf_clean_buf + + (size_t)k * SEMA_PAGE_SIZE, + SEMA_PAGE_SIZE)) + break; + changed++; + } + if (!svc_vas_lookup_h(vgpu, svc_surf_clean_hvas, va, &page, &ap) && + !svc_read_ap(vgpu, page & ~(u64)(SEMA_PAGE_SIZE - 1), + svc_ap_of(ap), cur, SEMA_PAGE_SIZE)) + ; + if (svc_surf_clean_restore_log < 16u) { + svc_surf_clean_restore_log++; + pr_info("SURF-CLEAN-RESTORE va=0x%llx pages=%llu changed=%u before8=%08x %08x after8=%08x %08x (screen code-heap quarantine)\n", + va, npages, changed, + ((const u32 *)before)[0], ((const u32 *)before)[1], + ((const u32 *)cur)[0], ((const u32 *)cur)[1]); + } + /* Keep the baseline armed: a later client context can upload the + * code heap again; the content gate only fires on the code + * signature, never on raster pixels. */ + mutex_unlock(&svc_surf_clean_mtx); + return (int)changed; +} + +static void svc_surf_prelude_capture(struct nv_vgpu_device *vgpu, + const u8 *buf, u32 len, u32 h_vaspace, + u32 h_object, u64 push_va) +{ + u32 off = 0; + + if (!READ_ONCE(svc_surf_pre_flag)) + return; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, k, n; + u32 v[5] = { 0, 0, 0, 0, 0 }; + u64 va, bytes, npages, sig; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt) { + if ((u64)off + 4u + (u64)cnt * 4u > (u64)len) + break; + span = 1u + cnt; + } + if (op != 1u || cnt < 5u || cnt >= SVC_SURF_PRE_WORDS || + addr < 0x800u || addr >= 0xc00u || + (addr & 0x3fu) != 0u) { + off += span * 4u; + continue; + } + for (k = 0; k < 5u; k++) + memcpy(&v[k], buf + off + 4u + 4u * k, 4); + va = ((u64)(v[0] & 0xffu) << 32) | (u64)(v[1] & 0xfffff000u); + bytes = (u64)v[2] * 4ull * (u64)v[3]; + npages = (bytes + 4095ull) >> 12; + if (!va || (va & 0xfffu) || v[4] != 0xe6u || !v[2] || !v[3] || + v[2] > 8192u || v[3] > 8192u || npages < 2ull || + npages > 4096ull) { + off += span * 4u; + continue; + } + n = span; + if (n > SVC_SURF_PRE_WORDS) + n = SVC_SURF_PRE_WORDS; + sig = svc_mme_run_sig(buf + off, n, hdr); + if (n == svc_surf_pre_n && sig == svc_surf_pre_sig) { + off += span * 4u; + continue; + } + /* + * r1506: the VAS that OWNS the whole surface span, not merely + * the one the bind happened to arrive on(see + * svc_surf_owner_vas). Every client's RM window shares the + * 0x1200xxxxx VAs; only a VAS that maps the first, middle and + * last page of the span is the real surface. + */ + { + u32 chosen = svc_surf_owner_vas(vgpu, h_vaspace, va, + npages, h_object, + push_va); + + if (chosen) + h_vaspace = chosen; + } + for (k = 0; k < n; k++) + memcpy(&svc_surf_pre_buf[k], buf + off + 4u * k, 4); + WRITE_ONCE(svc_surf_pre_n, n); + svc_surf_pre_sig = sig; + svc_surf_pre_va = va; + WRITE_ONCE(svc_surf_pre_npages, (u32)npages); + /* r1483/r1506: remember which client VAS this bind's surface + * belongs to (see svc_surf_pre_hvas): its page tables are the + * only authority for the surface's pages. */ + WRITE_ONCE(svc_surf_pre_hvas, h_vaspace); + WRITE_ONCE(svc_surf_pre_gen, svc_surf_pre_gen + 1); + if (svc_surf_pre_log < 8) { + svc_surf_pre_log++; + pr_info("sema-surf-pre: captured gen=%u words=%u va=0x%llx %ux%u fmt=0x%x hdr=%08x vas=0x%x owner=%s h=0x%x push=0x%llx\n", + svc_surf_pre_gen, n, va, v[2], v[3], v[4], + hdr, h_vaspace, + svc_surf_owner_name(h_object), h_object, + push_va); + } + svc_surf_clean_snapshot(vgpu, va, (u32)npages, h_vaspace); + off += span * 4u; + } +} + +static void svc_push_raw_dump(const u8 *buf, u32 len, u64 push_va, + u32 h_object) +{ + u32 off = 0, gi = 0; + + if (len > 8192) + return; + if (len < 512) { + if (!gsp_sema_push_raw_small || + svc_pushraw_small_n >= gsp_sema_push_raw_small) + return; + } else { + if (!gsp_sema_push_raw || svc_pushraw_n >= gsp_sema_push_raw) + return; + } + if (push_va == svc_pushraw_last_va && len == svc_pushraw_last_len) + return; + svc_pushraw_last_va = push_va; + svc_pushraw_last_len = len; + if (len < 512) { + svc_pushraw_small_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u (small)\n", + h_object, push_va, len, svc_pushraw_small_n); + } else { + svc_pushraw_n++; + pr_info("sema-pushraw: BEGIN h=0x%x va=0x%llx len=%u n=%u\n", + h_object, push_va, len, svc_pushraw_n); + } + while (off + 4 <= len && gi < 160) { + u32 hdr, op, cnt, subch, addr, v[4] = {0,0,0,0}, k, n; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 0x7; + cnt = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + addr = (hdr & 0x1fff) << 2; + n = cnt > 4 ? 4 : cnt; + for (k = 0; k < n && off + 8 + 4 * k <= len; k++) + memcpy(&v[k], buf + off + 4 + 4 * k, 4); + pr_info("sema-pushraw: [%u] off=0x%x op=%u sub=%u addr=0x%x cnt=%u v0=%08x v1=%08x v2=%08x v3=%08x\n", + gi, off, op, subch, addr, cnt, + v[0], v[1], v[2], v[3]); + if (op == 3 && cnt) { + off += 4 + cnt * 4; + gi++; + continue; + } + if (!cnt) { + gi++; + off += 4; + continue; + } + if (op == 1) { + off += 4 + cnt * 4; + gi++; + continue; + } + { + u32 p; + bool found = false; + + for (p = off + 4; p + 4 <= len; p += 4) { + u32 h2; + + memcpy(&h2, buf + p, 4); + if (((h2 >> 29) & 0x7) == 1 && + ((h2 >> 16) & 0xfff)) { + off = p; + found = true; + break; + } + } + if (!found) + off = len; + gi++; + } + } + pr_info("sema-pushraw: END h=0x%x va=0x%llx consumed=0x%x/%u\n", + h_object, push_va, off, len); + /* + * r1662 diag: the direct-GL GR init push reads a zero 640-byte head + * and the semantic decoder still finds its release; dump the real + * nonzero words once per handle so the notification method (if any) + * and the slot address can be identified. + */ + if (h_object == 0xbeef0100u && svc_pushraw_nz < 96) { + u32 p; + + for (p = 0; p + 4 <= len; p += 4) { + u32 w; + + memcpy(&w, buf + p, 4); + if (!w) + continue; + if (svc_pushraw_nz >= 96) + break; + svc_pushraw_nz++; + pr_info("sema-pushraw: NZ h=0x%x off=0x%x v=%08x\n", + h_object, p, w); + } + } +} + +/* + * r828: the guest kernel's RC watchdog (krcWatchdog_IMPL, 7 s notify + * timeout) posts a notifier cycle on its own channel (object 0x31415900) + * through the class2d subchannel (NVA06F_SUBCHANNEL_2D = 3): SET_NOTIFY_A/B + * name the notifier VA, NOTIFY (WRITE_ONLY) is the GPU's "write a zeroed + * NvNotification" - the driver set the status word to 0xFFFF and declares + * "GPU is probably locked" when it never comes back as 0. Observed: robust + * channels enabled by the GL client (GZDOOM), then NVRM krcWatchdog every + * 8 s and Xorg dead on its screen readback. The emulation must author the + * notification write, exactly like svc_notify_complete does for the + * work-submit token. + */ +#define SVC_2D_SUBCH 3 /* NVA06F_SUBCHANNEL_2D */ +#define SVC_2D_SET_NOTIFY_A 0x0104 /* payload: addr bits [56:32] */ +#define SVC_2D_SET_NOTIFY_B 0x0108 /* payload: addr bits [31:0] */ +#define SVC_2D_NOTIFY 0x010c /* payload: 0 = WRITE_ONLY */ + +static u64 svc_notify2d_done; +static u32 svc_notify2d_log; +/* + * r1608 A/B: the X DDX's IMMD class-2d NOTIFY ack (the nvFermiSync + * completion). Enabled by default; set 0 to reproduce the pre-r1608 + * dropped-NOTIFY behaviour (the Xorg mode-set WAIT) without a rebuild. + */ +static bool gsp_sema_2d_notify_immd = true; +module_param_named(sema_2d_notify_immd, gsp_sema_2d_notify_immd, bool, 0644); + +/* + * CONT.406: the direct-GL kick channel submits a short + * SET_OBJECT(TURING_A)+NOTIFY push before it ever arms a class-2d + * SET_NOTIFY_A/B, so the NOTIFY arrives with no class2d target and used to be + * dropped entirely: the push then had no release edge, the channel was + * consumed without a completion, and the client spun in its kick/fence wait. + * + * A NOTIFY with no programmed address is still this push's completion edge on + * real hardware (the class object's notify address register simply holds its + * creation value and the write goes nowhere). Treat it exactly like a + * completed NOTIFY - mark the release, post the channel event so the client's + * OS event fires, and count it - but do NOT write to any address: there is no + * target, and guessing one would corrupt an unrelated page. The 64-line + * witness is separate from the drop counter so the A/B stays readable. + */ +static u64 svc_kicknoaddr_done; +static u32 svc_kicknoaddr_log; + +static bool svc_notify2d_write(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, bool completion) +{ + /* 16 bytes = the NvNotification (status word at +14). */ + u8 zero[16] = { 0 }; + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (!ch->notify2d_va) { + /* + * r1652: the short kick form (NOTIFY without SET_NOTIFY) + * depends on the target carried from the channel's previous + * registration. Name the miss so a future loss is visible + * instead of a silent spin. + */ + if (gsp_sema_dbg && svc_notify2d_log < 64) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x NOTIFY with no class2d target (short kick before SET_NOTIFY; client spin)\n", + ch->h_object); + } + if (completion) { + /* CONT.406: complete the short kick anyway (no write). */ + svc_kicknoaddr_done++; + if (svc_kicknoaddr_log < 64) { + svc_kicknoaddr_log++; + pr_info("sema-notify2d: h=0x%x short-kick NOTIFY completed with no armed target (n=%llu; release+event)\n", + ch->h_object, svc_kicknoaddr_done); + } + gsp_rm_post_channel_event(vgpu, ch->h_object); + return true; + } + return false; + } + if (svc_lookup(vgpu, ch->notify2d_va, &pa, &ap)) { + if (gsp_sema_dbg && (svc_notify2d_log < 64 || + ch->h_object == 0xbfef0002)) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx unresolved (RC watchdog notifier; guest re-posts every second)\n", + ch->h_object, ch->notify2d_va); + } + return false; + } + /* + * SESSION 153-CONTINUED-18: read the first word for the witness + * (the kick member's timestamp slot: 0 = never GPU-written; the + * completed write makes it nonzero - see svc_kick_notify_exec). + */ + { + u32 o0 = 0; + + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), &o0, sizeof(o0))) + o0 = 0xffffffff; + if (svc_write_ap(vgpu, pa, svc_ap_of(ap), zero, sizeof(zero))) + return false; + svc_notify2d_done++; + if (gsp_sema_dbg && (svc_notify2d_log < 64 || + ch->h_object == 0xbfef0002)) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x va=0x%llx -> pa=0x%llx zeroed (w0 old=%08x; ack)\n", + ch->h_object, ch->notify2d_va, pa, o0); + } + } + /* + * r834: pin the page that accepted the ack; the same stale-map + * defense as the push-side pin in the decode. + */ + svc_pin_store(ch->notify2d_va & ~(u64)(SEMA_PAGE_SIZE - 1), + pa & ~(u64)(SEMA_PAGE_SIZE - 1), svc_ap_of(ap)); + /* + * r934: a NOTIFY decoded from a push is that push's completion + * edge on real hardware - post the channel event so the submitting + * client's OS event fires (without it the GL client's kick that + * arms 0x12000e0b0 and NOTIFYs never completes its submit wait + * and the app retries every ~22 s). The periodic idempotent ack + * (r832) passes completion=false so it stays silent. + */ + if (completion) + gsp_rm_post_channel_event(vgpu, ch->h_object); + return true; +} + +/* ------------------------------------------------------------------ */ +/* FERMI_TWOD (0x902d) executor */ +/* ------------------------------------------------------------------ */ +/* + * SESSION 153-CONTINUED-26: the X DDX paints its surfaces (the screen + * 0x120400000 1280x720 X8R8G8B8 and the cursor 0x120370000 256x256 + * A8R8G8B8) exclusively through subchannel-3 FERMI_TWOD methods: the + * SET_DST_* group, then RENDER_SOLID_PRIM (mode RECTS, color format, + * solid color, RENDER_SOLID_PRIM_POINT_X_Y packed pairs). Without this + * executor every 2D surface paint was a no-op and the surfaces the CE + * readback reads stayed unwritten (the screen byte pattern is all + * 0x00000000 - "the X server's drawing never executes"). + * + * Method map: cl902d.h (FERMI_TWOD_A); encodings: NVA06F_DMA_* + * (1 INC / 3 NON_INC / 4 IMMD_DATA, data in header bits 28:16 / 5 + * ONE_INC). Live-decoded per-frame dialog (guest RAM dump at + * 0x8d30c5f8): NON_INC 0x100 cnt17 (MME/global-render-enable window), + * INC 0x280 clip {0,0,0x7fff,0x7fff}, IMMD 0x204 layout, INC 0x214 + * pitch, INC 0x218 {w,h,upper,lower}, IMMD 0x200 fmt, IMMD 0x2ac + * operation, INC 0x580 {mode,fmt,color}, NON_INC 0x5e0 {packed x/y + * point pairs}, IMMD 0x10c NOTIFY, IMMD 0x100 NOP. + * + * The DST VA is (upper<<32)|lower resolved through the SAME operand + * machinery every other engine op uses (svc_fill), so the 2D painting + * and the DDX's own CE readback of the same VA always agree by + * construction. + */ +struct svc_twod_state { + u64 h_object; + u32 dst_fmt, dst_layout, dst_bsize, dst_pitch, dst_w, dst_h; + u64 dst_off; + u32 have_dst; + u32 operation, rop; + u32 prim_mode, prim_fmt, prim_color; + u32 pt_x, pt_y, have_pt; + u64 fills, fill_bytes, misses; + u64 last_fill_hash, last_fill_va; + u32 fill_skips; + u32 dst_log, fill_log, unk_log; +}; + +#define SVC_TWOD_STATES 4 +static struct svc_twod_state svc_twod_tab[SVC_TWOD_STATES]; + +unsigned gsp_sema_twod = 1; +module_param_named(sema_twod, gsp_sema_twod, uint, 0644); +MODULE_PARM_DESC(sema_twod, + "Execute FERMI_TWOD (0x902d) fills on subchannel 3 (default 1)"); + +static struct svc_twod_state *svc_twod_get(struct svc_channel *ch) +{ + u32 i, free_i = SVC_TWOD_STATES; + + for (i = 0; i < SVC_TWOD_STATES; i++) { + if (svc_twod_tab[i].h_object == ch->h_object) + return &svc_twod_tab[i]; + if (!svc_twod_tab[i].h_object && free_i == SVC_TWOD_STATES) + free_i = i; + } + if (free_i == SVC_TWOD_STATES) + free_i = 0; /* bounded table: steal slot 0 */ + memset(&svc_twod_tab[free_i], 0, sizeof(svc_twod_tab[0])); + svc_twod_tab[free_i].h_object = ch->h_object; + return &svc_twod_tab[free_i]; +} + +static u32 svc_twod_bpp(u32 fmt) +{ + switch (fmt) { + case 0xe8: case 0xf8: case 0xe9: case 0xfb: case 0xfc: case 0xee: + return 2; + case 0xf3: case 0xf4: case 0xf7: case 0x1c: + return 1; + default: + return 4; + } +} + +/* One RECTS primitive: fill the (xa,ya)-(xb,yb) axis-aligned rectangle. */ +static void svc_twod_fill(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + struct svc_twod_state *tw, u32 xa, u32 ya, u32 xb, + u32 yb, u64 push_va) +{ + u32 bpp = svc_twod_bpp(tw->dst_fmt); + u32 pitch, x0, x1, y0, y1, w, h; + u64 done = 0; + int rc; + + if (!tw->have_dst) { + if (tw->fill_log < 32) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x push=0x%llx SKIP (no DST set)\n", + ch->h_object, push_va); + } + return; + } + /* + * Pitch: the PITCH layout carries it explicitly (the cursor dialog + * pushes 0x400 for a 256-wide 4-bpp surface). The BLOCKLINEAR + * screen dialog sets only w/h; the DDX's own CE readback of the + * same surface rides pitch = w*bpp (5120 = 1280*4), so the linear + * approximation uses w*bpp there - painting and readback agree. + */ + pitch = (tw->dst_layout == 1 && tw->dst_pitch) ? tw->dst_pitch : + tw->dst_w * bpp; + if (!bpp || !pitch || !tw->dst_w || !tw->dst_h) { + if (tw->fill_log < 32) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x SKIP (bpp=%u pitch=%u w=%u h=%u)\n", + ch->h_object, bpp, pitch, tw->dst_w, + tw->dst_h); + } + return; + } + x0 = xa < xb ? xa : xb; + x1 = xa < xb ? xb : xa; + y0 = ya < yb ? ya : yb; + y1 = ya < yb ? yb : ya; + /* Surface-bound clamp (the DDX keeps CLIP at 0x7fff). */ + if (tw->dst_w >= 4 && tw->dst_w <= 0x8000) { + if (x0 >= tw->dst_w) + return; + if (x1 > tw->dst_w) + x1 = tw->dst_w; + } + if (tw->dst_h >= 4 && tw->dst_h <= 0x8000) { + if (y0 >= tw->dst_h) + return; + if (y1 > tw->dst_h) + y1 = tw->dst_h; + } + w = x1 - x0; + h = y1 - y0; + if (!w || !h) + return; + /* + * Replay dedupe: the DDX re-submits the same dialog push buffer + * for every repaint (the pump re-decodes ring entries), and the + * first pass of a reused slot is often a replay (allow=0). The + * fill is idempotent, so re-running is safe - but re-running on + * every replay pass would write megabytes per second for zero + * change. Execute once per (push VA, content hash); a changed + * push (new points/colors/surface) hashes differently and runs. + */ + if (tw->fills && push_va == tw->last_fill_va && + tw->last_fill_hash == svc_pass_hash) { + tw->fill_skips++; + return; + } + tw->last_fill_va = push_va; + tw->last_fill_hash = svc_pass_hash; + /* + * Row-wise fill through the shared operand machinery: the same + * resolution chain (shadow -> pin -> landing -> walk -> scan) the + * CE pitched fill uses, so the operand guards (LOWFILL / + * FILL-PT / FILL-HOLD) apply to 2D painting as well. + */ + rc = svc_fill(vgpu, tw->dst_off + (u64)y0 * pitch + (u64)x0 * bpp, + tw->prim_color, bpp, w, 0, CE_APERTURE_SYS, &done, + h, pitch); + tw->fills++; + if (!rc) + tw->fill_bytes += (u64)w * bpp * h; + else + tw->misses++; + if (tw->fill_log < 256) { + tw->fill_log++; + pr_info("sema-twod-fill: h=0x%x va=0x%llx lay=%u fmt=0x%x pitch=%u rect=%u,%u-%u,%u color=0x%x bpp=%u rows=%u done=%llu rc=%d push=0x%llx\n", + ch->h_object, tw->dst_off, tw->dst_layout, + tw->dst_fmt, pitch, x0, y0, x1, y1, tw->prim_color, + bpp, h, done, rc, push_va); + } +} + +static void svc_twod_method(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 m, u32 v, bool exec, + u64 push_va) +{ + struct svc_twod_state *tw = svc_twod_get(ch); + + switch (m) { + case 0x200: + tw->dst_fmt = v; + break; + case 0x204: + tw->dst_layout = v; + break; + case 0x208: + tw->dst_bsize = v; + break; + case 0x214: + tw->dst_pitch = v; + break; + case 0x218: + tw->dst_w = v; + break; + case 0x21c: + tw->dst_h = v; + break; + case 0x220: + tw->dst_off = (tw->dst_off & 0xffffffffULL) | + ((u64)(v & 0x1ffffff) << 32); + break; + case 0x224: + tw->dst_off = (tw->dst_off & ~0xffffffffULL) | v; + tw->have_dst = 1; + if (tw->dst_log < 64) { + tw->dst_log++; + pr_info("sema-twod-dst: h=0x%x va=0x%llx pitch=%u w=%u h=%u fmt=0x%x lay=%u bsz=0x%x\n", + ch->h_object, tw->dst_off, tw->dst_pitch, + tw->dst_w, tw->dst_h, tw->dst_fmt, + tw->dst_layout, tw->dst_bsize); + } + break; + case 0x2ac: + tw->operation = v; + break; + /* + * r1608: the X DDX's nvFermiSync (nvFermiXAASync) queues the + * class-2d completion as IMMD_DATA methods: SET_NOTIFY_A/B name + * the NvNotification VA and NOTIFY (0x10c, payload 0 = + * WRITE_ONLY) must author the zeroed 16-byte notification. The + * watchdog path below has handled the run-shaped form since r828; + * the IMMD form previously fell into the default case and was + * dropped, so the X server's post-selection 2D sync spun in + * "NVIDIA(GPU-0): WAIT (0, 8, 0x8000, ...)" forever (the status + * word stayed IN_PROGRESS 0x8000). The mode-set never completed, + * glsStartServer/glsDeviceHandle never ran, and the GL client's + * physical-device init (class/gpuGartSize/+0x18e0) stayed empty. + */ + case SVC_2D_SET_NOTIFY_A: + ch->notify2d_va = (u64)(v & 0x1ffffff) << 32; + break; + case SVC_2D_SET_NOTIFY_B: + ch->notify2d_va |= (u64)v; + break; + case SVC_2D_NOTIFY: + if (v == 0 && gsp_sema_2d_notify_immd) + svc_notify2d_write(vgpu, ch, true); + break; + case 0x2a0: + tw->rop = v; + break; + case 0x580: + tw->prim_mode = v; + break; + case 0x584: + tw->prim_fmt = v; + break; + case 0x588: + tw->prim_color = v; + break; + case 0x5e0: { + u32 x = v & 0xffff, y = (v >> 16) & 0xffff; + + if (tw->prim_mode == 4 /* RECTS: point pairs */) { + if (!tw->have_pt) { + tw->pt_x = x; + tw->pt_y = y; + tw->have_pt = 1; + } else { + tw->have_pt = 0; + if (exec && gsp_sema_twod) + svc_twod_fill(vgpu, ch, tw, + tw->pt_x, tw->pt_y, + x, y, push_va); + } + } else if (tw->unk_log < 256) { + tw->unk_log++; + pr_info("sema-twod-m: h=0x%x m=0x%x v=0x%x (point mode=%u)\n", + ch->h_object, m, v, tw->prim_mode); + } + break; + } + default: + if (tw->unk_log < 256) { + tw->unk_log++; + pr_info("sema-twod-m: h=0x%x m=0x%x v=0x%x\n", + ch->h_object, m, v); + } + break; + } +} + +/* + * Dispatch a whole method run's data words. `inc` selects the NVA06F + * DMA semantics: INC_METHOD/ONE_INC (op 1/5) advance the method address + * per word (a contiguous register run); NON_INC (op 3) does NOT - every + * data word goes to the SAME method, which is how the DDX streams the + * RECTS point list into RENDER_SOLID_PRIM_POINT_X_Y (0x5e0; the 2-point + * rect and the 4-point double-band fills are one NON_INC run each, and + * its 17-word 0x100 run is a NO_OPERATION padding block). + */ +static void svc_twod_run(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + u32 addr, bool inc, u32 count, const u8 *data, + u32 avail, bool exec, u64 push_va) +{ + u32 k; + + for (k = 0; k < count && (u64)(k + 1) * 4 <= avail; k++) { + u32 v; + + memcpy(&v, data + 4 * k, 4); + svc_twod_method(vgpu, ch, inc ? addr + 4 * k : addr, v, exec, + push_va); + } +} + +/* + * SESSION 148 perf (r862/r864): the RC watchdog's class2d notify cycle + * keep-alive that needs no push decode and no captured VA. Under the + * mirrored frame flow the watchdog ring reads go unreadable (rc=-2) for + * stretches, SET_NOTIFY never re-captures, notify2d_va stays 0, and the + * put != get gate of the r832 path misses - the guest's krcWatchdog then + * declares "GPU is probably locked" (Notify Timeout Seconds: 7) every + * ~8.2 s and the whole X/GL stack wedges (glxgears poll() never returns, + * late XOOM death as buffers pile up). + * + * Ground truth (r864, live RAM scan while wedged): BOTH the watchdog + * channel (0x31415900) and the X/GL channel (0x1000013) sit with their + * class2d notification ARMED - u16 status 0xFFFF at notifier_pa + 0x3E, + * i.e. the 16-byte NvNotification struct at slot 3 (notifier_pa + 0x30, + * status at +14 within the struct) - and NOTHING ever clears them + * (X channel notified=0 after 459 submissions). Slots 0..3 cover the + * error notifier (0), the work-submit token (1) and the class2d notify + * slot (3); the armed test is EXACTLY 0xFFFF so the fault-notifier + * payload slots (nonzero engine/exception codes) are never touched. + * + * Every pass, for every GR channel with a registered notifier (driver + * truth, always known): read slots 0..3; for each slot whose status is + * armed, write the zeroed 16-byte NvNotification (the GPU's completion + * write) and post the channel event - the same two effects + * svc_notify_complete() produces, but reached without any ring read. + * Idempotent: only an armed slot is written. + */ +#define SVC_NOTIFY_SCAN_SLOTS 4 + +/* + * CONT.408: 0xFFFF IS THE X DDX'S KICK-IDLE MARKER, NOT AN ARM. + * + * The DDX's own RM channels (the 0x01xxxxxx family, e.g. the X server's + * 0x01000014) drive their push buffer through nvidia_drv.so's + * pb_reserve/pb_flush path: pb_kickIfIdle (+0xb66c0) only calls the kick + * while *(u16 *)(pb->notifier + 0x0e) == 0xFFFF (signature 0xffff8000 is + * the 0x8000 arm pb_armNotifiers +0xb7250 writes at slot 0xb; completion + * is 0x0000). The r864 registration keep-alive below was written for the + * RC-watchdog/beef families where 0xFFFF *is* the armed value, and it + * zeroed every 0xFFFF slot on every pass. On the DDX family that + * destroyed the only value pb_kickIfIdle accepts: r1706 live (Xorg 99 % + * CPU in the kick ioctl, glxgears D on the RM rwlock) had notifier slot 0 + * at 0x0000 and re-zeroed within one pass of any 0xFFFF write (measured + * by writing the live page). Keep 0xFFFF untouched (and restore slot 0 to + * it) for this family; the 0x8000->0 completion still goes through the + * bfef/X-DDX sweep. + */ +static bool svc_ddx_notif_idle(const struct svc_channel *ch) +{ + return (ch->h_object & 0xff000000u) == 0x01000000u; +} + +/* + * CONT.409: the GL client's own channel (e.g. 0xbfef0002) uses the same + * NvPushBuffer kick-idle convention as the DDX: libnvidia-glcore arms its + * class-2D notification at slot 11 (VA ...d0b0) and gates its next kick on + * slot 0's status being 0xFFFF. The r864 keep-alive zeroed that 0xFFFF on + * every pass, so after the first setup kick the client could never kick + * again - the r1716 live probe wrote 0xFFFF to slot 0 and it was re-zeroed + * within one pass (2 s sample), while glxgears kept spinning in glcore + * with only 32/64-byte setup pushes and no draw stream. A channel that has + * captured a class-2D NOTIFY VA therefore owns a kick-idle page for slots + * 0..3; the RC-watchdog/beef kernel channels (which never arm notify2d) + * keep the r864 behaviour. + */ +static bool svc_kick_idle_page(const struct svc_channel *ch) +{ + return svc_ddx_notif_idle(ch) || ch->notify2d_va != 0; +} + +static u32 svc_ddx_idle_log; +static u64 svc_ddx_idle_n; + +static void svc_notify2d_keepalive(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 slots[16 * SVC_NOTIFY_SCAN_SLOTS]; + u8 zero[16] = { 0 }; + int slot; + + if (!ch->notifier_pa) + return; + if (svc_read_ap(vgpu, ch->notifier_pa, ch->notifier_as, + slots, sizeof(slots))) + return; + for (slot = 0; slot < SVC_NOTIFY_SCAN_SLOTS; slot++) { + u16 st; + + memcpy(&st, slots + slot * 16 + 14, 2); + /* + * CONT.408: on the X DDX family 0xFFFF is the kick-idle + * marker pb_kickIfIdle requires, never an arm. Restore it + * on slot 0 (the kick gate) instead of zeroing it, and only + * ack a real 0x8000 arm on the other slots. + */ + if (svc_kick_idle_page(ch)) { + if (slot == 0) { + u16 idle = 0xffff; + + if (st == 0xffff) + continue; + if (svc_write_ap(vgpu, + ch->notifier_pa, + ch->notifier_as, &idle, + sizeof(idle))) + continue; + svc_ddx_idle_n++; + if (svc_ddx_idle_log < 32) { + svc_ddx_idle_log++; + pr_info("sema-chan: kick idle h=0x%x pa=0x%llx restored 0x%04x -> 0xffff (n=%llu)\n", + ch->h_object, ch->notifier_pa, + st, + (unsigned long long) + svc_ddx_idle_n); + } + continue; + } + if (!(st & 0x8000)) + continue; /* 0xffff is idle here */ + } else if (st != 0xffff && !(st & 0x8000)) { + continue; /* not armed - nothing to ack */ + } + if (svc_write_ap(vgpu, ch->notifier_pa + slot * 16, + ch->notifier_as, zero, sizeof(zero))) + continue; + svc_notify2d_done++; + /* + * The GPU consumed the submission: post the channel event so + * the client's CPU-side RM signals the OS event the + * submitting thread waits on (a real GPU would raise an + * interrupt here). Mirrors svc_notify_complete(). + */ + gsp_rm_post_channel_event(vgpu, ch->h_object); + if (gsp_sema_dbg && svc_notify2d_log < 16) { + svc_notify2d_log++; + pr_info("sema-notify2d: h=0x%x notifier_pa=0x%llx slot=%d armed -> zeroed (registration-based ack)\n", + ch->h_object, ch->notifier_pa, slot); + } + } +} + +/* + * SESSION 153 CONT.6 (task 2): serve a DENIED DMA copy. + * + * The X server's GL loop re-pushes transfer-shaped launches whose copy + * lands on a pass the trust/replay rules deny (witness: + * `sema-dmatrace: denied h=0x1000013 push=0x12001c64c dma=0x0000030d ... + * in=0x122281200 out=0x120350000 ... replay=1 eng=0`). Both operands of + * this class sit inside the identity guest-RAM window (< svc_pa_max()), + * so the emulator can execute the copy itself: read the source GPA, write + * the destination GPA, verify the landing bytes. The copy is exactly the + * work the hardware copy engine was asked to perform - refusing it only + * stalls whatever consumes the destination (the driver's own fence/state + * progression). Exactly-once per (push_va, content hash); a refused + * shape (operand outside guest RAM, oversized, or failing the readback) + * falls through to the existing deny/hold machinery unchanged. + */ +#define SVC_SERVE_MAX_BYTES (4u << 20) +#define SVC_SERVE_HIST 64 +static bool gsp_sema_serve_denied = true; +module_param_named(sema_serve_denied, gsp_sema_serve_denied, bool, 0644); +MODULE_PARM_DESC(sema_serve_denied, + "Serve a denied transfer-shaped DMA copy between guest-RAM operands by executing it in the emulator (default 1; session 153 CONT.6)"); +static struct { + u64 push_va; + u64 hash; +} svc_serve_hist[SVC_SERVE_HIST]; +static u32 svc_serve_hist_next; +static u32 svc_serve_log; /* bounded served witness */ +static u32 svc_serve_ref_log; /* bounded refusal witness */ +static u64 svc_serve_bytes; +static u64 svc_serve_ops; + +static bool svc_serve_seen(u64 push_va, u64 hash) +{ + u32 i; + + for (i = 0; i < SVC_SERVE_HIST; i++) + if (svc_serve_hist[i].push_va == push_va && + svc_serve_hist[i].hash == hash) + return true; + return false; +} + +static void svc_serve_store(u64 push_va, u64 hash) +{ + u32 i = svc_serve_hist_next++ % SVC_SERVE_HIST; + + svc_serve_hist[i].push_va = push_va; + svc_serve_hist[i].hash = hash; +} + +static bool svc_serve_denied_copy(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, + const struct svc_push_state *st, u32 dma, + u64 push_va, u64 pass_hash) +{ + u8 bounce[256]; + u8 head[32]; + u64 total = st->line_length; + u64 off, bytes = 0; + u64 hash = 1469598103934665603ULL; + int rc = 0; + + if (!total || total > SVC_SERVE_MAX_BYTES) + return false; + if (st->offset_in + total > svc_pa_max() || + st->offset_out + total > svc_pa_max()) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: refused h=0x%x push=0x%llx in=0x%llx out=0x%llx line=%u dma=0x%08x (operand outside guest RAM)\n", + ch->h_object, push_va, st->offset_in, + st->offset_out, st->line_length, dma); + } + return false; + } + for (off = 0; off < total; off += sizeof(bounce)) { + size_t n = (size_t)min_t(u64, sizeof(bounce), total - off); + size_t b; + + rc = svc_read(vgpu, st->offset_in + off, bounce, n); + if (rc) + break; + for (b = 0; b < n; b++) { + hash ^= bounce[b]; + hash *= 1099511628211ULL; + } + rc = svc_write(vgpu, st->offset_out + off, bounce, n); + if (rc) + break; + bytes += n; + } + if (rc) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: refused h=0x%x push=0x%llx in=0x%llx out=0x%llx line=%u rc=%d (read/write failed)\n", + ch->h_object, push_va, st->offset_in, + st->offset_out, st->line_length, rc); + } + return false; + } + /* + * Verified landing: read the destination back and compare its first + * bytes with the source's. A mismatch means the GPA model of the + * operand pair is wrong for this push - say so and do not latch it + * as served (the deny path then keeps its historical behaviour). + */ + rc = svc_read(vgpu, st->offset_in, bounce, sizeof(head)); + if (!rc) + rc = svc_read(vgpu, st->offset_out, head, sizeof(head)); + if (rc || memcmp(head, bounce, sizeof(head))) { + if (gsp_sema_dbg && svc_serve_ref_log < 16) { + svc_serve_ref_log++; + pr_info("sema-serve: declined h=0x%x push=0x%llx out=0x%llx (readback mismatch)\n", + ch->h_object, push_va, st->offset_out); + } + return false; + } + svc_serve_store(push_va, pass_hash); + svc_serve_bytes += bytes; + svc_serve_ops++; + if (svc_serve_log < 128) { + svc_serve_log++; + pr_info("sema-serve: served h=0x%x push=0x%llx dma=0x%08x in=0x%llx out=0x%llx bytes=%llu hash=0x%llx verify=ok n=%llu\n", + ch->h_object, push_va, dma, st->offset_in, + st->offset_out, bytes, hash, svc_serve_ops); + } + return true; +} + +/* + * SESSION 154 r1506 (viewport calibration): dump every viewport-family + * method run a trusted push carries. The replayed 3D windows carry none + * (the client's per-frame stream is a mid-stream cut); the state lives in + * a once-per-context push the capture does not always preserve. This + * read-only census names the real values when they do appear: the 0x0a00 + * scale/offset block, the 0x0c00/0x0c04 clip rect, the 0x1924/0x192c/ + * 0x193c pixel/clip controls, and the 0x1618 BEGIN topology. Deduped by + * (method, value) so a hot frame does not flood the ring. + */ +#define SVC_GL_VP_LOG_MAX 48u +static u32 svc_gl_vp_sigs[SVC_GL_VP_LOG_MAX]; +static u64 svc_gl_vp_vals[SVC_GL_VP_LOG_MAX]; +static u32 svc_gl_vp_sig_n; + +static bool svc_gl_vp_is_viewport(u32 m) +{ + return (m >= 0x0a00u && m <= 0x0a14u) || m == 0x0c00u || + m == 0x0c04u || m == 0x1924u || m == 0x192cu || + m == 0x193cu || m == 0x1618u; +} + +static void svc_gl_vp_scan(const u8 *buf, u32 len, u64 push_va, u32 h_object) +{ + u32 off = 0; + + while (off + 4u <= len) { + u32 hdr, op, cnt, addr, span = 1u, k; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt && + (u64)off + 4u + (u64)cnt * 4u <= (u64)len) + span = 1u + cnt; + for (k = 0; k < span; k++) { + u32 m; + u32 v = 0; + u32 sig; + u64 key; + u32 s; + + if (k && op != 1u && op != 3u) + break; + m = addr + 4u * k; + if (op == 1u || op == 3u || op == 5u) { + if (!svc_gl_vp_is_viewport(m)) + continue; + if (k && (u64)off + 8u + (u64)k * 4u <= + (u64)len) + memcpy(&v, buf + off + 4u + 4u * k, 4); + } else if (op == 4u) { + if (!svc_gl_vp_is_viewport(addr)) + break; + m = addr; + v = cnt; + } else { + break; + } + sig = m ^ 0x9e3779b9u; + key = (u64)m << 32 | v; + for (s = 0; s < svc_gl_vp_sig_n; s++) + if (svc_gl_vp_sigs[s] == sig && + svc_gl_vp_vals[s] == key) + break; + if (s < svc_gl_vp_sig_n) + continue; + if (svc_gl_vp_sig_n < SVC_GL_VP_LOG_MAX) { + svc_gl_vp_sigs[svc_gl_vp_sig_n] = sig; + svc_gl_vp_vals[svc_gl_vp_sig_n] = key; + svc_gl_vp_sig_n++; + } + pr_info("sema-gl: VP-METHOD h=0x%x push=0x%llx method=0x%x op=%u value=%08x%s\n", + h_object, push_va, m, op, v, + (m == 0x1618u) ? " (BEGIN topology)" : ""); + } + off += span * 4u; + } +} + static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 len, u64 page_pa, bool allow_ops, struct svc_channel *ch, u32 entry_idx, @@ -8894,6 +21183,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, /* r391f: the inline-write once-only key needs the push's identity. */ svc_pass_hash = svc_push_hash(buf, len); + /* SESSION 149 R17 (M4): per-pass reset of the standalone 0x180/0x1b0 + * upload pair (the tracker pairs one run with its successor). */ + svc_gl_spot_dst = 0; + svc_gl_spot_bytes = 0; /* * r316: pending releases record the channel they were decoded for; @@ -8901,6 +21194,8 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * and the matmul r322 starved/hung note). */ svc.cur_ch = ch->h_object; + /* r835: decode-scoped VAS for channel-first VA resolution. */ + svc_cur_vas = ch->h_vaspace; /* * r34x: the acquire-wait capture is PER RING ENTRY. It used to be * reset only at SET_OBJECT, so a decode pass over a later entry (or @@ -8930,6 +21225,11 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, u32 ops_runs = 0; u32 ops_seq = 0; /* ops appended this pass (sequence numbers) */ u32 off = 0; + /* SESSION 153-CONTINUED-10: draw-window tracker. First method of + * the current C597 run on the attached subchannel after the last + * foreign-subchannel (2D/present) run; see the BEGIN recognizer + * below. Per pass, like the rest of the decode state. */ + u32 gl_win_start = 0; /* m164: fresh CLASS-2 QMD witness for this decode pass. */ svc_gr_qmd_seen = 0; @@ -8948,6 +21248,64 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, allow_ops = false; } + /* + * r481: verbatim method-stream dump for large GR-class pushes + * (default 0 = off). See svc_push_raw_dump. + * SESSION 153: the small-push variant (sema_push_raw_small) covers the + * GL client's 16..511 B C597 pushes with its own budget. + */ + if (gsp_sema_push_raw && ch && !ch->engine_type && len >= 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + else if (gsp_sema_push_raw_small && ch && !ch->engine_type && + len >= 16 && len < 512) + svc_push_raw_dump(buf, len, push_va, ch->h_object); + + /* SESSION 153-CONTINUED-52: MME-family census for the beef channel. */ + if (ch) + svc_mme_census(buf, len, push_va, ch->h_object); + + /* SESSION 153-CONTINUED-101: fixed-CB scratch (0x3460) watch, any channel. */ + if (ch) + svc_scratch18_watch(buf, len, push_va, ch->h_object); + + /* r1515: apply the client's C597 constant-buffer upload (0x2390 + * runs) into the fixed-CB image before this pass's launch. */ + if (ch) + svc_gl_cb_upload_scan(vgpu, ch->h_object, buf, len, push_va); + + /* r1475: feed the client's real fixed-CB page into the forced MME + * seed page before this pass's launch (beef GL channels only). */ + if (ch) + svc_gl_cb_copy(vgpu, ch->h_object); + + /* SESSION 153-CONTINUED-94: init-push capture + local-memory census. */ + if (ch) + svc_init_push_scan(buf, len, push_va, ch->h_object); + + /* + * SESSION 153-CONTINUED-53: MME context-setup prelude capture. Only a + * trusted resolution feeds the buffer (the bytes are replayed into a + * hardware context); identical re-uploads are deduped inside. + */ + if (ch && trusted) + svc_mme_prelude_capture(buf, len, push_va, ch->h_object); + /* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude capture (same + * trusted-only rule; the runs are replayed into a hardware context). + */ + if (ch && trusted) + svc_attr_prelude_capture(buf, len, push_va, ch->h_object); + /* + * SESSION 153-CONTINUED-62: surface (render-target) setup capture - + * channel-agnostic, the X-screen bind lives on the X server's own + * channel. See svc_surf_prelude_capture. + */ + if (ch && trusted) { + svc_surf_prelude_capture(vgpu, buf, len, ch->h_vaspace, + ch->h_object, push_va); + svc_gl_vp_scan(buf, len, push_va, ch->h_object); + } + memset(&st, 0, sizeof(st)); st.push_va = push_va; /* @@ -8966,8 +21324,19 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * capture: the pump consumes a capture immediately after the pass that * built it, so anything still here is stale. Replay/rescue decodes * must not wipe a capture the pump has not consumed yet. + * + * r548: a valid, never-offered candidate is NOT stale - it is a launch + * whose forward was DEFERRED (r541/r543 guards, backend busy): the + * pump deliberately left it for a bounded retry. Wiping it here let + * the retry passes re-decode the entry as a replay (candidate already + * captured) and then consume it without any forward or completion + * (the r547 saxpy stall: both launches deferred, ring wiped by an + * interleaved fresh decode, zero GR LAUNCH / GR-COMPLETE-SW for 16+ + * minutes). Keep it; the capture path (svc_gr_cand_capture) rotates to + * a free ring slot when a different launch needs the current one. */ - if (!replay && trusted && ch->engine_type == 0) + if (!replay && trusted && ch->engine_type == 0 && + (!svc_gr_cand.valid || svc_gr_cand.offered)) memset(&svc_gr_cand, 0, sizeof(svc_gr_cand)); svc_fwd_rel_n = 0; svc_fwd_defer_armed = false; @@ -9000,6 +21369,17 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, len >= 4 ? *(const u32 *)(buf + 0) : 0, len >= 8 ? *(const u32 *)(buf + 4) : 0); } + /* r1481: cache a raw shader image (SPH magic first word). */ + if (buf && len >= 128u && len <= sizeof(svc_gl_sph_cache[0])) { + u32 w0 = *(const u32 *)buf; + int ci = (w0 == 0x020a0481u) ? 1 : + (w0 == 0x000a1482u) ? 0 : -1; + + if (ci >= 0) { + memcpy(svc_gl_sph_cache[ci], buf, len); + svc_gl_sph_cache_len[ci] = len; + } + } if (gsp_sema_dbg && svc_dbg_streams < 96) { svc_dbg_streams++; pr_info("sema-dbg: stream h=0x%x eng=%u va=0x%llx len=%u allow=%d w=%08x %08x %08x %08x %08x %08x %08x %08x\n", @@ -9037,6 +21417,55 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } } + /* + * SESSION 153-CONTINUED-12: attach on the draw marker itself. + * + * The section recogniser below needs the sticky C597 bind + * (svc_gl_att_h) that the boot-class GLX init push carries. On a + * boot where the class is bound channel-side (inside the emulated + * RM channel setup) and no decoded push re-emits SET_OBJECT C597, + * every draw-shaped push is dropped and no 3D replay ever runs. + * Adopt the first trusted push that carries an NVC597_BEGIN IMMD + * (op 4, addr 0x1618, subchannel 0) as the attached channel; the + * per-pass gl_win_start tracker then runs over that same push from + * its start, exactly as it does after an explicit bind, so the + * window keeps the pre-BEGIN state methods (vertex stream bases, + * colour target, constant-buffer selectors). + */ + if (gsp_sema_gl_xlate && !ch->engine_type && trusted && + !svc_gl_att_has(ch->h_object, 0)) { + u32 o2 = 0; + + while (o2 + 4 <= len) { + u32 h2, op2, cnt2, sc2, ad2; + + memcpy(&h2, buf + o2, 4); + op2 = (h2 >> 29) & 0x7; + cnt2 = (h2 >> 16) & 0xfff; + sc2 = (h2 >> 13) & 0x7; + ad2 = (h2 & 0x1fff) << 2; + if (op2 == 4 && ad2 == 0x1618 && sc2 == 0) { + svc_gl_att_note(ch->h_object, 0); + if (gsp_sema_dbg && svc_gl_draw_log < 64) { + svc_gl_draw_log++; + pr_info("sema-gl: attach-on-BEGIN h=0x%x push=0x%llx off=0x%x (no explicit C597 bind seen)\n", + ch->h_object, push_va, o2); + } + break; + } + if (op2 == 4 || op2 == 7) { + o2 += 4; + continue; + } + if ((op2 == 1 || op2 == 3 || op2 == 5) && cnt2 && + cnt2 <= 0xfff && o2 + 4 + cnt2 * 4 <= len) { + o2 += 4 + cnt2 * 4; + continue; + } + o2 += 4; + } + } + while (off + 4 <= len) { u32 hdr = 0, op, count, subch, addr; u32 v0 = 0, v1 = 0, v2 = 0, v3 = 0, v4 = 0; @@ -9048,6 +21477,44 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, addr = (hdr & 0x1fff) << 2; /* + * SESSION 153-CONTINUED-10 (task 1): draw-window tracking on + * the attached C597 channel. The per-frame CONTENT pushes + * interleave the 2D/present ops (subchannel 3) and the 3D + * block (subchannel 0); the replayable 3D window starts at + * the first subchannel-0 method after the last subchannel-3 + * run. The SET_REPORT_SEMAPHORE arms (0x1b00..0x1b0c) are + * excluded: they name client semaphore VAs (0x1_2000c000) + * the replay channel cannot translate (the r929 v1 fault + * class), so they must stay outside the sliced window. + */ + if (gsp_sema_gl_xlate && !ch->engine_type) { + u32 as; + + /* + * CONT.411: record the engine subchannel this run + * addresses (cla06fsubch.h 0..4) for every attached + * channel, then track the draw window on the section's + * own method subchannel. The previous form keyed the + * tracker on the SET_OBJECT bind's subchannel, so a + * channel that binds C597 on the 3D subchannel and + * emits its runs on another engine subchannel reset + * the window on every one of its own runs. + */ + if (svc_gl_att_has_any(ch->h_object)) + svc_gl_att_run_note(ch->h_object, subch); + as = svc_gl.seen ? svc_gl.subch : + svc_gl_att_subch_of(ch->h_object); + if (as != 0xffu) { + if (subch != as) + gl_win_start = 0; + else if (!gl_win_start && hdr && + !(op == 1 && addr >= 0x1b00 && + addr <= 0x1b0c)) + gl_win_start = off; + } + } + + /* * fail4: name every big method-data run and every run whose * address is the LOAD_INLINE_DATA method (any op/subch - the * encoding of the 64 KB H2D upload is what we are pinning). @@ -9191,17 +21658,6 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, bytes, ch->h_object); /* - * fail4: hardware semantics - a - * LOAD_INLINE_DATA run advances the - * destination cursor by the bytes it - * wrote. Without this, the consecutive - * runs of a split 64 KB upload all land - * at the first destination and the page - * ends up with only the last run's - * words (the RAWA junk). - */ - st.offset_out += bytes; - /* * fail4: name the header that follows * each big LID run - the split 64 KB * upload's other payload carriers sit @@ -9229,6 +21685,25 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, trusted, allow_ops, st.have_out, st.line_length, ch->h_object); } + /* + * fail4 (r526): hardware semantics - a + * LOAD_INLINE_DATA run advances the destination + * cursor by the bytes it carries whether or not + * the once-only gate let the write run. Without + * this, the consecutive runs of a split upload + * all land at the first destination (the old + * RAWA bug), and worse: on a re-decode pass where + * the gate suppresses the FIRST run, the cursor + * stays at the first run's destination and the + * SECOND run executes there - its data (the + * buffer's bytes 8..16) overwriting offset 0 + * (the observed tgemm XDIFF: X[0..1] <- + * X[2..3], deterministic, every pass). The gate + * suppresses the write, not the method stream's + * positional semantics. + */ + if (bytes) + st.offset_out += bytes; } /* * Record the destination and the payload bytes for @@ -9275,6 +21750,120 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0); } + /* + * SESSION 153-CONTINUED-10 (task 1): the client's per-frame + * CONTENT pushes carry real C597 draws but no per-push + * SET_OBJECT (the class bind is sticky), so the section + * recognizer above never opened for them and the 3D replay + * lane never saw a draw. Recognise the draw marker itself: + * NVC597_BEGIN (0x1618, an IMMD method - its immediate is + * the topology in the count field) on the attached + * subchannel of the attached channel opens the section. + * SESSION 153-CONTINUED-26: the glxgears-era client draw + * pushes carry no 0x1618 marker - their first run is the + * C597 GL render-target setup (INC 0x840, {1, rtVA, w, h, + * fmt, blksz}) followed by the 0x1b0 draw list - so the RT + * setup opens the section too. Both shapes are gated on + * the sticky attachment, the channel class, and a closed + * section. + * The window starts at gl_win_start (the first C597 state + * method after the last foreign-subchannel run); the state + * methods before the marker are re-scanned so their operand + * VAs (vertex stream bases 0x1c00/0x1c10, colour target + * 0x0800, constant buffer selectors 0x2380) enter the + * candidate's page set - without them the backend's shadow + * relocation cannot rewrite the window and the engine + * faults on an unreachable guest VA (the r929 v2 class). + * The window then runs from gl_win_start to the pass end + * (the section close at the END method / push tail), which + * svc_gl_flush slices into the verbatim 3D replay. + */ + if (gsp_sema_gl_xlate && !ch->engine_type && + ((op == 4 && addr == 0x1618) || + (op == 1 && addr == 0x840 && count >= 4)) && + !svc_gl.seen && + svc_gl_att_accept_run(ch->h_object, subch)) { + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + u32 wstart = gl_win_start ? gl_win_start : off; + + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.seen = true; + svc_gl.subch = subch; + svc_gl.start_off = wstart; + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + /* + * SESSION 153-CONTINUED-26: when the RT-setup run is + * the opener, its address word IS the draw's render + * target - put it into the candidate page set, or the + * forwarded replay's shadow relocation cannot rewrite + * the RT VA and the physical launch faults/times out + * (r990: `GR forward failed rc=-110` on every client + * draw while the X section's 7-pointer capture ran). + * Must run AFTER the section reset above. + */ + if (op == 1 && addr == 0x840 && off + 12 <= len) { + /* + * SESSION 153-CONTINUED-27: the run is + * [upper, lower, w, h, fmt, blksz] - the VA + * is ((upper & 0xff) << 32) | lower exactly + * like every other pair recognizer in + * svc_gl_field. Reading only the low word + * added 0x20400000 for a surface whose true + * VA is 0x120400000 (the X screen), so the + * backend's shadow relocation MISSed it and + * the physical replay wrote the raw VA with + * no PTE chain (r992 FAULT_PDE @ +0xa000). + */ + u32 v0 = 0, v1 = 0; + + memcpy(&v0, buf + off + 4, 4); + memcpy(&v1, buf + off + 8, 4); + svc_gl_add_ptr(((u64)(v0 & 0xff) << 32) | + (u64)(v1 & 0xfffffffc)); + } + if (wstart < off) + svc_gl_scan_window(wstart, off, buf, len, + subch); + if (gsp_sema_dbg && svc_gl_draw_log < 64) { + svc_gl_draw_log++; + pr_info("sema-gl: section open (draw) h=0x%x push=0x%llx subch=%u start=0x%x begin=0x%x len=%u trusted=%d\n", + ch->h_object, push_va, subch, wstart, + off, len, trusted); + } + } + + /* + * SESSION 153-CONTINUED-11: SET_PIPELINE_SHADER(j) as an IMMD + * (op 4) method - the GL driver emits it as a 1U method, so + * the TYPE field rides in the header's immediate (bits 27:16, + * value = [ENABLE(0) | TYPE(7:4)]). Sticky per slot, like the + * run-shaped case in svc_gl_field. + */ + if (gsp_sema_gl_xlate && op == 4 && addr >= 0x2000 && + addr < 0x2800 && (addr & 0x3f) == 0 && + ((svc_gl.seen && subch == svc_gl.subch) || + svc_gl_att_has(ch->h_object, subch))) { + u32 j = (addr - 0x2000) / 0x40; + + if (j < 32) + svc_gl_shader_type[j] = + (u8)((((hdr >> 16) & 0xfff) >> 4) & 0xf); + } + /* + * SESSION 153-CONTINUED-26: FERMI_TWOD (0x902d) IMMD_DATA + * methods on subchannel 3 (SET_DST_FORMAT 0x200, layout, + * SET_OPERATION 0x2ac, NOTIFY 0x10c, ...). The value rides + * in the header's bits 28:16. State only; no execution. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && op == 4 && + addr >= 0x0100 && addr <= 0x1400) + svc_twod_method(vgpu, ch, addr, + (hdr >> 16) & 0xfff, + allow_ops || trusted, push_va); + if (op == 4 || op == 7) { /* Header-only methods (immediate data in the header * for op 4; end-of-push segment for op 7). */ @@ -9330,6 +21919,199 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, continue; } + /* + * SESSION 149 R14 (sema_gl_xlate): GL/3D direct-stream section + * recognition. A SET_OBJECT (op 1, addr 0, one class word) + * binds a class to a subchannel; C597 (TURING_A, the observed + * GL class) on the channel subchannel opens the section. Any + * other class bind closes it (the copy-class C5B5 and 2D 902D + * binds surrounding the GL push must not open sections). The + * collected VA operands live in svc_gl until the section closes + * at the end of the pass or at the next class bind. + */ + if (gsp_sema_gl_xlate && op == 1 && addr == 0 && count == 1 && + off + 8 <= len) { + u32 cls; + + memcpy(&cls, buf + off + 4, 4); + if (cls == 0xc597) { + /* + * SESSION 149 R15 (M1.1a): carry the last + * valid release across the section reset. + * It was decoded from the bracketing 24 B + * release pushes BEFORE this class bind; + * the old wipe-and-open here left rel_va + * at 0 at flushtime, so the trusted first + * pass of the one-shot setup push was + * rejected by the flush guard and the + * section was never captured (r883/r886: + * exactly one trusted decode of the 1524 B + * push, zero captures). + */ + u64 keep_rel = svc_gl.rel_va; + u32 keep_pay = svc_gl.rel_payload; + + if (svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, + len, push_va, trusted, + replay, off); + memset(&svc_gl, 0, sizeof(svc_gl)); + svc_gl.seen = true; + svc_gl.subch = subch; + svc_gl.start_off = off; + svc_gl.rel_va = keep_rel; + svc_gl.rel_payload = keep_pay; + /* SESSION 153-CONTINUED-10: the bind is sticky - + * remember it so the class-space recognizer + * (BEGIN 0x1618, below) accepts this channel's + * later per-push-CONTENT pushes that carry no + * SET_OBJECT of their own. + * SESSION 153-CONTINUED-13: per-channel table - + * a foreign channel's periodic C597 bind must + * not steal this channel's attachment. */ + svc_gl_att_note(ch->h_object, subch); + /* + * SESSION 153: witness every C597 section open + * (bounded) - it is the recognizer-side proof + * that the client's own draw pushes are seen; + * the capture itself is witnessed by + * "sema-gl: section captured". + */ + if (gsp_sema_dbg && svc_gl_open_log < 32) { + svc_gl_open_log++; + pr_info("sema-gl: section open h=0x%x push=0x%llx subch=%u trusted=%d\n", + ch->h_object, push_va, subch, + trusted); + } + } else { + if (svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, + len, push_va, trusted, + replay, off); + /* A different class bound over the attached + * subchannel replaces the C597 bind: drop the + * attachment so the class-space recognizer + * cannot misread methods of the new class. + * SESSION 153-CONTINUED-13: clears only this + * channel's entry. */ + svc_gl_att_clear(ch->h_object, subch); + } + } else if (gsp_sema_gl_xlate && svc_gl.seen && + (subch == svc_gl.subch || + subch == SVC_SUBCH_3D)) { + /* + * CONT.411: collect operands on the section's own + * method subchannel and on the 3D subchannel (0) - + * the class bind and the state methods it precedes + * live there even when the run that opened the window + * addresses another engine subchannel. + * + * Deliberately NOT every run subchannel: feeding a + * foreign 2D run (subch 3 on a bind-on-0 section) into + * svc_gl_field would interpret 2D method payloads as + * C597 state/pointers and corrupt the section's page + * set. The run-subchannel rule applies to OPENING a + * section (above), not to operand collection. + */ + svc_gl_field(addr, count, buf + off + 4, + len - off - 4); + } + + /* + * SESSION 149 R17 (M4): standalone GL-class upload apply. The + * context-init upload push (the one-shot program upload) need + * not sit inside a recognised C597 section, but it carries + * the same subchannel-0 0x180 (LINE_LENGTH_IN) + 0x1b0 + * (LAUNCH_DMA + LOAD_INLINE_DATA) i2m shape. Pair the last + * 0x180 destination with each following 0x1b0 payload and + * apply it through the shared inline-write path (the engine + * never ran these commands). Once-only by (push, content + * hash, dst, bytes); the witness line is bounded. + */ + if (gsp_sema_gl_xlate && ch->engine_type == 0 && subch == 0) { + if (op == 1 && addr == 0x180 && count >= 4) { + u32 w2, w3; + + memcpy(&w2, buf + off + 12, 4); + memcpy(&w3, buf + off + 16, 4); + svc_gl_spot_dst = + ((u64)(w2 & 0xff) << 32) | + (u64)(w3 & 0xfffffffc); + memcpy(&svc_gl_spot_bytes, buf + off + 4, 4); + } else if (op == 5 && addr == 0x1b0 && count > 1 && + svc_gl_spot_dst && + (trusted || allow_ops)) { + u32 bytes = (count - 1) * 4; + + if (bytes && !svc_inline_once_seen(push_va, + svc_pass_hash, svc_gl_spot_dst, + bytes)) { + svc_inline_once_store(push_va, + svc_pass_hash, + svc_gl_spot_dst, bytes); + (void)svc_grce_inline_write(vgpu, + svc_gl_spot_dst, + buf + off + 8, bytes, + ch->h_object); + /* + * SESSION 154: the context-init + * upload pair sits OUTSIDE any + * C597 section (start_off is + * later), so it never reaches + * svc_gl.up[]. The program page + * of a later draw still needs + * these bytes: the booter finds + * them through the GRCE + * destination ring and attaches + * them at the bind VA. Record the + * pair there too, or the client's + * real VS image stays invisible + * and every beef window executes + * an empty VTG page. + */ + { + struct svc_grce_rec *rr = + &svc_grce_dst_ring + [svc_grce_dst_ring_next % + SVC_GRCE_DST_MAX]; + + rr->dst_va = svc_gl_spot_dst; + rr->len = min_t(u32, bytes, + GSP_BRIDGE_INLINE_BYTES); + memcpy(rr->data, buf + off + 8, + rr->len); + svc_grce_dst_ring_next++; + } + if (gsp_sema_dbg && + svc_gl_lid_log < 24) { + svc_gl_lid_log++; + pr_info("sema-gl-lid: h=0x%x push=0x%llx dst=0x%llx bytes=%u len=%u trusted=%d first=%08x %08x %08x\n", + ch->h_object, push_va, + svc_gl_spot_dst, bytes, + svc_gl_spot_bytes, + trusted, + *(const u32 *)(buf + off + 8), + off + 16 <= len ? *(const u32 *)(buf + off + 12) : 0, + off + 20 <= len ? *(const u32 *)(buf + off + 16) : 0); + } + } + } + } + + /* + * SESSION 153-CONTINUED-26: FERMI_TWOD NON_INC/ONE_INC runs + * on subchannel 3. The dialogs' whole body is NON_INC (the + * 17-word context window and the point pairs); decode the + * data words as cl902d methods before the generic + * whole-run skip below consumes them. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && (op == 3 || op == 5) && + addr >= 0x0100 && addr <= 0x1400) + svc_twod_run(vgpu, ch, addr, op == 5 ? true : false, + count, buf + off + 4, len - off - 4, + allow_ops || trusted, push_va); + if (op == 3 || op == 5) { /* * NON_INC / ONE_INC methods carry their own data @@ -9362,6 +22144,27 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, memcpy(&v4, buf + off + 20, 4); /* + * SESSION 153-CONTINUED-26: FERMI_TWOD INC runs on + * subchannel 3 (the RENDER_SOLID_PRIM mode/format/color + * triplet at 0x580, the DST group, clip). The three + * notify shapes (0x100 cnt<=1, 0x104, 0x108, 0x10c) stay + * with the watchdog handler below; everything else in the + * cl902d space is decoded here and must not fall into the + * CE method chain. + */ + if (gsp_sema_twod && !ch->engine_type && + subch == SVC_2D_SUBCH && op == 1 && + addr >= 0x0100 && addr <= 0x1400 && + !(addr == 0x0100 && count <= 1) && + addr != 0x0104 && addr != 0x0108 && addr != 0x010c) { + svc_twod_run(vgpu, ch, addr, true, count, + buf + off + 4, len - off - 4, + allow_ops || trusted, push_va); + off += 4 + count * 4; + continue; + } + + /* * OFFSET_IN_UPPER/LOWER and OFFSET_OUT_UPPER/LOWER are * contiguous (0x400..0x40c) and the driver pushes them as a * single INC_METHOD run (count 2 or 4). Apply every word of @@ -9408,6 +22211,96 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, continue; } + if (subch == SVC_2D_SUBCH && count >= 1 && addr >= 0x0100 && + addr <= 0x010c) { + /* + * r828: the RC watchdog's notifier cycle (class2d + * methods on subchannel 3). See svc_notify2d_write: + * without the zeroed notification the guest's + * krcWatchdog declares the GPU locked after 7 s and + * the X/GL stack wedges on screen readback. + */ + switch (addr) { + case SVC_2D_SET_NOTIFY_A: + ch->notify2d_va = (u64)(v0 & 0x1ffffff) << 32; + /* + * r934: the GL client's kick pushes the + * pair as ONE count-2 run at 0x104 (the + * methods 0x104 + 0x108 are contiguous): + * `20026041 00000001 2000e0b0` = + * SET_NOTIFY_A=1, SET_NOTIFY_B=0x2000e0b0 + * -> notification VA 0x12000e0b0. Fold the + * second data word in; the watchdog's + * separate count-1 runs are unchanged. + */ + if (count >= 2) + ch->notify2d_va |= (u64)v1; + /* + * SESSION 153-CONTINUED-18 (task 1): when + * this push also carries the C597 kick + * section, remember the arm so the section + * flush can execute the NOTIFY write (the + * 2D ack below is the watchdog's zero-ack; + * the kick member needs the completed + * NvNotification authored). + */ + if (gsp_sema_gl_xlate && svc_gl.seen) + svc_gl.kick_notify_va = + ch->notify2d_va; + /* + * r834: this method stream reached us through + * a REAL read (svc_read_last_pa); pin its page + * so the watchdog's push translation cannot go + * stale later - the content-scanned map answers + * a zero page once the injected pin ages out, + * staging exactly the starvation the ack must + * not depend on. + */ + if (svc_read_last_pa) + svc_pin_store(push_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + svc_read_last_pa & + ~(u64)(SEMA_PAGE_SIZE - 1), + SVC_APERTURE_SYS); + break; + case SVC_2D_SET_NOTIFY_B: + ch->notify2d_va |= (u64)v0; + break; + case SVC_2D_NOTIFY: + if (count == 1 && v0 == 0) { /* WRITE_ONLY */ + /* + * SESSION 153-CONTINUED-18: mark the + * kick section's NOTIFY (payload 0) so + * svc_gl_flush executes the completion + * write at the armed VA - the executor + * authors the NvNotification the 2D + * zero-ack alone does not. + */ + if (gsp_sema_gl_xlate && svc_gl.seen) { + svc_gl.kick_notify_val = v0; + svc_gl.kick_notify_seen = true; + } + if (svc_notify2d_write(vgpu, ch, + true)) { + /* + * r934: the notifier write IS + * this push's release edge + * (the GL client's kick car- + * ries no semaphore form); + * mark it so the push line + * shows have_rel=1. + */ + st.have_rel = 1; + } + } + break; + default: + break; + } + off += 4 + count * 4; + continue; + } + if (subch == 0 && addr == 0) { /* SET_OBJECT on the channel subchannel: a push starts */ n_ops = 0; @@ -9430,12 +22323,22 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if ((op == HOST_SEM_OP_RELEASE || op == HOST_SEM_OP_REDUCTION) && payload && - va >= SEMA_VA_LO && va < SEMA_VA_HI) { + (va >= SEMA_VA_LO || va >= SVC_LOW_REL_VA_LO) && + va < SEMA_VA_HI) { if (page_pa) svc_mark_hot(page_pa); svc.pushes_seen++; st.rel_va = va; st.rel_payload = payload; + /* SESSION 149 R14: keep the last VALID release + * decoded on this channel for the GL section + * synthesizer (the 24 B release pushes bracket + * the large GL launch push in ring order). + */ + if (gsp_sema_gl_xlate) { + svc_gl.rel_va = va; + svc_gl.rel_payload = payload; + } svc_wait_rel_va = va; svc_wait_rel_payload = payload; svc_wait_rel_off = off; @@ -9452,6 +22355,12 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.ring_ops++; ops_runs++; } + /* r566: a held entry's unexecuted device-dst + * copies must be visible to the launch guard. */ + if (svc.op_tr_pending) + svc_dstpend_note_ops(ops, n_ops, + ops_seq - n_ops, + ops_done); n_ops = 0; /* * A release to a GR completion counter must go @@ -9493,7 +22402,16 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.va = va; or.payload = payload; or.entry_idx = entry_idx; - if (svc_fwd_defer_arm() && + /* + * r824: the deferral needs a + * candidate that OWNS this push + * (see svc_fwd_cand_owns_push) - + * a release from a push with no + * forwarding has nothing to wait + * for and applies inline. + */ + if (svc_fwd_defer_arm(push_va) && + svc_fwd_cand_owns_push(push_va) && !svc.op_tr_pending) { svc_opdef_drop(vgpu, ch, entry_idx); @@ -9617,7 +22535,9 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; va = ((u64)st.rs_upper << 32) | st.rs_lower; - if (va < SEMA_VA_LO || va >= SEMA_VA_HI) + if ((va < SEMA_VA_LO && + va < SVC_LOW_REL_VA_LO) || + va >= SEMA_VA_HI) break; if (page_pa) svc_mark_hot(page_pa); @@ -9653,6 +22573,10 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, svc.ring_ops++; ops_runs++; } + if (svc.op_tr_pending) + svc_dstpend_note_ops(ops, n_ops, + ops_seq - n_ops, + ops_done); n_ops = 0; /* * ISSUE-0002 (r62/r65): a captured @@ -9663,7 +22587,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * own it, and then it is recorded * instead of applied. */ - if (svc_fwd_defer_arm()) { + if (svc_fwd_defer_arm(push_va)) { svc_opdef_drop(vgpu, ch, entry_idx); if (svc_fwd_rel_n < SVC_FWD_REL_MAX) { @@ -9790,12 +22714,29 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } switch (addr) { case CE_M_SET_SEMAPHORE_A: + /* + * SESSION 153 CONT.8: the X DDX's CE release pushes + * the semaphore pair as ONE contiguous count-2 run at + * 0x240 (raw witness, h=0x1000013 va=0x120012434: + * `addr=0x240 cnt=2 v0=00000001 v1=2000c010` = the + * hi/lo halves of VA 0x12000c010, followed by a + * count-1 run at 0x248 with the payload). The old + * count>=3-only fold left sem_va_lo at 0, so every + * such release assembled the bogus VA 0x100000000 + * (hi<<32) and was deferred forever on an + * untranslatable address while the DDX's nvSemWait + * on the real slot spun (xwd -root / server-side + * readback wedged the whole Xorg). Fold the low + * half for count>=2; the payload stays on the 0x248 + * run (or v2 when a count-3 triple arrives). + */ st.sem_va_hi = v0; - if (count >= 3) { + if (count >= 2) { st.sem_va_lo = v1; - st.sem_payload = v2; st.have_sem = 1; } + if (count >= 3) + st.sem_payload = v2; break; case CE_M_SET_SEMAPHORE_B: st.sem_va_lo = v0; @@ -9873,23 +22814,170 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, break; case CE_M_LINE_LENGTH_IN: st.line_length = v0; + /* + * r458 THE memset2d root cause: the driver encodes + * LINE_LENGTH_IN (0x418) and the adjacent LINE_COUNT + * (0x41c) as ONE count=2 method group - observed raw + * push bytes `20028106 00001000 00000100` = header + * (subch 4, count 2, addr 0x418), line length 4096, + * line count 0x100 = 256. Reading only v0 left + * line_count = 0, so the 2D-copy builder below took + * the single-row path: one 4 KB op per push, and a + * 4 MB cudaMemcpy2D delivered exactly one page per + * 1 MB block (memset2d probe: per-MB ok=4096, + * bad=12.5M/12.58M). The GRCE variant below has + * always decoded the count=2 form; the CE path did + * not. + */ + if (count >= 2) + st.line_count = v1; break; /* r402: 2D geometry (cudaMemset2D / cudaMemcpy2D). */ case CE_M_LINE_COUNT: st.line_count = v0; break; case CE_M_PITCH_IN: + /* + * SESSION 153 CONT.9 (r943): the X DDX's per-frame + * present blit pushes PITCH_IN+PITCH_OUT+ + * LINE_LENGTH_IN as ONE count-3 INC run at 0x410 + * (raw witness, h=0x1000013 va=0x120015578: + * `addr=0x410 cnt=3 v0=00001400 v1=00000c80 + * v2=00000c80` = pitch_in 5120, pitch_out 3200, + * line_length 3200, followed by LINE_COUNT 10 at + * 0x41c). Reading only v0 left line_length 0, so + * the transfer-shaped LAUNCH_DMA (dma=0x30d xfer= + * pipelined multi-line) built NO op - both the + * allow-path builder and the r939 denied-copy serve + * gate on st.line_length - and every per-frame + * present copy was dropped (`sema-ldatrace ... + * line=0`). Same batched-run fix as r458 did for + * LINE_LENGTH_IN/LINE_COUNT at 0x418. + */ st.pitch_in = v0 & 0xffffff; st.have_pitch_in = 1; + if (count >= 2) { + st.pitch_out = v1 & 0xffffff; + st.have_pitch_out = 1; + } + if (count >= 3) + st.line_length = v2; break; case CE_M_PITCH_OUT: st.pitch_out = v0 & 0xffffff; st.have_pitch_out = 1; break; + /* + * SESSION 153 CONT.25: SRC_ORIGIN_X/Y (0x744/0x748) and + * DST_ORIGIN_X/Y (0x74c/0x750), pushed as a count-2 (X,Y) + * or count-4 (X,Y,X,Y) INC run (nv_xaa_kepler-style + * SetSrcOrigin: x already scaled by compSizeFactor == Bpp + * for this class). No other class uses these method + * offsets (cl902d/cla06f clear). + */ + case CE_M_SRC_ORIGIN_X: + case CE_M_SRC_ORIGIN_Y: + case CE_M_DST_ORIGIN_X: + case CE_M_DST_ORIGIN_Y: + { + u32 vv[4] = { v0, v1, v2, v3 }; + u32 k, n = count > 4 ? 4 : count; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case CE_M_SRC_ORIGIN_X: + st.src_origin_x = vv[k]; + st.have_src_origin = 1; + break; + case CE_M_SRC_ORIGIN_Y: + st.src_origin_y = vv[k]; + st.have_src_origin = 1; + break; + case CE_M_DST_ORIGIN_X: + st.dst_origin_x = vv[k]; + st.have_dst_origin = 1; + break; + case CE_M_DST_ORIGIN_Y: + st.dst_origin_y = vv[k]; + st.have_dst_origin = 1; + break; + } + } + } + break; case CE_M_LAUNCH_DMA: { u32 dma = v0; u32 transfer = DMA_DATA_TRANSFER_TYPE(dma); u32 n_ops_before = n_ops; + bool dma_served = false; + + /* + * SESSION 153 CONT.25: remember a full setup (this is + * the push that carried LINE_LENGTH + OFFSET_IN) as + * the channel's persistent CE state, then fill a + * transfer-shaped DELTA launch (the driver's per-strip + * follow-up: no LINE_LENGTH/OFFSET_IN of its own, + * OFFSET_OUT and the semaphore release present - the + * witnessed 52-byte pushes on h=0x1000013) from it so + * it builds its op instead of being silently dropped. + */ + if (allow_ops && st.line_length && st.have_in) + svc_ce_saved_store(&st, ch->h_object); + if (transfer != 0 && allow_ops && st.have_out && + st.have_sem && + (!st.line_length || !st.have_in)) { + struct svc_ce_saved *s = + svc_ce_saved_slot(ch->h_object); + + if (s && s->line_length && s->have_in) { + if (!st.line_length) { + st.line_length = + s->line_length; + st.line_count = + s->line_count; + } + if (!st.have_in) { + st.offset_in = + s->offset_in; + st.have_in = 1; + } + if (!st.have_pitch_in && + s->have_pitch_in) { + st.pitch_in = s->pitch_in; + st.have_pitch_in = 1; + } + if (!st.have_pitch_out && + s->have_pitch_out) { + st.pitch_out = s->pitch_out; + st.have_pitch_out = 1; + } + if (!st.have_src_mode && + s->have_src_mode) { + st.src_phys_mode = + s->src_phys_mode; + st.have_src_mode = 1; + } + if (!st.have_dst_mode && + s->have_dst_mode) { + st.dst_phys_mode = + s->dst_phys_mode; + st.have_dst_mode = 1; + } + if (gsp_sema_dbg && + svc_cedelta_log < 64) { + svc_cedelta_log++; + pr_info("sema-cedelta: h=0x%x push=0x%llx dma=0x%08x filled line=%u lc=%u in=0x%llx pi=%u po=%u sx=%u sy=%u (engine state inherited)\n", + ch->h_object, push_va, + dma, st.line_length, + st.line_count, + st.offset_in, + st.pitch_in, + st.pitch_out, + st.src_origin_x, + st.src_origin_y); + } + } + } if (gsp_sema_dbg && svc.lda_log < (gsp_sema_ops_log > 48 ? gsp_sema_ops_log : 48)) { @@ -9969,7 +23057,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.have_dst_mode && st.dst_phys_mode == CE_APERTURE_SYS && st.offset_out + (u64)st.line_length <= - SEMA_PA_MAX; + svc_pa_max(); /* r365: vidmem-window fill (see gsp_sema_winfill). */ bool winfill = gsp_sema_winfill && st.offset_out >= SVC_VIDMEM_VA_LO && @@ -9992,6 +23080,19 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, m->type = 0; m->dst = st.offset_out; + /* SESSION 153 CONT.25: blocklinear + * destination strip position. */ + if (st.have_dst_origin) { + u32 dp = (st.have_pitch_out && + st.pitch_out) ? + st.pitch_out : + st.line_length; + + m->dst += + (u64)st.dst_origin_y * + dp + + st.dst_origin_x; + } /* * r367: the driver's own memset push * (channel_utils.c) proves the @@ -10140,6 +23241,38 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, m->src_aperture = st.have_src_mode ? st.src_phys_mode : CE_APERTURE_VID; /* + * SESSION 153 CONT.25: a BLOCKLINEAR + * operand positions each strip with + * SRC/DST_ORIGIN (x in bytes, y in + * rows) instead of folding it into + * OFFSET_IN/OUT. The linear shadow + * holds the pixels linearly, so the + * effective operand is + * base + origin_y*pitch + origin_x. + */ + if (st.have_src_origin) { + u32 sp = (st.have_pitch_in && + st.pitch_in) ? + st.pitch_in : + st.line_length; + + m->src += + (u64)st.src_origin_y * + sp + + st.src_origin_x; + } + if (st.have_dst_origin) { + u32 dp = (st.have_pitch_out && + st.pitch_out) ? + st.pitch_out : + st.line_length; + + m->dst += + (u64)st.dst_origin_y * + dp + + st.dst_origin_x; + } + /* * r402: a 2D copy (cudaMemcpy2D): * LINE_COUNT rows of * LINE_LENGTH_IN bytes with @@ -10180,6 +23313,47 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, m->spitch = spitch; } } + /* + * r458 memset2d forensics: the + * driver's cudaMemcpy2D pushes decode + * LINE_LENGTH=4096 with PITCH_IN/OUT + * present but LINE_COUNT absent + * (lc=0), so only one 4 KB row per + * push executes and a 4 MB copy + * delivers 4 pages. Dump the raw push + * (bounded) for every pitch-shaped + * copy so the method stream that + * carries the row count is on record. + */ + if (gsp_sema_dbg && + (st.have_pitch_out || + st.have_pitch_in) && + svc.op2d_raw_log < 32) { + u32 dl = len < 220 ? len : 220; + + svc.op2d_raw_log++; + pr_info("sema-op2d-raw: h=0x%x push=0x%llx len=%u off=0x%x line=%u lc=%u po=%u/%u pi=%u/%u out=0x%llx in=0x%llx bytes=%u dma=0x%08x\n", + ch->h_object, push_va, + len, off, + st.line_length, + st.line_count, + st.have_pitch_out, + st.pitch_out, + st.have_pitch_in, + st.pitch_in, + m->dst, m->src, + (u32)m->bytes, dma); + /* %*phN caps at 64 bytes; + * print the push whole in + * 64-byte halves. */ + pr_info("sema-op2d-raw: raw[0] %*phN\n", + dl > 64 ? 64 : dl, + buf); + if (dl > 64) + pr_info("sema-op2d-raw: raw[1] %*phN\n", + dl - 64, + buf + 64); + } n_ops++; ops_seq++; } else if (gsp_sema_dbg && svc.dma_skip_log < 32) { @@ -10208,6 +23382,57 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, } /* + * SESSION 153 CONT.6 (task 2): execute the copy a + * denied pass would otherwise refuse (see + * svc_serve_denied_copy). Placement: after the + * allow-path builder (a built op means the copy is + * already owned by svc_run_ops), before the owed/ + * pending marks. A successful serve means nothing is + * owed - the dstpend note and the op_tr_pending mark + * below are suppressed via dma_served so the slot + * completes and later passes classify as replays. + */ + if (gsp_sema_serve_denied && transfer != 0 && + n_ops == n_ops_before && trusted && !replay && + !ch->engine_type && st.line_length && st.have_out && + st.have_in && !DMA_REMAP_ENABLE(dma) && + !svc_serve_seen(push_va, svc_pass_hash) && + svc_serve_denied_copy(vgpu, ch, &st, dma, push_va, + svc_pass_hash)) { + dma_served = true; + svc_dstpend_clear(st.offset_out); + } + + /* + * r567: a trusted, transfer-shaped push that decoded + * NO op this pass (its operands are not resolved yet: + * the r569 mail b-copy needed have_in, which the + * staging upload had not produced) must (a) keep its + * slot incomplete so a later pass runs the copy + * instead of replay-suppressing it forever (op_owed, + * the r540 rule), and (b) make its stale destination + * visible to the r541 launch guard, which defers any + * launch reading that page until the copy runs or the + * dstpend window expires. The release still applies + * (no hold is armed: a push whose trust never arrives + * must not wedge the pump) - the guard, not the + * release, provides the ordering. + */ + if (allow_ops && transfer != 0 && trusted && ch && + n_ops == n_ops_before && st.have_out && + st.offset_out >= SVC_VAS_FLOOR) { + svc_dstpend_note(st.offset_out); + svc.op_owed = 1; + if (svc.op_owed_log < 64) { + svc.op_owed_log++; + pr_info("sema-pend: OWED-PUSH h=0x%x push=0x%llx out=0x%llx line=%u (transfer-shaped, no op; dst noted for the launch guard)\n", + ch->h_object, push_va, + st.offset_out, + st.line_length); + } + } + + /* * r365 FILL-RAW: the fill class that still decodes no * op (window VA, no remap const decoded) - dump the * raw push and decode its method stream so the true @@ -10324,13 +23549,47 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * soon as the trust (or the operand translation) * arrives; the bounded SVC_OPDEF_HOLD_MS expiry keeps * the pre-existing skip as the last resort. + * + * r610: a REPLAY pass must be exempt. A replay is the + * pump re-walking an entry whose ops were already + * recorded complete for this identity (entry_replay = + * same && slot_ops_complete): re-marking it as pending + * re-arms a 3 s opdef hold for a copy that HAS run, + * and the fresh pass that follows re-executes the op - + * a replay/fresh/replay cycle (r609 boot, idx=94: the + * d_b H2D re-marked and re-run every ~0.3 s from + * 368841 while the vadd launch waited; the r606 + * captured held chain is the same shape). Measured + * r604-r609: 100% of op_tr_pending marks (27/27, + * 71/71, 72/72) came from replay passes of already- + * executed copies - each one stalled the whole copy + * channel behind it, which is the held-chain class. + * The r308 rationale (pre-r595 the eng==0 copy + * channel's fresh passes never admitted ops, so the + * replay re-mark WAS the retry hatch) no longer + * applies: r595's cap_trust admission runs a plausible + * copy on its fresh pass, and a genuinely failed copy + * keeps its slot incomplete through op_owed / the + * rc!=0 pending rule. The whole accounting block is + * skipped on replay: no dstpend note (the write ran), + * no staging re-snapshot, no hold. A fresh + * (non-replay) denied pass marks pending exactly as + * before. */ - if (!allow_ops && trusted && + if (!allow_ops && trusted && !replay && !dma_served && !ch->engine_type && transfer != 0 && st.line_length && st.have_out && st.have_in && !DMA_REMAP_ENABLE(dma) && st.offset_out >= SVC_VAS_FLOOR) { /* + * r541: the denied device-dst copy has NOT + * run; whatever release edge later consumes + * this entry, a GR launch must not read the + * target page until the copy executes (see + * svc_dstpend_note). + */ + svc_dstpend_note(st.offset_out); + /* * r314 (CONT.57 lead 2): capture the staging * source on the DENIED pass too. This is the * pipeline-order moment: the push was just @@ -10380,20 +23639,54 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * channel in 10 s hold cycles. */ if (entry_idx != SVC_ENTRY_NONE && - entry_idx < SVC_GPFIFO_MAX_BITS && - ch->slot_pend_tries[entry_idx] < 4) { - ch->slot_pend_tries[entry_idx]++; - svc.op_tr_pending = 1; - if (gsp_sema_dbg && - svc.pend_dbg_log < 512) { - svc.pend_dbg_log++; - pr_info("sema-pend: h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u replay=%d trusted=%d tries=%u -> op_tr_pending\n", - ch->h_object, push_va, - entry_idx, st.offset_out, - st.offset_in, - st.line_length, - replay, trusted, - ch->slot_pend_tries[entry_idx]); + entry_idx < SVC_GPFIFO_MAX_BITS) { + if (ch->slot_pend_tries[entry_idx] < 4) { + ch->slot_pend_tries[entry_idx]++; + svc.op_tr_pending = 1; + if (gsp_sema_dbg && + svc.pend_dbg_log < 512) { + svc.pend_dbg_log++; + pr_info("sema-pend: h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u replay=%d trusted=%d tries=%u -> op_tr_pending\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + replay, trusted, + ch->slot_pend_tries[entry_idx]); + } + } else { + /* + * r540: the cap stopped the + * re-hold, but the op is still + * owed. Leaving the slot + * complete would classify every + * later pass as a replay and + * suppress the copy forever - + * the client waits on a copy + * that never ran (the r539 + * ~5-minute device-init stall: + * a GRCE H2D denied while + * untrusted, capped at 4 tries, + * then replay-suppressed after + * the pin arrived). Mark the + * pass owed so the slot stays + * INCOMPLETE and the next + * fresh pass (injected or + * translated) executes the + * copy. No hold is armed, so a + * slot whose trust never arrives + * still cannot wedge the pump. + */ + svc.op_owed = 1; + if (svc.op_owed_log < 64) { + svc.op_owed_log++; + pr_info("sema-pend: OWED h=0x%x push=0x%llx idx=%u out=0x%llx in=0x%llx line=%u trusted=%d (cap hit; slot stays incomplete for a fresh retry)\n", + ch->h_object, push_va, + entry_idx, st.offset_out, + st.offset_in, + st.line_length, + trusted); + } } } } @@ -10406,6 +23699,11 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, if (n_ops != n_ops_before) { struct svc_memset_op *lm = &ops[n_ops - 1]; + /* r611: this channel carries transfer work + * (see ever_ops) - the stream-order + * watermark's scope. */ + ch->ever_ops = true; + /* * fail4 CONT.53: H2D-shaped copies (virtual * device destination, virtual host source) @@ -10504,7 +23802,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, * in one op"). Bounded. */ if (lm->type == 1) - pr_info("sema-op2d: h=0x%x dst=0x%llx src=0x%llx bytes=%u rows=%u dpitch=%u spitch=%u line=%u lc=%u have_po=%u po=%u have_pi=%u pi=%u\n", + pr_info("sema-op2d: h=0x%x dst=0x%llx src=0x%llx bytes=%llu rows=%u dpitch=%u spitch=%u line=%u lc=%u have_po=%u po=%u have_pi=%u pi=%u so=%u/%u do=%u/%u\n", ch->h_object, lm->dst, lm->src, lm->bytes, lm->rows, lm->dpitch, @@ -10514,7 +23812,11 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, st.have_pitch_out, st.pitch_out, st.have_pitch_in, - st.pitch_in); + st.pitch_in, + st.have_src_origin, + st.src_origin_y, + st.have_dst_origin, + st.dst_origin_y); } } @@ -10833,7 +24135,7 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, or.payload = st.sem_payload; or.entry_idx = entry_idx; or.dma = dma; - if (svc_fwd_defer_arm()) { + if (svc_fwd_defer_arm(push_va)) { svc_opdef_drop(vgpu, ch, entry_idx); if (svc_fwd_rel_n < SVC_FWD_REL_MAX) { @@ -10978,6 +24280,14 @@ static void svc_decode_stream(struct nv_vgpu_device *vgpu, const u8 *buf, off += 4 + count * 4; } + /* + * SESSION 149 R14: close an open GL/3D section when the pass ends - + * this is the normal capture point (one section per large GL push). + */ + if (gsp_sema_gl_xlate && svc_gl.seen) + svc_gl_flush(vgpu, ch, &st, buf, len, push_va, trusted, + replay, off); + if (gsp_sema_dbg && ch->h_object && svc.entry_log < 16384) { svc.entry_log++; @@ -11140,9 +24450,24 @@ static void svc_pending_add(u64 va, u32 payload, u32 op) * exist; drop them instead of resolving them against the next session. */ static void svc_gr_diag_reset(void); /* m150: defined with the fill diag */ +/* r604: defined next to the per-VA defer tables; clears both on session end + * (they are keyed by push VA alone and the UVM pool reuses those VAs). */ +static void svc_gr_defer_reset(void); static void svc_session_flush(void) { + /* r707x: per-client witness budgets. */ + svc_pushfail_log = 0; + svc_pinrej_log = 0; + svc_rdw_log = 0; + svc_pushdec_log = 0; + /* SESSION 132: util-sweep witnesses + their dedup keys are per-client. */ + svc_util_dec_log = 0; + svc_gr_util_log = 0; + memset(svc_util_dec_keys, 0, sizeof(svc_util_dec_keys)); + /* SESSION 143: the per-channel sweep-visibility keys are per-client. */ + svc_util_seen_n = 0; + memset(svc_util_seen_keys, 0, sizeof(svc_util_seen_keys)); unsigned long flags; u32 i; @@ -11161,6 +24486,7 @@ static void svc_session_flush(void) */ { bool live = false; + bool forced = READ_ONCE(svc_sess_flush_forced); spin_lock_irqsave(&svc.chan_lock, flags); for (i = 0; i < SVC_MAX_CHANNELS; i++) @@ -11169,15 +24495,25 @@ static void svc_session_flush(void) break; } spin_unlock_irqrestore(&svc.chan_lock, flags); - if (live) { + if (live && !forced) { pr_info("sema-svc: session flush skipped: live channel(s) present (deferred flush raced a new session registration)\n"); return; } + if (forced) { + svc_sess_flush_forced = 0; + pr_info("sema-svc: session flush (forced) executing with live=%d channel(s)\n", + live); + } } svc_gr_diag_reset(); /* m150: per-session diagnostic budgets */ + svc_gr_defer_reset(); /* r604: per-session defer latches */ + svc_grpark_reset(); /* r633: per-session parked GR releases */ + svc_gr_synth_ep_reset(); /* r633: per-session synthetic budgets */ + svc_dst_guard_reset(); /* r609: per-session device-dst windows */ svc.map_n = 0; svc_pb_n = 0; + svc_pb_gr_done = 0; svc_pd_found = 0; svc_pd_attempts = 0; svc_pd_best = 0; @@ -11244,6 +24580,7 @@ static void svc_session_flush(void) svc_pb_last = 0; svc_pb_last_n = 0; svc_gr_ring_done = 0; + svc_gr_ring_scan_done = 0; svc_ring_n = 0; /* * The learned tables are session state too: the primary VA->PA pin @@ -11269,8 +24606,37 @@ static void svc_session_flush(void) * name the previous client's pages (ISSUE-0002 fan-out). */ memset(svc_ipin_alt, 0, sizeof(svc_ipin_alt)); + /* + * r820: PROMOTE_CTX records name one client's kernel buffers, and a + * new client's channels re-announce their own (the same VAs can back + * with different PAs across sessions): session state. A flush skips + * entirely while any channel is live, so a running client's records + * cannot be wiped under it. + */ + memset(svc_promo, 0, sizeof(svc_promo)); + svc_promo_seq = 0; + svc_promo_hit_log = 0; memset(svc_cnt_der, 0, sizeof(svc_cnt_der)); svc_cnt_der_n = 0; + /* r551: delivered-release marks are per-session (same handles reused). */ + memset(svc_opdef_done, 0, sizeof(svc_opdef_done)); + svc_opdef_done_next = 0; + /* + * r565: the GR exactly-once memories are boot-scoped caches keyed + * (partly) by push VA, and the UVM pool reuses the same launch push + * VAs for every client of the boot. svc_gr_cap_hist matches on the VA + * ALONE (8 entries, no hash), so a previous client's capture could + * classify a later client's fresh launch as "already captured" and + * skip it (`LAUNCH-SKIP (replay; already captured=1)`); the hash-keyed + * svc_gr_seen and the content-keyed svc_inline_once are lower risk but + * share the same lifetime. Reset all three per session - they refill + * within the session from the live client's own pushes. + */ + memset(svc_gr_cap_hist, 0, sizeof(svc_gr_cap_hist)); + svc_gr_cap_hist_pos = 0; + memset(svc_gr_seen, 0, sizeof(svc_gr_seen)); + memset(svc_inline_once, 0, sizeof(svc_inline_once)); + svc_inline_once_next = 0; /* * Re-arm the diagnostic log budgets per session. The first client of * a boot would otherwise spend every budget and the next session's @@ -11334,6 +24700,16 @@ static void svc_session_flush(void) memset(svc_put_log_at, 0, sizeof(svc_put_log_at)); svc_userd_dbg = 0; memset(svc_userd_dbg_ch, 0, sizeof(svc_userd_dbg_ch)); + memset(svc_ce_firstsub_n, 0, sizeof(svc_ce_firstsub_n)); + memset(svc_ce_hold_since, 0, sizeof(svc_ce_hold_since)); + memset(svc_ce_firstsub_off, 0, sizeof(svc_ce_firstsub_off)); + svc_ce_initrel_done = 0; + memset(svc_ce_initrel_until, 0, sizeof(svc_ce_initrel_until)); + memset(svc_ce_initrel_next, 0, sizeof(svc_ce_initrel_next)); + memset(svc_ce_initrel_val, 0, sizeof(svc_ce_initrel_val)); + svc_ce_initrel_log = 0; + svc_fb_claim_reset(); + svc_ram_ring_owner_reset(); svc_userd_sweep_log = 0; memset(svc_userd_sweep_at, 0, sizeof(svc_userd_sweep_at)); memset(svc_userd_sweep_n, 0, sizeof(svc_userd_sweep_n)); @@ -11422,6 +24798,7 @@ static void svc_guest_reset_wipe(void) svc_pd_r_ev_hash = 0; svc_pd_r_n = 0; svc_pb_n = 0; + svc_pb_gr_done = 0; svc_pool_boot_n = 0; svc_pool_boot_next = 0; svc_pool_pa_n = 0; @@ -11758,7 +25135,6 @@ static u32 svc_apply_page(struct nv_vgpu_device *vgpu, u8 *buf, u64 pa_page) * *semaphore* address inside a release needs to resolve through the map. */ #define SVC_SCAN_WINDOW (32ULL << 20) /* 32 MB per pass */ -#define SVC_SCAN_CHUNK (64 * 1024) /* 16 pages per guest read */ #define SVC_MBUF_MAX 6 /* rescue pin candidates */ /* @@ -11807,8 +25183,8 @@ static void svc_apply_pushes(struct nv_vgpu_device *vgpu) if (!buf) return; } - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; + if (end > svc_pa_max()) + end = svc_pa_max(); svc_apply_hot(vgpu); @@ -11826,7 +25202,7 @@ static void svc_apply_pushes(struct nv_vgpu_device *vgpu) } } - svc_scan_cursor = (end >= SEMA_PA_MAX) ? 0 : end; + svc_scan_cursor = (end >= svc_pa_max()) ? 0 : end; } /* @@ -11853,9 +25229,9 @@ static void svc_apply_sweep_slice(struct nv_vgpu_device *vgpu) u32 pages = 0; bool complete; - if (end > SEMA_PA_MAX) - end = SEMA_PA_MAX; - complete = (end >= SEMA_PA_MAX); + if (end > svc_pa_max()) + end = svc_pa_max(); + complete = (end >= svc_pa_max()); /* * Back off after a full pass that applied nothing: a stalled channel @@ -11906,6 +25282,52 @@ static void svc_apply_sweep_slice(struct nv_vgpu_device *vgpu) } } +/* + * SESSION 153-CONT-34: watch the emulated BAR0 shadow's PCFG window + * (0x88000..0x89000) for doorbell writes the guest made directly to the + * mmap'd shadow (those never reach the trap path). Each changed word is fed + * to the doorbell path, whose only effect is waking the pump earlier - the + * pump itself stays per-pass. The first call for a guest snapshots the window + * without firing, so pre-existing register values (the emulation's own init + * writes) cannot ring. + */ +static u32 svc_bar0_pcfg_log; + +static void svc_bar0_pcfg_scan(struct nv_vgpu_device *vgpu) +{ + struct svc_guest_sess *sess; + size_t bsz = 0; + u8 *base = svc_bar0_shadow(vgpu, &bsz); + u32 i; + + if (!base || bsz < SVC_BAR0_PCFG_HI) + return; + sess = svc_guest_get(vgpu, true); + if (!sess) + return; + for (i = 0; i < SVC_BAR0_PCFG_WORDS; i++) { + u32 v; + + memcpy(&v, base + SVC_BAR0_PCFG_LO + i * 4, 4); + if (!sess->pcfg_have) { + sess->pcfg_snap[i] = v; + continue; + } + if (v == sess->pcfg_snap[i]) + continue; + sess->pcfg_snap[i] = v; + if (!v) + continue; /* a clear, not a doorbell */ + gsp_sema_service_doorbell(v); + if (svc_bar0_pcfg_log < 32) { + svc_bar0_pcfg_log++; + pr_info("vgsp-sema: PCFG doorbell (BAR0 shadow scan) off=0x%x val=0x%x\n", + SVC_BAR0_PCFG_LO + i * 4, v); + } + } + sess->pcfg_have = true; +} + static void svc_pb_note(struct nv_vgpu_device *vgpu, const u8 *buf, u64 pa_page) { u32 hdr, hi, lo, payload; @@ -12000,7 +25422,7 @@ static void svc_pb_discover(struct nv_vgpu_device *vgpu) svc_pb_passes++; svc_pb_last = jiffies; svc_pb_last_n = svc_pb_n; - for (gpa = 0; gpa < SEMA_PA_MAX; gpa += SVC_SCAN_CHUNK) { + for (gpa = 0; gpa < svc_pa_max(); gpa += SVC_SCAN_CHUNK) { u32 o; if (READ_ONCE(svc_sess_flush)) @@ -12230,9 +25652,23 @@ static int svc_pb_win_lookup(const struct svc_channel *ch, u64 va_page, { u32 slot, i; - if (!ch || !ch->engine_type || ch->ce_index >= 32) + if (!ch) return -ENOENT; - slot = 4 * ch->ce_index; + /* + * R10 (patch A): GR (eng==0) channels whose ring was content- + * validated own a ring-anchored window pair (svc_pb_pair_gr). + * The CE path is unchanged; a GR channel without a paired + * window is rejected exactly as before. + */ + if (!ch->engine_type) { + if (ch->gr_index >= 32 || !ch->gr_ram_ring) + return -ENOENT; + slot = SVC_PB_GR_SLOT_BASE | ch->gr_index; + } else { + if (ch->ce_index >= 32) + return -ENOENT; + slot = 4 * ch->ce_index; + } for (i = 0; i < svc_pb_n; i++) { u64 pa; @@ -12243,7 +25679,7 @@ static int svc_pb_win_lookup(const struct svc_channel *ch, u64 va_page, va_page >= svc_pb[i].va + 0x100000ULL) continue; pa = svc_pb[i].pa_page + (va_page - svc_pb[i].va); - if (pa >= SEMA_PA_MAX) + if (pa >= svc_pa_max()) return -ENOENT; *pa_page = pa; return 0; @@ -12301,7 +25737,14 @@ static bool svc_push_prefix_ok(const u8 *buf, size_t len) if (off + 4 > len) return true; /* all zeros: not refutable here */ op = (hdr >> 29) & 0x7; - if (op == 4 || op == 7) /* header-only methods */ + /* + * r715: op 0 (GRP0_USE_TERT: SET/STORE/USE_SUB_DEV_MASK) and op 2 + * (GRP2_USE_TERT) are 4-byte header-only formats per the class + * header; the GR submit pushes start with 0x0001fff0 + * (SET_SUBDEVICE_MASK), so a correct injected pin for such a page + * was rejected here and the read fell through to the walk. + */ + if (op == 0 || op == 2 || op == 4 || op == 7) /* header-only */ return true; if (op != 1 && op != 3 && op != 5) return false; @@ -12350,12 +25793,250 @@ static u32 svc_pushpg_split_log; * validated candidate). Reads stay defensively copied through * svc_read_ap()/vfio_dma_rw(), which enforce the aperture bounds. */ +/* r707d: bounded push-read failure witness (see below). */ +static u32 svc_pushfail_log; +/* + * r707e: last content-backed ring pair resolved by any channel. The VAS + * candidate verify is retried with it on EVERY pump pass: the parked + * COPY_SERVER_PDES candidate for the client's user VAS parks at + * device-create and must register before the submit's first push read + * (a failed GR entry read is never retried with a fresh VAS), but the ring + * resolution and the park race within one pass - the every-pass retry + * closes that race in a single boot. + */ +static u64 svc_lastgood_ring_va; +static u64 svc_lastgood_ring_pa; + + +/* + * SESSION 154 (M-GL1.2): the PA ceiling for the push read. The GL client's + * sysmem allocations live ABOVE SEMA_PA_MAX (the ring scan already reads to + * gsp_sema_gr_scan_top for exactly this client - r908: its ring page at gpa + * 0x231e5d000); the content anchor pinned its push page at pa 0x2328e8000 + * (r920, measured) and the old 6 GiB read bound then rejected the freshly + * pinned page (src=2 -> -EIO) while the stale window answered instead + * (src=3 garbage, every push "unreadable"). Use the same widened ceiling + * the scans use. + */ +static u64 svc_push_pa_ceiling(void) +{ + u64 top = READ_ONCE(gsp_sema_gr_scan_top); + u64 reg = gsp_phys_guest_ram_top(); + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PA_MAX) + top = SEMA_PA_MAX; + /* + * SESSION 153-CONTINUED-67: never let the ceiling exceed the highest + * GPA any registered guest-RAM range covers - a read above it can + * only return the backing file's padding zeros, never content. With + * the 8/10 GB guest layout (0-2 GB + 4-10 GB) the registered top is + * what it is and inside-range answers (like the r1127 stale walk at + * 0x23921d000) stay readable-but-zero; the repeated-zero + * invalidation above handles those, this bounds the rest. + */ + if (reg && reg < top) + top = reg; + return top; +} + +/* + * SESSION 153-CONTINUED-51: deterministic push-page resolution. + * + * The direct GL client's 5-7 KB window pushes routinely read back all + * zero through the trusted chain on some boots ("sema-pushzero ... + * rescued=0", r1033-class) while the SAME VA pages answer real content + * on other boots - the learned cache (lat) can shadow a page-table walk + * that has since moved, and the client reuses/reallocates its push VAs + * across frames. The launch-page chain (svc_gr_page_note) resolves + * through the ledger then a FRESH walk; the push read chain consults the + * learned cache FIRST, so a stale lat/pin entry shadows the fresh walk + * forever. When the trusted read answered an all-zero page, consult the + * same ledger/walk chain the launch pages use and, when a DIFFERENT page + * with non-zero content answers, adopt it AND heal the read sources + * (validated pin + lat) so every later read of this page is deterministic + * instead of a per-boot lottery. + * + * Only a page whose content differs from all-zero is adopted: a + * legitimately-zero page (padding) must never steal a correct + * translation. Returns 0 when a replacement page was adopted (pa_out / + * ap_out updated and `p` refilled), -ENOENT when nothing better exists. + */ +static int svc_push_page_heal(struct nv_vgpu_device *vgpu, u32 h_object, + u64 va, u64 net_pa, int net_ap, u8 *p, + size_t chunk, u64 *pa_out, int *ap_out) +{ + u64 page_va = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = va & (SEMA_PAGE_SIZE - 1); + u64 cur_page = net_pa & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 lpa = 0, wpa = 0; + u32 lap = 0; + int wap = SVC_APERTURE_SYS; + static u32 heal_log; + + /* 1. The ledger: the launch-page chain's landing authority. A page a + * host write or an earlier launch resolution landed on is the + * page the client's own chain agreed about - the read chain does + * not consult it at all. */ + if (!svc_ledger_lookup(page_va, &lpa, &lap) && + (lpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != cur_page) { + u32 map_ap = svc_ap_of((int)lap); + + if (svc_read_ap(vgpu, lpa + off, + map_ap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + p, chunk) == 0 && + memchr_inv(p, 0, chunk)) { + svc_pin_store(page_va, lpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)map_ap); + svc_lat_store(va, lpa + off, (int)map_ap); + *pa_out = lpa + off; + *ap_out = (int)map_ap; + if (heal_log < 64) { + heal_log++; + pr_info("sema-pushheal: h=0x%x va=0x%llx ledger pa=0x%llx (was 0x%llx, zero)\n", + h_object, (unsigned long long)va, + (unsigned long long)(lpa + off), + (unsigned long long)net_pa); + } + return 0; + } + } + + /* 2. A FRESH walk, bypassing the learned cache: the guest may have + * remapped the VA after the lat entry was written. */ + if (svc_walk_vaspaces(vgpu, page_va, &wpa, &wap) == 0 && + wpa != 0 && (wpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != cur_page) { + u32 map_ap = svc_ap_of(wap); + + if (svc_read_ap(vgpu, wpa + off, + map_ap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : SVC_APERTURE_SYS, + p, chunk) == 0 && + memchr_inv(p, 0, chunk)) { + svc_pin_store(page_va, wpa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)map_ap); + svc_lat_store(va, wpa + off, (int)map_ap); + *pa_out = wpa + off; + *ap_out = (int)map_ap; + if (heal_log < 64) { + heal_log++; + pr_info("sema-pushheal: h=0x%x va=0x%llx walk pa=0x%llx (was 0x%llx, zero)\n", + h_object, (unsigned long long)va, + (unsigned long long)(wpa + off), + (unsigned long long)net_pa); + } + return 0; + } + } + /* Nothing better: restore the zero state exactly (the caller's read + * filled p with zeros and must see zeros back). */ + memset(p, 0, chunk); + return -ENOENT; +} + +/* + * SESSION 153-CONTINUED-67: repeated all-zero read invalidation. + * + * The beef (direct GL 3D client) channel's pushes are re-read every pump + * pass; on the r1127/w2 boots the setup push and every draw window kept + * answering ALL-ZERO from the same cache-derived translation for the whole + * boot: "sema-pushdec: h=0xbeef0100 va=0x4001004 len=60216 src=4 + * pa=0x23921d000 ... unreadable=1 first8=00000000 00000000" and 64 + * "zero-head window ... (page mid-write; not captured)" rows - while the + * SAME VAs answered real content on other boots (r1126). A single zero + * read can be a legitimately-zero page (padding, a zeroed push slot), but + * the SAME page answering all-zero TWICE in a row through a cache-derived + * source is the stale-translation signature: the client zeroed, + * reallocated or rewrote the page and the cached pin/learned/walk answer + * keeps resolving to the dead page - every walk success re-stores its + * answer into the learned table, so nothing ever re-resolves. When the + * streak reaches two, drop the learned-cache entry (and a validated pin + * when that was the source) so the NEXT read re-runs the source chain + * (discovered map, registered-VAS walk, discovered walk, targeted + * resolve) instead of reusing the same answer; this read still returns + * what it read. Beef channels only (the GL client's push family this + * work targets); the drop is bounded and the witness bounded per boot. + */ +#define SVC_ZRW_MAX 256 +struct svc_zrw_rec { + u64 page_va; + u8 streak; +}; +static struct svc_zrw_rec svc_zrw_tab[SVC_ZRW_MAX]; +static u32 svc_zrw_next; +static u32 svc_zrw_log; + +static struct svc_zrw_rec *svc_zrw_slot(u64 page_va) +{ + u32 i; + + for (i = 0; i < SVC_ZRW_MAX; i++) + if (svc_zrw_tab[i].page_va == page_va) + return &svc_zrw_tab[i]; + i = svc_zrw_next++ % SVC_ZRW_MAX; + svc_zrw_tab[i].page_va = page_va; + svc_zrw_tab[i].streak = 0; + return &svc_zrw_tab[i]; +} + +static void svc_zrw_reset(u64 page_va) +{ + u32 i; + + for (i = 0; i < SVC_ZRW_MAX; i++) + if (svc_zrw_tab[i].page_va == page_va) { + svc_zrw_tab[i].streak = 0; + return; + } +} + +/* + * r1670: walk one VA through the VAS a channel declared at registration. + * The push-buffer read below uses this before the learned/pin chain: that + * chain is global over VASes, and several VASes map the same GPU VA (the GL + * client's CE channel 0xcaf00012 pushes at 0x120000000, which the learned + * map answered with a zero page from a different VAS while the channel's + * own VAS mapped the real method stream). Returns true on a walk hit. + */ +static bool svc_walk_vas_by_handle(struct nv_vgpu_device *vgpu, u32 h_vas, + u64 va, u64 *pa, int *ap) +{ + struct svc_vaspace local[SVC_MAX_VASPACES]; + unsigned long flags; + int i, n = 0; + + if (!h_vas) + return false; + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid && svc.vas[i].h_vaspace == h_vas) + local[n++] = svc.vas[i]; + spin_unlock_irqrestore(&svc.vas_lock, flags); + + for (i = 0; i < n; i++) { + int alt = local[i].ap == SVC_APERTURE_FB ? + SVC_APERTURE_SYS : SVC_APERTURE_FB; + + if (!svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, va, + local[i].ap, pa, ap, false) || + !svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, va, + alt, pa, ap, false)) + return true; + } + return false; +} + static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u64 va, void *buf, size_t len, bool skip_ipin) { u8 *p = buf; + /* r820: per-read; the decode reads it right after the call. */ + svc_read_last_promote = false; + while (len) { u64 page_va = va & ~(u64)(SEMA_PAGE_SIZE - 1); size_t chunk = min_t(size_t, len, @@ -12364,11 +26045,36 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, int ap = SVC_APERTURE_SYS; bool have_win, done = false; bool first = (p == buf); + bool rej = false; /* r1405: a source hit the PA ceiling */ int last_rc = -ENOENT; + int trusted_rc = 0; u32 src; have_win = !svc_pb_win_lookup(ch, page_va, &win_pa); + /* + * r1670: the VAS the channel declared at registration is the + * authority for its push buffer. The learned/pin chain below + * is global over VASes and can name a page from another + * address space that maps the same GPU VA. + */ + if (ch->h_vaspace) { + u64 vpa = 0; + int vap = SVC_APERTURE_SYS; + u64 in_page = va & (SEMA_PAGE_SIZE - 1); + + if (!svc_walk_vas_by_handle(vgpu, ch->h_vaspace, + page_va, &vpa, &vap) && + vpa && + !svc_read_ap(vgpu, vpa + in_page, vap, p, + chunk)) { + pa = vpa + in_page; + svc_read_last_src = 6; + svc_read_last_pa = pa; + done = true; + } + } + for (src = 0; src < 5 && !done; src++) { bool via_ipin = false; bool untrusted = false; @@ -12418,12 +26124,38 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, default: /* learned cache / discovered map / page * directories / targeted rm_mem resolve */ - if (svc_lookup_trusted(vgpu, va, &pa, &ap)) + trusted_rc = svc_lookup_trusted(vgpu, va, + &pa, &ap); + if (trusted_rc) continue; break; } - if (ap == SVC_APERTURE_SYS && pa >= SEMA_PA_MAX) { + /* + * r1405 (S153/r1404): the strict GL carrier pins are + * host-verified ground truth (guest pagemap), but they + * live ABOVE the registered guest-RAM top (8.0-8.35 GB + * vs a 6 GiB boot registration), so this bound refused + * every pinned page with -EIO before reading it and the + * 60 216 B carrier read never decoded. An injected pin + * (src 0) is exempt: vfio_dma_rw is the hard bound, and + * a stale pin above the top either reads (the desired + * case) or fails in svc_read_ap exactly as before. The + * other sources keep the conservative bound, and a + * refused source is named by the bounded witness. + */ + if (ap == SVC_APERTURE_SYS && + !via_ipin && + pa >= svc_push_pa_ceiling()) { + rej = true; + if (svc_pinrej_log < 96) { + svc_pinrej_log++; + pr_info("sema-pinrej: va=0x%llx pa=0x%llx src=%u ceiling=0x%llx\n", + (unsigned long long)va, + (unsigned long long)pa, + src, + (unsigned long long)svc_push_pa_ceiling()); + } last_rc = -EIO; continue; } @@ -12479,10 +26211,196 @@ static int svc_read_va_push_ex(struct nv_vgpu_device *vgpu, */ if (src == 3 && first && !memchr_inv(p, 0, chunk)) continue; + svc_read_last_src = (u8)src; + svc_read_last_pa = pa; + /* + * SESSION 153-CONTINUED-46: zero-content rescue for the + * direct GL client's push pages. r1032/r1033 evidence: + * the beef family's 5-7 KB draws routinely read back + * ALL ZERO from the trusted chain (latch/map/walk) - + * "sema-pushdec: h=0xbeef0100 va=0xcae2894 len=7148 + * rc=0 src=4 pa=0x13d181000 ... first8=00000000 + * 00000000" - and the capture is then skipped as a + * zero-head window, so no 3D replay ever launches + * (42 such skips on one boot). This is the r817 class + * of stale/wrong kernel-buffer translations: the + * driver's OWN PROMOTE_CTX statement holds the real + * backing. When the first chunk of a big GL push reads + * all zero from the trusted chain (src >= 2; the + * host-verified injected pins are ground truth and + * stay authoritative), probe the promote booking: a + * DIFFERENT page with non-zero content is used in + * place, the learned cache is healed with it, and the + * read is tagged src=9 so the pushdec witness shows + * the rescue. Bounded per boot. + */ + if (src >= 2 && first && len >= 512u && + (ch->h_object & 0xffff0000u) == 0xbeef0000u && + !memchr_inv(p, 0, chunk)) { + u64 ppa = 0; + int pap = SVC_APERTURE_SYS; + static u32 pushzero_log; + + if (!svc_promo_lookup(page_va, &ppa, &pap) && + ((ppa & ~(u64)(SEMA_PAGE_SIZE - 1)) != + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)))) { + u64 rpa = ppa + + (va & (SEMA_PAGE_SIZE - 1)); + int rrc = svc_read_ap(vgpu, rpa, pap, + p, chunk); + + if (!rrc && memchr_inv(p, 0, chunk)) { + pa = rpa; + svc_read_last_src = 9; + svc_read_last_pa = rpa; + svc_lat_store(page_va, ppa, + pap); + if (pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx promo pa=0x%llx rescued=1\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, + src, + (unsigned long long)pa, + (unsigned long long)ppa); + } + } else { + /* promo page unusable: put the + * zero state back exactly. */ + memset(p, 0, chunk); + if (!rrc && pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx promo pa=0x%llx rescued=0 (promo content zero)\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, + src, + (unsigned long long)pa, + (unsigned long long)ppa); + } + } + } else if (pushzero_log < 1024) { + pushzero_log++; + pr_info("sema-pushzero: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx rescued=0 (no distinct promo booking)\n", + ch->h_object, + (unsigned long long)va, + (unsigned)len, src, + (unsigned long long)pa); + } + } + /* + * SESSION 153-CONTINUED-51: when the zero-content + * rescue above did not produce non-zero bytes, the + * page must still be resolved deterministically - + * run the SAME ledger/fresh-walk chain the launch + * pages use (svc_gr_page_note) for this page. A + * different page with non-zero content is adopted, + * tagged src=10, and pinned so every later read of + * the page (continuation chunks and future frames) + * answers from the healed source instead of the + * learned cache. This is the r1069 class: every + * window push read zeros for a whole boot while the + * launch pages resolved fine. + */ + if (src >= 2 && + (ch->h_object & 0xffff0000u) == 0xbeef0000u && + (first ? len >= 512u : chunk >= 512u) && + !memchr_inv(p, 0, chunk)) { + u64 hpa = pa; + int hap = ap; + + if (!svc_push_page_heal(vgpu, ch->h_object, + va, pa, ap, p, + chunk, &hpa, &hap)) { + pa = hpa; + svc_read_last_src = 10; + svc_read_last_pa = hpa; + } + /* Not healed: the original (zero) source and + * its label stay; nothing was adopted. */ + } + /* + * r820: name whether the trusted answer came from a + * driver-declared PROMOTE_CTX buffer; the decode uses + * it for the zero-NOP acceptance (see the flag). + */ + if (src == 4) { + u64 promo_pa = 0; + int promo_ap = SVC_APERTURE_SYS; + + svc_read_last_promote = + !svc_promo_lookup(page_va, &promo_pa, + &promo_ap); + } + /* + * SESSION 153-CONTINUED-67: repeated all-zero read + * invalidation (see the svc_zrw block before this + * function). Count consecutive all-zero answers per + * page from cache-derived sources (validated pin / + * window / trusted chain); on the second in a row + * drop the learned-cache answer (and a validated pin + * when that answered) so the next read re-walks + * instead of reusing the same dead translation. The + * current read still returns what it read. + */ + if ((ch->h_object & 0xffff0000u) == 0xbeef0000u && + src >= 2) { + if (!memchr_inv(p, 0, chunk)) { + struct svc_zrw_rec *z = + svc_zrw_slot(page_va); + + if (z->streak < 0xff) + z->streak++; + if (z->streak >= 2) { + z->streak = 0; + svc_lat_drop(page_va); + if (src == 2) + svc_pin_drop(page_va); + if (svc_zrw_log < 64) { + svc_zrw_log++; + pr_info("sema-zrw: h=0x%x va=0x%llx page=0x%llx src=%u pa=0x%llx sub=%u all-zero x2 -> drop lat%s, re-walk next read\n", + ch->h_object, + (unsigned long long)va, + (unsigned long long)page_va, + src, + (unsigned long long)pa, + svc_trust_src, + src == 2 ? "+pin" : ""); + } + } + } else { + svc_zrw_reset(page_va); + } + } done = true; } - if (!done) + if (!done) { + svc_read_last_src = 0xff; + if (svc_pushfail_log < 96) { + u64 ipa = 0, apa = 0; + int iap = 0, aap = 0; + bool has_ipin = + (svc_ipin_lookup(va, &ipa, &iap) == 0); + bool has_alt = + (svc_ipin_alt_lookup(va, &apa, &aap) == 0); + + svc_pushfail_log++; + pr_info("sema-pushfail: va=0x%llx chunk=0x%llx page=0x%llx last_rc=%d trusted_rc=%d ipin=%d pa=0x%llx alt=%d pa=0x%llx win=%d pa=0x%llx ceil=0x%llx rej=%d\n", + (unsigned long long)va, + (unsigned long long)chunk, + (unsigned long long)page_va, + last_rc, trusted_rc, has_ipin, + (unsigned long long)ipa, + has_alt, + (unsigned long long)apa, + have_win, + (unsigned long long)win_pa, + (unsigned long long)svc_push_pa_ceiling(), + rej); + } return last_rc; + } p += chunk; va += chunk; len -= chunk; @@ -12498,29 +26416,345 @@ static int svc_read_va_push(struct nv_vgpu_device *vgpu, } /* + * r1780 (CONT.430f): dev overrides for the direct-GL USERD experiment. + * `sema_userd_swap` reads GP_PUT at +0x88 and GP_GET at +0x8c (the + * orientation measured on the direct-GL client's control page), and + * `sema_userd_override` ("0x<handle>,0x<pa>") forces a channel's USERD + * candidate to a known sysmem page. Both are diagnostic-only defaults + * off; the normal path is unchanged. + */ +static bool gsp_sema_userd_swap; +module_param_named(sema_userd_swap, gsp_sema_userd_swap, bool, 0644); +MODULE_PARM_DESC(sema_userd_swap, + "Read GP_PUT at +0x88 and GP_GET at +0x8c for the USERD probe (direct-GL layout experiment); default 0"); + +static int svc_userd_override_set(const char *val, const struct kernel_param *kp) +{ + u32 h = 0; + u64 pa = 0; + unsigned long flags; + int i, hits = 0; + char *buf, *comma; + + buf = kstrdup(val, GFP_KERNEL); + if (!buf) + return -ENOMEM; + comma = strchr(buf, ','); + if (!comma) { + kfree(buf); + return -EINVAL; + } + *comma = '\0'; + h = (u32)simple_strtoul(buf, NULL, 0); + pa = simple_strtoull(comma + 1, NULL, 0); + kfree(buf); + if (!h || !pa) + return -EINVAL; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || (u32)c->h_object != h) + continue; + c->userd_pa = pa; + c->userd_as = SVC_APERTURE_SYS; + c->userd_ok = true; + c->userd_reg = false; + hits++; + pr_info("sema-chan: USERD override h=0x%x pa=0x%llx (swap=%d)\n", + h, pa, gsp_sema_userd_swap ? 1 : 0); + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return hits ? 0 : -ENOENT; +} + +static const struct kernel_param_ops svc_userd_override_ops = { + .set = svc_userd_override_set, +}; +module_param_cb(sema_userd_override, &svc_userd_override_ops, NULL, 0200); +MODULE_PARM_DESC(sema_userd_override, + "Force a channel USERD page (handle,pa), diagnostic only"); + +/* * Read the GPFIFO get/put pair from a candidate USERD. Returns 0 when the * values are plausible for a channel with `entries` GPFIFO slots. */ -static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, - u32 entries, u32 *put, u32 *get) +static int svc_userd_probe_ex(struct nv_vgpu_device *vgpu, u64 pa, u32 as, + u32 entries, u32 *put, u32 *get, bool loose_get) { u32 p = 0, g = 0; + u32 put_off = gsp_sema_userd_swap ? SVC_USERD_GP_GET : SVC_USERD_GP_PUT; + u32 get_off = gsp_sema_userd_swap ? SVC_USERD_GP_PUT : SVC_USERD_GP_GET; if (!pa) return -EINVAL; - if (svc_read_ap(vgpu, pa + SVC_USERD_GP_PUT, as, &p, sizeof(p))) + if (svc_read_ap(vgpu, pa + put_off, as, &p, sizeof(p))) return -ENOENT; - if (svc_read_ap(vgpu, pa + SVC_USERD_GP_GET, as, &g, sizeof(g))) + if (svc_read_ap(vgpu, pa + get_off, as, &g, sizeof(g))) return -ENOENT; *put = p; *get = g; - if (p > entries || g > p) + /* + * SESSION 153-CONTINUED-38: the kernel channels' control page can + * carry an uninitialised GP_GET (measured 0x201400 at fb 0x18fe00+0x88 + * while GPPut was a live 3) - GP_GET is GPU-owned ("read only" in the + * Kepler control page) and the driver does not zero it, exactly the + * CE-layout lesson (ISSUE-0002 r47: "the GET word is GPU-owned and its + * initial content is not coherent"). The pump tracks its own position + * in last_get and rewrites GP_GET as it consumes, so `loose_get` + * callers (large GR kernel rings only) accept the block on the PUT + * alone. + */ + if (p > entries || (!loose_get && g > p)) return -EINVAL; if (!p && !g) return -EAGAIN; /* plausible but idle: callers decide */ return 0; } +static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, + u32 entries, u32 *put, u32 *get) +{ + return svc_userd_probe_ex(vgpu, pa, as, entries, put, get, false); +} + +/* + * CONT.405 - BAR1-alias resolution for a declared FBMEM USERD. + * + * The channel alloc params name the control block in the FB aperture + * (userdMem.base + userdOffset + index/page), but that is the allocation + * site, not where the client writes. The guest RM maps the USERD memdesc + * into its BAR1 aperture with an offset chosen by its own BAR1 VA + * allocator (kbusMapFbAperture -> dmaAllocMapping): the client then writes + * GP_GET/GP_PUT through that BAR1 window. The emulated BAR1 is an identity + * shadow of FB in this module, so the write lands at the BAR1 offset and + * the declared FB block stays fill (measured: X channel wire 0x190600 is + * 0x01010100 fill while the live pair sits at 0x910000 -> page bias + * 0x780000; both X registrations in r1699 latched the same 0x910000). + * + * The bias lives in the guest, so it cannot be derived here; it is learned + * once from a block the FB shape scan has already validated against a + * channel's ring (svc_chan_userd latch) and then applied to later FBMEM + * registrations as a preferred candidate. Learning is page-granular and + * bounded: a bias that is not page-aligned, zero, or outside the shadow is + * refused. Until it is learned, resolution falls back to the wire block, + * the wire page base, and the existing scans. + */ +static u64 svc_bar1_alias_bias; /* BAR1 page offset - FB page offset */ +static u32 svc_bar1_alias_log; + +static void svc_userd_alias_learn(struct svc_channel *ch, u64 alias_pa) +{ + u64 wire, wp, ap, bias; + + if (!ch || !ch->userd_wire || svc_bar1_alias_bias) + return; + if (ch->userd_as != SVC_APERTURE_FB || !alias_pa) + return; + wire = ch->userd_wire; + wp = wire & ~0xfffULL; + ap = alias_pa & ~0xfffULL; + if (!wp || ap <= wp) + return; + bias = ap - wp; + if ((bias & 0xfff) || bias >= 0x10000000ULL) + return; + WRITE_ONCE(svc_bar1_alias_bias, bias); + pr_info("vgsp-sema: userd alias bias learned 0x%llx (h=0x%x wire=0x%llx alias=0x%llx; applied to later FBMEM userd)\n", + bias, ch->h_object, wire, alias_pa); +} + +/* + * Return the BAR1-alias candidate for a channel's declared FBMEM USERD. + * The client writes the control block at the mapped page's base + 0x88, so + * the page granularity (not the wire's index/page offset) is what the + * alias names; svc_userd_probe_ex still validates the pair. + */ +static bool svc_userd_bar1_alias(struct svc_channel *ch, u64 *alias) +{ + u64 bias = READ_ONCE(svc_bar1_alias_bias); + u64 wire = ch->userd_wire; + + if (!wire || ch->userd_as != SVC_APERTURE_FB || !bias) + return false; + *alias = (wire & ~0xfffULL) + bias; + if (!*alias || *alias > 0x10000000ULL) + return false; + if (svc_bar1_alias_log < 64) { + svc_bar1_alias_log++; + pr_info("vgsp-sema: userd alias h=0x%x wire=0x%llx alias=0x%llx\n", + ch->h_object, wire, *alias); + } + return true; +} + +/* + * SESSION 153-CONTINUED-34 (BAR0-shadow USERD): the direct GL client's + * channel allocation declares its USERD in the FB aperture at a 0x18fc00/ + * 0x18fe00-class offset, but the guest's GP_PUT/GP_GET writes for those + * channels land in the EMULATED BAR0 register shadow at that same offset + * (the vGPU guest driver programs its usermode/userd block through the + * register window; the FB shadow at the same offset holds nothing). Every + * existing source reads the FB/SYS shadows or scans guest RAM, so the block + * is invisible: svc_chan_userd returns -ENOENT, the pump never iterates the + * channel's GPFIFO ring and the channel retires with submissions=0. + * + * Watch policy (same spirit as the RAM-scan liveness latch): read the pair + * at bar0_shadow + userd_pa. A nonzero, coherent PUT is the guest's own + * write - the shadow is zeroed at device creation and no other writer + * targets that offset - so: + * - PUT >= 2 (two publishes already): adopt immediately; + * - PUT == 1 with a different value later (movement): adopt; + * - PUT == 1 stable past SVC_BAR0_USERD_PATIENT_MS: adopt anyway, so a + * client waiting on its single first submission cannot deadlock the + * watch (the pump then consumes that very submission); + * - unreadable/zero/incoherent: keep waiting (other sources unchanged). + * Adoption sets the BAR0 aperture so every later USERD read/write + * (svc_read_ap/svc_write_ap) and the pump's GP_GET write-back hit the same + * block the guest is updating. The walk seed mirrors the RAM-scan latch: + * the newest entries the client is waiting on, bounded to entries-1 slots. + */ +#define SVC_BAR0_USERD_PATIENT_MS 3000u + +static bool svc_bar0_userd_try(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 *put, u32 *get) +{ + size_t bsz = 0; + u8 *base = svc_bar0_shadow(vgpu, &bsz); + u64 off = ch->userd_pa; + u32 p = 0, g = 0, span; + + if (!base || !off) + return false; + if (off + SVC_USERD_GP_PUT + 4 > bsz) + return false; + /* + * Only the FB-declared family is retried here: SYS-declared offsets + * are real guest PAs the ordinary paths already read, and the CUDA/ + * kick channels resolve through their own sources before this point. + */ + if (ch->userd_as != SVC_APERTURE_FB) + return false; + + memcpy(&p, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&g, base + off + SVC_USERD_GP_GET, 4); + + if (gsp_sema_dbg && (p || g) && ch->bar0_userd_log < 8) { + ch->bar0_userd_log++; + pr_info("sema-chan: BAR0 userd probe h=0x%x rank=%u off=0x%llx put=%u get=%u\n", + ch->h_object, ch->engine_type ? ch->ce_index + : ch->gr_index, + off, p, g); + } + /* Idle, incoherent or an implausible register-looking value. */ + if (!p || g > p || p > 0x200000u) + return false; + + if (ch->bar0_userd_watch_pa != off) { + ch->bar0_userd_watch_pa = off; + ch->bar0_userd_watch_put = p; + ch->bar0_userd_watch_at = jiffies; + if (p >= 2) + goto adopt; + return false; /* first sighting, p == 1: patience */ + } + if (p != ch->bar0_userd_watch_put) + goto adopt; /* movement: live writer */ + if (time_after(jiffies, ch->bar0_userd_watch_at + + msecs_to_jiffies(SVC_BAR0_USERD_PATIENT_MS))) + goto adopt; /* stable nonzero past the patience window */ + return false; + +adopt: + ch->userd_ok = true; + ch->userd_as = SVC_APERTURE_BAR0; + if (ch->gpfifo_entries > 1 && p) { + span = p < ch->gpfifo_entries ? p : ch->gpfifo_entries - 1; + ch->last_get = (p % ch->gpfifo_entries + + ch->gpfifo_entries - span) % ch->gpfifo_entries; + } else { + ch->last_get = 0; + } + ch->bar0_userd_watch_pa = 0; + *put = p; + *get = g; + pr_info("sema-chan: BAR0 USERD h=0x%x eng=%u ADOPT off=0x%llx put=%u get=%u ap=bar0 (walk seed last_get=%u; guest GP_PUT lives in the emulated BAR0 shadow)\n", + ch->h_object, ch->engine_type, off, p, g, + (u32)ch->last_get); + return true; +} + +/* r912: guest-RAM USERD shape scan (defined next to the ring RAM scan). */ +static bool svc_gr_ram_userd_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out, + u32 *put_out, u32 *get_out); + +/* + * SESSION 153-CONTINUED-37: pristine-block USERD fallback for a GR channel + * whose ring does not translate yet. + * + * Measured (r1013 fresh boot): the X driver's pushbuffer-test channel + * (class 0xc46f, 8192 entries, VA 0x120090000) reports its USERD as the FB + * memdesc base 0x30000, but the guest's GPPut write lands in emulated vidmem + * at 0xa0000 - the params' base is relative to a client pool the emulation + * cannot see (same class of problem the r496b sweep solves once the ring + * resolves). The block is pristine in the test window: the only non-zero + * bytes in [0x0, 0x100) are the GP_PUT dword at +0x8c (GP_GET at +0x88 is + * still 0, because the driver zeroes it before the NOP test). When the + * channel's VAS page directory has not registered yet, the r496b sweep + * cannot run (it validates a hit by reading the ring entries the put + * claims), so this bounded shape scan keeps the discovery alive: pending + * only (get < put), pristine page, unclaimed by any other channel, then + * adopt it exactly like the ring-validated sweep does. The r496b sweep + * still runs later and may re-adopt a ring-validated block; both paths + * claim their latch, so a foreign channel cannot take it in between. + */ +static bool svc_fb_userd_pristine_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *off_out, + u32 *put_out, u32 *get_out) +{ + u8 *base = vgpu->bars[1].vaddr; + u64 lim, off; + + if (!base || !ch->gpfifo_entries) + return false; + lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + for (off = 0; off + 0x100 <= lim; off += 0x1000) { + u32 hp, hg; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + if (!hp || hp > ch->gpfifo_entries) + continue; + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + /* Only a posed kick (get < put) - an idle/stale block stays + * with its owner. */ + if (hg >= hp) + continue; + if (memchr_inv(base + off, 0, SVC_USERD_GP_GET) || + memchr_inv(base + off + SVC_USERD_GP_GET + 8, 0, + 0x100 - SVC_USERD_GP_GET - 8)) + continue; + /* + * X-DMA-FIX: pass the candidate offset as both ring and USERD. + * The old (0, off) form could not match a live CE rank's + * convention ring (ring_off == conv), but it also made every + * later pristine scan foreign against the first pristine + * adopter (which stores ring_off=0): (0 == 0) matched the + * ring_off check unconditionally. Matching the actual offset + * against both stored fields keeps the CE convention + * reservation and the one-owner-per-block rule while allowing + * a second GR channel its own block. + */ + if (svc_fb_claim_foreign(vgpu, ch->h_object, off, off)) + continue; + *off_out = off; + *put_out = hp; + *get_out = hg; + return true; + } + return false; +} + /* * Locate a channel's USERD. The allocation parameters carry an offset that is * relative to a client pool the emulation cannot see, so it is only one @@ -12538,15 +26772,251 @@ static int svc_userd_probe(struct nv_vgpu_device *vgpu, u64 pa, u32 as, * aperture stays the channel's own; a wrong second aperture only costs a * failed probe (the values must still read as a coherent put/get pair). */ +/* + * SESSION 153-CONTINUED-38: the shared entry-window gate used by every GR + * ring/entry validator (defined next to svc_gr_ring_candidate below). + */ +static bool svc_gr_entry_win_ok(u64 ring_va, u64 eva); + +/* + * r1653: capture the class2d kick channel's notification target from the + * FIRST GPFIFO entry. The GL client arms the class2d subchannel with + * SET_NOTIFY_A/B + NOTIFY in entry 0 and then submits short kick pushes + * (NOTIFY only) that rely on the target staying programmed in the channel + * context. The pump's walk seed consumes only the newest entries ending at + * the live PUT, so on a fresh channel entry 0 can be skipped forever and + * every short kick has no address to author - the client then spins in its + * kick wait (r1652: h=0xbfef0002, "NOTIFY with no class2d target"). Decode + * the skipped entries here, once, before the seed. + */ +static bool svc_kick_scan_push(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 va, u32 len) +{ + u8 buf[512]; + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + u32 n = min_t(u32, len, sizeof(buf)); + u32 off = 0; + + if (svc_lookup(vgpu, va, &pa, &ap) && + svc_resolve_va(vgpu, va, &pa, &ap)) + return false; + if (svc_read_ap(vgpu, pa, svc_ap_of(ap), buf, n)) + return false; + while (off + 4 <= n) { + u32 hdr, op, count, subch, addr, k; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 7; + addr = (hdr & 0x1fff) << 2; + if (!hdr) { + off += 4; + continue; + } + if (op == 1 || op == 3 || op == 5) { + if (!count || off + 4 + count * 4 > n) + break; + if (subch == 3) { + for (k = 0; k < count; k++) { + u32 m = op == 3 ? addr : addr + 4 * k; + u32 v; + + memcpy(&v, buf + off + 4 + 4 * k, 4); + if (m == 0x104) + ch->notify2d_va = + (u64)(v & 0x1ffffff) << 32; + else if (m == 0x108) { + ch->notify2d_va |= v; + if (ch->notify2d_va) + return true; + } + } + } + off += 4 + count * 4; + continue; + } + off += 4; + } + return ch->notify2d_va != 0; +} + +static void svc_kick_capture_arm(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 ring_pa, + int ring_ap, u32 uput) +{ + u32 e, nscan; + + if (!ch->engine_type || ch->h_object < 0xbfef0000u || + ch->h_object > 0xbfef00ffu || ch->notify2d_va || !ring_pa) + return; + /* + * r1653: the arm is normally entry 0, so a missing/zero live PUT + * must not block the decode. Bound the scan by the ring size and + * the live PUT when known. + */ + nscan = ch->gpfifo_entries ? ch->gpfifo_entries : 2; + if (uput && uput < nscan) + nscan = uput; + if (nscan > 8) + nscan = 8; + for (e = 0; e < nscan && !ch->notify2d_va; e++) { + u8 ent[8]; + u32 e0, e1; + u64 pva; + u32 plen; + + if (svc_read_ap(vgpu, ring_pa + (u64)e * 8, ring_ap, ent, + sizeof(ent))) + continue; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + pva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + plen = ((e1 >> 10) & 0x1fffff) << 2; + if (pva && plen >= 4 && plen <= 0x10000) + svc_kick_scan_push(vgpu, ch, pva, plen); + } + if (ch->notify2d_va) + pr_info("sema-chan: kick arm captured h=0x%x notify2d=0x%llx from ring entry (first-discovery decode)\n", + ch->h_object, ch->notify2d_va); +} + +/* + * r1659: direct-GL CE USERD adoption. The beef0101/0102 channels declare a + * real FB control page (userdMem.base + userdOffset: 0x244f800 / 0x244f400) + * while their GPFIFO ring is mapped in guest RAM. svc_chan_userd only + * probed ring+0x10000 in the ring's aperture and the 16 MB BAR0 shadow, so + * the declared page was never read; their ring also starts at slot 2 + * (entry 0 stays zero), which the entry-0 layout gate can never accept. + * Read the declared page through the FB shadow and adopt it when its PUT is + * plausible and the newest claimed slot decodes into this channel's low-VA + * push window. Holes below the live PUT are allowed (the client's CE ring + * has unwritten slots between its publishes). + */ +static bool svc_ce_declared_userd(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 *put, u32 *get) +{ + u32 p = 0, g = 0, k, slot, valid = 0, newest, newest_ok = 0; + u32 scan_end; + + if (!ch->userd_pa || !ch->gpfifo_entries) + return false; + if (svc_userd_probe_ex(vgpu, ch->userd_pa, ch->userd_as, + ch->gpfifo_entries, &p, &g, true)) + return false; /* unreadable/idle/incoherent: retry later */ + if (!p) + return false; + newest = (p - 1) % ch->gpfifo_entries; + /* + * Validate the newest claimed slot first: that is the entry the + * client is waiting on, and a stale/foreign control page cannot + * name a push in this channel's own ring window. Allow a few holes + * below the newest (the live CE ring publishes at slots 2/4/5). + */ + for (k = 0; k < 8 && k < p && !newest_ok; k++) { + u8 e[8]; + u32 e0, e1, len; + u64 eva; + + slot = (p - 1 - k) % ch->gpfifo_entries; + if (svc_read_va(vgpu, ch->gpfifo_va + + (u64)slot * SVC_GPFIFO_ENTRY_SIZE, + e, sizeof(e))) + continue; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!eva && !len) + continue; /* unwritten hole */ + if (!eva || len < 4 || len > 0x100000 || + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) + return false; + newest_ok = 1; + } + if (!newest_ok) + return false; + scan_end = p < 64 ? p : 64; + for (k = 0; k < scan_end; k++) { + u8 e[8]; + u32 e0, e1, len; + u64 eva; + + slot = k % ch->gpfifo_entries; + if (svc_read_va(vgpu, ch->gpfifo_va + + (u64)slot * SVC_GPFIFO_ENTRY_SIZE, + e, sizeof(e))) + continue; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!eva && !len) + continue; /* unwritten hole */ + if (!eva || len < 4 || len > 0x100000 || + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) + return false; + valid++; + } + if (!valid) + return false; + ch->userd_ok = true; + ch->userd_wait_fresh = p >= 1; + ch->last_get = 0; /* live sequence starts at slot 0; holes skip */ + *put = p; + *get = g; + pr_info("sema-chan: CE USERD h=0x%x eng=%u ADOPT declared pa=0x%llx as=%u put=%u get=%u (newest slot %u validated)\n", + ch->h_object, ch->engine_type, ch->userd_pa, ch->userd_as, + p, g, newest); + return true; +} + +/* + * CONT.438: bounded witness for FB-sweep candidates skipped because another + * live channel already owns their page. + */ +static u32 svc_foreign_userd_skip_log; + static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u64 ring_pa, int ring_ap, u32 *put, u32 *get, int rrc) { - u64 cand[8]; - u32 as[8], nc = 0, i; + u64 cand[16]; + u32 as[16], nc = 0, i; + /* + * X-DMA-FIX (2026-11-10): "translated" is not "validated". svc_lookup + * can answer from a stale pin or a foreign VAS with a page whose + * GPFIFO entries are zero/foreign; the r496b FB sweep then rejects + * every candidate while ring_pa != 0, and the old !ring_pa pristine + * gate skipped the fallback in exactly that state. The channel then + * stayed without a USERD, the pump never wrote GP_GET, and the Xorg + * DDX pushbuffer NOP test timed out (10 s) -> "Failed to initialize + * DMA". Track whether the sweep actually ran and whether any entry + * decoded into this channel's push window so the pristine scan can + * still run for a resolved-but-unvalidated ring. + */ + bool ring_entry_ok = false; /* a ring entry decoded in this sweep */ + bool ring_sweep_ran = false; /* the r496b FB sweep ran this call */ int ring_alt = ring_ap == SVC_APERTURE_FB ? SVC_APERTURE_SYS : SVC_APERTURE_FB; /* + * r820 diag: the Xorg DMA-test window is 10 s and the latch hinges on + * this function's bounded scan re-runs (userd_scans < 8); a pass + * cadence stretched by a rebuild can spend every scan before the VAS + * candidate registers (r820: the ring resolved at +1 s but no latch + * line followed). One bounded row per call names the state the scan + * decides on. + */ + if (gsp_sema_dbg && !ch->engine_type && svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp h=0x%x eng=%u scans=%u rrc=%d ring_pa=0x%llx ring_ap=%d userd_pa=0x%llx as=%d ok=%u\n", + ch->h_object, ch->engine_type, ch->userd_scans, + rrc, ring_pa, ring_ap, ch->userd_pa, ch->userd_as, + ch->userd_ok ? 1 : 0); + } + + /* * ISSUE-0002 (r40): prefer the layout-derived CE USERD and accept it * while it is still idle. In a second session the registered userdMem * offset (pool-relative, see below) aliases a DIFFERENT rank's real @@ -12676,15 +27146,49 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * at the layout offset. */ if (ch->engine_type != 0) { + /* + * r1659: the direct-GL CE channels declare their control + * page in FB (0x244f800/0x244f400) and publish from slot 2, + * so the entry-0 layout gate above never fires. Adopt the + * declared page when the newest slot it names validates. + */ + if (svc_ce_declared_userd(vgpu, ch, put, get)) + return 0; + /* + * SESSION 153-CONT-34: before giving up on a CE channel, try + * the registered offset read as a BAR0-shadow USERD - the GL + * client's copy/2D channels declare the same FB offsets as its + * GR channel and their GP_PUT writes land in the same shadow. + */ + if (svc_bar0_userd_try(vgpu, ch, put, get)) + return 0; svc_userd_sweep_sched(ch); - return -ENOENT; + /* + * r1659: the direct-GL CE channels' real control page is + * not the declared pool-relative offset; find it with the + * same VRAM shape scan the GR path uses (now hole-tolerant + * for CE). Jump past the GR-only candidate/pin paths. + */ + goto ce_fb_scan; } if (ring_pa) { - cand[nc] = ring_pa + 0x2000; as[nc++] = ring_ap; - cand[nc] = ring_pa + 0x2000; as[nc++] = ring_alt; - cand[nc] = ring_pa + 0x1000; as[nc++] = ring_ap; - cand[nc] = ring_pa + 0x1000; as[nc++] = ring_alt; + /* + * r1780 (V38): the +0x2000/+0x1000 convention places the + * USERD after a 1024-entry (0x2000-byte) ring. A 32768-entry + * GPFIFO spans 0x40000 bytes, so those addresses are GPFIFO + * slots, not a control block; the probe still "passed" + * (put=25205/get=512 from slot 1041) and the pump then wrote + * GP_GET into the ring. Only use the convention for small + * rings and let the declared/RAM-scan candidates serve the + * large ones. + */ + if (ch->gpfifo_entries <= 1024) { + cand[nc] = ring_pa + 0x2000; as[nc++] = ring_ap; + cand[nc] = ring_pa + 0x2000; as[nc++] = ring_alt; + cand[nc] = ring_pa + 0x1000; as[nc++] = ring_ap; + cand[nc] = ring_pa + 0x1000; as[nc++] = ring_alt; + } /* * UVM's CE channel allocations place the USERD one 0x10000 * block after the ring (observed for all eight CE channels: @@ -12699,6 +27203,27 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, as[nc++] = ring_alt; } } + /* + * CONT.405: BAR1-alias candidates FIRST for a declared FBMEM USERD. + * The client writes its control block through the guest RM's BAR1 + * mapping, so the alias page can carry the live pair while the + * declared FB block stays fill. The learned bias makes the alias a + * direct translation (X: 0x190000 -> 0x910000); the wire page base + * covers the case where the alloc carried an index offset the client + * did not apply to the block it publishes. Both are still validated + * by svc_userd_probe_ex / the candidate gates below. + */ + if (ch->userd_as == SVC_APERTURE_FB && ch->userd_wire) { + u64 alias = 0; + + if (svc_userd_bar1_alias(ch, &alias)) { + cand[nc] = alias; as[nc++] = SVC_APERTURE_FB; + } + if ((ch->userd_wire & ~0xfffULL) != ch->userd_pa) { + cand[nc] = ch->userd_wire & ~0xfffULL; + as[nc++] = SVC_APERTURE_FB; + } + } if (ch->userd_pa) { cand[nc] = ch->userd_pa; as[nc++] = ch->userd_as; cand[nc] = ch->userd_pa; @@ -12707,10 +27232,42 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, } for (i = 0; i < nc; i++) { + bool loose = !ch->engine_type && ch->gpfifo_entries > 1024; + if (!cand[i]) continue; - if (!svc_userd_probe(vgpu, cand[i], as[i], ch->gpfifo_entries, - put, get)) { + if (!svc_userd_probe_ex(vgpu, cand[i], as[i], + ch->gpfifo_entries, put, get, loose)) { + /* + * SESSION 153-CONTINUED-44 (r1026): a GR kernel + * channel's candidate read of (put == 0, arbitrary + * get) is not a block the client publishes to. The + * cleared-screen fill pattern (alternating + * 0xff000000 / 0x00000000 pixels) yields exactly + * (put=0, get=0xff000000) at every 8-byte aligned + * offset, and the loose large-ring acceptance + * (g > p allowed) latched it: r1025's every + * beef0100 cycle adopted fb 0x20fc00 with put=0 + * forever (`submissions=0`, client recycles at + * 12 s) while the pre-walk-first boot's FB shape + * scan adopted the block carrying the client's + * real puts (`put=3/14/6`, submissions 29->208). + * Skip zero-put candidates for GR channels: the + * resolution re-runs every pass, and the FB/RAM + * shape scans (which require put >= 1) then adopt + * the block the client actually publishes on its + * first push. CE channels keep their layout rules + * (idle 0/0 acceptance is a documented CE case). + */ + if (!ch->engine_type && !*put) { + if (svc_zero_put_skip_log < 32) { + svc_zero_put_skip_log++; + pr_info("sema-chan: userd candidate 0x%llx/%u skipped for h=0x%x (put=0 get=%u; not a published block)\n", + cand[i], as[i], + ch->h_object, *get); + } + continue; + } /* ISSUE-0002 r46: name a fallback latch (CE wrong block). */ if (gsp_sema_dbg && ch->engine_type && ch->ce_index < 32 && @@ -12728,6 +27285,562 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, return 0; } } + + /* + * r912 (M-GL1.2): a GR channel whose ring was content-validated in + * guest RAM (gr_ram_ring) has its USERD in guest RAM too - the + * FB-shadow sweep below can never reach it. The guest-RAM shape + * scan is sliced and budgeted like the ring scan; it validates + * every entry the candidate's put claims against the ring window + * and admits the block only after its PUT advances across sightings + * (see svc_gr_ram_userd_scan; SESSION 154 liveness). The budget + * gate stays open while candidates are under watch, so the + * liveness confirmation cannot be starved by slice accounting. + * SESSION 153-CONTINUED-20: for the kick family it also stays open + * while the bounded sweep re-arm budget lasts - the watch set can + * go empty for stretches (all frozen pool entries timed out), and + * the in-scan re-arm needs a next call to re-run the sweep that + * finds the advancing block. + */ + /* + * SESSION 153-CONTINUED-38: kernel-VA rings (the direct GL client's + * 0x4000000-class channels) resolve through the channel's own VAS + * walk, never the guest-RAM content scan, so gr_ram_ring stays false + * and the RAM USERD discovery below would never run for exactly the + * channels that need it (CONT-37: beef0100-0102 ok=0, sub=0, then + * the alloc/free retry loop). Anchor the walk-resolved ring as a pin + * - only after entry0 decodes plausibly, so a stale walk answer is + * not cemented - and let the scan run on either anchor. The scan's + * entry validation still proves ownership against the ring. + */ + if (!ch->engine_type && ring_pa && !ch->gr_ram_ring && !ch->userd_ok) { + u8 re[SVC_GPFIFO_ENTRY_SIZE]; + u32 r0, r1; + u64 rva; + u32 rlen; + + if (!svc_read_ap(vgpu, ring_pa, ring_ap, re, sizeof(re))) { + memcpy(&r0, re, 4); + memcpy(&r1, re + 4, 4); + rva = ((u64)(r1 & 0xff) << 32) | + (u64)(r0 & 0xfffffffc); + rlen = ((r1 >> 10) & 0x1fffff) << 2; + if (!(rva & 0xfffff) && rlen >= 4 && rlen <= 0x10000 && + svc_gr_entry_win_ok(ch->gpfifo_va, rva)) { + svc_pin_store(ch->gpfifo_va, ring_pa, ring_ap); + svc_lat_store(ch->gpfifo_va, ring_pa, ring_ap); + } + } + } + if (!ch->engine_type && (ch->gr_ram_ring || ring_pa) && !ch->userd_ok && + (ch->gr_ram_userd_passes < svc_gr_ram_pass_budget() || + ch->gr_ram_userd_watch_n || + ((ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu) && + ch->gr_ram_userd_kick_sweeps < 64))) { + u64 upa = 0; + u32 uput = 0, uget = 0; + + if (svc_gr_ram_userd_scan(vgpu, ch, &upa, &uput, &uget)) { + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_SYS; + ch->userd_ok = true; + /* r1653: decode the skipped first entries for the + * class2d kick arm before the walk seed below. */ + svc_kick_capture_arm(vgpu, ch, ring_pa, ring_ap, + uput); + /* + * SESSION 154 (M-GL1.2): seed the walk at the live + * block's PUT. The client's put is free-running + * (measured 0->3 with entries=2), so the pump's + * `put %= entries` alone can collapse onto + * last_get and consume NOTHING - r914 stayed at + * submissions=0 for minutes with put=12 % 2 = 0 and + * last_get=0. Consume up to entries-1 positions + * ENDING at the PUT: the newest slots the client is + * waiting on; normal slot-space tracking resumes + * from there. + */ + if (ch->gpfifo_entries > 1 && uput) { + u32 span = uput < ch->gpfifo_entries ? + uput : ch->gpfifo_entries - 1; + + ch->last_get = (uput % ch->gpfifo_entries + + ch->gpfifo_entries - span) % + ch->gpfifo_entries; + } else { + ch->last_get = 0; + } + ch->gr_ram_put_seeded = uput; + ch->gr_ram_userd_latch_put = uput; + ch->gr_ram_userd_latch_at = jiffies; + ch->gr_ram_userd_confirmed = false; + *put = uput; + *get = uget; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by RAM scan -> pa=0x%llx (put=%u get=%u%s; entries ring-validated; speculative, pump confirms; walk seed last_get=%u)\n", + ch->h_object, ch->gr_index, upa, uput, uget, + uget < uput ? "" : ", idle", + (u32)ch->last_get); + return 0; + } + } + + /* + * SESSION 153-CONTINUED-76: name the gated-out case too. The churning + * GL client's failing instance showed neither a sweep line nor an + * adoption; if the ring parameter never resolves post-re-registration + * the whole r496b sweep is skipped and the failure is invisible. + */ + if (gsp_sema_dbg && !ch->engine_type && !ring_pa && !ch->userd_ok && + (ch->fb_userd_scan_n == 0u || ch->fb_userd_scan_n == 64u) && + (ch->h_object >= 0xbeef0000u && ch->h_object <= 0xbeefffffu)) { + ch->fb_userd_scan_n++; + pr_info("sema-chan: userdp no-sweep h=0x%x ring_pa=0 n=%u gr_ram_ring=%d (r496b gated on an unresolved ring)\n", + ch->h_object, ch->fb_userd_scan_n, + ch->gr_ram_ring ? 1 : 0); + } + + /* + * r496b: no candidate matched. For a GR channel whose ring is known + * (translated and pinned - the sysmem content scan or the VRAM + * conventions), the USERD may still be elsewhere: the X/GL driver + * allocates its USERD in VRAM and the CPU RM maps it through a BAR1 + * window, so the driver's GP_PUT write lands at a BAR1 offset that is + * neither the registered memdesc base nor ring-derived. Scan the + * emulated VRAM for the USERD shape and validate each hit against the + * ring: the put it claims must correspond to entries whose pushbuffer + * VAs all sit in the ring's allocation window. A foreign channel's + * USERD cannot pass that test. One scan per channel (the result + * persists through the pump's write-back), logged on latch. + */ + /* + * r820: the latch scan is capped per channel so a channel that + * never anchors cannot sweep forever. But the resolution the scan + * needs can arrive LATE: the Xorg GL channel's ring resolves only + * when the client's VAS page directory registers (COPY_SERVER_PDES + * age-out), and r820 showed every one of the 8 pre-resolution scans + * spent before that moment (the pass cadence stretched by a + * rebuild), so the 10 s DMA-test window closed with the ring + * resolved and no retry left. Once ring_pa is non-zero the scan is + * cheap (the sweep validates against a resolved ring), so it may + * keep retrying up to a wider cap; the client's own timeout still + * bounds the channel's life. + */ +ce_fb_scan: + if (ring_pa && + (!ch->engine_type || + (ch->h_object & 0xfffff000u) == 0xbeef0000u) && + ch->fb_userd_scan_n < 512 && + (!ch->fb_userd_scan_at || + time_after_eq(jiffies, + (unsigned long)ch->fb_userd_scan_at + + msecs_to_jiffies(25)))) { + u64 off, lim; + u8 *base = vgpu->bars[1].vaddr; + u64 best = 0, best_dist = ~0ULL; + u32 best_put = 0, best_get = 0; + bool best_pending = false; + u32 gate_n = 0, vfail_n = 0; + /* + * SESSION 153-CONTINUED-76: post-re-registration ring reads. + * The client churns its channels every ~12 s and a ring pin + * inherited from the dead instance can shadow the walk in + * svc_lookup; validated FELL through the stale pin made every + * candidate fail and the re-registered channel never got its + * USERD (top-level contention.75). Resolve the ring page + * through the client's OWN VAS walk once per sweep and retry + * each failed entry read there: the walk is the authority for + * the live session, the pin only a cache. + */ + u64 wp = 0; + int wap = SVC_APERTURE_SYS; + bool wtry = false, wused = false; + u32 woff = (u32)(ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + + /* + * CONT.438 (patch A): a candidate on a 4K page that another + * live channel already owns is not this channel's USERD. The + * live X/GLX channel 0x1000013 re-adopted the RC watchdog + * channel 0x31415900's block at fb 0x8f0000 every pass (its + * own USERD is client-allocated, so the wire userdMem is + * zero); the foreign pair then never matched the client's + * progress. Snapshot the other channels' USERD pages once per + * sweep and refuse their pages as candidates. Failing closed + * here is deliberate: a foreign block must not become the + * write target for GP_GET. + */ + u64 foreign_userd_pages[SVC_MAX_CHANNELS]; + u32 foreign_userd_as[SVC_MAX_CHANNELS]; + unsigned long chflags; + u32 foreign_userd_n = 0; + u32 oi; + + spin_lock_irqsave(&svc.chan_lock, chflags); + for (oi = 0; + oi < SVC_MAX_CHANNELS && + foreign_userd_n < SVC_MAX_CHANNELS; oi++) { + struct svc_channel *other = &svc.chan[oi]; + + if (!other->valid || other->vgpu != vgpu || + !other->userd_pa) + continue; + if (other->h_object == ch->h_object && + other->vgpu == ch->vgpu) + continue; + foreign_userd_pages[foreign_userd_n] = + other->userd_pa & ~0xfffULL; + foreign_userd_as[foreign_userd_n] = + other->userd_as; + foreign_userd_n++; + } + spin_unlock_irqrestore(&svc.chan_lock, chflags); + + ch->fb_userd_scan_at = jiffies; + ch->fb_userd_scan_n++; + ch->userd_scans++; + ring_sweep_ran = true; + lim = min_t(u64, svc_fb_shadow_size(vgpu), + 0x20000000ULL); + if (base && lim) { + for (off = 0; off + 0xa0 <= lim; off += 0x200) { + u32 hp, hg; + bool ok = true; + bool newest_ok = false; + u32 k; + u64 dist; + bool pending; + int rck = -1; + u64 rkva = 0; + u32 rklen = 0; + u64 page; + bool foreign = false; + + memcpy(&hp, base + off + SVC_USERD_GP_PUT, 4); + memcpy(&hg, base + off + SVC_USERD_GP_GET, 4); + /* CONT-38: large GR kernel rings may carry an + * uninitialised GPU-owned GP_GET; the PUT alone + * gates there (see svc_userd_probe_ex). r1659: + * CE control pages also carry a GPU-owned GET, + * but bound it to one ring so a garbage pair + * (r1660: put=2880 get=188746560) cannot pass. */ + if (!hp || hp > ch->gpfifo_entries || + (hg > hp && + (ch->engine_type ? + hg > hp + ch->gpfifo_entries : + ch->gpfifo_entries <= 1024))) + continue; + gate_n++; + /* + * CONT.438: the page-ownership gate. A block + * on another live channel's USERD page cannot + * be this channel's control block. + */ + page = off & ~0xfffULL; + for (oi = 0; oi < foreign_userd_n; oi++) { + /* + * CONT.438b: the same numeric PA can + * exist in sysmem and vidmem (c23: + * X's 0x8f0000 sysmem block vs the + * RC watchdog's 0x8f0000 FB block). + * Only a same-aperture owner makes + * the candidate foreign. + */ + if (foreign_userd_pages[oi] == page && + foreign_userd_as[oi] == ch->userd_as) { + foreign = true; + break; + } + } + if (foreign) { + if (svc_foreign_userd_skip_log < 16) { + svc_foreign_userd_skip_log++; + pr_info("sema-chan: userdp h=0x%x skip fb 0x%llx (page owned by another live channel; put=%u get=%u)\n", + ch->h_object, off, hp, + hg); + } + continue; + } + /* Validate: every claimed entry decodes to a + * pushbuffer VA in the ring's window. */ + for (k = 0; k < hp && k < 64; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + int rdrc; + + rdrc = svc_read_va(vgpu, + ch->gpfifo_va + + (u64)k * 8, + e, sizeof(e)); + if (rdrc) { + /* + * CONT-76 walk fallback: a stale + * or missing pin must not veto a + * candidate the live session's own + * page tables can answer. Resolve + * the ring page once per sweep and + * re-read the entry from it. + */ + if (!wtry && !wp) { + wtry = true; + if (svc_walk_vaspaces(vgpu, + ch->gpfifo_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + &wp, &wap)) + wp = 0; + } + if (wp && + (u64)woff + (u64)k * 8 + 8 <= + SEMA_PAGE_SIZE) { + rdrc = svc_read_ap(vgpu, + wp + woff + + (u64)k * 8, + wap, e, sizeof(e)); + if (!rdrc) + wused = true; + } + } + if (rdrc) { + ok = false; + rck = (int)k; + /* -1: read failure rc; else the + * window check below. */ + rkva = (u64)rdrc; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + /* + * Every entry the put claims must name a + * VA in the ring's allocation window - + * pushbuffers below the ring, the + * progress tracker above it; the + * kernel-VA family is accepted in its + * own low window (CONT-38). + */ + /* + * r1659: the direct-GL CE ring starts + * at slot 2 (entry 0/1 stay zero until + * the client's first publishes); an + * unwritten hole is not a window + * failure for a CE channel. + */ + if (!eva && !elen && + (ch->engine_type || + (ch->h_object & 0xfffff000u) == + 0xbeef0000u)) + continue; + if (!eva || + (!svc_gr_entry_win_ok( + ch->gpfifo_va, eva) && + !((ch->h_object & 0xfffff000u) == + 0xbeef0000u && + eva >= SEMA_VA_LO && + eva < SEMA_VA_HI)) || + elen < 4 || elen > 0x100000) { + ok = false; + rck = (int)k; + rkva = eva; + rklen = elen; + break; + } + /* + * X-DMA-FIX: an entry that decodes into + * this channel's push window proves the + * resolved page really carries this + * channel's GPFIFO. This is the only + * "ring validated" signal the pristine + * gate below may use; wused only says the + * walk fallback was needed. + */ + ring_entry_ok = true; + if (ch->engine_type && + k == (u32)((hp - 1) % + ch->gpfifo_entries)) + newest_ok = true; + } + /* r1659: a CE control page must account + * for the slot the client is waiting on. */ + if (ch->engine_type && !newest_ok) + ok = false; + if (!ok) { + /* + * r820 diag: name why a scanned + * block was rejected (read rc or + * the entry VA/len window). + */ + vfail_n++; + if (gsp_sema_dbg && !ch->engine_type && + svc_userdp_log < 512) { + svc_userdp_log++; + pr_info("sema-chan: userdp scan off=0x%llx put=%u rejected k=%d (read_rc=%lld va=0x%llx len=%u)\n", + off, hp, rck, + (long long)rkva, + rkva, rklen); + } + continue; + } + /* + * Several blocks can validate: a previous + * session leaves its old USERD holding the same + * put/entry values while the new channel's block + * sits elsewhere (observed r504: the re- + * registered channel re-scanned, adopted the + * stale block, and its DMA test timed out). + * Prefer a block with work PENDING (get < put); + * among equals prefer the one nearest the + * registered memdesc offset (the driver's own + * hint). The pump's GP_GET write-back goes to + * the adopted block, so a stale adoption strands + * the client's poll. + */ + dist = off > ch->userd_pa ? off - ch->userd_pa + : ch->userd_pa - off; + pending = hg < hp; + if (best && + (best_pending > pending || + (best_pending == pending && dist >= best_dist))) + continue; + best = off; + best_dist = dist; + best_put = hp; + best_get = hg; + best_pending = pending; + } + if (best) { + /* + * CONT.405: this latch is ring-validated, so + * it is the ground truth for the FB->BAR1 + * page bias. Learn it once and apply it to + * later FBMEM registrations. + */ + svc_userd_alias_learn(ch, best); + ch->userd_pa = best; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + *put = best_put; + *get = best_get; + /* + * r1780 (V32): seed the speculative latch with + * the adopted block's own PUT. The previous + * code left latch_put at 0 (or at a previous + * RAM-scan candidate's value), so the pump's + * judge saw put != latch_put on the very first + * pass and rejected/cleared the just-adopted + * USERD - the ~10 Hz adopt/reject spin that + * starved the X server's channel (r1132 class, + * reproduced on r1780 c31: "speculative latch + * REJECTED (put=86 latch=512)" every pass). + * With the seed, the first judge pass is a + * no-op and a forward step confirms; a decrease + * or the 15 s no-advance bound still rejects a + * stale block. + */ + ch->gr_ram_userd_latch_put = best_put; + ch->gr_ram_userd_latch_at = jiffies; + ch->gr_ram_userd_confirmed = false; + pr_info("sema-chan: %s USERD h=0x%x rank=%u found by VRAM shape scan at fb 0x%llx (put=%u get=%u%s, ring-validated)\n", + ch->engine_type ? "CE" : "GR", + ch->h_object, + ch->engine_type ? ch->ce_index + : ch->gr_index, best, + best_put, best_get, + best_pending ? "" : ", idle"); + return 0; + } + /* + * r820 diag: the sweep found nothing adoptable. gates + * counts put/get-coherent blocks seen, vfails the ones + * rejected by the ring-entry validation. + * + * SESSION 153-CONTINUED-76: the global r820 budget + * (svc_userdp_log < 512) was spent by other channels + * before the churning GL client's ranks ever swept, so + * its post-re-registration failure was invisible. Log + * the sweep result for GR ranks at fixed per-instance + * scan numbers (3 lines per instance max) regardless of + * the global budget, with the walk answer included so + * the stale-pin vs dead-walk distinction is in the + * record. + */ + if (gsp_sema_dbg && + (svc_userdp_log < 512 || + (!ch->engine_type && + (ch->fb_userd_scan_n == 4u || + ch->fb_userd_scan_n == 64u || + ch->fb_userd_scan_n == 256u)))) { + svc_userdp_log++; + pr_info("sema-chan: userdp sweep h=0x%x scans=%u gates=%u vfails=%u best=0x%llx ring_pa=0x%llx walk=0x%llx used=%d\n", + ch->h_object, ch->userd_scans, + gate_n, vfail_n, best, ring_pa, wp, + wused ? 1 : 0); + } + } + } + /* + * SESSION 153-CONTINUED-37: the channel's ring VA does not translate + * yet (the client's VAS page directory registers ~1 s after channel + * creation; until then svc_lookup cannot resolve it and the + * ring-validated r496b sweep above never runs). The driver's DMA + * test only needs the block it is polling: a freshly kicked pristine + * FB block (see svc_fb_userd_pristine_scan). Adopt it so the pump + * starts iterating entries as soon as the ring walk lands, inside + * the driver's 10 s window. + */ + /* + * X-DMA-FIX (2026-11-10): run the pristine scan when the ring is not + * translated (!ring_pa, as before) OR when the r496b sweep ran this + * call on a resolved ring but could not validate a single entry + * (ring_sweep_ran && !ring_entry_ok). A RAM-anchored ring + * (gr_ram_ring) keeps its own guest-RAM USERD scan and is excluded. + */ + if (!ch->engine_type && + (!ring_pa || + (ring_sweep_ran && !ring_entry_ok && !ch->gr_ram_ring)) && + ch->fb_userd_scan_n < 512 && + (!ch->fb_userd_scan_at || ring_sweep_ran || + time_after_eq(jiffies, + (unsigned long)ch->fb_userd_scan_at + + msecs_to_jiffies(25)))) { + u64 off = 0; + u32 p = 0, g = 0; + + /* + * X-DMA-FIX: the sweep above already spent this call's scan + * slot. Counting the combined call twice would halve the + * 512-scan (12.8 s at 25 ms) window the X 10 s DMA test + * relies on. Only a standalone pristine scan pays here. + */ + if (!ring_sweep_ran) { + ch->fb_userd_scan_at = jiffies; + ch->fb_userd_scan_n++; + } + if (svc_fb_userd_pristine_scan(vgpu, ch, &off, &p, &g)) { + ch->userd_pa = off; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_fb_claim_add(ch->h_object, 0, off); + *put = p; + *get = g; + pr_info("sema-chan: GR USERD h=0x%x rank=%u found by pristine-FB scan at fb 0x%llx (put=%u get=%u; ring va 0x%llx pa 0x%llx %s)\n", + ch->h_object, ch->gr_index, off, p, g, + ch->gpfifo_va, ring_pa, + ring_pa ? "resolved but not validated" + : "not translated yet"); + return 0; + } + } + /* + * SESSION 153-CONT-34: last resort for the GL client family - retry the + * registered offset as a BAR0-shadow USERD. Every other source (layout, + * ring-derived, RAM scan, FB sweep) has already missed the block whose + * GP_PUT writes land in the emulated register window. + */ + if (svc_bar0_userd_try(vgpu, ch, put, get)) + return 0; return -ENOENT; } @@ -12754,6 +27867,555 @@ static int svc_chan_userd(struct nv_vgpu_device *vgpu, struct svc_channel *ch, * pump's completion clear before the normal path resumes. */ #define SVC_NOTIFIER_ERR_HOLD (2 * HZ) +/* + * SESSION 134: CE first-submission capture. + * + * The CE (copy-engine) rings are placed by the guest's allocator at a + * location that varies per boot and is NOT reliably described by the + * channel's registered gpfifo VA through the discovered page tables (r763 + * matched the fixed FB convention and the sweep decoded the inline-SASS + * entry; r765-r772 read zeros everywhere, the CE channels never latched, + * the driver's inline LAUNCH_DMA code downloads never executed and the + * frames stayed black). The logged instance where ring content and USERD + * were both live is the channel's first notifier clear - the guest's + * doorbell for its first submission, processed by the pump in the same + * pass that writes the ring. At that instant this hook: + * 1. reads entry0 through the channel's own translation (svc_lookup), + * 2. reads entry0 through the trusted walk (the VAS may have just + * landed since the registration read), + * 3. scans the shadow (0x100 step, first 64 MB, the GR-scan bound) for + * a ring-shaped page whose entry0 names a 1MB-aligned push VA in the + * CE window [0x120000000,0x121000000), and for every hit runs the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate + * whose USERD reads coherent (or idle 0/0) is PINNED and LATCHED + * exactly like svc_ring_autopin's CE path. + * Bounded: at most 4 captures per CE channel, 16 witness rows per boot; + * resets with the session (svc_session_flush). + */ +static u64 svc_ce_firstsub_log; + +/* + * SESSION 134: bounded completion hold for unlatched CE channels. + * + * The clear write in svc_notify_complete is what makes the guest treat the + * submission as consumed; the driver then recycles the ring and its content + * is gone before the client's VAS lands (sema_vas_cand_ms), leaving the CE + * ring latch with nothing readable (r769-r774: rrc=-2 at the first + * submission, the walk answering a zero page later). Holding the notifier + * busy for a bounded window keeps the driver's submission in flight - and + * with it the ring content that the regular per-pass latch path + * (svc_chan_userd through svc_lookup) needs - until the VAS registers and + * the ring reads coherently. 0 = off (the pre-SESSION-134 behaviour). + */ +static u64 svc_ce_hold_ms; +module_param_named(sema_ce_hold_ms, svc_ce_hold_ms, ullong, 0644); +MODULE_PARM_DESC(sema_ce_hold_ms, "Hold an unlatched CE channel's completion open for up to N ms so the ring stays live for the latch (0 = off)"); + +/* + * SESSION 134c rank-aware first-submission capture (see the capture below). + * The CE block convention: ring at 0x80000 + 0x20000 * rank, USERD at + * ring + 0x10000 (observed for every CE channel across boots; the same + * convention svc_chan_userd applies in its regular per-pass latch). + */ +#define SVC_CE_RING_FB_BASE 0x80000ULL +#define SVC_CE_RING_FB_STRIDE 0x20000ULL + +/* + * One candidate offset for the CE first-submission capture: validate the + * ring shape (`scan_hits` counts shape hits) and, on a shape hit, run the + * standard USERD gate at +0x10000/+0x2000/+0x1000/+0x0; a candidate whose + * USERD reads coherent (or idle 0/0) is PINNED and LATCHED exactly like + * svc_ring_autopin's CE path. Returns true when latched. + */ +static bool svc_ce_firstsub_try_off(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u8 *sbase, + u64 lim, u64 off, u32 *scan_hits, + u64 *latch_off, u64 *latch_upa, + u32 *latch_put, u32 *latch_get, + bool own_block) +{ + static const u64 udeltas[] = { 0x10000, 0x2000, 0x1000, 0x0 }; + u32 h0, h1, hlen; + u64 hva; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + if ((hva & 0xfffff) || hva < 0x120000000ULL || + hva >= 0x121000000ULL || hlen < 4 || hlen > 0x10000) + return false; + (*scan_hits)++; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + int prc; + + if (upa + 4 > lim) + continue; + prc = svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3); + /* + * SESSION 144d: the rank's OWN convention block (own_block, + * the +0x10000 candidate at the rank's convention offset) is + * latched even when its stale content probes incoherent + * (g > p) or idle - the guest rewrites put/get on its next + * submission and the pump's per-channel last_get processes + * entry0 exactly once. r813: client #2's convention block at + * fb 0x90000 read put=1/get=146 (g > p -> -EINVAL) at the + * capture instant, the coherence gate rejected it, and the + * never-used 0x82000 zero block was latched instead, so the + * client's first push never drained and it hung. The gates + * below stay for every heuristic candidate (other deltas and + * the fallback scan), preserving the SESSION 135 protection. + */ + if (!own_block || ui != 0) { + if (prc && prc != -EAGAIN) + continue; + } + /* + * SESSION 135: at the FIRST-SUBMISSION instant the channel's + * submission is by definition unconsumed - the notifier clear + * this capture runs before is what would let the driver treat + * it as consumed. A candidate whose USERD reads a consumed + * state (get >= put && put > 0) cannot be this channel's live + * just-submitted work; it is a PREVIOUS session's leftover + * block surviving in VRAM (r785: the capture latched the dead + * CUDA client's caf00012 block at fb 0x80000, put=1 get=1, + * for the Vulkan channel beef0101 - the decoded entry was the + * old session's, the client stalled on its expectations). + * Idle 0/0 stays acceptable (svc_userd_probe's own rule). + */ + /* + * SESSION 144c: this consumed-state rejection is for FOREIGN + * blocks found by the heuristic deltas. The rank's OWN + * convention block (+0x10000) legitimately carries the + * previous session's consumed state until the guest + * re-initialises the reused allocation - rejecting it made + * the r812 second client fall through to a never-used + * 0x82000 block and stall forever (cefirst row: off=0x80000 + * upa=0x82000 put=0 get=0 while the client waited for the + * first push's completion). The pump's per-channel last_get + * still processes entry0 exactly once, so latching the + * convention block with stale put/get is correct. + */ + if (!own_block && p3 && p3 == g3) + continue; + /* latch exactly like the CE autopin */ + svc_pin_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + set_bit(ch->ce_index, &svc_ring_done); + svc_ring_n++; + *latch_off = off; + *latch_upa = upa; + *latch_put = p3; + *latch_get = g3; + return true; + } + return false; +} + +/* + * r1644: only the guest's UVM channel pool polls the semaphore page at + * 0x121000000 -- its CE rings are registered at 0x121010000 + 0x30000*rank + * (the CE GPFIFO window). Other CE channels (the RC + * watchdog at 0x120000400, the CeUtils scrubber at 0x120064000, ...) never + * wait on a pool slot, so authoring their rank slot would corrupt UVM's + * tracker words. Keep the fallback to the pool's own ring window. + */ +static bool svc_ce_uvm_pool_channel(const struct svc_channel *ch) +{ + return ch && ch->engine_type && ch->ce_index < 32 && + ch->gpfifo_va >= 0x121000000ULL && + ch->gpfifo_va < 0x121800000ULL; +} + +/* Decode one GPFIFO entry as a CE push: 1 MB-aligned VA in the CE window. */ +static bool svc_ce_push_entry(const u8 *e, u64 *va, u32 *len) +{ + u32 e0, e1; + + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + *va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + *len = ((e1 >> 10) & 0x1fffff) << 2; + return !(*va & 0xfffff) && *va >= 0x120000000ULL && + *va < 0x121000000ULL && *len >= 4 && *len <= 0x10000; +} + +/* + * r1644: resolve a CE channel's init GPFIFO entry without relying on the + * ring VA's translation alone. Candidates, in order: + * 1. the client's own page tables (walk_vaspaces / walk_discovered) for + * the registered GPFIFO VA: the learned/pin layers can answer with a + * stale page at the first submission (r1644: the cached answer was + * 0x22b0000 while the VAS walk named 0x2640000), and the walks are the + * ground truth for the VA the guest actually submits through; + * 2. the registered GPFIFO VA through the trusted resolver (the historical + * source; may answer a zero page when the guest's BAR1 ring writes and + * its GMMU mapping disagree, the SESSION 134 class); + * 3. the allocation's own USERD memdesc minus the CE layout deltas + * (0x10000 is the observed ring-to-USERD distance; 0x2000/0x1000/0 + * cover the combined-allocation layouts). The USERD is the guest's + * declaration, so this is the one source that uses alloc data. + * Entry 0 is tried first (channel_init is a pristine channel's first push); + * when the registered USERD names a put, the newest claimed entry (put-1) + * is tried as well. A hit is evidence for the log and names the exact + * entry/ring; the release itself is authored from the rank (the entry read + * is not a latch gate). + */ +static bool svc_ce_init_entry(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put, + u64 *ring_pa, int *ring_ap, + u64 *entry_va, u32 *entry_len, + u32 *entry_idx) +{ + static const u64 udeltas[] = { 0x10000, 0x2000, 0x1000, 0x0 }; + u64 cand[ARRAY_SIZE(udeltas) + 1]; + u32 as[ARRAY_SIZE(udeltas) + 1]; + u32 nc = 0, i, k; + u64 va = 0; + u32 len = 0; + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u64 vpage = ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1); + + if (!svc_walk_vaspaces(vgpu, vpage, &wpa, &wap) && wpa) { + cand[nc] = wpa + (ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + as[nc++] = (u32)wap; + } + wpa = 0; + wap = SVC_APERTURE_SYS; + if (!svc_walk_discovered(vgpu, vpage, &wpa, &wap) && wpa) { + cand[nc] = wpa + (ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + as[nc++] = (u32)wap; + } + if (!svc_lookup(vgpu, ch->gpfifo_va, ring_pa, ring_ap) && *ring_pa) { + cand[nc] = *ring_pa; + as[nc++] = (u32)*ring_ap; + } + if (ch->userd_pa) { + for (i = 0; i < ARRAY_SIZE(udeltas); i++) { + if (ch->userd_pa < udeltas[i]) + continue; + cand[nc] = ch->userd_pa - udeltas[i]; + as[nc++] = ch->userd_as; + } + } + for (i = 0; i < nc; i++) { + u32 idxs[2]; + u32 nidx = 0; + + idxs[nidx++] = 0; + if (put && put <= ch->gpfifo_entries) { + k = (put - 1) % ch->gpfifo_entries; + if (k) + idxs[nidx++] = k; + } + for (k = 0; k < nidx; k++) { + if (svc_read_ap(vgpu, cand[i] + + (u64)idxs[k] * SVC_GPFIFO_ENTRY_SIZE, + as[i], e, sizeof(e))) + continue; + if (!svc_ce_push_entry(e, &va, &len)) + continue; + *ring_pa = cand[i]; + *ring_ap = (int)as[i]; + *entry_va = va; + *entry_len = len; + *entry_idx = idxs[k]; + return true; + } + } + return false; +} + +/* + * r1644: author the UVM init push's tracker release for a CE channel whose + * ring/USERD discovery failed. The work-submit notifier edge (or a put read + * from the allocation's USERD) is the submission evidence; the release is + * then the channel's own pool slot (rank) with payload 1, exactly what + * channel_init() waits for. svc_sem_write() runs the pool resolver with its + * pin / boot-cache / walk candidates and the dual-page fan-out, and queues + * the release for the retry pass when no translation exists yet, so the + * write reaches the page the guest polls (observed 0x110b44000) even when + * the FB candidate (0x2600000 family) is the one the translations name. + * One-shot per CE rank per session. Diagnostic names rank, USERD PA/AP and + * put on both the found and not-found paths. + */ +static void svc_ce_init_rel_author(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, bool submitted, + const char *why) +{ + u64 tva, ring_pa = 0, entry_va = 0, upa; + int ring_ap = SVC_APERTURE_SYS; + u32 entry_len = 0, entry_idx = 0; + u32 put = 0, get = 0, uas; + int prc = -1; + bool put_ok = false, found; + int wrc; + + /* + * r1645: do not gate on ch->userd_ok. A latched/known USERD does not + * imply the init push was decoded (rank 5 latched with put=get=0); + * the one-shot rank bit is the only duplicate guard needed. + */ + if (!svc_ce_uvm_pool_channel(ch)) + return; + if (test_bit(ch->ce_index, &svc_ce_initrel_done)) + return; + + upa = ch->userd_pa; + uas = ch->userd_as; + /* + * The allocation's USERD is the guest's own declaration of the + * channel's GP_PUT block. It can be pool-relative on some boots (the + * r46 lesson), so its put is evidence, never a latch gate. + */ + if (upa && !svc_read_ap(vgpu, upa + SVC_USERD_GP_PUT, uas, + &put, sizeof(put))) { + (void)svc_read_ap(vgpu, upa + SVC_USERD_GP_GET, uas, + &get, sizeof(get)); + if (put <= ch->gpfifo_entries) + put_ok = true; + } + if (!put_ok) { + /* The CE layout keeps the control block at ring + 0x10000. */ + u64 rpa = 0; + int rap = SVC_APERTURE_SYS; + + if (!svc_lookup(vgpu, ch->gpfifo_va, &rpa, &rap) && rpa) { + prc = svc_userd_probe(vgpu, rpa + 0x10000, rap, + ch->gpfifo_entries, &put, &get); + if (prc != -ENOENT && prc != -EINVAL) { + upa = rpa + 0x10000; + uas = (u32)rap; + put_ok = true; + } + } + } + if (!submitted && put_ok && put) + submitted = true; + + found = svc_ce_init_entry(vgpu, ch, put_ok ? put : 0, &ring_pa, + &ring_ap, &entry_va, &entry_len, &entry_idx); + + if (svc_ce_initrel_log < 64) { + svc_ce_initrel_log++; + if (found) + pr_info("sema-ceinit: FOUND h=0x%x rank=%u gpfifo=0x%llx userd=0x%llx/%u put=%u get=%u ring=0x%llx/%d entry=%u va=0x%llx len=%u (%s)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + upa, uas, put, get, ring_pa, ring_ap, + entry_idx, entry_va, entry_len, why); + else + pr_info("sema-ceinit: NOTFOUND h=0x%x rank=%u gpfifo=0x%llx userd=0x%llx/%u put=%u get=%u (%s%s)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + upa, uas, put, get, why, + submitted ? "" : ", no submission edge"); + } + if (!submitted) + return; + if (test_and_set_bit(ch->ce_index, &svc_ce_initrel_done)) + return; + /* + * r1650: keep re-fanning the rank slot for a short window. The UVM + * pool page can move (and a per-client VAS directory can be + * discovered) seconds after the first submission, so a single + * one-shot write can race the translation; the counter is + * forward-only, so the pump's periodic re-write is harmless. + */ + svc_ce_initrel_until[ch->ce_index] = jiffies + 10 * HZ; + svc_ce_initrel_next[ch->ce_index] = + jiffies + msecs_to_jiffies(250); + + /* + * UVM's per-channel tracking semaphore is pool slot rank (allocation + * order); the init push's value is 1. Forward-only and idempotent: a + * slot that already reads >= 1 is left alone by the applier. + */ + tva = SVC_UVM_POOL_MAP_BASE + 4ULL * (u64)ch->ce_index; + svc.cur_ch = ch->h_object; + wrc = svc_sem_write(vgpu, ch->h_object, tva, 1); + svc_ce_initrel_val[ch->ce_index] = 1; + pr_info("sema-ceinit: h=0x%x rank=%u tracker=0x%llx val=1 %s (%s; init push release)\n", + ch->h_object, ch->ce_index, tva, + wrc == 0 ? "ok" : (wrc == -ENOENT ? "queued" : "failed"), why); +} + +static void svc_ce_firstsub_capture(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 ring_pa = 0, wpa = 0; + u32 e0 = 0, e1 = 0, walk_e0 = 0, walk_e1 = 0; + u32 scan_hits = 0, latched = 0; + u64 latch_off = 0, latch_upa = 0; + u32 latch_put = 0, latch_get = 0; + int ring_ap = 0, wap = SVC_APERTURE_SYS; + int rrc = -1, wrc = -1; + + if (ch->engine_type == 0 || ch->ce_index >= 32) + return; + if (svc_ce_firstsub_n[ch->ce_index] >= 4) + return; + svc_ce_firstsub_n[ch->ce_index]++; + + if (sbase && lim >= 0x1000) { + /* 1. the channel's own translation at this instant. */ + rrc = svc_lookup(vgpu, ch->gpfifo_va, &ring_pa, &ring_ap); + if (!rrc && ring_pa) { + u8 e[8]; + + if (!svc_read_ap(vgpu, ring_pa, ring_ap, e, + sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + } + } + /* 2. the trusted walk at this instant. */ + if (!svc_lookup_trusted(vgpu, ch->gpfifo_va, &wpa, &wap)) { + u8 e[8]; + + wrc = 0; + if (!svc_read_ap(vgpu, wpa, svc_ap_of(wap), e, + sizeof(e))) { + memcpy(&walk_e0, e, 4); + memcpy(&walk_e1, e + 4, 4); + } + } + /* 3. shadow scan for a CE-window ring page. */ + { + /* + * SESSION 134c rank-aware order: try the rank's + * convention offset first, then the bounded scan, + * and never accept an offset another CE rank already + * latched. The rank-blind first-hit latched rank 0's + * block for rank 1 on r779 (both rows read + * off=0x80000 upa=0x90000), so the rank-1 channel's + * first release decoded from rank 0's ring/page and + * was stale-dropped (cur=1 already) - the client's + * rank-1 tracker wait never completed and it spun + * forever. A rank whose block is not in the shadow + * at this instant now simply stays unlatched for the + * regular per-pass CE latch instead of stealing a + * foreign block. + */ + u64 exp = SVC_CE_RING_FB_BASE + + SVC_CE_RING_FB_STRIDE * ch->ce_index; + u64 off; + + latched = svc_ce_firstsub_try_off(vgpu, ch, sbase, + lim, exp, &scan_hits, + &latch_off, &latch_upa, + &latch_put, &latch_get, + true); + for (off = 0; !latched && off + 0x90 <= lim; + off += 0x100) { + unsigned int ri; + bool foreign = false; + + if (off == exp) + continue; + for (ri = 0; ri < 32; ri++) { + if (ri == ch->ce_index || + !svc_ce_firstsub_off[ri]) + continue; + if (svc_ce_firstsub_off[ri] == off) { + foreign = true; + break; + } + } + if (foreign) + continue; + /* + * SESSION 144: symmetric protection - a CE + * fallback scan must not steal a block another + * channel (CE rank or GR) already claims. + */ + if (svc_fb_claim_foreign(vgpu, ch->h_object, off, + off + 0x10000)) + continue; + latched = svc_ce_firstsub_try_off(vgpu, ch, + sbase, lim, + off, &scan_hits, + &latch_off, + &latch_upa, + &latch_put, + &latch_get, + false); + } + if (latched) { + svc_ce_firstsub_off[ch->ce_index] = latch_off; + /* + * SESSION 144: publish the claim so a GR rank + * whose own ring is not written yet cannot + * latch this CE block (r805-r808: every CUDA + * GR rank stole the CE rank-0 block and the + * launch was never captured). + */ + svc_fb_claim_add(ch->h_object, latch_off, + latch_upa); + } + } + } + + if (gsp_sema_dbg && svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u n=%u rrc=%d rpa=0x%llx rap=%d e0=0x%x e1=0x%x wrc=%d wpa=0x%llx wap=%d we0=0x%x we1=0x%x scan_hits=%u latch=%u off=0x%llx upa=0x%llx put=%u get=%u\n", + ch->h_object, ch->ce_index, + svc_ce_firstsub_n[ch->ce_index], rrc, ring_pa, + ring_ap, e0, e1, wrc, wpa, wap, walk_e0, walk_e1, + scan_hits, latched, latch_off, latch_upa, + latch_put, latch_get); + } + if (latched) { + pr_info("sema-cefirst: LATCHED h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd 0x%llx put=%u get=%u; first-submission capture)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + latch_off, latch_upa, latch_put, latch_get); + } + /* + * r1645: a latch is not a release. Rank 5 latched its fb ring + * (0xa50000) while put/get stayed 0 and the pinned push buffer never + * produced a decode, so the guest spun on slot 5 forever. The + * first-submission notifier edge proves the guest submitted the init + * push, so author the rank-slot release for every UVM pool channel, + * latched or not. One-shot per rank and forward-only, so the normal + * decode path is unaffected when it does land. + */ + svc_ce_init_rel_author(vgpu, ch, true, "first-submission"); +} + +/* + * SESSION 153-CONTINUED-41 (mission: advance the client's completed side + * host-side): the direct GL client's kernel channels (h=0xbeef01xx) arm + * their OWN completion sentinel. Glcore's submit pump writes 0xffff into + * a u16 at notifier-page+0x0e+i*16 (i=0..5) and spins until the word + * changes (glcore+0xB1A313 scan loop; live: armed at +0x2e during the + * yield wait, SESSION 40/41). On real hardware the RM completion + * overwrites that word; the emulation never did, so every ladder attempt + * timed out (~11 s), the client fell back to no-op frames and GP_PUT was + * never stored. Proof by host-side emission: blindly writing 0 over the + * armed word took a live client from ~23.4 s/frame to ~11.7 s/frame. + * The channel notifier proper starts at `base` (page+0x60 for this + * family), so the client's six 16-byte records are just below it; clear + * any that read 0xffff. Only the client family is touched - the X DDX's + * bfef-family member page uses a different all-zero wait discipline and + * a 0xffff there means a delivered error, so it must not be wiped. + */ +static bool gsp_sema_client_slot = true; +module_param_named(sema_client_slot, gsp_sema_client_slot, bool, 0644); +MODULE_PARM_DESC(sema_client_slot, + "Clear the GL client's armed 0xffff completion slots (beef-family notifier pages)"); +static u32 svc_client_slot_n; + static void svc_notify_complete(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -12765,6 +28427,32 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, if (!ch->notifier_pa) return; + if (gsp_sema_client_slot && + (ch->h_object & 0xfffff000u) == 0xbeef0000u) { + u64 page = base & ~((u64)SEMA_PAGE_SIZE - 1); + int i; + + for (i = 0; i < 6; i++) { + u64 pa = page + 0x0e + (u64)i * SVC_NOTIFY_SIZE; + u16 st = 0, z = 0; + + if (pa + sizeof(st) > base) + break; + if (svc_read_ap(vgpu, pa, ch->notifier_as, + &st, sizeof(st))) + continue; + if (st != 0xffff) + continue; + if (!svc_write_ap(vgpu, pa, ch->notifier_as, + &z, sizeof(z))) { + svc_client_slot_n++; + if (gsp_sema_dbg && svc_client_slot_n <= 64) + pr_info("sema-chan: client-slot cleared h=0x%x slot=%d pa=0x%llx\n", + ch->h_object, i, pa); + } + } + } + /* * P6c C3: while a delivered fault notifier is latched the pump must * not clear the error slot. The latch clears when the guest ack reads @@ -12799,6 +28487,15 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, svc_read_ap(vgpu, pa + SVC_NOTIFY_INFO32_OFF, ch->notifier_as, &info32, 4)) continue; + /* + * CONT.408: the X DDX family's slot 0 carries the kick-idle + * marker 0xFFFF (pb_kickIfIdle). It is free, not busy - + * zeroing it blocked every later kick (r1706 live). A real + * 0x8000 arm on slot 1 still counts. + */ + if (svc_kick_idle_page(ch) && slot == 0 && + status == 0xffff && !info32) + continue; if (status || info32) busy = 1; } @@ -12806,9 +28503,58 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, return; /* + * SESSION 134: a visible doorbell means a submission just landed - + * for a CE-class channel this is the one instant its ring content + * and USERD are both live. Capture/snapshot it BEFORE the clear + * write below (the clear is what makes the guest treat the + * submission as complete and recycle the ring), see + * svc_ce_firstsub_capture. + */ + if (ch->engine_type) + svc_ce_firstsub_capture(vgpu, ch); + + /* + * SESSION 134: keep an unlatched CE submission open so the ring + * content survives until the latch can read it (see + * svc_ce_hold_ms). The regular per-pass latch path retries while + * the hold runs; the hold expires un-latched into the old + * behaviour, so a channel the emulation can never resolve cannot + * strand the guest. + */ + if (ch->engine_type && !ch->userd_ok && svc_ce_hold_ms && + ch->ce_index < 32) { + if (!svc_ce_hold_since[ch->ce_index]) + svc_ce_hold_since[ch->ce_index] = jiffies; + if (time_before(jiffies, + svc_ce_hold_since[ch->ce_index] + + msecs_to_jiffies(svc_ce_hold_ms))) + return; + if (svc_ce_firstsub_log < 16) { + svc_ce_firstsub_log++; + pr_info("sema-cefirst: h=0x%x rank=%u completion hold expired (%llu ms) unlatched; clearing\n", + ch->h_object, ch->ce_index, svc_ce_hold_ms); + } + svc_ce_hold_since[ch->ce_index] = jiffies; + } + + /* * Clear both slots: they are adjacent, and whichever one the client is * polling must read as completed for a healthy channel. + * + * CONT.408: the X DDX family's slot 0 is the 0xFFFF kick-idle marker + * (pb_kickIfIdle), never a completion slot - clear only the + * work-submit-token slot (1) there, or the DDX's push-buffer kick + * gate dies with the clear (r1706 live). */ + if (svc_kick_idle_page(ch)) { + if (svc_write_ap(vgpu, base + SVC_NOTIFY_SIZE, + ch->notifier_as, zero, SVC_NOTIFY_SIZE)) + return; + gsp_rm_post_channel_event(vgpu, ch->h_object); + svc.post_hint_until = jiffies + + msecs_to_jiffies(SVC_POST_HINT_MS); + return; + } if (!svc_write_ap(vgpu, base, ch->notifier_as, zero, sizeof(zero))) { ch->notified++; if (gsp_sema_dbg && @@ -12830,6 +28576,202 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, } /* + * SESSION 153-CONTINUED-42 (mission: converge the client-slot clear so a + * ladder attempt COMPLETES instead of recycling). + * + * CONT-41 found the protocol: glcore's submit pump arms a 0xffff sentinel in + * one of six u16 records just below the beef-family channel-notifier base + * (page+0x0e+i*16) and treats ANY other value as the completion. The kernel + * clear landed inside svc_notify_complete, but that hook only runs on passes + * where a completion is being processed - a client whose sentinel is armed + * while the pump has nothing else to complete still sat out its ~11 s + * deadline (r1023: attempt 2/3 had the clears logged and still retired + * submissions=0; only attempt 1 - cleared while its pushes were live - + * submitted). This sweep runs once per service pass for every live + * beef-family channel and clears ANY armed 0xffff in the six records, so + * every ladder phase releases within a pass regardless of what else the + * pump is doing. The bfef-family (X DDX member page) keeps its 0xffff + * delivered-error semantics and is not touched. + * + * Bounded: 16 channels per pass, first 96 clear events logged with the slot + * index (arm-phase tracing), then every 4096th. The log's slot index is + * what identifies which phase the client is in when it arms each record. + */ +static u32 svc_client_slot_log_n; +static unsigned long svc_client_slot_at; +/* + * r1662: glcore's kick wait (570.144: wait_kick_complete 0xa03930, predicate + * 0x9fde70) polls a u32 == 0xffffffff at notify_va + 0x200 + seq*16 + 8. + * The client clears the record to 0 and writes its tag at + * notify_va + 8 + idx*4 before submitting; the GPU completion writes the + * payload. The emulation only cleared the u16 status below the notifier, so + * every kick sat out its deadline. Complete every tagged record (the client + * clears before reuse, so completing a stale one is harmless). + */ +static bool gsp_sema_kick_complete = true; +module_param_named(sema_kick_complete, gsp_sema_kick_complete, bool, 0644); +MODULE_PARM_DESC(sema_kick_complete, + "Author the GL client's kick notification payload 0xffffffff (default 1)"); +static unsigned long svc_kick_complete_at; +static u32 svc_kick_complete_n; + +static void svc_client_slot_sweep(struct nv_vgpu_device *vgpu) +{ + struct { + u32 h; + u64 base; + u32 as; + } tab[16]; + unsigned long flags; + int i, n = 0; + + if (!gsp_sema_client_slot) + return; + /* ~1 pass per tick while the pump runs tight; plenty for an ~11 s wait. */ + if (time_before(jiffies, svc_client_slot_at)) + return; + svc_client_slot_at = jiffies + msecs_to_jiffies(2); + + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS && n < ARRAY_SIZE(tab); i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu || !c->notifier_pa) + continue; + /* + * CONT.387: the X DDX's own RM channels (h=0x01000014 and + * siblings, allocated by our RM for the server's nvidia.ko + * clients) use the same push-buffer NvNotification contract + * as the bfef family: pb_armNotifiers() writes 0x8000 at + * notif+0xe before a submit and pb_wait() polls it back to 0. + * The bfef-only filter skipped them, so the DDX's 8 s submit + * wait expired (Xorg log "WAIT (2, 8, 0x8000, ...)" then + * "Wait for channel idle timed out") even though the class2d + * NOTIFY ack had already zeroed the same page at decode time + * (before the arm). Include the 0x01xxxxxx RM-channel family + * in the sweep so the armed record is completed after the arm. + */ + if ((c->h_object & 0xfffff000u) != 0xbeef0000u && + (c->h_object & 0xfffff000u) != 0xbfef0000u && + (c->h_object & 0xff000000u) != 0x01000000u) + continue; + tab[n].h = c->h_object; + tab[n].base = c->notifier_pa; + tab[n].as = c->notifier_as; + n++; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + + for (i = 0; i < n; i++) { + u64 page = tab[i].base & ~((u64)SEMA_PAGE_SIZE - 1); + int k; + + /* + * The bfef family (the GLX client's kick channel) is the X + * DDX's 2D channel. Its NvNotification status halfword is + * marked 0x8000 (IN_PROGRESS) before a submit and the GPU + * (or RM) must write it back to 0; the emulation never did, + * so the DDX's 8 s submit wait expired and its recovery + * path (dc3d0) released the channel group and then + * dereferenced it (SIGSEGV at 0x28 in nvidia_drv.so). + * Complete every 0x8000 record in the notifier page. A + * 0xffff here is the delivered-error value and is left + * alone (the beef loop below owns the 0xffff semantics). + */ + if (((tab[i].h & 0xfffff000u) == 0xbfef0000u || + (tab[i].h & 0xff000000u) == 0x01000000u) && + time_after_eq(jiffies, svc_kick_complete_at)) { + static u8 npage[SEMA_PAGE_SIZE]; + static u32 bfef_done_n; + + if (!svc_read_ap(vgpu, page, tab[i].as, npage, + sizeof(npage))) { + for (k = 0; k < SEMA_PAGE_SIZE / 16; k++) { + u16 *st = (u16 *)(npage + 0x0e + + (u64)k * 16); + u16 z = 0; + + if (*st != 0x8000) + continue; + if (svc_write_ap(vgpu, + page + 0x0e + + (u64)k * 16, + tab[i].as, &z, + sizeof(z))) + continue; + bfef_done_n++; + if (gsp_sema_dbg && bfef_done_n <= 64) + pr_info("sema-chan: bfef notif completed h=0x%x slot=%d pa=0x%llx n=%u\n", + tab[i].h, k, + page + 0x0e + + (u64)k * 16, + bfef_done_n); + } + } + } + + if ((tab[i].h & 0xfffff000u) != 0xbeef0000u) + continue; + for (k = 0; k < 6; k++) { + u64 pa = page + 0x0e + (u64)k * SVC_NOTIFY_SIZE; + u16 st = 0, z = 0; + + if (pa + sizeof(st) > tab[i].base) + break; + if (svc_read_ap(vgpu, pa, tab[i].as, &st, sizeof(st))) + continue; + if (st != 0xffff) + continue; + if (svc_write_ap(vgpu, pa, tab[i].as, &z, sizeof(z))) + continue; + svc_client_slot_n++; + svc_client_slot_log_n++; + if (gsp_sema_dbg && (svc_client_slot_log_n <= 96 || + (svc_client_slot_log_n % 4096) == 0)) + pr_info("sema-chan: client-slot cleared(sweep) h=0x%x slot=%d pa=0x%llx n=%u\n", + tab[i].h, k, pa, svc_client_slot_log_n); + } + /* + * r1662: complete the kick-slot records (payload at + * notify+0x200+seq*16+8 == 0xffffffff). Rate-limited to + * ~100 Hz; the client's wait slice is 1 ms but its overall + * deadline is seconds, so a few passes of latency are fine. + */ + if (gsp_sema_kick_complete && + time_after_eq(jiffies, svc_kick_complete_at)) { + u32 done = 0xffffffffu; + int q; + + for (q = 0; q < 120; q++) { + u32 tag = 0; + + if (svc_read_ap(vgpu, + tab[i].base + 8 + (u64)q * 4, + tab[i].as, &tag, sizeof(tag))) + continue; + if (!tag) + continue; + if (svc_write_ap(vgpu, + tab[i].base + 0x200 + + (u64)q * SVC_NOTIFY_SIZE + 8, + tab[i].as, &done, + sizeof(done))) + continue; + svc_kick_complete_n++; + if (gsp_sema_dbg && svc_kick_complete_n <= 64) + pr_info("sema-chan: kick-slot completed h=0x%x seq=%d pa=0x%llx\n", + tab[i].h, q, + tab[i].base + 0x200 + + (u64)q * SVC_NOTIFY_SIZE + 8); + } + } + } + if (gsp_sema_kick_complete && + time_after_eq(jiffies, svc_kick_complete_at)) + svc_kick_complete_at = jiffies + msecs_to_jiffies(10); +} + +/* * Run the work the guest has submitted on each registered channel. * * A submission is visible as GPPut moving ahead of our own read position in @@ -12848,8 +28790,6 @@ static void svc_notify_complete(struct nv_vgpu_device *vgpu, * against the ring's first GPFIFO entry, so a layout change costs a failed * pin, not a wrong one. */ -#define SVC_CE_RING_FB_BASE 0x80000ULL -#define SVC_CE_RING_FB_STRIDE 0x20000ULL static unsigned long svc_ring_done; /* CE ranks whose ring is pinned */ static bool svc_fbs_scan_done; /* one-shot FB entry scan diagnostic */ @@ -12885,6 +28825,86 @@ static int svc_chan_entry0(struct nv_vgpu_device *vgpu, struct svc_channel *ch, bool gsp_sema_gr_rings = true; /* + * SESSION 132 dead-entry guard for ring candidates. + * + * The r759 false positive: a vidmem candidate (0xa0000) whose first GPFIFO + * entry (push 0x120100000/40) passed the plausibility gate - the USERD + * probe hit the channel's REAL USERD at 0xb0000, which the VRAM shape scan + * had just found 0x10000 after the candidate by coincidence - was pinned + * as the GR ring while the client's own page tables resolve the ring VA to + * a guest-RAM page (0x11623d000) whose entry0 is live (push 0x4000000/ + * 0x8bc, decoded by the util sweep). The pump then read the pinned page, + * could not fetch the entry's push (all-zero through every source), and + * skipped the entry after 39 failed rescues - the live ring stayed + * undrained and every upload behind it invisible. + * + * A candidate whose first entry cannot deliver a method stream is not this + * channel's ring. The push head is read through the client's own page + * tables (guest truth); a push that cannot be resolved at all does NOT + * mark the entry dead (the VAS may not be registered yet on an early + * pass, and vidmem-backed pushes are not in the client's tables at all). + */ +static bool svc_ring_entry_dead(struct nv_vgpu_device *vgpu, u32 e0, u32 e1) +{ + u64 va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + u32 len = ((e1 >> 10) & 0x1fffff) << 2; + u8 head[16]; + + if (!va || len < 8 || len > 0x10000) + return false; /* unusable entry; the gate rejects it */ + if (svc_read_va_trusted(vgpu, va, head, sizeof(head))) + return false; /* cannot tell: do not reject */ + return !memchr_inv(head, 0, sizeof(head)); +} + +/* + * SESSION 132: does the client's own page table (VAS walk only - never the + * pin/lat/map sources, so a bad pin cannot vote for itself) resolve the + * ring VA to a page whose first entry is LIVE? Used to reject a dead + * candidate only when a live alternative provably exists. + */ +static bool svc_ring_walk_live(struct nv_vgpu_device *vgpu, u64 gpfifo_va) +{ + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 we[8]; + u32 w0, w1; + + if (svc_walk_vaspaces(vgpu, gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &wpa, &wap)) + return false; /* walk cannot resolve: no verdict */ + if (svc_read_ap(vgpu, wpa, wap, we, sizeof(we))) + return false; + memcpy(&w0, we, 4); + memcpy(&w1, we + 4, 4); + return !svc_ring_entry_dead(vgpu, w0, w1); +} + +/* + * SESSION 153-CONTINUED-38: kernel-VA ring entry window. The direct GL + * client's kernel channels (h 0xbeef01xx) place their GPFIFO ring and + * pushbuffers in the driver's LOW aperture (0x4000000-class, + * SVC_LOW_REL_VA_LO) instead of the userspace GPU-VA window (>= 4 GiB, + * SEMA_VA_LO) every GR gate below assumed, so their entries could never + * validate and the channels never resolved a USERD (the CONT-37 alloc/free + * retry loop). An entry VA in the low family is this channel's push only + * when it sits in the same contiguous allocation as the ring VA; accept it + * within a 256 MB window around the ring, never outside the low family. + */ +static bool svc_gr_entry_win_ok(u64 ring_va, u64 eva) +{ + if (eva + SVC_GR_RAM_WIN / 2 >= ring_va && + eva < ring_va + SVC_GR_RAM_WIN / 2) + return true; + if (eva >= SVC_LOW_REL_VA_LO && + eva < SVC_LOW_REL_VA_LO + 0x10000000ULL && + eva + 0x10000000ULL > ring_va && + eva < ring_va + 0x10000000ULL) + return true; + return false; +} + +/* * Validate a candidate translation for a GR ring: the first GPFIFO entry * must decode to a 1 MB aligned push in the client VA window with a sane * length, and the USERD at +SVC_GR_USERD_OFF must be readable through the @@ -12907,8 +28927,19 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, memcpy(&e1, e + 4, 4); *va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); *len = ((e1 >> 10) & 0x1fffff) << 2; - if ((*va & 0xfffff) || *va < SEMA_VA_LO || *va >= SEMA_VA_HI || - *len < 4 || *len > 0x10000) + if ((*va & 0xfffff) || *len < 4 || *len > 0x10000) + return false; + /* SESSION 153-CONTINUED-38: low-aperture kernel-VA ring entries. */ + if ((*va < SEMA_VA_LO || *va >= SEMA_VA_HI) && + !svc_gr_entry_win_ok(ch->gpfifo_va, *va)) + return false; + /* + * SESSION 132: a candidate whose entry0 push provably reads all-zero + * through the client's own page tables cannot deliver a method + * stream (the pump's rescue path eventually skips exactly these) - + * reject it so the autopin falls through to a live candidate. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) return false; /* * Gate: the USERD must be readable. rc==0 is an active get/put @@ -12920,6 +28951,20 @@ static bool svc_gr_ring_candidate(struct nv_vgpu_device *vgpu, *get = 0; prc = svc_userd_probe(vgpu, cand_pa + SVC_GR_USERD_OFF, cand_ap, ch->gpfifo_entries, put, get); + /* + * SESSION 132: the driver places the GPFIFO and the USERD + * independently (r759: ring in guest RAM 0x11623d000, USERD in + * vidmem 0xb0000), so a walk-resolved ring cannot pass a + * same-aperture USERD probe. Accept the channel's own USERD when + * one is already known - the put/get liveness gate is unchanged. + */ + if (prc != 0 && prc != -EAGAIN && ch->userd_ok && ch->userd_pa && + ch->userd_pa != cand_pa + SVC_GR_USERD_OFF) { + *put = 0; + *get = 0; + prc = svc_userd_probe(vgpu, ch->userd_pa, ch->userd_as, + ch->gpfifo_entries, put, get); + } return prc == 0 || prc == -EAGAIN; } @@ -13373,7 +29418,7 @@ static void svc_gr_content_discover(struct nv_vgpu_device *vgpu) if (!buf) return; - for (gpa = SEMA_PA_MAX - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { + for (gpa = svc_pa_max() - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { if (svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) goto next_chunk; for (o = 0; o + 24 <= SVC_SCAN_CHUNK; o += 4) { @@ -13573,6 +29618,88 @@ bool gsp_sema_wait_gate = 1; unsigned gsp_sema_copy_dst_guard_ms = 3000; /* + * r609: the m169 first-touch window is per PAGE, but the stale-table risk it + * covers is per CLIENT TABLE BUILD. The driver's H2D relocation burst on the + * eng==0 copy channel writes N distinct first-write pages back to back, and + * because the pump holds the FIRST held entry and the rest of the chain + * decodes only after it drains, each page paid its own full window, + * serialized (r606 red draw: `sema-pend: hold h=0x5c000059 idx=90` held + * 362167.7->362170.8, idx=92 held 362177.7->362182.9; the b H2D entry ran + * 362182.882, 326 ms AFTER the launch forward -> `VADD bad=32735 + * csum=8139264`, c = a + 0). The guard's bet - a fresh pin lands inside the + * window - is a property of the client's table state, not of one page: once + * ONE device-dst page of a channel has waited the full window with NO pin + * arriving (every observed release logs "without a pin"), the same bet for + * the next page of the same burst is already lost. Age the channel for + * gsp_sema_copy_dst_age_ms after such a release; further first-write + * device-dst pages of that channel inside the window skip the hold and run + * with the walk translation - the exact translation the expired guard would + * have used anyway. The first page of every burst still pays the full + * window, so the anti-stale bet is unchanged; only the N x window + * serialization is removed (r606 class: ~20 s of chain -> one window). + * Keyed by h_object like every channel table; the session flush clears it, + * so a recycled handle can never inherit the previous client's ageing + * (the r604 lifetime lesson). 0 disables (= per-page windows, pre-r609). + */ +unsigned gsp_sema_copy_dst_age_ms = 3000; +#define SVC_DST_AGE_SLOTS 16 +static u32 svc_dst_age_ch[SVC_DST_AGE_SLOTS]; +static unsigned long svc_dst_age_j[SVC_DST_AGE_SLOTS]; +static u32 svc_dst_age_pos; +static u32 svc_dst_age_log; + +/* r609: the current decode channel's guard window expired without a pin. */ +static void svc_dst_guard_age(void) +{ + u32 i; + + if (!gsp_sema_copy_dst_age_ms) + return; + for (i = 0; i < SVC_DST_AGE_SLOTS; i++) + if (svc_dst_age_ch[i] == svc.cur_ch) { + svc_dst_age_j[i] = jiffies; + return; + } + i = svc_dst_age_pos++ % SVC_DST_AGE_SLOTS; + svc_dst_age_ch[i] = svc.cur_ch; + svc_dst_age_j[i] = jiffies; +} + +/* r609: is the current decode channel inside its post-expiry ageing window? */ +static bool svc_dst_guard_aged(void) +{ + unsigned long now = jiffies; + u32 i; + + if (!gsp_sema_copy_dst_age_ms || !svc.cur_ch) + return false; + for (i = 0; i < SVC_DST_AGE_SLOTS; i++) + if (svc_dst_age_ch[i] == svc.cur_ch && + time_before(now, svc_dst_age_j[i] + + msecs_to_jiffies(gsp_sema_copy_dst_age_ms))) + return true; + return false; +} + +/* + * r609: decide whether a first-write device-dst page pays the m169 window. + * An aged channel skips it (the burst's first page already paid and no pin + * arrived); otherwise the normal per-page window runs. + */ +static bool svc_dst_guard_should_hold(u64 va) +{ + if (svc_dst_guard_aged()) { + if (svc_dst_age_log < 16) { + svc_dst_age_log++; + pr_info("sema-svc: device-dst guard age-skipped dst=0x%llx h=0x%x (same-channel burst after an un-pinned release; walk translation)\n", + va, svc.cur_ch); + } + return false; + } + return svc_dst_guard_hold(va); +} + +/* * r231 READ-ORDER guard (default 0: witness only). A device-src copy whose * source VA has a PENDING device-dst write hold (the m169 guard window is * still open) reads the PRE-write content: the writer's bytes land only @@ -13621,6 +29748,7 @@ static bool svc_dst_guard_hold(u64 va) if (svc_dst_guard[i].first) { svc_dst_guard[i].first = 0; svc.copy_dst_relaxed++; + svc_dst_guard_age(); /* r609: burst ageing */ pr_info("sema-svc: device-dst guard released without a pin va=0x%llx (using the walk translation)\n", page); } @@ -13637,6 +29765,24 @@ static bool svc_dst_guard_hold(u64 va) } /* + * r609: clear the m169 page windows and the r609 ageing ring with the + * session. Both are keyed by VA page / h_object alone, and both would + * otherwise pre-age the next client's first touch (a page slot still inside + * its window makes the new client's first device-dst write skip the + * anti-stale wait; an ageing stamp makes the new channel's whole first + * burst skip it). The stale-table risk this guard covers is precisely the + * new-session condition, so the windows must start fresh. + */ +static void svc_dst_guard_reset(void) +{ + memset(svc_dst_guard, 0, sizeof(svc_dst_guard)); + svc_dst_guard_next = 0; + memset(svc_dst_age_ch, 0, sizeof(svc_dst_age_ch)); + memset(svc_dst_age_j, 0, sizeof(svc_dst_age_j)); + svc_dst_age_pos = 0; +} + +/* * r231 READ-ORDER guard: true while a held device-dst write for this VA * page is still inside its bounded window (the writer's bytes have not * landed yet). See the gsp_sema_read_order_guard declaration. @@ -13761,8 +29907,2319 @@ unsigned gsp_sema_ce_grace_ms = 25000; */ unsigned gsp_sema_push_hold_ms = 2500; +/* + * SESSION 153-CONTINUED-54: GR setup-push hold. + * + * The client's 12288 B C597 MME-setup push ("LOAD_MME_* program upload") + * reads unreadable/all-zero on the first-read lottery boots and used to be + * consumed ("skipping stale entry ... after 1 failed rescues") before the + * demand channel's injected pin could land, so the MME prelude capture + * missed the macro program for the whole boot. When such a push's first + * read fails, demand every 4K VA page of the push through the stalled-VA + * demand channel (pin_gr_stream harvests the rows and the guest resolves + + * injects the pages) and hold the entry pending PAST the rescue-consume + * until the pin lands (the cheap-wait guard re-reads when it does) or this + * window expires. Bounded by an absolute per-(va,slot) deadline; 0 + * disables the hold (the demand rows are still emitted). + * + * r1083-r1087: DEFAULT 0 (off). A 20 s hold blocks the whole ring behind + * the held entry, and on the three hold-active boots (r1083/r1084/r1085) + * the GL client derailed into the black-dud/churn class, while the r1087 + * boot (hold=0) read the beef windows fine (the wall moved back to the + * incomplete MME prelude + dispatch stall). The demand channel also cannot + * yet resolve the GL pushbuffer VA families (0x4000000+ / 0x42xxxxx), so + * the pin the hold waits for never lands and the hold only delays. Re-enable + * (e.g. sema_gr_setup_hold_ms=20000) once the guest content resolver covers + * those VA families. + */ +unsigned gsp_sema_gr_setup_hold_ms; + +/* + * SESSION 155 CONT 8: per-boot draw-window hold deadline for the same + * parameter. While it is open, an immature beef window is busy-deferred + * (candidate survives, pump re-offers it) instead of being software- + * completed by IMMATURE-SKIP, so the pump keeps walking the held setup-push + * entry and a late pin/upload can still mature the capture. Armed once per + * module load (= per boot) and bounded by the clamped param (<= 2000 ms). + */ +static unsigned long svc_setup_hold_wait_j; +static bool svc_setup_hold_exp_logged; + static unsigned long svc_gr_ring_tried; /* ranks that got a diagnostic line */ +/* + * SESSION 153-CONTINUED-65: per-boot map of every DISTINCT beacon + * (0xbeef0xxx) GPFIFO entry the pump walks - (idx, va, len), first + * sighting order, dedup table of 512 triples (the ring re-walks entries + * every pass, so without dedup this floods). The r1124 setup-hold bound + * itself to entries 8 (0x4001004, 60216 B) and 15 (0x3200000, 153600 B); + * the client's 0x114 instruction-RAM program lives in the 0x3200000 + * family window, so binding the hold/demand/capture class to the ENTRY + * (not a size heuristic) starts from this map. Service-thread only. + */ +static void svc_beef_entry_note(u32 h, u32 idx, u64 va, u32 len) +{ + static u64 sva[512]; + static u32 sidx[512]; + static u32 slen[512]; + static u32 n; + u32 i; + + for (i = 0; i < n; i++) + if (sva[i] == va && sidx[i] == idx && slen[i] == len) + return; + if (n >= 512u) + return; + sva[n] = va; + sidx[n] = idx; + slen[n] = len; + n++; + pr_info("sema-chan: beef entry h=0x%x idx=%u va=0x%llx len=%u\n", + h, idx, va, len); +} + +/* + * Is this a GR push whose readability the MME-setup capture depends on? + * The client's C597 setup push is 12288 B; any GR push of that size is a + * candidate. A smaller push qualifies only when it is READABLE and its + * method runs hit the MME setup families (the unreadable case has no bytes + * to scan, hence the length rule alone). + * + * SESSION 153-CONTINUED-65: the MME macro-upload family (0x3200000..) + * is setup-class by VA alone. The r1124 entry 15 (va=0x3200000, + * len=153600) carries the client's 0x114 instruction-RAM program - the + * guest RAM image of its window measures code=712 words with an exact-end + * stream walk - and the ubounded-tail demand must follow the ENTRY, not a + * size heuristic: the class match is what keeps the hold, the 4K-page + * demand and the strict pin anchored on the entry that actually feeds + * svc_mme_prelude_capture the 0x114 runs. + */ +static bool svc_gr_setup_push(u64 va, u32 len, const u8 *buf) +{ + u32 off = 0; + + if (va >= 0x3200000u && va < 0x3300000u) + return true; + if (len >= 12288u) + return true; + if (!buf) + return false; + while (off + 4u <= len) { + u32 hdr, op, cnt, addr; + + memcpy(&hdr, buf + off, 4); + op = (hdr >> 29) & 7u; + cnt = (hdr >> 16) & 0xfffu; + addr = (hdr & 0x1fffu) << 2; + if ((op == 1u || op == 3u || op == 5u) && cnt && + svc_mme_setup_run_addr(addr)) + return true; + off += ((op == 1u || op == 3u || op == 5u) && cnt) ? + (1u + cnt) * 4u : 4u; + } + return false; +} + +/* + * Demand every 4K page of the push and arm the per-(va,slot) hold. The + * demand is the existing stalled-VA channel: svc_log_stalled_va rows are + * deduped and harvested by tools/guest-diag/pin_gr_stream.py, which asks + * the guest scanner to resolve the VA set and inject the pages. Only one + * arm line per (va,slot) is logged. The loop is bounded by `page_cap` (the + * 12288 B setup push needs 3; the cap keeps a bulk push from eating the + * 256-entry stalled-VA budget). + * + * SESSION 153-CONTINUED-55: the cap was 0x8000 (8 pages). The GL client's + * observed setup push is 60216 B (16 pages); pages 1..8 were demanded but + * pages 9..15 never were, so the host's per-chunk read of the entry had no + * source for the tail chunks and the whole read failed (unreadable=1, the + * hold re-armed forever). 0x10000 covered that push within the stalled-VA + * budget. + * + * SESSION 153-CONTINUED-65: the macro-upload carrier (r1124 entry 15, + * va=0x3200000) is 153600 B = 38 pages; 0x10000 demanded only 16 of them, + * so even a resolved window could never be read whole. Raise to 0x40000 + * (256 pages) - bounded by the 512-slot demand ring and the 256-entry + * stalled-VA table, both of which keep their own eviction. + */ +static void svc_gr_setup_hold_arm(struct svc_channel *ch, u64 va, u32 idx, + u32 len) +{ + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u32 poff; + + if (!va) + return; + /* + * SESSION 153-CONTINUED-76: the page demand is DECOUPLED from the + * hold. The r1157 live A/B proved the 20 s hold derails the client + * (the ring stalls behind the unreadable setup entry and the client's + * ~10 s fence timeout churns the channel), while with the demand also + * disabled (early return) the unreadable setup push's pages are never + * demanded, never pinned, and the 0x114/0x11c/0x120 macro-program + * runs stay zeros. Keep emitting the demand rows for the entry with + * the hold off; one emission per (va,slot). + */ + if (ch->gr_setup_hold_va != va || ch->gr_setup_hold_idx != idx) { + ch->gr_setup_hold_va = va; + ch->gr_setup_hold_idx = idx; + /* + * r1405 (S153/r1404): the push-read failure/reject witnesses + * are per setup-hold generation, not per boot. On r1404 the + * 96-slot budget was spent 23 s BEFORE the first strict pin + * landed, so no post-pin rejection row could ever print and + * the ceiling had to be found offline. Reset both counters on + * every (re-)arm so the next generation's failures are always + * visible; each generation is still bounded by its own cap. + */ + svc_pushfail_log = 0; + svc_pinrej_log = 0; + /* + * SESSION 155 CONT 8: the hold is hard-bounded at 2000 ms. + * Longer values (the old r1083-r1087 20 s experiment) block + * the whole ring behind the unreadable setup entry and derail + * the client; 0 still disables the hold (demand rows remain). + */ + if (gsp_sema_gr_setup_hold_ms) { + unsigned hold_ms = min_t(unsigned, + READ_ONCE(gsp_sema_gr_setup_hold_ms), + 2000u); + + ch->gr_setup_hold_until = jiffies + + msecs_to_jiffies(hold_ms); + } else { + ch->gr_setup_hold_until = 0; + } + pr_info("sema-chan: GR setup hold h=0x%x entry=%u va=0x%llx len=%u armed (demanding 4K pages; hold=%u ms clamped<=2000; pending until the pin lands)\n", + ch->h_object, idx, va, len, + ch->gr_setup_hold_until ? + jiffies_to_msecs(ch->gr_setup_hold_until - jiffies) : 0u); + } + for (poff = 0; poff < len && poff < 0x40000u; poff += SEMA_PAGE_SIZE) { + svc_log_stalled_va(page + poff); + /* + * SESSION 153-CONTINUED-56: also publish the page in the + * debugfs demand set the pin tool reads directly, so the + * demand no longer depends on the dmesg row surviving the + * ring rotation under vgsp-sema spam. + * + * SESSION 153-CONTINUED-64: the entry's own page (poff == 0) + * additionally carries (off, len) - the exact [va, va+len) + * window the host read will parse - so the guest resolver + * can validate a candidate page range with the host's own + * full method-stream parse before injecting the pin. + */ + svc_setup_demand_record_ex(page + poff, + poff == 0 ? + (u32)(va & (SEMA_PAGE_SIZE - 1)) : 0u, + poff == 0 ? len : 0u); + } +} + +static bool svc_gr_setup_hold_active(const struct svc_channel *ch, u64 va, + u32 idx) +{ + return ch->gr_setup_hold_until && + ch->gr_setup_hold_va == va && + ch->gr_setup_hold_idx == idx && + time_before(jiffies, ch->gr_setup_hold_until); +} + +static void svc_gr_setup_hold_clear(struct svc_channel *ch) +{ + ch->gr_setup_hold_until = 0; + ch->gr_setup_hold_va = 0; + ch->gr_setup_hold_idx = ~0u; +} + +/* + * r1780 (V37): eager push-page pins at the ring-latch instant. + * + * The direct-GL client rebuilds its VAS every few seconds while its GL init + * retries; a push-VA walk is only possible while the directory incarnation + * that the ring scan just content-verified is still live. The RAM content + * scan pins the ring page, but the pushbuffer pages were left to a later + * walk that almost always landed after the next rebuild (measured c37/c39: + * the verified pdb answered the ring at +0 s and ap0 at +15..20 s; the + * 59556-byte setup push then read as an implausible window guess and was + * skipped). Resolve and pin the entry-referenced push pages inside the same + * latch, while the just-verified directory is provably alive. + * + * Only successful walks are pinned. The first page of each push must also + * pass the method-stream prefix parse, so a directory that maps the ring to + * garbage cannot pin arbitrary pages. Bounded (32 entries / 64 pages per + * latch); later reads still validate the full stream. + */ +static bool gsp_sema_pin_pushes = true; +static unsigned long svc_push_pin_at; +module_param_named(sema_pin_pushes, gsp_sema_pin_pushes, bool, 0644); +MODULE_PARM_DESC(sema_pin_pushes, + "Pin the ring entry's pushbuffer pages through the freshly verified VAS at the ring-latch instant (default 1)"); + +/* + * r1780 (V38): pin every page of a just-resolved GPFIFO ring. + * + * The pump's entry fetch is VA-based and page-granular: only the base page + * that svc_lookup() answered is pinned, so a 32768-entry (64-page) ring + * silently breaks at idx=512 (`svc.gpfifo_nomap++`, no log) and no push is + * ever decoded. Walk the whole ring through the same registered VAS right + * when the translation is live and pin each page; the pins survive the + * client's VAS churn. Rate-limited: once a page resolves, the lookup skips + * it, so steady state costs only the loop. + */ +static void svc_pin_gr_range(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 ring_pa, int ring_ap) +{ + u64 nbytes = (u64)ch->gpfifo_entries * SVC_GPFIFO_ENTRY_SIZE; + u64 npages, i; + int pinned = 0; + + if (!ring_pa || !ch->gpfifo_va) + return; + npages = DIV_ROUND_UP(nbytes, (u64)SEMA_PAGE_SIZE); + if (npages > 64) + npages = 64; + for (i = 0; i < npages; i++) { + u64 va = (ch->gpfifo_va + i * SEMA_PAGE_SIZE) & + ~(u64)(SEMA_PAGE_SIZE - 1); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (!svc_lookup(vgpu, va, &pa, &ap)) + continue; + if (svc_walk_vaspaces(vgpu, va, &pa, &ap)) + continue; + svc_pin_store(va, pa, ap); + svc_lat_store(va, pa, ap); + pinned++; + } + if (pinned) + pr_info_ratelimited("sema-chan: ring-range pin h=0x%x va=0x%llx pages=%d/%llu ring_pa=0x%llx ap=%d\n", + (u32)ch->h_object, ch->gpfifo_va, pinned, + npages, ring_pa, ring_ap); +} + +static void svc_pin_ring_pushes(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 ring_pa, int ring_ap) +{ + u8 e[8]; + int i, pinned = 0, pushes = 0; + static unsigned long last_log; + + if (!gsp_sema_pin_pushes) + return; + for (i = 0; i < 32 && pinned < 64; i++) { + u32 e0, e1, len, off; + u64 va; + + if (svc_read_va(vgpu, ch->gpfifo_va + (u64)i * 8, + e, sizeof(e))) + break; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va || len < 4 || len > 0x40000) + break; + for (off = 0; off < len && pinned < 64; off += SEMA_PAGE_SIZE) { + u64 page = (va + off) & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + + if (svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + break; + if (!off) { + u8 pre[64]; + u32 plen = min_t(u32, len, sizeof(pre)); + + if (svc_read_ap(vgpu, wpa, wap, pre, plen) || + !svc_method_stream_plausible(pre, plen)) + break; + } + svc_pin_store(page, wpa, wap); + svc_lat_store(page, wpa, wap); + pinned++; + } + pushes++; + } + if (pinned && time_after(jiffies, last_log)) { + last_log = jiffies + HZ; + pr_info("sema-chan: ring-latch push pins h=0x%x ring_pa=0x%llx entries=%d pinned_pages=%d (VAS alive at latch)\n", + ch->h_object, ring_pa, pushes, pinned); + } +} + +static bool svc_gr_ram_ring_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out, + bool owner_filter, bool sparse) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 start = ch->gr_ram_cursor; + u64 end; + u64 gpa; + /* + * r908: the GPFIFO entries start AT the registered ring VA, so their + * in-page offset is the ring VA's own offset (X server ring + * 0x120090000 -> offset 0; RC watchdog 0x120000400 -> 0x400; the GL + * client 0x1203cfff0 -> 0xff0, measured). The scan used to test only + * page offset 0, so any ring whose VA is not page-aligned could never + * match no matter how good the entry contents were. + */ + u32 eoff = (u32)(ch->gpfifo_va & (SEMA_PAGE_SIZE - 1)); + /* CONT.400: the GL client's kernel channels use the RM low aperture. */ + bool low = (ch->gpfifo_va >= SVC_LOW_REL_VA_LO && + ch->gpfifo_va < 0x10000000ULL) || + ((ch->h_object & 0xfffff000u) == 0xbeef0000u); + int i; + unsigned long flags; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) { + if (gsp_sema_dbg && + (ch->h_object & 0xfffff000u) == 0xbeef0000u) { + static unsigned long rfail_at; + static u64 rfail_gpa; + + if (time_after(jiffies, rfail_at) || + rfail_gpa != gpa) { + rfail_at = jiffies + HZ; + rfail_gpa = gpa; + pr_info("sema-chan: gr-scan READ FAIL h=0x%x gpa=0x%llx chunk=0x%llx cursor=0x%llx\n", + ch->h_object, gpa, chunk, + ch->gr_ram_cursor); + } + } + continue; + } + for (o = 0; o + eoff + 2 * SVC_GPFIFO_ENTRY_SIZE <= chunk; + o += SEMA_PAGE_SIZE) { + u32 e0, e1, f0, f1; + u64 va0, va1; + u32 len0, len1; + + memcpy(&e0, buf + o + eoff, 4); + memcpy(&e1, buf + o + eoff + 4, 4); + va0 = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len0 = ((e1 >> 10) & 0x1fffff) << 2; + /* + * entry0: a pushbuffer VA in the ring's allocation + * window - just below the ring for the long-standing + * convention (X ring: 0x120010000 for ring + * 0x120090000), or just above it (r908: the GL + * client's PROMOTE_CTX ordering shows the main + * pushbuffer above the ring on this boot; accept + * both sides, still bounded to +/-1 MB of the ring). + */ + /* + * CONT.400: the direct-GL client's kernel channels live + * in the RM low aperture (ring 0x4020000, pushbuffers + * 0x4000000/0x4001004/0x4250000/0xcae2894). Those push + * VAs are far below SEMA_VA_LO and outside +/-1 MB of + * the ring, so the high-family window below rejected + * every one of them and the ring could never latch - + * the GL draw stream was never decoded. For the low + * family accept any push VA in the low aperture and a + * much wider ring window; keep the high-family rules + * exactly as they were. + */ + if (low) { + if (va0 < SVC_LOW_REL_VA_LO || + va0 >= SVC_LOW_REL_VA_LO + 0x10000000ULL) + continue; + /* + * r1780 (CONT.430d): entry0 is the main + * pushbuffer next to the ring. Bound it to + * +/-8 MB of the registered ring VA so a + * dense/transient page elsewhere in the + * 64 MB low arena cannot masquerade as the + * ring (measured false latches at + * PA 0x108587000/0x1100b6000 while the real + * ring sat at PA 0x35331b000). + */ + if (va0 + 0x800000ULL < ch->gpfifo_va || + va0 > ch->gpfifo_va + 0x800000ULL) + continue; + } else { + if (va0 + SVC_GR_RAM_WIN / 2 < ch->gpfifo_va || + va0 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + if (va0 < SEMA_VA_LO || va0 >= SEMA_VA_HI) + continue; + } + if (len0 < 4 || len0 > 0x100000) + continue; + memcpy(&f0, buf + o + eoff + 8, 4); + memcpy(&f1, buf + o + eoff + 12, 4); + va1 = ((u64)(f1 & 0xff) << 32) | + (u64)(f0 & 0xfffffffc); + len1 = ((f1 >> 10) & 0x1fffff) << 2; + /* + * entry1: the progress tracker above entry0 - or the + * SAME push submitted again (r908 measured on the GL + * client's ring: {push=0x1203c0000,len=64} then + * {push=0x1203c0000,len=32}). Never below entry0; + * still bounded by the window above the ring. + */ + if (low) { + if (va1 < va0 || + va1 >= SVC_LOW_REL_VA_LO + 0x10000000ULL) + continue; + } else { + if (va1 < va0 || + va1 > ch->gpfifo_va + SVC_GR_RAM_WIN / 2) + continue; + } + if (len1 < 4 || len1 > 0x100000) + continue; + /* + * r979: kick-family ring discrimination. Shape alone + * admitted a foreign table page ahead of the real ring + * on r978 (pa=0x104fa9000, entries naming a bogus + * 0x12048b824/0x40000 push) while the true ring's + * entry pair names the client's per-frame record VA + * (SVC_GL_KICK_PUSH_VA, the r908-documented GL + * pushbuffer, constant across r908-r978). A + * kick-family candidate must name the record; the + * lookalike class then never latches. Non-kick + * channels keep the shape gate unchanged. + */ + if (svc_kick_family(ch) && + va0 != SVC_GL_KICK_PUSH_VA && + va1 != SVC_GL_KICK_PUSH_VA) + continue; + /* + * r1777 (V12): a CE-sibling scan must not latch a page + * another channel's ring already owns (the GR page + * passes the same low-family shape). Skip and keep + * scanning this slice; the true page may sit later. + */ + if (owner_filter && + svc_ram_ring_owner_foreign(ch->h_object, gpa + o)) + continue; + /* + * r1777 (V12): sparse-page gate for the CE family. A + * real GPFIFO ring page carries a bounded run of + * queued entries and then zero words (the r1774 GR + * page: ~17 entries then zeros); the false-positive + * class the loose shape+window rule admits (r1777 c1: + * CE 0xbeef0101 latched va=0x1131860c/393216) is a + * DENSE data array where every 8-byte slot decodes. + * Require an all-zero entry slot within the first 128 + * entries of the run. + */ + /* + * r1780 (CONT.430d): apply the terminus gate to the + * low-aperture family too. The direct-GL ring page + * carries a bounded run of queued entries and then + * zero words (measured: entry0 va=0x4000000 len=2260 + * then zeros at slot 23), while a dense data page in + * the same low arena decoded on every slot and was + * latched first at PA 0x108587000 - the pump then + * read phantom entries, the GET/PUT mismatched and + * the guest corrupted itself (bad pmd/OOM) chasing + * them. A real ring cannot be dense past slot 128. + */ + if (sparse || low) { + u32 z; + bool term = false; + + for (z = 2; z < 128; z++) { + u32 zoff = o + eoff + + z * SVC_GPFIFO_ENTRY_SIZE; + u32 q0, q1; + + if ((u64)zoff + 8 > chunk) + break; + memcpy(&q0, buf + zoff, 4); + memcpy(&q1, buf + zoff + 4, 4); + if (!q0 && !q1) { + term = true; + break; + } + } + if (!term) + continue; + } + *pa_out = gpa + o; + return true; + } + } + ch->gr_ram_cursor = (end >= top) ? 0 : end; + ch->gr_ram_passes++; + if (ch->gr_ram_cursor == 0 && !ch->gr_ram_logged) { + ch->gr_ram_logged = true; + if (ch->engine_type) + pr_info("sema-chan: CE ring h=0x%x rank=%u RAM content scan found no candidate in %u slice(s) (va=0x%llx)\n", + ch->h_object, ch->ce_index, ch->gr_ram_passes, + ch->gpfifo_va); + else + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan found no candidate in %u slice(s) (va=0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_passes, + ch->gpfifo_va); + } + /* + * Persist the cursor immediately: the pump's stalled-userd path + * `continue`s past the end-of-pass write-back, so without this the + * scan would restart from slice 0 every pass and never reach the + * ring's page. + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + /* SESSION 149 R15: the M1.1b keep-alive budget state. */ + svc.chan[i].gr_ram_kick = ch->gr_ram_kick; + svc.chan[i].gr_ram_refreshes = ch->gr_ram_refreshes; + svc.chan[i].gr_ram_refresh_at = ch->gr_ram_refresh_at; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + return false; +} + +/* + * SESSION 153-CONTINUED-19 (kick-family USERD acceptance). + * + * The glxgears/X GLX client's kick channel (h 0xbfef00xx, GR engine, ring + * found by the guest-RAM content scan) posts a per-frame completion kick + * the client's flush loop paces on. Its USERD is a sysmem control block + * the driver allocates far from the ring - r971 measured the ring at pa + * 0x12f9ed000 and the live USERD at pa 0x1052ae000 (~681 MB away; the + * same nomination was +0.6..+2.3 GB across r966-r971) - while the general + * RAM USERD scan's allocation-locality gate only admits candidates within + * +/-64 MB of the ring (SESSION 153-CONT-7, built for the r936-r939 + * nine-decoy class). That gate rejected the channel's real USERD on every + * boot since the gate landed, the scan retired ("RAM scan found no + * candidate in 20 slice(s)"), the pump never iterated the kick entries + * and the kick-notify lane never ran. For this family the ring-entry + * match (every PUT-claimed ring entry must decode to a pushbuffer VA + * inside the ring's own allocation window) is the ownership proof; the + * locality distance is not - so the gate is waived for these channels and + * the distance is logged instead. The pump's speculative-latch + * confirmation (a +1/+2 PUT step, svc_pump_channels) remains the final + * judge, and the rejected-latch memory prevents a frozen block from ever + * pinning the scan. + */ +static bool svc_kick_family(const struct svc_channel *ch) +{ + return ch && !ch->engine_type && + ch->h_object >= 0xbfef0000u && ch->h_object <= 0xbfef00ffu; +} + +/* + * Re-validate the kick channel's ring latch each pass (bounded to once a + * second per channel). The RAM content scan latches a ring page once; if + * the driver later zeroes or reuses that page (the r966-r971 class the + * design brief describes: the USERD scan then validates every candidate + * against a dead ring and adopts none), the latch is dropped here so the + * content scan re-runs and re-latches the channel's current ring page. + * The stale entry pair is logged on re-arm so the transition is visible + * in the record. A live ring (a first entry that still decodes to a + * window-bounded pushbuffer VA) is a cheap no-op. + */ +static void svc_kick_ring_revalidate(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0; + unsigned long flags; + int i; + + if (!svc_kick_family(ch) || !ch->gr_ram_ring || ch->userd_ok) + return; + if (!time_after(jiffies, ch->gr_kick_ring_reval_at)) + return; + ch->gr_kick_ring_reval_at = jiffies + msecs_to_jiffies(1000); + if (!svc_read_va(vgpu, ch->gpfifo_va, e, sizeof(e))) { + u64 eva; + u32 elen; + + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + elen = ((e1 >> 10) & 0x1fffff) << 2; + if (eva && elen >= 4 && elen <= 0x100000) + return; /* ring page live; nothing to do */ + } + /* + * The latched page no longer decodes entries: drop the one-shot + * latch state so the RAM ring content scan re-runs. The pin is left + * in place - a stale pin only shadows VA reads, while the content + * scan validates against raw RAM and re-pins on a hit. + */ + clear_bit(ch->gr_index, &svc_gr_ring_done); + clear_bit(ch->gr_index, &svc_pb_gr_done); + ch->gr_ram_cursor = 0; + ch->gr_ram_passes = 0; + ch->gr_ram_logged = false; + ch->gr_ram_ring = false; + pr_info("sema-chan: kick ring re-arm h=0x%x va=0x%llx (latched entry e0=0x%x e1=0x%x no longer decodes; content scan re-runs)\n", + ch->h_object, ch->gpfifo_va, e0, e1); + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[i].gr_ram_passes = ch->gr_ram_passes; + svc.chan[i].gr_ram_logged = ch->gr_ram_logged; + svc.chan[i].gr_ram_ring = ch->gr_ram_ring; + svc.chan[i].gr_kick_ring_reval_at = + ch->gr_kick_ring_reval_at; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); +} + +/* + * SESSION 153-CONTINUED-75: GR ring stale-pin self-heal. + * + * The GR ring pin can outlive the page it names. The client churns its + * channels every ~12 s (fence timeouts), frees the ring allocation, and the + * driver's allocator reuses the PA. Live evidence (r1156): the beef ring + * pin 0x238041000 later held glxgears vertex/matrix floats and every + * "entry" the pump decoded from it was float data - the phantom held push + * va=0x3e4ccccc len=123916 is exactly the float pair (0x3e4ccccd = 0.2f, + * 0x81e40e00). A stale pin shadows the client's own page tables in + * svc_lookup, so the channel runs on phantom entries and never retires. + * + * Once per second, when the current translation's entry0 no longer decodes + * as a ring entry (dead, zeroed or float junk), ask the client's own VAS + * walk for the ring VA. If the walk resolves a DIFFERENT page whose entry0 + * DOES decode through the same window/length gate every adoption path uses, + * re-pin to that page and log it. With no walk, or a walk page that also + * fails the gate, nothing is touched: the heal can only replace a + * provably-dead translation with a provably-plausible one. + */ +static u32 svc_gr_heal_log; +static u64 svc_gr_heal_n; + +static bool svc_gr_entry_plausible(const struct svc_channel *ch, u32 e0, + u32 e1) +{ + u64 eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + u32 elen = ((e1 >> 10) & 0x1fffff) << 2; + /* + * CONT.400: the low-aperture GL client's pushes are 4-byte granular + * (0x4001004, 0x4250000, 0xcae2894); the 1 MB alignment test below is + * a high-family convention and rejected every one of them. Keep it + * for the high family only. + */ + bool low = (ch->gpfifo_va >= SVC_LOW_REL_VA_LO && + ch->gpfifo_va < 0x10000000ULL) || + ((ch->h_object & 0xfffff000u) == 0xbeef0000u); + + if ((low ? (eva & 0x3u) : (eva & 0xfffffu)) || elen < 4 || + elen > 0x10000) + return false; + if (low) { + if (eva < SVC_LOW_REL_VA_LO || + eva >= SVC_LOW_REL_VA_LO + 0x10000000ULL) + return false; + } else if ((eva < SEMA_VA_LO || eva >= SEMA_VA_HI) && + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) { + return false; + } + return true; +} + +static void svc_gr_ring_stale_heal(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0; + u64 wpa = 0, cpa = 0; + int wap = SVC_APERTURE_SYS, cap = SVC_APERTURE_SYS; + + if (!ch->gpfifo_va || ch->engine_type || ch->gr_index >= 32) + return; + if (!time_after(jiffies, svc_gr_heal_at[ch->gr_index])) + return; + svc_gr_heal_at[ch->gr_index] = jiffies + msecs_to_jiffies(1000); + + /* + * Entry0 through the CURRENT translation (the pin shadows the walk in + * svc_lookup). An unreadable read gets the walk check too - the pin + * may name freed memory the read path rejects. + */ + if (!svc_read_va(vgpu, ch->gpfifo_va, e, sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if (svc_gr_entry_plausible(ch, e0, e1)) + return; /* live ring; nothing to do */ + } else { + e0 = 0; + e1 = 0; + } + + if (svc_pin_lookup_any(ch->gpfifo_va, &cpa, &cap)) + cpa = 0; + + /* The client's own page tables answer the ring VA right now? */ + if (svc_walk_vaspaces(vgpu, ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + &wpa, &wap)) + return; /* no walk: no verdict, leave the pin alone */ + if (cpa && (wpa & ~0xfffULL) == (cpa & ~0xfffULL)) + return; /* walk agrees with the implausible pin: dead either way */ + if (svc_read_ap(vgpu, wpa, wap, e, sizeof(e))) + return; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if (!svc_gr_entry_plausible(ch, e0, e1)) + return; /* the walk page is not a live ring either */ + + svc_pin_store(ch->gpfifo_va, wpa, wap); + svc_lat_store(ch->gpfifo_va, wpa, wap); + svc_gr_heal_n++; + if (svc_gr_heal_log < 16) { + svc_gr_heal_log++; + pr_info("sema-chan: GR ring re-pin h=0x%x va=0x%llx -> pa=0x%llx ap=%d (stale translation 0x%llx no longer decodes; walk entry0 va=0x%llx len=%u)\n", + ch->h_object, ch->gpfifo_va, wpa, wap, cpa, + ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc), + ((e1 >> 10) & 0x1fffff) << 2); + } +} + +/* + * r912 (M-GL1.2): guest-RAM USERD shape scan for a GR channel whose ring + * was content-validated in guest RAM (gr_ram_ring; svc_gr_ram_ring_scan). + * Such a client's USERD is a system-memory object like its ring, so the + * FB-shadow shape scan inside svc_chan_userd can never reach it: the + * registered userd value is pool-relative, every FB candidate reads + * garbage, and the pump never iterates the GPFIFO entries ("unregistered + * (submissions=0)"). Scan guest RAM for the USERD control block shape + * (GET at +0x88, PUT at +0x8c - SVC_USERD_GP_*), with the same sliced, + * cursor-resumed budget as the ring scan (one SVC_GR_RAM_SLICE slice per + * pass, svc_gr_ram_pass_budget() slices before giving up). + * + * A candidate passes the shape and entry gates first (every entry its PUT + * claims must decode to a pushbuffer VA inside the ring's allocation + * window: entry k at ring VA + (k % entries)*8, within +/-SVC_GR_RAM_WIN/2 + * of the ring VA with a sane length), but admission is by LIVENESS, not by + * shape (SESSION 154): the GET/PUT gate alone admitted frozen or churning + * lookalike pages (r914: a low-RAM table whose +0x88/+0x8c words happened + * to read 12/8 and later 8/4 - adopted, the pump then read PUT=12 with + * entries=2, the modulo collapsed to 0 and NOTHING was consumed while the + * client retried for minutes). A passing candidate is WATCHED across + * passes; only a block whose PUT takes the client's own small step (+1 or + * +2) on two separate sightings is adopted as the live control block (the + * measured live block advances 1->2->3; the lookalikes sit still or jump + * by 4). The watch runs every call, so it keeps working past the slice + * budget (the gate in svc_chan_userd keeps calling while a watch is live). + */ +static bool svc_gr_ram_userd_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 *pa_out, + u32 *put_out, u32 *get_out) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 ring_pa = 0; + u64 start = ch->gr_ram_userd_cursor; + u64 end, gpa; + u32 entries = ch->gpfifo_entries; + u64 live_pa = 0; + u32 live_put = 0, live_get = 0, live_prev = 0; + int ring_ap = SVC_APERTURE_SYS; + bool kick; + u32 wi; + int i; + unsigned long flags; + + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE || !entries) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + kick = svc_kick_family(ch); + /* + * SESSION 153-CONTINUED-19: re-check the kick channel's ring latch + * before validating candidates against it (svc_kick_ring_revalidate + * re-latches a ring page the driver has zeroed since the latch; see + * its comment). + */ + svc_kick_ring_revalidate(vgpu, ch); + (void)svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, &ring_ap); + /* + * SESSION 154 (M-GL1.2): liveness re-check of the watched candidates. + * Cheap (one 4-byte PUT read each, every pass) and the primary adoption + * site: a watched block whose PUT takes the client's own single +1 step + * is the live control block (the measured live block advances 1->2->3 + * in 2-8 s steps; the registration window is 25-45 s, so one step must + * suffice). A block whose PUT jumps by a larger amount or decreases is + * a churning lookalike (r914's pa 0x9e200 marched -4; the dense + * lookalike pools churn +14) and its streak resets; one that goes + * unreadable or zero is dropped. + */ + for (wi = 0; wi < ch->gr_ram_userd_watch_n; ) { + u64 wpa = ch->gr_ram_userd_watch[wi]; + u32 hp = 0, hg = 0, last; + + /* + * SESSION 153-CONT-7 (ownership by allocation locality): + * drop a watched candidate that sits far outside the + * channel's ring allocation. r939 measured: the ring at pa + * 0x1225ac000, the client's sysmem blocks (memfd) at pa + * 0x136b9c000..0x136eca000 (~21 MB away), while the + * latched pair at pa 0x106e8xxxx sat ~460 MB from the ring, + * mapped by NO process, and never advanced on any pass + * (two frozen snapshots that cost a 15 s reject/re-latch + * cycle all boot). A control block this far from its own + * ring is not this channel's USERD. SESSION 153-CONT-19: + * waived for the kick family - its real USERD is measured + * 0.6-2.3 GB from the ring, and the ring-entry match (not + * proximity) is the ownership proof there (see + * svc_kick_family). + */ + if (ring_pa && !kick && + (wpa < ring_pa - (SVC_GR_RAM_WIN << 5) || + wpa > ring_pa + (SVC_GR_RAM_WIN << 5))) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + continue; + } + if (svc_read(vgpu, wpa + SVC_USERD_GP_PUT, &hp, + sizeof(hp)) || !hp || hp > 0x100000) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + continue; + } + last = ch->gr_ram_userd_watch_put[wi]; + if (hp == last) { + /* + * SESSION 153-CONTINUED-20: stale-entry timeout. A + * kick-family watched block whose PUT has not moved + * for the bound is a frozen lookalike (the r972/r973 + * descriptor-pool class, put=7 forever) - drop it so + * the watch set keeps room for the advancing control + * block. Non-kick entries keep the old semantics + * (they are only added by the speculative latch's + * own re-arm today). + */ + if (kick && time_after(jiffies, + ch->gr_ram_userd_watch_at[wi] + + msecs_to_jiffies(SVC_KICK_WATCH_STALE_MS))) { + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + wi++; + continue; + } + if (kick && hp < last) { + /* A reset is churn: drop (same class as the jump). */ + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + if (!kick && hp != last + 1) { + /* + * SESSION 154: a jump or a reset is the churn + * signature (the r916-r919 lookalike pools move by + * +4/-4/+14 or collapse to zero between sightings) - + * drop the entry instead of keeping a dead watch slot + * that would evict newer, better candidates. + */ + ch->gr_ram_userd_watch_n--; + ch->gr_ram_userd_watch[wi] = + ch->gr_ram_userd_watch[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_put[wi] = + ch->gr_ram_userd_watch_put[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_hits[wi] = + ch->gr_ram_userd_watch_hits[ch->gr_ram_userd_watch_n]; + ch->gr_ram_userd_watch_at[wi] = + ch->gr_ram_userd_watch_at[ch->gr_ram_userd_watch_n]; + continue; + } + ch->gr_ram_userd_watch_put[wi] = hp; + ch->gr_ram_userd_watch_at[wi] = jiffies; + /* + * SESSION 153-CONTINUED-20: THE kick-family adoption rule - + * any positive PUT step across passes. The real control block + * is authored by the client's frame loop (measured 140/s), so + * between two sightings its PUT steps by several positions; + * the frozen descriptor-pool entries never step and time out + * above. Do NOT require the +1 step here: for this family a + * free-running put is the LIVE signature, a stall is the + * lookalike. + */ + if (kick) { + (void)svc_read(vgpu, wpa + SVC_USERD_GP_GET, &hg, + sizeof(hg)); + if (hg > hp) + continue; /* impossible; keep watching */ + live_pa = wpa; + live_put = hp; + live_get = hg; + live_prev = last; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family ADOPT on PUT step (put %u -> %u across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_prev, + live_put, live_pa, live_put, live_get); + break; + } + /* + * SESSION 154: ONE observed +1 step adopts. The live block + * advances ~2-8 s per step (measured) while the client's + * channel registration window is ~25-45 s, so waiting for + * a second step can outlive the window; a churning lookalike + * moves in larger jumps (measured +4/-4/+14) and a random + * page cannot produce this exact one-step signature with + * the shape+entry gates already applied. + */ + ch->gr_ram_userd_watch_hits[wi]++; + (void)svc_read(vgpu, wpa + SVC_USERD_GP_GET, &hg, + sizeof(hg)); + if (!hg || hg > hp) { + ch->gr_ram_userd_watch_hits[wi] = 0; + wi++; + continue; + } + live_pa = wpa; + live_put = hp; + live_get = hg; + live_prev = last; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u LIVE (put %u -> %u across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_prev, live_put, + live_pa, live_put, live_get); + break; + } + if (!live_pa) + for (gpa = start; gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 0x90 <= chunk; o += 0x100) { + u32 hp, hg, k; + bool ok = true; + + memcpy(&hg, buf + o + SVC_USERD_GP_GET, 4); + memcpy(&hp, buf + o + SVC_USERD_GP_PUT, 4); + /* + * r914: the control block lives in a near-empty page + * (only the put/get pair and a few engine-owned words + * are written - the measured X-channel FB block and + * the GL client's live sysmem block both leave the + * leading 0x80 bytes zero). The shape gate alone is + * far too weak: a low-RAM page at pa 0x7c000 carrying + * coincidental small numbers at +0x88/+0x8c was + * latched on r913 (the pass then read its put + * forever and never advanced). SESSION 154: the put + * cap is 8 - the GL client's live block cycles 0..3 + * with entries=2 (measured), while the r914 false + * latches read 12/8 and 16/16 and every previously + * measured lookalike counted 16 or more. Also accept + * a free-running put (the old put <= entries bound + * would reject the live block). SESSION 153- + * CONTINUED-20: for the kick family the cap is + * LIFTED to the watch loop's sanity bound (0x100000) + * - its control block is authored by the client's + * frame loop and its PUT free-runs with the frame + * rate; the step-based watch adoption (not the cap) + * is the discriminator there. + */ + /* + * SESSION 153-CONTINUED-38: the cap is also lifted for + * LARGE rings (the kernel channels' 32768-entry + * GPFIFOs): their put is a real free-running count and + * can sit far above 8 while the client waits (measured + * put=16 on beef0100). The remaining gates (near-empty + * page, entry validation against the ring, and the + * pump's confirm/reject) carry the ownership proof. + */ + if (!hp || + hp > (kick || entries > 1024 ? 0x100000 : 8) || + (hg > hp && entries <= 1024)) + continue; + { + u32 z = 0, zi; + + for (zi = 0; zi < 0x80; zi++) + if (!buf[o + zi]) + z++; + if (z < 0x60) + continue; + } + /* + * Never adopt the ring page itself: the ring VA's + * own page holds GPFIFO entries and ring metadata, + * and a latch there would make the pump's GP_GET + * write-back corrupt it. + */ + if (ring_pa && (gpa + o) == + (ring_pa & ~(u64)(SEMA_PAGE_SIZE - 1))) + continue; + /* + * SESSION 153-CONT-7: allocation-locality gate - the + * same 64 MB ring neighbourhood as the watch loop + * above. Excludes the r936-r939 nine-decoy pair + * (~460 MB from the ring) before it can occupy a + * watch slot and burn a reject/re-latch cycle. + * SESSION 153-CONT-19: waived for the kick family + * (h 0xbfef00xx) - its real USERD is measured far + * outside this window, so admission there is by the + * ring-entry match alone (the entry loop below) and + * the pump's +1/+2-step confirmation; the distance + * is logged at the latch. + */ + if (ring_pa && !kick && + (gpa + o < ring_pa - (SVC_GR_RAM_WIN << 5) || + gpa + o > ring_pa + (SVC_GR_RAM_WIN << 5))) + continue; + for (k = 0; k < hp && k < 8; k++) { + u8 e[8]; + u64 eva; + u32 elen, a0, a1; + /* + * r914: entry slot j of a free-running put is + * j % entries, not j. + */ + u32 slot = k % entries; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)slot * SVC_GPFIFO_ENTRY_SIZE, + e, sizeof(e))) { + ok = false; + break; + } + memcpy(&a0, e, 4); + memcpy(&a1, e + 4, 4); + eva = ((u64)(a1 & 0xff) << 32) | + (u64)(a0 & 0xfffffffc); + elen = ((a1 >> 10) & 0x1fffff) << 2; + if (!eva && !elen && + (ch->h_object & 0xfffff000u) == 0xbeef0000u) + continue; + if (!eva || + (!svc_gr_entry_win_ok(ch->gpfifo_va, eva) && + !((ch->h_object & 0xfffff000u) == 0xbeef0000u && + eva >= SEMA_VA_LO && eva < SEMA_VA_HI)) || + elen < 4 || elen > 0x100000) { + ok = false; + break; + } + } + if (!ok) + continue; + /* + * SESSION 154: the live control block lives in a + * NEAR-EMPTY page (measured: put/get plus two more + * bytes non-zero - r914's live block; the X-channel FB + * block and the GL sysmem block both leave the leading + * 0x80 bytes zero). The r916-r919 candidate floods all + * came from dense pages (a dozen+ shape hits per page + * in tables and pools), so a page with more than 32 + * non-zero bytes is rejected before it can occupy a + * watch slot. + */ + { + u32 pstart = o & ~0xfffu; + u32 zc = 0, zi2; + + for (zi2 = 0; zi2 < SEMA_PAGE_SIZE; zi2++) + if (!buf[pstart + zi2]) + zc++; + if (zc < SEMA_PAGE_SIZE - 32) + continue; + } + /* + * SESSION 153-CONT-5: a previously rejected block at + * the same PUT is a frozen lookalike - do not spend a + * (re-armed) scan budget re-latching it, or the budget + * loops on the same stale window forever. A DIFFERENT + * PUT is content change (the block moved): adopt it + * again so a real control block that appears late is + * still picked up. + */ + if ((gpa + o) == ch->gr_ram_userd_reject_pa && + hp == ch->gr_ram_userd_reject_put) + continue; + /* + * SESSION 153-CONTINUED-20 (watch-adopt-on-PUT- + * step): a kick-family candidate is NOT adopted on + * sight. This family's RAM is littered with frozen + * descriptor-pool lookalikes whose +0x88/+0x8c + * words happen to form the put/get shape (r972/973: + * put=7 frozen, 65 pump rejections in one boot, + * the latch alternating between two pool entries). + * Add the candidate to the watch set (deduped, LRU + * replace when full) and KEEP scanning the slice; + * the watch loop at the top adopts the block only + * when its PUT takes a step across passes, and a + * stale-entry timeout (SVC_KICK_WATCH_STALE_MS) + * evicts frozen entries so the pool can never crowd + * out the real control block. The r971-r973 + * immediate latch is retired for this family. + */ + if (kick) { + u32 wn = ch->gr_ram_userd_watch_n; + bool known = false; + u32 wj; + + for (wj = 0; wj < wn; wj++) + if (ch->gr_ram_userd_watch[wj] == + gpa + o) { + known = true; + break; + } + if (!known) { + if (wn >= SVC_GR_USERD_WATCH) { + u32 old = 0, si; + + for (si = 1; si < wn; si++) + if (time_before( + ch->gr_ram_userd_watch_at[si], + ch->gr_ram_userd_watch_at[old])) + old = si; + wn = old; + } + ch->gr_ram_userd_watch[wn] = gpa + o; + ch->gr_ram_userd_watch_put[wn] = hp; + ch->gr_ram_userd_watch_hits[wn] = 0; + ch->gr_ram_userd_watch_at[wn] = jiffies; + if (wn == ch->gr_ram_userd_watch_n) + ch->gr_ram_userd_watch_n++; + if (ch->gr_ram_userd_watch_log < 48) { + ch->gr_ram_userd_watch_log++; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family watch[%u] pa=0x%llx put=%u get=%u (ring_pa=0x%llx delta=%lld MB; ring-entry validated; adopt on PUT step)\n", + ch->h_object, + ch->gr_index, wn, + gpa + o, hp, hg, + ring_pa, + (long long)(gpa + o - ring_pa) / + (1024 * 1024)); + } + } + continue; /* keep scanning; adopt is deferred */ + } + /* + * SESSION 154 (M-GL1.2): a candidate that passed the + * shape gate, the entry validation, the page-density + * gate and the put cap is latched SPECULATIVELY. The + * old code latched on shape alone and a frozen/ + * churning lookalike could hold the walk forever + * (r913: pa 0x7c000; r914: pa 0x9e200 at put=12); + * the strengthened gates reject those classes, and the + * pump now CONFIRMS the latch from its own per-pass + * read: a +1/+2 step confirms the block as the live + * one, a reset/jump or no movement within the bound + * unlatches it so the scan can adopt a better + * candidate (see svc_pump_channels). + */ + live_pa = gpa + o; + live_put = hp; + live_get = hg; + ch->gr_ram_userd_watch_n = 0; + pr_info("sema-chan: GR USERD h=0x%x rank=%u LIVE (put step across passes) -> pa=0x%llx put=%u get=%u\n", + ch->h_object, ch->gr_index, live_pa, + live_put, live_get); + break; + } + if (live_pa) + break; + } + ch->gr_ram_userd_cursor = (end >= top) ? 0 : end; + ch->gr_ram_userd_passes++; + /* + * SESSION 153-CONTINUED-20: kick-family sweep re-arm. When every + * watched candidate has gone stale (frozen pool) and the slice + * budget is spent, one bounded re-arm keeps the sweep alive so the + * real advancing block can still be found; capped like the reject + * counter so a truly dead channel stops sweeping in bounded time. + */ + if (kick && !live_pa && !ch->gr_ram_userd_watch_n && + ch->gr_ram_userd_passes >= svc_gr_ram_pass_budget() && + ch->gr_ram_userd_kick_sweeps < 64) { + ch->gr_ram_userd_kick_sweeps++; + ch->gr_ram_userd_passes = 0; + ch->gr_ram_userd_logged = false; + pr_info("sema-chan: GR USERD h=0x%x rank=%u kick-family sweep re-arm %u/64 (watch empty, budget spent; content scan re-runs)\n", + ch->h_object, ch->gr_index, + ch->gr_ram_userd_kick_sweeps); + } + if (ch->gr_ram_userd_cursor == 0 && !ch->gr_ram_userd_logged && + !ch->gr_ram_userd_watch_n) { + ch->gr_ram_userd_logged = true; + pr_info("sema-chan: GR USERD h=0x%x rank=%u RAM scan found no candidate in %u slice(s) (top 0x%llx)\n", + ch->h_object, ch->gr_index, ch->gr_ram_userd_passes, + top); + } + /* + * Persist the cursor immediately: the stalled-userd pump path + * `continue`s past the end-of-pass write-back (the same reason + * svc_gr_ram_ring_scan persists its cursor). + */ + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + u32 wj; + + svc.chan[i].gr_ram_userd_cursor = + ch->gr_ram_userd_cursor; + svc.chan[i].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[i].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + svc.chan[i].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[i].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[i].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + svc.chan[i].gr_ram_userd_watch_n = + ch->gr_ram_userd_watch_n; + svc.chan[i].gr_ram_userd_watch_log = + ch->gr_ram_userd_watch_log; + svc.chan[i].gr_ram_userd_kick_sweeps = + ch->gr_ram_userd_kick_sweeps; + for (wj = 0; wj < SVC_GR_USERD_WATCH; wj++) { + svc.chan[i].gr_ram_userd_watch[wj] = + ch->gr_ram_userd_watch[wj]; + svc.chan[i].gr_ram_userd_watch_put[wj] = + ch->gr_ram_userd_watch_put[wj]; + svc.chan[i].gr_ram_userd_watch_hits[wj] = + ch->gr_ram_userd_watch_hits[wj]; + svc.chan[i].gr_ram_userd_watch_at[wj] = + ch->gr_ram_userd_watch_at[wj]; + } + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!live_pa) + return false; + *pa_out = live_pa; + *put_out = live_put; + *get_out = live_get; + return true; +} + +/* + * SESSION 154 (M-GL1.2): content anchor for a GR client push whose + * translation went stale. r913: the entry's read resolved through a + * VALIDATED pin that kept answering for a page the guest had since reused + * (`first8=00000001` through src=2) while the trusted walk answered from a + * wrong VASPACE (`pa=0x5c0000 aperture=vidmem plaus=0`) and the client's + * real push bytes sat in RAM on a page no registered directory covered. + * The client's submit pushes carry two stable anchors - the first method + * pair (SET_SUBDEVICE_MASK 0x0001fff0, measured on every GR submit push of + * this driver) and the release-group signature pair measured deep in the + * same pushes - so sweep guest RAM for them (one SVC_GR_RAM_SLICE slice per + * call, cursor-resumed, a few sweeps per entry VA), validate the page by + * re-reading it (prefix at the entry's own page offset + full method-stream + * parse + the signature present in the same page - a stale fragment page + * carrying only the signature fails), then pin it with svc_pin_store so the + * pump's retry reads the REAL bytes like any validated page. + */ +static const u8 svc_gr_anchor_pre[8] = { + 0xf0, 0xff, 0x01, 0x00, 0x00, 0x00, 0x01, 0x20, +}; +static const u8 svc_gr_anchor_sig[16] = { + 0x43, 0x60, 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, + 0x40, 0x60, 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, +}; +#define SVC_GR_ANCHOR_SWEEPS 3 + +/* + * r977 (SESSION 153-CONTINUED-21): the kick family's per-frame submit + * record is a fixed method stream opening with the SET_SUBDEVICE_MASK + * pair 0x0001fff0 0x20010000 (LE bytes f0 ff 01 00 00 00 01 20; the + * bytes r975's anchor pin answered and r973 decoded to n=512 execs). + * Every boot r966-r976 read that VA through some untrusted translation + * and, because a plausible-looking garbage page (r976: first8=9864be60 + * ffffffff at pa=0x10353b000 via the src=3 window guess) passes the + * generic method-stream parse, the entry was consumed with the wrong + * bytes and the kick lane stayed silent for the whole session. For + * this family the canonical prefix IS the content identity: a read + * that lacks it came from a stale/foreign page no matter how + * plausible it parses, and must take the rescue chain (VAS walk -> + * content anchor -> pin). + */ +static bool svc_kick_canonical(const u8 *buf, size_t len) +{ + return len >= sizeof(svc_gr_anchor_pre) && + !memcmp(buf, svc_gr_anchor_pre, sizeof(svc_gr_anchor_pre)); +} + +static bool svc_gr_anchor_scan(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u64 va, u32 len) +{ + static u8 *buf; + u64 top = gsp_sema_gr_scan_top; + u64 start = ch->gr_anchor_cursor; + u64 end, gpa, page_pa = 0; + u32 poff = (u32)(va & (SEMA_PAGE_SIZE - 1)); + int i; + unsigned long flags; + + if (len > SEMA_PAGE_SIZE) + return false; + if (ch->gr_anchor_va != va) { + ch->gr_anchor_va = va; + ch->gr_anchor_sweeps = 0; + ch->gr_anchor_cursor = 0; + start = 0; + } + if (ch->gr_anchor_sweeps >= SVC_GR_ANCHOR_SWEEPS) + return false; + if (top > SVC_GR_SCAN_TOP_MAX) + top = SVC_GR_SCAN_TOP_MAX; + if (top < SEMA_PAGE_SIZE || !len) + return false; + if (start >= top) + start = 0; + end = start + SVC_GR_RAM_SLICE; + if (end > top) + end = top; + if (!buf) { + buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + if (!buf) + return false; + } + if (!ch->gr_anchor_logged) { + ch->gr_anchor_logged = true; + pr_info("sema-anchor: h=0x%x va=0x%llx len=%u content anchor scan started\n", + ch->h_object, va, len); + } + for (gpa = start; !page_pa && gpa + SEMA_PAGE_SIZE <= end; + gpa += SVC_SCAN_CHUNK) { + u64 chunk = min_t(u64, SVC_SCAN_CHUNK, end - gpa); + u32 o; + + if (svc_read(vgpu, gpa, buf, chunk)) + continue; + for (o = 0; o + 24 <= chunk; o += 4) { + /* + * (a) the push's own first method pair - it sits at + * the entry VA's page offset, so only a hit whose + * page offset matches the entry's is a candidate. + */ + if (!memcmp(buf + o, svc_gr_anchor_pre, + sizeof(svc_gr_anchor_pre)) && + ((gpa + o) & (SEMA_PAGE_SIZE - 1)) == poff) { + page_pa = gpa + o - poff; + break; + } + /* + * (b) the release-group signature measured deep in + * the client's submit pushes; the page is validated + * below before it is pinned. + */ + if (!memcmp(buf + o, svc_gr_anchor_sig, + sizeof(svc_gr_anchor_sig))) { + page_pa = (gpa + o) & + ~(u64)(SEMA_PAGE_SIZE - 1); + break; + } + } + } + ch->gr_anchor_cursor = (end >= top) ? 0 : end; + if (ch->gr_anchor_cursor == 0) + ch->gr_anchor_sweeps++; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == ch->h_object) { + svc.chan[i].gr_anchor_cursor = ch->gr_anchor_cursor; + svc.chan[i].gr_anchor_va = ch->gr_anchor_va; + svc.chan[i].gr_anchor_sweeps = ch->gr_anchor_sweeps; + svc.chan[i].gr_anchor_logged = ch->gr_anchor_logged; + break; + } + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (!page_pa) + return false; + /* + * Validate the candidate BEFORE pinning: the push's first method + * pair at the entry's own page offset, a full method-stream parse, + * and the release-group signature present in the same page (the + * remnant fragment at r914's pa 0x84aef000 carries the signature + * but fails the prefix/parse gate). + */ + { + u8 *pb = kmalloc(SEMA_PAGE_SIZE, GFP_KERNEL); + u32 n = min_t(u32, len, SEMA_PAGE_SIZE - poff); + bool ok = false; + + if (pb) { + const u8 *sig = NULL; + + ok = !svc_read(vgpu, page_pa, pb, SEMA_PAGE_SIZE); + if (ok) { + u32 z; + + for (z = 0; z + 16 <= SEMA_PAGE_SIZE; z++) + if (!memcmp(pb + z, svc_gr_anchor_sig, + sizeof(svc_gr_anchor_sig))) { + sig = pb + z; + break; + } + } + ok = ok && sig && + !memcmp(pb + poff, svc_gr_anchor_pre, + sizeof(svc_gr_anchor_pre)) && + svc_push_prefix_ok(pb + poff, n) && + svc_method_stream_plausible(pb + poff, n); + kfree(pb); + } + if (!ok) { + pr_info("sema-anchor: h=0x%x va=0x%llx candidate pa=0x%llx rejected (content gate)\n", + ch->h_object, va, page_pa); + return false; + } + } + svc_pin_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), page_pa, + SVC_APERTURE_SYS); + svc_lat_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), page_pa, + SVC_APERTURE_SYS); + pr_info("sema-anchor: h=0x%x va=0x%llx -> pa=0x%llx (release-group signature anchored; pinned)\n", + ch->h_object, va, page_pa); + return true; +} + +/* + * r705h: host-triggered pdb discovery. The automatic latch + * (svc_vas_candidates_verify + svc_xvas_pdb_discover) only runs from the RAM + * ring scan, which sweeps [0,6GB) before the client's pushes exist and then + * gives up for the boot; and per-client VAS state is flushed at session end. + * This entry point lets a host tool that CONTENT-LOCATED a verified + * {ring VA -> ring PA} pair (the ring page's GPFIFO entries are its + * fingerprint) hand the pair to the service while the client is alive, so: + * 1. every parked COPY_SERVER_PDES candidate is verified against the pair + * (fast: a walk each), and + * 2. if none verifies, the vidmem walk-root sweep anchors on the pair. + * Registered VASes make the client's pushbuffer VAs walkable and the pump + * can then decode the submit's pushes (the fence release among them). + */ +static u64 svc_pdb_disc_va; +static u64 svc_pdb_disc_pa; +static bool svc_pdb_disc_pending; + +/* + * r707r: bounded channel rewind (debug). After the VAS pdb anchor lands, + * entries skipped before it never re-decode on their own (their unread + * window expires into a skip), so the release they carry would be lost for + * that put epoch. A host tool asks for a rewind (sema_chan_rewind: a hex + * h_object, or "all"); the pump re-processes the channel from slot 0 with + * the consumed-slot bitmap and the per-entry rescue state cleared. + */ +static u32 svc_rewind_h; +static bool svc_rewind_all; +static bool svc_rewind_pending; + +int gsp_sema_service_chan_rewind(u32 h_object) +{ + if (!svc.task) + return -ENODEV; + svc_rewind_h = h_object; + svc_rewind_all = (h_object == 0xffffffffu); + svc_rewind_pending = true; + wake_up_process(svc.task); + return 0; +} + +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa) +{ + if (!svc.task) + return -ENODEV; + svc_pdb_disc_va = ring_va & ~0xfffULL; + svc_pdb_disc_pa = ring_pa & ~0xfffULL; + svc_pdb_disc_pending = true; + wake_up_process(svc.task); + return 0; +} + +/* + * R10 (patch B): one candidate walk-root probe. Shared by the vidmem + * and guest-sysmem phases of svc_xvas_pdb_discover: a candidate + * registers only when its walk reproduces the content-verified + * ring_va -> ring_pa pair, so a wrong candidate costs a rejected walk, + * never a wrong registration. + */ +static bool svc_xvas_try_root(struct nv_vgpu_device *vgpu, u64 pdb, + u32 entries, bool vidmem, u64 ring_va, + u64 ring_pa) +{ + struct sema_vaspace_desc d; + u64 pa = 0; + int ap = 0; + + if (svc_gmmu_walk(vgpu, pdb, entries, ring_va & ~0xfffULL, + vidmem ? SVC_APERTURE_FB : SVC_APERTURE_SYS, + &pa, &ap, false) || + (pa & ~0xfffULL) != (ring_pa & ~0xfffULL)) + return false; + + memset(&d, 0, sizeof(d)); + d.h_vaspace = 0x5a5a0000u | (u32)(ring_va >> 20); + d.pdb = pdb; + d.num_entries = entries; + d.b_vidmem = vidmem; + d.verified = true; /* CONT.400: ring VA->PA verified */ + gsp_sema_service_vaspace_add(&d); + pr_info("sema-chan: client VAS page directory discovered pdb=0x%llx entries=%u (%s root; ring VA 0x%llx -> pa 0x%llx verified; the client's other VAs are now walkable)\n", + pdb, entries, vidmem ? "vidmem" : "sysmem", ring_va, ring_pa); + return true; +} + +/* + * r496f: discover a kernel-RM client VAS's page directory by verification. + * The CPU RM builds internal-VAS page tables and never publishes them + * (SET_PAGE_DIRECTORY is an externally owned / UVM-only RPC), so the + * pushbuffer pages and the progress semaphore of an X/GL channel cannot + * be translated from any existing source. Try each 4 KB page as the + * walk root and accept the first whose walk reproduces the ring VA -> + * ring PA pair we already validated by content; registering it with the + * VAS table makes every other VA of the same client resolvable through + * the existing walker. + * + * R10 (patch B): bounded guest-sysmem phase. A sysmem client (the X/GL + * kernel-RM channel whose ring is guest RAM) can keep its internal-VAS + * page tables in guest RAM too, where the vidmem sweep cannot see them + * (r880: ring VA 0x120090000 -> pa 0x10643d000 content-validated, no + * root ever reproduced it). Candidates come only from guest-sysmem + * ranges this module already knows: + * - the ring's allocation window, +/- SVC_GR_RAM_WIN/2 around the + * caller's content-verified ring PA (SVC_GR_RAM_WIN, the span + * svc_gr_ram_ring_scan validated the client's ring/pushes in), and + * - the merged guest-allocation map svc.map[]/svc.map_n (struct + * sema_map_entry - "the allocations the guest has described to us", + * see svc_apply_mapped), capped to SVC_XVAS_SYS_MAP_MAX entries. + * Both are additionally bounded by SEMA_PA_MAX (guest RAM); every + * candidate passes the identical verification (svc_xvas_try_root), so a + * wrong candidate cannot be registered. + */ +#define SVC_XVAS_SYS_MAP_MAX 16384 +static void svc_xvas_pdb_discover(struct nv_vgpu_device *vgpu, u64 ring_va, + u64 ring_pa) +{ + static const u32 ent_tab[] = { 4, 8, 16, 32, 64, 2, 128, 256, 512 }; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + unsigned int ei; + + /* Vidmem shadow sweep (r496f; unchanged behaviour). */ + if (vgpu->bars[1].vaddr && lim) { + for (off = 0; off + 0x1000 <= lim; off += 0x1000) + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], true, + ring_va, ring_pa)) + return; + } + + /* + * R10 (patch B): bounded guest-sysmem phase. The ring allocation + * window first (the strongest guess - the ring scan's own + * convention), then the merged allocation map, skipping pages the + * window phase already covered. + */ + if (ring_pa >= SVC_GR_RAM_WIN / 2 && + ring_pa + SVC_GR_RAM_WIN / 2 <= svc_pa_max()) { + u64 lo = (ring_pa - SVC_GR_RAM_WIN / 2) & ~0xfffULL; + u64 hi = (ring_pa + SVC_GR_RAM_WIN / 2) & ~0xfffULL; + + for (off = lo; off < hi; off += 0x1000) + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], false, + ring_va, ring_pa)) + return; + } + { + u32 nmap = min_t(u32, svc.map_n, SVC_XVAS_SYS_MAP_MAX); + u32 mi; + + for (mi = 0; mi < nmap; mi++) { + off = svc.map[mi].pa_page & ~0xfffULL; + if (!off || off >= svc_pa_max()) + continue; + if (ring_pa >= SVC_GR_RAM_WIN / 2 && + off >= ring_pa - SVC_GR_RAM_WIN / 2 && + off < ring_pa + SVC_GR_RAM_WIN / 2) + continue; /* window phase covered it */ + for (ei = 0; ei < ARRAY_SIZE(ent_tab); ei++) + if (svc_xvas_try_root(vgpu, off, + ent_tab[ei], false, + ring_va, ring_pa)) + return; + } + } + /* + * r1780 (CONT.431): the COPY_SERVER_PDES candidates are the only + * known addresses of the client's page-directory incarnations, and + * the correct incarnation is frequently allocated next to an + * offered one (the driver rebuilds the directory in the same pool). + * Scan a bounded window around every candidate pdb for a root that + * reproduces the verified ring pair and register it under the + * candidate's VAS handle, so the channel's own VAS lookup resolves + * the ring even when the offered pdb itself is stale. The walk is + * the proof; a wrong page cannot register. + */ + { + static const u32 cand_ent[] = { 4, 8, 16, 32 }; + u32 ci, ei2; + + for (ci = 0; ci < SVC_VAS_CAND_MAX; ci++) { + u64 base = svc_vas_cand[ci].pdb; + u64 lo, hi, poff; + u32 hvas = svc_vas_cand[ci].h_vaspace; + + if (!base) + continue; + lo = (base > 0x400000ULL) ? base - 0x400000ULL + : 0x1000ULL; + hi = base + 0x400000ULL; + if (hi > svc_pa_max()) + hi = svc_pa_max(); + for (poff = lo & ~0xfffULL; poff < hi; + poff += 0x1000ULL) { + for (ei2 = 0; ei2 < ARRAY_SIZE(cand_ent); + ei2++) { + if (!svc_xvas_try_root(vgpu, poff, + cand_ent[ei2], + false, ring_va, + ring_pa)) + continue; + if (hvas) { + struct sema_vaspace_desc d; + + memset(&d, 0, sizeof(d)); + d.h_vaspace = hvas; + d.pdb = poff; + d.num_entries = cand_ent[ei2]; + d.b_vidmem = false; + d.verified = true; + gsp_sema_service_vaspace_add(&d); + pr_info("sema-svc: VAS 0x%x registered from candidate-neighbourhood pdb=0x%llx entries=%u (ring 0x%llx -> 0x%llx verified)\n", + hvas, poff, + cand_ent[ei2], + ring_va, ring_pa); + } + return; + } + } + } + } +} + +/* + * X-DMA-FIX (2026-11-10): register the channel's own parked + * COPY_SERVER_PDES candidate as soon as a pump pass can content-validate it + * against this channel's GPFIFO, instead of waiting for the age-out. The + * existing verifier is anchored to the current ring_pa; when the lookup + * answered a stale/foreign page that pair is wrong and the candidate never + * matches until the 1000 ms age-out registers it UNVERIFIED (r1585: first + * USERD sweep at +15 ms, age-out at +1.36 s, X's DMA test window 10 s). + * Anchor on the channel's own gpfifo_va and its entry 0 content instead. + */ +static unsigned long svc_promote_diag_at; +/* + * r1772 (IDLE-DMA-RCA F3): budget for the "RING-VAS ... mapped by VAS" + * success print in the probe loop below. During the r1771 c18 stall it + * repeated unbudgeted at ~12 Hz (1,034 lines) on the same service thread + * that must author the kick/idle lane. + */ +static unsigned long svc_ringvas_diag_at; +/* + * r1773 (IDLE-DMA-RCA F3 extension): the parked-candidate walk probe + * (`VAS-PROMOTE ... cand ... walk rc=`) was UNBUDGETED for the beef family. + * During the GZDoom V_Init stall the direct-GL client channel's ring VA + * (low arena, e.g. gpfifo 0x42dc000) failed the candidate walk against four + * parked directories every ~40 ms - 100+ pr_info/s plus the walks on the + * single service thread that must also author the kick/idle lane. Budget + * the print to 2 s and back the whole probe off to 250 ms after a failed + * sweep (reset on any promotion). sema_vas_promote_backoff=0 restores the + * pre-fix behaviour for A/B. + */ +static bool gsp_sema_vas_promote_backoff = true; +module_param_named(sema_vas_promote_backoff, gsp_sema_vas_promote_backoff, + bool, 0644); +MODULE_PARM_DESC(sema_vas_promote_backoff, + "Back off the beef-family VAS ring-promote walk after a failed sweep (default 1)"); +static unsigned long svc_vaspromote_diag_at; +static unsigned long svc_vaspromote_backoff_at; +/* + * r1774 (V9): the failed-sweep backoff above was a SINGLE shared stamp, so + * whichever beef-family channel swept first re-armed it every pass and + * starved its siblings (r1773 c31: 0xbeef0101 swept 92 times, 0xbeef0100 + * only 8 - the GR channel whose ring the GLX lane needs). Remember the + * channel that failed and gate only that one; a different beef channel + * always gets its own sweep. + */ +static u32 svc_vaspromote_backoff_h; +/* + * r1774 (V9): when 0, restore the pre-V9 rule where a beef-family channel + * whose VAS is registered VERIFIED skips the ring walk (A/B knob). When 1 + * (default) the direct-GL family always re-walks the ring VA: the client + * rebuilds its page directory every few seconds and the mapping for the + * low-arena ring appears IN the directory only after the client populates + * it (r1773 inc A: the same pdb answered zeros at +3.3 s and the live + * ring pair at +6.6 s), so a verified-but-stale directory must not end the + * probe for this family. + */ +static bool gsp_sema_vas_client_ring = true; +module_param_named(sema_vas_client_ring, gsp_sema_vas_client_ring, bool, 0644); +MODULE_PARM_DESC(sema_vas_client_ring, + "Always re-probe the direct-GL client's low-arena ring VA (default 1; 0 = legacy verified-skip)"); +/* + * r1774 (V9): per-channel rate limit for the direct-GL always-probe so the + * re-walk (up to 16 parked candidates + every registered VAS) runs at most + * ~10/s per channel instead of every service pass; 100 ms still catches the + * client's late ring mapping well inside the DDX's 5 s idle window, and a + * successful probe re-pins the ring so the pump's svc_lookup is served from + * the pin until the next probe. + */ +#define SVC_VASPROMOTE_RL 8 +static struct { + u32 h; + unsigned long at; +} svc_vaspromote_rl[SVC_VASPROMOTE_RL]; + +static bool svc_vaspromote_rl_allow(u32 h) +{ + unsigned long now = jiffies; + int i, free_i = -1; + + for (i = 0; i < SVC_VASPROMOTE_RL; i++) { + if (svc_vaspromote_rl[i].h == h) { + if (time_before(now, svc_vaspromote_rl[i].at)) + return false; + svc_vaspromote_rl[i].at = now + + msecs_to_jiffies(100); + return true; + } + if (!svc_vaspromote_rl[i].h && free_i < 0) + free_i = i; + } + if (free_i >= 0) { + svc_vaspromote_rl[free_i].h = h; + svc_vaspromote_rl[free_i].at = now + msecs_to_jiffies(100); + } + return true; +} + +static void svc_chan_vas_candidate_promote(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u32 h = ch->h_vaspace; + bool beef = ((ch->h_object & 0xfffff000u) == 0xbeef0000u); + int i, pass; + + /* + * X-DMA-FIX: the GR client channel declares hVASpace==0 (attach to + * the TSG); its real VAS is published by its CE siblings. Without + * this fallback the helper returned early for exactly the channel + * whose ring we need to resolve. + */ + if (!h) + h = svc_channel_owner_vas(vgpu, ch->h_object); + /* + * CONT.400: gate on VERIFIED, not merely known. The bounded age-out + * registers the parked directory UNVERIFIED (so the client can move) + * but the translation is frequently wrong; we must keep trying to + * upgrade it by content until a walk proves it. Only a verified + * directory lets us skip the walk. + */ + if (!h || !ch->gpfifo_va || ch->gpfifo_entries < 2) { + /* CONT.400 diag: name the exact early-return reason. */ + if (beef && + time_after(jiffies, svc_promote_diag_at)) { + svc_promote_diag_at = jiffies + msecs_to_jiffies(2000); + pr_info("sema-svc: VAS-PROMOTE h=0x%x skip (h=0x%x gpfifo=0x%llx entries=%u)\n", + ch->h_object, h, + ch->gpfifo_va, ch->gpfifo_entries); + } + return; + } + if (svc_vaspace_verified(h)) { + /* + * CONT.401: see svc_vaspace_stale_candidate(). Only a verified + * directory with no contradicting parked candidate lets us + * skip the content walk; a rebuilt one is demoted first. + * + * r1774 (V9): the direct-GL family is exempted (see the + * sema_vas_client_ring comment) - its directory churns and + * the ring mapping lands late, so "verified" is not a + * sufficient proof that the ring VA is resolvable NOW. + */ + if (!svc_vaspace_stale_candidate(h) && + !(beef && gsp_sema_vas_client_ring)) { + if (beef && + time_after(jiffies, svc_promote_diag_at)) { + svc_promote_diag_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: VAS-PROMOTE h=0x%x skip (h=0x%x verified=1 gpfifo=0x%llx entries=%u)\n", + ch->h_object, h, + ch->gpfifo_va, ch->gpfifo_entries); + } + return; + } + /* + * r1776 (V11): with the always-probe the direct-GL family + * re-validates its ring content every sweep, so demoting the + * stored directory on a churning parked candidate is redundant + * - and now harmful: keeping >=2 parks alive means a + * different-pdb candidate would re-trigger demote (and the + * svc.lat reset) on every sweep. Other clients keep CONT.401. + */ + if (svc_vaspace_stale_candidate(h) && + !(beef && gsp_sema_vas_client_ring)) { + pr_info("sema-svc: VAS 0x%x verified directory is stale (rebuilt pdb parked); demoting for re-validation\n", + h); + svc_vaspace_demote(h); + } + /* + * r1780 (V35): beef-family dead-directory demote. The guest + * frees/rebuilds the client VAS every few seconds; a verified + * entry whose page now reads all-zero cannot map anything and + * must yield to the freshest parked candidate. Cooldown 2 s + * keeps this from chasing its own tail. + */ + if (beef && gsp_sema_vas_client_ring) { + static unsigned long beef_demote_at; + u64 rpdb = 0; + int rap = 0; + + if (svc_vas_registered_pdb(h, &rpdb, &rap) && + svc_pdb_reads_zero(vgpu, rpdb, rap) && + time_after(jiffies, beef_demote_at)) { + beef_demote_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: VAS 0x%x verified directory pdb=0x%llx reads all-zero (guest rebuilt); demoting for re-validation\n", + h, rpdb); + svc_vaspace_demote(h); + } + } + } + /* + * r1773: bound the failed-sweep storm for the beef family (see the + * backoff_at comment above). A promotion (below) clears the window. + * r1774: per-channel (see svc_vaspromote_backoff_h). + */ + if (gsp_sema_vas_promote_backoff && beef && + ch->h_object == svc_vaspromote_backoff_h && + time_before(jiffies, svc_vaspromote_backoff_at)) + return; + /* + * r1774 (V9): rate-limit the direct-GL always-probe (see the + * svc_vaspromote_rl comment). Non-beef channels keep the legacy + * per-pass probe cadence. + */ + if (beef && gsp_sema_vas_client_ring && + !svc_vaspromote_rl_allow(ch->h_object)) + return; + /* + * r1777 (V12): directory-incarnation handling. The client parks ~19 + * pdbs per churn cycle; the freshest incarnation is the one most + * likely to map the ring VA NOW, so walk it first (pass 0) and the + * rest afterwards (pass 1). A parked pdb whose L0 reads as a freed/ + * reused page (kernel-pointer pattern or all-zero) is skipped before + * the walk - r1776: walking such a directory fabricated the rc=-6 + * "mapped but all-zero" ring page. + */ + for (pass = 0; pass < 2; pass++) { + int newest = -1; + + if (pass == 0) { + unsigned long best = 0; + int j; + + for (j = 0; j < SVC_VAS_CAND_MAX; j++) { + if (!svc_vas_cand[j].valid || + svc_vas_cand[j].h_vaspace != h) + continue; + if (newest < 0 || + time_after(svc_vas_cand[j].stamp, best)) { + best = svc_vas_cand[j].stamp; + newest = j; + } + } + if (newest < 0) + break; + } + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0, len; + u64 eva; + struct sema_vaspace_desc d; + int wrc; + + if (!svc_vas_cand[i].valid || svc_vas_cand[i].h_vaspace != h) + continue; + if (pass == 0 && i != newest) + continue; + if (pass == 1 && i == newest) + continue; + if (gsp_sema_ce_ram && + !svc_pdb_l0_sane(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS)) { + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u && + time_after(jiffies, svc_vaspromote_diag_at)) { + svc_vaspromote_diag_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: VAS-PROMOTE h=0x%x cand pdb=0x%llx L0 sanity failed (freed/reused directory)\n", + ch->h_object, svc_vas_cand[i].pdb); + } + continue; + } + wrc = svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, + ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1), + svc_vas_cand[i].vidmem ? SVC_APERTURE_FB + : SVC_APERTURE_SYS, + &wpa, &wap, false); + if (wrc) { + /* CONT.400 diag: why the beef ring walk misses. + * r1773: budgeted (2 s) - unbudgeted this printed + * 100+/s during the GZDoom V_Init stall. */ + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u && + time_after(jiffies, svc_vaspromote_diag_at)) { + svc_vaspromote_diag_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: VAS-PROMOTE h=0x%x cand pdb=0x%llx entries=%u walk rc=%d (gpfifo 0x%llx)\n", + ch->h_object, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, wrc, + ch->gpfifo_va); + } + continue; + } + if (svc_read_ap(vgpu, wpa, wap, e, sizeof(e))) + continue; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!eva || len < 4 || len > 0x10000 || + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) { + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u) + pr_info("sema-svc: VAS-PROMOTE h=0x%x cand pdb=0x%llx -> wpa=0x%llx entry0=%08x %08x eva=0x%llx len=%u REJECT\n", + ch->h_object, svc_vas_cand[i].pdb, + wpa, e0, e1, eva, len); + continue; + } + memset(&d, 0, sizeof(d)); + d.h_vaspace = h; + d.pdb = svc_vas_cand[i].pdb; + d.num_entries = svc_vas_cand[i].entries; + d.b_vidmem = svc_vas_cand[i].vidmem; + d.verified = true; /* CONT.400: content-proven ring walk */ + /* + * r1776 (V11): keep >=2 fresh same-VAS parks. r1775 CE wall: + * the GR channel content-validated and CONSUMED the only fresh + * directory ~0.1 s before the CE ring page content landed; the + * CE siblings were then left with the stale parked pdb and + * spun "cand ... walk rc=-2" until teardown (submissions=0). + * Keep this candidate parked while fewer than two other + * same-VAS candidates remain so a sibling can re-validate + * through the same directory later. Re-register only when the + * stored directory differs (avoid clearing svc.lat/log spam). + */ + if (beef) { + unsigned nsame = 0; + bool reg_log = false; + int j; + + for (j = 0; j < SVC_VAS_CAND_MAX; j++) + if (j != i && svc_vas_cand[j].valid && + svc_vas_cand[j].h_vaspace == h) + nsame++; + if (nsame < 2) { + /* + * Keep this candidate parked so a sibling can + * re-validate through the same directory after + * this channel moves on. Re-register only when + * the stored (verified) directory is not + * already this pdb. + */ + if (!svc_vaspace_pdb_match(h, + svc_vas_cand[i].pdb)) { + gsp_sema_service_vaspace_add(&d); + reg_log = true; + } else + svc_vas_cand[i].stamp = jiffies; + } else { + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + reg_log = true; + } + svc_vaspromote_backoff_at = 0; + svc_vaspromote_backoff_h = 0; + if (reg_log) + pr_info("sema-svc: VAS 0x%x registered at channel 0x%x turn (parked candidate content-validated%s: gpfifo 0x%llx entry0 va=0x%llx len=%u)\n", + h, ch->h_object, + nsame < 2 ? ", kept parked" : "", + ch->gpfifo_va, eva, len); + return; + } + gsp_sema_service_vaspace_add(&d); + svc_vas_cand[i].valid = false; + svc_vaspromote_backoff_at = 0; /* r1773: promotion clears the backoff */ + svc_vaspromote_backoff_h = 0; /* r1774 (V9) */ + pr_info("sema-svc: VAS 0x%x registered at channel 0x%x turn (parked candidate content-validated: gpfifo 0x%llx entry0 va=0x%llx len=%u)\n", + h, ch->h_object, ch->gpfifo_va, eva, len); + return; + } + } /* end two-pass (newest-first) candidate walk */ + /* + * r1773: no parked candidate mapped the ring this sweep; back the + * beef-family probe off for 250 ms so a persistently-unresolvable + * client ring cannot re-walk + re-log every service pass (the V_Init + * stall). r1774 (V9): only THIS channel backs off - the shared stamp + * let one failing sibling starve the GR channel's probe. + */ + if (gsp_sema_vas_promote_backoff && beef) { + svc_vaspromote_backoff_h = ch->h_object; + svc_vaspromote_backoff_at = jiffies + msecs_to_jiffies(250); + } + + /* + * r1776 (V11): CE SIBLING SWEEP - when a beef-family channel (the + * direct-GL CE rings 0xbeef0101/0102, and the GR sibling as a + * fallback) misses its ring through every same-VAS parked candidate, + * walk ALL parked candidates regardless of owner VAS. The client + * rebuilds its page directory every few seconds and parks ~19 pdbs + * per cycle under churning handles, so the directory that maps the + * CE ring VA can sit in a slot whose h_vaspace does not match the + * channel's (r1775: CE 0x4100000 never mapped; GR 0x4020000 mapped + * only through a later fresh candidate). The walk stays content- + * gated (entry-0 push plausibility + window bound), so a foreign + * directory can only win by actually mapping THIS channel's ring. + * The candidate is NOT consumed: its owner still needs it, and the + * CE siblings re-walk it until their ring page content lands. + */ + if (beef) { + for (i = 0; i < SVC_VAS_CAND_MAX; i++) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0, len; + u64 eva; + int wrc; + + if (!svc_vas_cand[i].valid || + svc_vas_cand[i].h_vaspace == h) + continue; + wrc = svc_gmmu_walk(vgpu, svc_vas_cand[i].pdb, + svc_vas_cand[i].entries, + ch->gpfifo_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + svc_vas_cand[i].vidmem + ? SVC_APERTURE_FB + : SVC_APERTURE_SYS, + &wpa, &wap, false); + if (wrc) + continue; + if (svc_read_ap(vgpu, wpa, wap, e, sizeof(e))) + continue; + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!eva || len < 4 || len > 0x10000 || + !svc_gr_entry_win_ok(ch->gpfifo_va, eva)) + continue; + svc_pin_store(ch->gpfifo_va, wpa, wap); + svc_lat_store(ch->gpfifo_va, wpa, wap); + svc_cur_vas = svc_vas_cand[i].h_vaspace; + svc_vaspromote_backoff_at = 0; /* r1773 */ + svc_vaspromote_backoff_h = 0; /* r1774 (V9) */ + if (time_after(jiffies, svc_ringvas_diag_at)) { + svc_ringvas_diag_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: RING-CAND h=0x%x gpfifo=0x%llx mapped by parked VAS 0x%x cand (pdb=0x%llx ap=%d) -> pa=0x%llx entry0 va=0x%llx len=%u\n", + ch->h_object, ch->gpfifo_va, + svc_vas_cand[i].h_vaspace, + svc_vas_cand[i].pdb, wap, wpa, + eva, len); + } + return; + } + } + + /* + * CONT.400: the parked candidate is not always the directory that maps + * the ring (the GL client's GR ring 0x4020000 lives in the RM low + * aperture; its client VAS 0xbeef0300 maps the pushbuffers but not + * necessarily the ring). Probe EVERY registered directory for the ring + * VA and adopt the first that yields a plausible entry-0 push. This is + * the missing link that kept the beef GR channel ring_pa=0 forever, so + * the whole GL draw stream was never decoded (black screen + 0.2 FPS). + */ + { + unsigned long flags; + int k; + /* + * r1776 (V11): keep the LAST per-directory failure reason for + * the miss line below. r1775 saw only "NOT mapped" - the CE + * ring walk through the fresh directory actually returned rc=0 + * with an all-zero entry page during the early window, which + * the bare line hid. rc: walk error, -5 read error, -6 mapped + * but entry implausible (e0/e1 recorded). + */ + int last_rc = 0; + u32 last_vas = 0; + u64 last_wpa = 0; + u32 last_e0 = 0, last_e1 = 0; + + spin_lock_irqsave(&svc.vas_lock, flags); + for (k = 0; k < SVC_MAX_VASPACES; k++) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + u8 e[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0 = 0, e1 = 0, elen; + u64 eva; + int wrc; + struct svc_vaspace snap; + + if (!svc.vas[k].valid) + continue; + snap = svc.vas[k]; + spin_unlock_irqrestore(&svc.vas_lock, flags); + /* + * r1777 (V12): a registered directory can still be a + * freed/reused page (the UNVERIFIED age-out + * registrations). Skip walking an L0 that reads as + * a kernel pointer or all-zero: the r1776 rc=-6 + * "mapped but all-zero" ring page came through + * exactly such an incarnation. + */ + if (gsp_sema_ce_ram && + !svc_pdb_l0_sane(vgpu, snap.pdb, snap.ap)) { + last_rc = -7; + last_vas = snap.h_vaspace; + last_wpa = 0; + last_e0 = last_e1 = 0; + spin_lock_irqsave(&svc.vas_lock, flags); + continue; + } + wrc = svc_gmmu_walk(vgpu, snap.pdb, snap.entries, + ch->gpfifo_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + snap.ap, &wpa, &wap, false); + if (wrc) { + last_rc = wrc; + last_vas = snap.h_vaspace; + last_wpa = 0; + last_e0 = last_e1 = 0; + } else if (svc_read_ap(vgpu, wpa, wap, e, sizeof(e))) { + last_rc = -5; + last_vas = snap.h_vaspace; + last_wpa = wpa; + last_e0 = last_e1 = 0; + } else { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + elen = ((e1 >> 10) & 0x1fffff) << 2; + if (eva && elen >= 4 && elen <= 0x10000 && + svc_gr_entry_win_ok(ch->gpfifo_va, eva)) { + if (time_after(jiffies, + svc_ringvas_diag_at)) { + svc_ringvas_diag_at = jiffies + + msecs_to_jiffies(2000); + pr_info("sema-svc: RING-VAS h=0x%x gpfifo=0x%llx mapped by VAS 0x%x (pdb=0x%llx ap=%d) -> pa=0x%llx entry0 va=0x%llx len=%u\n", + ch->h_object, + ch->gpfifo_va, + snap.h_vaspace, + snap.pdb, + snap.ap, wpa, eva, + elen); + } + /* Pin the ring through this directory. */ + svc_pin_store(ch->gpfifo_va, wpa, wap); + svc_lat_store(ch->gpfifo_va, wpa, wap); + svc_cur_vas = snap.h_vaspace; + svc_vaspromote_backoff_at = 0; /* r1773 */ + svc_vaspromote_backoff_h = 0; /* r1774 (V9) */ + return; + } + last_rc = -6; + last_vas = snap.h_vaspace; + last_wpa = wpa; + last_e0 = e0; + last_e1 = e1; + } + spin_lock_irqsave(&svc.vas_lock, flags); + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u && + time_after(jiffies, svc_promote_diag_at)) { + svc_promote_diag_at = jiffies + msecs_to_jiffies(2000); + pr_info("sema-svc: RING-VAS h=0x%x gpfifo=0x%llx NOT mapped by any registered VAS (last vas=0x%x walk rc=%d pa=0x%llx e0=%08x e1=%08x)\n", + ch->h_object, ch->gpfifo_va, last_vas, + last_rc, last_wpa, last_e0, last_e1); + } + } +} + static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { @@ -13796,6 +32253,42 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, return; /* wait until the CE channels are known */ /* + * SESSION 148 perf (r858/r859 evidence): a GR channel's ring page + * sits at (notifier_pa page + 0x400). Verified for the RC watchdog + * across r848/r853/r858; extended to all GR channels after r859 + * showed the X channel (0x1000013) NEVER resolving its ring (no + * auto-pin all boot, entries decoded from garbage: va=0x41c000/ + * 0x2017a20/0x10104 with lens 20464..49152, unreadable, skipped; + * ~11 s per frame). The candidate probe (entry0 in the channel's + * own alloc + USERD shape) content-gates every derivation, so a + * wrong guess costs a failed probe - exactly the current behaviour. + */ + if (!ch->engine_type && ch->notifier_pa) { + u64 rpa = (ch->notifier_pa & ~(u64)(SEMA_PAGE_SIZE - 1)) + + 0x400; + + if (svc_gr_ring_candidate(vgpu, ch, rpa, SVC_APERTURE_SYS, + &va, &len, &put, &get)) { + svc_pin_store(ch->gpfifo_va, rpa, SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, rpa, SVC_APERTURE_SYS); + if (ch->userd_reg) + ch->userd_ok = true; + else { + ch->userd_pa = rpa + SVC_GR_USERD_OFF; + ch->userd_as = SVC_APERTURE_SYS; + ch->userd_ok = true; + } + svc_gr_ring_base = rpa; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=1 (notifier-derived; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + rpa, va, len, put, get); + return; + } + } + + /* * Guest truth first: the channel registration carries the USERD * memdesc base for THIS session's allocation, and the ring sits one * USERD block before it (observed for every GR channel across @@ -13850,11 +32343,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, ch->gpfifo_entries, &put, &get)) continue; + /* + * SESSION 132: reject a dead candidate when + * the client's own page tables provably + * resolve the ring VA to a live page (the + * r759 vidmem false positive). + */ + if (svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va)) + continue; svc_pin_store(ch->gpfifo_va, cand, cap); svc_lat_store(ch->gpfifo_va, cand, cap); ch->userd_as = cap; ch->userd_ok = true; svc_gr_ring_base = cand; + if (cap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, cand, + ch->userd_pa); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (userd-derived d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -13885,13 +32390,17 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, len >= 4 && len <= 0x10000 && !svc_userd_probe(vgpu, pa + SVC_GR_USERD_OFF, SVC_APERTURE_FB, ch->gpfifo_entries, - &put, &get)) { + &put, &get) && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); ch->userd_pa = pa + SVC_GR_USERD_OFF; ch->userd_as = SVC_APERTURE_FB; ch->userd_ok = true; svc_gr_ring_base = base; + svc_fb_claim_add(ch->h_object, pa, + pa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -13923,7 +32432,9 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, (u64)(e0 & 0xfffffffc); len = ((e1 >> 10) & 0x1fffff) << 2; if (!(va & 0xfffff) && va >= SEMA_VA_LO && - va < SEMA_VA_HI && len >= 4 && len <= 0x10000) { + va < SEMA_VA_HI && len >= 4 && len <= 0x10000 && + !(svc_ring_entry_dead(vgpu, e0, e1) && + svc_ring_walk_live(vgpu, ch->gpfifo_va))) { /* * Best effort only: a wrapped GP_PUT * legitimately exceeds the entry count, so a @@ -13938,10 +32449,22 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get); svc_pin_store(ch->gpfifo_va, pin_pa, pin_ap); svc_lat_store(ch->gpfifo_va, pin_pa, pin_ap); - ch->userd_pa = pin_pa + SVC_GR_USERD_OFF; - ch->userd_as = svc_ap_of(pin_ap); - ch->userd_ok = true; + /* r831: keep a registered real-PA userd. */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else if (ch->gpfifo_entries <= 1024) { + ch->userd_pa = + pin_pa + SVC_GR_USERD_OFF; + ch->userd_as = svc_ap_of(pin_ap); + ch->userd_ok = true; + } + /* V38: large ring needs the generic resolver + * (declared USERD or RAM scan), not slot data. */ svc_gr_ring_base = base; + if (svc_ap_of(pin_ap) == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, pin_pa, + pin_pa + + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -13973,10 +32496,23 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, &get)) { svc_pin_store(ch->gpfifo_va, wpa, wap); svc_lat_store(ch->gpfifo_va, wpa, wap); - ch->userd_pa = wpa + SVC_GR_USERD_OFF; - ch->userd_as = wap; - ch->userd_ok = true; + /* + * r831: the ring+0x2000 userd convention is a guess + * for the CUDA GR layout; a channel whose REGISTERED + * userd is a real PA (the RC watchdog) keeps its own + * (the candidate gate above probes it as fallback). + */ + if (ch->userd_reg) { + ch->userd_ok = true; + } else { + ch->userd_pa = wpa + SVC_GR_USERD_OFF; + ch->userd_as = wap; + ch->userd_ok = true; + } svc_gr_ring_base = wpa; + if (wap == SVC_APERTURE_FB) + svc_fb_claim_add(ch->h_object, wpa, + wpa + SVC_GR_USERD_OFF); set_bit(ch->gr_index, &svc_gr_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> pa=0x%llx ap=%d (trusted walk; entry0 va=0x%llx len=%u put=%u get=%u)\n", @@ -13986,6 +32522,228 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, } } + /* + * SESSION 133: bounded shadow ring scan fallback. + * + * The CE-class rings live in VIDMEM and the FB convention only + * matches when the guest's VRAM allocator happens to place them at + * the conventional offsets (r763: matched, fb 0x80000/0xa0000, the + * channel decoded; r765: read zeros, the ring VA resolved to a + * stale RAM page whose entry0 was dead, the CE channel never + * latched and its inline SASS copies never decoded - black + * frames). When every deterministic candidate has failed, scan the + * shadow for a ring-shaped page: entry0 must name a 1 MB aligned + * push VA in the client window with a sane length, must not be + * provably dead (svc_ring_entry_dead - the r759 false-positive + * guard), and the USERD at the observed deltas (+0x10000 for the + * CE family, +0x2000 for GR) must read a coherent put/get. One + * bounded scan per rank per boot; a false candidate still costs + * only a failed probe. + */ + if (!test_bit(ch->gr_index, &svc_gr_ring_scan_done)) { + u8 *sbase = vgpu->bars[1].vaddr; + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x4000000ULL); + u64 off; + + /* + * SESSION 133b: the rings are NOT page-aligned (r767: the + * USERD shape scan found the channel blocks at 0xfee00/ + * 0xff000/0xff200, so the rings sit at unaligned offsets like + * 0xefe00); step by 0x100 and RETRY (rate-limited) - a first + * pass can run before the client has written the ring or the + * USERD, and a consumed once-bit would never rescan. + */ + if (time_before(jiffies, + (unsigned long)svc_gr_ring_scan_last[ch->gr_index] + + msecs_to_jiffies(1500))) + goto skip_shadow_scan; + svc_gr_ring_scan_last[ch->gr_index] = jiffies; + if (sbase && lim > 0x1000) { + static const u64 udeltas[] = { 0x10000, SVC_GR_USERD_OFF, + 0x1000 }; + + /* + * SESSION 133c pass 1: find the USERD by its OWN + * shape (put@+0x8c, get@+0x88 - the find_userd + * layout) and derive the ring from it. The USERD + * words appear at channel creation and stay; the + * ring entry may be written later, so this pass is + * the stabler trigger. Each hit is validated by the + * derived ring's entry0 (shape + not-dead), so a + * coincidental put/get pair costs a failed check. + */ + for (off = 0; off + 0x90 <= lim; off += 0x100) { + u32 up, ug; + unsigned int ui; + + memcpy(&up, sbase + off + 0x8c, 4); + if (up < 1 || up > 65535) + continue; + memcpy(&ug, sbase + off + 0x88, 4); + if (ug > up) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 roff; + u32 h0, h1, p3 = 0, g3 = 0; + u64 hva; + u32 hlen; + + if (off < udeltas[ui]) + continue; + roff = off - udeltas[ui]; + memcpy(&h0, sbase + roff, 4); + memcpy(&h1, sbase + roff + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + /* + * r1780 (V33): the direct-GL + * client's ring lives in the + * LOW aperture (push VAs + * 0x4000000-class, e.g. + * 0x4000100/0x4000000) and + * its entries are not 1 MB + * aligned. The old high-family + * gate could never latch it, so + * the shadow scan never found + * the ring while the client's + * USERD was live (put=150 on + * r1780 c32). Accept the low + * family through the same + * entry-window test the RAM scan + * uses. + */ + if (!hva || hlen < 4 || + hlen > 0x100000) + continue; + if (!((hva >= SEMA_VA_LO && + hva < SEMA_VA_HI && + !(hva & 0xfffff)) || + svc_gr_entry_win_ok(ch->gpfifo_va, + hva))) + continue; + /* + * r1780 (V39): the direct-GL ring's + * pushbuffer shares its low-aperture + * allocation window with the ring + * (measured: ring 0x4020000, pushes + * 0x4023000/0x4024004). A page-table + * page full of PTE-shaped dwords passed + * the broad entry-window test and was + * latched as the ring on c42 + * (entry0 0xe107a20 len=49156). Require + * entry0 within 64 MiB of the ring VA; + * the real pushes are within 16 KiB. + */ + if (hva + 0x4000000ULL < ch->gpfifo_va || + hva > ch->gpfifo_va + 0x4000000ULL) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + /* + * SESSION 144: never latch a foreign + * block. Without this every CUDA GR + * rank stole the CE rank-0 block while + * its own ring was unwritten + * (r805-r808); staying unlatched here + * lets the FB-convention path latch the + * real ring once the client writes it. + */ + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + roff, off)) + continue; + if (svc_userd_probe(vgpu, off, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, roff, + SVC_APERTURE_FB); + ch->userd_pa = off; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = roff; + svc_fb_claim_add(ch->h_object, roff, off); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (userd-shape scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, roff, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + /* pass 2: ring-entry-shaped pages (below) */ + for (off = 0; off + 8 <= lim; off += 0x100) { + u32 h0, h1; + u64 hva; + u32 hlen; + unsigned int ui; + + memcpy(&h0, sbase + off, 4); + memcpy(&h1, sbase + off + 4, 4); + hva = ((u64)(h1 & 0xff) << 32) | + (u64)(h0 & 0xfffffffc); + hlen = ((h1 >> 10) & 0x1fffff) << 2; + /* r1780 (V33): accept the low-aperture + * direct-GL family too (see pass 1). */ + if (!hva || hlen < 4 || hlen > 0x100000) + continue; + if (!((hva >= SEMA_VA_LO && + hva < SEMA_VA_HI && + !(hva & 0xfffff)) || + svc_gr_entry_win_ok(ch->gpfifo_va, hva))) + continue; + if (svc_ring_entry_dead(vgpu, h0, h1)) + continue; + for (ui = 0; ui < ARRAY_SIZE(udeltas); ui++) { + u64 upa = off + udeltas[ui]; + u32 p3 = 0, g3 = 0; + + if (upa + 4 > lim) + continue; + if (svc_fb_claim_foreign(vgpu, + ch->h_object, + off, upa)) + continue; + if (svc_userd_probe(vgpu, upa, + SVC_APERTURE_FB, + ch->gpfifo_entries, + &p3, &g3)) + continue; + svc_pin_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + svc_lat_store(ch->gpfifo_va, off, + SVC_APERTURE_FB); + ch->userd_pa = upa; + ch->userd_as = SVC_APERTURE_FB; + ch->userd_ok = true; + svc_gr_ring_base = off; + svc_fb_claim_add(ch->h_object, off, + upa); + set_bit(ch->gr_index, + &svc_gr_ring_done); + set_bit(ch->gr_index, + &svc_gr_ring_scan_done); + svc_ring_n++; + pr_info("sema-chan: auto-pinned GR ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (shadow scan d=0x%llx; entry0 va=0x%llx len=%u put=%u get=%u)\n", + ch->h_object, ch->gr_index, + ch->gpfifo_va, off, + udeltas[ui], hva, hlen, p3, + g3); + return; + } + } + } + } + if (!test_and_set_bit(ch->gr_index, &svc_gr_ring_tried)) { u8 d1[8] = { 0 }, d2[8] = { 0 }; int cap = ch->userd_as ? ch->userd_as : SVC_APERTURE_SYS; @@ -14039,15 +32797,261 @@ static void svc_gr_ring_autopin(struct nv_vgpu_device *vgpu, pr_info("sema-chan: fbs scan done h=0x%x hits=%d (lim 0x%llx)\n", ch->h_object, hits, lim); } + + /* + * r495 diagnostic (X/GL channel): the X driver's GR channel + * registers its USERD at a memdesc base that may be an FB + * offset or a BAR1 window. The driver's kickoff writes GP_PUT + * through its CPU mapping of that object; if the write landed + * in the emulated FB shadow, scanning for the USERD shape + * (GP_PUT in [1..entries] at +0x8c, GP_GET <= GP_PUT at +0x90, + * both nonzero somewhere in the 0x200 block) names the real + * offset. One-shot per channel, diagnostic only. + */ + if (gsp_sema_dbg && vgpu->bars[1].vaddr && + svc_fb_shadow_size(vgpu) && ch->gpfifo_entries <= 0x20000) { + u64 lim = min_t(u64, svc_fb_shadow_size(vgpu), 0x20000000ULL); + u64 off; + int hits = 0; + + for (off = 0; off + 0xa0 <= lim && hits < 8; + off += 0x200) { + u32 hp, hg; + + memcpy(&hp, vgpu->bars[1].vaddr + off + 0x8c, 4); + memcpy(&hg, vgpu->bars[1].vaddr + off + 0x90, 4); + if (!hp || hp > ch->gpfifo_entries || hg > hp) + continue; + pr_info("sema-chan: userd scan hit off=0x%llx put=%u get=%u (blk %08x %08x %08x %08x)\n", + off, hp, hg, + *(u32 *)(vgpu->bars[1].vaddr + off + 0x80), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x84), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x88), + *(u32 *)(vgpu->bars[1].vaddr + off + 0x94)); + hits++; + } + pr_info("sema-chan: userd scan done h=0x%x hits=%d (lim 0x%llx)\n", + ch->h_object, hits, lim); + } + } + +skip_shadow_scan: + /* + * r496b: guest-RAM content scan for the ring (see the gr_ram_cursor + * field comment). The driver's kickoff writes GPFIFO entries whose + * pushbuffer VAs sit just BELOW the ring VA in the same allocation + * (main pushbuffer first, GPFIFO after - nvidia-push-priv.h): entry0 + * in (ring_va - 0x100000, ring_va), entry1 above entry0 and up to + * ring_va + 0x100000 (the progress tracker). r908 relaxations, + * measured on the GL client ring (va=0x1203cfff0, gpa 0x231e5d000): + * the entry pair is tested at the ring VA's own in-page offset (the + * entries start AT the registered VA), entry0 may sit either side of + * the ring within the window, and entry1 may repeat entry0's VA + * (that client queues {push,64} then {push,32}). A page whose entry + * pair both name such VAs with sane lengths is the ring; pin it + * (validated) and its allocation window (guesses) so the ring entry + * and pushbuffer reads resolve. Sliced so the service thread stays + * responsive; gives up per channel after svc_gr_ram_pass_budget() + * slices (two sweeps of sema_gr_scan_top). + * + * SESSION 147 (r840 evidence): the scan must NOT run for the kernel + * RC-watchdog channel (h=0x31415900). Its ring is a plain kernel + * allocation whose VA is always resolvable through the driver- + * published VAS walk - the walk is the correct, deterministic + * source. On r840 the walk lost a registration race, the scan ran + * first and latched a FOREIGN page (pa=0x101cc3000, garbage + * entries: raw=0xffff88f10403e000), and the bad latch poisoned + * every later entry/push read: the driver's DMA test never saw its + * push complete and Xorg died with "Failed to initialize DMA" + * (the same boot passed on r835/836/838 where the trusted walk + * won the race). Gating the scan keeps the per-pass walk retry + * alive for the watchdog; the walk landing late is indistinguishable + * from the r838 flow, and never landing is no worse than a poisoned + * latch. + */ + if (ch->h_object != 0x31415900u && + ch->gr_ram_passes < svc_gr_ram_pass_budget() && + !test_bit(ch->gr_index, &svc_gr_ring_done)) { + u64 hit_pa = 0; + bool hit; + + hit = svc_gr_ram_ring_scan(vgpu, ch, &hit_pa, false, false); + if (gsp_sema_dbg && (ch->h_object & 0xfffff000u) == 0xbeef0000u) { + static unsigned long dbg_at; + + if (time_after(jiffies, dbg_at)) { + dbg_at = jiffies + HZ; + pr_info("sema-chan: gr-scan h=0x%x rank=%u cursor=0x%llx passes=%u hit=%d pa=0x%llx\n", + ch->h_object, ch->gr_index, + ch->gr_ram_cursor, ch->gr_ram_passes, + hit ? 1 : 0, hit ? hit_pa : 0); + } + } + if (hit) { + /* + * Pin ONLY the content-validated ring page (validated, + * not a window guess). The pushbuffer pages and the + * tracker push of this client resolve through the VAS + * page directory registered from COPY_SERVER_PDES + * (r496g) - the surface is NONCONTIGUOUS, so a linear + * window guess is wrong for them and, being a + * pin_lookup_any hit, would shadow the correct walk in + * svc_lookup_trusted (observed r502: the tracker push + * read zeros through the wrong window page and the + * driver's DMA poll never completed). + */ + svc_pin_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, hit_pa, + SVC_APERTURE_SYS); + svc_ram_ring_owner_add(ch->h_object, hit_pa); + ch->gr_ram_logged = true; + ch->gr_ram_ring = true; + set_bit(ch->gr_index, &svc_gr_ring_done); + svc_ring_n++; + /* + * r496j: with the ring content-validated, verify the + * parked COPY_SERVER_PDES candidates against it and + * register the directory that reproduces the ring's + * PA; its walk then resolves the client's other VAs + * (pushbuffer pages, the progress semaphore the + * driver polls). r496f's blind vidmem sweep stays as + * a fallback for directories the control never + * published. + */ + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + hit_pa); + svc_xvas_pdb_discover(vgpu, ch->gpfifo_va, hit_pa); + /* r1780 (V37): pin the pushes NOW, while the + * verified directory incarnation is still live. */ + svc_pin_ring_pushes(vgpu, ch, hit_pa, + SVC_APERTURE_SYS); + /* + * Persist the latch immediately: a later stalled + * path in this pass may `continue` past the + * end-of-pass write-back, and the next pass's + * snapshot must carry gr_ram_ring (the zero-NOP + * exception and the no-capture-hold skip key on + * it). + */ + { + unsigned long flags2; + int i2; + + spin_lock_irqsave(&svc.chan_lock, flags2); + for (i2 = 0; i2 < SVC_MAX_CHANNELS; i2++) + if (svc.chan[i2].valid && + svc.chan[i2].vgpu == vgpu && + svc.chan[i2].h_object == + ch->h_object) { + svc.chan[i2].gr_ram_ring = true; + svc.chan[i2].gr_ram_logged = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags2); + } + pr_info("sema-chan: GR ring h=0x%x rank=%u found by RAM content scan va=0x%llx -> pa=0x%llx (allocation window pinned; entry/push reads resolve through it)\n", + ch->h_object, ch->gr_index, ch->gpfifo_va, + hit_pa); + return; + } } } +/* + * r1777 (V12): gate for the direct-GL CE-family ring probe below. Default + * OFF: the live A/B latched false positives on dense data pages and the + * L0-sanity predicate still needs tightening before it may guard the GR + * lane (see the wedge/CE RCA notes). sema_ce_ram=1 enables the probe; the + * committed default preserves the validated pre-V12 behaviour. + */ +module_param_named(sema_ce_ram, gsp_sema_ce_ram, bool, 0644); +MODULE_PARM_DESC(sema_ce_ram, + "Discover the direct-GL CE sibling rings by guest-RAM content scan + identity probe (default 0 = pre-V12 A/B)"); + +/* + * r1777 (V12): direct-GL CE-family ring discovery (beef 0xbeef01xx). + * + * The direct-GL client registers a GR channel (0xbeef0100, ring 0x4020000) + * and CE siblings (0xbeef0101/0102, rings 0x4100000/0x4160000) that share + * VAS 0xbeef0300. The GR ring resolves (registered directory + RAM content + * scan + auto-pin); the CE rings never did (r1775/r1776: every parked or + * registered directory incarnation either misses the ring VA - walk rc=-2 - + * or maps a zeroed page, and the CE branch of svc_ring_autopin never tried + * either of the two mechanisms that resolved the GR ring). This probe runs + * in that branch (beef CE channels only, gated by sema_ce_ram) and: + * 1. reads the ring VA raw - the RM low aperture is an identity-mapped + * VA==PA window (the pump's GR path uses the same trick) - and pins + * va->va when entry0 decodes as a plausible low-family push; + * 2. reuses the same guest-RAM content scan loop the GR family uses + * (svc_gr_ram_ring_scan) with the low-aperture push validator, but + * refuses pages another channel's RAM ring already owns + * (svc_ram_ring_owner_*: the GR page passes the same shape, r1776). + * On a latch the ring VA is pinned like the GR scan latch, so the pump's + * entry/USERD reads resolve through the page. + */ +static bool svc_ce_beef_ring_probe(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + u8 e[2 * SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1; + u64 hit_pa = 0; + + if (!ch->gpfifo_va) + return false; + /* 1. identity raw read (the low aperture is a VA==PA window). */ + if (!svc_read(vgpu, ch->gpfifo_va, e, sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if ((e0 || e1) && svc_gr_entry_plausible(ch, e0, e1) && + !svc_ram_ring_owner_foreign(ch->h_object, ch->gpfifo_va)) { + svc_pin_store(ch->gpfifo_va, ch->gpfifo_va, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, ch->gpfifo_va, + SVC_APERTURE_SYS); + svc_ram_ring_owner_add(ch->h_object, ch->gpfifo_va); + set_bit(ch->ce_index, &svc_ring_done); + svc_ring_n++; + pr_info("sema-chan: CE ring h=0x%x rank=%u identity-probed va=0x%llx -> pa=0x%llx (entry0 va=0x%llx len=%u)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, + ch->gpfifo_va, + ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc), + ((e1 >> 10) & 0x1fffff) << 2); + return true; + } + } + /* 2. guest-RAM content scan (same loop the GR family uses). */ + if (ch->gr_ram_passes < svc_gr_ram_pass_budget() && + svc_gr_ram_ring_scan(vgpu, ch, &hit_pa, true, true)) { + u64 eva = 0; + u32 elen = 0; + + if (!svc_read(vgpu, hit_pa, e, 8)) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + eva = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + elen = ((e1 >> 10) & 0x1fffff) << 2; + } + svc_pin_store(ch->gpfifo_va, hit_pa, SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, hit_pa, SVC_APERTURE_SYS); + svc_ram_ring_owner_add(ch->h_object, hit_pa); + set_bit(ch->ce_index, &svc_ring_done); + svc_ring_n++; + pr_info("sema-chan: CE ring h=0x%x rank=%u found by RAM content scan va=0x%llx -> pa=0x%llx (sibling pin; entry0 va=0x%llx len=%u)\n", + ch->h_object, ch->ce_index, ch->gpfifo_va, hit_pa, + eva, elen); + return true; + } + return false; +} + static void svc_ring_autopin(struct nv_vgpu_device *vgpu, struct svc_channel *ch) { u64 pa = 0, va = 0; - int ap = SVC_APERTURE_SYS; - u8 e[SVC_GPFIFO_ENTRY_SIZE]; + int ap = SVC_APERTURE_SYS; u8 e[SVC_GPFIFO_ENTRY_SIZE]; u32 e0, e1, len; if (!ch->engine_type) { @@ -14063,6 +33067,16 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, svc_ring_n++; return; } + /* + * r1777 (V12): the direct-GL CE siblings get the same two discovery + * sources the GR ring has (content scan + identity probe) before the + * FB convention probe below, which can never describe their + * low-aperture rings. + */ + if (gsp_sema_ce_ram && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + svc_ce_beef_ring_probe(vgpu, ch)) + return; pa = SVC_CE_RING_FB_BASE + SVC_CE_RING_FB_STRIDE * ch->ce_index; if (svc_read_ap(vgpu, pa, SVC_APERTURE_FB, e, sizeof(e))) @@ -14077,6 +33091,13 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, svc_pin_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); svc_lat_store(ch->gpfifo_va, pa, SVC_APERTURE_FB); + /* + * SESSION 144b: record the block ownership here too - CE ranks that + * latch through this plain autopin path (not the first-submission + * capture) were invisible to the GR fallback scan, which then stole + * their block (r809: CE rank 1 at fb 0xa0000). + */ + svc_fb_claim_add(ch->h_object, pa, pa + 0x10000); set_bit(ch->ce_index, &svc_ring_done); svc_ring_n++; pr_info("sema-chan: auto-pinned ring h=0x%x rank=%u va=0x%llx -> fb 0x%llx (entry0 va=0x%llx len=%u)\n", @@ -14084,6 +33105,90 @@ static void svc_ring_autopin(struct nv_vgpu_device *vgpu, } /* + * R10 (patch A): content-anchored ring window for a GR (eng==0) channel. + * + * A GR channel whose ring was found by the guest-RAM content scan + * (gr_ram_ring; svc_gr_ram_ring_scan) has exactly one content-verified + * anchor: the ring page itself, pinned by the scan. The scan's contract + * bounds the channel's pushbuffer VAs to +/- SVC_GR_RAM_WIN/2 around the + * ring VA (entry0 just below the ring, the progress tracker above), so + * install the window as two one-megabyte svc_pb[] entries with the same + * +1 slope convention CE uses (cross-checked on the RC watchdog r880: + * ring 0x120000400 -> pa 0x10567a400, its push 0x120000000 -> + * pa 0x10567a000). + * + * Strictly additive: only eng==0 channels whose ring was content- + * validated are eligible - the kernel-RM sysmem clients the scan exists + * for. CUDA channels keep their verified VAS/walk sources; the RC + * watchdog is excluded from the scan gate. The entries are consumed + * only by svc_pb_win_lookup, i.e. as an untrusted last-resort source in + * the push read, after every per-page validated source missed, with the + * existing all-zero first-chunk rejection still applied. No pin, lat + * or other lookup consumer sees these guesses. + */ +static void svc_pb_pair_gr(struct svc_channel *ch) +{ + u64 ring_pa = 0, ring_va; + int ring_ap = SVC_APERTURE_SYS; + u32 want, n = 0; + + if (ch->gr_index >= 32 || !ch->gr_ram_ring || !ch->gpfifo_va) + return; + if (test_bit(ch->gr_index, &svc_pb_gr_done)) + return; + if (svc_pin_lookup_any(ch->gpfifo_va, &ring_pa, &ring_ap)) + return; + ring_pa &= ~(u64)(SEMA_PAGE_SIZE - 1); + if (!ring_pa || ring_pa >= svc_pa_max()) + return; + ring_va = ch->gpfifo_va & ~(u64)(SEMA_PAGE_SIZE - 1); + want = SVC_PB_GR_SLOT_BASE | ch->gr_index; + + /* + * r914: replace this rank's previous pair instead of appending. + * svc_pb_win_lookup returns the FIRST matching slot entry, so a + * stale earlier pair (previous attempt's ring PA) would keep + * winning over the refreshed one; svc_chan_rearm_rank re-arms this + * pairing per (re-)registration for exactly that reason. + */ + { + u32 i; + + for (i = 0; i < svc_pb_n; i++) { + if (svc_pb[i].slot == want) { + svc_pb[i] = svc_pb[svc_pb_n - 1]; + svc_pb_n--; + i--; + } + } + } + + /* Lower half: [ring - 1 MB, ring) -> [ring_pa - 1 MB, ring_pa). */ + if (svc_pb_n < SVC_PB_MAX && ring_va >= SVC_GR_RAM_WIN / 2 && + ring_pa >= SVC_GR_RAM_WIN / 2) { + svc_pb[svc_pb_n].va = ring_va - SVC_GR_RAM_WIN / 2; + svc_pb[svc_pb_n].pa_page = ring_pa - SVC_GR_RAM_WIN / 2; + svc_pb[svc_pb_n].slot = want; + svc_pb_n++; + n++; + } + /* Upper half: [ring, ring + 1 MB) -> [ring_pa, ring_pa + 1 MB). */ + if (svc_pb_n < SVC_PB_MAX && + ring_pa + SVC_GR_RAM_WIN / 2 < svc_pa_max()) { + svc_pb[svc_pb_n].va = ring_va; + svc_pb[svc_pb_n].pa_page = ring_pa; + svc_pb[svc_pb_n].slot = want; + svc_pb_n++; + n++; + } + if (!n) + return; + set_bit(ch->gr_index, &svc_pb_gr_done); + pr_info("sema-svc: GR ring window paired h=0x%x rank=%u ring va=0x%llx -> pa=0x%llx (%u entries; untrusted last-resort push window)\n", + ch->h_object, ch->gr_index, ring_va, ring_pa, n); +} + +/* * Pair a channel's push buffer with its base page: the ring's first entry * carries the push-buffer VA, the base page was found by content and its * release names the channel's slot (4 bytes per rank). @@ -14093,7 +33198,11 @@ static void svc_pb_pair(struct nv_vgpu_device *vgpu, struct svc_channel *ch) u64 hit_pa = 0, va = 0; u32 len = 0, i, hit_i = 0, want, off; - if (!ch->engine_type || ch->ce_index >= 32) + if (!ch->engine_type) { + svc_pb_pair_gr(ch); + return; + } + if (ch->ce_index >= 32) return; if (test_bit(ch->ce_index, &svc_pb_done)) return; @@ -14455,7 +33564,7 @@ static int svc_rescue_run_try(struct nv_vgpu_device *vgpu, break; } pa = (u64)((s64)anchor_pa + delta); - if (pa + SEMA_PAGE_SIZE > SEMA_PA_MAX) { + if (pa + SEMA_PAGE_SIZE > svc_pa_max()) { rc = -EINVAL; break; } @@ -14601,6 +33710,65 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, u32 ncand = 0, i; /* + * SESSION 148 perf (measured on r843): content-verified full-chain + * rescue, usable by GR channels too. The CE-only gate below refused + * every GR push outright, yet the same svc_lookup the diagnostic + * probe uses resolves the stalled entries correctly (X's frame + * pushes at 0x120000514 -> vidmem shadow pa=0x200514 with plausible + * stream content) while the pump's own read went through a stale + * shadowing translation, failed plausibility, and skipped the entry + * - one full retry window per push, the measured 22.4 s/frame + * glxgears cadence. Resolve via svc_lookup, verify the bytes parse + * as a method stream, then pin the verified page so the pump's + * re-read lands on it. Cheap: one chain lookup + one page read. + */ + if (entry_va && entry_len && entry_len <= SEMA_PAGE_SIZE) { + u64 mpa = 0; + int map = SVC_APERTURE_SYS; + u8 *vb = kmalloc(entry_len, GFP_KERNEL); + int lrc = -ENOMEM, rrc = -ENOMEM; + bool plaus = false; + + if (vb) { + lrc = svc_lookup(vgpu, entry_va, &mpa, &map); + if (!lrc) + rrc = svc_read_ap(vgpu, mpa, map, vb, + entry_len); + if (!rrc) + plaus = svc_method_stream_plausible(vb, + entry_len); + if (gsp_sema_trace_h && + (u32)ch->h_object == (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: maprescue h=0x%x va=0x%llx len=%u lookup=%d pa=0x%llx ap=%d read=%d plaus=%d\n", + ch->h_object, entry_va, entry_len, + lrc, mpa, map, rrc, plaus); + } + if (!lrc && !rrc && plaus) { + svc_pin_store(entry_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + mpa & + ~(u64)(SEMA_PAGE_SIZE - 1), + map); + svc_lat_store(entry_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + mpa & + ~(u64)(SEMA_PAGE_SIZE - 1), + map); + svc.rescue_map_ok++; + if (svc.rescue_map_ok <= 8) + pr_info("sema-svc: rescue via map h=0x%x va=0x%llx -> pa=0x%llx ap=%d (verified stream)\n", + ch->h_object, entry_va, mpa, + map); + kfree(vb); + return 0; + } + kfree(vb); + } + } + + /* * The matcher needs only the channel rank (slot) and the release-push * shape; it does NOT need the pool semaphore translation that * expected_sem_payload is learned from. Requiring the learned payload @@ -14732,8 +33900,8 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, lo = ALIGN(svc_pb[i].pa_page > (512ULL << 20) ? svc_pb[i].pa_page - (512ULL << 20) : (u64)SVC_SCAN_CHUNK, SVC_SCAN_CHUNK); - if (hi > SEMA_PA_MAX - SVC_SCAN_CHUNK) - hi = SEMA_PA_MAX - SVC_SCAN_CHUNK; + if (hi > svc_pa_max() - SVC_SCAN_CHUNK) + hi = svc_pa_max() - SVC_SCAN_CHUNK; if (lo <= hi && !svc_rescue_scan_range(vgpu, ch, entry_va, entry_len, slot, hi, lo, buf, cand, cand_va)) { @@ -14758,7 +33926,7 @@ static int svc_rescue_push_page(struct nv_vgpu_device *vgpu, * few hundred MB. */ i = svc_rescue_scan_range(vgpu, ch, entry_va, entry_len, slot, - SEMA_PA_MAX - SVC_SCAN_CHUNK, 0, buf, cand, + svc_pa_max() - SVC_SCAN_CHUNK, 0, buf, cand, cand_va); kfree(buf); if (!i) @@ -14904,6 +34072,16 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, */ u64 pa = ch->release_pa ? ch->release_pa : ch->counter_pa; + if (gsp_sema_trace_h && (u32)ch->h_object == (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: pub h=0x%x put=%u last=%u armed=%d pa=0x%llx cnt=%u gate=%s\n", + ch->h_object, put, (u32)ch->last_get, ch->gr_pub_armed, + pa, ch->counter_last, + !pa ? "no-pa" : + (ch->last_get != put) ? "outstanding" : + (!ch->gr_pub_armed ? "unarmed" : "publish")); + } if (!pa) return; /* @@ -14958,6 +34136,25 @@ static void svc_counter_publish(struct nv_vgpu_device *vgpu, return; val = put; + /* + * r471: never advance the guest's completion word past a push whose + * content was still unwritten when its entry was consumed. The + * clamp position is the lowest all-zero GR slot (gr_bad_put = slot + * + 1; 0 = none). Without it the publish can satisfy a D2H wait + * before its copy ran and the client drains a reused staging buffer + * holding the previous copy's bytes (streams_events h_c[2] = c0). + * The clamp heals when the rewinds re-read the slot with real + * content; the real completions still arrive through the decoded + * releases. + */ + if (!ch->engine_type && ch->gr_bad_put && val >= ch->gr_bad_put) { + val = ch->gr_bad_put - 1; + if (gsp_sema_dbg && svc.counter_pub_dbg < 96) { + svc.counter_pub_dbg++; + pr_info("sema-diag: pub-clamp h=0x%x pa=0x%llx val=%u (bad=%u) cur=%u\n", + ch->h_object, pa, val, ch->gr_bad_put, cur); + } + } if (val == ch->counter_last) return; /* @@ -15086,9 +34283,15 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch sizeof(back)) && back == nv) { delivered = true; - if (svc_pool_sem_va(pool_va)) - svc_pool_pa_store(pool_va, spa, + if (svc_pool_sem_va(pool_va)) { + svc_pool_pa_store(vgpu, pool_va, spa, svc_ap_of(sap)); + /* r539: reach the secondary candidate + * page too (pin/walk disagreement). */ + svc_pool_boot_fanout(vgpu, + ch->h_object, + pool_va, nv); + } svc.releases_applied++; if (!gsp_sema_fake_rel) svc.releases_recovered++; @@ -15105,6 +34308,22 @@ static bool svc_fake_release(struct nv_vgpu_device *vgpu, struct svc_channel *ch svc_pending_add(pool_va, deliver, 0); svc.releases_forwarded++; } + /* + * r576b DIAG: witness every authored/queued recovery release and the + * target it chose. The r575 atomics wedge (a UVM tracker wait for + * payload 34 that stalled at 25 while the ring was consumed) needs + * this to prove which VA the fakes targeted when the client's pool + * stopped advancing - last_rel_va is learned from readable pushes and + * a foreign release VA there would send every later recovery to the + * wrong semaphore. + */ + if (gsp_sema_dbg && svc_fake_rel_log < 256) { + svc_fake_rel_log++; + pr_info("sema-fake: h=0x%x pool=0x%llx deliver=%u delivered=%d have_pa=%d last_rel=0x%llx exp=%u\n", + ch->h_object, pool_va, deliver, delivered ? 1 : 0, + have_pa ? 1 : 0, ch->last_rel_va, + ch->expected_sem_payload); + } svc.releases_faked++; svc_rel_count++; ch->expected_sem_payload++; @@ -15143,6 +34362,179 @@ static bool svc_entry_want_reached(struct nv_vgpu_device *vgpu, return cur >= ch->entry_want; } +/* ------------------------------------------------------------------ */ +/* Native publish-counter advance (SESSION 153-CONTINUED-4, pace wall) */ +/* ------------------------------------------------------------------ */ + +/* + * Two CPU-polled publish words gate the GLX client's per-frame pace; both + * live in the X driver's shared pages (CPU-mapped from its memfd) and both + * were proven to be needed: a guest daemon poking only the 0x44000-offset + * page sustained 140 FPS while the other word was already latched at 3; + * resetting both to 2 collapsed to the 22.4 s give-up ladder, and an + * ordered re-poke showed each word releasing part of the stall (page2 3 + + * main 2 -> still stalled; main 3 -> "603 frames in 5.0 seconds"). Each + * page is discovered by content and the counter word is at +0x8. + * + * Page signatures: a boot-varying magic at +0x4 (observed 0x0e836365 and + * 0xc228e806) is useless, but the handle words around it are stable: + * +0x10 family 0xc1d0xxxx and +0x14 == 0xcaf00006 on both observed boots. + * Page B's sparse notifier state is +0x208==0x80, +0x20c==1, +0x210==0. + * Both signatures were verified unique over the full 8 GiB RAM scan on the + * r935 image (and the legacy magic on r934) - exactly one match each. + */ +#define SVC_PUB_A_HANDLE_LO 0x14 /* stable client handle word */ +#define SVC_PUB_A_HANDLE_LO_V 0xcaf00006u +#define SVC_PUB_A_HANDLE_HI 0x10 /* 0xc1d0xxxx family */ +#define SVC_PUB_A_HANDLE_MASK 0xfffff000u +#define SVC_PUB_A_HANDLE_HI_V 0xc1d00000u +#define SVC_PUB_A_MAGIC_OFF 0x4 /* legacy boot-varying magic */ +#define SVC_PUB_A_MAGIC 0x0e836365u +#define SVC_PUB_B_OFF 0x208 /* sparse 2D notifier state */ +#define SVC_PUB_B_208 0x80u +#define SVC_PUB_B_20C 0x1u +#define SVC_PUB_WORD_OFF 0x8 +#define SVC_PUB_VALUE 3u +#define SVC_PUB_SANE_MAX 0x100u /* counter plausibility bound */ +#define SVC_PUB_SCAN_CAP 16 /* attempts before giving up per boot */ + +static u64 svc_pub_pa_a; +static u64 svc_pub_pa_b; +static unsigned long svc_pub_next_scan; +static u32 svc_pub_attempts; +static u32 svc_pub_writes; + +/* + * Content discovery for both publish pages. The pages are created per + * session by the X driver and their GPAs move per boot, so find them by + * structure (the r934 full-RAM scan proved each signature matches exactly + * one page in all of guest RAM). Descending walk: the driver allocates in + * the high band, so the pages are reached within the first chunks. + */ +static void svc_pub_scan(struct nv_vgpu_device *vgpu) +{ + u8 *buf = kmalloc(SVC_SCAN_CHUNK, GFP_KERNEL); + u64 gpa; + + if (!buf) + return; + svc_pub_attempts++; + for (gpa = svc_pa_max() - SVC_SCAN_CHUNK;; gpa -= SVC_SCAN_CHUNK) { + if (!svc_read(vgpu, gpa, buf, SVC_SCAN_CHUNK)) { + u32 o; + + for (o = 0; o + SEMA_PAGE_SIZE <= SVC_SCAN_CHUNK; + o += SEMA_PAGE_SIZE) { + u32 w4, w8, w10, w14, w208, w20c, w210; + + if (!svc_pub_pa_a) { + memcpy(&w8, buf + o + SVC_PUB_WORD_OFF, 4); + if (w8 <= SVC_PUB_SANE_MAX) { + memcpy(&w4, buf + o + SVC_PUB_A_MAGIC_OFF, 4); + memcpy(&w10, buf + o + SVC_PUB_A_HANDLE_HI, 4); + memcpy(&w14, buf + o + SVC_PUB_A_HANDLE_LO, 4); + if (w4 == SVC_PUB_A_MAGIC || + (w14 == SVC_PUB_A_HANDLE_LO_V && + (w10 & SVC_PUB_A_HANDLE_MASK) == + SVC_PUB_A_HANDLE_HI_V)) + svc_pub_pa_a = gpa + o; + } + } + if (!svc_pub_pa_b) { + memcpy(&w208, buf + o + SVC_PUB_B_OFF, 4); + if (w208 == SVC_PUB_B_208) { + memcpy(&w20c, buf + o + SVC_PUB_B_OFF + 4, 4); + memcpy(&w210, buf + o + SVC_PUB_B_OFF + 8, 4); + memcpy(&w8, buf + o + SVC_PUB_WORD_OFF, 4); + if (w20c == SVC_PUB_B_20C && + w210 == 0 && + w8 <= SVC_PUB_SANE_MAX) + svc_pub_pa_b = gpa + o; + } + } + if (svc_pub_pa_a && svc_pub_pa_b) + goto out; + } + } + if (gpa == 0) + break; + } +out: + kfree(buf); + if (svc_pub_pa_a || svc_pub_pa_b) + pr_info("sema-pubadv: publish pages discovered a=0x%llx b=0x%llx (attempt %u)\n", + svc_pub_pa_a, svc_pub_pa_b, svc_pub_attempts); + else if (svc_pub_attempts <= 8) + pr_info("sema-pubadv: publish pages not found yet (attempt %u)\n", + svc_pub_attempts); +} + +/* + * Write-3 + readback for one target. Returns false when the target is + * unreadable or non-coherent so a stale/foreign page can be dropped and + * re-discovered (never persist a wrong write target). + */ +static bool svc_pub_write_one(struct nv_vgpu_device *vgpu, u64 pa, u32 *back) +{ + u32 v = SVC_PUB_VALUE; + int rc; + + *back = 0; + rc = svc_write(vgpu, pa + SVC_PUB_WORD_OFF, &v, sizeof(v)); + if (!rc) + rc = svc_read(vgpu, pa + SVC_PUB_WORD_OFF, back, sizeof(*back)); + return !rc && *back == v; +} + +/* + * Called from svc_replay_note (pump retire path) for every consumed slot. + * Only kick-family channels (the GLX client's user channels, h=0xbfef00xx) + * drive the publish counters; every other lane is untouched. + */ +static void svc_pub_advance(struct svc_channel *ch) +{ + u32 back_a = 0, back_b = 0; + bool ok_a, ok_b; + + if (!gsp_sema_pub_adv || !ch || !ch->vgpu) + return; + if (ch->h_object < 0xbfef0000u || ch->h_object > 0xbfef00ffu) + return; + if (!svc_pub_pa_a && !svc_pub_pa_b) { + if (gsp_sema_pub_pa) { + svc_pub_pa_a = gsp_sema_pub_pa & + ~((u64)SEMA_PAGE_SIZE - 1); + } else if (svc_pub_attempts < SVC_PUB_SCAN_CAP && + time_after_eq(jiffies, svc_pub_next_scan)) { + svc_pub_scan(ch->vgpu); + svc_pub_next_scan = jiffies + 30 * HZ; + } + if (!svc_pub_pa_a && !svc_pub_pa_b) + return; + } + ok_a = !svc_pub_pa_a || + svc_pub_write_one(ch->vgpu, svc_pub_pa_a, &back_a); + ok_b = !svc_pub_pa_b || + svc_pub_write_one(ch->vgpu, svc_pub_pa_b, &back_b); + svc_pub_writes++; + if (!ok_a && !gsp_sema_pub_pa) { + pr_warn("sema-pubadv: page a write rejected (pa=0x%llx back=%u); dropping page\n", + svc_pub_pa_a, back_a); + svc_pub_pa_a = 0; + svc_pub_next_scan = jiffies; + } + if (!ok_b && !gsp_sema_pub_pa) { + pr_warn("sema-pubadv: page b write rejected (pa=0x%llx back=%u); dropping page\n", + svc_pub_pa_b, back_b); + svc_pub_pa_b = 0; + svc_pub_next_scan = jiffies; + } + if (svc_pub_writes <= 8 || (svc_pub_writes & 0x3ff) == 0) + pr_info("sema-pubadv: h=0x%x a=0x%llx/%s b=0x%llx/%s n=%u\n", + ch->h_object, svc_pub_pa_a, ok_a ? "3" : "ERR", + svc_pub_pa_b, ok_b ? "3" : "ERR", svc_pub_writes); +} + /* Mark a consumed GPFIFO slot for the current put epoch. */ static void svc_replay_note(struct svc_channel *ch, u32 idx, u32 put) { @@ -15175,6 +34567,11 @@ static void svc_replay_note(struct svc_channel *ch, u32 idx, u32 put) svc_opdef_finish_ch(ch, idx); if (idx < SVC_GPFIFO_MAX_BITS) set_bit(idx, ch->slot_done_epoch); + /* + * SESSION 153-CONTINUED-4: retirement of a kick-family slot advances + * the GLX client's publish counter (see svc_pub_advance). + */ + svc_pub_advance(ch); } /* Was this slot already consumed in the current put epoch? */ @@ -15322,11 +34719,29 @@ struct svc_fwd_req { u32 gr_dropped; /* GRFIX: pages the fill's last pass could not * resolve (0 = launch fully resolved); read * by the absolute-bound expiry log */ + bool gr_arg_partial; /* r543: an ARGUMENT pointer resolved 0 + * pages while other arguments resolved - the + * launch would run with a stale/zero operand + * (the r542 matmul NOWRITE wedge: fe_esr= + * 0x80000000, engine fetch OK, wrote 0, + * client read zeros). The request path holds + * such a launch once per push VA through the + * bounded busy-defer retry. */ u64 gr_push_va; u32 gr_push_len; u32 gr_push_n; u32 gr_push_qmd_hdr; u32 gr_push_qmd_end; + /* + * r929 (GL 3D path): the captured raw 3D method window (gr_raw + * words gr_stream_off..gr_stream_off+gr_stream_n) replays verbatim; + * see struct gsp_phys_gr_submit.stream. + */ + bool gr_stream_3d; + u32 gr_stream_off; + u32 gr_stream_n; + u32 gr_stream_rel; /* stream-relative dword offset of this + * window (0 = the section's first chunk) */ u32 gr_raw[SVC_GR_PUSH_WORDS]; u64 gr_qmd_dev_va; /* @@ -15350,6 +34765,10 @@ struct svc_fwd_req { u64 gr_rel_pa; u8 gr_rel_ap; /* SVC_APERTURE_* of gr_rel_pa */ u64 gr_prog_va; + /* CONT-87: the window's own program binds (see svc_gr_cand_rec). */ + u32 gr_n_prog_binds; + u64 gr_prog_bind_va[SVC_GR_CAND_PROG_BINDS]; + u32 gr_prog_bind_slot[SVC_GR_CAND_PROG_BINDS]; u64 gr_param_va; u32 gr_param_size; u32 gr_qmd[SVC_GR_QMD_WORDS]; @@ -15365,6 +34784,217 @@ struct svc_fwd_req { static struct svc_fwd_req svc_fwd_req; +/* + * SESSION 153-CONTINUED-61: MISS-ABORT re-arm. The pre-doorbell abort + * (GSP_PHYS_GR_RC_MISSABORT) fires when a 3D-window operand's page is still + * unresolved; the page is demanded through the pin path and resolves within + * the pin cycle (tens of seconds). The historical handling consumed the + * captured candidate and software-completed the client (releases + QMD + * word), so no later forward ever re-ran that chunk and a store-bearing + * window never reached the engine (CONT-60). With the re-arm the candidate + * is copied back into the candidate ring unoffered, the client's releases + * are held back, and a bounded retry tick (svc_gr_missre_tick, called from + * svc_pump_channels) re-offers the same chunk; once the pins land the real + * launch runs and the engine (hold success edge) authors the releases. + * Bounded: after SVC_GR_MISSRE_DEAD_MS / SVC_GR_MISSRE_MAX_TRIES the + * historical consume-with-software-completion path stands so a page that + * never resolves cannot hang the client. Service-thread only. + */ +#define SVC_GR_MISSRE_INT_MS 1500u +#define SVC_GR_MISSRE_DEAD_MS 55000u +#define SVC_GR_MISSRE_MAX_TRIES 64u +static struct { + bool active; + u64 push_va; + u32 h_object; + u32 tries; + unsigned long first_at; + unsigned long last_try; +} svc_gr_missre; +static u32 svc_gr_missre_log; +static u64 svc_gr_missre_arm_n, svc_gr_missre_ok_n, svc_gr_missre_exp_n; + +static bool svc_gr_request(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch); + +/* + * Episode complete: the re-armed launch ran for real. Clears the retry + * state so the tick stops re-offering. Called from the success edges + * (svc_fwd_outcome_check / svc_fwd_hold_poll). + */ +static void svc_gr_missre_done(void) +{ + if (!svc_gr_missre.active || + svc_gr_missre.push_va != svc_fwd_req.gr_push_va) + return; + svc_gr_missre.active = false; + svc_gr_missre_ok_n++; + pr_info("vgsp-gr: MISS-RETRY converged push=0x%llx h=0x%x tries=%u ok=%llu (real launch ran after the pins landed)\n", + svc_fwd_req.gr_push_va, svc_fwd_req.gr_h_object, + svc_gr_missre.tries, svc_gr_missre_ok_n); +} + +/* + * Copy the just-aborted request's captured launch back into the candidate + * ring as an unoffered candidate (the re-arm). Returns false when the + * episode bound has expired or the ring has no free slot; the caller then + * falls through to the historical software completion. `rel`/`nrel` are + * the attempt's push-level release records (restored from the park table + * by svc_gr_request and drained by its caller): they are re-parked so the + * retried request restores the same set - the GPFIFO entry is consumed + * with this edge and will not re-decode. + */ +static bool svc_gr_missre_arm(const struct svc_fwd_rel *rel, u8 nrel) +{ + struct svc_gr_cand_rec *c = NULL; + unsigned long now = jiffies; + u32 slot; + + if (!svc_gr_missre.active || + svc_gr_missre.push_va != svc_fwd_req.gr_push_va || + svc_gr_missre.h_object != svc_fwd_req.gr_h_object) { + svc_gr_missre.active = true; + svc_gr_missre.push_va = svc_fwd_req.gr_push_va; + svc_gr_missre.h_object = svc_fwd_req.gr_h_object; + svc_gr_missre.tries = 0; + svc_gr_missre.first_at = now; + } + if (svc_gr_missre.tries >= SVC_GR_MISSRE_MAX_TRIES || + time_after(now, svc_gr_missre.first_at + + msecs_to_jiffies(SVC_GR_MISSRE_DEAD_MS))) { + u32 ri; + + svc_gr_missre_exp_n++; + pr_warn("vgsp-gr: MISS-RETRY episode expired push=0x%llx h=0x%x tries=%u waited_ms=%lu (falling back to software completion)\n", + svc_gr_missre.push_va, svc_gr_missre.h_object, + svc_gr_missre.tries, + jiffies_to_msecs(now - svc_gr_missre.first_at)); + /* + * Drop the lingering armed slot(s) for this launch: an + * unoffered candidate surviving the episode could be offered + * much later with a stale window snapshot (the client has + * since reused its buffers - the double-run class). + */ + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + struct svc_gr_cand_rec *cc = &svc_gr_cand_ring[ri]; + + if (cc->valid && !cc->offered && + cc->push_va == svc_gr_missre.push_va && + cc->h_object == svc_gr_missre.h_object) { + cc->offered = true; + cc->valid = false; + } + } + svc_gr_missre.active = false; + return false; + } + for (slot = 0; slot < SVC_GR_CAND_RING; slot++) { + if (!svc_gr_cand_ring[slot].valid || + svc_gr_cand_ring[slot].offered) { + c = &svc_gr_cand_ring[slot]; + break; + } + } + if (!c) + return false; /* ring full: fall back historically */ + memset(c, 0, sizeof(*c)); + c->valid = true; + c->offered = false; + c->rel_done = false; + c->captured_jiffies = now; + c->h_object = svc_fwd_req.gr_h_object; + c->push_va = svc_fwd_req.gr_push_va; + c->push_len = svc_fwd_req.gr_push_len; + c->push_n = svc_fwd_req.gr_push_n; + c->push_qmd_hdr = svc_fwd_req.gr_push_qmd_hdr; + c->push_qmd_end = svc_fwd_req.gr_push_qmd_end; + c->stream_3d = svc_fwd_req.gr_stream_3d; + c->stream_off = svc_fwd_req.gr_stream_off; + c->stream_n = svc_fwd_req.gr_stream_n; + c->stream_rel = svc_fwd_req.gr_stream_rel; + memcpy(c->raw, svc_fwd_req.gr_raw, sizeof(c->raw)); + c->qmd_dev_va = svc_fwd_req.gr_qmd_dev_va; + memcpy(c->qmd, svc_fwd_req.gr_qmd, sizeof(c->qmd)); + c->n_inline = svc_fwd_req.gr_n_inline; + if (c->n_inline > SVC_GR_INLINE_MAX) + c->n_inline = SVC_GR_INLINE_MAX; + memcpy(c->inline_recs, svc_fwd_req.gr_inline, + sizeof(c->inline_recs)); + c->prog_va = svc_fwd_req.gr_prog_va; + c->param_va = svc_fwd_req.gr_param_va; + c->param_size = svc_fwd_req.gr_param_size; + c->rel_va = svc_fwd_req.gr_rel_va; + c->rel_payload = svc_fwd_req.gr_rel_payload; + c->rel_flags = svc_fwd_req.gr_rel_flags; + if (rel && nrel) + svc_grpark_put(svc_fwd_req.vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, rel, nrel); + svc_gr_missre_arm_n++; + if (svc_gr_missre_log < 96) { + svc_gr_missre_log++; + pr_info("vgsp-gr: MISS-RETRY re-arm push=0x%llx h=0x%x slot=%u window=%u nrel=%u arm=%llu (candidate kept unoffered; retry tick re-offers after the pins)\n", + c->push_va, c->h_object, slot, c->stream_n, + (unsigned)nrel, svc_gr_missre_arm_n); + } + return true; +} + +/* + * The retry tick (called per pass from svc_pump_channels for the tracked + * channel): re-offer the re-armed candidate on a bounded cadence. Parks + * svc_gr_cand_pos on the tracked slot so svc_gr_request's oldest-unoffered + * scan selects exactly this candidate; svc_gr_request itself defers on + * busy/inflight and no-ops for channels without a matching candidate. + */ +static void svc_gr_missre_tick(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch) +{ + u32 ti; + + if (ch->engine_type || !svc_gr_missre.active || + svc_gr_missre.h_object != ch->h_object || + !gsp_phys_gr_forward_enabled() || + !time_after_eq(jiffies, svc_gr_missre.last_try + + msecs_to_jiffies(SVC_GR_MISSRE_INT_MS))) + return; + if (svc_gr_missre.tries >= SVC_GR_MISSRE_MAX_TRIES || + time_after(jiffies, svc_gr_missre.first_at + + msecs_to_jiffies(SVC_GR_MISSRE_DEAD_MS))) + return; /* the next abort edge expires the episode */ + svc_gr_missre.last_try = jiffies; + svc_gr_missre.tries++; + for (ti = 0; ti < SVC_GR_CAND_RING; ti++) { + struct svc_gr_cand_rec *cc = &svc_gr_cand_ring[ti]; + + if (cc->valid && !cc->offered && + cc->push_va == svc_gr_missre.push_va && + cc->h_object == svc_gr_missre.h_object) { + svc_gr_cand_pos = ti; + break; + } + } + if (svc_gr_missre_log < 192) { + svc_gr_missre_log++; + pr_info("vgsp-gr: MISS-RETRY tick try=%u push=0x%llx h=0x%x (re-offering the armed candidate)\n", + svc_gr_missre.tries, svc_gr_missre.push_va, + svc_gr_missre.h_object); + } + if (svc_gr_request(vgpu, ch)) { + /* + * A tick-queued request never gets an entry hold (the GPFIFO + * entry was consumed with the first abort - there is nothing + * left for the entry walk to hold). The per-push release + * records svc_gr_request just restored are re-parked so they + * survive the work item's lifetime; svc_fwd_outcome_check + * applies them when the retried launch's outcome lands. + */ + if (svc_fwd_defer_armed && svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, ch->h_object, + svc_gr_missre.push_va, + svc_fwd_rel, svc_fwd_rel_n); + } +} + static struct workqueue_struct *svc_fwd_wq; static unsigned long svc_fwd_state; static u64 svc_fwd_requests; @@ -15799,8 +35429,17 @@ static void svc_fwd_rel_apply_all(struct nv_vgpu_device *vgpu, * Bounded hold: comfortably above the observed ~1 s pin latency and well * inside the 25 s CE grace, so a genuinely dead operand delays its * channel by this much and then falls back to the old skip-and-release. + * + * r549: 10000 -> 3000. r548 evidence: a reused channel handle whose new + * client submits ~65 init entries with device-dst ops no resolver covers + * walked one entry per 10 s (the journal shows OPDEF-EXPIRED exactly + * every 10.5 s, idx 7 -> 18) while the client spun at 100 % CPU - the + * release set the client waits on advances one entry per hold, so the + * bound is the stall multiplier. Ops that do resolve land in 0.5-2 s; + * 3 s keeps the m163 guarantee (a failed device copy is still retried + * once after its pins arrive) and caps the dead-op walk 3.3x shorter. */ -#define SVC_OPDEF_HOLD_MS 10000u +#define SVC_OPDEF_HOLD_MS 3000u struct svc_opdef_ent { bool active; @@ -15876,13 +35515,85 @@ static void svc_opdef_clear(struct svc_opdef_ent *e, u32 caller) e->n ? e->rel[0].va : 0ull, e->n ? e->rel[0].payload : 0u); } - /* r444: the entry is being consumed; no retry can need its - * staging snapshot any more (see svc_copy_snap_free_owner). */ - if (e->active) - svc_copy_snap_free_owner(e->h_object, e->entry_idx); + /* + * r456/r457: the snapshot lifecycle is now "capture once, consume + * on the copy's success, evict on ring-slot reuse, TTL". Every + * opdef_clear caller can still re-decode the entry: + * - APPLY (0): the pass's own op execution (which consumes the + * snapshot on success) follows the apply; + * - FINISH (1): the entry can be REPLAYED after a consume (the + * pin-batch rewind replays entries while mappings stream in), + * and an owed copy still needs the pipeline-order bytes; + * - FLUSH (2): the op stays skipped this pass but a later replay + * may run it. + * The r457 mode-C evidence: the b1 H2D's snapshot was captured at + * 191607.816388 with b1's bytes (first4=00000000 3e800000 ...), + * FINISH freed it on the entry's consume, the rewind replay + * re-captured at 191608.798838 from the REFILLED staging + * (first4=00000000 3d800000 ... = b2) and the deferred execution + * wrote b2's bytes into db1 (dst16_after=000000000000803d...). + * The r444 leak the old free guarded against is already covered + * by the owner key (a stale snapshot can only ever serve its own + * (ch, idx, hash)), the slot-reuse eviction in + * svc_copy_snap_capture, the ops_done skip free in svc_run_ops + * and the 30 s TTL. + */ memset(e, 0, sizeof(*e)); } +/* + * r551: remember releases DELIVERED by the hold-expiry paths (flush and + * sweep) by (channel, entry, va, payload) for a short window. The pump + * re-decodes a parked entry on every pass; without this mark each re-decode + * recorded a fresh opdef slot and held the entry again - r550 + * victory_strided: SWEEP-EXPIRED every ~3 s for the SAME entry six times + * (entries 41..43, ~18-21 s per entry, OPDEF-EXPIRED only after six + * cycles), so a ~40-entry init sequence walked for 12+ minutes with the + * client at 100 % CPU. The release is a monotonic completion signal and + * every delivery path is forward-only, so re-applying it is a no-op: a + * re-decode of an already-delivered entry must neither withhold the + * release again nor re-arm the hold. + */ +static void svc_opdef_done_note(u32 h, u32 entry_idx, const struct svc_fwd_rel *r) +{ + u32 i = svc_opdef_done_next++ % SVC_OPDEF_DONE_SLOTS; + + svc_opdef_done[i].h_object = h; + svc_opdef_done[i].entry_idx = entry_idx; + svc_opdef_done[i].va = r->va; + svc_opdef_done[i].payload = r->payload; + svc_opdef_done[i].j = jiffies; +} + +static bool svc_opdef_done_seen(u32 h, u32 entry_idx, const struct svc_fwd_rel *r) +{ + u32 i; + + for (i = 0; i < SVC_OPDEF_DONE_SLOTS; i++) + if (svc_opdef_done[i].h_object == h && + svc_opdef_done[i].entry_idx == entry_idx && + svc_opdef_done[i].va == r->va && + svc_opdef_done[i].payload == r->payload && + time_before(jiffies, svc_opdef_done[i].j + + msecs_to_jiffies(SVC_OPDEF_DONE_TTL_MS))) + return true; + return false; +} + +/* Any release of this entry delivered recently? (hold gate) */ +static bool svc_opdef_done_entry(u32 h, u32 entry_idx) +{ + u32 i; + + for (i = 0; i < SVC_OPDEF_DONE_SLOTS; i++) + if (svc_opdef_done[i].h_object == h && + svc_opdef_done[i].entry_idx == entry_idx && + time_before(jiffies, svc_opdef_done[i].j + + msecs_to_jiffies(SVC_OPDEF_DONE_TTL_MS))) + return true; + return false; +} + static bool svc_opdef_contains(const struct svc_opdef_ent *e, const struct svc_fwd_rel *r) { @@ -16011,6 +35722,16 @@ static bool svc_opdef_release(struct nv_vgpu_device *vgpu, if (svc.op_tr_pending) { if (rescue) return e != NULL; + /* + * r551: this release was already delivered forward-only by the + * hold-expiry flush/sweep; the pump's re-decode of the parked + * entry must not withhold it (or re-arm the hold) again. + */ + if (svc_opdef_done_seen(ch->h_object, entry_idx, r)) { + svc_rel_va_seen = r->va; + svc_rel_count++; + return true; + } if (!e) { e = svc_opdef_alloc(vgpu, ch->h_object, entry_idx); if (!e) @@ -16065,6 +35786,9 @@ static bool svc_opdef_hold(struct nv_vgpu_device *vgpu, struct svc_channel *ch, { struct svc_opdef_ent *e = svc_opdef_find(vgpu, ch->h_object, entry_idx); + /* r551: a delivered entry is consumed, never held again. */ + if (svc_opdef_done_entry(ch->h_object, entry_idx)) + return false; if (!e) { e = svc_opdef_alloc(vgpu, ch->h_object, entry_idx); if (!e) @@ -16122,7 +35846,11 @@ static void svc_opdef_flush(struct nv_vgpu_device *vgpu, struct svc_channel *ch, if (!e) return; if (e->n) { + u8 i; + svc_fwd_rel_apply_all(vgpu, ch, e->rel, e->n); + for (i = 0; i < e->n; i++) + svc_opdef_done_note(ch->h_object, entry_idx, &e->rel[i]); pr_info("sema-opdef: EXPIRED h=0x%x entry=%u n=%u applied (operand never translated; op stays skipped)\n", ch->h_object, entry_idx, e->n); } @@ -16273,6 +36001,53 @@ static void svc_opdef_reset_all(void) * once per service iteration: deliver forward-only to the still-live owner * channel, count a LOSS when the owner is gone. */ +/* + * r563b: forced variant of the expiry sweep for the stall self-heal. The + * regular sweep skips slots younger than SVC_OPDEF_HOLD_MS; when the pump + * has already been stuck for SVC_STALL_WATCH_MS, age no longer matters - + * deliver every withheld release forward-only (recording the delivered + * marks so a re-decode cannot re-hold) and clear the slots. This is the + * same delivery the slot's own 3 s bound would perform, just executed at + * the watchdog's discretion. + */ +static void svc_opdef_force_flush(void) +{ + int i; + + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + u8 k; + + if (!e->active || !e->n) + continue; + for (k = 0; k < e->n; k++) + svc_opdef_done_note(e->h_object, e->entry_idx, + &e->rel[k]); + { + static struct svc_channel ch_scratch; + unsigned long flags; + struct svc_channel *local = NULL; + int c; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + ch_scratch = svc.chan[c]; + local = &ch_scratch; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + if (local) + svc_fwd_rel_apply_all(e->vgpu, local, + e->rel, e->n); + } + svc.opdef_expired++; + svc_opdef_clear(e, SVC_OPDEF_CLR_SWEEP); + } +} + static void svc_opdef_expire_sweep(void) { /* @@ -16306,7 +36081,13 @@ static void svc_opdef_expire_sweep(void) } spin_unlock_irqrestore(&svc.chan_lock, flags); if (local) { + u8 i; + svc_fwd_rel_apply_all(e->vgpu, local, e->rel, e->n); + for (i = 0; i < e->n; i++) + svc_opdef_done_note(e->h_object, + (u32)e->entry_idx, + &e->rel[i]); svc.opdef_expired++; pr_info_ratelimited("sema-opdef: SWEEP-EXPIRED h=0x%x entry=%u n=%u applied forward-only (entry parked past the hold window)\n", e->h_object, e->entry_idx, e->n); @@ -16336,6 +36117,225 @@ static void svc_fwd_hold_direct_note(void) } /* + * SESSION 127 C5C0 translator: pending GRCE copy-out list. + * + * The Vulkan compute client submits its frame work in the pre-built + * continuation beyond GP_PUT: beec0100 idx23 carries the inline-QMD launch + * (0x31c = 0x4b85f, 64-word 0x320 run) and idx24 carries the TURING_DMA_COPY_A + * (0xC5B5, subch 4) image->buffer copies that un-tile the storage image into + * the client's sysmem readback buffer. Both pushes are only visible to the + * sema_gr_ahead sweep (see svc_gr_ahead_push), not to the put-bounded main + * walk. + * + * Copy ordering: the copy must read the image AFTER the forwarded launch + * completed (the engine's output is copied back into the VA shadow on the + * completion edge). The sweep only records the copy; it is executed inside + * svc_fwd_hold_gr_release, i.e. after the backend published done and after + * the copy-out, immediately before the launch's guest release word is + * authored - so the client's fence never observes a pre-render image. + * A bounded fallback run (sweep, when no forward is owed) keeps the list + * from stalling when the launch was refused. + */ +#define SVC_GR_COPY_PEND 64 +struct svc_gr_copy_pend { + bool valid; + u32 h_object; + u64 push_va; + u64 hash; + u64 src; + u64 dst; + u32 bytes; /* per-row byte count */ + u32 rows; + u32 dpitch; + u32 spitch; + u32 dma; /* LAUNCH_DMA word (remap/transfer bits, logging) */ + u32 remap; /* SET_REMAP_COMPONENTS (logging) */ +}; +static struct svc_gr_copy_pend svc_gr_copy_pends[SVC_GR_COPY_PEND]; +static u32 svc_gr_copy_pend_next; +static u64 svc_gr_copy_run_total; +static u64 svc_gr_copy_run_log; + +/* + * SESSION 153 CONT.25: recently-executed copy identities. The sweep + * re-decodes the ring continuation every pass, so an entry it just ran + * would be re-added (the valid-dedup above only covers live entries) and + * re-run forever - with a bounded per-pass run count that pins the runner + * on the same first slots. The content hash changes whenever the driver + * rewrites the push (new frame: the semaphore payload advances), so a + * (push_va, hash, src, dst, bytes, rows) match means "this exact strip + * already ran this frame". + */ +#define SVC_GR_COPY_DONE_N 512 +struct svc_gr_copy_done_rec { + u64 push_va, hash, src, dst; + u32 bytes, rows; +}; +static struct svc_gr_copy_done_rec svc_gr_copy_done[SVC_GR_COPY_DONE_N]; +static u32 svc_gr_copy_done_next; + +static bool svc_gr_copy_done_lookup(u64 push_va, u64 hash, u64 src, + u64 dst, u32 bytes, u32 rows) +{ + u32 i; + + for (i = 0; i < SVC_GR_COPY_DONE_N; i++) + if (svc_gr_copy_done[i].push_va == push_va && + svc_gr_copy_done[i].hash == hash && + svc_gr_copy_done[i].src == src && + svc_gr_copy_done[i].dst == dst && + svc_gr_copy_done[i].bytes == bytes && + svc_gr_copy_done[i].rows == rows) + return true; + return false; +} + +static void svc_gr_copy_done_store(u64 push_va, u64 hash, u64 src, + u64 dst, u32 bytes, u32 rows) +{ + struct svc_gr_copy_done_rec *d = + &svc_gr_copy_done[svc_gr_copy_done_next++ % + SVC_GR_COPY_DONE_N]; + + d->push_va = push_va; + d->hash = hash; + d->src = src; + d->dst = dst; + d->bytes = bytes; + d->rows = rows; +} + +static void svc_gr_copy_pend_add(u32 h_object, u64 push_va, u64 hash, + u64 src, u64 dst, u32 bytes, u32 rows, + u32 dpitch, u32 spitch, u32 dma, u32 remap) +{ + u32 i; + + if (!bytes || !rows || !src || !dst) + return; + if (svc_gr_copy_done_lookup(push_va, hash, src, dst, bytes, rows)) + return; /* this exact strip already ran for this content */ + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + + if (p->valid && p->push_va == push_va && p->src == src && + p->dst == dst && p->bytes == bytes && p->rows == rows) + return; /* already recorded (idempotent re-sweep) */ + } + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + if (p->valid) + continue; /* keep live ones; take a free slot */ + p->valid = true; + p->h_object = h_object; + p->push_va = push_va; + p->hash = hash; + p->src = src; + p->dst = dst; + p->bytes = bytes; + p->rows = rows; + p->dpitch = dpitch ? dpitch : bytes; + p->spitch = spitch ? spitch : bytes; + p->dma = dma; + p->remap = remap; + pr_info("sema-gr: copy pend h=0x%x push=0x%llx src=0x%llx dst=0x%llx row=0x%x rows=%u dpitch=0x%x spitch=0x%x dma=0x%x remap=0x%x\n", + h_object, push_va, src, dst, bytes, rows, p->dpitch, + p->spitch, dma, remap); + return; + } + /* All slots live: overwrite the oldest position. */ + { + struct svc_gr_copy_pend *p = + &svc_gr_copy_pends[svc_gr_copy_pend_next++ % + SVC_GR_COPY_PEND]; + + pr_warn_ratelimited("sema-gr: copy pend overflow (dropping push=0x%llx)\n", + p->push_va); + memset(p, 0, sizeof(*p)); + } +} + +/* + * Execute and retire every pending GRCE copy. Called from the completion + * edge (svc_fwd_hold_gr_release) and, as a bounded fallback, from the sweep + * when no forward is owed. The executor is the same svc_copy() the CE/GCE + * op path uses (handles vidmem shadow sources and sysmem destinations). + */ +/* + * SESSION 153 CONT.25: bounded per-call run count for the pending + * copy-out list. The X DDX readback keeps at most two strips in flight + * (the two scratch halves), so running the whole queue in one sweep pass + * overruns the server's consumption and every frame reads the newest + * strips only. Pace at a few per pass; 0 = unlimited (historical). + */ +static uint gsp_sema_gr_copy_run_max; +module_param_named(sema_gr_copy_run_max, gsp_sema_gr_copy_run_max, uint, 0644); +MODULE_PARM_DESC(sema_gr_copy_run_max, + "Max pending GRCE copy-outs executed per sweep pass (0 = unlimited)"); + +static void svc_gr_copy_pend_run(struct nv_vgpu_device *vgpu) +{ + u32 i, ran = 0; + + for (i = 0; i < SVC_GR_COPY_PEND; i++) { + if (gsp_sema_gr_copy_run_max && + ran >= gsp_sema_gr_copy_run_max) + break; + struct svc_gr_copy_pend *p = &svc_gr_copy_pends[i]; + u64 done = 0; + int rc; + + if (!p->valid) + continue; + ran++; + rc = svc_copy(vgpu, p->dst, p->src, p->bytes, 0, + SVC_APERTURE_SYS, 0, SVC_APERTURE_SYS, &done, + p->rows, p->dpitch, p->spitch, p->h_object, 0, + p->hash); + svc_gr_copy_run_total++; + if (rc == 0 || rc == SVC_COPY_FORWARDED) { + p->valid = false; + svc_gr_copy_done_store(p->push_va, p->hash, p->src, + p->dst, p->bytes, p->rows); + if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy done h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rows=%u rc=%d\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, p->rows, rc); + } + } else if (svc_gr_copy_run_log < 64) { + svc_gr_copy_run_log++; + pr_info("sema-gr: copy retry h=0x%x push=0x%llx dst=0x%llx bytes=0x%llx rc=%d done=0x%llx\n", + p->h_object, p->push_va, p->dst, + (u64)p->bytes * p->rows, rc, done); + } + } +} + +/* + * Is this release VA owned by a live, not-yet-authored launch candidate + * captured from THIS push? (Only the launch's own push's release is + * withheld; a continuation fence entry elsewhere in the ring keeps the + * historical immediate-forward behaviour.) + */ +static bool svc_gr_ahead_owned_rel_va(u64 va, u64 push_va) +{ + u32 ri; + + for (ri = 0; ri < SVC_GR_CAND_RING; ri++) { + if (svc_gr_cand_ring[ri].valid && + !svc_gr_cand_ring[ri].rel_done && + svc_gr_cand_ring[ri].rel_va == va && + svc_gr_cand_ring[ri].push_va == push_va) + return true; + } + return false; +} + +/* * Author a GR launch completion release (the QMD-level semaphore the * physical engine writes when the launch completes) in software, * forward-only. Used when a hold resolves without a successful hardware @@ -16383,9 +36383,30 @@ static void svc_fwd_hold_gr_release(struct nv_vgpu_device *vgpu, { if (!svc_fwd_req.gr) return; + /* + * SESSION 127 C5C0 translator: the launch is complete and its output + * pages were copied back into the VA shadow on this edge - run any + * pending GRCE copy-outs (image -> linear readback buffer) BEFORE + * authoring the client's release word, so the fence can never clear + * on a pre-render image. + */ + svc_gr_copy_pend_run(vgpu); svc_gr_sw_release(vgpu, ch ? ch->h_object : 0, svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload); + /* + * CONT.400: a GR launch completion is an RM event edge, not only a + * memory word. The guest client waits in the RM software event queue + * (NV_ESC_RM_GET_EVENT_DATA), populated only by a GSP 0x1003 + * EVENT_POST_EVENT record. The release word alone leaves that waiter + * polling at its ~1 s retry cadence - the observed 0.2 FPS. Post the + * event AFTER the release is visible so the guest wakes immediately. + */ + if (ch && ch->h_object) { + gsp_rm_post_channel_event_force(vgpu, ch->h_object); + svc.post_hint_until = + jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); + } } /* @@ -16410,6 +36431,87 @@ static void svc_fwd_hold_gr_release(struct nv_vgpu_device *vgpu, * applies the recorded releases AND the QMD word, bounded by the existing * hold timeout), so this check stands down for it. */ + +/* + * r618: synth-fault for a refused/failed GR launch. The r616 experiment + * (manual grfaultpoke on the live r616b matmul wedge - fill `0/5 high + * argument ptr(s)`, `GR submit refused: degenerate data set n_data=2 + * (min 16)`, sw release authored, client spun in cudaDeviceSynchronize + * forever) proved the edge: the channel error-notifier write + * (status=0xffff + channel event) ends the client's wait with + * `CUDA_ERROR unspecified launch failure` in seconds. Deliver the same + * edge automatically on every FAILED forward so a refused launch is an + * observable error, never an unbounded hang. One-line rollback: + * gsp_sema_refusal_fault=0. + */ +bool gsp_sema_refusal_fault = true; +module_param_named(sema_refusal_fault, gsp_sema_refusal_fault, bool, 0644); +MODULE_PARM_DESC(sema_refusal_fault, + "Deliver a synthetic channel error edge on a failed/refused GR forward (default 1; 0 = legacy silent sw release)"); + +/* + * r638: break the CE-watermark/wait-gate mutual citation (see + * svc_wait_held_entry and svc_ce_stream_backlog): a channel whose + * unconsumed head entry is wait-gate held is a consumer-side wait, not a + * producer backlog, and is skipped while the candidate's operands are fully + * resolved and no decoded device write is pending on a data page. Rollback + * to the r636 behaviour: sema_mutual_skip=0. + */ +bool gsp_sema_mutual_skip = true; +module_param_named(sema_mutual_skip, gsp_sema_mutual_skip, bool, 0644); +MODULE_PARM_DESC(sema_mutual_skip, + "Skip a wait-held consumer head in the CE stream-order watermark (default 1; 0 = r636 behaviour)"); + +static void svc_gr_fault_synth(struct nv_vgpu_device *vgpu, u32 h_object) +{ + u8 notif[SVC_NOTIFY_SIZE] = {0}; + u16 status = SVC_NOTIFY_ERROR_STATUS; + u32 info32 = 0x1; /* generic engine exception, as proven live */ + unsigned long flags; + struct svc_channel *ch = NULL; + u64 pa; + u32 as, eng; + int i; + + if (!gsp_sema_refusal_fault || !h_object) + return; + spin_lock_irqsave(&svc.chan_lock, flags); + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].h_object == h_object && + svc.chan[i].notifier_pa) { + ch = &svc.chan[i]; + break; + } + } + if (!ch) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + return; + } + pa = ch->notifier_pa; + as = ch->notifier_as; + eng = ch->engine_type; + ch->notifier_err = true; + ch->notifier_err_until = jiffies + SVC_NOTIFIER_ERR_HOLD; + ch->notifier_err_put = ch->put_prev; + spin_unlock_irqrestore(&svc.chan_lock, flags); + + memcpy(notif + SVC_NOTIFY_INFO32_OFF, &info32, 4); + { + u16 info16 = (u16)eng; + + memcpy(notif + SVC_NOTIFY_INFO16_OFF, &info16, 2); + } + memcpy(notif + SVC_NOTIFY_STATUS_OFF, &status, 2); + if (svc_write_ap(vgpu, pa, as, notif, sizeof(notif))) { + pr_info_ratelimited("vgsp-fault: SYNTH-FAULT notifier write failed ch=0x%x pa=0x%llx as=%u\n", + h_object, pa, as); + return; + } + pr_warn("vgsp-fault: SYNTH-FAULT ch=0x%x engine=%u pa=0x%llx status=0x%x except=0x%x (refused forwarded launch -> client-visible error)\n", + h_object, eng, pa, status, info32); + gsp_rm_post_channel_event(vgpu, h_object); +} static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) { bool done = smp_load_acquire(&svc_fwd_req.done); @@ -16425,8 +36527,57 @@ static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) * semaphore and the client's QMD word is still behind (the same edge * the hold's success path covers for held requests). */ - if (svc_fwd_req.hw_rc == 0 && !svc_fwd_req.gr_rel_unobserved) - return; + if (svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT) { + /* + * SESSION 153-CONTINUED-61: MISS-ABORT without a hold - keep + * the candidate armed for a real re-forward and hold the + * client's release back (it completes on the retried + * launch's own edges). Only the expired episode falls + * through to the historical software completion below. + */ + if (svc_gr_missre_arm(NULL, 0)) { + svc_fwd_req.gr_swrel_done = true; + return; + } + } + /* + * SESSION 153-CONTINUED-61: a tick-queued retried launch carries no + * entry hold, so its per-push release records survive in the park + * table (re-parked by svc_gr_missre_tick). Deliver them here on ANY + * final outcome - success, fault or the expired retry episode - so + * the client's tracking words always complete. Exactly-once: the + * park entry is dropped on delivery and the latch below stops any + * later pass from re-running this outcome. + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + static struct svc_channel shim; /* single-threaded service + * path; BSS keeps the kernel + * stack safe */ + u8 pn = 0; + + if (svc_grpark_get(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, parked, &pn)) { + memset(&shim, 0, sizeof(shim)); + shim.valid = true; + shim.h_object = svc_fwd_req.gr_h_object; + svc_gr_copy_pend_run(vgpu); + svc_fwd_rel_apply_all(vgpu, &shim, parked, pn); + svc_grpark_drop(vgpu, svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va); + svc.fwd_rel_applied++; + pr_info("vgsp-gr: MISS-RETRY records delivered h=0x%x push=0x%llx nrel=%u (no-entry retry completion)\n", + svc_fwd_req.gr_h_object, + svc_fwd_req.gr_push_va, (unsigned)pn); + } + } + if (svc_fwd_req.hw_rc == 0) { + svc_gr_missre_done(); + if (!svc_fwd_req.gr_rel_unobserved) { + svc_fwd_req.gr_swrel_done = true; + return; + } + } /* * The outcome is final; latch it first so no later pass, hold edge or * consume tail accounts it again (the release write itself is @@ -16439,14 +36590,47 @@ static void svc_fwd_outcome_check(struct nv_vgpu_device *vgpu) svc_fwd_req.gr_push_va, waited); return; } + /* + * SESSION 127 C5C0 translator: this edge authors the guest release + * without a hold - run any pending GRCE copy-outs first, exactly like + * the hold's success edge, so the fence cannot clear on a pre-render + * image. + */ + svc_gr_copy_pend_run(vgpu); svc_gr_sw_release(vgpu, svc_fwd_req.gr_h_object, svc_fwd_req.gr_rel_va, svc_fwd_req.gr_rel_payload); svc.fwd_gr_stall_swrel++; - if (svc_fwd_req.hw_rc != 0) + /* CONT.400: wake the RM-event waiter on this failure edge too. */ + if (svc_fwd_req.gr_h_object) { + gsp_rm_post_channel_event_force(vgpu, svc_fwd_req.gr_h_object); + svc.post_hint_until = + jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); + } + if (svc_fwd_req.hw_rc == -EBADMSG) { + /* + * SESSION 153-CONTINUED-60: the backend skipped an all-zero 3D + * window (ZEROWIN-SKIP) - nothing was submitted, so there is no + * engine error to report. Author the software release and do + * NOT raise the synthetic fault: the client sees a clean + * completion for a frame whose window had no content, and the + * engine is never handed the zero method stream that used to + * poison it (trap 0x1168 -> grfifo=0xbadf1201, r1107). + */ + pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (zero window skipped; QMD release authored in software, no synth fault)\n", + svc_fwd_req.gr_push_va, waited); + } else if (svc_fwd_req.hw_rc != 0) { pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (failed forwarded launch, rc=%d; QMD release authored in software)\n", svc_fwd_req.gr_push_va, waited, svc_fwd_req.hw_rc); - else + /* + * r618: the software release alone completes only launches the + * engine actually ran; a refused/failed one left the client in + * an unbounded cudaDeviceSynchronize. Deliver the synthetic + * engine-error edge (proven live by the r616 grfaultpoke test) + * so the failed launch is client-observable. + */ + svc_gr_fault_synth(vgpu, svc_fwd_req.gr_h_object); + } else pr_warn("vgsp-gr: GR-STALL-backend va=0x%llx waited_ms=%lu action=proceed (successful launch whose release0 could not target the guest page; QMD release authored in software)\n", svc_fwd_req.gr_push_va, waited); pr_info("vgsp-gr: GR-COMPLETE-SW h=0x%x push=0x%llx rel=0x%llx pay=0x%x waited_ms=%lu n=%llu (backend outcome without a hold)\n", @@ -16522,6 +36706,19 @@ static void svc_fwd_hold_engage(struct nv_vgpu_device *vgpu, svc_fwd_hold.relay_armed = svc_fwd_req.relay; svc_fwd_hold.relay_index = svc_fwd_req.relay_index; svc_fwd_hold.fault_delivered = false; + /* + * r633 diag: the held record set, so a later release-delivery gap + * (the streams2 spin class) can be told apart from a hold that + * engaged empty / with another entry's records. Bounded. + */ + if (svc_grpark_log < 64) { + svc_grpark_log++; + pr_info("vgsp-work: hold engage h=0x%x entry=%u nrel=%u va0=0x%llx pay0=%u gr=%d\n", + ch->h_object, entry_idx, n, + n ? svc_fwd_hold.rel[0].va : 0ull, + n ? svc_fwd_hold.rel[0].payload : 0u, + svc_fwd_req.gr ? 1 : 0); + } /* r420: a fresh hold starts with no WEDGE-QUIET deferral. */ svc_fwd_hold.paused_until = 0; svc_fwd_hold.pause_logged = false; @@ -16697,6 +36894,20 @@ static void svc_fwd_hold_expire_check(struct nv_vgpu_device *vgpu, done = smp_load_acquire(&svc_fwd_req.done); hv = svc_fwd_hold.vgpu ? svc_fwd_hold.vgpu : vgpu; /* + * SESSION 153-CONTINUED-61: a MISS-ABORT hold expiring with the + * outcome known takes the re-arm edge instead of the software + * completion (the same verdict svc_fwd_hold_poll applies); only an + * episode past its bound falls through to the historical authorship + * below. + */ + if (done && svc_fwd_req.gr && + svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT && + svc_gr_missre_arm(svc_fwd_hold.rel, svc_fwd_hold.nrel)) { + svc_fwd_req.gr_swrel_done = true; + svc_fwd_hold_clear(); + return; + } + /* * P6c C3: a relayed hold whose work item already published a timeout * gets the decoded engine fault delivered before the software releases * land (the late-release recheck inside the helper protects against a @@ -16755,6 +36966,32 @@ static void svc_fwd_hold_expire_check(struct nv_vgpu_device *vgpu, * mirrors the normal consume tail; returns true when the entry was consumed. * When the copy is still in flight it returns false and the entry stays. */ +static u32 svc_gr_arm_log; /* CONT.408 bounded epoch-arm witness */ + +/* + * CONT.408: arm the GR publish epoch from a proof-of-live-session edge. + * The channel object is the pump's mutable pass-local copy (the callers + * see it const for read-mostly use); the end-of-pass write-back persists + * the flag. A queued/completed launch whose candidate was captured from + * this channel's own guest ring proves the ring belongs to the live + * session, so the residual stale-epoch watchdog (put > 1 && + * last_get == put && !gr_pub_armed) must not declare a wedge on it. + */ +static void svc_gr_epoch_arm(const struct svc_channel *ch, u64 push_va, + const char *why) +{ + struct svc_channel *m = (struct svc_channel *)ch; + + if (m->engine_type || m->gr_pub_armed) + return; + m->gr_pub_armed = true; + if (svc_gr_arm_log < 64) { + svc_gr_arm_log++; + pr_info("sema-chan: GR epoch armed h=0x%x by %s push=0x%llx (residual watchdog defused)\n", + m->h_object, why, push_va); + } +} + static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, struct svc_channel *ch, u32 put) { @@ -16768,6 +37005,21 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, if (!done && !expired && !aborted) return false; if (done && svc_fwd_req.hw_rc == 0) { + svc_gr_missre_done(); + /* + * CONT.408: a successful forwarded GR launch - including the + * verbatim 3D STREAM replay the X/DDX channel runs - is + * proof this channel's ring belongs to the LIVE session: the + * candidate was captured from the guest's own push and the + * engine executed it. Arm the epoch here so the residual + * stale-epoch watchdog cannot declare a wedge on a channel + * that is demonstrably live (r1704-r1707: the X channel was + * discovered with residual put=2, never dropped, and the + * watchdog fired ~3 s after the last replay while the DDX + * still polled its push buffer). + */ + svc_gr_epoch_arm(ch, svc_fwd_req.gr_push_va, + "forwarded launch"); /* * P6c: on the relay success edge, re-read the guest-visible * semaphore through the software view. Only a word that @@ -16815,6 +37067,23 @@ static bool svc_fwd_hold_poll(struct nv_vgpu_device *vgpu, svc_fwd_hold.nrel); svc.fwd_rel_applied++; } + } else if (done && svc_fwd_req.gr && + svc_fwd_req.hw_rc == GSP_PHYS_GR_RC_MISSABORT && + svc_gr_missre_arm(svc_fwd_hold.rel, svc_fwd_hold.nrel)) { + /* + * SESSION 153-CONTINUED-61: the launch aborted pre-doorbell + * because a window operand's page is still unresolved. Do NOT + * software-complete it: the candidate is re-armed for a real + * re-forward once the pin demand resolves the page (the retry + * tick re-offers it) and the client's releases stay unauthored + * (the retried launch's own edges author them on success). + * The attempt's push-level records were re-parked by the arm. + * Bounded by the episode deadline; an expired episode falls + * into the historical failure path below. gr_swrel_done + * latches the request outcome so the pass-top check cannot + * author the release later. + */ + svc_fwd_req.gr_swrel_done = true; } else { /* * P6b wave 2: failure/timeout/abort fallback - author the @@ -17103,6 +37372,14 @@ static void svc_fwd_gr_work(void) pages[i].guest_va = pg->va; pages[i].hpa = hpa; pages[i].qmd_word = pg->qmd_word; + /* + * r1503 (mission task 1): carry the aperture + device PA to the + * booter so every inline/image write can name its target in the + * decisive receipt (FB offset vs SYS PA). The vidmem window is + * 256 MB, so an FB offset fits the low 30 bits. + */ + pages[i].reserved = (pg->ap == SVC_APERTURE_FB) ? + (0x80000000u | ((u32)pg->pa & 0x3fffffffu)) : 0u; } if (svc_fwd_req.gr_rel_va && @@ -17144,6 +37421,26 @@ static void svc_fwd_gr_work(void) rel_unobserved = true; } svc_fwd_req.gr_rel_unobserved = rel_unobserved; + /* + * r633 diag: the release0 resolution chain, so a redirect + * ("release0 could not target the guest page") names its cause + * (rel_pa unresolved at capture vs HPA unresolved here vs the + * forward-only zeroing). Bounded. + */ + { + static u32 gr_relhpa_log; + + if (gr_relhpa_log < 32) { + gr_relhpa_log++; + pr_info("vgsp-work: gr-rel0 va=0x%llx pa=0x%llx ap=%u hpa=0x%llx off=%u word=%08x pay=%u unobs=%d\n", + svc_fwd_req.gr_rel_va, + svc_fwd_req.gr_rel_pa, + svc_fwd_req.gr_rel_ap, rel_hpa, roff, + rel_hpa ? READ_ONCE(*(u32 *)((u8 *)__va(rel_hpa) + roff)) : 0, + svc_fwd_req.gr_rel_payload, + rel_unobserved ? 1 : 0); + } + } } gs.qmd = svc_fwd_req.gr_qmd; @@ -17169,11 +37466,34 @@ static void svc_fwd_gr_work(void) u64 dhpa = 0; u32 doff = 0; - if (!dg->pa) - continue; - if (svc_fwd_operand_hpa(svc_fwd_req.vgpu, dkind, dg->pa, 0, - SEMA_PAGE_SIZE, &dhpa, &doff) || doff) - continue; + if (dg->shadow) { + /* + * r632: the page was served from the VA shadow lane + * (no resolver and no landing answered for it). The + * shadow page is the page's storage: hand its host + * address to the bounce machinery exactly like a + * resolved guest page. The copy-in stages the client's + * own bytes (H2D'd or zeroed) from it, and the + * copy-out writes the engine's results back into it - + * the same page the D2H copies read through the + * shadow-first rule. Module-owned memory throughout; + * no guest page is involved. + */ + struct svc_vas_rec *vr = svc_vas_find(dg->va); + + if (!vr || !vr->data) + continue; + dhpa = (u64)virt_to_phys(vr->data) & + ~(u64)(SEMA_PAGE_SIZE - 1); + dkind = SVC_APERTURE_SYS; + } else { + if (!dg->pa) + continue; + if (svc_fwd_operand_hpa(svc_fwd_req.vgpu, dkind, + dg->pa, 0, SEMA_PAGE_SIZE, + &dhpa, &doff) || doff) + continue; + } if (gs.n_data >= SVC_GR_DATA_MAX) break; data[gs.n_data].va = dg->va; @@ -17303,6 +37623,30 @@ static void svc_fwd_gr_work(void) gs.raw_words = svc_fwd_req.gr_push_n; gs.raw_qmd_hdr = svc_fwd_req.gr_push_qmd_hdr; gs.raw_qmd_end = svc_fwd_req.gr_push_qmd_end; + /* + * r929 (Path A): a 3D stream capture replays its raw window verbatim; + * the backend emits SET_OBJECT C597 + the window + the release tail + * instead of the compute prelude/QMD. The window is sliced from the + * section's C597 bind; see svc_gl_flush. + */ + if (svc_fwd_req.gr_stream_3d) { + gs.stream = svc_fwd_req.gr_raw + svc_fwd_req.gr_stream_off; + gs.stream_n = svc_fwd_req.gr_stream_n; + gs.stream_rel = svc_fwd_req.gr_stream_rel; + } + gs.prog_va = svc_fwd_req.gr_prog_va; /* CONT-87: family binds */ + gs.n_prog_binds = svc_fwd_req.gr_n_prog_binds; + { + u32 pb; + + for (pb = 0; pb < svc_fwd_req.gr_n_prog_binds && pb < 8u; + pb++) { + gs.prog_bind_va[pb] = + svc_fwd_req.gr_prog_bind_va[pb]; + gs.prog_bind_slot[pb] = + svc_fwd_req.gr_prog_bind_slot[pb]; + } + } gs.h_object = svc_fwd_req.gr_h_object; /* @@ -17600,23 +37944,71 @@ static void svc_fwd_relay_select(struct svc_fwd_req *req) */ static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, u64 *pa_out, u32 *ap_out); -static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) +/* r1497: deliver the sysmem backing of an empty FB fetch page. */ +bool gsp_sema_gr_fetch_sys_into(u64 va, u64 fb_pa, u64 *gpa_out); +bool gsp_sema_gr_fetch_backing(u64 va); +static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word, + bool vas_first) { u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); u64 pa = 0; u32 ap = 0; + u64 vpa = 0; + u32 vap = 0; + bool vas_ok = false; u32 i; /* r372 diagnostic: which branch resolved this launch page. */ const char *src = "?"; if (!page) return; + /* + * r1503 (mission task 1): the emitting channel's own page tables are + * the only truth for its program-family and inline-destination pages. + * The global launch-page chain below is keyed by VA ALONE and prefers + * a live injected pin; every client's RM window reuses the same VAs + * (0x1200xxxxx and the 4xx_xxxx program family), so a pin belonging + * to another client's allocation can answer for this one. That is how + * the program image delivery landed in a scanout/window-buffer page + * (FB 0x18f000, inside the captured window's buffer) instead of the + * client's own program page: the writer was INLINE-APPLY, the VA was + * the program bind VA, and the pin was another VAS's allocation. + * Resolve these pages through the channel VAS first; on a miss the + * historical chain still runs unchanged, so a VA the walk cannot + * reach keeps the proven replay. + */ + if (vas_first && READ_ONCE(svc_cur_vas) && + !svc_vas_lookup_h(vgpu, READ_ONCE(svc_cur_vas), page, + &vpa, &vap)) { + vpa &= ~(u64)(SEMA_PAGE_SIZE - 1); + vas_ok = true; + } for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { if (svc_fwd_req.gr_pages[i].va != page) continue; if (svc_fwd_req.gr_pages[i].qmd_word == SVC_GR_QMD_WORD_NONE && qmd_word != SVC_GR_QMD_WORD_NONE) svc_fwd_req.gr_pages[i].qmd_word = qmd_word; + if (vas_ok) { + u64 gpa = svc_fwd_req.gr_pages[i].pa; + u32 gap = svc_fwd_req.gr_pages[i].ap; + + if (gpa && (gpa != vpa || gap != vap)) { + static u32 vasprio_dedupe_log; + + if (vasprio_dedupe_log < 16u) { + vasprio_dedupe_log++; + pr_info("sema-gr: PAGE-VAS-PRIORITY-DEDUPE qmd=%u va=0x%llx vas=0x%x vas_pa=0x%llx ap=%u was=0x%llx ap=%u (using vas)\n", + qmd_word, page, + READ_ONCE(svc_cur_vas), vpa, + vap, gpa, gap); + } + svc_ledger_store(page, vpa, vap); + svc_pin_store(page, vpa, (int)vap); + } + svc_fwd_req.gr_pages[i].pa = vpa; + svc_fwd_req.gr_pages[i].ap = (u8)vap; + } return; } if (svc_fwd_req.gr_n_pages >= SVC_GR_PAGES_MAX) { @@ -17626,6 +38018,36 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) i = svc_fwd_req.gr_n_pages++; svc_fwd_req.gr_pages[i].va = page; svc_fwd_req.gr_pages[i].qmd_word = qmd_word; + if (vas_ok) { + u64 gpa = 0; + u32 gap = 0; + const char *gsrc = "none"; + + if (!svc_ipin_lookup(page, &gpa, &gap)) + gsrc = "ipin"; + else if (!svc_ledger_lookup(page, &gpa, &gap)) + gsrc = "ledger"; + else if (!svc_gr_src_probe(vgpu, page, &gpa, &gap)) + gsrc = "probe"; + if (gpa && ((gpa & ~(u64)(SEMA_PAGE_SIZE - 1)) != vpa || + gap != vap)) { + static u32 vasprio_log; + + if (vasprio_log < 24u) { + vasprio_log++; + pr_info("sema-gr: PAGE-VAS-PRIORITY qmd=%u va=0x%llx vas=0x%x vas_pa=0x%llx ap=%u %s_pa=0x%llx ap=%u (using vas)\n", + qmd_word, page, READ_ONCE(svc_cur_vas), + vpa, vap, gsrc, + gpa & ~(u64)(SEMA_PAGE_SIZE - 1), + gap); + } + } + svc_fwd_req.gr_pages[i].pa = vpa; + svc_fwd_req.gr_pages[i].ap = (u8)vap; + svc_ledger_store(page, vpa, vap); + svc_pin_store(page, vpa, (int)vap); + return; + } /* * m181 (relocation/ledger unification): the unified landing table is * the authority for a page a host write already landed on. The CE @@ -17719,8 +38141,34 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) goto note_done; } } - if (svc_resolve_va(vgpu, page, &pa, &ap)) - return; + if (svc_resolve_va(vgpu, page, &pa, &ap)) { + /* + * SESSION 127 C5C0 translator (gated): a client-VAS page. The + * Vulkan + * QMD's program, parameter and descriptor pages live in the + * client's own page tables (vidmem aperture) and the trusted + * sources - pins, learned cache, discovered map - do not + * carry them, which made the fill refuse the launch with + * -ENOENT. Resolve through the registered-VAS walk instead: + * the exact path walk_probe proves (r707w). + */ + u64 wpa = 0; + u32 wap = SVC_APERTURE_SYS; + + if (!gsp_sema_gr_xlate || + svc_walk_vaspaces(vgpu, page, &wpa, &wap)) + return; + pa = wpa; + ap = svc_ap_of(wap); + svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); + svc_ledger_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (u32)svc_ap_of(ap)); + svc_pin_store(page, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + (int)svc_ap_of(ap)); + src = "vas"; + goto note_done; + } svc_fwd_req.gr_pages[i].pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_pages[i].ap = (u8)svc_ap_of(ap); /* @@ -17741,6 +38189,29 @@ static void svc_gr_page_note(struct nv_vgpu_device *vgpu, u64 va, u32 qmd_word) src = "walk"; note_done: /* + * r1497: an FB launch page whose emulated BAR1 shadow is empty is a + * page the engine will fetch as zeros (r1496 wall). If the same VA + * also has a sysmem mapping with real bytes, deliver them into the + * shadow page now, before the launch maps/copies it. The fb_pa/ap + * stay unchanged, so every downstream identity (QMD relocation, + * copy-out, ledger) is untouched. + */ + if (svc_fwd_req.gr_pages[i].ap == SVC_APERTURE_FB && + qmd_word != SVC_GR_QMD_WORD_NONE) { + u64 gpa = 0; + + if (gsp_sema_gr_fetch_sys_into(page, + svc_fwd_req.gr_pages[i].pa, + &gpa)) { + svc.fwd_fetch_backing++; + if (svc.fwd_fetch_backing <= 32) + pr_info("sema-gr: FETCH-BACKING note va=0x%llx fb_pa=0x%llx gpa=0x%llx qmd=%u n=%llu\n", + page, svc_fwd_req.gr_pages[i].pa, + gpa, qmd_word, + svc.fwd_fetch_backing); + } + } + /* * r372 diagnostic: the exact (pa, ap) and branch the launch page * resolved through - the value the engine's hpa and the copy-out use. * A D2H readback that lands elsewhere (the sum=0 class) names its @@ -17764,6 +38235,17 @@ static bool svc_gr_page_inline_covers(u64 page) ~(u64)(SEMA_PAGE_SIZE - 1)) == page) return true; } + /* + * SESSION 153-CONTINUED-91: the request's records may have been + * re-rooted onto this launch's bind targets (svc_gr_inline_retarget), + * so a page the request names can be covered by an inline write even + * when the candidate's original record page differs. + */ + for (i = 0; i < svc_fwd_req.gr_n_inline; i++) { + if ((svc_fwd_req.gr_inline[i].dst_va & + ~(u64)(SEMA_PAGE_SIZE - 1)) == page) + return true; + } return false; } @@ -17820,9 +38302,50 @@ static bool svc_gr_page_inline_covers(u64 page) */ #define SVC_GR_DATA_BOUNCE_MAX (GSP_PHYS_RMCALL_GR_DATA_BOUNCE_PAGES / 2u) #define SVC_GR_DATA_BOUNCE_MAX_OLD 205u +/* + * SESSION 153-CONTINUED-63: render-target page tracking (see + * gsp_sema_surf_va_get above). The captured colour target is seeded as a + * tracked data candidate so (a) GR-DEC n_data >= 1, (b) the retire wait + * counts the engine's stores into the target ("N/N pages changed") and + * (c) the copy-out ships the changed pages back to the guest's own screen + * pages. Bounded to one full 1280x720 RGB surface plus slack; the + * completeness record (va + n) arms the booter's rec-over-alias operand + * preference only when EVERY page resolved, so a partial span can never + * hand the engine a base pointer walking past the tracked run. + */ +#define SVC_GR_SURF_TRACK_MAX 1024u +static u64 svc_gr_surf_rec_va; +static u32 svc_gr_surf_rec_n; +static u32 svc_gr_surf_rec_ok; +static u32 svc_gr_surf_rec_log; + +bool gsp_sema_gr_surf_recs(u64 *va, u32 *n) +{ + if (!READ_ONCE(svc_gr_surf_rec_ok)) + return false; + if (va) + *va = svc_gr_surf_rec_va; + if (n) + *n = svc_gr_surf_rec_n; + return true; +} + static u32 svc_gr_data_log; /* m142: bounded per-page attribution log */ static u32 svc_gr_src_log; /* m150: bounded per-candidate source log */ +static u32 svc_gr_shadow_fill_log; /* r632: bounded shadow-serve log */ +/* + * r632 (the red-draw resolution class): caps on SYNTHETIC shadow pages a + * launch may create for argument candidates whose VAs no resolver answers + * (the output-buffer class). Existing shadow content (the client's own H2D + * writes) serves without a quota. The per-candidate cap bounds how far a + * span may guess past a buffer's true end (the first page past the cap ends + * the span through the historical drop path); the per-launch cap keeps the + * bounce window (16384 pages) safe. The real output operands are argument + * candidates and are collected first. + */ +#define SVC_GR_SHADOW_SYNTH_PER_CAND 1024u +#define SVC_GR_SHADOW_SYNTH_MAX 4096u static bool svc_gr_data_stalled; /* m150: high data args unresolved */ /* * m156 (R1): the GR request was refused because a forward was already @@ -17909,6 +38432,193 @@ static const char *svc_gr_src_probe(struct nv_vgpu_device *vgpu, u64 va, } /* + * r1497: fetch-page sysmem backing. + * + * THE WALL (r1496): the active indexed draw's vertex/index buffer resolves + * through the client's own VAS to a vidmem page (FB aperture, low offset) + * but the emulated BAR1 shadow at that offset is entirely zero - the upload + * never passed through an emulated path (no inline write, no CE copy the + * host decoded), so only the real FBMEM has the bytes. The client's object + * is nevertheless also mapped in the channel's page tables with a SYS + * answer (the CPU/staging copy): walking every registered VAS for the same + * VA finds a guest-RAM GPA whose page holds the real vertex/index data + * (r1497 live proof: VA 0x44ff000/0x44fea00 -> GPA 0x124c89000/0x12c77e00, + * first8 of the latter 00000000 3f800000 32f0b842 c0166666 ...). + * + * When the FB shadow page is all-zero, copy that SYS page into the shadow + * page the engine already reads. The launch map, the MME stream line and + * the copy-out keep their existing identities; only the empty page's + * content changes. A shadow page with any byte set is never touched, so + * every working stream and the CB seed keep their current behaviour. + * + * The helper runs in process context (service thread or the GR work item); + * the walks may sleep in vfio_dma_rw exactly like svc_walk_vaspaces. + */ +static bool svc_page_nonzero(const u8 *p, size_t n) +{ + size_t i; + + for (i = 0; i + 8u <= n; i += 8u) { + u64 v; + + memcpy(&v, p + i, 8); + if (v) + return true; + } + for (; i < n; i++) + if (p[i]) + return true; + return false; +} + +static int svc_gr_sys_backing_walk(struct nv_vgpu_device *vgpu, u64 va, + u64 *gpa_out) +{ + struct svc_vaspace local[SVC_MAX_VASPACES]; + unsigned long flags; + unsigned long stamps[SVC_MAX_VASPACES]; + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + u8 tmp[32]; + int i, n = 0; + + /* + * The channel currently being serviced is the most specific answer; + * it is the VAS that produced the FB mapping in the first place, so + * ask it first for the alternate (SYS) encoding of the same VA. + */ + { + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (!svc_chanvas_lookup(vgpu, page, &pa, &ap) && + ap == SVC_APERTURE_SYS && + !svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, tmp, + sizeof(tmp)) && + svc_page_nonzero(tmp, sizeof(tmp))) { + *gpa_out = pa; + return 0; + } + } + + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) { + if (!svc.vas[i].valid) + continue; + local[n] = svc.vas[i]; + stamps[n] = local[n].stamp; + n++; + } + spin_unlock_irqrestore(&svc.vas_lock, flags); + + /* Newest registration first, same authority order as svc_walk_vaspaces. */ + for (i = 1; i < n; i++) { + struct svc_vaspace k = local[i]; + unsigned long ks = stamps[i]; + int j = i - 1; + + while (j >= 0 && time_before(stamps[j], ks)) { + local[j + 1] = local[j]; + stamps[j + 1] = stamps[j]; + j--; + } + local[j + 1] = k; + stamps[j + 1] = ks; + } + + for (i = 0; i < n; i++) { + int alt = local[i].ap == SVC_APERTURE_FB ? SVC_APERTURE_SYS + : SVC_APERTURE_FB; + int taps[2]; + int t; + + taps[0] = local[i].ap; + taps[1] = alt; + for (t = 0; t < 2; t++) { + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (svc_gmmu_walk(vgpu, local[i].pdb, + local[i].entries, page, taps[t], + &pa, &ap, false)) + continue; + if (ap != SVC_APERTURE_SYS) + continue; + if (svc_read_ap(vgpu, pa, SVC_APERTURE_SYS, tmp, + sizeof(tmp))) + continue; + if (!svc_page_nonzero(tmp, sizeof(tmp))) + continue; + *gpa_out = pa; + return 0; + } + } + return -ENOENT; +} + +bool gsp_sema_gr_fetch_sys_into(u64 va, u64 fb_pa, u64 *gpa_out) +{ + struct nv_vgpu_device *vgpu = svc_fwd_req.vgpu; + static u32 back_log; + u64 gpa = 0; + u8 *base, *dst; + u32 wfb[8] = { 0 }, wsys[8] = { 0 }; + u32 k; + + if (!vgpu || !gpa_out) + return false; + base = vgpu->bars[1].vaddr; + if (!base || fb_pa + SEMA_PAGE_SIZE > svc_fb_shadow_size(vgpu)) + return false; + dst = base + fb_pa; + if (svc_page_nonzero(dst, SEMA_PAGE_SIZE)) + return false; + if (svc_gr_sys_backing_walk(vgpu, va, &gpa)) + return false; + /* + * Take the first 8 words of the SYS candidate before the copy so the + * receipt shows both sides of the delivery even when the copy itself + * is what the engine later reads. + */ + if (!svc_read_ap(vgpu, gpa, SVC_APERTURE_SYS, wsys, sizeof(wsys))) + ; + for (k = 0; k < 8u; k++) + memcpy(&wfb[k], dst + (size_t)k * 4u, 4); + if (svc_read_ap(vgpu, gpa, SVC_APERTURE_SYS, dst, SEMA_PAGE_SIZE)) + return false; + *gpa_out = gpa; + if (back_log < 64u) { + back_log++; + pr_info("vgsp-sema: FETCH-SYSBACK va=0x%llx fb_pa=0x%llx gpa=0x%llx pages=1 first8_fb=%08x %08x %08x %08x %08x %08x %08x %08x first8_sys=%08x %08x %08x %08x %08x %08x %08x %08x\n", + va, fb_pa, gpa, + wfb[0], wfb[1], wfb[2], wfb[3], wfb[4], wfb[5], + wfb[6], wfb[7], wsys[0], wsys[1], wsys[2], wsys[3], + wsys[4], wsys[5], wsys[6], wsys[7]); + } + return true; +} + +/* + * Resolver wrapper for callers that only have the VA (the booter's MME + * stream-line builder). Resolve the VA the same way the relocation chain + * does; only an FB answer whose shadow page is empty is a candidate. + */ +bool gsp_sema_gr_fetch_backing(u64 va) +{ + struct nv_vgpu_device *vgpu = svc_fwd_req.vgpu; + u64 pa = 0, gpa = 0; + u32 ap = 0; + + if (!vgpu) + return false; + if (svc_gr_src_probe(vgpu, va & ~(u64)(SEMA_PAGE_SIZE - 1), &pa, &ap)) + return false; + if (ap != SVC_APERTURE_FB) + return false; + return gsp_sema_gr_fetch_sys_into(va, pa & ~(u64)(SEMA_PAGE_SIZE - 1), + &gpa); +} + +/* * relfix PART 3: append one attempt to the resolver's diagnostic chain * ("name:rc,"). Bounded; the tail is simply dropped once full. */ @@ -18064,6 +38774,16 @@ bool gsp_sema_gr_write_guard(u64 va, u64 hpa, const char *path) if (!va || !hpa) return true; + /* + * r632: a VA with a shadow page uses the shadow as its storage (the + * copy paths resolve it there before any other source, and the launch + * path serves its pages from it). The engine's writeback targets + * exactly that page, so a later pin or discovery answer must not skip + * the write - skipping it would strand the engine's output in the + * bounce while the D2H copies still read the shadow. + */ + if (svc_vas_has(va)) + return true; if (!svc_ipin_lookup(page, &ipa, &iap)) { u32 kind = svc_ap_of(iap) == SVC_APERTURE_FB ? SVC_APERTURE_FB : SVC_APERTURE_SYS; @@ -18185,6 +38905,14 @@ static void svc_gr_scan_cand(u64 *wins, u32 *n_cand, const u8 *data, u32 bytes) if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* SESSION 139: float-shaped words are scalars, not device VAs + * (see the scan-site note; r789's 480.0f = 0x43f00000). */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < *n_cand; w++) if (wins[w] == wv) @@ -18269,6 +38997,18 @@ static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) * forever (the same identity bug class as the opdef=955/13/0 * streams2 wedge). The first hold arms; others ride it; a fresh * bound arms once the previous one has passed. + * + * r509: the bound must actually REPORT expiry. The historical arming + * rule re-armed the deadline on the very call that detected expiry + * and then compared against the NEW deadline - so the function could + * never return true, and a data-stalled launch whose arguments never + * resolved held its entry forever (r508 witness: matmul plus ten + * sibling channels in "GR hold ... data args unresolved" with zero + * GR-STALL-hold-bound lines for 10+ minutes; the designed + * retry-forward + consume never ran). The expiry is now one-shot: + * push the deadline, latch gr_hold_bound_hit for the r351 block of + * the same pass, return true. A fresh arm clears the latch (and the + * once-per-launch retry log). */ if (!ch->gr_hold_dead_va || time_after_eq(jiffies, ch->gr_hold_deadline)) { @@ -18276,6 +39016,8 @@ static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) ch->gr_hold_dead_since = jiffies; ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); ch->gr_hold_exp_logged = false; + ch->gr_hold_bound_hit = false; + return false; } /* * r420: pause the per-launch consumption bound while the backend @@ -18298,7 +39040,266 @@ static bool svc_gr_hold_bound_expired(struct svc_channel *ch, u64 launch_va) launch_va); return false; } - return time_after_eq(jiffies, ch->gr_hold_deadline); + if (time_after_eq(jiffies, ch->gr_hold_deadline)) { + /* + * r509: one-shot expiry. Push the deadline so the hold window + * restarts for the caller's retry, latch the verdict for the + * second call site of this pass (the r351 retry-forward + * block), and report expired. + */ + ch->gr_hold_deadline = jiffies + msecs_to_jiffies(cap_ms); + ch->gr_hold_bound_hit = true; + return true; + } + return false; +} + +/* + * r632 (the red-draw resolution class): serve a data page from the VA shadow + * lane when the resolution chain (ipin > resolver > ledger) misses. + * + * The red draw's chain is: a client session whose device VAs no source + * answers (the FB-aperture directory walk returns rc=-2 for every published + * VA space, the pin stream delivers no pair, the discovery re-validation + * fails) - the client's H2D copies then land in the shadow lane (the copy's + * documented fallback), and the launch fill drops every operand, the hold + * expires, the data set is degenerate and the refusal faults the client. + * + * Serving the page from the shadow instead keeps one story across the whole + * pipeline: the copy paths already read/write a shadowed VA in the shadow + * (svc_vas_has is checked before every resolver there), so the fill hands the + * engine the client's own H2D bytes (or a zeroed page for an output the + * client never wrote), and the submit build passes the shadow page's host + * address to the bounce machinery. The engine's writeback lands back in that + * same page, which is exactly what the D2H copies read. The shadow is + * module-owned memory: no guest page is ever touched on this path, and the + * pages are freed with the session like every other shadow page. + * + * Existing shadow content serves unconditionally (the client wrote it). + * A synthetic page (an output buffer no copy has touched) is created only + * for ANCHORED argument candidates - candidates whose span already has at + * least one page the resolvers answered - under both a per-candidate cap + * (the span has no other way to know where the buffer ends - the cap bounds + * the guess and the first page failing the cap ends the span through the + * historical drop path) and a per-launch cap (the bounce window must + * survive). With sema_no_shadow=1 every fallback is refused and the caller + * keeps its historical -ENOENT/drop path. + * + * r633 (the r632 cap validation): the r632 caps were per-FILL-CALL, and the + * fill re-runs on every retry pass of a held launch - one probe=none junk + * candidate could therefore mint another 1024 synthetic pages per pass and + * the 8192-page shadow table filled within seconds (r632 mail: va-shadow + * 8192/8192 with all pages synthetic, from the two 16 MB guessed spans of + * candidates no resolver ever answered). Two changes bound this: + * - the ANCHOR GATE: synthetic pages exist only to extend a span that + * real evidence has already anchored (>= 1 page resolved through + * ipin/resolve/ledger). A candidate whose every page is answerless is + * the historical drop path - its guessed span is not a buffer; + * - the budget is PERSISTED PER LAUNCH EPISODE, keyed by (push VA, span + * base), so the caps hold across the episode's passes, not per call. + * The anchor gate is what keeps the genuinely useful case working: the + * red-draw chain's H2D-shadowed operand pages serve through the CONTENT + * path (unconditionally), and a real buffer's unresolved tail is anchored + * by its resolved head. + */ +#define SVC_GR_SYNTH_EP_SLOTS 16 +static struct svc_gr_synth_ep { + u64 push_va; + u64 base; /* span base; 0 = the launch-level total record */ + u32 n; + /* + * r635: the cross-mint counter (launch-level anchor, see + * svc_gr_shadow_fill_serve). Tracked separately from n so a cross + * mint can never consume the anchored budgets and vice versa. + */ + u32 cross; +} svc_gr_synth_ep[SVC_GR_SYNTH_EP_SLOTS]; +static u32 svc_gr_synth_ep_pos; + +static struct svc_gr_synth_ep *svc_gr_synth_ep_get(u64 push_va, u64 base, + bool create) +{ + u32 i, slot = ~0u; + + if (!push_va) + return NULL; + for (i = 0; i < SVC_GR_SYNTH_EP_SLOTS; i++) { + if (svc_gr_synth_ep[i].push_va == push_va && + svc_gr_synth_ep[i].base == base) + return &svc_gr_synth_ep[i]; + if (slot == ~0u && !svc_gr_synth_ep[i].push_va) + slot = i; + } + if (!create) + return NULL; + if (slot == ~0u) + slot = svc_gr_synth_ep_pos++ % SVC_GR_SYNTH_EP_SLOTS; + memset(&svc_gr_synth_ep[slot], 0, sizeof(svc_gr_synth_ep[slot])); + svc_gr_synth_ep[slot].push_va = push_va; + svc_gr_synth_ep[slot].base = base; + return &svc_gr_synth_ep[slot]; +} + +static u32 svc_gr_synth_ep_launch_n(u64 push_va) +{ + struct svc_gr_synth_ep *lep = + svc_gr_synth_ep_get(push_va, 0, false); + + return lep ? lep->n : 0; +} + +static void svc_gr_synth_ep_reset(void) +{ + memset(svc_gr_synth_ep, 0, sizeof(svc_gr_synth_ep)); + svc_gr_synth_ep_pos = 0; +} + +/* + * r635 (host-side operand resolution without the scanner): the r634 red + * draws measured the remaining shape - a client whose INPUT buffers are + * shadow-CONTENT served (its own H2D bytes, the copy took the shadow lane) + * while its OUTPUT buffer (an argument candidate the client never wrote) + * still failed closed, because the r633 anchor gate only counted a page + * answered through ipin/resolve/ledger and every page of the output span + * was answerless. Two additions, both host-side (no scanner/pin needed): + * + * - a CONTENT-served page anchors its span exactly like a trusted answer: + * the client's own H2D bytes are first-hand truth, and the output tail + * of a contiguous allocation (matmul d_a|d_b|d_c, streams2's buffers) + * then mints synthetic pages through the unchanged per-candidate / + * per-launch budgets instead of dropping; + * - a LAUNCH-LEVEL anchor (cross): once any argument page of the launch + * is served from content or trust, an answerless ARG candidate whose + * span holds nothing at all (a non-contiguous output allocation) may + * still mint a strictly smaller cross-budget page set. This is what + * makes output buffers resolvable regardless of candidate order or + * allocation placement. + * + * Guards kept from r632/r633: only argument candidates mint, budgets are + * per launch episode, and every synthetic mint additionally respects + * SVC_GR_SHADOW_TABLE_GUARD - once the shared session shadow table is + * three-quarters full, no guess may displace the headroom real client + * content (H2D writes) still needs; content serves unconditionally. + */ +#define SVC_GR_SHADOW_NONE 0 +#define SVC_GR_SHADOW_CONTENT 1 +#define SVC_GR_SHADOW_SYNTH 2 +#define SVC_GR_SHADOW_SYNTH_CROSS 3 +/* + * Cross budgets: the r633 proven caps (1024 per candidate / 4096 per + * launch). They apply to CONTENT-anchored spans; the flood class they + * must never re-open (r632: 8192/8192 synthetic pages from answerless + * junk spans) is excluded by the r635 content gate - a green boot has + * no written shadow content at all, so the cross mint is off there, and + * an answerless junk span on a red boot has no head to anchor it. + */ +#define SVC_GR_SHADOW_CROSS_PER_CAND 1024u +#define SVC_GR_SHADOW_CROSS_MAX 4096u +#define SVC_GR_SHADOW_TABLE_GUARD (SVC_VAS_SLOTS * 3u / 4u) + +static int svc_gr_shadow_fill_serve(u64 va, bool is_arg, bool trusted, + bool span_content, bool launch_content, + u64 push_va, u64 base, bool *real_content) +{ + struct svc_gr_synth_ep *ep = NULL; + bool have; + + if (real_content) + *real_content = false; + if (!svc_vas_can_shadow(va)) + return SVC_GR_SHADOW_NONE; + have = svc_vas_has(va); + if (!have) { + if (!is_arg) + return SVC_GR_SHADOW_NONE; + if (trusted) { + /* + * r633 path: a trusted answer (ipin/resolver/ledger) + * anchors the span; the large r633 budgets apply. + */ + if (svc_vas_n >= SVC_GR_SHADOW_TABLE_GUARD) + return SVC_GR_SHADOW_NONE; + ep = svc_gr_synth_ep_get(push_va, base, true); + if (!ep || ep->n >= SVC_GR_SHADOW_SYNTH_PER_CAND) + return SVC_GR_SHADOW_NONE; + if (svc_gr_synth_ep_launch_n(push_va) >= + SVC_GR_SHADOW_SYNTH_MAX) + return SVC_GR_SHADOW_NONE; + } else if (span_content || launch_content) { + /* + * r635 path: the anchor is the client's own written + * content (its H2D took the shadow lane), not a + * resolver answer. The cross budgets apply - smaller, + * because this serves the output tail of a red-draw + * client, not a trusted span. + */ + struct svc_gr_synth_ep *lep; + + if (svc_vas_n >= SVC_GR_SHADOW_TABLE_GUARD) + return SVC_GR_SHADOW_NONE; + ep = svc_gr_synth_ep_get(push_va, base, true); + if (ep && ep->cross >= SVC_GR_SHADOW_CROSS_PER_CAND) + return SVC_GR_SHADOW_NONE; + lep = svc_gr_synth_ep_get(push_va, 0, false); + if (lep && lep->cross >= SVC_GR_SHADOW_CROSS_MAX) + return SVC_GR_SHADOW_NONE; + } else { + return SVC_GR_SHADOW_NONE; + } + } + { + struct svc_vas_rec *r = svc_vas_get(va, true); + + if (!r) + return SVC_GR_SHADOW_NONE; + if (have) { + /* + * r635: real_content tells the caller whether these + * bytes are the client's own WRITES (anchoring + * material) or a synthetic zero page minted by an + * earlier pass (servable, but never an anchor - the + * guard against the self-feeding mint loop). + */ + if (real_content) + *real_content = r->content != 0; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (content%s)\n", + va, r->content ? "" : ", synthetic"); + } + return SVC_GR_SHADOW_CONTENT; + } + } + if (trusted) { + struct svc_gr_synth_ep *lep = + svc_gr_synth_ep_get(push_va, 0, true); + + ep->n++; + if (lep) + lep->n++; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (synthetic; cand=%u launch=%u)\n", + va, ep->n, + svc_gr_synth_ep_launch_n(push_va)); + } + return SVC_GR_SHADOW_SYNTH; + } + { + struct svc_gr_synth_ep *lep = svc_gr_synth_ep_get(push_va, 0, + true); + + if (ep) + ep->cross++; + if (lep) + lep->cross++; + if (svc_gr_shadow_fill_log < 32) { + svc_gr_shadow_fill_log++; + pr_info("sema-svc: gr data page served from the shadow lane va=0x%llx (synthetic cross; cand=%u launch=%u)\n", + va, ep ? ep->cross : 0, lep ? lep->cross : 0); + } + } + return SVC_GR_SHADOW_SYNTH_CROSS; } static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, @@ -18306,6 +39307,15 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, { u64 wins[SVC_GR_CAND_MAX]; bool is_arg[SVC_GR_CAND_MAX]; /* m150: candidate came from the arg scan */ + /* + * r635: launch-level CONTENT anchor for the host-side operand + * fallback (see the pre-pass and svc_gr_shadow_fill_serve). Armed + * only by the client's own WRITTEN shadow content (H2D bytes that + * took the shadow lane) - a resolver/pin/ledger answer is NOT + * enough, so a green boot cannot enable the cross mint; consumed + * by the cross-mint for answerless argument candidates. + */ + bool launch_arg_content = false; u32 n_cand; u32 off = 0, i, w; u8 kind; @@ -18383,6 +39393,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_n_data = 0; svc_fwd_req.gr_dropped = 0; + svc_fwd_req.gr_arg_partial = false; svc_fwd_req.gr_n_data_wins = 0; svc_gr_data_stalled = false; memset(is_arg, 0, sizeof(is_arg)); @@ -18394,6 +39405,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, */ n_cand = 0; /* + * SESSION 153-CONTINUED-63: the captured colour target (the surface + * the replayed draw rasterizes into) becomes a tracked data + * candidate FIRST, so its pages are in the launch's data set even + * when the window/content scans fill every other slot (the beef + * windows' 24 scan candidates are all unresolvable junk). Odd: the + * copy-in stages the guest page into the bounce, the engine's + * stores land in the bounce, the retire wait sees them (GR-DEC + * n_data>=1 -> "N/N pages changed") and the copy-out writes the + * changed pages back to the guest's own pages. Completeness is + * recorded for the booter's rec-over-alias preference. + */ + u64 surf_cand_va = 0; + u32 surf_cand_span = 0; + bool surf_cand_used = false; + + WRITE_ONCE(svc_gr_surf_rec_ok, 0); + if (gsp_sema_surf_va_get(&surf_cand_va, &surf_cand_span)) { + if (surf_cand_span > SVC_GR_SURF_TRACK_MAX) + surf_cand_span = SVC_GR_SURF_TRACK_MAX; + if (surf_cand_va && !(surf_cand_va & (SEMA_PAGE_SIZE - 1)) && + surf_cand_span && n_cand < SVC_GR_CAND_MAX) { + wins[n_cand] = surf_cand_va; + is_arg[n_cand] = false; + n_cand++; + } + } + /* * m121: kernel-argument pointers first. The launched kernel's device * pointers live in the param page (qmd_word 32, c[0x0]) at byte * offsets >= 0x150 (m120: mail_kernel out@0x160, msg@0x168; vadd @@ -18407,18 +39445,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; if (pp->qmd_word != 32u || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } for (k = 0x150u / 8u; k < 256; k++) { u64 v = get_unaligned_le64(page + k * 8); u64 wv; @@ -18426,6 +39479,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -18460,18 +39530,33 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; if (pp->qmd_word != 32u || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } for (k = 0; k < SEMA_PAGE_SIZE / 8; k++) { u64 v, wv; @@ -18481,6 +39566,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -18493,19 +39595,38 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, const struct svc_gr_page *pp = &svc_fwd_req.gr_pages[i]; u64 hpa = 0; const u8 *page; + u8 pbuf[SEMA_PAGE_SIZE]; u32 k; - if (pp->qmd_word < 32 || pp->qmd_word > 47 || + /* SESSION 139: include word 48 (see the r788 finding: the + * word-34 pointer dedupes into the program's 4K page whose + * entry carries qmd_word=48; the old 32..47-even filter left + * that page unscanned). */ + if (pp->qmd_word < 32 || pp->qmd_word > 48 || (pp->qmd_word & 1) || !pp->pa) continue; - kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : - SVC_APERTURE_SYS; - if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, - SEMA_PAGE_SIZE, &hpa, &off) || off) - continue; - page = (const u8 *)__va(hpa); - if (!page) - continue; + /* + * SESSION 140: read the page through the pin-aware reader + * first. The r791 evidence: the QMD page (va 0x4b85000) + * resolves to a vidmem shadow that reads empty, while the + * content the capture decoded comes from the push-buffer + * discovery's pinned staging copy - so a raw host mapping of + * pp->pa sees zeros and the pointer values never become + * candidates. svc_read_va consults ipin/pin/lat before the + * walk and returns the staged bytes. + */ + if (!svc_read_va(vgpu, pp->va, pbuf, sizeof(pbuf))) { + page = pbuf; + } else { + kind = (pp->ap == SVC_APERTURE_FB) ? SVC_APERTURE_FB : + SVC_APERTURE_SYS; + if (svc_fwd_operand_hpa(vgpu, kind, pp->pa, 0, + SEMA_PAGE_SIZE, &hpa, &off) || off) + continue; + page = (const u8 *)__va(hpa); + if (!page) + continue; + } /* * relfix PART 3: whole page, matching the rewrite's scan * range (it walks 0x0..0xfff qword-pairs); the old 0..0x7ff @@ -18518,6 +39639,23 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (v < 0x10000000ULL || v >= (1ULL << 47) || (v & 3)) continue; + /* + * SESSION 139: reject float-shaped words. The launch + * param page carries f32 scalars (r789: 0.2f, 640.0f, + * 480.0f at +0x630) whose bit patterns pass the + * VA-shape filter (0x43f00000 = 480.0f) and were the + * fill's only candidate - an unresolvable "arg base". + * A plausible f32 magnitude has an exponent byte in + * ~[0x6c..0x8f] (2^-19..2^16); device VAs (2^28..2^47) + * as f32 land in subnormal/tiny exponents, so this + * does not reject pointers. + */ + { + u32 fexp = (u32)(v >> 23) & 0xffu; + + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + } wv = v & ~(u64)(SEMA_PAGE_SIZE - 1); for (w = 0; w < n_cand; w++) if (wins[w] == wv) @@ -18529,6 +39667,90 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, * inline-captured bytes as well (see svc_gr_scan_inline). */ svc_gr_scan_inline(wins, &n_cand, pp->va); } + /* + * SESSION 140 (the n_data wiring): seed the candidate set with the + * launch's OWN QMD pointer pairs (words 34..47, already decoded into + * gr_qmd from the push's inline-QMD run). These ARE the launch's + * data-window pointers - the r792 evidence: the capture holds + * ptr[0]=0xdff0febc00 but the fill's page scans read only empty + * shadow pages (the QMD page's vidmem shadow is zero; its content + * never arrives - the invisible-vidmem-write class), so the pointer + * values could never become candidates and n_data stayed 0 (refused + * degenerate). The values need no page scan - use them directly. + */ + { + u32 w2; + + for (w2 = 0; w2 + 1 < 16; w2 += 2) { + u64 pv = (u32)svc_fwd_req.gr_qmd[34 + w2] | + ((u64)(svc_fwd_req.gr_qmd[35 + w2] & + 0x1ffffu) << 32); + u64 pw; + u32 fexp; + + if (pv < 0x10000000ULL || pv >= (1ULL << 47) || + (pv & 3)) + continue; + fexp = (u32)(pv >> 23) & 0xffu; + if (fexp >= 0x6cu && fexp <= 0x8fu) + continue; + pw = pv & ~(u64)(SEMA_PAGE_SIZE - 1); + for (w = 0; w < n_cand; w++) + if (wins[w] == pw) + break; + if (w == n_cand && n_cand < ARRAY_SIZE(wins)) { + wins[n_cand++] = pw; + is_arg[n_cand - 1] = true; + if (svc_gr_pages_log < 64) { + svc_gr_pages_log++; + pr_info("sema-gr: qmd-ptr seed va=0x%llx (word %u) h=0x%x\n", + pw, 34 + w2, + svc_fwd_req.gr_h_object); + } + } + } + } + if (gsp_sema_dbg && svc_gr_pages_log < 48) { + svc_gr_pages_log++; + pr_info("sema-gr: fill cand n=%u pages=%u push=0x%llx\n", + n_cand, svc_fwd_req.gr_n_pages, + svc_fwd_req.gr_push_va); + for (w = 0; w < n_cand && w < 12; w++) + pr_info("sema-gr: win[%u] va=0x%llx arg=%u\n", + w, wins[w], is_arg[w]); + } + /* + * r635: the launch-level CONTENT anchor for the host-side fallback, + * computed BEFORE the span loop so candidate order cannot decide. + * An argument candidate whose span already holds the client's own + * WRITTEN shadow bytes (r->content; a synthetic zero page does not + * count) marks the launch's operands as host-visible; an answerless + * ARG candidate (the output buffer the client never wrote, possibly + * not contiguous with the inputs) may then mint a strictly bounded + * cross-budget page set instead of failing the whole launch closed. + * Only the first SVC_GR_CROSS_ANCHOR_PROBE pages of each span are + * consulted - a real H2D'd buffer always has its head written, and + * the probe is a pure hash lookup. Trusted answers (pin/walk/ledger) + * do NOT arm this: on a green boot the cross mint stays off. + */ +#define SVC_GR_CROSS_ANCHOR_PROBE 64u + { + bool found = false; + + for (w = 0; w < n_cand && !found; w++) { + u32 p; + if (!is_arg[w] || wins[w] < 0x40000000000ULL) + continue; + for (p = 0; p < SVC_GR_CROSS_ANCHOR_PROBE; p++) { + if (svc_vas_has_content(wins[w] + + (u64)p * SEMA_PAGE_SIZE)) { + found = true; + break; + } + } + } + launch_arg_content = found; + } /* m110/B-2: one contiguous run of pages per candidate pointer, * relocated into the active engine-visible data region (grown region * 0x0a000000+ with gsp_phys_gr_data_region=1 - the default - or the @@ -18538,6 +39760,15 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, u64 base = wins[w]; u32 p; u32 before = svc_fwd_req.gr_n_data; + bool cand_anchored = false; /* r633: >= 1 page of this span + * resolved through a trusted + * source (ipin/resolve/ledger); + * the r633 large synthetic + * budgets only apply here */ + bool cand_content = false; /* r635: >= 1 page of this span + * holds the client's own + * WRITTEN shadow bytes; the + * cross budgets extend it */ bool first_unresolved = false; /* m170: span's first page failed */ u64 pba = 0; u32 pba_ap = 0; @@ -18591,6 +39822,28 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, continue; } span = (base >= 0x40000000000ULL) ? per_ptr : 1u; + /* + * SESSION 153-CONTINUED-63: the render-target candidate keeps + * its full surface span (it is a low VA, so the generic span + * rule above would give it ONE page and the engine's raster + * walk would leave the tracked run at the first page + * crossing). surf_cand_used marks the pass so the tail can + * record completeness only for the run that actually filled. + */ + if (surf_cand_va && base == surf_cand_va) { + span = surf_cand_span; + surf_cand_used = true; + } + /* + * r1483: the captured surface's OWN VAS answers its pages + * first (see svc_surf_pre_hvas). The global pin/ledger layers + * are VA-keyed and another client's allocation sits at the + * same 0x1200xxxxx VA, which is how the render-target recs + * became 10 pages of the client's code/operand buffer. + */ + u32 surf_vas_h = (surf_cand_va && base == surf_cand_va && + READ_ONCE(svc_surf_vas_first)) ? + READ_ONCE(svc_surf_pre_hvas) : 0; if (is_arg[w] && base >= 0x40000000000ULL) n_arg_hi++; @@ -18611,6 +39864,10 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, u64 pa = 0; u32 ap = 0, x; struct svc_gr_data *rec; + bool shadow_serve = false; + bool surf_vas_answered = false; /* r1483 */ + int serve = SVC_GR_SHADOW_NONE; + bool real_content = false; /* r635 */ if (va >= (1ULL << 47)) break; @@ -18631,10 +39888,27 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, { u64 ipa = 0; int iap = SVC_APERTURE_SYS; - - if (!svc_ipin_lookup(va, &ipa, &iap)) { + u64 vpa = 0; + int vap = SVC_APERTURE_SYS; + + if (surf_vas_h && + !svc_vas_lookup_h(vgpu, surf_vas_h, va, + &vpa, &vap)) { + static u32 surf_vas_log; + + pa = vpa; + ap = (u32)vap; + cand_anchored = true; + surf_vas_answered = true; + if (surf_vas_log < 8u) { + surf_vas_log++; + pr_info("sema-gr: surf-vas h=0x%x va=0x%llx -> pa=0x%llx ap=%u (captured-surface VAS wins over the VA-keyed pin chain)\n", + surf_vas_h, va, pa, ap); + } + } else if (!svc_ipin_lookup(va, &ipa, &iap)) { pa = ipa; ap = (u32)iap; + cand_anchored = true; } else if (svc_resolve_va(vgpu, va, &pa, &ap)) { /* * C1 (stall/data class): the page may @@ -18653,6 +39927,58 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (!svc_ledger_lookup(va, &lpa, &lap)) { pa = lpa; ap = lap; + cand_anchored = true; + } else if ((serve = svc_gr_shadow_fill_serve( + va, is_arg[w], + cand_anchored, + cand_content, + launch_arg_content, + svc_gr_cand.push_va, + wins[w], + &real_content)) != + SVC_GR_SHADOW_NONE) { + /* + * r632/r633: no resolver, no + * landing - serve the page + * from the shadow lane + * instead of dropping it. + * Existing H2D shadow + * content serves always; a + * SYNTHETIC page only when + * the span is anchored by a + * resolved page and the + * per-episode budgets hold. + * The rec is created with + * pa=0 + shadow=1 below; + * the submit build hands + * the shadow page to the + * bounce machinery. Never + * breaks the span: a + * shadow-served page is a + * resolved page. + */ + shadow_serve = true; + /* + * r635: REAL content (the + * client's own written H2D + * bytes, r->content) anchors + * the span for the cross + * budgets and arms the + * launch-level content + * anchor. A synthetic zero + * page served from an + * earlier pass is NOT an + * anchor (the self-feeding + * mint guard). + */ + if (serve == + SVC_GR_SHADOW_CONTENT && + real_content) { + cand_content = true; + if (is_arg[w]) + launch_arg_content = + true; + } } else { /* * m170 (R4a): the very first @@ -18714,6 +40040,21 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, * page). */ svc_log_stalled_va(va); + if (is_arg[w]) + svc_fwd_req.gr_arg_partial = true; + /* + * SESSION 153-CONTINUED-63: a + * render-target page that does + * not resolve must NOT hold + * the launch (the target is + * an output, not an operand; + * its alias fallback keeps + * today's behaviour), and the + * broken span must not be + * recorded as complete. + */ + if (surf_cand_va && wins[w] == surf_cand_va) + break; if (dropped_pages < 4) { drop_va[dropped_pages] = va; drop_kind[dropped_pages] = @@ -18723,6 +40064,13 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, dropped_pages++; break; } + } else { + /* r633: the trusted resolver answered + * this page - the span is anchored + * (the r633 large budgets apply; + * the r635 cross anchor stays + * content-only). */ + cand_anchored = true; } } /* @@ -18734,7 +40082,7 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, * aperture-aware (a SYS page and an FB page with the * same numeric address are not the same page). */ - if (va >= SVC_VAS_FLOOR) { + if (va >= SVC_VAS_FLOOR && !surf_vas_answered) { u64 lpa = 0; u32 lap = 0; u64 ipa = 0; @@ -18794,10 +40142,13 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, } rec = &svc_fwd_req.gr_data[svc_fwd_req.gr_n_data]; rec->va = va; - rec->pa = pa & ~(u64)(SEMA_PAGE_SIZE - 1); + rec->pa = shadow_serve ? 0 : + (pa & ~(u64)(SEMA_PAGE_SIZE - 1)); rec->reloc_va = reloc_base + ((u64)svc_fwd_req.gr_n_data << 12); - rec->ap = (u8)svc_ap_of(ap); + rec->ap = shadow_serve ? SVC_APERTURE_SYS : + (u8)svc_ap_of(ap); + rec->shadow = shadow_serve ? 1 : 0; svc_fwd_req.gr_n_data++; /* * r349: record the launch's own resolution as the @@ -18811,8 +40162,12 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, * said SYS 0x7c00000; the D2H read followed the * ledger and returned zeros while the correct sum * sat on the written page). + * + * r632: a shadow-served rec has no guest page to + * record - the shadow is its storage, and the copy + * paths already prefer the shadow for these VAs. */ - if (va >= SVC_VAS_FLOOR) + if (!shadow_serve && va >= SVC_VAS_FLOOR) svc_ledger_store(va, rec->pa, rec->ap); /* * m142: bounded per-page attribution log. A wrong data @@ -18822,8 +40177,9 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, */ if (svc_gr_data_log < 192) { svc_gr_data_log++; - pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u\n", - va, rec->pa, rec->ap); + pr_info("vgsp-sema: svc-gr data page va=0x%llx -> pa=0x%llx ap=%u%s\n", + va, rec->pa, rec->ap, + shadow_serve ? " (shadow lane)" : ""); } } (void)base; @@ -18861,7 +40217,14 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, if (is_arg[w] && svc_fwd_req.gr_n_data == before && svc_gr_src_log < 64) { int vi; - char vb[240]; + /* r614 diag: the r613b red-boot post-mortem could not + * tell which registered pdb would have answered an + * unresolved arg base (the 240-byte buffer truncated + * the walk list at three entries, and only the red + * boot's misses were ever printed). A larger buffer + * with a clamped append keeps every registered + * VASpace's walk verdict visible. */ + char vb[512]; int o = 0; svc_gr_src_log++; @@ -18876,11 +40239,12 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, svc.vas[vi].entries, base, svc.vas[vi].ap, &wa, &wap, gsp_sema_walk_trace != 0); - o += snprintf(vb + o, sizeof(vb) - o, - " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", - vi, svc.vas[vi].pdb, - svc.vas[vi].entries, - svc.vas[vi].ap, wrc, wa); + if (o < (int)sizeof(vb) - 1) + o += snprintf(vb + o, sizeof(vb) - o, + " vas[%d] pdb=0x%llx ent=%u ap=%d rc=%d pa=0x%llx", + vi, svc.vas[vi].pdb, + svc.vas[vi].entries, + svc.vas[vi].ap, wrc, wa); } pr_info("vgsp-sema: svc-gr arg-miss base=0x%llx probe=%s n_cand=%u pd_found=%u ipin_n=%u disc=0x%llx/%d%s\n", base, pba_src, n_cand, svc_pd_found, @@ -18913,6 +40277,35 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, (n_arg_hi > 0 && n_arg_hi_res == 0); svc_fwd_req.gr_dropped = dropped_pages; /* + * SESSION 153-CONTINUED-63: record the render-target tracking run - + * only when the pass actually seeded it AND every requested page + * produced a data rec contiguous from the base (the booter then + * rewrites the target's operand pair to the bounce base; a partial + * run falls back to the identity alias so the engine can never walk + * past the tracked pages). + */ + if (surf_cand_used) { + u32 hit = 0; + + for (i = 0; i < svc_fwd_req.gr_n_data; i++) { + u64 rva = svc_fwd_req.gr_data[i].va; + + if (rva >= surf_cand_va && + rva < surf_cand_va + + ((u64)surf_cand_span << 12)) + hit++; + } + if (hit == surf_cand_span) { + svc_gr_surf_rec_va = surf_cand_va; + svc_gr_surf_rec_n = surf_cand_span; + WRITE_ONCE(svc_gr_surf_rec_ok, 1); + } else if (svc_gr_surf_rec_log < 16) { + svc_gr_surf_rec_log++; + pr_info("vgsp-sema: surf-rec incomplete va=0x%llx want=%u got=%u (target falls back to the identity alias)\n", + surf_cand_va, surf_cand_span, hit); + } + } + /* * GRFIX FIX1e (r215): gsp_phys_gr_fill_hold is the master gate. * Default 0 = LOG-ONLY: the fill NEVER holds on dropped pages - the * stalled verdict keeps only the historical pre-8b058884 aggregate @@ -18966,10 +40359,29 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, u32 dl; svc_gr_drop_log_va = svc_fwd_req.gr_push_va; - for (dl = 0; dl < 4 && dl < dropped_pages; dl++) + for (dl = 0; dl < 4 && dl < dropped_pages; dl++) { + /* + * r585: DEMAND the dropped pages through the + * pin channel. The host pin stream parses the + * "stalled VA #N: 0x..." lines (svc_log_stalled_va) + * and pushes each VA to the guest scanner within + * ~3 s, whose pin then lands inside the launch's + * bounded argument-defer window - the r584 matmul + * draw dropped 18 staging pages that were NEVER + * demanded (the GRFILL-DROP line format is + * invisible to the host's demand regex), the r543 + * defer window expired, and the r564 degenerate + * refusal SW-completed a launch that wrote + * nothing (MATMUL bad=65536 csum=0). One demand + * per page per boot (svc_log_stalled_va dedups); + * one episode per launch push VA (the marker + * above). + */ + svc_log_stalled_va(drop_va[dl]); pr_info("vgsp-gr: GRFILL-DROP va=0x%llx kind=%c idx=%u\n", drop_va[dl], drop_kind[dl], drop_idx[dl]); + } } } if (svc_gr_data_stalled) @@ -18987,6 +40399,92 @@ static void svc_gr_data_fill(struct nv_vgpu_device *vgpu, n_cand, svc_fwd_req.gr_n_data, reloc_base); } +static u32 svc_gr_inline_retarget_log; + +/* + * SESSION 153-CONTINUED-91: delivery follows the launch. + * + * The captured program-family inline records (the real-upload payloads and + * the authored stage images) are keyed to the program family of the section + * that captured them, but that family is sticky guest state: the client + * cycles push slots whose content names a different program family per slot + * (observed live: records at 0x4350000/0x4300000 while the launched window + * binds 0x42d0000/0x41c0000/0x4400000). The record pages are noted into the + * launch page set, so the delivery lands on pages no bind names, the + * engine's own bind VAs keep whatever stale content the guest memory holds, + * and the DS latches `Shader Program Header N Error` (r1297: zero + * INLINE-APPLY, fetch page all zeros). + * + * Re-root every record onto the launch's own bind targets by in-page + * offset: the window's own binds (prog_bind_va) first, then the launch's + * program family at the record's own offset (which is also where the + * synthesized j=5/j=2 stage binds sit, family+0x0 / family+0x200). A record + * that continues the previous record's page (the page-crossing tail of a + * split real-upload) follows its head's new target so the split delivery + * stays contiguous. A record already on a build-bind page is left alone; + * the pass is idempotent and a no-op whenever the families already agree. + */ +static void svc_gr_inline_retarget(u64 prog_va, const u64 *binds, u32 n_binds) +{ + u64 lfam = prog_va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 prev_page = 0, prev_new = 0; + u32 i, moved = 0; + + if (!lfam || !svc_gr_cand.n_inline) + return; + for (i = 0; i < svc_gr_cand.n_inline && i < SVC_GR_INLINE_MAX; i++) { + struct svc_gr_inline *inl = &svc_gr_cand.inline_recs[i]; + u64 page = inl->dst_va & ~(u64)(SEMA_PAGE_SIZE - 1); + u64 off = inl->dst_va & (SEMA_PAGE_SIZE - 1); + u64 target = 0; + u32 b; + + if (!page) + continue; + /* Page-crossing tail: follow the head's new location. */ + if (prev_page && page == prev_page + SEMA_PAGE_SIZE && !off) { + u64 nv = (prev_new & ~(u64)(SEMA_PAGE_SIZE - 1)) + + SEMA_PAGE_SIZE; + + if (nv != inl->dst_va) { + inl->dst_va = nv; + moved++; + } + prev_page = page; + prev_new = inl->dst_va; + continue; + } + /* A build bind whose in-page offset matches this record. */ + for (b = 0; b < n_binds && b < SVC_GR_CAND_PROG_BINDS; b++) { + if (binds[b] && + (binds[b] & (SEMA_PAGE_SIZE - 1)) == off) { + target = binds[b]; + break; + } + } + /* Else the launch's own family at the record's offset. */ + if (!target) + target = lfam + off; + if ((target & ~(u64)(SEMA_PAGE_SIZE - 1)) != page) { + inl->dst_va = (target & ~(u64)(SEMA_PAGE_SIZE - 1)) + + (inl->dst_va - page); + moved++; + } + prev_page = page; + prev_new = inl->dst_va; + } + if (moved && svc_gr_inline_retarget_log < 16u) { + svc_gr_inline_retarget_log++; + pr_info("sema-gr: INLINE-RETARGET prog=0x%llx lfam=0x%llx binds=%u moved=%u/%u\n", + prog_va, lfam, n_binds, moved, + svc_gr_cand.n_inline); + for (i = 0; i < svc_gr_cand.n_inline && i < 8u; i++) + pr_info("sema-gr: inl[%u] dst=0x%llx len=%u\n", i, + svc_gr_cand.inline_recs[i].dst_va, + svc_gr_cand.inline_recs[i].len); + } +} + static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, const struct svc_channel *ch) { @@ -19013,6 +40511,23 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_push_n = svc_gr_cand.push_n; svc_fwd_req.gr_push_qmd_hdr = svc_gr_cand.push_qmd_hdr; svc_fwd_req.gr_push_qmd_end = svc_gr_cand.push_qmd_end; + /* r929 (Path A): carry the 3D method-window identity to the backend. */ + svc_fwd_req.gr_stream_3d = svc_gr_cand.stream_3d; + svc_fwd_req.gr_stream_off = svc_gr_cand.stream_off; + svc_fwd_req.gr_stream_n = svc_gr_cand.stream_n; + svc_fwd_req.gr_stream_rel = svc_gr_cand.stream_rel; + svc_fwd_req.gr_n_prog_binds = svc_gr_cand.n_prog_binds; + { + u32 pb; + + for (pb = 0; pb < svc_gr_cand.n_prog_binds && + pb < SVC_GR_CAND_PROG_BINDS; pb++) { + svc_fwd_req.gr_prog_bind_va[pb] = + svc_gr_cand.prog_bind_va[pb]; + svc_fwd_req.gr_prog_bind_slot[pb] = + svc_gr_cand.prog_bind_slot[pb]; + } + } memcpy(svc_fwd_req.gr_raw, svc_gr_cand.raw, sizeof(svc_fwd_req.gr_raw)); svc_fwd_req.gr_qmd_dev_va = svc_gr_cand.qmd_dev_va; @@ -19027,7 +40542,12 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, u64 qpa = 0; u32 qap = SVC_APERTURE_SYS; - if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap)) { + if (!svc_resolve_va(vgpu, svc_gr_cand.qmd_dev_va, &qpa, &qap) || + (gsp_sema_gr_xlate && + !svc_walk_vaspaces(vgpu, + svc_gr_cand.qmd_dev_va & + ~(u64)(SEMA_PAGE_SIZE - 1), + &qpa, &qap))) { svc_fwd_req.gr_qmd_pa = qpa & ~(u64)(SEMA_PAGE_SIZE - 1); svc_fwd_req.gr_qmd_ap = (u8)svc_ap_of(qap); @@ -19075,6 +40595,15 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_param_size = svc_gr_cand.param_size; memcpy(svc_fwd_req.gr_qmd, svc_gr_cand.qmd, sizeof(svc_fwd_req.gr_qmd)); + /* + * SESSION 153-CONTINUED-91: re-root the captured program-family inline + * records onto this launch's own bind targets before they are copied + * into the request (see svc_gr_inline_retarget). The candidate's + * records are retargeted in place so the page-note loop below and + * svc_gr_scan_inline() both follow the delivered VAs. + */ + svc_gr_inline_retarget(svc_gr_cand.prog_va, svc_gr_cand.prog_bind_va, + svc_gr_cand.n_prog_binds); svc_fwd_req.gr_n_inline = svc_gr_cand.n_inline; if (svc_fwd_req.gr_n_inline > SVC_GR_INLINE_MAX) svc_fwd_req.gr_n_inline = SVC_GR_INLINE_MAX; @@ -19084,15 +40613,78 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, svc_fwd_req.gr_page_overflow = false; if (svc_gr_cand.param_va) - svc_gr_page_note(vgpu, svc_gr_cand.param_va, 32); + svc_gr_page_note(vgpu, svc_gr_cand.param_va, 32, false); if (svc_gr_cand.prog_va) - svc_gr_page_note(vgpu, svc_gr_cand.prog_va, 48); + svc_gr_page_note(vgpu, svc_gr_cand.prog_va, 48, true); + /* + * SESSION 140: the QMD page itself carries the launch's pointer + * VALUES (QMD words 32..49 live inside it - the param, the + * pointer-region pairs and the program). Without this note the fill + * never scans it, so the pointer values (r790: word 34 = + * 0xdff0febc00) can never become candidates and n_data stays 0 + * (refused: degenerate data set). Noted under word 34 so the + * 32..48-even bank-page scan covers it. + */ + if (svc_gr_cand.qmd_dev_va) + svc_gr_page_note(vgpu, svc_gr_cand.qmd_dev_va, 34, false); for (i = 0; i < svc_gr_cand.n_pointer; i++) svc_gr_page_note(vgpu, svc_gr_cand.pointer_va[i], - svc_gr_cand.pointer_word[i]); + svc_gr_cand.pointer_word[i], false); for (i = 0; i < svc_gr_cand.n_inline; i++) svc_gr_page_note(vgpu, svc_gr_cand.inline_recs[i].dst_va, - SVC_GR_QMD_WORD_NONE); + SVC_GR_QMD_WORD_NONE, true); + /* + * SESSION 153-CONTINUED-57: attr-prelude operand pages. The captured + * attr/pipeline prelude (see svc_attr_prelude_capture) binds + * SET_PIPELINE_PROGRAM_ADDRESS pairs whose VAs were set in earlier + * windows; the launch builder relocates the emitted pairs to + * shadow-pool VAs - which requires each page to be IN this page set, + * or gsp_phys_rmcall_gr_3d_reloc MISSes and the emitted address stays + * a raw client VA the DS cannot fetch an SPH through. Append them + * (deduped by page inside svc_gr_page_note), soft: an unresolved or + * overflowing page is rolled back so the fill's ENOENT/overflow + * checks cannot refuse the launch over a prelude nicety. + */ + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u) { + u32 apn = 0, ai; + const u64 *aps = gsp_sema_attr_prelude_ptrs(&apn); + static u32 svc_attr_ptr_log; + + for (ai = 0; ai < apn; ai++) { + u32 before = svc_fwd_req.gr_n_pages; + + if (svc_fwd_req.gr_n_pages + 2u >= SVC_GR_PAGES_MAX) + break; + svc_gr_page_note(vgpu, aps[ai], SVC_GR_QMD_WORD_NONE, true); + if (svc_fwd_req.gr_n_pages > before && + !svc_fwd_req.gr_pages[svc_fwd_req.gr_n_pages - 1].pa) + svc_fwd_req.gr_n_pages = before; + else if (svc_attr_ptr_log < 16) { + svc_attr_ptr_log++; + pr_info("sema-gr: attr-prelude page va=0x%llx noted (n_pages=%u)\n", + aps[ai], svc_fwd_req.gr_n_pages); + } + } + } + if (gsp_sema_dbg && svc_gr_pages_log < 24) { + svc_gr_pages_log++; + pr_info("sema-gr: cand h=0x%x param=0x%llx prog=0x%llx n_ptr=%u inline=%u n_pages=%u push=0x%llx\n", + svc_gr_cand.h_object, svc_gr_cand.param_va, + svc_gr_cand.prog_va, svc_gr_cand.n_pointer, + svc_gr_cand.n_inline, svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + for (i = 0; i < svc_gr_cand.n_pointer && i < 8; i++) + pr_info("sema-gr: ptr[%u] va=0x%llx word=%u\n", i, + svc_gr_cand.pointer_va[i], + svc_gr_cand.pointer_word[i]); + for (i = 0; i < svc_fwd_req.gr_n_pages && i < 12; i++) { + const struct svc_gr_page *pg = + &svc_fwd_req.gr_pages[i]; + + pr_info("sema-gr: page[%u] va=0x%llx pa=0x%llx ap=%u qmd_word=%u\n", + i, pg->va, pg->pa, pg->ap, pg->qmd_word); + } + } if (svc_fwd_req.gr_page_overflow || !svc_fwd_req.gr_n_pages) return -E2BIG; for (i = 0; i < svc_fwd_req.gr_n_pages; i++) { @@ -19100,10 +40692,19 @@ static int svc_gr_req_fill(struct nv_vgpu_device *vgpu, if (pg->pa) continue; - if (pg->qmd_word == SVC_GR_QMD_WORD_NONE) - return -ENOENT; - if (!svc_gr_page_inline_covers(pg->va)) + if (pg->qmd_word == SVC_GR_QMD_WORD_NONE || + !svc_gr_page_inline_covers(pg->va)) { + if (svc.fwd_gr_enoe_log < 8) { + svc.fwd_gr_enoe_log++; + pr_info("vgsp-work: GR fill ENOENT va=0x%llx qmd_word=%u inline=%u covered=%d n_pages=%u push=0x%llx\n", + pg->va, pg->qmd_word, + svc_gr_cand.n_inline, + svc_gr_page_inline_covers(pg->va), + svc_fwd_req.gr_n_pages, + svc_gr_cand.push_va); + } return -ENOENT; + } } /* * GRFIX FIX1c: the fill hosts the self-enforcing decision point and @@ -19151,6 +40752,635 @@ static void svc_gr_req_bridge_fallback(struct nv_vgpu_device *vgpu) */ static void svc_handle_map_add(void); +/* + * r543: one bounded deferral per push VA for a partially-resolved argument + * set (see svc_fwd_req.gr_arg_partial). The latch keeps a genuine junk + * pointer word from adding a retry window to every launch: the first time a + * push is seen with a partial argument set it is deferred (pins land within + * ~1-2 s in the healthy case); the same push VA seen again inside the + * window proceeds through the historical forward. A re-used ring slot after + * the window gets its own deferral. + * + * r548: the latch is a bounded per-VA table, not a single slot. A client + * with TWO in-flight launches (victory_saxpy r547: push VAs A and B + * alternating re-decodes) made the single slot ping-pong: every call saw + * "a different VA than last time" and deferred again - both launches + * deferred forever, the client's cudaDeviceSynchronize never completed + * (held launches, zero GR LAUNCH lines, zero GR-COMPLETE). The table gives + * each VA exactly one deferral inside the window; the second sighting of + * any VA proceeds (bounded, cannot hang), interleaved launches included. + */ +#define SVC_GR_ARG_DEFER_MS 30000u +/* + * r554: 64, not 8. The latch must remember every launch push VA of the + * current client (the UVM pool allocates ~8-10 channels and each command + * buffer reuses two launch push slots, so >8 distinct VAs are in rotation); + * with 8 slots each new VA evicted another and the "one deferral per VA" + * guarantee thrashed - r553 soak: victory_reduce re-deferred the same push + * VA (0x20043be64) every pass for 430 s ("partial argument set: 7-8 pages + * unresolved") and never forwarded. 64 covers a full client with margin; + * the table is per-boot tiny (64 * 16 B). + */ +#define SVC_GR_ARG_DEFER_SLOTS 64 +static u64 svc_gr_arg_defer_va[SVC_GR_ARG_DEFER_SLOTS]; +static unsigned long svc_gr_arg_defer_j[SVC_GR_ARG_DEFER_SLOTS]; +static u32 svc_gr_arg_defer_pos; + +static bool svc_gr_arg_defer_ok(u64 push_va) +{ + u32 i; + + for (i = 0; i < SVC_GR_ARG_DEFER_SLOTS; i++) { + if (svc_gr_arg_defer_va[i] == push_va && + time_before(jiffies, svc_gr_arg_defer_j[i] + + msecs_to_jiffies(SVC_GR_ARG_DEFER_MS))) + return false; /* deferred once already: forward */ + } + i = svc_gr_arg_defer_pos++ % SVC_GR_ARG_DEFER_SLOTS; + svc_gr_arg_defer_va[i] = push_va; + svc_gr_arg_defer_j[i] = jiffies; + return true; +} + +/* + * r556: the ABSOLUTE per-launch deferral cap. Every defer path (r541 + * dstpend, r543 partial args, m150 data-stalled hold, busy-defer re-holds) + * is individually bounded, but the bounds interact and re-arm across + * re-decode passes; the r553/r555 boots show clients still walking these + * loops for 400+ s (victory_reduce 430 s, victory_saxpy 254 s at write + * time, work_guard deferred climbing by the thousand). This is the single + * backstop that makes "a launch can never be deferred for more than + * SVC_GR_DEFER_CAP_MS" true regardless of which guard keeps winning: the + * first sighting of a push VA arms an absolute deadline; once it passes, + * svc_gr_defer_expired() returns true and every defer site forwards + * best-effort. 15 s is well above the 0.5-2 s pin latency the deferrals + * exist to cover and well below every client-side timeout (>= 60 s). + */ +#define SVC_GR_DEFER_CAP_MS 15000u +static u64 svc_gr_defer_cap_va[SVC_GR_ARG_DEFER_SLOTS]; +static unsigned long svc_gr_defer_cap_j[SVC_GR_ARG_DEFER_SLOTS]; +static u32 svc_gr_defer_cap_pos; + +static bool svc_gr_defer_expired(u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_ARG_DEFER_SLOTS; i++) { + if (svc_gr_defer_cap_va[i] == push_va) + return !time_before(jiffies, svc_gr_defer_cap_j[i] + + msecs_to_jiffies(SVC_GR_DEFER_CAP_MS)); + } + i = svc_gr_defer_cap_pos++ % SVC_GR_ARG_DEFER_SLOTS; + svc_gr_defer_cap_va[i] = push_va; + svc_gr_defer_cap_j[i] = jiffies; + return false; +} + +/* + * SESSION 153-CONTINUED-57: first-launch prelude-maturity gate. + * + * r1094's second failure mode: the FIRST beef launch of the boot ran with + * `MME-PRELUDE emitted gen=2 words=9` (the client's setup push had not + * become readable yet; the capture matured to 443 words only ~90 s later, + * after the premature launch had already poisoned the lane with MME + * ILLEGAL_OPCODE). The 5 ms sked sampler recorded `mme_esr=80000004 + * mme_mac=0x30` at +5 ms. The mirror of that boot (r1091) read the setup + * push BEFORE the first draw and launched with words=512: MME clean. + * + * This gate withholds a beef forward while the captured MME prelude is + * immature (no macro program yet), so the first window that reaches the + * engine carries the client's real setup. The pump's busy-defer machinery + * holds the entry and retries; once the setup push lands (the setup-hold + * demand/pin path is already armed for it) the capture matures and the + * launch proceeds. Bounded: after sema_gr_prelude_gate_ms the gate stops + * firing for the boot (fail-open, one loud log) so a client that never + * uploads a macro program cannot livelock the forward path. + */ +static unsigned gsp_sema_gr_prelude_gate_ms = 60000u; +module_param_named(sema_gr_prelude_gate_ms, gsp_sema_gr_prelude_gate_ms, + uint, 0644); +MODULE_PARM_DESC(sema_gr_prelude_gate_ms, + "Withhold the first beef GR forwards while the captured MME prelude is immature (no 0x114 macro program yet); fail-open after this many ms (default 60000, 0 = off)"); +static unsigned long svc_gr_prelude_gate_j; +static u64 svc_gr_prelude_gate_defer; +static u64 svc_gr_prelude_gate_exp; + +/* + * SESSION 153-CONTINUED-62: immature-MME skip. The prelude gate above + * HOLDS the forward while the client's macro program is missing, but its + * bounded fail-open window used to forward anyway - and a window whose + * CALL_MME_MACRO runs have no program poisons the engine on contact + * (r1094 and r1114: `MME-PRELUDE emitted gen=1 words=9` -> + * mme_esr=0x80000004 ILLEGAL_OPCODE -> grfifo=0xbadf1201, whole lane + * wedged for the boot). The client needs no engine action to advance + * (the software release lanes author every completion it waits on), so an + * immature-MME beef window is CONSUMED with the software release instead + * of being forwarded into the poison. The moment the setup push lands the + * capture matures and the launches proceed normally. Default 1; A/B with + * 0 (the historical fail-open forward). + */ +static unsigned gsp_sema_mme_immature_skip = 1u; +module_param_named(sema_mme_immature_skip, gsp_sema_mme_immature_skip, + uint, 0644); +MODULE_PARM_DESC(sema_mme_immature_skip, + "Software-complete a beef 3D window instead of forwarding it while the captured MME prelude has no macro program yet (avoids the ILLEGAL_OPCODE engine poison the fail-open forward causes; default 1)"); + +/* + * SESSION 153-CONTINUED-59: attr-program settle gate. The first beat + * window of a capture generation binds program pairs (method 0x2014/0x2054) + * whose operand pages must be IN the launch page set or the builder's + * shadow relocation MISSes and the emitted address stays a raw client VA + * the engine cannot fetch (r1104: `pair addr=0x2054 va=0x4350900 -> MISS` + * -> MMU Fault HUBCLIENT_PD @0x4350000 -> guest channel reset). The pages + * resolve through the pin demand path with a multi-second latency, so a + * launch that runs within a second of the attr prelude's program-VA set + * last growing races that resolution. This gate withholds beef forwards + * for sema_attr_settle_ms after the set last grew (bounded by the same + * gate window as the prelude gate), giving the pins time; the busy-defer + * keeps the candidate so the launch goes out once settled. + */ +static unsigned gsp_sema_attr_settle_ms = 8000u; +module_param_named(sema_attr_settle_ms, gsp_sema_attr_settle_ms, uint, 0644); +MODULE_PARM_DESC(sema_attr_settle_ms, + "Withhold beef GR forwards for this long after the attr prelude program-VA set last grew, so window pair pages resolve before the first launch (default 8000, 0 = off)"); +static unsigned long svc_attr_settle_j; +static u32 svc_attr_settle_n; + +static bool svc_gr_attr_settle_wait(void) +{ + u32 apn = 0; + + if (!gsp_sema_attr_settle_ms) + return false; + (void)gsp_sema_attr_prelude_ptrs(&apn); + if (!apn) + return false; + if (apn != svc_attr_settle_n) { + svc_attr_settle_n = apn; + svc_attr_settle_j = jiffies; + } + if (!svc_gr_prelude_gate_j) + svc_gr_prelude_gate_j = jiffies; + if (time_after(jiffies, svc_gr_prelude_gate_j + + msecs_to_jiffies(gsp_sema_gr_prelude_gate_ms))) + return false; /* overall fail-open with the prelude gate */ + return time_before(jiffies, svc_attr_settle_j + + msecs_to_jiffies(gsp_sema_attr_settle_ms)); +} + +static bool svc_gr_prelude_gate_active(const struct svc_channel *ch) +{ + if ((ch->h_object & 0xfffff000u) != 0xbeef0000u) + return false; + /* + * SESSION 153-CONTINUED-59: attr-program settle first (fires even + * with the MME prelude already mature - that is the r1104 state). + */ + if (svc_gr_attr_settle_wait()) + return true; + if (!READ_ONCE(svc_mme_prelude_flag) || !gsp_sema_gr_prelude_gate_ms) + return false; + if (gsp_sema_mme_prelude_mature()) + return false; + if (!svc_gr_prelude_gate_j) + svc_gr_prelude_gate_j = jiffies; + if (time_after(jiffies, svc_gr_prelude_gate_j + + msecs_to_jiffies(gsp_sema_gr_prelude_gate_ms))) { + if (!svc_gr_prelude_gate_exp) { + svc_gr_prelude_gate_exp = 1; + pr_warn("vgsp-work: prelude gate EXPIRED h=0x%x (MME prelude still immature after %u ms; forwarding anyway)\n", + ch->h_object, gsp_sema_gr_prelude_gate_ms); + } + return false; + } + return true; +} + +/* + * r604: the r543 arg-defer latch and the r556 absolute defer cap are keyed + * by push VA ALONE, and the UVM pool reuses the same launch push VAs for + * every client of a boot - the exact lifetime bug the session flush already + * fixes for the r565 exactly-once memories (svc_gr_cap_hist, svc_gr_seen, + * svc_inline_once: "matches on the VA ALONE ... a previous client's + * capture could classify a later client's fresh launch"). Carried across + * sessions, the r556 stamp arms once - at the previous client's first + * sighting - so a new client's first decision for a reused VA already reads + * defer_expired()=true: every partial-argument defer fails closed (r586 + * path), the engine never launches, the QMD release is authored in + * software, and the client reads stale/zero buffers. Witnessed r603 family + * boot: the first mail client (fresh VAs 0x20043be94/0x20043c364) deferred + * and launched (`GUEST GR LAUNCH OK`), while every later mail client (VAs + * 0x20043beac/0x20043c37c reused from earlier boot runs) saw defer_exp=1 + * at its FIRST decision and `GR forward REFUSED at defer cap ... argument + * pages still unresolved` for both launches; its "green" was the previous + * client's results still sitting in the reused device pages. + * + * Reset both tables with the session: within a session the original + * bounds stand unchanged (one deferred sighting per VA in the arg window, + * one absolute 15 s cap per VA lifetime); a new session starts with fresh + * push generations and must not inherit either stamp. + */ +static void svc_gr_defer_reset(void) +{ + memset(svc_gr_arg_defer_va, 0, sizeof(svc_gr_arg_defer_va)); + memset(svc_gr_arg_defer_j, 0, sizeof(svc_gr_arg_defer_j)); + svc_gr_arg_defer_pos = 0; + memset(svc_gr_defer_cap_va, 0, sizeof(svc_gr_defer_cap_va)); + memset(svc_gr_defer_cap_j, 0, sizeof(svc_gr_defer_cap_j)); + svc_gr_defer_cap_pos = 0; +} + +/* + * r548: is a forward still OWED for this push VA - a captured, never + * offered candidate sitting in the candidate ring? The r547 saxpy stall: + * both launches were deferred by the r541/r543 guards, the candidate ring + * held them as unoffered, and the later passes re-decoded the entries as + * REPLAYS (LAUNCH-SKIP; already captured=1). The replay consume then + * dropped the entries without forwarding them and without authoring any + * completion (no GR LAUNCH, no GR-COMPLETE-SW) - the client's + * cudaDeviceSynchronize waited forever while the pump re-walked the ring. + * The pump's no-capture hold must therefore also fire for a replay whose + * forward is owed, forcing one fresh capture + request per bounded window. + */ +static bool svc_gr_cand_push_owed(u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) + if (svc_gr_cand_ring[i].valid && + !svc_gr_cand_ring[i].offered && + svc_gr_cand_ring[i].push_va == push_va) + return true; + return false; +} + +/* + * r548: last-resort resolution of an owed forward when the bounded + * no-capture hold expires without the launch ever being submitted - the + * client's completion word must not be left to a launch that never ran. + * Author the candidate's QMD release in software (forward-only) and mark + * the candidate consumed so a later pass can never forward it late onto + * refilled buffers (the r59 double-run class). Returns true when a pending + * candidate for this channel+push VA was resolved. + */ +static bool svc_gr_owed_resolve(struct nv_vgpu_device *vgpu, + const struct svc_channel *ch, u64 push_va) +{ + u32 i; + + if (!push_va) + return false; + for (i = 0; i < SVC_GR_CAND_RING; i++) { + struct svc_gr_cand_rec *c = &svc_gr_cand_ring[i]; + + if (!c->valid || c->offered || + c->push_va != push_va || + c->h_object != ch->h_object) + continue; + svc_gr_sw_release(vgpu, ch->h_object, c->rel_va, + c->rel_payload); + c->offered = true; + svc.gr_owed_resolved++; + return true; + } + return false; +} + +/* + * r541: does any of the filled launch's data pages carry an unexecuted + * device write (see svc_dstpend_note)? Runs on the service thread right + * after the fill, so svc_fwd_req.gr_data holds this launch's records. + */ +static bool svc_gr_dstpend_overlap(void) +{ + u32 i; + + for (i = 0; i < svc_fwd_req.gr_n_data; i++) + if (svc_dstpend_hit(svc_fwd_req.gr_data[i].va)) + return true; + return false; +} + +/* + * r568: a CE channel entry with a withheld release is a device write that + * has NOT executed yet. A GR launch must not read any of the client's + * device pages while such a write is pending. The page-level dstpend guard + * cannot cover this when the launch's argument base was dropped + * (GRFILL-DROP): the launch's data set then does not contain the buffer's + * pages, so the overlap check is blind to them - the r571 mail VADD read b + * 1.5 s before the b H2D executed (copy-in 319818.68, b copy 319820.16) and + * computed c = a + 0. Gate the launch on the coarser but complete + * condition: any active opdef slot whose channel is a CE channel means at + * least one copy/fill is unexecuted. Bounded by the same busy-defer + + * absolute 15 s cap machinery as every other guard, so a genuinely stuck + * copy delays a launch by the cap and no more. GR channels are excluded + * (a launch's own withheld release must not defer itself). + */ +static bool svc_copy_pending(void) +{ + unsigned long flags; + bool pending = false; + u32 i, c; + + for (i = 0; i < SVC_OPDEF_SLOTS && !pending; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active || !e->n) + continue; + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) + if (svc.chan[c].valid && + svc.chan[c].vgpu == e->vgpu && + svc.chan[c].h_object == e->h_object) { + if (svc.chan[c].engine_type) + pending = true; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + } + return pending; +} + +/* + * r590: this pass's channel snapshot, so the CE watermark guard can read the + * pump's CURRENT consumption position (local.last_get) instead of the live + * table's previous-pass merge. The pump is single-threaded and walks CE + * channels before GR ones, so a GR decision always sees every CE channel's + * consumed position for this pass. + */ +static struct svc_channel *svc_pass_local; +static u32 svc_pass_local_n; + +/* + * r590: does any ring of this device still hold entries this pass's pump has + * not consumed? Reads the ring's live GP_PUT from its own USERD and compares + * it (modulo the ring size) with the snapshot's last_get. A mismatch means + * the client submitted copies/fills the pump has not decoded yet - exactly + * the stream-order backlog the r541/r568 guards cannot see. + * + * r607 (r606 red-draw capture, final form of r590b): the engine_type==0 + * filter skipped the channel the guard exists for. The driver's H2D + * relocation copies ride a COMPUTE-class channel (h=0x5c000059, eng=0 - + * the r595 finding), and the captured red draw shows the launch forwarding + * while that channel held three published, unconsumed entries behind an + * opdef operand hold: + * 362177.927 GR-DEC push=0x20043c394 dropped=9 partial=1 dstpend=0 + * copypend=0 ceback=0 + * 362178.001 peek-backlog h=0x5c000059 eng=0 put=95 last=92 + * 362182.145 peek-backlog h=0x5c000059 eng=0 put=95 last=92 + * 362182.556 GR LAUNCH submitted (forwarded at the one-shot arg-defer + * retry) -> VADD bad=32735 csum=8139264 (c = a + 0) + * 362182.882 ce-soft-copy va=d_b executed, 326 ms AFTER the forward + * The chain was held by `sema-pend: hold h=0x5c000059 idx=90 ... (operand + * pending)`; the wait group (0x20480fff0 pay=84) passed every pass, so the + * only blindness was the class filter - ceback=0 while the copy channel + * was three entries behind. Consider every other channel of the device; + * the launching channel itself is excluded (its own ring legitimately runs + * ahead with future entries). r590's validated part is unchanged: same + * pass snapshot, same bounded busy-defer, historical forward at the + * absolute cap (this guard itself never refuses; the r586 fail-closed + * path is a separate branch that fires only when argument pages never + * resolve). r595's GR-class ops admission has since cut the hold + * resolution to 0.5-4.7 s (the captured 4.7 s), well inside the 15 s cap. + * + * Channels whose USERD is unreadable are skipped (their work cannot be + * ordered by this mechanism anyway). + * + * r611: only channels that carry transfer ops are considered (ever_ops). + * The r607 blanket "every other ring" caught sibling GR LAUNCH channels + * whose rings hold the client's own future launches: streams2's two stream + * channels (h=0x5c00001f/23) each deferred on the other's published entry + * for the full 15 s cap, then the r586 fail-closed refused both kernels + * (measured r610c: every streams2 launch REFUSED, bad=65534). A launch + * channel decodes no ops (measured r606/r609/r610c: ops appear only on the + * 0x5c000059 copy family and the CE ranks), so the flag separates them + * cleanly while keeping the r606 red-draw copy channel in scope. + */ +static bool svc_ce_stream_backlog(struct nv_vgpu_device *vgpu, + const struct svc_channel *self) +{ + u32 k; + + if (!svc_pass_local || !svc_pass_local_n) + return false; + for (k = 0; k < svc_pass_local_n; k++) { + const struct svc_channel *c = &svc_pass_local[k]; + u32 put = 0; + + if (!c->valid || (vgpu && c->vgpu != vgpu)) + continue; + if (self && c->h_object == self->h_object) + continue; + if (!c->ever_ops) + continue; + if (!c->gpfifo_entries || !c->userd_pa) + continue; + if (svc_read_ap(vgpu ? vgpu : c->vgpu, + c->userd_pa + SVC_USERD_GP_PUT, + c->userd_as, &put, sizeof(put))) + continue; + put %= c->gpfifo_entries; + if (put != ((u32)c->last_get % c->gpfifo_entries)) { + /* + * r638 mutual-cycle gate. The r612/r636 captures + * showed a provable deadlock: a launch L is deferred + * by this watermark because channel X holds an + * unconsumed entry; X's head entry is held by the + * bounded wait gate because its acquire's release is + * authored by L itself, which cannot forward until + * this watermark clears - both halves cite each + * other, and only the 15 s defer cap / the wait's + * fail-open break it (the copy runs ~20 s late; the + * launch finds its operand pages empty). Such a head + * is a consumer-side wait, not a producer backlog: + * skip it and let the forward install the release. + * Bounded by the same guards the watermark's + * consumers already trust: skip only while the + * candidate's operands are fully resolved (no + * stalled fill pages - a launch that still needs + * producer data stays protected) and no decoded + * device write is pending on a data page. Rollback: + * sema_mutual_skip=0. + */ + if (gsp_sema_mutual_skip && + svc_gr_forward_pending() && + !svc_gr_data_stalled && + !svc_gr_dstpend_overlap() && + svc_wait_held_entry(c->h_object, + (u32)c->last_get % c->gpfifo_entries)) { + svc.ce_mutual_skips++; + if (svc.ce_mutual_log < 32) { + svc.ce_mutual_log++; + pr_info("vgsp-work: watermark skip h=0x%x head=%u (head is wait-gate held; forward unblocks its release)\n", + c->h_object, + (u32)c->last_get % + c->gpfifo_entries); + } + continue; + } + return true; + } + } + return false; +} + +/* + * r598 diag: the driver's ring WRITE pointer, not the GPFIFO put. The r568 + * draw needs the copy entries that are already written into the ring but + * whose GP_PUT has not been published yet. When the put says the ring is + * drained, peek at the slots immediately after the consumption position and + * log their {va,len} plus whether the slot already executed that exact + * entry. Bounded per launch, gated by gsp_phys_gr_peek. + */ +u32 gsp_phys_gr_peek; + +static void svc_gr_peek_lookahead(struct nv_vgpu_device *vgpu, + const struct svc_channel *self) +{ + u32 k, logged = 0, backlog_logged = 0; + + if (!gsp_phys_gr_peek || !svc_pass_local) + return; + /* + * r604 diag: 8 was too small. svc_pass_local carries the six CE ranks + * (engine_type 11/12/13) FIRST and the GR-class channels after, so the + * cap was always spent before the copy-carrying GR-class channels + * (0x5c000059 family - the r595 finding: "the driver's relocation / + * H2D copies live on a compute channel") could be logged. Session 48 + * concluded "the copy channel's lookahead read empty" from captures + * where that channel was never in the logged set at all. 24 covers + * every drained channel of a live client with margin; the lookahead + * depth 6 covers the copy behind the launch's own small pushes. + * + * r604: also name every channel the drained-only filter SKIPS because + * its live GP_PUT reads ahead of the pass consumption position. Those + * are exactly the channels the r590 CE watermark watches - but that + * guard skips engine_type==0, so a published backlog on a GR-class + * copy channel is invisible to every existing guard; the capture must + * show it so a red decision can be attributed to write-ahead (put == + * last, entries logged below) vs published-backlog (put != last, + * logged here) without guessing. + */ + for (k = 0; k < svc_pass_local_n && logged < 24; k++) { + const struct svc_channel *c = &svc_pass_local[k]; + u32 put = 0, j; + + if (!c->valid || c->h_object == self->h_object || + !c->gpfifo_va || !c->gpfifo_entries || !c->userd_pa) + continue; + if (svc_read_ap(vgpu, c->userd_pa + SVC_USERD_GP_PUT, + c->userd_as, &put, sizeof(put))) + continue; + put %= c->gpfifo_entries; + if (put != ((u32)c->last_get % c->gpfifo_entries)) { + /* real published backlog */ + if (backlog_logged < 12) { + backlog_logged++; + pr_info("vgsp-gr: peek-backlog h=0x%x eng=%u put=%u last=%u\n", + c->h_object, c->engine_type, put, + (u32)c->last_get); + } + continue; + } + pr_info("vgsp-gr: peek h=0x%x eng=%u put=%u last=%u\n", + c->h_object, c->engine_type, put, (u32)c->last_get); + for (j = 1; j <= 6; j++) { + u32 idx = ((u32)c->last_get + j) % c->gpfifo_entries; + u8 ent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len; + u64 va; + + if (svc_read_va(vgpu, + c->gpfifo_va + + (u64)idx * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | + (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va || !len || len > SVC_PUSH_MAX) + break; + pr_info("vgsp-gr: peek +%u idx=%u va=0x%llx len=%u ident_match=%d\n", + j, idx, va, len, + c->slot_ops_ident[idx] == + ((u64)e0 | ((u64)e1 << 32))); + } + logged++; + } +} + +/* + * r576 DIAG: the GR launch-decision witness (gsp_phys_gr_dec_log=N budget + * per boot). Logged from svc_gr_request right after the fill, for launches + * whose fill saw dropped pages / a partial argument set or whose decision + * point carries a dstpend/copy-pending verdict - i.e. exactly the draws the + * r566/r567/r568 guards can defer or (on their blind spots) forward. The + * data-page list is the operand evidence: each VA -> guest PA the engine + * will read; a wrong-but-valid PA here (vs the page the client's H2D wrote + * for the same VA) is the failure mechanism the readback lottery draws. + */ +unsigned gsp_phys_gr_dec_log_max; +static unsigned svc_gr_dec_log; + +static void svc_gr_dec_dump(const struct svc_channel *ch) +{ + bool dstpend, copypend, ceback; + u32 i, n = 0; + + if (svc_gr_dec_log >= gsp_phys_gr_dec_log_max) + return; + svc_gr_dec_log++; + + dstpend = svc_gr_dstpend_overlap(); + copypend = svc_copy_pending(); + ceback = svc_ce_stream_backlog(NULL, ch); + + pr_info("vgsp-gr: GR-DEC h=0x%x push=0x%llx n_data=%u dropped=%u partial=%d stalled=%d dstpend=%d copypend=%d ceback=%d defer_exp=%d\n", + ch->h_object, svc_gr_cand.push_va, svc_fwd_req.gr_n_data, + svc_fwd_req.gr_dropped, + svc_fwd_req.gr_arg_partial ? 1 : 0, + svc_gr_data_stalled ? 1 : 0, + dstpend ? 1 : 0, copypend ? 1 : 0, + ceback ? 1 : 0, + svc_gr_defer_expired(svc_gr_cand.push_va) ? 1 : 0); + + /* + * r577: raised from 32 to 256 pages - the mail VADD's fill carried + * n_data=969 pages and the red-draw divergence (a wrong-but-similar + * input page) can sit beyond the first 32 pages. + */ + for (i = 0; i < svc_fwd_req.gr_n_data && i < 256; i++) + pr_info("vgsp-gr: GR-DEC data[%u] va=0x%llx pa=0x%llx ap=%u reloc=0x%llx\n", + i, svc_fwd_req.gr_data[i].va, + svc_fwd_req.gr_data[i].pa, svc_fwd_req.gr_data[i].ap, + svc_fwd_req.gr_data[i].reloc_va); + + for (i = 0, n = 0; i < svc_dstpend_n && n < 16; i++) { + if (!svc_dstpend[i].va_page) + continue; + n++; + pr_info("vgsp-gr: GR-DEC dstpend va=0x%llx age_ms=%u\n", + svc_dstpend[i].va_page, + jiffies_to_msecs(jiffies - svc_dstpend[i].since)); + } + + for (i = 0, n = 0; i < SVC_OPDEF_SLOTS && n < 8; i++) { + if (!svc_opdef[i].active || !svc_opdef[i].n) + continue; + n++; + pr_info("vgsp-gr: GR-DEC opdef h=0x%x entry=%u n=%u age_ms=%u\n", + svc_opdef[i].h_object, svc_opdef[i].entry_idx, + svc_opdef[i].n, + jiffies_to_msecs(jiffies - svc_opdef[i].since)); + } +} + static bool svc_gr_request(struct nv_vgpu_device *vgpu, const struct svc_channel *ch) { @@ -19181,11 +41411,25 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, } } if (!gsp_phys_gr_forward_enabled() || !svc_gr_cand.valid || - svc_gr_cand.h_object != ch->h_object) + svc_gr_cand.h_object != ch->h_object) { + if (svc.fwd_gr_nocand_log < 8) { + svc.fwd_gr_nocand_log++; + pr_info("vgsp-work: GR request without candidate h=0x%x (forward=%d cand_valid=%d cand_h=0x%x offered=%d)\n", + ch->h_object, gsp_phys_gr_forward_enabled(), + svc_gr_cand.valid, svc_gr_cand.h_object, + svc_gr_cand.offered); + } return false; + } svc_gr_data_stalled = false; /* m150: only the fill sets this */ svc.fwd_gr_req++; if (!gsp_phys_backend_gr_ready() || !gsp_phys_guest_mem_enabled()) { + svc.fwd_gr_nordy++; + if ((svc.fwd_gr_nordy & 0x3F) == 1) + pr_info("vgsp-work: GR request refused: backend not ready h=0x%x ready=%d gmem=%d n=%llu\n", + ch->h_object, gsp_phys_backend_gr_ready(), + gsp_phys_guest_mem_enabled(), + svc.fwd_gr_nordy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; svc_gr_bridge_fallback(vgpu); @@ -19196,6 +41440,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, * request's done/hw_rc are still being read. The capture falls back. */ if (svc_fwd_hold.active) { + svc.fwd_gr_holdbusy++; + if ((svc.fwd_gr_holdbusy & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: hold active h=0x%x (hold ch=0x%x n=%llu)\n", + ch->h_object, svc_fwd_hold.ch_handle, + svc.fwd_gr_holdbusy); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; /* @@ -19216,6 +41465,11 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, if (svc_fwd_paused || test_and_set_bit_lock(SVC_FWD_INFLIGHT, &svc_fwd_state)) { mutex_unlock(&svc_fwd_lock); + svc.fwd_gr_inflight++; + if ((svc.fwd_gr_inflight & 0x3F) == 1) + pr_info("vgsp-work: GR request busy: inflight h=0x%x (state=0x%lx n=%llu)\n", + ch->h_object, svc_fwd_state, + svc.fwd_gr_inflight); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; if (!gsp_cuda_bridge_enabled()) { @@ -19226,6 +41480,112 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, return false; } /* + * SESSION 153-CONTINUED-57: first-launch prelude-maturity gate (see + * svc_gr_prelude_gate_active). Defer the forward while the captured + * MME setup is incomplete: the candidate survives (busy-defer), the + * pump re-decodes the entry and retries, and the launch goes out + * with the client's real macro program instead of poisoning the + * lane. Skipped for every non-beef channel and after the bounded + * fail-open window. + */ + if (svc_gr_prelude_gate_active(ch)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc_gr_prelude_gate_defer++; + if (svc_gr_prelude_gate_defer <= 16 || + (svc_gr_prelude_gate_defer & 0x3F) == 1) + pr_info("vgsp-work: prelude gate h=0x%x push=0x%llx deferred (MME prelude immature; capture will gate this launch until the setup push lands) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, + svc_gr_prelude_gate_defer); + return false; + } + /* + * r1445 5c: submit-side FE-busy deferral of the load-only carrier. + * A 0 return from the launch builder cannot express this (it is the + * block length and a zero-length block is never publishable), so the + * gate lives here, the same busy-defer shape as the prelude gate + * above: nothing is emitted or submitted, no release is authored, + * the candidate survives and the pump re-offers it. The predicate + * (gsp_phys_gr_mme_fe_defer_wanted) is the builder's own load-only + * decision plus the 0x400504 FE-idle read, bounded per generation so + * a never-idle FE falls back to the historical carrier. + */ + if (svc_gr_cand.stream_3d && + gsp_phys_gr_mme_fe_defer_wanted(ch->h_object, + svc_gr_cand.stream_n)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + return false; + } + /* + * SESSION 153-CONTINUED-62: the gate's fail-open edge (see + * gsp_sema_mme_immature_skip). Past the gate window a beef window + * whose macro program is STILL missing must not be forwarded - the + * engine poisons on the first CALL_MME_MACRO without a program. + * Consume it with the software release instead; the client advances, + * and the lane survives until the setup push matures the capture. + */ + if (READ_ONCE(gsp_sema_mme_immature_skip) && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + READ_ONCE(svc_mme_prelude_flag) && + !gsp_sema_mme_prelude_mature()) { + unsigned hold_ms = + min_t(unsigned, READ_ONCE(gsp_sema_gr_setup_hold_ms), + 2000u); + + /* + * SESSION 155 CONT 8: bounded setup-push draw-window hold. + * Instead of immediately software-completing an immature + * beef window, keep the candidate pending (busy-defer, no + * bridge fallback) while a per-boot deadline of + * sema_gr_setup_hold_ms (clamped <= 2000 ms) is open. The + * pump re-offers the window on every pass and keeps walking + * the held setup-push entry, so a pin/upload that lands + * inside the window matures the capture and the next retry + * forwards the client's real window. Past the deadline the + * historical IMMATURE-SKIP resumes, one loud receipt. + */ + if (hold_ms && !svc_setup_hold_wait_j) + svc_setup_hold_wait_j = jiffies + + msecs_to_jiffies(hold_ms); + if (hold_ms && time_before(jiffies, svc_setup_hold_wait_j)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc_gr_prelude_gate_defer++; + if (svc_gr_prelude_gate_defer <= 24 || + (svc_gr_prelude_gate_defer & 0xFF) == 1) + pr_info("vgsp-work: SETUP-HOLD-DEFER h=0x%x push=0x%llx (MME prelude immature; holding the window for the setup pin, remaining=%u ms) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, + jiffies_to_msecs(svc_setup_hold_wait_j - + jiffies), + svc_gr_prelude_gate_defer); + return false; + } + if (hold_ms && !svc_setup_hold_exp_logged) { + svc_setup_hold_exp_logged = true; + pr_info("vgsp-work: SETUP-HOLD-EXPIRED h=0x%x push=0x%llx (hold=%u ms elapsed, prelude still immature; resuming IMMATURE-SKIP) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, hold_ms, + svc_gr_prelude_gate_defer); + } + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc.fwd_gr_refused++; + svc_gr_prelude_gate_defer++; + if (svc_gr_prelude_gate_defer <= 24 || + (svc_gr_prelude_gate_defer & 0xFF) == 1) + pr_info("vgsp-work: MME-IMMATURE-SKIP h=0x%x push=0x%llx (macro program not captured yet; window software-completed, no engine contact) n=%llu\n", + ch->h_object, svc_gr_cand.push_va, + svc_gr_prelude_gate_defer); + svc_gr_bridge_fallback(vgpu); + return false; + } + /* * m143: drain the host injection queue before resolving the launch's * pages. The watcher's pins for the client's a/b/c buffers can land * between the service loop's map_add drain and this capture; the fill @@ -19242,13 +41602,272 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; svc.fwd_gr_fallback++; - if ((svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) + if (svc.fwd_gr_refused < 8 || + (svc.fwd_gr_fallback & SVC_FWD_LOG_MASK) == 1) pr_info("vgsp-work: GR forward request refused rc=%d h=0x%x push=0x%llx; bridge fallback\n", rc, ch->h_object, svc_gr_cand.push_va); svc_gr_bridge_fallback(vgpu); return false; } /* + * r576 DIAG: witness the decision inputs of the draws the guards + * care about (see svc_gr_dec_dump). + */ + if (gsp_phys_gr_dec_log_max && + (svc_fwd_req.gr_dropped > 0 || svc_fwd_req.gr_arg_partial || + svc_gr_data_stalled || svc_gr_dstpend_overlap() || + svc_copy_pending() || svc_ce_stream_backlog(vgpu, ch))) + svc_gr_dec_dump(ch); + /* + * r541: launch-ordering guard. The fill's data pages are what the + * engine will read; if any of them is the target of a device copy + * that decoded but has NOT executed yet (svc_dstpend_note - the + * smiley/streams2 class: the client's memcpy completed on its + * release while the H2D op was still held, and the launch read the + * PREVIOUS page), do not forward: return the busy-defer verdict so + * the pump holds the entry through its bounded retry window and + * re-captures once the copy has run (or the dstpend window expired, + * which restores the historical forward). Bounded by the same + * busy-defer + gr_hold machinery as every other busy refusal, so it + * can never hang the launch. + */ + if (svc_gr_dstpend_overlap()) { + if (!svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (data page has an unexecuted device write; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + /* + * r583: the absolute defer cap expired while a data page still + * overlaps a pending device write (r581 saxpy live: GR-DEC + * dstpend=1 defer_exp=1 -> forwarded anyway -> bad=524121 + * with dstpend ages 9.5-85 s). Never forward a KNOWN-stale + * operand: fail-closed - author the candidate's QMD + * completion in software (forward-only, latched so the + * consume tail cannot author it twice) and refuse the + * engine launch. The client completes; the engine stays + * away from a page whose writer never ran. Witnessed. + */ + svc.fwd_gr_refused++; + svc.fwd_gr_closed++; + if (svc_gr_cand.rel_va) { + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc_gr_cand.rel_done = true; + } + /* + * r633: deliver the parked push-level release records too - a + * fail-closed refusal still completes the client's tracking + * word (forward-only; a record already applied is skipped). + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, + &parked_n)) { + svc_fwd_rel_apply_all(vgpu, ch, parked, + parked_n); + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + } + } + svc_gr_cand.offered = true; + svc_gr_cand.valid = false; + if (svc.fwd_gr_closed_log < 32) { + svc.fwd_gr_closed_log++; + pr_warn("vgsp-work: GR forward REFUSED at defer cap h=0x%x push=0x%llx (data page still has an unexecuted device write; fail-closed, QMD release authored in software)\n", + ch->h_object, svc_gr_cand.push_va); + } + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + return false; /* no busy-defer: the pump consumes; the + * candidate is owned and sw-completed */ + } + /* + * r590: the CE submission watermark. A launch must not be forwarded + * while a CE ring still holds entries the pump has not consumed yet: + * those are the client's stream-ordered copies, and an entry the + * pump has not decoded cannot be seen by the r541 dstpend guard + * (decoded copies only) nor by the r568 opdef gate (decoded holds + * only). The r589 mail VADD drew exactly this: the h_b H2D entries + * sat behind a translation-held CE entry, the pump's last read of the + * ring's GP_PUT was 92 while the guest had already submitted 95, the + * GR channel was processed later in the same pass and the launch was + * forwarded with the b page still zero (c = a + 0). + * + * The watermark is read LIVE (USERD GP_PUT) against THIS pass's + * consumption position (the pump runs CE channels before GR ones, so + * local.last_get already includes everything this pass consumed). + * r588's proxy failed because it compared the live table's stale + * last_get against a put sampled in a different epoch; both sides of + * this comparison come from the same channel and the same pass. + * + * Bounded by the normal busy-defer window; at the absolute cap the + * historical forward stands (witnessed) - the guard delays a launch, + * it never refuses one, so it cannot recreate r588's mail refusal. + */ + svc_gr_peek_lookahead(vgpu, ch); + if (svc_ce_stream_backlog(vgpu, ch)) { + if (!svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (CE ring has unconsumed entries; stream-order watermark)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + svc.fwd_gr_cewm_cap++; + if (svc.fwd_gr_cewm_log < 32) { + svc.fwd_gr_cewm_log++; + pr_warn("vgsp-work: GR forward at defer cap h=0x%x push=0x%llx (CE ring still has unconsumed entries; historical forward)\n", + ch->h_object, svc_gr_cand.push_va); + } + } + /* + * r568: the coarse companion gate - any unexecuted CE write at all + * (see svc_copy_pending). Covers the dropped-argument-base case the + * page-level guard cannot see (the r571 mail b-copy). + */ + if (svc_copy_pending() && + (svc_fwd_req.gr_arg_partial || svc_fwd_req.gr_dropped > 0) && + !svc_gr_defer_expired(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_dstpend++; + if (svc.fwd_gr_dstpend_log < 32) { + svc.fwd_gr_dstpend_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (a CE write is unexecuted; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va); + } + return false; + } + /* + * r543: a partially-resolved ARGUMENT set must not forward either - + * the engine would run with a stale/zero operand (the r542 matmul + * wedge: 2 of 13 high arg bases resolved 0 pages, the fill proceeded + * "fill hold disabled (log-only)", the engine fetched the QMD and + * wrote nothing: NOWRITE-AMBIGUOUS fe_esr=0x80000000, client read + * zeros). Defer once per push VA through the bounded busy-defer + * retry so the pins can land; the latch (svc_gr_arg_defer_ok) keeps + * a genuinely junk pointer word from paying the window on every + * launch. After the retry the historical forward stands - the guard + * delays, it cannot hang. + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + !svc_gr_defer_expired(svc_gr_cand.push_va) && + svc_gr_arg_defer_ok(svc_gr_cand.push_va)) { + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + svc_gr_busy_defer = true; + svc.fwd_gr_refused++; + svc.fwd_gr_argdefer++; + if (svc.fwd_gr_argdefer_log < 32) { + svc.fwd_gr_argdefer_log++; + pr_info("vgsp-work: GR forward deferred h=0x%x push=0x%llx (partial argument set: %u page(s) unresolved; bounded retry)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped); + } + return false; + } + /* + * r586: the defer cap expired with argument pages still unresolved - + * fail-closed like the r583 dstpend cap. The historical path + * forwarded the launch with the partial set and the engine read + * stale/zero operands (r585 boot saxpy: n_data=969 dropped=7 + * partial=1 -> forwarded at the cap -> bad=1047527 csum=130523290; + * r584 matmul: n_data=2 dropped=18 -> r564 degenerate refusal -> + * no-write). Refuse the forward, author the completion in software, + * consume; the dropped pages were already demanded through the pin + * channel (r585) so the retry path had its chance first. + * + * r612: refuse only a DEGENERATE fill - one whose dropped set is at + * least as large as the resolved set (r584's 2 resolved / 18 + * dropped). A launch with a full resolved window (969 resolved / + * 9 dropped) whose unresolved candidates are still unresolved at + * the absolute cap has them stalled in EVERY source for the whole + * window; they are the param page's decoy/foreign pointers (the + * r611 streams2 first-client capture: candidates 0x783299000000 / + * 0x783297000000, `stalled VA` from the first fill, no pin, no + * walk, no ledger, never resolving in the run) - waiting longer + * cannot help, and refusing converts a launch whose real operands + * all resolved into a guaranteed no-write client failure (streams2 + * bad=65534). At the cap the copy-related guards (dstpend, CE + * watermark) have already cleared or the refusal branches above + * win first, so a non-degenerate fill forwards historically, + * loudly witnessed. + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + svc_gr_defer_expired(svc_gr_cand.push_va) && + svc_fwd_req.gr_n_data < svc_fwd_req.gr_dropped) { + svc.fwd_gr_refused++; + svc.fwd_gr_closed++; + if (svc_gr_cand.rel_va) { + svc_gr_sw_release(vgpu, ch->h_object, + svc_gr_cand.rel_va, + svc_gr_cand.rel_payload); + svc_gr_cand.rel_done = true; + } + /* r633: deliver the parked push-level release records too. */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, + &parked_n)) { + svc_fwd_rel_apply_all(vgpu, ch, parked, + parked_n); + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + } + } + svc_gr_cand.offered = true; + svc_gr_cand.valid = false; + if (svc.fwd_gr_closed_log < 32) { + svc.fwd_gr_closed_log++; + pr_warn("vgsp-work: GR forward REFUSED at defer cap h=0x%x push=0x%llx (argument pages still unresolved: %u; fail-closed, QMD release authored in software)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped); + } + clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); + mutex_unlock(&svc_fwd_lock); + return false; /* no busy-defer: the pump consumes; the + * candidate is owned and sw-completed */ + } + /* + * r612: see the refusal note above - a non-degenerate fill at the cap + * forwards historically (witnessed once per budget slot). + */ + if (svc_fwd_req.gr_arg_partial && !svc_gr_data_stalled && + svc_gr_defer_expired(svc_gr_cand.push_va)) { + svc.fwd_gr_argfw++; + if (svc.fwd_gr_argfw_log < 32) { + svc.fwd_gr_argfw_log++; + pr_info("vgsp-work: GR forward at defer cap h=0x%x push=0x%llx (partial argument set: %u dropped of %u resolved; stalled candidates never resolve, forwarding historically)\n", + ch->h_object, svc_gr_cand.push_va, + svc_fwd_req.gr_dropped, + svc_fwd_req.gr_n_data); + } + } + /* * m150: the fill found high argument pointers that no source can * translate (the second-client zeros class). Do NOT forward a launch * whose kernel would dereference unresolved pointers: leave the @@ -19299,7 +41918,8 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, } } } - if (svc_gr_data_stalled) { + if (svc_gr_data_stalled && + !svc_gr_defer_expired(svc_gr_cand.push_va)) { clear_bit_unlock(SVC_FWD_INFLIGHT, &svc_fwd_state); mutex_unlock(&svc_fwd_lock); svc.fwd_gr_refused++; @@ -19311,6 +41931,45 @@ static bool svc_gr_request(struct nv_vgpu_device *vgpu, return false; /* candidate kept: the pump retries the entry */ } /* + * r633: this launch may have been held in an earlier pass whose decode + * recorded its push-level release records - records that are per-decode + * state and were parked at the hold's break. Restore them now so the + * caller's hold-engage holds THIS push's records (a queued launch from + * the capture ring is not accompanied by a re-decode, so the current + * svc_fwd_rel[] can be empty or belong to another entry; the r632 mail + * hang: the vadd push's tracking release never landed and the client + * spun in cudaDeviceSynchronize). + */ + { + static struct svc_fwd_rel parked[SVC_FWD_REL_MAX]; + u8 parked_n = 0; + + if (svc_grpark_get(vgpu, ch->h_object, + svc_gr_cand.push_va, parked, &parked_n)) { + if (parked_n > SVC_FWD_REL_MAX) + parked_n = SVC_FWD_REL_MAX; + memcpy(svc_fwd_rel, parked, sizeof(svc_fwd_rel)); + svc_fwd_rel_n = parked_n; + svc_fwd_defer_armed = true; + svc_grpark_drop(vgpu, ch->h_object, + svc_gr_cand.push_va); + if (svc_grpark_log < 32) { + svc_grpark_log++; + pr_info("vgsp-work: GR parked releases restored h=0x%x push=0x%llx n=%u va0=0x%llx pay0=%u\n", + ch->h_object, svc_gr_cand.push_va, + parked_n, parked[0].va, + parked[0].payload); + } + } + } + /* + * CONT.408: this launch is being queued from a candidate captured + * out of this channel's own guest ring - proof the session is live. + * Arm the publish epoch (defuses the residual stale-epoch watchdog) + * exactly like the fresh-put-drop rewind does. + */ + svc_gr_epoch_arm(ch, svc_gr_cand.push_va, "queued launch"); + /* * The candidate is consumed: the request copy owns it from here, so a * later capture/replay cannot re-offer it and the bridge is only * reached through the request's own latch. @@ -19918,8 +42577,1324 @@ static void svc_gr_stale_wedge_check(struct nv_vgpu_device *vgpu, put, get, ch->submissions, waited, reason); } +/* + * --------------------------------------------------------------------- + * SESSION 126 - FENCE FOLD: channel-anchored sweep of a GR-class + * channel's PRE-BUILT ring continuation. + * + * The Vulkan client's fence waits never clear unaided because the + * driver's kick boundary (GP_PUT) stops at the first batch while the ring + * already carries the continuation entries it pre-built for the next + * submits - and the fence's own INC-5 RELEASE group (observed + * [2005c017][0x45ba090][0][payload 1][0][0x02100001]) lives in one of + * those entries, so the pump's GP_PUT-bounded walk never decodes it + * (r715/r716: put=11, entries 12..47 carried content, no dmesg row ever + * mentioned 0x45ba). The host-side fenceword_assist wrote the wait word + * directly; this fold makes the service author it from the driver's own + * push bytes. + * + * Doctrine: unlike the disabled page-content sweep (svc_apply_page_data, + * off since the false-positive corruption incidents), this sweep is + * CHANNEL-ANCHORED: it only walks entries reachable through the ring's + * validated GPFIFO translation, only accepts entries whose bytes pass the + * strict method-stream gate, and only applies the semaphore release + * groups it finds - forward-only, through the standard pending ledger + * (svc_pending_add). No consumption bookkeeping is touched (last_get, + * put_prev, slot_done_epoch and the replay state stay as the pump left + * them), so a later real kick of the same entries decodes them normally; + * repeated sweeps are idempotent (the pending ledger dedups (va,payload) + * and the apply path is forward-only). + * + * Gated by the same live-epoch arm as the counter publish + * (ch->gr_pub_armed) so a residual ring from a previous session cannot + * author releases before the session switch is proven. + * --------------------------------------------------------------------- + */ +unsigned gsp_sema_gr_ahead_max = 64; +module_param_named(sema_gr_ahead, gsp_sema_gr_ahead_max, uint, 0644); +MODULE_PARM_DESC(sema_gr_ahead, + "GR-class channels: max GPFIFO entries beyond GP_PUT whose channel-anchored pushes are swept for semaphore releases (0=off)"); + + +static u32 svc_gr_ahead_log; +static u64 svc_gr_ahead_entries; +static u64 svc_gr_ahead_rels; +static u64 svc_gr_ahead_skips; +/* SESSION 127: releases withheld because a live candidate owns the word. */ +static u64 svc_gr_ahead_owned; +/* SESSION 127: bounded witness for the sweep-caller's launch request. */ +static u32 svc_gr_ahead_req_log; +/* SESSION 128: inline (LOAD_INLINE_DATA) writes authored by the sweep. */ +static u64 svc_gr_ahead_inline_writes; + +/* + * SESSION 129 util-channel witness: the utility channel's subchannel-2 + * LOAD_INLINE_DATA arm never fired (writes=0 on r737) while the same arm + * fired for in-push payloads on the GR channel - i.e. the failure sits in + * the split (tail-served) shape. This struct carries the per-entry story + * out of the translator for one bounded, deduplicated witness line: + * which read source answered (attempt A = the channel path, 1 = the direct + * registered-VAS walk), whether the strict parse accepted the entry, and + * what the LID arm did (saw/fired/refused a still-zero payload). + */ +struct svc_util_wit { + u32 p_src; /* attempt that won: 0 = channel, 1 = VAS walk, 0xff none */ + u32 pa_ok; /* attempt-A parse (method count) */ + u32 pb_ok; /* attempt-B parse (method count) */ + u32 t_src; /* tail read source: 0 = channel, 1 = walk, 0xff none */ + u32 t_len; /* tail length served */ + u32 t0, t1; /* SESSION 130: first two words of the served tail */ + bool saw_lid; /* a LOAD_INLINE_DATA arm was reached */ + bool fired; /* ... and authored its write */ + bool zero; /* ... but the tail payload was still all-zero (pending) */ + u32 dst; + u32 line_len; + u32 line_lc; + u32 count; +}; +static struct svc_util_wit svc_util_wit; +static u64 svc_util_pending_zero; + +/* + * SESSION 127 C5C0 translator, bounded tail materialization buffer: a + * LOAD_INLINE_DATA run whose payload overruns the entry end is served from + * the next ring entry (the driver splits large inline runs into a methods + * entry plus a dedicated data slot entry). 4 KB covers the observed + * param-bank write (1568 B) in full and the first page of the big uploads. + */ +#define SVC_AHEAD_TAIL_INLINE_MAX 32768 +static u8 svc_ahead_inline_buf[SVC_AHEAD_TAIL_INLINE_MAX]; +static u64 svc_ahead_inline_trunc; + +/* + * SESSION 128: one-copy materialization bound for the utility/CE sweep. + * The engine-side copy-in the sweep replays (a TURING_DMA_COPY_A pitch + * copy from a client-owned staging buffer into the vidmem shadow) moves + * the driver's code/constant uploads; the observed SASS/code-heap class is + * tens of KB, so 4 MB is a generous cap that still refuses a bogus + * geometry before it can walk the shadow. + */ +#define SVC_AHEAD_COPY_IN_MAX (4ULL << 20) + +/* + * SESSION 128: materialize one TURING_DMA_COPY_A pitch copy-in that an + * unprocessed copy-class channel carried - src bytes live in the client's + * staging buffer (guest RAM, resolved through the trusted walk like any + * operand page), dst is the vidmem page the driver staged the upload for + * (code heap / constant store). This is the emulated engine write for the + * copy the real CE would perform; it is once-only keyed by (push_va, + * content hash, dst, bytes) so re-scans of a stable ring are free. + */ +static u64 svc_ahead_copy_in_skip; +static void svc_ahead_copy_in(struct nv_vgpu_device *vgpu, u32 ch_h, + u64 push_va, u64 hash, u64 src_va, u64 dst_va, + u32 row, u32 rows, u32 dma) +{ + static u8 cb[65536]; + u64 bytes = (u64)row * (rows ? rows : 1); + u64 done = 0; + u64 spa = 0, dpa = 0; + int sap = SVC_APERTURE_SYS, dap = SVC_APERTURE_SYS; + int rc = 0; + bool src_ok, dst_ok; + + if (!bytes || src_va < SVC_LOW_REL_VA_LO || src_va >= SEMA_VA_HI || + dst_va < SVC_LOW_REL_VA_LO || dst_va >= SEMA_VA_HI) + return; + /* + * Only the upload direction is materialized: the source must be a + * resolved client page in sysmem/guest RAM (the staging buffer the + * client CPU-wrote), and the destination must be vidmem (resolved to + * the FB aperture) or a not-yet-resolved VA whose bytes the VA + * shadow will hold. A resolved SYS destination is a copy-OUT (the + * engine reading vidmem into a client buffer): replaying it here + * would overwrite the client's buffer with shadow garbage before the + * producing launch ran, so it is skipped - the GR channels' copy-out + * machinery owns that direction. + */ + src_ok = !svc_lookup(vgpu, src_va, &spa, &sap) && sap != SVC_APERTURE_FB; + dst_ok = svc_lookup(vgpu, dst_va, &dpa, &dap) || + dap == SVC_APERTURE_FB; + if (!src_ok || !dst_ok) { + svc_ahead_copy_in_skip++; + if (svc_ahead_copy_in_skip <= 8) + pr_info("sema-utilsweep: copy-in skipped (src_ok=%d dst_ok=%d) push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx\n", + src_ok, dst_ok, push_va, src_va, dst_va, + bytes); + return; + } + if (bytes > SVC_AHEAD_COPY_IN_MAX) { + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-utilsweep: copy-in too large push=0x%llx src=0x%llx dst=0x%llx bytes=0x%llx cap=0x%llx\n", + push_va, src_va, dst_va, bytes, + SVC_AHEAD_COPY_IN_MAX); + return; + } + if (svc_inline_once_seen(push_va, hash, dst_va, (u32)bytes)) + return; + svc_inline_once_store(push_va, hash, dst_va, (u32)bytes); + while (done < bytes) { + u32 chunk = bytes - done > sizeof(cb) ? + (u32)sizeof(cb) : (u32)(bytes - done); + + rc = svc_read_va(vgpu, src_va + done, cb, chunk); + if (rc) { + rc = -EIO; + break; + } + rc = svc_grce_inline_write(vgpu, dst_va + done, cb, chunk, + ch_h); + if (rc) + break; + done += chunk; + } + svc_gr_ahead_inline_writes++; + pr_info("sema-utilsweep: copy-in src=0x%llx -> dst=0x%llx bytes=0x%llx dma=0x%x done=0x%llx rc=%d h=0x%x\n", + src_va, dst_va, bytes, dma, done, rc, ch_h); +} + +/* One 4-byte word of a method stream that may continue into the tail. */ +static u32 svc_ahead_w32(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + u32 off) +{ + u32 v = 0; + + if (off + 4 <= len) + memcpy(&v, buf + off, 4); + else if (tail && off >= len && (u64)(off - len) + 4 <= taillen) + memcpy(&v, tail + (off - len), 4); + return v; +} + +/* + * Strict method-stream parse with a one-entry tail (see the sweep comment). + * A run whose payload overruns this entry's end may be completed from the + * next ring entry; anything else invalid means the page is not a method + * stream (the same false-positive arbiter as svc_stream_full_ok). Returns + * the method count, or 0; *used_tail reports that the tail served a + * trailing data run (nothing after it is parsed). + */ +static u32 svc_ahead_parse(const u8 *buf, u32 len, const u8 *tail, u32 taillen, + bool *used_tail) +{ + u32 off = 0, methods = 0; + + *used_tail = false; + for (;;) { + u32 hdr, op, count; + u64 end; + + if (off + 4 > len) + break; + memcpy(&hdr, buf + off, 4); + if (hdr == 0) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + count = (hdr >> 16) & 0xfff; + switch (op) { + case 1: /* INC_METHOD */ + case 3: /* NON_INC_METHOD */ + case 5: /* ONE_INC */ + if (!count || count > 0xfff) + return 0; + end = (u64)off + 4 + (u64)count * 4; + if (end > len) { + if (!tail || (u64)(end - len) > taillen) + return 0; + *used_tail = true; + off = len; + goto done; + } + off = (u32)end; + break; + case 0: /* GRP0_USE_TERT */ + case 2: /* GRP2_USE_TERT */ + case 4: /* IMMD_DATA_METHOD */ + case 7: /* END_PB_SEGMENT */ + off += 4; + break; + default: + return 0; + } + if (++methods > 8192) + return 0; + } +done: + while (off < len && buf[off] == 0) + off++; + return (methods > 0 && (off == len || *used_tail)) ? methods : 0; +} + +/* + * SESSION 127 C5C0 translator sweep pass over one coherent push. Besides the + * original release extraction (svc_gr_ahead_push, below), it recognizes, for + * pre-built continuation entries: + * + * - the C5C0 inline-QMD window (SET_INLINE_QMD_ADDRESS_A/B + the 64-word + * LOAD_INLINE_QMD_DATA run) and feeds it to the standard svc_qmd_accum + * capture, so the physical GR forward (the CUDA QMD path) fires for a + * launch that never reaches the GP_PUT-bounded main walk - observed + * beec0100 idx23: `0x31c = 0x4b85f` + `0x320 x 64`; + * - GRCE / KEPLER_INLINE_TO_MEMORY_B LOAD_INLINE_DATA runs (op 3 at + * 0x1b4, offsets from the 0x180/0x184/0x188/0x18c group), materialized + * into the VA shadow exactly like the main decode does - observed + * beef0100 idx00 (256 B -> 0x423f000, payload in idx01) and idx08 + * (12288 B -> 0x4448000, payload in idx09), and beec0100 idx23 + * (1568 B -> 0x4b86020, payload in-push); + * - TURING_DMA_COPY_A (0xC5B5, subch >= 2) copy operands + * (0x400/0x404 in, 0x408/0x40c out, 0x410/0x414 pitch, 0x418/0x41c + * line length/count, 0x300 LAUNCH_DMA, 0x700..0x708 remap) recorded as + * pending copy-outs for the completion edge (svc_gr_copy_pend_run) - + * observed beec0100 idx24: image 0xdfd8d60000 -> readback 0x050b0000, + * 640x480 rows. + * + * Returns the number of release groups queued (as before). The buffer must + * pass the strict tail-aware parse first; a page that is not a method + * stream is not processed at all. + */ +static u32 svc_gr_ahead_push(struct nv_vgpu_device *vgpu, struct svc_channel *ch, + const u8 *buf, u32 len, const u8 *tail, + u32 taillen, u64 push_va, bool inline_only) +{ + static struct svc_push_state stq; /* service thread only */ + static u32 dw[66]; /* QMD window words */ + u32 off = 0, nrel = 0, methods, i; + u64 off_in = 0, off_out = 0; + u32 have_in = 0, have_out = 0; + u32 pitch_in = 0, pitch_out = 0; + u32 line_len = 0, line_count = 0; + u32 src_origin_x = 0, src_origin_y = 0; + u32 dst_origin_x = 0, dst_origin_y = 0; + u32 have_src_origin = 0, have_dst_origin = 0; + u32 last_dma = 0, last_remap = 0; + u32 cp_subch = ~0u; + bool used_tail = false; + bool forward = gsp_phys_gr_forward_enabled(); + bool xlate = gsp_sema_gr_xlate; + + if (!xlate) { + /* Historical sweep: releases from exactly-covered entries. */ + tail = NULL; + taillen = 0; + } + methods = svc_ahead_parse(buf, len, tail, taillen, &used_tail); + if (!methods) + return 0; + /* + * r1515: the init-push read worth catching is exactly the entry the + * main decode may skip as stale. Apply the client's C597 fixed-CB + * upload (0x2390 runs) from this translated push as well, so the + * beef channel's setup entry feeds the CB image even on boots where + * only the util/continuation sweep reads it. + */ + svc_gl_cb_upload_scan(vgpu, ch->h_object, buf, len, push_va); + + memset(&stq, 0, sizeof(stq)); + stq.push_va = push_va; + + while (off + 4 <= len) { + u32 hdr, op, addr, count, subch; + u32 w0, w1, w2, w3, w4; + u64 end; + + memcpy(&hdr, buf + off, 4); + if (!hdr) { + off += 4; + continue; + } + op = (hdr >> 29) & 0x7; + addr = (hdr & 0x1fff) << 2; + count = (hdr >> 16) & 0xfff; + subch = (hdr >> 13) & 0x7; + if (op == 0 || op == 2 || op == 4 || op == 7) { + off += 4; + continue; + } + if (op != 1 && op != 3 && op != 5) + break; + if (!count || count > 0xfff) + break; + end = (u64)off + 4 + (u64)count * 4; + /* + * Tail-served trailing run: process from the head word, then + * stop (the tail is a data slot, nothing parses after it). + */ + w0 = svc_ahead_w32(buf, len, tail, taillen, off + 4); + w1 = svc_ahead_w32(buf, len, tail, taillen, off + 8); + w2 = svc_ahead_w32(buf, len, tail, taillen, off + 12); + w3 = svc_ahead_w32(buf, len, tail, taillen, off + 16); + w4 = svc_ahead_w32(buf, len, tail, taillen, off + 20); + + /* + * C5C0 inline-QMD window (mirrors the main decode's intercept): + * only accumulate once the run declares itself a QMD window. + */ + if (xlate && forward && !inline_only && subch >= 1 && + addr >= SVC_QMD_ADDR_A && + addr <= SVC_QMD_DATA + (SVC_QMD_WORDS - 1) * 4 && + (stq.qmd_have || + (addr <= SVC_QMD_ADDR_B && + (u64)addr + (u64)count * 4 > SVC_QMD_ADDR_A))) { + u32 n = count > 66 ? 66 : count; + + for (i = 0; i < n; i++) + dw[i] = svc_ahead_w32(buf, len, tail, taillen, + off + 4 + 4 * i); + svc_qmd_accum(vgpu, ch, &stq, addr, n, + (const u8 *)dw, buf, len, true, false); + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY_B LOAD_INLINE_DATA: materialize the + * DMA write (the engine would do this; the emulation authors + * the bytes into the VA shadow so the forwarded launch's + * operand pages - parameter banks - are real). + */ + if (xlate && op == 3 && subch >= 1 && + addr == GRCE_M_LOAD_INLINE_DATA && + have_out && line_len) { + u64 nbytes = (u64)line_len * + (line_count ? line_count : 1); + u32 cap = count * 4; + u32 bytes, k, words; + u64 h = svc_push_hash(buf, len); + bool tail_served = end > len; + bool refuse_zero = false; + + if (nbytes > cap) + nbytes = cap; + bytes = (u32)nbytes; + if (bytes > SVC_AHEAD_TAIL_INLINE_MAX) { + bytes = SVC_AHEAD_TAIL_INLINE_MAX; + svc_ahead_inline_trunc++; + if (svc_ahead_inline_trunc <= 8) + pr_warn("sema-ahead: inline write truncated push=0x%llx dst=0x%llx want=%llu cap=%u\n", + push_va, off_out, nbytes, + SVC_AHEAD_TAIL_INLINE_MAX); + } + words = (bytes + 3) / 4; + for (k = 0; k < words; k++) + memcpy(svc_ahead_inline_buf + 4 * k, + &(u32){ svc_ahead_w32(buf, len, tail, + taillen, + off + 4 + 4 * k) }, + 4); + /* + * SESSION 129: a tail-served payload whose head is still + * all zeros means the tail source answered with an empty + * page (a stale pin shadow, or the driver has not staged + * the bytes yet) - authoring zeros over the destination + * would corrupt the code heap. Leave the write pending + * (no once-store, the sweep retries every pass) until a + * nonzero payload arrives. In-push payloads are never + * gated: the bytes are part of the flushed submit. + */ + if (inline_only && tail_served && bytes && + !memchr_inv(svc_ahead_inline_buf, 0, + min_t(u32, bytes, 16))) + refuse_zero = true; + if (inline_only) { + svc_util_wit.saw_lid = true; + svc_util_wit.dst = (u32)off_out; + svc_util_wit.line_len = line_len; + svc_util_wit.line_lc = line_count; + svc_util_wit.count = count; + if (refuse_zero) + svc_util_wit.zero = true; + } + if (refuse_zero) { + /* + * SESSION 130: a zero-headed tail-served + * payload is applied under a VARIANT once-key + * (h | 1) instead of being refused forever: + * the destination is currently empty, so + * authoring zeros is idempotent and correct + * for the driver's internal zero buffer (the + * observed 256 B utility i2m = exactly + * GF100_ZERO_BUFFER_SIZE), while a later + * non-zero payload for the same entry still + * matches its own (h) key and lands - the + * refusal can no longer stall the arm + * (r737/r739: `writes=0` with the payload + * page stably zero). + */ + svc_util_pending_zero++; + if (bytes && !svc_inline_once_seen(push_va, + h | 1, + off_out, + bytes)) { + svc_inline_once_store(push_va, h | 1, + off_out, bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, + off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + } else if (bytes && !svc_inline_once_seen(push_va, h, + off_out, + bytes)) { + svc_inline_once_store(push_va, h, off_out, + bytes); + svc_gr_ahead_inline_writes++; + if (inline_only) + svc_util_wit.fired = true; + (void)svc_grce_inline_write(vgpu, off_out, + svc_ahead_inline_buf, + bytes, + ch->h_object); + } + /* + * Record the destination and the fresh payload bytes + * for the launch fill's inline-served pages, exactly + * like the main decode's LOAD_INLINE_DATA arm: the + * QMD capture later in this entry collects the ring + * and the fill can serve the parameter bank from it + * even while its shadow page is still empty. + */ + { + struct svc_grce_rec *rec = + &svc_grce_dst_ring[svc_grce_dst_ring_next % + SVC_GRCE_DST_MAX]; + + /* + * A refused (still-zero) tail payload is not + * recorded either: the next pass re-reads the + * tail and the real bytes land then. + */ + if (!refuse_zero) { + rec->dst_va = off_out; + rec->len = min_t(u32, bytes, + GSP_BRIDGE_INLINE_BYTES); + memcpy(rec->data, svc_ahead_inline_buf, + rec->len); + svc_grce_dst_ring_next++; + } + } + if (end > len) + break; + off += 4 + count * 4; + continue; + } + + /* + * GRCE / INLINE_TO_MEMORY operand group (0x180..0x18c), for + * subchannel 1 AND 2. The mem2mem subchannel (2, the + * A140/KEPLER_INLINE_TO_MEMORY_B upload lane the utility + * channels drive) carries the same operand methods as the + * compute subchannel (1). This group MUST be parsed BEFORE + * the TURING-copy branch below: that branch's `subch >= 2` + * guard also matches subchannel 2 and would swallow the + * 0x180..0x18c methods unread, leaving have_out/line_len + * unset - the exact reason the utility channel's subch-2 + * LOAD_INLINE_DATA arm never fired (r737/r739 witness: + * `lidn=1 subch=2 ... saw=0 dst=0 ll=0`). + */ + if (xlate && (subch == 1 || subch == 2) && + addr >= 0x180 && addr <= 0x18c && count) { + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x180: + line_len = vv[k]; + break; + case 0x184: + line_count = vv[k]; + break; + case 0x188: + off_out = (off_out & 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x18c: + off_out = (off_out & + ~0xffffffffULL) | vv[k]; + have_out = 1; + break; + default: + break; + } + } + } else if (xlate && subch >= 2) { + /* TURING_DMA_COPY_A operand runs (subch >= 2). */ + u32 k, n = count > 4 ? 4 : count; + u32 vv[4] = { w0, w1, w2, w3 }; + + if (addr >= 0x400 && addr <= 0x41c && n) { + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case 0x400: + off_in = (off_in & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_in = 1; + break; + case 0x404: + off_in = (off_in & + ~0xffffffffULL) | + vv[k]; + have_in = 1; + break; + case 0x408: + off_out = (off_out & + 0xffffffffULL) | + ((u64)(vv[k] & 0x1ffff) + << 32); + have_out = 1; + break; + case 0x40c: + off_out = (off_out & + ~0xffffffffULL) | + vv[k]; + have_out = 1; + break; + case 0x410: + pitch_in = vv[k]; + break; + case 0x414: + pitch_out = vv[k]; + break; + case 0x418: + line_len = vv[k]; + break; + case 0x41c: + line_count = vv[k]; + break; + default: + break; + } + } + } else if (addr >= 0x700 && addr <= 0x708 && n) { + for (k = 0; k < n; k++) { + if (addr + 4 * k == 0x700) + last_remap = (last_remap & + ~0xffffu) | + (vv[k] & 0xffffu); + else if (addr + 4 * k == 0x708) + last_remap = (last_remap & + 0xffffu) | + ((vv[k] & 0xffffu) + << 16); + } + } else if (addr >= CE_M_SRC_ORIGIN_X && + addr <= CE_M_DST_ORIGIN_Y && n) { + /* + * SESSION 153 CONT.25: the X DDX readback + * pair positions each 6-row strip with + * SRC_ORIGIN (x bytes / y rows) instead of + * folding it into OFFSET_IN - the same + * methods the main walk now decodes. The + * strip y steps 6 per push (A: 120, B: 126, + * A: 132, ...) and the copy must read + * OFFSET_IN + origin_y*PITCH_IN. + */ + for (k = 0; k < n; k++) { + switch (addr + 4 * k) { + case CE_M_SRC_ORIGIN_X: + src_origin_x = vv[k]; + have_src_origin = 1; + break; + case CE_M_SRC_ORIGIN_Y: + src_origin_y = vv[k]; + have_src_origin = 1; + break; + case CE_M_DST_ORIGIN_X: + dst_origin_x = vv[k]; + have_dst_origin = 1; + break; + case CE_M_DST_ORIGIN_Y: + dst_origin_y = vv[k]; + have_dst_origin = 1; + break; + } + } + } else if (addr == 0x300 && count) { + u32 dma = w0; + + /* + * DATA_TRANSFER_TYPE bits 1:0; 0 = NONE (a + * release-only / semaphore op - no data + * movement, nothing to record). + * + * SESSION 153 CONT.25: remember a full setup + * (LINE_LENGTH + OFFSET_IN present) as the + * channel's persistent CE state and fill a + * DELTA launch (the X DDX readback pair's + * second push: no setup of its own, only + * SRC_ORIGIN + SEM_A/PAYLOAD + OFFSET_OUT + + * LAUNCH_DMA) from it, exactly like the main + * walk - without this every odd strip of the + * frame is dropped. + */ + last_dma = dma; + if (have_in && off_in && line_len) { + struct svc_push_state tst; + + memset(&tst, 0, sizeof(tst)); + tst.offset_in = off_in; + tst.pitch_in = pitch_in; + tst.pitch_out = pitch_out; + tst.line_length = line_len; + tst.line_count = line_count; + svc_ce_saved_store(&tst, + ch->h_object); + } + if ((dma & 0x3) && have_out && off_out && + (!have_in || !line_len)) { + struct svc_ce_saved *sv = + svc_ce_saved_slot( + ch->h_object); + + if (sv && sv->line_length && + sv->have_in) { + if (!line_len) { + line_len = + sv->line_length; + line_count = + sv->line_count; + } + if (!have_in) { + off_in = + sv->offset_in; + have_in = 1; + } + if (!pitch_in) + pitch_in = + sv->pitch_in; + if (!pitch_out) + pitch_out = + sv->pitch_out; + } + } + if ((dma & 0x3) && have_in && have_out && + line_len && line_count && + off_in && off_out) { + /* + * SESSION 153 CONT.25: fold the + * per-strip SRC/DST_ORIGIN into the + * operands (bytes on X, rows on Y; + * the linear shadow makes + * base + y*pitch + x the truth). + * line_len is BYTES for C5B5 + * (LINE_LENGTH_IN semantics), not + * dwords: the old *4 inflated every + * strip 4x. + */ + u64 src_f = off_in, dst_f = off_out; + + if (have_src_origin) + src_f += + (u64)src_origin_y * + (pitch_in ? + pitch_in : + line_len) + + src_origin_x; + if (have_dst_origin) + dst_f += + (u64)dst_origin_y * + (pitch_out ? + pitch_out : + line_len) + + dst_origin_x; + if (inline_only) + svc_ahead_copy_in(vgpu, + ch->h_object, push_va, + svc_push_hash(buf, + len), + src_f, dst_f, + line_len, + line_count, dma); + else if (forward) { + svc_gr_copy_pend_add(ch->h_object, + push_va, svc_push_hash(buf, + len), + src_f, dst_f, + line_len, + line_count, + pitch_out ? pitch_out : + line_len, + pitch_in, dma, + last_remap); + cp_subch = subch; + } + } + } + } + + if (!inline_only && addr == HOST_M_SEM_ADDR_LO && count >= 5) { + u64 va; + + va = ((u64)(w1 & 0xff) << 32) | + (u64)(w0 & 0xfffffffc); + op = w4 & 0x7; + if ((op == HOST_SEM_OP_RELEASE || + op == HOST_SEM_OP_REDUCTION) && w2 && + va >= SVC_LOW_REL_VA_LO && va < SEMA_VA_HI) { + /* + * A release targeting a live launch candidate's + * own QMD release word is owned by the + * completion edge (post-copy-out); sweeping it + * here would clear the client's fence before + * the engine output landed. + */ + if (xlate && forward && + svc_gr_ahead_owned_rel_va(va, push_va)) { + svc_gr_ahead_owned++; + } else { + svc_pending_add(va, w2, + op == HOST_SEM_OP_REDUCTION ? + 1 : 0); + nrel++; + } + } + } + if (end > len) + break; + off += 4 + count * 4; + } + (void)cp_subch; + return nrel; +} + +static void svc_gr_ahead_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 put) +{ + static u8 abuf[2048]; + static u8 nbuf[2048]; + u32 i, maxent, nrel = 0, swept = 0, nskip = 0; + + if (!put || !ch->gpfifo_entries) + return; + maxent = min_t(u32, gsp_sema_gr_ahead_max, 512); + for (i = 0; i < maxent; i++) { + u32 idx = (put + i) % ch->gpfifo_entries; + u8 ent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen, tail_len = 0; + u64 va; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)idx * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + break; /* unwritten slot: end of the continuation */ + if (!va || !len || len > SVC_PUSH_MAX) + break; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + break; + rlen = min_t(u32, len, sizeof(abuf)); + if (svc_read_va_push(vgpu, ch, va, abuf, rlen)) { + /* + * Unreadable here (pre-anchor translation) is a + * reason to stop this pass: the continuation is not + * proven coherent, so nothing is authored. + */ + nskip++; + break; + } + /* + * SESSION 127 C5C0 translator: read the next ring entry as + * the tail of a stream whose final inline run overruns this + * entry (the driver splits large LOAD_INLINE_DATA payloads + * into a dedicated data slot entry; observed beef0100 + * idx00->idx01 and idx08->idx09). + */ + { + u32 nidx = (put + i + 1) % ch->gpfifo_entries; + u8 nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 ne0, ne1, nlen; + u64 nva; + + if (!svc_read_va(vgpu, + ch->gpfifo_va + + (u64)nidx * SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent))) { + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + if (nva && nlen < SVC_PUSH_MAX) { + tail_len = min_t(u32, nlen, + sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, + nbuf, tail_len)) + tail_len = 0; + } + } + } + swept++; + nrel += svc_gr_ahead_push(vgpu, ch, abuf, rlen, nbuf, + tail_len, va, false); + } + svc_gr_ahead_entries += swept; + svc_gr_ahead_rels += nrel; + svc_gr_ahead_skips += nskip; + if (swept && (nrel || svc_gr_ahead_log < 64)) { + svc_gr_ahead_log++; + pr_info("sema-ahead: h=0x%x put=%u swept=%u rels=%u owned=%llu skipped=%u (total rels=%llu)\n", + ch->h_object, put, swept, nrel, svc_gr_ahead_owned, + nskip, svc_gr_ahead_rels); + } +} + +/* + * SESSION 129: read a VA straight through the registered-VAS walks only + * (newest registration first), one page at a time - the exact path + * walk_probe and sema_probe prove. The util sweep's channel read can be + * shadowed by a stale injected pin / window guess for the utility VAs + * (observed r737: the subchannel-2 LID arm never fired although the push + * page content is real and the VAS walk resolves both the methods entry + * and its payload slot); this helper is the arbitration fallback that + * bypasses those sources entirely. + */ +static int svc_util_walk_read(struct nv_vgpu_device *vgpu, u64 va, void *buf, + size_t len) +{ + u8 *p = buf; + + while (len) { + u64 page = va & ~(u64)(SEMA_PAGE_SIZE - 1); + size_t off = va & (SEMA_PAGE_SIZE - 1); + size_t chunk = min_t(size_t, len, SEMA_PAGE_SIZE - off); + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + + if (svc_walk_vaspaces(vgpu, page, &pa, &ap)) + return -ENOENT; + if (svc_read_ap(vgpu, pa + off, svc_ap_of(ap), p, chunk)) + return -EIO; + p += chunk; + va += chunk; + len -= chunk; + } + return 0; +} + +/* + * SESSION 129: does this entry window carry a LOAD_INLINE_DATA header? Used + * by the bounded util-sweep witness to select only the entries that can + * exercise the arm (one per distinct content, so a stable ring logs once). + */ +static u32 svc_util_lid_scan(const u8 *buf, u32 len, u32 *first_subch) +{ + u32 k, n = 0; + + *first_subch = 0xff; + for (k = 0; k + 4 <= len; k += 4) { + u32 hdr; + + memcpy(&hdr, buf + k, 4); + if (((hdr >> 29) & 0x7) == 3 && + ((hdr & 0x1fff) << 2) == GRCE_M_LOAD_INLINE_DATA) { + if (*first_subch == 0xff) + *first_subch = (hdr >> 13) & 0x7; + n++; + } + } + return n; +} + +/* + * SESSION 128: utility-channel inline-download materialization. + * + * The driver's upload/utility channels (the MEM2MEM/A140 + C5B5 + 2D + * composite: observed h=0xbeef0100 with gpfifo=0x4020000) are registered + * with an unresolved USERD (userd_ok=0), so the put-bounded entry loop in + * svc_pump_channels never decodes their pushes and the LOAD_INLINE_DATA + * downloads they carry (the client's code-heap/SASS upload, command-stream + * staging) never land anywhere the emulation can see: the destinations + * stay zero in the vidmem shadow. + * + * This pass scans a bounded window of the channel's GPFIFO directly (the + * gpfifo VA resolves through the trusted walk even when the USERD does + * not) and feeds each entry + its successor (the source-view call carrying + * the payload) through the proven sweep decoder in inline-only mode: the + * GRCE/INLINE_TO_MEMORY operand group (0x180..0x18c on subch 1/2), + * LAUNCH_DMA and the LOAD_INLINE_DATA run are materialized by + * svc_grce_inline_write; releases, QMD windows and copy-pend records are + * skipped so nothing about the client's fence/completion state changes. + * Once-only keyed by (push_va, content hash, dst, bytes), so a stable ring + * costs only the reads. Requires sema_gr_xlate; bounded by + * sema_gr_util_scan entries per channel per pass (0 = off). + * + * SESSION 129 read arbitration: attempt A is the channel path (unchanged); + * when A fails to read, or reads content the strict parse rejects, attempt + * B re-reads the push and the tail through svc_util_walk_read and the + * parse arbitrates - a source that yields a method stream wins. The r737 + * evidence (arm never fired while the content is real) pointed at exactly + * this: a shadowed channel read consumed the entry without a parseable + * buffer while the walk resolves the true pages. + */ +static void svc_gr_util_sweep_n(struct nv_vgpu_device *vgpu, + struct svc_channel *ch, u32 maxent_override) +{ +/* + * The observed utility-channel pushes are the ~17 KB setup block and the + * two split uploads (256 B command stream + 12 KB code/heap payload); a + * 64 KB window covers every one of them with room to spare while keeping + * the two static buffers modest. + */ +#define SVC_UTIL_BUF (64 * 1024) + static u8 ubuf[SVC_UTIL_BUF]; + static u8 nbuf[SVC_UTIL_BUF]; + static u8 bbuf[SVC_UTIL_BUF]; + static u8 tbuf[SVC_UTIL_BUF]; + static u64 svc_gr_util_entries; + static u64 svc_gr_util_writes; + static u64 svc_util_tail_alt; /* SESSION 130: tails re-read through + * the direct walk because the + * channel read was zero-headed */ + static u64 svc_util_tail_try; /* ... and the walk attempts (alt is + * the subset that yielded non-zero) */ + u32 i, maxent = maxent_override ? + min_t(u32, maxent_override, 64) : + min_t(u32, gsp_sema_gr_util_scan, 64); + u32 swept = 0, nmade = 0; + bool first_sight = false; + + if (!gsp_sema_gr_xlate || !maxent) + return; + if (!ch->gpfifo_va || !ch->gpfifo_entries) { + /* SESSION 143: name every channel the sweep cannot cover. */ + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + pr_info("sema-utilsweep: skip h=0x%x eng=%u reason=no-gpfifo gpfifo=0x%llx entries=%u userd=%d\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0); + } + return; + } + { + u32 k; + + for (k = 0; k < svc_util_seen_n; k++) + if (svc_util_seen_keys[k] == ch->h_object) + break; + if (k == svc_util_seen_n && + svc_util_seen_n < ARRAY_SIZE(svc_util_seen_keys)) { + svc_util_seen_keys[svc_util_seen_n++] = ch->h_object; + first_sight = true; + pr_info("sema-utilsweep: begin h=0x%x eng=%u gpfifo=0x%llx entries=%u userd=%d cid=%u\n", + ch->h_object, ch->engine_type, ch->gpfifo_va, + ch->gpfifo_entries, ch->userd_ok ? 1 : 0, + ch->cid); + } + } + for (i = 0; i < maxent && i < ch->gpfifo_entries; i++) { + u8 ent[SVC_GPFIFO_ENTRY_SIZE], nent[SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1, len, rlen; + u64 va, before; + + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + ent, sizeof(ent))) + break; + memcpy(&e0, ent, 4); + memcpy(&e1, ent + 4, 4); + va = ((u64)(e1 & 0xff) << 32) | (u64)(e0 & 0xfffffffc); + len = ((e1 >> 10) & 0x1fffff) << 2; + if (!va && !len) + continue; /* unwritten slot */ + if (!va || !len || len > SVC_PUSH_MAX) + continue; + if (va < SVC_LOW_REL_VA_LO || va >= SEMA_VA_HI) + continue; + /* + * SESSION 132 stale-translation arbitration: when the + * channel-path entry is dead (its push provably reads + * all-zero through the client's page tables) while the + * direct VAS walk answers a live entry on the same ring + * slot, the channel path is reading a stale page (the r759 + * pin false positive) - the walk's entry is the channel's + * ring. A dead entry with no live walk alternative is left + * to the existing push arbitration below. + */ + if (svc_ring_entry_dead(vgpu, e0, e1)) { + u8 went[SVC_GPFIFO_ENTRY_SIZE]; + + if (!svc_util_walk_read(vgpu, + ch->gpfifo_va + + (u64)i * SVC_GPFIFO_ENTRY_SIZE, + went, sizeof(went))) { + u32 we0, we1; + u64 wva; + u32 wlen; + + memcpy(&we0, went, 4); + memcpy(&we1, went + 4, 4); + wva = ((u64)(we1 & 0xff) << 32) | + (u64)(we0 & 0xfffffffc); + wlen = ((we1 >> 10) & 0x1fffff) << 2; + if (!svc_ring_entry_dead(vgpu, we0, we1) && + wva && wlen && wlen <= SVC_PUSH_MAX && + wva >= SVC_LOW_REL_VA_LO && + wva < SEMA_VA_HI) { + e0 = we0; + e1 = we1; + va = wva; + len = wlen; + } + } + } + rlen = min_t(u32, len, sizeof(ubuf)); + /* + * SESSION 129 read arbitration + bounded witness (see the + * sweep's header comment). Attempt A: the channel path, + * unchanged. Attempt B: the direct registered-VAS walk, used + * when A cannot read or cannot parse; the strict parse then + * arbitrates which content is the method stream. + */ + { + u32 push_rc, parse_a = 0, parse_b = 0; + u32 tl_a = 0, tl_b = 0; + u32 lidn, lidsc = 0xff, slot; + u64 nva = 0; + u32 nlen = 0; + u64 hkey; + u8 *pbuf = ubuf; + bool ut = false, seen = false; + + push_rc = svc_read_va_push(vgpu, ch, va, ubuf, + rlen); + /* + * The payload of a trailing LOAD_INLINE_DATA run rides + * the next GPFIFO entry (the driver's source-view + * GPFIFO call, observed beef0100 idx00->idx01 and + * idx08->idx09): read it as the tail exactly like + * svc_gr_ahead_sweep does. + */ + if (svc_read_va(vgpu, + ch->gpfifo_va + + (u64)((i + 1) % ch->gpfifo_entries) * + SVC_GPFIFO_ENTRY_SIZE, + nent, sizeof(nent)) == 0) { + u32 ne0, ne1; + + memcpy(&ne0, nent, 4); + memcpy(&ne1, nent + 4, 4); + nva = ((u64)(ne1 & 0xff) << 32) | + (u64)(ne0 & 0xfffffffc); + nlen = ((ne1 >> 10) & 0x1fffff) << 2; + } + if (!push_rc && nva && nlen && nlen < SVC_PUSH_MAX) { + tl_a = min_t(u32, nlen, sizeof(nbuf)); + if (svc_read_va_push(vgpu, ch, nva, nbuf, + tl_a)) + tl_a = 0; + } + if (!push_rc) + parse_a = svc_ahead_parse(ubuf, rlen, nbuf, + tl_a, &ut); + if (push_rc || !parse_a) { + bool bpush = false; + + if (nva && nlen && nlen < SVC_PUSH_MAX) { + u32 want = min_t(u32, nlen, + sizeof(nbuf)); + + if (!svc_util_walk_read(vgpu, nva, + nbuf, + want)) + tl_b = want; + } + if (!svc_util_walk_read(vgpu, va, bbuf, + rlen)) { + bpush = true; + parse_b = svc_ahead_parse(bbuf, rlen, + nbuf, tl_b, + &ut); + /* + * A's buffer is authoritative only + * when A actually read it; otherwise + * (or when only B parses) B's walk + * content is the stream. + */ + if (push_rc || + (parse_b && !parse_a)) { + pbuf = bbuf; + tl_a = tl_b; + } + } + if (push_rc && !bpush) + continue; /* unreadable both ways */ + } + /* + * SESSION 130 tail-content arbitration: a tail-served + * payload whose channel head is still all-zero is not + * the staged bytes (the page the channel translation + * points at is empty, or the read is shadowed by a + * stale pin). Try the direct registered-VAS walk - + * the same path attempt B proves - before handing + * the tail to the LID arm, whose zero-headed refusal + * would otherwise stall the upload every pass. + */ + if (tl_a && nva && + !memchr_inv(nbuf, 0, min_t(u32, tl_a, 16))) { + u32 want = tl_a; + + svc_util_tail_try++; + if (!svc_util_walk_read(vgpu, nva, tbuf, + want) && + memchr_inv(tbuf, 0, + min_t(u32, want, 16))) { + memcpy(nbuf, tbuf, want); + svc_util_tail_alt++; + } + } + memset(&svc_util_wit, 0, sizeof(svc_util_wit)); + if (tl_a >= 4) + memcpy(&svc_util_wit.t0, nbuf, 4); + if (tl_a >= 8) + memcpy(&svc_util_wit.t1, nbuf + 4, 4); + before = svc_gr_ahead_inline_writes; + (void)svc_gr_ahead_push(vgpu, ch, pbuf, rlen, nbuf, + tl_a, va, true); + if (svc_gr_ahead_inline_writes != before) + nmade++; + swept++; + /* + * Bounded witness, one line per distinct LID-carrying + * (or unparseable) entry content: which attempt was + * used, the parses, the tail resolution and what the + * LID arm did. + */ + lidn = svc_util_lid_scan(pbuf, rlen, &lidsc); + if (!lidn && parse_a) + continue; + hkey = svc_push_hash(pbuf, rlen); + for (slot = 0; slot < 32; slot++) + if (svc_util_dec_keys[slot].h == hkey && + svc_util_dec_keys[slot].va == va) { + seen = true; + break; + } + if (seen || svc_util_dec_log >= 96) + continue; + slot = svc_util_dec_log & 31; + svc_util_dec_keys[slot].va = va; + svc_util_dec_keys[slot].h = hkey; + svc_util_dec_log++; + pr_info("sema-utildec: h=0x%x i=%u va=0x%llx len=0x%x p_src=%u pa=%u pb=%u tail_va=0x%llx tail=0x%x t8=%08x %08x alt=%llu/%llu lidn=%u subch=%u saw=%d fired=%d zero=%d dst=0x%x ll=%u lc=%u f8=%08x %08x\n", + ch->h_object, i, va, rlen, + pbuf == bbuf ? 1u : 0u, parse_a, parse_b, + nva, tl_a, svc_util_wit.t0, svc_util_wit.t1, + svc_util_tail_alt, svc_util_tail_try, + lidn, lidsc, + svc_util_wit.saw_lid ? 1 : 0, + svc_util_wit.fired ? 1 : 0, + svc_util_wit.zero ? 1 : 0, + svc_util_wit.dst, svc_util_wit.line_len, + svc_util_wit.line_lc, + rlen >= 4 ? *(const u32 *)pbuf : 0, + rlen >= 8 ? *(const u32 *)(pbuf + 4) : 0); + } + } + svc_gr_util_entries += swept; + svc_gr_util_writes += nmade; + if (first_sight && !swept) + pr_info("sema-utilsweep: read=none h=0x%x gpfifo=0x%llx entries=%u\n", + ch->h_object, ch->gpfifo_va, ch->gpfifo_entries); + if (swept && (nmade || svc_gr_util_log < 128)) { + svc_gr_util_log++; + pr_info("sema-utilsweep: h=0x%x swept=%u writes=%u (total %llu/%llu)\n", + ch->h_object, swept, nmade, svc_gr_util_entries, + svc_gr_util_writes); + } +} + +static void svc_gr_util_sweep(struct nv_vgpu_device *vgpu, + struct svc_channel *ch) +{ + svc_gr_util_sweep_n(vgpu, ch, 0); +} + +/* + * Wrap-aware PUT progress for the speculative latch. + * + * A USERD adopted by the FB/VRAM shape scan carries GPFIFO ring indices in + * [0, entries] (the shape gate rejects put > entries; `entries` is the + * wrapped/empty value). Such a producer legitimately wraps entries -> 0: + * h=0x1000013 (ent=65536, latch=65536) later reads put=12, and the raw + * `put > latch` judge saw a decrease, rejected the live block and cleared + * userd_ok - the X/DDX channel then waited forever in "Failed to idle + * DMA". For that class compare modulo entries (entries is a power of two): + * a step of 1..entries/2 is forward progress; anything else (backward + * jump or more than half the ring) keeps the old reject path. + * + * Channels whose PUT is a free-running counter keep the raw comparison: + * the guest-RAM scan (SYS aperture, gr_ram_ring) explicitly admits puts + * far above entries (measured put=12 with entries=2 and put=16 with + * entries=32768), the kick family (0xbfef00xx) authors its own free- + * running count, and a non-power-of-two ring has no usable mask. A + * one-slot ring is excluded too: every put aliases, so there is no + * modulo progress to measure. + */ +static bool svc_userd_put_wrap(const struct svc_channel *ch) +{ + u32 entries = ch->gpfifo_entries; + + return entries >= 2 && !(entries & (entries - 1)) && + ch->userd_as == SVC_APERTURE_FB && !svc_kick_family(ch); +} + static void svc_pump_channels(struct nv_vgpu_device *vgpu) { + if (svc_pdb_disc_pending) { + svc_pdb_disc_pending = false; + pr_info("sema-pdb-disc: requested va=0x%llx pa=0x%llx\n", + svc_pdb_disc_va, svc_pdb_disc_pa); + svc_vas_candidates_verify(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + svc_xvas_pdb_discover(vgpu, svc_pdb_disc_va, + svc_pdb_disc_pa); + } + /* + * r1780 (CONT.430g): clear the GLX shared-surface deadlock before the + * channel loop (see the sema_gl_shsurf_watch comment). + */ + { + static unsigned long watch_at; + u64 sva = svc_gl.rel_va ? svc_gl.rel_va : SVC_GLSHSURF_VA; + + if ((gsp_sema_gl_shsurf_watch || gsp_sema_gl_shsurf_kick) && + time_after(jiffies, watch_at)) { + watch_at = jiffies + HZ; + gsp_sema_gl_shsurf_kick = false; + (void)svc_gl_shsurf_sync(vgpu, 0, sva, 0); + if (sva != SVC_GLSHSURF_VA) + (void)svc_gl_shsurf_sync(vgpu, 0, + SVC_GLSHSURF_VA, 0); + if (gsp_sema_gl_shsurf_pin) + svc_gl_shsurf_pin_sync(vgpu); + } + } /* Only ever called from the service thread. */ static struct svc_channel local[SVC_MAX_CHANNELS]; /* @@ -19933,10 +43908,111 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) bool stalled = false; spin_lock_irqsave(&svc.chan_lock, flags); + if (svc_rewind_pending) { + svc_rewind_pending = false; + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu) + continue; + if (!svc_rewind_all && c->h_object != svc_rewind_h) + continue; + pr_info("sema-chan: rewind h=0x%x last_get=%u put=%u (slot bitmap + rescue state cleared)\n", + c->h_object, + (unsigned int)c->last_get, + (unsigned int)c->put_prev); + bitmap_zero(c->slot_done_epoch, SVC_GPFIFO_MAX_BITS); + c->last_get = 0; + c->rescue_fail = 0; + c->unread_va = 0; + c->unread_until = 0; + c->defer_since = 0; + svc_gr_setup_hold_clear(c); + } + } + /* + * r707e: every-pass candidate verify with the last-good ring pair. + * Cheap (<=16 parked candidates, one walk each) and idempotent; the + * first pass after the user-VAS candidate parks registers it, so the + * client's very first push reads resolve. + */ + if (svc_lastgood_ring_pa) + svc_vas_candidates_verify(vgpu, svc_lastgood_ring_va, + svc_lastgood_ring_pa); + /* + * r507o stream-order: walk the CE channels BEFORE the GR channels. + * The client's stream order places its H2D copies before the kernel + * that consumes them, but the channels register in handle order (the + * RM's GR channel early, the copy-ce channels later), and a pass that + * decoded the launch before the CE channel's entries ran let the + * forward workqueue execute the launch's copy-in against pre-copy + * data (r480-sep: copy-in 115 ms before its stream's H2D soft + * copies; r481 proved no host semaphore links the two, so the + * emulation must enforce stream order itself). The CE copies execute + * on this same service thread, so consuming the CE entries first + * fixes the inversion directly - no cross-channel hold, no latency + * tax. Stable partition; relative order within each class preserved. + */ for (i = 0; i < SVC_MAX_CHANNELS; i++) { - if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu) + if (svc.chan[i].valid && svc.chan[i].vgpu == vgpu && + svc.chan[i].engine_type) local[n++] = svc.chan[i]; } + /* + * r605 stream-order, the eng==0 half of the r507o fix. The same + * argument that made CE channels walk before GR ones applies WITHIN + * the GR class: the driver's relocation/H2D copies ride a + * later-registered compute-class channel (h=0x5c000059 family, + * eng=0 - the r595 finding), while the launch channel registers + * early (0x5c000019). The stable partition walked the launch first, + * so at a launch decision the copy channel's PUBLISHED backlog was + * unconsumed and invisible to every guard: the r590 watermark skips + * engine_type==0, and the r541/r568 gates only see DECODED copies. + * r604 capture (fresh boot, peek armed): at the vadd's second + * decision `peek-backlog h=0x5c000059 eng=0 put=95 last=94` - the + * b-copy entry was already in the ring with GP_PUT ahead of the + * pump, executed 8 ms after the decision and 320 ms before the + * launch's copy-in, and the run was green only because the + * partial-argument defer had stretched the window; pins landing one + * pass earlier would have forwarded against the pre-copy page + * (c = a + 0, the r568 red draw). Newest-registered first + * (descending h_object) walks the copy / push-library channels + * before the compute channels, so every launch decision sees the + * copy ring consumed to its live write pointer (GP_PUT): the + * write-pointer drain, by ordering. No deferral, no cap + * interaction, no speculative execution - the pump still consumes + * each entry exactly once through its normal per-slot identity + * bookkeeping (slot_ops_ident/done/complete/hash), so nothing needs + * mirroring. A client whose compute channel registers last simply + * keeps the historical order (no worse). + */ + { + u32 prev_h = 0xffffffffu; + int picked; + + for (picked = 0; picked < SVC_MAX_CHANNELS; picked++) { + int best = -1; + u32 best_h = 0; + + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *c = &svc.chan[i]; + + if (!c->valid || c->vgpu != vgpu || + c->engine_type) + continue; + if (c->h_object >= prev_h) + continue; + if (best < 0 || c->h_object > best_h) { + best = i; + best_h = c->h_object; + } + } + if (best < 0) + break; + local[n++] = svc.chan[best]; + prev_h = best_h; + } + } spin_unlock_irqrestore(&svc.chan_lock, flags); n_gr = SVC_GR_COUNTER_TOTAL; @@ -19980,6 +44056,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_drop_gone(vgpu, local, n); + /* + * r590: publish this pass's snapshot for the CE submission-watermark + * guard (svc_ce_stream_backlog). CE channels were copied first, so a + * GR launch's decision sees the pass-current consumed positions. + */ + svc_pass_local = local; + svc_pass_local_n = (u32)n; + for (i = 0; i < n; i++) { struct svc_channel *ch = &local[i]; u64 ring_pa = 0; @@ -20005,6 +44089,16 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!ch->engine_type) { u32 pput = 0; + /* + * SESSION 153-CONTINUED-38: set the channel's VAS + * BEFORE the autoreg derive, not after the ring + * lookup. The counter slot must be disambiguated + * through THIS channel's page tables (two clients + * register the same counter VA; CONT-37 collision), + * and the ring lookup + USERD discovery below must + * also resolve through the channel's own VAS first. + */ + svc_cur_vas = ch->h_vaspace; svc_counter_autoreg(vgpu, ch, n_gr); svc_counter_flush_pending(vgpu, ch); /* @@ -20041,8 +44135,23 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc_fwd_hold.ch_handle == ch->h_object) && !svc_read_ap(vgpu, ch->userd_pa + SVC_USERD_GP_PUT, - ch->userd_as, &pput, sizeof(pput))) + ch->userd_as, &pput, sizeof(pput))) { + if (gsp_sema_trace_h && + (u32)ch->h_object == + (u32)gsp_sema_trace_h && + time_after(jiffies, svc_perf_last)) { + svc_perf_last = jiffies + HZ; + pr_info("sema-perf: pass h=0x%x pput=%u last=%u armed=%d cnt=%u sub=%llu held=%llu starved=%llu\n", + ch->h_object, pput, + (u32)ch->last_get, + ch->gr_pub_armed, + ch->counter_last, + ch->submissions, + svc.gpfifo_held, + svc.gpfifo_starved); + } svc_counter_publish(vgpu, ch, pput); + } } /* @@ -20063,15 +44172,187 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (svc_fwd_hold.active && svc_fwd_hold.vgpu == vgpu) svc_fwd_hold_expire_check(vgpu, ch); + /* + * SESSION 153-CONTINUED-61: MISS-ABORT retry tick. A re-armed + * candidate (svc_gr_missre) is re-offered on a bounded cadence + * until the pin demand resolves its pages; the full decode path + * is deliberately NOT involved (the candidate ring slot is the + * queue). svc_gr_request no-ops for other channels and defers on + * busy/inflight. Placed before the early `continue`s so the retry + * still runs when the USERD read happens to fail this pass. + */ + svc_gr_missre_tick(vgpu, ch); + if (!ch->gpfifo_va || !ch->gpfifo_entries) continue; + /* + * r1659: every channel turn answers its own VAs first. The + * GR block below also does this before its autoreg derive; + * CE channels need it before USERD discovery so the declared + * control page can be validated against this channel's ring + * entries through this channel's VAS. + */ + svc_cur_vas = ch->h_vaspace; + /* + * CONT.395/X-DMA-FIX: hVASpace==0 is the driver's "attach to + * the TSG address space". The GR client channel (0xbeef0100) + * publishes vas=0 while its CE siblings (0xbeef0101/0102) carry + * the real VAS 0xbeef0300; without this fallback svc_lookup + * cannot resolve the GR ring (ring_pa=0) and the beef draw + * stream is never decoded, so the fixed-CB scan never runs. + */ + if (!svc_cur_vas) + svc_cur_vas = svc_channel_owner_vas(vgpu, ch->h_object); + + /* + * X-DMA-FIX: the channel's own parked VAS must be usable before + * the first ring lookup/USERD sweep; the age-out registers it + * only after the X client's 10 s window. Content-gate it + * against this channel's GPFIFO. + */ + /* + * CONT.401: re-arm the GR ring content scan while the client + * is live, BEFORE the USERD probe. When the USERD block never + * passes svc_userd_probe() (ok=0, the exact unresolved-ring + * state) the pump `continue`s further down and never reaches + * the keep-alive at the end of the loop, so the bounded scan + * budget (40 slices) retires and the ring is never found. Read + * the raw GP_PUT from the registered USERD block and grant a + * fresh budget whenever it moves. + */ + if (!ch->engine_type && ch->h_object != 0x31415900u && + ch->gr_index < 32 && !ch->gr_ram_ring && ch->userd_pa && + !test_bit(ch->gr_index, &svc_gr_ring_done)) { + u32 rput = 0; + + if (!svc_read_ap(vgpu, ch->userd_pa + SVC_USERD_GP_PUT, + ch->userd_as, &rput, sizeof(rput)) && + rput && rput != ch->gr_ram_kick) { + ch->gr_ram_kick = rput; + ch->gr_ram_passes = 0; + ch->gr_ram_cursor = 0; + ch->gr_ram_refresh_at = jiffies + + msecs_to_jiffies(SVC_GR_RAM_REFRESH_MS); + } + } + /* + * r1780 (CONT.430b): persistent direct-GL ring scan. The + * beef GR client ring can register BEFORE its first kick + * (empty ring: the content scan legitimately finds nothing) + * while its USERD block is still unresolved, so the block + * above never re-arms the scan and the slice budget retires + * long before the client submits its first push. Measured + * r1780: GZDoom's ring 0xbeef0100 registered at t+0 with an + * empty ring, the 40-slice budget expired during the ~60 s + * GL-init window, and the ring was never found (the X DDX + * then hit its refcount idle timeout). Keep a slow budget + * refresh alive for the beef GR family while the ring is + * unresolved and no submission was observed; the scan stays + * sliced/bounded, the cursor is preserved so a sweep still + * completes, and any content hit latches gr_ram_ring. + */ + if (gsp_sema_gr_ram_keep && !ch->engine_type && + ch->h_object != 0x31415900u && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + ch->gr_index < 32 && !ch->gr_ram_ring && !ch->submissions && + !test_bit(ch->gr_index, &svc_gr_ring_done) && + time_after(jiffies, ch->gr_ram_refresh_at)) { + ch->gr_ram_passes = 0; + ch->gr_ram_refresh_at = jiffies + msecs_to_jiffies(2000); + } + svc_chan_vas_candidate_promote(vgpu, ch); svc_ring_autopin(vgpu, ch); + /* + * SESSION 153-CONTINUED-75: heal a GR ring whose translation + * has gone stale (the client's churn frees the ring page and + * the allocator reuses it for data; the stale pin then feeds + * the pump phantom entries - see svc_gr_ring_stale_heal). + */ + svc_gr_ring_stale_heal(vgpu, ch); rrc = svc_lookup(vgpu, ch->gpfifo_va, &ring_pa, &ring_ap); + /* + * r1682 (CONT.393): the GL client's GR kernel ring lives in + * the RM low aperture (0x4000000..0xfffffff), an identity- + * mapped kernel window (GPU VA == guest PA). No published + * client VAS directory maps it, so the walk - and every + * ring-gated USERD discovery below - fails with ring_pa=0. + * For the 0xbeef0000 GR family, prove the ring content with a + * raw PA read (PA==VA) and adopt that identity translation. + * Re-run every pass: the ring VA churns, so a new VA simply + * re-proves on the next turn (the stale pin is keyed to the + * old VA and is ignored). + * r1777 (V12): extended to the beef CE siblings (same low- + * aperture family, same identity window), with the RAM-ring + * ownership gate so the GR/CE pages cannot alias each other. + */ + if (rrc && + (ch->h_object & 0xfffff000u) == 0xbeef0000u && + ch->gpfifo_va >= SVC_LOW_REL_VA_LO && + ch->gpfifo_va < 0x10000000ULL && + !svc_ram_ring_owner_foreign(ch->h_object, ch->gpfifo_va)) { + u8 e[2 * SVC_GPFIFO_ENTRY_SIZE]; + u32 e0, e1; + + if (!svc_read(vgpu, ch->gpfifo_va, e, sizeof(e))) { + memcpy(&e0, e, 4); + memcpy(&e1, e + 4, 4); + if ((e0 || e1) && + svc_gr_entry_plausible(ch, e0, e1)) { + svc_pin_store(ch->gpfifo_va, + ch->gpfifo_va, + SVC_APERTURE_SYS); + svc_lat_store(ch->gpfifo_va, + ch->gpfifo_va, + SVC_APERTURE_SYS); + svc_ram_ring_owner_add( + ch->h_object, ch->gpfifo_va); + ring_pa = ch->gpfifo_va; + ring_ap = SVC_APERTURE_SYS; + rrc = 0; + if (gsp_sema_dbg) + pr_info("vgsp-sema: low-ring identity h=0x%x va=0x%llx\n", + ch->h_object, + (unsigned long long) + ch->gpfifo_va); + } + } + } + if (!rrc) svc_pb_pair(vgpu, ch); + if (!rrc && ring_pa) { + /* V38: pin the whole ring while the translation + * is live, then the pushes it references. */ + svc_pin_gr_range(vgpu, ch, ring_pa, ring_ap); + if (time_after(jiffies, svc_push_pin_at)) { + svc_push_pin_at = jiffies + + msecs_to_jiffies(500); + svc_pin_ring_pushes(vgpu, ch, ring_pa, + ring_ap); + } + } + if (!rrc && ring_pa) { + svc_lastgood_ring_va = ch->gpfifo_va; + svc_lastgood_ring_pa = ring_pa; + } + /* + * r707f (FOLD): the moment ANY channel's ring resolves through + * the channel object, verify the parked COPY_SERVER_PDES VAS + * candidates against that content-backed pair. The candidates + * are parked at device-create time, before the client's compute + * channels first submit, so registering here lands the VAS page + * directory BEFORE the submit's entries get their first push + * read - the window the host-triggered path could not win (the + * entry's first read and the ring's first resolution happen in + * the same pass; a pair that arrives later cannot help, because + * a failed GR entry read is never retried with the fresh VAS). + */ + if (!rrc) + svc_vas_candidates_verify(vgpu, ch->gpfifo_va, + ring_pa); /* * ISSUE-0002 (r47 diag): run the deferred CE USERD sweep for a @@ -20092,6 +44373,56 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) jiffies + 27 * HZ; } + /* + * SESSION 128: utility/upload channels carry the engine's + * inline code/data downloads (LOAD_INLINE_DATA on the + * mem2mem subchannel) and the copy-class channels (eng=11/12) + * carry the driver's pitch copy-ins (staging -> code heap / + * constant store). With an unresolved USERD the put-bounded + * entry loop below never decodes them (the !userd_ok branch + * continues before it); with a resolved USERD the main + * decode's LOAD_INLINE_DATA/operand arms still only accept + * subchannel 1, so the subchannel-2 downloads are missed + * either way. Scan every channel's GPFIFO directly here and + * materialize the downloads (see svc_gr_util_sweep); the + * inline-only mode plus the once-only key make the overlap + * with the main loop and the GR sweep a no-op. + */ + if (gsp_sema_gr_util_scan && gsp_sema_gr_xlate) + svc_gr_util_sweep(vgpu, ch); + /* + * SESSION 154 r1509: the beef group's one-shot init uploads + * (the GL client's LOAD_INLINE_DATA code/constant/vertex + * images) sit at the HEAD of the ring, but the first observed + * GP_PUT can already be far past them (r1508: put=8982 at + * registration), so the continuation sweep above starts + * beyond the uploads and the fetch pages stay empty. Scan the + * head once per channel with the same read-arbitrated, + * tail-aware, inline-only decoder: it materializes the + * LOAD_INLINE_DATA/copy-ins and records the destination ring, + * while releases and QMD work stay untouched. + */ + if (gsp_sema_gr_xlate && !ch->head_inline_done && + (ch->h_object & 0xfffff000u) == 0xbeef0000u) { + ch->head_inline_done = true; + svc_gr_util_sweep_n(vgpu, ch, 64); + } + + /* + * SESSION 153-CONT-7: a kick-family channel whose USERD + * stays unresolved (the r939/r940 decoys are excluded by + * the allocation-locality gate and no in-window control + * block exists) still owes its pace publication: the GL + * client polls the publish words to pace its render loop, + * and the publish advance was otherwise reachable only + * through the decoy consume path. Author it per pass while + * the USERD is missing; once a userd exists the normal + * retire path carries it again. + */ + if (!ch->engine_type && !ch->userd_ok && + ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu) + svc_pub_advance(ch); if (!ch->userd_ok) { if (svc_chan_userd(vgpu, ch, rrc ? 0 : ring_pa, rrc ? -1 : ring_ap, &put, &get, @@ -20103,6 +44434,35 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * Treat as stalled so the content sweep looks for * the releases in RAM. */ + if (ch->engine_type && ch->notified) + svc_ce_init_rel_author(vgpu, ch, true, + "userd-miss"); + /* + * r1663: the X DDX's kick channel has no + * discoverable USERD (2-entry ring at a page + * end, FB userd never written), yet its ring + * PA resolves and entry 0 carries the + * class2d SET_NOTIFY target. Capture it from + * the ring head before stalling so the kick + * lane can author the completion the X + * server waits on. + */ + if (!ch->engine_type && !ch->notify2d_va && + ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu && ring_pa) { + svc_kick_capture_arm(vgpu, ch, ring_pa, + ring_ap, 0); + /* + * The captured target is the class2d + * completion word; author the zeroed + * NvNotification the guest waits on + * (the ring walk that would carry the + * NOTIFY never runs without a USERD). + */ + if (ch->notify2d_va) + (void)svc_notify2d_write(vgpu, + ch, true); + } stalled = true; continue; } @@ -20116,6 +44476,69 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * r832: the RC watchdog's notifier ack must not depend on the + * ring reads. Its VA is known from the channel's first decode + * (ch->notify2d_va, captured from SET_NOTIFY_A/B) and the ack + * is the same zeroed NvNotification the decode path writes: + * the guest re-arms status=0xFFFF every cycle and checks it + * back to 0 within 7 s. Under heavy GL traffic the watchdog + * ring's translation can go stale for stretches (its entries + * then read as zeros and are consumed as empty), which starved + * exactly this write and wedged the guest with "GPU is + * probably locked" while every other channel kept running. + * Writing the ack on every pass that sees work pending is + * idempotent. + * + * SESSION 153-CONTINUED-19: NOT for the kick family. Their + * notify2d_va is the kick record's own NOTIFY target (the + * shared completion word the GL client polls), and the kick + * lane authors that word (a completed NvNotification with a + * nonzero timestamp, svc_kick_notify_exec). The per-pass ack + * zeroed it 14029 times on r972 - the last kick write + * survived at most one pass (~60 ms) before being wiped, so + * the client's poll kept seeing "not fired". The GPU never + * clears this notification; the client does (re-arm). + */ + if (!ch->engine_type && ch->notify2d_va && put != get && + !svc_kick_family(ch)) + svc_notify2d_write(vgpu, ch, false); + /* + * SESSION 148 (r864): registration-based keep-alive that + * needs no decode - ack the driver-registered notifier when + * it is armed. Catches the stretches where the watchdog ring + * reads go unreadable and notify2d_va was never captured. + */ + if (!ch->engine_type) + svc_notify2d_keepalive(vgpu, ch); + + /* + * SESSION 149 R15 (M1.1b): GR RAM-scan keep-alive. The boot + * budget can retire before the client creates its ring (r886: + * 16 slices spent, no pairing). A GPFIFO put advance proves + * the client is live NOW, so grant a fresh bounded budget - + * capped and spaced so an unfound ring cannot keep the scan + * hot. The scan itself stops on its own once the ring is + * content-validated (gr_ram_ring). + */ + if (!ch->engine_type && ch->h_object != 0x31415900u && + ch->gr_index < 32 && + !test_bit(ch->gr_index, &svc_gr_ring_done) && + ch->gr_ram_passes >= svc_gr_ram_pass_budget() && + ch->gr_ram_refreshes < SVC_GR_RAM_REFRESH_MAX && + time_after_eq(jiffies, ch->gr_ram_refresh_at) && + ch->gr_ram_kick != put) { + ch->gr_ram_kick = put; + ch->gr_ram_refreshes++; + ch->gr_ram_refresh_at = jiffies + + msecs_to_jiffies(SVC_GR_RAM_REFRESH_MS); + ch->gr_ram_passes = 0; + pr_info("sema-chan: GR ring h=0x%x rank=%u RAM content scan keep-alive %u/%u (put=%u moved; budget reset)\n", + ch->h_object, ch->gr_index, + ch->gr_ram_refreshes, SVC_GR_RAM_REFRESH_MAX, + put); + } + + /* * ISSUE-0002 diag: put snapshots for CE channels, bounded per * session. The throttle is indexed by rank (the channel struct * iterated here is a local copy, so a per-channel stamp never @@ -20136,6 +44559,65 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * r1653: the class2d kick channel's notification target is + * armed in the first GPFIFO entry, which the walk seed can + * skip. When the live PUT is known and the target is still + * missing, decode the head entries once here (the pump has + * the channel's GPFIFO VA and PUT). Without it every short + * kick push has no address and the GL client spins. + */ + if (!ch->engine_type && !ch->notify2d_va && put && + ch->h_object >= 0xbfef0000u && + ch->h_object <= 0xbfef00ffu) { + if (ring_pa) { + svc_kick_capture_arm(vgpu, ch, ring_pa, + ring_ap, put); + } else { + u64 rpa = 0; + int rap = SVC_APERTURE_SYS; + + if (!svc_lookup(vgpu, ch->gpfifo_va, &rpa, + &rap)) + svc_kick_capture_arm(vgpu, ch, rpa, + rap, put); + } + } + + /* + * r1650: keep the UVM rank-slot release moving for a short + * window. Two effects need it: the pool page can move / a + * per-client VAS directory can be discovered seconds later + * (r1649: the one-shot write landed on a stale page before + * the live directory existed), and the channel's init + * sequence queues more than one push (r1651 gdb: the rank-0 + * tracker entry waited for value 2 while the authored word + * read 1; a manual write of 2 cleared uvm_push_end_and_wait). + * Forward-only and idempotent, one attempt per 250 ms per + * rank for 10 s, escalating 1 -> 2 -> 3 -> 4: the channel's + * control-entry + init pushes are boot-time configuration, + * so completing them host-side is the same shortcut the + * first release already uses. Values beyond that need the + * real CE push decode and are not authored here. + */ + if (ch->engine_type && ch->ce_index < 32 && + test_bit(ch->ce_index, &svc_ce_initrel_done) && + time_before(jiffies, svc_ce_initrel_until[ch->ce_index]) && + time_after_eq(jiffies, + svc_ce_initrel_next[ch->ce_index])) { + u64 tva2 = SVC_UVM_POOL_MAP_BASE + + 4ULL * (u64)ch->ce_index; + + svc_ce_initrel_next[ch->ce_index] = + jiffies + msecs_to_jiffies(250); + if (svc_ce_initrel_val[ch->ce_index] < 4) { + svc_ce_initrel_val[ch->ce_index]++; + svc_pool_write_all_vas(vgpu, ch->h_object, + tva2, + svc_ce_initrel_val[ch->ce_index]); + } + } + + /* * P6c C3: a latched fault notifier is released when the guest * submits more work (GPFIFO put advanced past the value at * delivery), alongside the guest-ack read in @@ -20170,8 +44652,218 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) /* Initialize expected semaphore payload from the pool (if mapped). */ svc_init_expected_payload(vgpu, ch); - if (put >= ch->gpfifo_entries) + /* SESSION 153-CONTINUED-20: the family test shared by the + * kick-confirm relaxation and the churn monitor below. */ + bool kfam = svc_kick_family(ch); + + /* + * SESSION 154 (M-GL1.2): speculative-latch confirmation for + * the RAM USERD. The scan latches a gated candidate right + * away (fast walk start); the pump is the judge: a +1/+2 + * step of the block's PUT confirms it as the client's live + * control block, while a jump, a reset or no movement within + * the bound rejects it so the scan can adopt a better + * candidate. The live block advances 1->2->3 in 2-10 s + * (measured); the client's own release waits are ~11 s, so + * the no-advance bound must sit above those. + * + * SESSION 153-CONTINUED-20: for the kick family ANY positive + * PUT step confirms (its put is authored by the client's + * frame loop), only a decrease or the no-advance bound + * rejects - and a CONFIRMED kick latch is not exempt from the + * churn rule: the r974 false confirm (pa=0x11c62e000, put + * walked 1->2->3, then reset to 0/1 and toggled for minutes) + * shows a slowly resetting pool block can pass any + * single-step test. Track its PUT while confirmed and + * unlatch on the first decrease - a client-authored + * free-running put never decreases (the rewind path owns the + * session-reset case). + */ + if (kfam && ch->gr_ram_userd_confirmed && + put > ch->gr_ram_userd_latch_put) + ch->gr_ram_userd_latch_put = put; + /* + * SESSION 153-CONTINUED-38: the latch confirmation must also + * judge a block adopted through the RAM scan for a + * walk-resolved ring (gr_ram_ring false; gr_ram_userd_latch_at + * stamps the latch), or a wrong low-aperture latch would stick + * forever. + */ + if (!ch->engine_type && ch->userd_ok && + (ch->gr_ram_ring || ch->gr_ram_userd_latch_at) && + (!ch->gr_ram_userd_confirmed || + (kfam && put < ch->gr_ram_userd_latch_put))) { + bool stepped; + bool wrap = svc_userd_put_wrap(ch); + u32 wstep = wrap ? + (put - ch->gr_ram_userd_latch_put) & + (ch->gpfifo_entries - 1) : 0; + bool moved = wrap ? wstep != 0 : + put != ch->gr_ram_userd_latch_put; + + if (!ch->gr_ram_userd_latch_at) + ch->gr_ram_userd_latch_at = jiffies; + /* + * r1780 (V32): any forward PUT movement is progress. + * The old +1/+2 step test for non-kick channels + * rejected a block whose PUT advanced by more than + * two entries between pump passes (the X server's + * 2D stream does exactly that after a burst), which + * cleared userd_ok and produced an adopt/reject spin + * at ~10 Hz. A forward jump is not evidence of a + * stale latch: only a decrease (wrong actor / reused + * block) or the 15 s no-advance bound is. + * + * WRAP-FIX: for a ring-index PUT (FB shape scan, + * power-of-two entries) "forward" is modulo entries. + * h=0x1000013 legitimately wrapped 65536 -> 12 and + * the raw compare rejected the live block; see + * svc_userd_put_wrap(). + */ + stepped = wrap ? + (wstep && wstep <= ch->gpfifo_entries / 2) : + put > ch->gr_ram_userd_latch_put; + if (stepped) { + ch->gr_ram_userd_confirmed = true; + if (wrap && + put < ch->gr_ram_userd_latch_put) + pr_info_ratelimited("sema-chan: GR USERD h=0x%x pa=0x%llx speculative latch wrap accepted (put %u latch %u ent %u step %u)\n", + ch->h_object, ch->userd_pa, + put, + ch->gr_ram_userd_latch_put, + ch->gpfifo_entries, wstep); + pr_info("sema-chan: GR USERD h=0x%x pa=0x%llx speculative latch CONFIRMED (put %u -> %u)\n", + ch->h_object, ch->userd_pa, + ch->gr_ram_userd_latch_put, put); + } else if (kfam && (gsp_sema_kick_latch_keep || + ch->gr_ram_userd_confirmed) && + put == 0 && + ch->gr_ram_userd_latch_put != 0 && + ch->gr_ram_userd_keeps < 64) { + /* + * CONT.407: a CONFIRMED kick-family block that + * the owner zeroes is a put reset, not a dead + * latch. Keep the block (verified by the +1 + * step above), re-arm at the reset baseline + * and let the next authored step re-confirm; + * r1701-r1703 lost a working USERD exactly + * here 7-8 s before the frame loop's next + * kick. Bounded keeps keep a truly churning + * zeroing block on the old reject path. + * + * r1772 (IDLE-DMA-RCA F2): also keep an + * ADOPTED-but-unconfirmed latch (r1771 c18: + * ADOPT put 2->3 at [42774.3], REJECTED put=0 + * latch=3 7 s later, kick lane dark 78 s + * after; the ADOPT itself was ring-entry- + * validated). sema_kick_latch_keep=0 restores + * the pre-fix condition for A/B. + */ + ch->gr_ram_userd_keeps++; + if (ch->gr_ram_userd_keep_log < 8) { + ch->gr_ram_userd_keep_log++; + pr_info("userd latch keep h=0x%x blk=0x%llx put=%u\n", + ch->h_object, ch->userd_pa, + put); + } + ch->gr_ram_userd_latch_put = put; + ch->gr_ram_userd_latch_at = jiffies; + } else if (moved || + time_after(jiffies, + ch->gr_ram_userd_latch_at + + msecs_to_jiffies(15000))) { + unsigned long lflags; + int li; + + pr_info("sema-chan: GR USERD h=0x%x pa=0x%llx speculative latch REJECTED (put=%u latch=%u%s)\n", + ch->h_object, ch->userd_pa, put, + ch->gr_ram_userd_latch_put, + moved ? "" : " no advance in 15 s"); + /* + * SESSION 153-CONT-5: one stale window must not + * kill the pipeline. Remember the rejected + * (pa, put) so the scan never re-latches the same + * frozen block, re-arm the scan budget (bounded) + * so a REAL control block that appears later can + * still be adopted, and only then drop the USERD. + * r936: the rejection retired the 20-slice budget + * and the kick channel never re-latched anything. + */ + ch->gr_ram_userd_reject_pa = ch->userd_pa; + ch->gr_ram_userd_reject_put = + ch->gr_ram_userd_latch_put; + if (ch->gr_ram_userd_rejects < 64) { + ch->gr_ram_userd_rejects++; + ch->gr_ram_userd_passes = 0; + ch->gr_ram_userd_logged = false; + } + ch->userd_ok = false; + ch->userd_pa = 0; + ch->userd_as = 0; + ch->gr_ram_userd_latch_at = 0; + ch->gr_ram_userd_confirmed = false; + ch->gr_ram_put_seeded = 0; + ch->last_get = 0; + /* + * Persist immediately: the pump `continue`s + * here, skipping the end-of-pass write-back + * for this channel. + */ + spin_lock_irqsave(&svc.chan_lock, lflags); + for (li = 0; li < SVC_MAX_CHANNELS; li++) + if (svc.chan[li].valid && + svc.chan[li].vgpu == vgpu && + svc.chan[li].h_object == + ch->h_object) { + svc.chan[li].userd_ok = false; + svc.chan[li].userd_pa = 0; + svc.chan[li].userd_as = 0; + svc.chan[li].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[li].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + svc.chan[li].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[li].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[li].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + break; + } + spin_unlock_irqrestore(&svc.chan_lock, lflags); + continue; + } + } + if (put >= ch->gpfifo_entries) { + u32 put_raw = put; + put %= ch->gpfifo_entries; + /* + * SESSION 154 (M-GL1.2): a free-running producer (the + * GL client's control block counts past any small ring + * - put=12 with entries=2, measured) can collapse + * onto the pump's own position by the modulo: an even + * number of positions ahead and last_get == put, so + * the walk consumes NOTHING and the client's release + * is never applied (r914: submissions stayed 0 while + * the client retried for minutes). One-shot per put + * value: step the walk back one slot so it consumes + * the newest entry; normal slot-space tracking + * resumes from there and the next put value walks + * on. + */ + if (ch->gpfifo_entries > 1 && + put == (u32)ch->last_get && + put_raw != ch->gr_ram_put_seeded) { + ch->gr_ram_put_seeded = put_raw; + ch->last_get = (put + ch->gpfifo_entries - 1) % + ch->gpfifo_entries; + pr_info_ratelimited("sema-chan: GR free-running put h=0x%x put=%u get=%u ent=%u -> walk seed last=%u (modulo collapse consumed nothing)\n", + ch->h_object, put_raw, get, + ch->gpfifo_entries, + (u32)ch->last_get); + } + } /* * A wrapped put means every slot may hold a new entry: clear @@ -20309,6 +45001,60 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->put_prev = put; /* + * SESSION 126 fence fold: sweep the ring continuation beyond + * GP_PUT for semaphore releases (see svc_gr_ahead_sweep). + * GR-class channels only, live-armed epochs only, and only + * with the ring translation proven through the channel + * object (rrc == 0). + */ + if (!ch->engine_type && ch->gr_pub_armed && + gsp_sema_gr_ahead_max && !rrc && ring_pa) { + svc_gr_ahead_sweep(vgpu, ch, put); + /* + * SESSION 127 C5C0 translator: offer a launch the + * sweep captured from the continuation (the Vulkan + * client's inline-QMD push lives beyond GP_PUT, so + * the main walk's per-entry request never sees it). + * svc_gr_request is a no-op without a matching + * unoffered candidate and keeps it on a busy + * verdict, so a bounded per-pass retry here is the + * sweep-captured launch's entire dispatch path. + */ + if (gsp_sema_gr_xlate && + gsp_phys_gr_forward_enabled()) { + if (svc_gr_cand.valid && + svc_gr_cand.h_object == ch->h_object && + svc_gr_ahead_req_log < 8) { + svc_gr_ahead_req_log++; + pr_info("sema-ahead: request h=0x%x push=0x%llx offered=%d\n", + ch->h_object, + svc_gr_cand.push_va, + svc_gr_cand.offered); + } + (void)svc_gr_request(vgpu, ch); + } + /* + * Fallback copy-out retirement: when no forward is + * owed (the launch was already consumed or the + * backend is off), run any pending GRCE copies so + * they cannot sit forever (the completion edge is + * the primary path). + * + * SESSION 153 CONT.25: also run when a forward IS + * owed - the X DDX readback pair's copies have no + * forwarded launch behind them, and a stuck + * unoffered candidate (valid && !offered) blocked + * this fallback forever: 1443 copy pends, 0 runs, + * the server's per-copy completion wait timing out + * on every strip. The completion edge still runs + * them first when it fires; the runner retires + * each entry once, so running early is bounded. + */ + if (gsp_sema_gr_xlate) + svc_gr_copy_pend_run(vgpu); + } + + /* * STALLFREE r225: stale-epoch / stale-replay wedge watchdog. * Runs before the entry loop so a wedged session epoch is * recovered (and logged) instead of spinning: see @@ -20342,6 +45088,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since = 0; ch->unread_va = 0; ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); svc_replay_note(ch, (u32)ch->last_get, put); ch->last_get = (ch->last_get + 1) % ch->gpfifo_entries; @@ -20350,6 +45097,14 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) get_before = ch->last_get; applied_before = svc.releases_applied; + /* r835: this channel's VAS answers its VAs first (see + * svc_chanvas_lookup). Refreshed per channel turn; a channel + * without a known VAS clears it back to the old chain. */ + svc_cur_vas = ch->h_vaspace; + /* CONT.395: hVASpace==0 attaches to the TSG VAS - use the + * sibling/owner VAS so releases resolve for the GR client. */ + if (!svc_cur_vas) + svc_cur_vas = svc_channel_owner_vas(vgpu, ch->h_object); /* * C4 (GAP-1 H1): the GR walk witness - does put advance, and @@ -20446,6 +45201,11 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gpfifo_va + idx * SVC_GPFIFO_ENTRY_SIZE, entry, sizeof(entry))) { + if (gsp_sema_dbg) + pr_info_ratelimited("sema-chan: ring entry unreadable h=0x%x idx=%llu va=0x%llx ring_pa=0x%llx ap=%d\n", + (u32)ch->h_object, idx, + ch->gpfifo_va + idx * 8, + ring_pa, ring_ap); svc.gpfifo_nomap++; break; } @@ -20467,10 +45227,17 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ { u64 ident = (u64)e0 | ((u64)e1 << 32); - bool same = ident && + /* + * r505b: the per-slot op bookkeeping arrays are + * SVC_GPFIFO_MAX_BITS (1024) entries; the X/GL + * channel's ring is 8192 slots. Bound every slot + * index here or a large ring writes out of + * bounds. + */ + bool same = ident && idx < SVC_GPFIFO_MAX_BITS && ch->slot_ops_ident[idx] == ident; - entry_ident = ident; + entry_ident = (idx < SVC_GPFIFO_MAX_BITS) ? ident : 0; /* * A complete prior pass suppresses every op. * A partial pass (an operand's targeted scan @@ -20507,6 +45274,49 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) } /* + * SESSION 153-CONTINUED-65: per-boot map of every + * distinct beacon entry (see svc_beef_entry_note) - + * the setup-hold/capture binding reads this list. + */ + if ((ch->h_object & 0xfffff000u) == 0xbeef0000u) + svc_beef_entry_note(ch->h_object, (u32)idx, + va, len); + + /* + * r829: a zero-length / zero-VA entry is a bad read + * (a stale or wrong translation), never work. The old + * path consumed it as an empty push and advanced + * GPGet; observed r830 on the RC watchdog channel: + * two bogus zero entries consumed through a + * mis-resolved ring page pushed the userd's GPGet past + * the driver's own GPPut, every later userd probe then + * read get>put and the channel's REAL userd was + * rejected - the channel stopped being serviced and + * the guest declared the GPU locked. Hold the entry + * instead (bounded; the per-channel counter persists + * through the pass snapshot); a corrected translation + * (the ring auto-pin lands within seconds) re-reads + * it. Past the cap the old consume-as-empty behavior + * resumes so an unresolvable ring cannot livelock the + * channel. + */ + if (va == 0 || len == 0) { + if (ch->zero_held < 512 && !svc_hold_bypass()) { + ch->zero_held++; + svc.gpfifo_held++; + if (gsp_sema_dbg && + svc_zero_entry_log < 16) { + svc_zero_entry_log++; + pr_info("sema-chan: zero-entry hold h=0x%x idx=%llu raw=%08x%08x (bad ring translation; not consumed)\n", + ch->h_object, idx, + e1, e0); + } + stalled = true; + continue; + } + } + + /* * Cheap wait for an entry already inside its * expensive-wait window: read + rescue + hold were * already paid for this VA, and a pin-batch rewind @@ -20519,7 +45329,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * exists for the page, the put advances (a new * push at the same VA), or the window expires. */ - if (ch->engine_type && ch->unread_va == va && + if ((ch->engine_type || + svc_gr_setup_hold_active(ch, va, (u32)idx)) && + ch->unread_va == va && ch->unread_idx == idx && time_before(jiffies, ch->unread_until)) { u64 tpa = 0, vpa = 0; @@ -20554,11 +45366,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (pb) { int rescue_rc; - bool unreadable = - svc_read_va_push(vgpu, ch, va, - pb, len) != 0; + int rd_rc; + bool unreadable; + bool zero_nop = false; + bool displaced = false; + bool have_ipin = false; + + svc_read_last_src = 0xff; + svc_read_last_pa = 0; + rd_rc = svc_read_va_push(vgpu, ch, va, + pb, len); + unreadable = rd_rc != 0; + if (gsp_sema_dbg && svc_rdw_log < 6000 && + !ch->engine_type) { + svc_rdw_log++; + pr_info("sema-rdw: h=0x%x va=0x%llx len=%u rc=%d src=%u\n", + ch->h_object, va, len, + rd_rc, + (unsigned)svc_read_last_src); + } bool injected = false; bool allow_ops = (ch->engine_type != 0); + bool kick_badpg = false; /* * A resolved buffer that does not start @@ -20581,9 +45410,126 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u64 ipa; int iap; - injected = (svc_ipin_lookup(va, &ipa, - &iap) == 0); - if (!svc_method_stream_plausible(pb, len)) { + have_ipin = + (svc_ipin_lookup(va, &ipa, + &iap) == 0); + /* + * r715: the entry's treatment + * follows the source that + * ACTUALLY answered the read, + * not a fresh page-granularity + * ipin lookup. The read falls + * through to the trusted walk + * (src 4) whenever an ipin's + * page fails its first-chunk + * guard, and taking the + * injected branch then skipped + * the only source that could + * answer: the entry was + * consumed without ever + * consulting the registered + * VAS walk (session 124). + */ + injected = have_ipin && + (svc_read_last_src == 0); + /* + * r496e: an all-zero push on a + * content-scanned-ring channel is the + * driver's NOP sanity push (one zero + * dword - NVA06F_DMA_NOP), not a bad + * translation: accept it and let the + * decode find no ops. + * r820/r826: the same push arrives on + * KERNEL channels (nvidia-push's + * __nvPushTestPushBuffer writes the + * NOP into the kernel pushbuffer); its + * promote entry is a booking, so the + * trusted walk answers - a SMALL + * all-zero GR push resolved by the + * trusted chain (src=4) is the NOP. + * Real launches are large (a QMD push + * is hundreds of bytes), so the size + * bound keeps an unread legitimate + * push from being consumed as empty. + */ + /* + * r934 (SESSION 153-CONT-3): + * bound the gr_ram_ring + * zero-NOP acceptance by + * SIZE. The NOP sanity push + * is one zero dword + * (NVA06F_DMA_NOP; observed + * len=4 on h=0x1000013); + * without the bound a + * 64-byte ALL-ZERO read + * from a stale validated pin + * (the GL client's kick page + * after the guest reused it, + * r933: src=2 + * pa=0x1285a3000 zeros while + * the real kick 0001fff0 + * 20010000 sat at + * pa=0x12c94b000) was + * consumed as empty and the + * entry never fell through + * to the VAS walk that + * answers correctly. With + * the bound the implausible + * push takes the r707v + * viawalk retry (and, if + * that fails, the stale pin + * is dropped by the + * rescue path). + */ + zero_nop = (ch->gr_ram_ring || + (!ch->engine_type && + svc_read_last_src == 4)) && + len >= 4 && len <= 16 && + !memchr_inv(pb, 0, len); + + /* + * r977/r979: a read of the + * kick family's per-frame + * record (SVC_GL_KICK_PUSH_VA; + * r978 showed the channel also + * carries bulk pushes that + * must NOT be held to this + * content identity - a 256 KB + * push at 0x12048b824) whose + * bytes lack the canonical + * submit-record prefix is a + * wrong page no matter how + * plausible it parses (r976: + * src=3 window garbage + * 9864be60 ffffffff consumed + * as a valid push and the kick + * lane ran zero execs). Force + * the implausible path so the + * VAS walk / content anchor / + * pin-drop chain below + * re-anchors the page. + */ + kick_badpg = !zero_nop && + svc_kick_family(ch) && + va == SVC_GL_KICK_PUSH_VA && + len <= 128 && + !svc_kick_canonical(pb, len); + if (kick_badpg) { + svc.kick_badpg++; + if (svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-kickpg: h=0x%x va=0x%llx len=%u src=%u pa=0x%llx first8=%08x %08x (non-canonical kick page; rescue chain)\n", + ch->h_object, va, len, + (unsigned)svc_read_last_src, + svc_read_last_pa, + len >= 4 ? *(const u32 *)pb : 0, + len >= 8 ? *(const u32 *)(pb + 4) : 0); + } + } + + if ((!svc_method_stream_plausible(pb, len) || + kick_badpg) && + !zero_nop) { svc.false_trans++; if (injected) { u8 *fb = kmalloc(len, @@ -20611,7 +45557,134 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(fb); } } else { - unreadable = true; + /* + * r707v: a primary-source + * hit (injected pin, window + * guess, lat cache) whose + * content is implausible + * must not shadow the + * registered VAS walk - a + * stale pin otherwise makes + * every push read "resolve" + * to the wrong page, fail + * plausibility and turn the + * entry unreadable forever. + * Retry once with the pins + * bypassed so the trusted + * walk (svc_walk_vaspaces) + * answers; accept only a + * plausible stream. + */ + u8 *vb = kmalloc(len, + GFP_KERNEL); + + if (vb) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + bool ok = false; + + /* + * r707w: consult ONLY the + * registered VAS walk (the + * exact path walk_probe + * proves) - not + * svc_lookup_trusted, whose + * pin/lat hits would answer + * with the same wrong page + * again and keep the entry + * unreadable. + */ + int wr = svc_walk_vaspaces(vgpu, + va & ~0xfffULL, + &wpa, &wap); + int rr = wr ? wr : svc_read_ap(vgpu, + wpa + (va & 0xfff), + wap == SVC_APERTURE_FB ? + SVC_APERTURE_FB : + SVC_APERTURE_SYS, + vb, len); + bool pl = !rr ? svc_method_stream_plausible(vb, len) + : false; + + /* + * r977: for a kick-family + * entry a plausible walk + * answer still has to carry + * the canonical prefix - a + * wrong-VAS garbage page + * parses plausibly (r976) and + * would re-shadow the real + * page here. + */ + if (pl && kick_badpg && + !svc_kick_canonical(vb, len)) + pl = false; + + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk rc=%d pa=0x%llx plaus=%d\n", + ch->h_object, va, + wr ? wr : rr, + wpa + (va & 0xfff), + pl ? 1 : 0); + } + if (!wr && !rr && pl) { + memcpy(pb, vb, len); + svc.pushpg_fallback++; + ok = true; + /* + * r977: a kick-family + * entry whose page the + * walk re-anchored is + * pinned so later passes + * read the canonical page + * directly (src=2) instead + * of re-paying the walk + * while the content anchor + * catches up (r975 needed + * ~78 s for the anchor; the + * walk answered in seconds). + */ + if (kick_badpg && + wap == SVC_APERTURE_SYS && + svc_kick_canonical(vb, len)) + svc_pin_store(va & ~(u64)(SEMA_PAGE_SIZE - 1), + wpa, + SVC_APERTURE_SYS); + if (gsp_sema_dbg && + svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx viawalk ok pa=0x%llx\n", + ch->h_object, va, + wpa + (va & 0xfff)); + } + } + kfree(vb); + if (!ok && + !ch->engine_type) { + u64 fbo = 0; + + if (svc_fb_push_rescue(vgpu, + va, len, + pb, &fbo)) { + ok = true; + injected = false; + svc_pin_store( + va & ~(u64)(SEMA_PAGE_SIZE - 1), + fbo, + SVC_APERTURE_FB); + svc_lat_store( + va & ~(u64)(SEMA_PAGE_SIZE - 1), + fbo, + SVC_APERTURE_FB); + } + } + if (!ok) + unreadable = true; + } else { + unreadable = true; + } } /* * Still implausible (or @@ -20627,18 +45700,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) u8 *db = kmalloc(len, GFP_KERNEL); - if (db && - svc_push_read_displaced(vgpu, - ch->h_object, va, - db, len)) { - memcpy(pb, db, len); - injected = false; - unreadable = false; - } - kfree(db); - } - } - } + if (db && + svc_push_read_displaced(vgpu, + ch->h_object, va, + db, len)) { + memcpy(pb, db, len); + injected = false; + unreadable = false; + displaced = true; + } + kfree(db); + } + } + } /* * The entry's page was unreadable through @@ -20657,11 +45731,87 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) memcpy(pb, db, len); injected = false; unreadable = false; + displaced = true; } kfree(db); } /* + * SESSION 154 (M-GL1.2): a GR RAM-ring + * client push whose every primary + * source failed (r913: a validated pin + * kept answering for a page the guest + * reused and the wrong-VASPACE walk + * answer was implausible). The + * client's submit pushes carry stable + * content anchors, so sweep RAM for + * the page, validate it, pin it and + * re-read the entry through the fresh + * pin. When the anchor has not found + * the page yet, drop a stale validated + * pin (the read that failed came from + * src=2) so later reads prefer the + * walk and the anchor over it. + */ + if (unreadable && !ch->engine_type && + ch->gr_ram_ring && va && len) { + if (svc_gr_anchor_scan(vgpu, ch, + va, len) && + !svc_read_va_push(vgpu, ch, + va, pb, + len) && + svc_method_stream_plausible(pb, + len) && + (!(svc_kick_family(ch) && + va == SVC_GL_KICK_PUSH_VA && + len <= 128) || + svc_kick_canonical(pb, + len))) { + ch->rescue_last = 0; + ch->push_hold_until = 0; + svc.gpfifo_rescued++; + unreadable = false; + } else if (svc_read_last_src == 2) { + svc_pin_drop(va); + if (svc_pushpg_fb_log < 64) { + svc_pushpg_fb_log++; + pr_info("sema-pushpg: h=0x%x va=0x%llx stale validated pin dropped (content failed every source)\n", + ch->h_object, + va); + } + } + } + + /* + * r715 pushdec witness: name the branch + * the entry takes straight after the + * read, before the rescue/hold paths. + * Session 124 had an entry consumed + * between the read (rc=0 src=4) and the + * decode with no row naming why; this + * line makes the branch and the content + * (first 8 bytes) visible on every read. + * Budget: 512 rows per client (reset in + * svc_session_flush). + */ + if (gsp_sema_dbg && svc_pushdec_log < 2048) { + svc_pushdec_log++; + pr_info("sema-pushdec: h=0x%x va=0x%llx len=%u src=%u sub=%u pa=0x%llx plausible=%d injected=%d ipin=%d zero_nop=%d displaced=%d unreadable=%d first8=%08x %08x\n", + ch->h_object, va, len, + (unsigned)svc_read_last_src, + svc_trust_src, + svc_read_last_pa, + svc_method_stream_plausible(pb, len) ? 1 : 0, + injected ? 1 : 0, + have_ipin ? 1 : 0, + zero_nop ? 1 : 0, + displaced ? 1 : 0, + unreadable ? 1 : 0, + len >= 4 ? *(const u32 *)pb : 0, + len >= 8 ? *(const u32 *)(pb + 4) : 0); + } + + /* * Content rescue before any fake-release * fallback (CE channels): the push bytes * are in guest RAM even when the VA->PA @@ -20700,6 +45850,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!unreadable) { ch->push_hold_until = 0; + svc_gr_setup_hold_clear(ch); /* * The entry words repeated, but the * push bytes may not have: a client @@ -20807,20 +45958,86 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) pr_info("vgsp-gr: GR rehold capture idx=%llu va=0x%llx\n", (unsigned long long)idx, va); } - if (!cap_trust && !entry_replay && + /* + * SESSION 149 R15 + * (M1.1a): a rewind + * replay whose slot + * identity AND content + * hash both match the + * recorded pass is the + * same push re-read + * (entry words alone + * cannot tell it from a + * rewritten push - the + * hash can, and the + * r354 checks above + * cleared entry_replay + * on every mismatch, so + * a surviving replay + * plus the explicit hash + * compare here is the + * verified case). + * Admit it as trusted so + * the GL section / + * synthesis capture can + * still be made when the + * first pass never + * captured (the r883 + * one-shot setup push + * decoded exactly once, + * as a replay, and the + * old suppression threw + * that single chance + * away). Ops stay + * suppressed: + * entry_replay keeps + * them off at the + * decode call, and the + * capture sites hold + * their own exactly-once + * guards. + */ + if (!cap_trust && !ch->engine_type && gsp_phys_gr_forward_enabled() && !unreadable && svc_method_stream_plausible(pb, - len)) - cap_trust = true; + len)) { + if (!entry_replay || + (entry_ident && + ch->slot_ops_hash[idx] == + svc_push_hash(pb, len))) + cap_trust = true; + } svc_entry_outcome("decode", ch, idx, va, len); svc.op_tr_pending = 0; + svc.op_owed = 0; + /* + * r595: a GR-class channel + * (engine_type 0) carries the + * driver's own relocation / + * H2D copies as well as its + * launches, but ops were only + * allowed for CE channels (or + * an injected push page). The + * copies then never executed + * and their releases were + * applied by the hold path - + * exactly the r568 b-copy + * class. Admit a GR push's + * ops when its content is a + * plausible method stream + * (cap_trust is that same + * content arbiter, set just + * above), never on a replay + * or a rehold. + */ svc_decode_stream(vgpu, pb, len, 0, - (allow_ops || injected) && + (allow_ops || injected || + cap_trust) && !entry_replay && !rehold, ch, @@ -20852,8 +46069,20 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) entry_ident; ch->slot_ops_done[idx] = entry_done; + /* + * r540: an owed + * denied op keeps + * the slot + * incomplete too - + * otherwise every + * later pass is a + * replay and the + * op can never run + * (see op_owed). + */ ch->slot_ops_complete[idx] = - svc.op_tr_pending ? + (svc.op_tr_pending || + svc.op_owed) ? 0 : 1; ch->slot_ops_hash[idx] = svc_push_hash(pb, @@ -20878,7 +46107,28 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * data), so the hold and the * deferral move together. */ + /* + * r558: the opdef hold is the one + * pump hold path without the + * absolute per-launch bound. r557 + * STALL-DUMP evidence: the pump + * consumed NOTHING for 20+ s while + * held/opdef counters climbed + * (opdef=3880/16/64, held=155) - + * a held entry whose operand never + * resolves was re-held every pass + * and the whole pump froze on it. + * Consult the same absolute + * per-launch-VA bound the GR hold + * paths use (svc_gr_hold_bound_ + * expired, one-shot r509): once it + * fires the entry flushes and is + * consumed instead of held again. + */ if (svc.op_tr_pending && + !svc_gr_hold_bound_expired(ch, + svc_gr_launch_va(ch)) && + !svc_hold_bypass() && svc_opdef_hold(vgpu, ch, (u32)idx, va)) { @@ -20911,7 +46161,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) kfree(pb); break; } - if (svc.op_tr_pending) + if (svc.op_tr_pending) { /* * Deadline hit: the * operand never @@ -20923,6 +46173,34 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ svc_opdef_flush(vgpu, ch, (u32)idx); + /* + * r505b: mark the slot's + * ops complete. The op was + * permanently skipped, but + * the decode above stored + * complete=0 (op_tr_pending + * was set), so every pin- + * batch rewind replay of + * this slot re-decoded it, + * re-set op_tr_pending and + * re-held the entry in a + * livelock (multikernel on + * r505: opdef held=18359, + * expired=61, the client + * waiting forever while the + * pump looped on slot 1). + * Nothing is owed after the + * flush, so a replay must + * skip the ops. + */ + if ((u32)idx < + SVC_GPFIFO_MAX_BITS) { + ch->slot_ops_complete[idx] = + 1; + ch->slot_ops_done[idx] = + entry_done; + } + } rel_seen = (svc_rel_count != rel_n_before); { bool fwd_guest_queued; @@ -21010,7 +46288,8 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch); if (fwd_guest_queued) { svc.fwd_gr_defer++; - } else if (!svc_gr_hold_bound_expired(ch, + } else if (!svc_hold_bypass() && + !svc_gr_hold_bound_expired(ch, svc_gr_launch_va(ch)) && (ch->gr_hold_idx != (u32)idx || time_before(jiffies, @@ -21053,6 +46332,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_hold_idx = (u32)idx; ch->gr_hold_until = jiffies + msecs_to_jiffies(hold_ms); + /* + * r633: this break leaves + * before the release block - + * park this pass's records + * with the launch so the + * eventual forward's hold + * delivers them. + */ + if (svc_fwd_defer_armed && + svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, + ch->h_object, + svc_gr_cand.push_va, + svc_fwd_rel, + svc_fwd_rel_n); svc.fwd_gr_defer++; if (svc_busy_defer_log < 64) { svc_busy_defer_log++; @@ -21104,6 +46398,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!fwd_guest_queued && !ch->engine_type && svc_gr_data_stalled && + !svc_hold_bypass() && !svc_gr_hold_bound_expired(ch, svc_gr_launch_va(ch)) && (ch->gr_hold_idx != (u32)idx || @@ -21129,6 +46424,25 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_hold_idx = (u32)idx; ch->gr_hold_until = jiffies + msecs_to_jiffies(hold_ms); + /* + * r633: this break leaves + * before the release block - + * park this pass's records + * with the launch so the + * eventual forward's hold + * delivers them (the r632 + * mail hang: the vadd push's + * tracking release was lost + * here and the client spun + * at cudaDeviceSynchronize). + */ + if (svc_fwd_defer_armed && + svc_fwd_rel_n > 0) + svc_grpark_put(vgpu, + ch->h_object, + svc_gr_cand.push_va, + svc_fwd_rel, + svc_fwd_rel_n); svc.fwd_gr_defer++; if (svc_gr_hold_log < 16) { svc_gr_hold_log++; @@ -21158,10 +46472,15 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) if (!fwd_guest_queued && !ch->engine_type && svc_gr_data_stalled && - svc_gr_hold_bound_expired(ch, - svc_gr_launch_va(ch)) && + ch->gr_hold_bound_hit && !ch->gr_hold_exp_logged) { ch->gr_hold_exp_logged = true; + /* r509: the latch is this pass's only + * report of the expiry; clear it so the + * NEXT expiry cycle (gr_hold_exp_logged + * now set) falls through to the + * historical consume. */ + ch->gr_hold_bound_hit = false; svc.fwd_gr_hold_exp++; /* * r351: the bound firing used to @@ -21309,11 +46628,83 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) * through to the historical * consume so the channel can * never livelock. + * + * r470: an ALL-ZERO push page is + * the same "not the client's work + * yet" condition - the ring entry + * is written (the previous + * session's or the client's own + * slot reuse) but the push + * CONTENT has not landed, so the + * decode sees no QMD and the + * entry would be consumed as a + * no-op; the end-of-pass counter + * publish then advances the + * guest's completion word while + * the real push was never read, + * and the client drains a + * staging buffer that still holds + * the PREVIOUS copy's bytes + * (streams_events: h_c[2] = c0 + * because the c2 D2H's + * completion was published 4.5 s + * before its copy ran, the ring + * slots having been decoded as + * all-zero pushes). Deliberately + * NARROWER than the CE-shaped + * plausibility verdict: compute + * pushes legitimately fail that + * matcher, so only an entirely + * zero page counts as unwritten. */ + { + bool all_zero = false; + + if (!ch->engine_type && + va && len) { + u32 zi; + + all_zero = true; + for (zi = 0; zi < len; + zi++) + if (pb[zi]) { + all_zero = + false; + break; + } + } + /* + * r471: publish clamp bookkeeping. + * A zero-content GR entry that is + * held or consumed must never let + * the end-of-pass counter publish + * advance the guest's completion + * word past it (the client would + * satisfy its readback wait before + * the real push/copy ran - the + * staging buffer still holds the + * previous copy's bytes). Record + * the lowest zero slot; a decode + * that reads non-zero content + * clears the clamp. + */ + if (!ch->engine_type) { + if (all_zero) { + if (!ch->gr_bad_put || + (u32)idx + 1 < + ch->gr_bad_put) + ch->gr_bad_put = + (u32)idx + 1; + } else if (va && len) { + ch->gr_bad_put = 0; + } + } if (!ch->engine_type && va && len && gsp_phys_gr_forward_enabled() && - svc_gr_qmd_seen > 0 && - !entry_replay && + (svc_gr_qmd_seen > 0 || + (all_zero && !zero_nop)) && + (!entry_replay || + svc_gr_cand_push_owed(va)) && !(svc_gr_cand.valid && svc_gr_cand.h_object == ch->h_object && @@ -21329,12 +46720,34 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) 1 : 0; ch->gr_nocap_since = jiffies; + /* + * SESSION 153-CONTINUED-54: an + * all-zero setup push gets the + * same 4K-page demand + pending + * hold as the unreadable one. + */ + if (svc_gr_setup_push(va, len, pb)) + svc_gr_setup_hold_arm(ch, + va, + (u32)idx, + len); } - if (gsp_sema_gr_nocap_hold_ms && + { + u32 nocap_ms = + gsp_sema_gr_nocap_hold_ms; + + if (svc_gr_setup_push(va, len, pb) && + gsp_sema_gr_setup_hold_ms > + nocap_ms) + nocap_ms = + gsp_sema_gr_setup_hold_ms; + if (nocap_ms && + !ch->gr_ram_ring && time_before(jiffies, ch->gr_nocap_since + msecs_to_jiffies( - gsp_sema_gr_nocap_hold_ms))) { + nocap_ms)) && + !svc_hold_bypass()) { /* Force a fresh capture on * the retry: a replay decode * would suppress it. */ @@ -21344,13 +46757,15 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.gr_nocap_held++; if (svc_gr_nocap_log < 16) { svc_gr_nocap_log++; - pr_info("sema-chan: GR no-capture hold h=0x%x entry=%u va=0x%llx len=%u replay=%d qmd_seen=%u (forcing capture; window %u ms)\n", + pr_info("sema-chan: GR no-capture hold h=0x%x entry=%u va=0x%llx len=%u replay=%d owed=%d qmd_seen=%u (forcing capture; window %u ms)\n", ch->h_object, (u32)idx, va, len, ch->gr_nocap_replay, + svc_gr_cand_push_owed(va) ? + 1 : 0, svc_gr_qmd_seen, - gsp_sema_gr_nocap_hold_ms); + nocap_ms); } kfree(pb); break; @@ -21358,11 +46773,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) /* Window expired: consume * exactly as the pre-change * tail does, with a proof - * line. */ + * line. r548: a forward + * still owed at expiry is + * software-completed first so + * the entry is never consumed + * without a completion. */ + int owed_done = + svc_gr_owed_resolve(vgpu, ch, va); + svc.gr_nocap_consumed++; - pr_info_ratelimited("sema-chan: GR no-capture hold expired h=0x%x entry=%u va=0x%llx len=%u (consuming without a launch)\n", + pr_info_ratelimited("sema-chan: GR no-capture hold expired h=0x%x entry=%u va=0x%llx len=%u owed_resolved=%d (consuming without a launch)\n", ch->h_object, - (u32)idx, va, len); + (u32)idx, va, len, + owed_done); + } + } } /* * A CE push that should carry this @@ -21420,8 +46845,21 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch, idx, va, true, true, NULL); + /* + * r577: for a CE channel only a + * pool-shaped release target is + * this channel's tracking slot. A + * rescued stale page can decode a + * release to a foreign VA (r575: + * 0x12006c004 pay=60); counting + * that as the tracking release + * advanced the expectation without + * delivering to the pool. + */ rel_seen = - (svc_rel_count != rel_n2); + (svc_rel_count != rel_n2) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); /* * The rescue re-synced the * expectation down to the RAM @@ -21448,16 +46886,66 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) true, NULL); rel_seen = - (svc_rel_count != rel_n2); + (svc_rel_count != rel_n2) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); } + /* + * r581: the rescue decoded a + * release but the r577 gate + * refused it as a tracking slot + * (rel_seen false while the + * counter moved) - the content + * is provably stale/foreign, no + * work may run from it, and + * waiting the 8 s translation + * grace per entry crawls the + * client's init for minutes + * (r577/r580 boots: 'skipping + * stale entry ... after 8000 ms + * grace' every ~8 s). Backdate + * the deferral so the consume + * path delivers the pool step + * through the exact-slot + * fallback immediately. + */ + if (!rel_seen && + svc_rel_count != rel_n2 && + ch->engine_type && + ch->expected_sem_payload) + ch->defer_since = jiffies - + msecs_to_jiffies(gsp_sema_ce_grace_ms + 100); } } if (rel_seen) { - ch->last_rel_va = - svc_rel_va_seen; + /* + * r577: the exact slot every + * recovery release targets; only + * a pool-shaped VA may claim it + * for a CE channel. The r575 + * wedge adopted a foreign rescued + * release (0x12006c004) here and + * every later recovery went away + * from the client's pool + * (0x121000000 stayed at 25 while + * the client waited for 34; the + * exact-slot poke recovered it). + */ + if (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)) + ch->last_rel_va = + svc_rel_va_seen; + else if (svc_ce_foreign_log < 16) { + svc_ce_foreign_log++; + pr_info("sema-chan: foreign release VA 0x%llx refused as tracking slot h=0x%x eng=%u (recovery stays on the pool slot)\n", + svc_rel_va_seen, + ch->h_object, + ch->engine_type); + } ch->unread_va = 0; ch->unread_until = 0; } + ch->submissions++; svc.gpfifo_entries_done++; if (ch->engine_type && ch->expected_sem_payload) { @@ -21543,6 +47031,41 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) idx, va, len); svc_log_stalled_va(va); /* + * SESSION 153-CONTINUED-54: + * demand the setup push's 4K + * pages and hold it pending. + */ + if (!ch->engine_type && + svc_gr_setup_push(va, len, NULL)) + svc_gr_setup_hold_arm(ch, va, + (u32)idx, + len); + /* + * GR stale-skip clock: the + * rescue-failure skip gate below + * requires defer_since, yet every + * GR arming site is gated on a + * nonzero expected_sem_payload. A + * zero-expectation GR entry (its + * release was already delivered) + * therefore never satisfied the + * gate and was re-read on every + * pass while the channel ring + * stalled behind it (live + * evidence: one 96-byte entry + * re-read 362 times over ~18 s; + * ~8 s of ring stall per frame + * with frames at 0.045 FPS). Arm + * the clock at first sighting; the + * skip window itself stays + * governed by gsp_sema_gr_grace_ms. + */ + if (!ch->engine_type && + !ch->expected_sem_payload && + !ch->defer_since) + ch->defer_since = jiffies - + msecs_to_jiffies(gsp_sema_gr_grace_ms + 1); + /* * Open the cheap-wait window * for this entry (see the * loop-top guard) so rewind @@ -21566,6 +47089,19 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->unread_ipin_pa = ipa; } /* + * SESSION 153-CONTINUED-54: a held + * GR setup push waits cheaply for + * the whole hold window (the fresh- + * pin check still releases it the + * moment the demanded pin lands). + */ + if (svc_gr_setup_hold_active(ch, va, + (u32)idx) && + time_before(ch->unread_until, + ch->gr_setup_hold_until)) + ch->unread_until = + ch->gr_setup_hold_until; + /* * Bounded work-before-release * hold (CE): the deferred retry * probe shows the page's @@ -21691,6 +47227,8 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) /* Rescue succeeded, retry the read. */ if (!svc_read_va_push(vgpu, ch, va, pb, len)) { + bool track_rel = false; + /* * Never run work decoded from a * rescued page: the rescue matches @@ -21715,14 +47253,43 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) false, false, NULL); - if (svc_rel_count != rel_n_before) + /* + * r577: only a pool-shaped + * release counts as this CE + * channel's tracking release + * (see the norel-path comment). + * A foreign target must not + * claim the exact slot, and the + * pool step must still be + * delivered. + */ + track_rel = + (svc_rel_count != rel_n_before) && + (!ch->engine_type || + svc_pool_sem_va(svc_rel_va_seen)); + if (track_rel) ch->last_rel_va = svc_rel_va_seen; + else if (svc_rel_count != rel_n_before && + svc_ce_foreign_log < 16) { + svc_ce_foreign_log++; + pr_info("sema-chan: foreign release VA 0x%llx refused (rescue) h=0x%x eng=%u (pool step via fallback)\n", + svc_rel_va_seen, + ch->h_object, + ch->engine_type); + } } ch->submissions++; svc.gpfifo_entries_done++; - if (ch->engine_type && ch->expected_sem_payload) - ch->expected_sem_payload++; + if (ch->engine_type && + ch->expected_sem_payload) { + if (track_rel) + ch->expected_sem_payload++; + else if (!gsp_sema_fake_rel) + svc_fake_release(vgpu, ch); + else + ch->expected_sem_payload++; + } } else { svc.gpfifo_nomap++; kfree(pb); @@ -21855,7 +47422,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since && time_after(jiffies, ch->defer_since + - msecs_to_jiffies(gsp_sema_gr_grace_ms))) { + msecs_to_jiffies(gsp_sema_gr_grace_ms)) && + !svc_gr_setup_hold_active(ch, va, + (u32)idx)) { /* * The slot no longer holds this * push (overwritten buffer): the @@ -21900,6 +47469,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->defer_since = 0; ch->unread_va = 0; ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); /* * m177: deliver any * withheld set even @@ -21999,6 +47569,7 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_nocap_va = 0; ch->unread_va = 0; ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); /* GRFIX FIX1c (r214): do NOT reset the per-launch * bound record here - a consume of an unrelated entry * would re-arm the held launch's bound every pass @@ -22026,8 +47597,9 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->userd_pa + SVC_USERD_GP_PUT, ch->userd_as, &pnow, sizeof(pnow)); svc.counter_pub_dbg++; - pr_info("sema-chan: gr-get-back h=0x%x wrote=%u read=%u put=%u\n", - ch->h_object, g, grad, pnow); + pr_info("sema-chan: gr-get-back h=0x%x userd=0x%llx/%u wrote=%u read=%u put=%u\n", + ch->h_object, ch->userd_pa, + ch->userd_as, g, grad, pnow); } } @@ -22073,6 +47645,86 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].userd_pa = ch->userd_pa; svc.chan[k].userd_as = ch->userd_as; svc.chan[k].userd_ok = ch->userd_ok; + /* r496b: persist the RAM-scan cursor and the + * one-shot USERD-scan latch so they survive + * the per-pass snapshot copy. */ + svc.chan[k].gr_ram_cursor = ch->gr_ram_cursor; + svc.chan[k].gr_ram_passes = ch->gr_ram_passes; + svc.chan[k].gr_ram_logged = ch->gr_ram_logged; + /* r912: the guest-RAM USERD scan cursor (also + * persisted immediately inside the scan - the + * stalled path skips this write-back). */ + svc.chan[k].gr_ram_userd_cursor = + ch->gr_ram_userd_cursor; + svc.chan[k].gr_ram_userd_passes = + ch->gr_ram_userd_passes; + svc.chan[k].gr_ram_userd_logged = + ch->gr_ram_userd_logged; + /* SESSION 154 (M-GL1.2): liveness watch + the + * free-running-put seed + the push anchor + * cursor all survive the snapshot copy. */ + { + u32 wj; + + svc.chan[k].gr_ram_userd_watch_n = + ch->gr_ram_userd_watch_n; + svc.chan[k].gr_ram_userd_watch_log = + ch->gr_ram_userd_watch_log; + svc.chan[k].gr_ram_userd_kick_sweeps = + ch->gr_ram_userd_kick_sweeps; + for (wj = 0; wj < SVC_GR_USERD_WATCH; + wj++) { + svc.chan[k].gr_ram_userd_watch[wj] = + ch->gr_ram_userd_watch[wj]; + svc.chan[k].gr_ram_userd_watch_put[wj] = + ch->gr_ram_userd_watch_put[wj]; + svc.chan[k].gr_ram_userd_watch_hits[wj] = + ch->gr_ram_userd_watch_hits[wj]; + svc.chan[k].gr_ram_userd_watch_at[wj] = + ch->gr_ram_userd_watch_at[wj]; + } + } + svc.chan[k].gr_ram_put_seeded = + ch->gr_ram_put_seeded; + svc.chan[k].gr_ram_userd_latch_put = + ch->gr_ram_userd_latch_put; + svc.chan[k].gr_ram_userd_latch_at = + ch->gr_ram_userd_latch_at; + svc.chan[k].gr_ram_userd_confirmed = + ch->gr_ram_userd_confirmed; + /* CONT.407: keep-across-put-reset state. */ + svc.chan[k].gr_ram_userd_keeps = + ch->gr_ram_userd_keeps; + svc.chan[k].gr_ram_userd_keep_log = + ch->gr_ram_userd_keep_log; + /* SESSION 153-CONT-5: rejected-latch memory. */ + svc.chan[k].gr_ram_userd_reject_pa = + ch->gr_ram_userd_reject_pa; + svc.chan[k].gr_ram_userd_reject_put = + ch->gr_ram_userd_reject_put; + svc.chan[k].gr_ram_userd_rejects = + ch->gr_ram_userd_rejects; + svc.chan[k].gr_anchor_cursor = ch->gr_anchor_cursor; + svc.chan[k].gr_anchor_va = ch->gr_anchor_va; + svc.chan[k].gr_anchor_sweeps = ch->gr_anchor_sweeps; + svc.chan[k].gr_anchor_logged = ch->gr_anchor_logged; + /* SESSION 149 R15: M1.1b keep-alive state. */ + svc.chan[k].gr_ram_kick = ch->gr_ram_kick; + svc.chan[k].gr_ram_refreshes = + ch->gr_ram_refreshes; + svc.chan[k].gr_ram_refresh_at = + ch->gr_ram_refresh_at; + svc.chan[k].gr_ram_ring = ch->gr_ram_ring; + svc.chan[k].gr_kick_ring_reval_at = + ch->gr_kick_ring_reval_at; + svc.chan[k].userd_scans = ch->userd_scans; + svc.chan[k].fb_userd_scan_at = + ch->fb_userd_scan_at; + svc.chan[k].fb_userd_scan_n = + ch->fb_userd_scan_n; + svc.chan[k].notify2d_va = ch->notify2d_va; + svc.chan[k].zero_held = ch->zero_held; + svc.chan[k].userd_reg = ch->userd_reg; svc.chan[k].logged = ch->logged; svc.chan[k].logged_expected = ch->logged_expected; svc.chan[k].notified = ch->notified; @@ -22099,6 +47751,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) svc.chan[k].push_hold_until = ch->push_hold_until; svc.chan[k].defer_since = ch->defer_since; svc.chan[k].put_prev = ch->put_prev; + /* r471: the publish clamp watermark is set + * and cleared on the pump thread; persist it + * so the next pass still clamps. */ + svc.chan[k].gr_bad_put = ch->gr_bad_put; /* * Persist the cheap-wait window and the content * watermark: the pump works on a snapshot, so @@ -22146,6 +47802,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) ch->gr_hold_deadline; svc.chan[k].gr_hold_exp_logged = ch->gr_hold_exp_logged; + /* r509: the one-shot expiry latch is snapshot + * state too. */ + svc.chan[k].gr_hold_bound_hit = + ch->gr_hold_bound_hit; /* r361: the opdef absolute-bound record is * snapshot state too (the GRFIX lesson) - * without the write-back the deadline would @@ -22280,6 +47940,10 @@ static void svc_pump_channels(struct nv_vgpu_device *vgpu) */ if (stalled) svc_apply_sweep_slice(vgpu); + + /* r590: the pass snapshot is only valid while the loop runs. */ + svc_pass_local = NULL; + svc_pass_local_n = 0; } /* ------------------------------------------------------------------ */ @@ -22474,12 +48138,13 @@ static void svc_handle_chan_dump(struct nv_vgpu_device *vgpu) svc_read_ap(vgpu, ch->notifier_pa + 16 + 8, ch->notifier_as, &info1, 4); } - pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", + pr_info("sema-chan-dump: h=0x%x eng=%u gpfifo=0x%llx ent=%u userd=0x%llx as=%u ok=%d get=%u sub=%llu exp=%u rel=0x%llx crel=0x%llx armed=%d notif=0x%llx as=%u slot0{info=0x%x st=0x%x} slot1{info=0x%x st=0x%x} notified=%llu\n", ch->h_object, ch->engine_type, ch->gpfifo_va, ch->gpfifo_entries, ch->userd_pa, ch->userd_as, ch->userd_ok, ch->last_get, ch->submissions, ch->expected_sem_payload, ch->last_rel_va, ch->counter_pa, + ch->gr_pub_armed, ch->notifier_pa, ch->notifier_as, info0, st0, info1, st1, ch->notified); @@ -22521,6 +48186,23 @@ int gsp_sema_service_dump_channels(void) #define SVC_PIN_REWIND 8 #define SVC_MAP_ADD_MAX 4096 +/* + * SESSION 153-CONTINUED-84: optional length guard for injected mappings. + * A stale producer on the host (observed 2026-11-01: a periodic writer + * enqueueing (0x4001004,0x11f168000,0x10000) every ~2 s from an unknown + * leftover process) re-applies a whole-64K contiguous mapping over the + * per-page SA-carrier pins and can win the read of the client's push + * region, leaving every page past the first reading zeros (r1240/r1260: + * the replayed prelude captured only the first 15 SA slots while r1236/ + * r1245/r1250 captured all 55). When set, requests longer than this many + * bytes are refused with a bounded log, so only per-page (4K) pins land. + * 0 = accept any length (the historical behaviour). + */ +static u32 gsp_sema_map_add_max; +module_param_named(sema_map_add_max, gsp_sema_map_add_max, uint, 0644); +MODULE_PARM_DESC(sema_map_add_max, + "Refuse injected mapping requests longer than this many bytes (0 = off; set 4096 to defeat stale whole-window injections)"); + struct svc_map_add_req { u64 va; u64 pa; @@ -22568,17 +48250,32 @@ static bool svc_va_hits_chan_ring(u64 va) return hit; } +/* + * r561: the drain is BOUNDED per call. r559 soak evidence (victory_saxpy, + * 167 s at 100% CPU, pump alive but consuming nothing): the guest pin + * stream flooded the injection queue (svc_handle_map_add: 1867-6101 + * callbacks suppressed per burst) and the old unbounded for(;;) drain + * spent every service pass processing injections - the ring walk never + * ran, no GPFIFO entry was consumed, and the client spun. The queue is + * drained again on every pass (and from svc_gr_request / svc_run_ops), so + * a bounded batch loses nothing; it just guarantees the ring walk gets + * its cycles under an injection flood. + */ +#define SVC_MAP_ADD_BATCH 64u static void svc_handle_map_add(void) { unsigned long flags; bool rewound = false; u32 i; + u32 batch = 0; for (;;) { struct svc_map_add_req req; u32 ap; u64 off; + if (batch++ >= SVC_MAP_ADD_BATCH) + break; spin_lock_irqsave(&svc_map_add_lock, flags); if (svc_map_add_tail == svc_map_add_head) { spin_unlock_irqrestore(&svc_map_add_lock, flags); @@ -22589,6 +48286,24 @@ static void svc_handle_map_add(void) svc_map_add_tail++; spin_unlock_irqrestore(&svc_map_add_lock, flags); + /* + * SESSION 153-CONTINUED-84: refuse oversized requests when + * the guard is set (see the sema_map_add_max comment) - the + * page pins from pin_loop2 (len 0x1000) are unaffected. + */ + if (READ_ONCE(gsp_sema_map_add_max) && + req.len > (u64)READ_ONCE(gsp_sema_map_add_max)) { + static u32 map_rej_log; + + if (map_rej_log < 8) { + map_rej_log++; + pr_info("sema-svc: injected mapping va=0x%llx pa=0x%llx len=0x%llx REFUSED (len > sema_map_add_max=%u)\n", + req.va, req.pa, req.len, + READ_ONCE(gsp_sema_map_add_max)); + } + continue; + } + ap = req.ap ? req.ap : SVC_APERTURE_SYS; for (off = 0; off < (req.len ? req.len : 1); off += SEMA_PAGE_SIZE) { @@ -22612,7 +48327,7 @@ static void svc_handle_map_add(void) */ if (svc_va_hits_chan_ring(va)) { svc_ring_pin_skips++; - if (svc_ring_pin_skips <= 8) + if (svc_ring_pin_skips <= 64) pr_info("sema-svc: injected pin va=0x%llx DROPPED (live channel ring range; ring reads stay on the channel's own translation)\n", va); continue; @@ -22766,6 +48481,262 @@ int gsp_sema_service_add_map(u64 gpu_va, u64 pa, u64 len, u32 ap) } /* + * X/GL frame-pacer mirror (SESSION 148 perf, proven on r861): the X + * server's present path waits for a GPU-progress word - a lone u32 + * (format 0x14 | base nibble at bits 16..19 | consumed-dword index in + * bits 0..17) in its own guest-RAM page - to reach its per-frame + * submitted-byte counter ((word & 0x3ffff) << 2 == counter). The + * emulation does not execute the 2D method streams, so nothing ever + * advances the word: every frame paid the driver's full ~10 s wait + * timeout (xorg.log "(EE) NVIDIA(GPU-0): WAIT (2, 8, 0x8000, 0x8d4, + * 0x<growing>)" every 10 s -> glxgears 0.09 FPS). A gdb poke of the + * word to (0x140000 | counter >> 2) made the wait complete in 3 ms + * ("WAIT (0, ... <equal>, <equal>)" appears ms after the timeout print) + * and silenced all further timeouts for the session. + * + * This kthread mirrors the driver's counter word (guest PA cnt_pa, a + * u32 in the X process's channel object that advances by the frame's + * submitted bytes) into the progress word (guest PA mem_pa) every few + * ms, preserving the word's high bits. Both PAs are discovered per boot + * (content/handle search in guest RAM) and injected with the + * sema_xw_mirror parameter; "0,0" stops the thread. Pure tuning aid for + * the 1.0.0 demo path - no product logic depends on it. + */ +static u64 svc_xw_cnt_pa; +static u64 svc_xw_mem_pa; +static struct task_struct *svc_xw_task; + +static int svc_xw_mirror_loop(void *unused) +{ + u32 orig_top = 0; + u32 ticks = 0; + + while (!kthread_should_stop()) { + struct nv_vgpu_device *d, *vgpu = NULL; + u32 c = 0, cur = 0; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (vgpu && + !svc_read_ap(vgpu, svc_xw_cnt_pa, SVC_APERTURE_SYS, + &c, sizeof(c)) && + !svc_read_ap(vgpu, svc_xw_mem_pa, SVC_APERTURE_SYS, + &cur, sizeof(cur))) { + /* + * SESSION 148 (r867): member = counter (dword index, + * high bits preserved). The plain tracking law is + * the correct one: the driver's re-check cadence + * (~3 ms-scale re-probes) gives the mirror room to + * settle on the new counter between the driver's + * increment and its first probe. + * + * SESSION 148 r873 page-reuse guard: the word page + * is only ours while the driver's progress word + * lives there. When the page is freed/reused (X + * server died, batch BO recycled) its content + * changes and the base bits (bits 16..23 = 0x10 or + * 0x14) vanish. r872 evidence: the mirror kept + * writing 0x26d into a page that had become another + * process's PTE table (guest "Bad page map ... + * pmd:102656067" pointing at exactly the mirrored + * page's PA), corrupting unrelated guest memory. + * NEVER write once the base bits have changed; stop + * the thread so nothing more touches the page. + */ + u32 top = cur & 0xfffc0000u; + + if (!orig_top) { + if (!top) { + /* not a progress word: write nothing */ + msleep(3); + continue; + } + orig_top = top; + } + if (top != orig_top) { + pr_warn("sema-xw: mirror STOP: word page 0x%llx repurposed (top 0x%x -> 0x%x)\n", + (unsigned long long)svc_xw_mem_pa, + orig_top, cur); + break; + } + if (!(++ticks & 63)) { + /* counter-side guard: channel signature */ + u64 sig = 0; + + if (svc_read_ap(vgpu, svc_xw_cnt_pa - 0x50, + SVC_APERTURE_SYS, &sig, + sizeof(sig)) || + sig != 0x0100001300000001ULL) { + pr_warn("sema-xw: mirror STOP: counter object 0x%llx repurposed (sig=0x%llx)\n", + (unsigned long long)svc_xw_cnt_pa, + (unsigned long long)sig); + break; + } + } + { + u32 m = top | ((c >> 2) & 0x3ffffu); + u16 st = 0; + + if (m != cur) + svc_write_ap(vgpu, svc_xw_mem_pa, + SVC_APERTURE_SYS, &m, + sizeof(m)); + /* + * SESSION 148 r873: submit-kick completion. + * The X submit path arms member status + * 0x8000 at value_word+0xE (gdb-proven layout: + * value @member+0, status @member+0xE) and + * polls until it reads 0; nothing else ever + * clears it, so each kick paid the driver's + * full timeout. Clear it here, right after + * the value write, so the poll passes ms + * after the frame is submitted. + */ + if (!svc_read_ap(vgpu, svc_xw_mem_pa + 0xE, + SVC_APERTURE_SYS, &st, + sizeof(st)) && + st == 0x8000) { + u16 z = 0; + + svc_write_ap(vgpu, svc_xw_mem_pa + 0xE, + SVC_APERTURE_SYS, &z, + sizeof(z)); + if (!(++ticks % 256)) + pr_info("sema-xw: mirror alive (clears=%u cnt=0x%x)\n", + ticks, c); + } + } + } + msleep(3); + } + svc_xw_task = NULL; + return 0; +} + +/* + * X/GL submit-kick completion clearer (SESSION 148 r873). The X server's + * submit path (decoded at nvidia_drv dc170) arms a 16-byte member - + * u32 value at +0x0 (0 while pending) and u16 status 0x8000 at +0xE - and + * then polls dc330 until EVERY member status reads 0; nothing in the + * emulation ever writes that completion, so each kick paid the driver's + * full retry/timeout cycle. The member lives in a page whose page offset + * is 0xB0 (membase is page-aligned), so the status word is at page + * offset 0xBE. The finder in guest RAM reports that PA during an armed + * window; this thread clears status 0x8000 -> 0 (and only that exact + * pattern) so the kick poll passes within a few ms. Pure tuning aid - + * no product logic depends on it. + */ +static u64 svc_xw_comp_pa; /* PA of the u16 status word (page off 0xBE) */ +static struct task_struct *svc_xw_comp_task; + +static int svc_xw_comp_loop(void *unused) +{ + u64 n = 0; + + while (!kthread_should_stop()) { + struct nv_vgpu_device *d, *vgpu = NULL; + u32 val = 1; + u16 st = 0; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (vgpu && + !svc_read_ap(vgpu, svc_xw_comp_pa - 0xE, + SVC_APERTURE_SYS, &val, sizeof(val)) && + !svc_read_ap(vgpu, svc_xw_comp_pa, SVC_APERTURE_SYS, + &st, sizeof(st))) { + /* + * Clear ONLY the exact pending pattern (value still + * 0, status 0x8000). Anything else is either a + * completed member (status 0) or a repurposed page - + * leave it alone. + */ + if (st == 0x8000 && val == 0) { + u16 z = 0; + + svc_write_ap(vgpu, svc_xw_comp_pa, + SVC_APERTURE_SYS, &z, sizeof(z)); + if (!(++n % 25)) + pr_info("sema-xw: comp clears=%llu\n", + (unsigned long long)n); + } + } + msleep(2); + } + svc_xw_comp_task = NULL; + return 0; +} + +int gsp_sema_service_xw_comp(u64 comp_pa) +{ + if (!comp_pa) { + if (svc_xw_comp_task) { + kthread_stop(svc_xw_comp_task); + svc_xw_comp_task = NULL; + pr_info("sema-xw: comp stopped\n"); + } + return 0; + } + if (comp_pa & 1) + return -EINVAL; + svc_xw_comp_pa = comp_pa; + if (!svc_xw_comp_task) { + svc_xw_comp_task = kthread_run(svc_xw_comp_loop, NULL, + "vgsp-xwcomp"); + if (IS_ERR(svc_xw_comp_task)) { + int err = PTR_ERR(svc_xw_comp_task); + + svc_xw_comp_task = NULL; + return err; + } + } + pr_info("sema-xw: comp status_pa=0x%llx running\n", + (unsigned long long)comp_pa); + return 0; +} + +int gsp_sema_service_xw_mirror(u64 cnt_pa, u64 mem_pa) +{ + if (!cnt_pa || !mem_pa) { + if (svc_xw_task) { + kthread_stop(svc_xw_task); + svc_xw_task = NULL; + pr_info("sema-xw: mirror stopped\n"); + } + return 0; + } + if ((cnt_pa | mem_pa) & 0x3) + return -EINVAL; + svc_xw_cnt_pa = cnt_pa; + svc_xw_mem_pa = mem_pa; + if (!svc_xw_task) { + svc_xw_task = kthread_run(svc_xw_mirror_loop, NULL, + "vgsp-xwmirror"); + if (IS_ERR(svc_xw_task)) { + int err = PTR_ERR(svc_xw_task); + + svc_xw_task = NULL; + return err; + } + } + pr_info("sema-xw: mirror cnt=0x%llx -> mem=0x%llx running\n", + (unsigned long long)cnt_pa, (unsigned long long)mem_pa); + return 0; +} + +/* * Dump a range of the emulated vidmem (BAR1 shadow) into the kernel log. * Debug aid for the CE data path: verifies whether a decoded fill/copy * actually landed where the CE operand said it should. @@ -22851,6 +48822,138 @@ int gsp_sema_service_find_vidmem(u64 pattern, u64 max_hits) } /* + * Whole-vidmem page-hash bracket. Snapshot a hash of every 4 KB page of the + * first open device's BAR1 shadow, then re-hash later and report the pages + * whose content changed. This proves whether an engine wrote any page of the + * emulated vidmem, including pages the output copy-out does not sample. + * The hash array is allocated once per shadow size and kept until the + * service stops. + */ +static DEFINE_MUTEX(gsp_sema_vidmem_hash_lock); +static u64 *gsp_sema_vidmem_hash; +static unsigned int gsp_sema_vidmem_hash_pages; + +static inline u64 gsp_sema_vidmem_fnv1a64(const u8 *p, size_t len) +{ + u64 h = 0xcbf29ce484222325ULL; + size_t i; + + for (i = 0; i < len; i++) { + h ^= p[i]; + h *= 0x100000001b3ULL; + } + return h; +} + +/* + * Resolve the shadow to bracket: the first mdev a guest has open, the same + * resolution the vidmem debug helpers use. Only whole 4 KB pages inside + * bars[1] are hashed. + */ +static int gsp_sema_vidmem_hash_target(u8 **base, u64 *size) +{ + struct nv_vgpu_device *vgpu = NULL; + struct nv_vgpu_device *d; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (!vgpu || !vgpu->bars[1].vaddr || !vgpu->bars[1].size) + return -ENODEV; + *base = vgpu->bars[1].vaddr; + *size = vgpu->bars[1].size; + return 0; +} + +int gsp_sema_service_vidmem_hash_snapshot(void) +{ + u8 *base; + u64 size; + unsigned int pages, i; + int ret; + + ret = gsp_sema_vidmem_hash_target(&base, &size); + if (ret) + return ret; + pages = (unsigned int)(size >> SEMA_PAGE_SHIFT); + if (!pages) + return -ENODEV; + + mutex_lock(&gsp_sema_vidmem_hash_lock); + if (!gsp_sema_vidmem_hash || gsp_sema_vidmem_hash_pages != pages) { + u64 *hashes; + + hashes = vmalloc((size_t)pages * sizeof(*hashes)); + if (!hashes) { + ret = -ENOMEM; + goto out_unlock; + } + vfree(gsp_sema_vidmem_hash); + gsp_sema_vidmem_hash = hashes; + gsp_sema_vidmem_hash_pages = pages; + } + for (i = 0; i < pages; i++) + gsp_sema_vidmem_hash[i] = gsp_sema_vidmem_fnv1a64( + base + ((u64)i << SEMA_PAGE_SHIFT), SEMA_PAGE_SIZE); + ret = 0; +out_unlock: + mutex_unlock(&gsp_sema_vidmem_hash_lock); + return ret; +} + +int gsp_sema_service_vidmem_hash_diff(const char *tag, unsigned int max_pages) +{ + static DEFINE_RATELIMIT_STATE(detail_rs, 5 * HZ, 4); + u8 *base; + u64 size; + unsigned int pages, i, changed = 0; + const char *name = tag ? tag : "diff"; + bool detail; + int ret; + + ret = gsp_sema_vidmem_hash_target(&base, &size); + if (ret) + return ret; + pages = (unsigned int)(size >> SEMA_PAGE_SHIFT); + if (!pages) + return -ENODEV; + if (max_pages > 256) + max_pages = 256; + + mutex_lock(&gsp_sema_vidmem_hash_lock); + if (!gsp_sema_vidmem_hash || gsp_sema_vidmem_hash_pages != pages) { + mutex_unlock(&gsp_sema_vidmem_hash_lock); + return -EINVAL; + } + detail = __ratelimit(&detail_rs); + for (i = 0; i < pages; i++) { + const u8 *p = base + ((u64)i << SEMA_PAGE_SHIFT); + u64 h = gsp_sema_vidmem_fnv1a64(p, SEMA_PAGE_SIZE); + + if (h == gsp_sema_vidmem_hash[i]) + continue; + changed++; + if (detail && changed <= max_pages) { + u64 w0; + + memcpy(&w0, p, sizeof(w0)); + pr_info("vgsp-gr: VIDMEM-BRACKET page=0x%x off=0x%llx w0=0x%016llx hash=0x%016llx\n", + i, (unsigned long long)i << SEMA_PAGE_SHIFT, + w0, h); + } + } + mutex_unlock(&gsp_sema_vidmem_hash_lock); + pr_info_ratelimited("vgsp-gr: VIDMEM-BRACKET %s changed=%u/%u\n", + name, changed, pages); + return (int)changed; +} + +/* * Register a channel's GPU completion counter (guest PA of the u32 the GPU * would advance as pushes complete). Called from the sema_counter debug * parameter; plain state update, so it runs in the caller's context. @@ -22863,7 +48966,7 @@ int gsp_sema_service_counter_add(u32 h_object, u64 pa) if (!svc.task) return -ENODEV; /* The counter is a u32: require a 4-byte aligned guest PA. */ - if (!h_object || !pa || (pa & 0x3) || pa >= SEMA_PA_MAX) + if (!h_object || !pa || (pa & 0x3) || pa >= svc_pa_max()) return -EINVAL; spin_lock_irqsave(&svc.chan_lock, flags); @@ -22890,9 +48993,14 @@ int gsp_sema_service_counter_add(u32 h_object, u64 pa) /* * Register the guest PMA scrubber's channel-buffer words. Called from the * sema_scrubbuf_add debug parameter with the PAs the guest scanner resolved - * (vmallocinfo vmap area + kcore page-table walk). fin_pa is the finish - * payload the guest's scrub wait polls; hsem_pa is the channel's host - * semaphore (channelWaitForFreeEntry polls it). Plain state update. + * (vmallocinfo vmap area + page-table walk). fin_pa is the finish payload + * the guest's scrub wait polls; hsem_pa is the channel's host semaphore + * (channelWaitForFreeEntry polls it). Plain state update. + * + * CONT.409: one record per channel-buffer page. The CeUtils scrubber and + * the X DDX's own CE/PMA channel buffer can both exist in one guest, and + * the scanner re-emits when a buffer moves. fin_pa=hsem_pa=0 clears every + * scanner-registered record of the open guest (the buffer's vmap is gone). * * Dual-guest: the words are written through a specific guest's address space * (see svc_scrub_pass), so the registration binds to the FIRST open mdev - @@ -22904,13 +49012,40 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa) struct nv_vgpu_device *vgpu = NULL; struct nv_vgpu_device *d; struct svc_scrub *s; + int i; if (!svc.task) return -ENODEV; + if (!fin_pa && !hsem_pa) { + /* Clear: the scanner saw its channel-buffer vmap disappear. */ + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(d, &nv_vgpu_driver.devices, list) { + if (atomic_read(&d->vfio_dev_open_count) > 0) { + vgpu = d; + break; + } + } + mutex_unlock(&nv_vgpu_driver.lock); + if (!vgpu) + return 0; + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + if (svc_scrubs[i].vgpu != vgpu || + !svc_scrubs[i].registered) + continue; + if (svc_scrubs[i].userd_ok) { + /* Keep the stashed CeUtils geometry. */ + svc_scrubs[i].registered = false; + continue; + } + memset(&svc_scrubs[i], 0, sizeof(svc_scrubs[i])); + } + pr_info("sema-svc: scrubber buffer registration cleared\n"); + return 0; + } /* Both words are u32: require 4-byte alignment and a RAM-shaped PA. */ - if (!fin_pa || (fin_pa & 0x3) || fin_pa >= SEMA_PA_MAX) + if (!fin_pa || (fin_pa & 0x3) || fin_pa >= svc_pa_max()) return -EINVAL; - if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= SEMA_PA_MAX) + if (!hsem_pa || (hsem_pa & 0x3) || hsem_pa >= svc_pa_max()) return -EINVAL; mutex_lock(&nv_vgpu_driver.lock); @@ -22925,20 +49060,82 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa) pr_warn_ratelimited("sema-svc: scrubber buffer registration ignored: no open mdev to bind it to\n"); return -ENODEV; } - s = svc_scrub_get(vgpu, true); + s = svc_scrub_reg(vgpu, hsem_pa); if (!s) return -ENOSPC; + s->registered = true; + if (s->hsem_pa != hsem_pa) + svc_scrub_ring_reset(s); /* CONT.419: new geometry */ s->fin_pa = fin_pa; s->hsem_pa = hsem_pa; s->pa_ok = true; + s->pa_at = jiffies; + s->seeded = false; /* idle-pair authoring runs on the next pass */ s->pa_gen++; - pr_info("sema-svc: scrubber buffer registered fin_pa=0x%llx hsem_pa=0x%llx (gen %u)\n", - fin_pa, hsem_pa, s->pa_gen); + pr_info("sema-svc: scrubber buffer registered fin_pa=0x%llx hsem_pa=0x%llx (gen %u slot %d userd_ok=%d)\n", + fin_pa, hsem_pa, s->pa_gen, (int)(s - svc_scrubs), + s->userd_ok); return 0; } /* + * Resolve a scrubber channel-buffer VA to guest RAM, preferring the RM + * kernel VAS (h=0xc, the address space the guest's own CPU mapping of the + * channel buffer lives in). Several VA spaces can map the same VA: the + * client's VAS (e.g. beef0300) maps it to its own emulated vidmem object, + * and the kernel VAS maps the real channel buffer in RAM. Picking the first + * walk success (newest VAS first) therefore picked a vidmem page and the + * authored finish word never reached the guest. Returns true on a RAM hit. + */ +static bool svc_scrub_resolve_va(struct nv_vgpu_device *vgpu, u64 va, + u64 *pa, int *ap, u32 *hvas) +{ + struct svc_vaspace local[SVC_MAX_VASPACES]; + unsigned long flags; + u64 best_pa = 0; + int best_ap = SVC_APERTURE_SYS; + u32 best_h = 0; + bool found = false; + int i, n = 0; + + spin_lock_irqsave(&svc.vas_lock, flags); + for (i = 0; i < SVC_MAX_VASPACES; i++) + if (svc.vas[i].valid) + local[n++] = svc.vas[i]; + spin_unlock_irqrestore(&svc.vas_lock, flags); + + for (i = 0; i < n; i++) { + u64 wpa = 0; + int wap = SVC_APERTURE_SYS; + + if (svc_gmmu_walk(vgpu, local[i].pdb, local[i].entries, va, + local[i].ap, &wpa, &wap, false) || !wpa) + continue; + if (svc_ap_of(wap) != SVC_APERTURE_SYS) + continue; + if (local[i].h_vaspace == 0xc) { + *pa = wpa; + *ap = wap; + *hvas = local[i].h_vaspace; + return true; + } + if (!found) { + best_pa = wpa; + best_ap = wap; + best_h = local[i].h_vaspace; + found = true; + } + } + if (found) { + *pa = best_pa; + *ap = best_ap; + *hvas = best_h; + } + return found; +} + +/* * Author the CeUtils scrubber channel's progress once per service pass. * * The physical CE would advance, per completed scrub block, the channel's @@ -22960,25 +49157,273 @@ int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa) * for; correctness impact is limited to stale data in pages the guest * believes scrubbed. */ -static void svc_scrub_pass(struct nv_vgpu_device *vgpu) +static void svc_scrub_one(struct nv_vgpu_device *vgpu, struct svc_scrub *s) { - struct svc_scrub *s = svc_scrub_get(vgpu, false); u32 put = 0; u64 delta; - if (!s || !s->userd_ok || !s->pa_ok || !s->entries) + if (!s) + return; + /* + * CONT.409: a scanner-registered channel-buffer page is the vmap the + * guest RM actually polls. Do not walk the GPU VAS for it: after the + * guest RM re-creates its kernel VAS the h=0xc directory can still + * resolve a stale page (r1712: the pass kept authoring 0x126b60000 + * while the guest polled 0x124fec000, so the X scrub wait could never + * complete). Release the wait every pass (CONT.408/409/414/419): the + * sema word carries the all-ones value because 0xffffffff can never + * equal a putIndex below the ring size, while the finish word mirrors + * only real submitted counts (r1766/r1767 RCA: any fabricated delta + * underflows the guest's scrubListSize; 0xffffffff made its next PMA + * allocation request 24*0xffffffff bytes - see svc_scrub_fin_next); + * with no new submissions it writes back the word it read (a no-op). + * Idempotent when sema/fin already hold the release values. + */ + if (s->registered && !s->userd_ok) { + u32 sema = 0; + u32 fin = 0; + u32 fin_new; + u32 idle = 0xffffffffu; + bool ok = true; + + if (!s->pa_ok) + return; + /* + * CONT.419: never author the release pair into a channel's + * GPFIFO ring page (a scanner vmap candidate can register it + * as a sema/fin pair; the all-ones write clobbered GPFIFO + * entry 0 of the real channel - see svc_scrub_is_ring_page). + */ + if (svc_scrub_is_ring_page(s)) { + if (!s->ring_logged) { + s->ring_logged = true; + pr_info("sema-svc: scrub ring-page registration ignored hsem_pa=0x%llx (GPFIFO of a stashed channel)\n", + s->hsem_pa); + } + return; + } + ok &= !svc_read_ap(vgpu, s->hsem_pa, SVC_APERTURE_SYS, + &sema, sizeof(sema)); + ok &= !svc_read_ap(vgpu, s->fin_pa, SVC_APERTURE_SYS, + &fin, sizeof(fin)); + /* + * CONT.419 content fallback: a GPFIFO ring page registered as + * a sema/fin pair starts with entry 0 (push VA low dword + + * flags). A real channel sema word is a consumed entry count, + * far below the push-VA window (0x20000000) - so a "sema" that + * reads like a push VA means this page is a ring and must + * never be written. + */ + if (ok && svc_scrub_push_lo(sema)) { + if (!s->ring_logged) { + s->ring_logged = true; + pr_info("sema-svc: scrub ring-page registration ignored hsem_pa=0x%llx (entry-shaped word)\n", + s->hsem_pa); + } + return; + } + /* + * Mirror real submissions only (r1765/r1766/r1767 RCA): the + * finish payload is folded by the guest + * (ceutilsUpdateProgress) and must never carry 0xffffffff + * (24*0xffffffff-byte next allocation) nor any fabricated + * delta (guest scrubListSize underflow); the sema word keeps + * the all-ones release. + */ + fin_new = gsp_sema_scrub_ramp ? svc_scrub_fin_next(s, fin) + : idle; + if (ok && sema == idle && fin == fin_new) + return; /* already released */ + if (s->log_budget < 16) { + s->log_budget++; + pr_info("sema-svc: scrub reg author hsem_pa=0x%llx fin_pa=0x%llx (was 0x%x/0x%x)\n", + s->hsem_pa, s->fin_pa, sema, fin); + } + svc_write_ap(vgpu, s->hsem_pa, SVC_APERTURE_SYS, &idle, + sizeof(idle)); + svc_write_ap(vgpu, s->fin_pa, SVC_APERTURE_SYS, &fin_new, + sizeof(fin_new)); + if (gsp_sema_scrub_ramp) + s->fin_release = fin_new; + return; + } + /* + * r1665: the guest scanner never registered the scrubber's finish + * payload, so the X DDX's vidheap scrub wait (channelFillPbFastScrub + * / channelWaitForFinishPayload) never completed. The channel buffer + * is pb (0x64000) + GPFIFO (entries*8) + sema (4) + finish (4); the + * scrubber's gpfifo VA = pb + pbSize, so hsem = gpfifo_va + + * entries*8 and fin = hsem + 4. Resolve them through the learned map + * or the VAS walk and register in place. + */ + if (s->userd_ok && s->gpfifo_va && s->entries) { + u64 hva = s->gpfifo_va + (u64)s->entries * 8; + u64 pa = 0; + int ap = SVC_APERTURE_SYS; + u32 hv = 0; + + /* + * r1667: resolve through the GPU page tables FIRST. The learned + * VA->PA pair map can name a stale page for a reused VA (the guest + * polls its scrubber finish word through the kernel VAS h=0xc, + * which the walk resolves correctly, while svc_lookup returned an + * old page from a previous allocation and the scrub wait never + * completed). Fall back to the pair map only when no walkable VAS + * maps the VA. + * + * r1668: re-resolve on EVERY pass, not just the first. The guest + * re-creates the RM kernel VAS (h=0xc) and re-maps the scrubber + * buffer under the same GPU VA when its RM resets (observed after + * the first GZDoom run: the same gpfifo VA 0x120064000 moved from + * PA 0x127e6a000 to 0x1122e9000). A once-only resolution then + * authored a page the guest no longer polls and the scrub wait + * never completed, which in turn kept the guest's RM lock held and + * froze the status-queue consumer (the pump's event posts then + * blocked for 200 ms each, starving this very pass). + */ + if (svc_scrub_resolve_va(vgpu, hva, &pa, &ap, &hv) && pa) { + /* + * CONT.409: a scanner-registered PA is the page the + * guest RM actually polls; never replace it with a + * walk result. After the guest re-creates its kernel + * VAS the h=0xc directory can keep resolving an old + * page for the reused VA (r1712: walk said + * 0x126b60000 while the guest polled 0x124fec000), + * so adopting the walk would silently re-break the + * wait the registration just fixed. + */ + if (s->registered && pa != s->hsem_pa) { + if (s->log_budget < 24) { + s->log_budget++; + pr_info("sema-svc: scrub walk disagrees hsem_va=0x%llx walk_pa=0x%llx reg_pa=0x%llx (kept registered)\n", + hva, pa, s->hsem_pa); + } + } else if (!s->pa_ok || pa != s->hsem_pa) { + if (s->log_budget < 24) { + s->log_budget++; + pr_info("sema-svc: scrub geometry hsem_va=0x%llx fin_va=0x%llx -> hsem_pa=0x%llx fin_pa=0x%llx vas=0x%x\n", + hva, hva + 4, pa, pa + 4, hv); + } + svc_scrub_ring_reset(s); /* CONT.419 */ + s->hsem_pa = pa; + s->fin_pa = pa + 4; + } + /* + * CONT.409: the scanner may have registered this same + * page before the stash bound (the scanner runs from + * boot; the 4096-entry channel arrives later), which + * leaves a registered-only duplicate record for the same + * words. Two records authoring one page ping-pong between + * the put-based (sema = put-1) and idle (0xffffffff) + * values, and a wait that samples the wrong half can + * still block. The stash record governs; drop the + * duplicate. + */ + { + int j; + + for (j = 0; j < SVC_MAX_GUESTS * SVC_SCRUB_RECS; j++) { + if (&svc_scrubs[j] == s || + svc_scrubs[j].vgpu != vgpu) + continue; + if (svc_scrubs[j].registered && + !svc_scrubs[j].userd_ok && + svc_scrubs[j].hsem_pa == s->hsem_pa) { + /* + * CONT.416: merge the scanner + * registration onto the stash + * record instead of dropping it. + * Dropping it left this record + * registered=false, so the next + * authoring pass wrote the + * blocking pair (sema=entries-1, + * fin=0) to the very page the + * guest waited on (r1749/r1752: + * Xorg's PMA scrub wait never + * ended). The wait needs the + * all-ones release, which only + * registered records author. + */ + if (!s->registered || + s->fin_pa != svc_scrubs[j].fin_pa) { + s->registered = true; + s->fin_pa = svc_scrubs[j].fin_pa; + s->pa_gen++; + s->seeded = false; + svc_scrub_ring_reset(s); /* CONT.419 */ + } + pr_info("sema-svc: scrub dedup slot %d (stash owns hsem_pa=0x%llx; registration merged)\n", + j, s->hsem_pa); + memset(&svc_scrubs[j], 0, + sizeof(svc_scrubs[j])); + } + } + } + s->pa_ok = true; + s->pa_at = jiffies; + } else if (!s->pa_ok) { + /* Last resort: the learned pair map, RAM only. */ + if (!svc_lookup(vgpu, hva, &pa, &ap) && pa && + svc_ap_of(ap) == SVC_APERTURE_SYS) { + svc_scrub_ring_reset(s); /* CONT.419 */ + s->hsem_pa = pa; + s->fin_pa = pa + 4; + s->pa_ok = true; + s->pa_at = jiffies; + } + } + } + /* + * CONT.408: never author through a stale geometry. svc_scrub_pass + * re-resolves the VAs above on every pass; if that walk stops + * succeeding the old PA may already belong to another allocation + * (the r1710 guest panic class). Two seconds is far longer than the + * per-pass cadence, so a healthy boot never trips it. + */ + if (!s->userd_ok || !s->pa_ok || !s->entries || + time_after(jiffies, s->pa_at + msecs_to_jiffies(2000))) { + /* m112b: silent-return diagnostic (bounded per boot). */ + if (s->log_budget < 8) { + s->log_budget++; + pr_info("sema-svc: scrub pass idle vgpu=%px slot=%d userd_ok=%d pa_ok=%d entries=%u gen=%u\n", + (void *)vgpu, (int)(s - svc_scrubs), + s->userd_ok, s->pa_ok, s->entries, + s->session_gen); + } return; + } if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as, &put, sizeof(put))) { /* Retry through the alternate aperture (as registration did). */ if (svc_read_ap(vgpu, s->userd_pa + SVC_USERD_GP_PUT, s->userd_as == SVC_APERTURE_FB ? SVC_APERTURE_SYS : SVC_APERTURE_FB, - &put, sizeof(put))) + &put, sizeof(put))) { + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass read-fail userd=0x%llx as=%u\n", + s->userd_pa, s->userd_as); + } + /* + * CONT.408 A/B: authoring the sema/finish from a + * stale geometry when the USERD is unreadable wrote a + * repurposed page and panicked the guest (r1710, + * Fatal exception in interrupt, RIP 0x0). Never write + * without a readable put; the free-run below still + * covers the readable-but-frozen case. + */ return; + } } - if (put >= s->entries) + if (put >= s->entries) { + /* m112b: junk-translation diagnostic (bounded). */ + if (s->log_budget < 12) { + s->log_budget++; + pr_info("sema-svc: scrub pass junk put=%u entries=%u userd=0x%llx\n", + put, s->entries, s->userd_pa); + } return; /* junk translation; retry next pass */ + } if (!s->seeded) { s->seeded = true; @@ -22987,38 +49432,344 @@ static void svc_scrub_pass(struct nv_vgpu_device *vgpu) * count until the ring wraps. Seeding from it adopts any * pre-host scrub work (the boot session's scrubbers). */ s->submitted = put; + /* + * CONT.408: the host semaphore is the LAST CONSUMED entry, + * not the put position: channelWaitForFreeEntry blocks while + * sema == (lastSubmittedEntry + 1) % entries, so authoring + * the put itself is exactly the value the just-submitted + * entry has and the wait can never pass. Start one behind. + */ + s->consumed = put ? put - 1 : s->entries - 1; } else if (put != s->put_last) { delta = (put + s->entries - s->put_last) % s->entries; if (!delta) delta = s->entries; /* exact wrap */ s->submitted += delta; s->put_last = put; - } else if (s->submitted == s->authored) { + s->consumed = put ? put - 1 : s->entries - 1; + } else if (s->submitted == s->authored && + s->authored_gen == s->pa_gen && + !s->registered) { + /* + * CONT.414: re-author the release pair every pass while a + * scanner-registered buffer has no observed submissions, so a + * guest rewrite of the polled word cannot re-block the wait. + */ return; /* nothing new; no guest writes */ } if (s->log_budget < 64) { s->log_budget++; - pr_info("sema-svc: scrub author put=%u submitted=%llu fin_pa=0x%llx\n", - put, s->submitted, s->fin_pa); + pr_info("sema-svc: scrub author put=%u consumed=%u submitted=%llu fin_pa=0x%llx\n", + put, s->consumed, s->submitted, s->fin_pa); } /* - * Author (forward-only). The USERD GP_GET and host-sema writes mark - * the submissions consumed; the finish write releases the scrub - * wait. Written only when GP_PUT advanced (or on the seeding pass), - * so the per-pass cost is zero while the scrubber idles. + * Author (forward-only). The USERD GP_GET marks the entries consumed; + * the host sema is the last consumed index the free-entry check reads + * (CONT.408: authoring the put itself made sema == putIndex and the + * wait could never pass), and the finish write releases the scrub + * wait. Written only when GP_PUT advanced (or on the seeding pass) so + * the per-pass cost is zero while the scrubber idles. */ svc_write_ap(vgpu, s->userd_pa + SVC_USERD_GP_GET, s->userd_as, &put, sizeof(put)); - svc_write_ap(vgpu, s->hsem_pa, SVC_APERTURE_SYS, &put, sizeof(put)); + /* + * CONT.414: a scanner-registered channel buffer with no observed + * USERD GP_PUT submissions is the X DDX's vidheap scrub wait: the + * guest RM polls the finish payload at channel buffer + 0x6c004 + * (channelWaitForFinishPayload) while the host still reads GP_PUT=0, + * so the put-based forward authoring would write sema=entries-1 and + * fin=0 - the exact values that leave channelWaitForFreeEntry and + * channelWaitForFinishPayload blocked (r1729: Xorg holds the NVRM + * RM API write lock in that wait, glxgears and nvidia-modeset D on + * os_acquire_rwlock_write). Author the proven all-ones release pair + * (CONT.408/409) until a real put advance gives us progress to + * mirror; 0xffffffff can never equal a putIndex below the ring size + * and the folded finish counter is at its maximum. + */ { + /* + * CONT.416: a scanner-registered record always takes the + * all-ones release. The PMA scrub wait (mem_scrub + * _scrubWaitAndSave; finish word at vmap+0x6c004) has no + * timeout, and the registered page is by definition the one + * the guest polled. The put-based values remain for + * stash-only records (no scanner registration). + * + * CONT.417/419: the sema word keeps the all-ones free + * release while no submissions were observed and stays + * put-based after, so an active scrubber still sees forward + * progress; the finish word mirrors real submitted counts + * instead of all-ones or any fabricated delta + * (r1765/r1766/r1767 RCA: the guest FOLDS it, 0xffffffff + * makes a 96 GiB allocation and a fabricated delta underflows + * scrubListSize - see svc_scrub_fin_next; with no new + * submissions it is a no-op). 0xffffffff can never equal a + * putIndex below the ring size, so the free-entry wait cannot + * block either way. + */ + bool rel_sema = s->registered && !s->submitted; + u32 idle = 0xffffffffu; + u32 sema = rel_sema ? idle : s->consumed; u32 fin = (u32)s->submitted; + if (s->registered) { + /* + * Mirror real submissions only: never 0xffffffff + * (24*0xffffffff-byte next allocation) and never a + * fabricated delta (scrubListSize underflow); sema + * keeps the all-ones release. + * + * CONT.419: with no observed USERD submissions the + * release source is the channel's GPFIFO ring - the + * guest's own submission ledger. fin advances to the + * counted pushes (forward-only), which is exactly the + * guest's lastSubmittedWorkId: every pending wait + * releases, the reported delta equals the pending + * list and scrubListSize lands on 0 (never underflow). + * A blind +1 tick cannot work here: no host-observable + * release edge exists, and overshooting by one past + * the pending list underflows the guest (measured: + * fin=12 with lastSubmitted=11 -> soft-lock). + */ + if (gsp_sema_scrub_ramp) { + u32 cur = 0; + + svc_read_ap(vgpu, s->fin_pa, SVC_APERTURE_SYS, + &cur, sizeof(cur)); + fin = svc_scrub_fin_next(s, cur); + if (!s->submitted) + fin = svc_scrub_fin_next_ring(vgpu, s, + fin); + } else { + fin = idle; /* legacy A/B */ + } + } + + svc_write_ap(vgpu, s->hsem_pa, SVC_APERTURE_SYS, &sema, + sizeof(sema)); svc_write_ap(vgpu, s->fin_pa, SVC_APERTURE_SYS, &fin, sizeof(fin)); + if (s->registered && gsp_sema_scrub_ramp) + s->fin_release = fin; } s->authored = s->submitted; + s->authored_gen = s->pa_gen; +} + +/* + * CONT.409: author every channel-buffer record of this guest. The stashed + * CeUtils scrubber and any scanner-registered buffer (the X DDX's own + * CE/PMA channel) are independent; a singleton record could only serve + * one of them, and the other's wait then never completed. + */ +static void svc_scrub_pass(struct nv_vgpu_device *vgpu) +{ + int i; + + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) + if (svc_scrubs[i].vgpu == vgpu) + svc_scrub_one(vgpu, &svc_scrubs[i]); +} + +/* + * r1668: run the scrub step from inside a blocked event post. + * + * When the guest's status-queue consumer is frozen (typically because the + * X server's PMA scrub wait holds the RM lock the event callback needs), + * the pump's critical-event posts wait out their full ring-full budget + * (200 ms each) before dropping. A pass can then take many seconds, and + * the next pass's scrub step - the only thing that can release the scrub + * wait - is starved. This hook is called from the send path's wait loop, + * on the service thread itself, so it runs the same state machine without + * any concurrency. It is a no-op from any other thread (RPC replies run + * on the GSP emulation thread). + */ +void gsp_sema_service_scrub_pump(struct nv_vgpu_device *vgpu) +{ + if (!svc.task || !vgpu || current != svc.task) + return; + svc_scrub_pass(vgpu); +} + +/* + * r539: init-stall watchdog. The r530-r538 stall boots all share one shape: + * the pump keeps looping (rebuilds, pending retries) but the entry loop stops + * consuming anything, a client spins at DEVICE (uvm channel_pool_add -> + * uvm_push_end_and_wait) and every counter the guest waits on is frozen. The + * historical evidence for that state was spread over many bounded logs; this + * dump collects the whole stall state in one place when the entry loop makes + * no progress for SVC_STALL_WATCH_MS while withheld work exists. Bounded: + * SVC_STALL_WATCH_DUMPS per boot, one per re-armed window. + */ +#define SVC_STALL_WATCH_MS 20000u +#define SVC_STALL_WATCH_DUMPS 4u + +static unsigned long svc_stall_watch_last; +static u64 svc_stall_watch_done; +static u32 svc_stall_watch_dumps; + +/* + * r563: stall-watchdog self-heal. The r562 STALL-DUMP shape (pump alive, + * no entry consumed for 20 s, held/opdef counters frozen) has been closed + * one hole at a time (r548..r558), but every remaining interaction hole + * has the same end state: some hold condition keeps winning and the walk + * breaks on the same entry forever. When the watchdog fires, arm a small + * budget of forced consumes: the next hold decisions that would otherwise + * park an entry are bypassed, the entry is consumed with its withheld + * releases applied forward-only (and the GR QMD completion authored by the + * existing tail), and the client's wait advances. Bounded by the dump + * cadence (<= 4 per boot) and by the budget (4 hold bypasses per arming), + * so a healthy boot never sees it and a stalled one cannot livelock. + */ +static u32 svc_force_consume; + +static inline bool svc_hold_bypass(void) +{ + if (!svc_force_consume) + return false; + svc_force_consume--; + pr_info_ratelimited("vgsp-sema: STALL-HEAL: hold bypassed (forced consume, budget left %u)\n", + svc_force_consume); + return true; +} + +static void svc_stall_watch_dump(void) +{ + u32 i; + + pr_err("vgsp-sema: STALL-DUMP #%u: no GPFIFO entry consumed for %us (done=%llu held=%llu opdef=%llu/%llu/%llu pending=%u starved=%llu)\n", + svc_stall_watch_dumps, SVC_STALL_WATCH_MS / 1000u, + svc.gpfifo_entries_done, svc.gpfifo_held, + svc.opdef_held, svc.opdef_applied, svc.opdef_expired, + svc_pending_n, svc.gpfifo_starved); + for (i = 0; i < svc_pending_n && i < 16; i++) + pr_err("vgsp-sema: STALL-DUMP pending[%u] h=0x%x va=0x%llx pay=%u op=%u tries=%u\n", + i, svc_pending[i].h, svc_pending[i].va, + svc_pending[i].payload, svc_pending[i].op, + svc_pending[i].tries); + for (i = 0; i < SVC_OPDEF_SLOTS; i++) { + struct svc_opdef_ent *e = &svc_opdef[i]; + + if (!e->active) + continue; + pr_err("vgsp-sema: STALL-DUMP opdef[%u] h=0x%x entry=%u n=%u age_ms=%u first_va=0x%llx pay=%u\n", + i, e->h_object, e->entry_idx, (unsigned)e->n, + jiffies_to_msecs(jiffies - e->since), + e->n ? e->rel[0].va : 0ull, + e->n ? e->rel[0].payload : 0u); + } + for (i = 0; i < svc_pool_boot_n; i++) { + struct svc_pool_boot_pa *b = &svc_pool_boot[i]; + + if (!b->va_page) + continue; + pr_err("vgsp-sema: STALL-DUMP pool[%u] va=0x%llx pa=0x%llx ap=%d pa2=0x%llx ap2=%d age_ms=%u\n", + i, b->va_page, b->pa_page, b->ap, b->pa2_page, + b->ap2, jiffies_to_msecs(jiffies - b->seen)); + } + pr_err("vgsp-sema: STALL-DUMP pool_dual=%llu alt=%llu conflicts=%u residue=%llu\n", + svc.pool_dual_applied, svc.pool_alt_applied, + svc_pool_boot_conflicts, svc.pool_residue); +} + +static void svc_stall_watch_check(void) +{ + if (svc.gpfifo_entries_done != svc_stall_watch_done) { + svc_stall_watch_done = svc.gpfifo_entries_done; + svc_stall_watch_last = jiffies; + return; + } + if (!svc_stall_watch_last) + svc_stall_watch_last = jiffies; + if (!time_after(jiffies, svc_stall_watch_last + + msecs_to_jiffies(SVC_STALL_WATCH_MS))) + return; + svc_stall_watch_last = jiffies; /* re-arm the window */ + if (svc_stall_watch_dumps >= SVC_STALL_WATCH_DUMPS) + return; + /* + * r563c: gate on ACTIVE withheld work, not on the cumulative counters. + * r566 evidence: the dump fired four times with done=8929 and + * opdef=215/7/5 frozen - but the ring was EMPTY (every channel + * last_get == put: the clients were simply waiting and no entry was + * pending), and the forced flush had nothing to deliver. `opdef_held` + * is a lifetime counter (215 holds applied/expired long ago), not + * proof of a current hold. A real stall has either a pending + * translation (svc_pending_n) or an ACTIVE opdef slot with releases + * recorded (e->active && e->n) - only then is there something the + * pump is withholding. + */ + { + bool work = svc_pending_n != 0; + u32 wi; + + for (wi = 0; wi < SVC_OPDEF_SLOTS && !work; wi++) + if (svc_opdef[wi].active && svc_opdef[wi].n) + work = true; + if (!work) + return; + } + svc_stall_watch_dumps++; + svc_stall_watch_dump(); + /* + * r563: self-heal after the dump. The r564 boot showed the force + * budget alone is not enough: the walk broke BEFORE the instrumented + * hold sites (no STALL-HEAL lines while held=727 climbed), so clear + * the state blockers every break site consults, and force-deliver + * the withheld opdef slots: + * - per channel: the cheap-wait unread window (its break at the + * loop top) and the GR hold/nocap indices; + * - the opdef slots: deliver forward-only and mark delivered, so + * the next decode cannot re-hold (the same edge the 3 s bound + * would run); + * - the force-consume budget stays as the belt for the hold sites. + */ + svc_opdef_force_flush(); + { + unsigned long flags; + u32 c; + + spin_lock_irqsave(&svc.chan_lock, flags); + for (c = 0; c < SVC_MAX_CHANNELS; c++) { + struct svc_channel *ch = &svc.chan[c]; + + if (!ch->valid) + continue; + ch->unread_va = 0; + ch->unread_until = 0; + svc_gr_setup_hold_clear(ch); + ch->gr_hold_idx = ~0u; + ch->gr_hold_until = 0; + ch->gr_nocap_idx = ~0u; + ch->gr_nocap_va = 0; + } + spin_unlock_irqrestore(&svc.chan_lock, flags); + } + pr_err("vgsp-sema: STALL-HEAL: forced opdef flush + walk-state clear (dumps=%u)\n", + svc_stall_watch_dumps); + svc_force_consume = 4; +} + +/* + * ISSUE-0002 client-session boundary aid: force a session flush that + * executes even while channels are live. The channel-close trigger only + * fires when the table empties, which never happens on a guest where the + * driver-global kernel channels (RC watchdog 0x31415900, the tiny notifier + * channel) stay valid; later client sessions then inherit the previous + * session's map/pin/pool discovery state and wedge (false translations -> + * failed rescues -> driver DMA-init timeout). Raised through the + * sema_force_flush module parameter so the boundary can be tested before + * wiring automatic detection. + */ +void gsp_sema_service_sess_flush_force(void) +{ + WRITE_ONCE(svc_sess_flush_forced, 1); + WRITE_ONCE(svc_sess_flush, 1); + if (READ_ONCE(svc.task)) + wake_up_process(svc.task); + pr_info("sema-svc: forced session flush requested\n"); } static int svc_thread(void *arg) @@ -23097,6 +49848,11 @@ static int svc_thread(void *arg) u64 done_before; vgpu = vgpus[g]; + svc_shared_hold(vgpu); /* CONT.36: GL slot-word keeper */ + /* CONT.42: release any armed client completion sentinel + * on every pass, not only when a completion is being + * processed (see svc_client_slot_sweep). */ + svc_client_slot_sweep(vgpu); svc_handle_probe(vgpu); svc_handle_walk_probe(vgpu); svc_handle_chan_dump(vgpu); @@ -23122,6 +49878,13 @@ static int svc_thread(void *arg) } /* + * r537: bounded fallback for a parked VAS candidate + * whose content-verified ring walk never anchored + * (see svc_vas_candidates_age_out). + */ + svc_vas_candidates_age_out(vgpu); + + /* * Channel-driven execution: notice submissions as they * are made and run the pushed work immediately, so the * guest's UVM progress waits complete in milliseconds @@ -23136,6 +49899,16 @@ static int svc_thread(void *arg) pumped = true; /* + * r1621: display-engine progress. The guest's + * nvidia-modeset kicks its EVO channels through the + * BAR0 UDISP control windows; consume the push buffers, + * publish GET and complete each UPDATE's core notifier. + * The boot display-init and every Xorg mode set wait on + * that completion. + */ + gsp_disp_engine_pump(vgpu); + + /* * Host CUDA bridge: marshal one queued launch into a * daemon job and resolve completions. Inert until a * launch is seen. @@ -23161,6 +49934,9 @@ static int svc_thread(void *arg) svc_apply_sweep_slice(vgpu); } + /* r539: one stall dump when the entry loop stops progressing. */ + svc_stall_watch_check(); + /* * While the pump is making progress the client is streaming * pushes and waiting on their releases: loop tightly and skip @@ -23170,6 +49946,32 @@ static int svc_thread(void *arg) * progress. (With two guests, any guest's progress keeps the * loop tight; the next iteration pumps both.) */ + /* + * r580: the completion hint must also run while the pump is + * making progress - the fast path below skips it, and a + * completion burst saturates the event burst budget and drops + * wake-ups (r579 boot: events_dropped=724; the sentinel + * "no postable event (events_n=8 dropped=724)" repeated every + * 0.5 s). A client that starts waiting mid-burst needs the + * 0.5 s re-post even under load; the per-event dedup keeps the + * extra volume bounded. + */ + for (g = 0; g < n_vgpus; g++) { + vgpu = vgpus[g]; + if (time_before(jiffies, + (unsigned long)svc.post_hint_until) && + time_after(jiffies, + (unsigned long)svc.last_hint + HZ / 2)) { + int rc; + + svc.last_hint = jiffies; + rc = gsp_rm_post_all_events(vgpu); + if (gsp_sema_dbg) + pr_info_ratelimited("sema-svc: completion hint rc=%d\n", + rc); + } + } + if (pumped) { msleep(1); continue; @@ -23181,30 +49983,22 @@ static int svc_thread(void *arg) /* * Push-buffer base discovery: a chunked walk of guest * RAM finds the pages whose first push names a - * channel, so a channel that appears with work - * already waiting is pinned and pumped without - * waiting for the rolling sweep. */ svc_pb_discover(vgpu); /* - * Completion hint: re-post the events for a few - * seconds after a completion, so a client that starts - * waiting after the edge-triggered wake-up still - * observes it. + * SESSION 153-CONT-34: doorbell writes the guest made + * directly into the mmap'd BAR0 shadow (PCFG window) - + * feed them to the doorbell path so the pump is woken + * without waiting for the next pass boundary. */ - if (time_before(jiffies, - (unsigned long)svc.post_hint_until) && - time_after(jiffies, - (unsigned long)svc.last_hint + HZ / 2)) { - int rc; + svc_bar0_pcfg_scan(vgpu); - svc.last_hint = jiffies; - rc = gsp_rm_post_all_events(vgpu); - if (gsp_sema_dbg) - pr_info_ratelimited("sema-svc: completion hint rc=%d\n", - rc); - } + /* + * r580: the completion hint was hoisted above the + * pumped fast path (it must run under load too); it is + * not repeated here. + */ /* * Fast path for the pushes themselves: every mapped @@ -23305,8 +50099,11 @@ int gsp_sema_service_start(void) * while gsp_phys_work_forward is enabled; the software path needs no * workqueue at all. */ + if (gsp_disp_engine_init()) + pr_warn("sema-svc: display engine init failed\n"); svc.task = kthread_run(svc_thread, NULL, "gsp-sema-svc"); if (IS_ERR(svc.task)) { + gsp_disp_engine_fini(); int ret = PTR_ERR(svc.task); svc.task = NULL; @@ -23314,6 +50111,8 @@ int gsp_sema_service_start(void) } if (gsp_cuda_bridge_init()) pr_warn("sema-svc: cuda bridge debugfs unavailable\n"); + if (svc_setup_demand_debugfs_init()) + pr_warn("sema-svc: setup-demand debugfs unavailable\n"); return 0; } @@ -23335,6 +50134,18 @@ void gsp_sema_service_stop(void) svc.task = NULL; } /* + * CONT.419: the ring-ledger snapshots are lazily allocated per record + * slot; free them here (the thread that reads them is gone). + */ + { + int i; + + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) { + kfree(svc_scrub_ring_snap[i]); + svc_scrub_ring_snap[i] = NULL; + } + } + /* * A work item canceled before it ran never cleared INFLIGHT; clear * it here so the next session is not refused by a stranded bit. */ @@ -23348,9 +50159,17 @@ void gsp_sema_service_stop(void) svc_fwd_wq = NULL; } /* Thread is gone: release the bridge's pending jobs and debugfs files. */ + gsp_disp_engine_fini(); gsp_cuda_bridge_fini(); + svc_setup_demand_debugfs_fini(); /* A fresh session must not inherit shadowed bytes from the last one. */ svc_vas_free_all(); + /* Drop the bracket hash array; a later session allocates it again. */ + mutex_lock(&gsp_sema_vidmem_hash_lock); + vfree(gsp_sema_vidmem_hash); + gsp_sema_vidmem_hash = NULL; + gsp_sema_vidmem_hash_pages = 0; + mutex_unlock(&gsp_sema_vidmem_hash_lock); } /* @@ -23377,6 +50196,9 @@ void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu) if (!vgpu) return; + /* r1621: drop the guest's display channels with the rest of its state. */ + gsp_disp_engine_vgpu_cleanup(vgpu); + /* * The forward work item runs on its own kthread and may be using the * request's vgpu; quiesce waits for it (bounded by the submit's @@ -23436,7 +50258,7 @@ void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu) if (svc_guests[i].vgpu == vgpu) memset(&svc_guests[i], 0, sizeof(svc_guests[i])); spin_unlock_irqrestore(&svc_guest_lock, flags); - for (i = 0; i < SVC_MAX_GUESTS; i++) + for (i = 0; i < SVC_MAX_GUESTS * SVC_SCRUB_RECS; i++) if (svc_scrubs[i].vgpu == vgpu) memset(&svc_scrubs[i], 0, sizeof(svc_scrubs[i])); @@ -23642,3 +50464,68 @@ int gsp_sema_service_sem_write(struct nv_vgpu_device *vgpu, u32 h, u64 va, svc.post_hint_until = jiffies + msecs_to_jiffies(SVC_POST_HINT_MS); return rc; } + +/* + * r616 diag: manual fault poke for the refused-launch hang experiment. A GR + * push that the fill cannot resolve is refused (degenerate data set) and the + * entry is consumed with a software release; the authored release lands in + * the guest page (measured: kcore read 0x6 == pay on r613d) yet the guest + * driver never completes cudaDeviceSynchronize for a launch that never ran. + * This writes the channel's error notifier in the exact shape the relayed- + * fault edge uses (info32=exceptType, info16=engine, status=0xffff, latched + * by notifier_err) and posts the channel event, so a hung client's kernel + * can be poked into observing an engine error and ending its wait. Debug + * only; harmless when no client is hung (the pump's completion clear resumes + * after SVC_NOTIFIER_ERR_HOLD). h_object 0 = every valid channel. + */ +int gsp_sema_channel_fault_poke(struct nv_vgpu_device *vgpu, u32 h_object) +{ + u8 notif[SVC_NOTIFY_SIZE] = {0}; + u16 status = SVC_NOTIFY_ERROR_STATUS; + u32 info32 = 0x1; /* generic engine exception */ + int i, done = 0; + + if (!vgpu) + return -ENODEV; + memcpy(notif + SVC_NOTIFY_INFO32_OFF, &info32, 4); + memcpy(notif + SVC_NOTIFY_STATUS_OFF, &status, 2); + + for (i = 0; i < SVC_MAX_CHANNELS; i++) { + struct svc_channel *ch = &svc.chan[i]; + unsigned long flags; + u64 pa; + u32 as, eng, h; + u16 info16; + + spin_lock_irqsave(&svc.chan_lock, flags); + if (!ch->valid || ch->vgpu != vgpu || !ch->notifier_pa || + (h_object && ch->h_object != h_object)) { + spin_unlock_irqrestore(&svc.chan_lock, flags); + continue; + } + h = ch->h_object; + pa = ch->notifier_pa; + as = ch->notifier_as; + eng = ch->engine_type; + /* Latch exactly like the relayed-fault edge so the pump's + * completion clear cannot wipe the poke before the guest + * reads it. */ + ch->notifier_err = true; + ch->notifier_err_until = jiffies + SVC_NOTIFIER_ERR_HOLD; + ch->notifier_err_put = ch->put_prev; + spin_unlock_irqrestore(&svc.chan_lock, flags); + + info16 = (u16)eng; + memcpy(notif + SVC_NOTIFY_INFO16_OFF, &info16, 2); + if (svc_write_ap(vgpu, pa, as, notif, sizeof(notif))) { + pr_info("vgsp-fault: POKE ch=0x%x notifier write failed pa=0x%llx as=%u\n", + h, pa, as); + continue; + } + pr_info("vgsp-fault: POKE ch=0x%x engine=%u pa=0x%llx as=%u status=0x%x except=0x%x (manual experiment)\n", + h, eng, pa, as, status, info32); + gsp_rm_post_channel_event(vgpu, h); + done++; + } + return done ? done : -ENOENT; +} diff --git a/kernel/gsp_sema_service.h b/kernel/gsp_sema_service.h index 3765cd2..36f9902 100644 --- a/kernel/gsp_sema_service.h +++ b/kernel/gsp_sema_service.h @@ -31,6 +31,7 @@ struct sema_channel_desc { u32 h_object; u32 engine_type; u32 cid; /* RM channel id (the doorbell token vector) */ + u32 h_vaspace; /* r835: VAS handle the channel was allocated in */ u64 gpfifo_va; u32 gpfifo_entries; u64 userd_pa; /* NvA06FControl: GPGet @0x88, GPPut @0x8c */ @@ -44,6 +45,14 @@ void gsp_sema_service_channel_add(const struct sema_channel_desc *desc); void gsp_sema_service_channel_del(struct nv_vgpu_device *vgpu, u32 h_object); /* + * r616 diag: manually write a channel's error notifier (engine-exception + * shape) and post its channel event, to test whether a refused launch's + * hung client can be ended by a fault-conversion edge. h_object 0 = every + * valid channel of the device. Debug only. + */ +int gsp_sema_channel_fault_poke(struct nv_vgpu_device *vgpu, u32 h_object); + +/* * Release every piece of service state that belongs to one mdev before its * guest session disappears: drop the mdev's channels (per-vgpu session end), * clear a release hold that names it (never dereference a removed device), @@ -64,6 +73,15 @@ void gsp_sema_service_vgpu_cleanup(struct nv_vgpu_device *vgpu); int gsp_sema_service_scrubbuf_add(u64 fin_pa, u64 hsem_pa); /* + * Author the CeUtils scrubber's progress from the GSP-RM send path's + * ring-full wait loop (r1668). Called on the sema service thread only; it + * lets the scrub completion run while the pump is blocked posting a + * critical event into a status ring the guest cannot drain (the guest's RM + * consumer is frozen behind the very scrub wait this releases). + */ +void gsp_sema_service_scrub_pump(struct nv_vgpu_device *vgpu); + +/* * A GPU VA space whose top-level page directory the driver has published (the * driver builds its own page tables for the address spaces it owns, then hands * the directory's physical address to the GPU through SET_PAGE_DIRECTORY). @@ -77,15 +95,77 @@ struct sema_vaspace_desc { u64 pdb; u32 num_entries; bool b_vidmem; + /* + * CONT.400: true when the directory was proven against this client's + * own content-validated ring (or published by RM itself). A directory + * registered only by the bounded age-out is UNVERIFIED; a verified + * registration must never be downgraded by a later unverified one. + */ + bool verified; }; void gsp_sema_service_vaspace_add(const struct sema_vaspace_desc *desc); void gsp_sema_service_vaspace_del(u32 h_vaspace); +/* + * CONT.407: a client's DUP_OBJECT of a VAS gives the destination client a + * NEW handle for the SAME page directory (measured: the GL client dups X's + * VAS 0x10005 as 0xbfef0001, then allocates its GR channel with + * hVASpace=0xbfef0001). Register the destination handle as an alias of the + * source's directory so the channel's gpfifo VA resolves through the + * client's own VAS instead of falling through to another registered + * directory (observed bogus 0x1bcfff0/ap=2 answer). Returns true when the + * source handle was a registered VAS and the alias was installed. + */ +bool gsp_sema_service_vaspace_alias(u32 h_dst, u32 h_src); + +/* + * r820: a driver-declared kernel-buffer translation from an + * NV2080_CTRL_CMD_GPU_PROMOTE_CTX entry (gpuVirtAddr/gpuPhysAddr/size/ + * physAttr of one context buffer). These buffers - the nvidia-push + * pushbuffer, the progress-tracker segment, the FECS/patch/access-map + * blocks - are kernel-owned, so the userspace pin scanner cannot cover + * them and the page-table walk can answer with a stale/wrong page + * (r817 Xorg GL init: the walk resolved the 64 KB pushbuffer VA to a + * zero page while the promote entry named the real one). The sema + * service consults these records before the learned cache and the walks + * in svc_lookup_trusted; newest statement per VA wins. VA-only bookings + * (zero pa/size) and bNonmapped entries (va=0) are ignored. Records are + * session state, cleared by svc_session_flush. + */ +void gsp_sema_service_promote_add(u64 va, u64 pa, u64 size, u32 phys_attr, + u32 buffer_id); +/* + * SESSION 153-CONTINUED-95: latest PROMOTE_CTX statement per + * (engine, bufferId), kept for the launch-time CTX-BIND receipt. MAIN is + * bufferId 0, PATCH bufferId 2 (NV2080_CTRL_GPU_PROMOTE_CTX_BUFFER_ID_*). + * The note side never fails; the get side returns -ENOENT until the guest + * driver has promoted that buffer. + */ +void gsp_sema_service_ctx_promote_note(u32 engine, u32 buffer_id, u64 pa, + u64 va, u64 size, u32 phys_attr, + u32 b_init); +int gsp_sema_service_ctx_promote_get(u32 engine, u32 buffer_id, u64 *pa, + u64 *va, u64 *size, u32 *phys_attr, + u32 *b_init); +/* + * r496j: park a page-directory candidate (from COPY_SERVER_PDES) until a + * content-validated walk proves it belongs to a live client; only then is + * it registered (a blind registration let stale kernel directories answer + * other clients' VAs in the walker's first-success order). + */ +void gsp_sema_service_vaspace_candidate_add(u32 h_vaspace, u64 pdb, + u32 entries, bool vidmem); /* m177: guest RM reload = new guest boot; wipe boot-scoped discovery state * (direct-map base, content-scanned tables) on the svc thread. Client * session flushes keep that state - see svc_session_flush. */ void gsp_sema_service_guest_reset(void); +/* + * ISSUE-0002 aid: force a client-session flush that executes even while + * channels are live (see addendum 8 and the sema_force_flush parameter). + */ +void gsp_sema_service_sess_flush_force(void); + int gsp_sema_service_start(void); void gsp_sema_service_stop(void); int gsp_sema_service_probe(u64 gpu_va, u32 count); @@ -93,7 +173,21 @@ int gsp_sema_service_walk_probe(u64 gpu_va); int gsp_sema_service_dump_channels(void); int gsp_sema_service_add_map(u64 gpu_va, u64 pa, u64 len, u32 ap); int gsp_sema_service_dump_vidmem(u64 addr, u64 len); +int gsp_sema_service_xw_mirror(u64 cnt_pa, u64 mem_pa); +int gsp_sema_service_xw_comp(u64 comp_pa); int gsp_sema_service_find_vidmem(u64 pattern, u64 max_hits); +int gsp_sema_service_vidmem_hash_snapshot(void); +int gsp_sema_service_vidmem_hash_diff(const char *tag, + unsigned int max_pages); +int gsp_sema_service_pdb_discover(u64 ring_va, u64 ring_pa); +/* + * r1780 (V34): bounded page-directory scan for a content-verified + * {VA -> PA} pair (FB shadow first, then guest RAM up to ram_lim), and a + * direct VAS registration for a directory found out-of-band. + */ +int gsp_sema_service_pdb_scan(u64 va, u64 pa, u32 h_vas, u64 ram_lim); +void gsp_sema_service_vas_force(u32 h_vas, u64 pdb, u32 entries, bool vid); +int gsp_sema_service_chan_rewind(u32 h_object); int gsp_sema_service_counter_add(u32 h_object, u64 pa); void gsp_sema_service_doorbell(u32 token); @@ -191,6 +285,16 @@ void gsp_sema_service_fwd_gr_stats(u64 *req, u64 *ok, u64 *fallback, u64 *refused, u64 *timeout, u64 *fault); /* + * SESSION 153-CONTINUED-94: the first large beef-channel push of the boot, + * preserved verbatim for the gsp-phys-init-push debugfs dump (the client's + * pushbuffer is transient in guest RAM). Returns the buffer pointer and + * fills len/h/va; lmem_seen counts any SET_SHADER_LOCAL_MEMORY_A..D + * (0x790..0x79c) runs seen in beef pushes so far (0 = the GL client never + * emitted the download-state local-memory block). + */ +const u8 *gsp_sema_init_cap_get(u32 *len, u32 *h, u64 *va, u32 *lmem_seen); + +/* * Discover and pin GR (compute) channel rings after the CE region of the * channel video-memory allocation. Set false to leave GR rings untranslated * (the pump then only runs the CE channels). @@ -251,6 +355,7 @@ extern u64 gsp_sema_copy_dst2; /* CE pending-entry grace window in ms before the release fallback fires. */ extern unsigned gsp_sema_ce_grace_ms; +extern u32 gsp_phys_gr_peek; /* * Bounded work-before-release hold (ms) for an unreadable CE entry whose @@ -258,6 +363,8 @@ extern unsigned gsp_sema_ce_grace_ms; * retry probe). Absolute deadline per entry, never reset; 0 disables. */ extern unsigned gsp_sema_push_hold_ms; +/* SESSION 153-CONTINUED-54: GR setup-push (MME-setup / >=12 KB) hold. */ +extern unsigned gsp_sema_gr_setup_hold_ms; /* * Grace window (ms) an untranslatable GR GPFIFO entry is left pending before @@ -284,6 +391,17 @@ extern unsigned gsp_phys_gr_hold_max_ms; * hold and its absolute bound apply. */ extern unsigned gsp_phys_gr_fill_hold; + +/* + * r576 DIAG: per-boot witness budget for the GR launch-decision dump + * (default 0 = off). Nonzero logs, for the first N launches whose fill saw + * dropped pages / a partial argument set or whose decision point had a + * dstpend/copy-pending verdict: the decision verdicts, the launch's + * resolved data pages (VA -> guest PA), the live deferred-write records and + * the pending CE opdef entries - the evidence needed to explain a red draw + * of the readback lottery. + */ +extern unsigned gsp_phys_gr_dec_log_max; extern unsigned gsp_sema_walk_trace; /* @@ -309,10 +427,17 @@ extern bool gsp_sema_pool_hold_fresh; * walk/RAM scan until a pin confirms (default on). */ extern bool gsp_sema_copy_dst_pin_guard; +/* r609: after a device-dst guard window expires WITHOUT a pin, skip the + * window for further first-write device-dst pages of the same channel for + * this long (ms). 0 restores the per-page windows (pre-r609). */ +extern unsigned gsp_sema_copy_dst_age_ms; extern bool gsp_sema_wait_gate; /* r401: bounded acquire-wait hold (fail-open) in ms; must exceed the * emulated launch completion latency for copy-after-launch ordering. */ extern unsigned gsp_sema_wait_hold_ms; +/* r481: verbatim method-stream dump for the first N large GR pushes + * (default 0 off). Diagnostic only; no behaviour change. */ +extern unsigned gsp_sema_push_raw; /* * r384 (guest boot-corruption lottery): fill-destination guards (default @@ -383,6 +508,185 @@ struct gsp_sema_gr_reloc { }; int gsp_sema_gr_reloc_resolve(u64 va, struct gsp_sema_gr_reloc *out); +/* + * r1497: when a fetch VA resolves to an FB page whose emulated shadow is + * empty, copy the same VA's sysmem backing (found by walking every + * registered VAS) into that page. gsp_sema_gr_fetch_sys_into() takes the + * already-resolved FB PA; gsp_sema_gr_fetch_backing() resolves the VA the + * way the relocation chain does and is the booter's entry point. Both + * return true only when bytes were delivered. + */ +bool gsp_sema_gr_fetch_sys_into(u64 va, u64 fb_pa, u64 *gpa_out); +bool gsp_sema_gr_fetch_backing(u64 va); + +/* + * SESSION 153-CONTINUED-59: record one VA (page-aligned internally) in the + * pin-demand set exposed at /sys/kernel/debug/vgsp/sema_setup_demand, so + * pin_gr_stream demands its page in the guest. The GR launch builder calls + * this when a window/attr operand page cannot be resolved at build time + * (the r1104/r1105 MISS -> MMU-fault class); the demand gives the next + * forward a resolvable page instead of a lane-killing raw guest VA. + */ +void gsp_sema_demand_note(u64 va); + +/* + * SESSION 153-CONTINUED-53: MME context-setup prelude accessor for the GR + * launch builder. Returns the captured setup runs (C597 MME family + * 0x114..0x124/0x234/0x550..0x568/0x3400..0x34fc, verbatim, first- + * encounter order), the word count in *n and the capture generation in + * *gen. The builder preplays the words once per generation, ahead of the + * first 3D window replay, so the ladder context's MME has its macro + * program before CALL_MME_MACRO runs. Reading the generation before and + * after the copy detects a concurrent append (the service thread only + * appends; a mismatch just defers emission to the next launch). *n is 0 + * until the client's setup push has been decoded. + */ +const u32 *gsp_sema_mme_prelude_get(u32 *n, u32 *gen); + +/* + * SESSION 153-CONTINUED-57: is the captured MME prelude mature enough to + * launch against - i.e. does it hold the client's macro program (>= 64 + * captured 0x114 code words)? The r1094 premature-emission poison + * (MME ILLEGAL_OPCODE at +5 ms, capture still at 9 scratch words) is + * gated on this: the sema forward path withholds beef launches while it + * reads false, bounded by sema_gr_prelude_gate_ms. + */ +bool gsp_sema_mme_prelude_mature(void); + +/* + * SESSION 153-CONTINUED-57: attr/pipeline state prelude accessor (C597 + * families 0x1100..0x11fc and 0x2000..0x205c, verbatim, first-encounter + * order, deduped). Same contract as gsp_sema_mme_prelude_get: preplay once + * per generation ahead of the first 3D window replay. The runs carry the + * pipeline program binds whose SPH the draw shader validates, so the + * builder also registers the program VAs (gsp_sema_attr_prelude_ptrs) in + * the launch page set to make the address relocation land. + */ +const u32 *gsp_sema_attr_prelude_get(u32 *n, u32 *gen); +const u64 *gsp_sema_attr_prelude_ptrs(u32 *n); + +/* + * SESSION 153-CONTINUED-62: surface (render-target) setup accessor. Holds + * the last captured C597 colour-target bind run (INC 800+j*40, [upper, + * lower, w, h, fmt, blksz], fmt 0xe6, 2..4096 pages) from any trusted + * push. The GR launch builder replays it - aliased/relocated exactly like + * the window - ahead of a 3D window that carries no usable render-target + * bind of its own; without it the replayed draw rasterizes into an + * unbound target and stores nothing (the observed 0/N retire). + */ +const u32 *gsp_sema_surf_prelude_get(u32 *n, u32 *gen); + +/* + * SESSION 153-CONTINUED-63: surface geometry for render-target tracking + * (base VA + page count of the captured colour target). + */ +bool gsp_sema_surf_va_get(u64 *va, u32 *npages); + +/* + * SESSION 154 r1510: quarantine the guest code-heap/shader upload that can + * overlap the captured X screen (BAR1 mmap writes the emulator cannot + * trap). The surface capture snapshots the screen; a later launch whose + * screen page 0 carries the code signature restores the surface from that + * snapshot before the engine runs (the baseline stays armed so a later + * client context's upload is caught too). Returns the pages restored (0 + * when there is no pollution or no baseline). + */ +int gsp_sema_surf_clean_restore(u64 va, u64 npages); + +/* + * SESSION 153-CONTINUED-63: the fill's completed render-target tracking run + * (base VA + page count), true only when every page of the surface resolved + * into contiguous data recs this pass. The booter prefers these recs over + * the identity surface alias when rewriting the target's operand pair, so + * the engine's stores land in the bounce (witness + copy-out) instead of + * the guest pages directly. + */ +bool gsp_sema_gr_surf_recs(u64 *va, u32 *n); + +/* + * SESSION 153-CONTINUED-63: the captured MME START-ADDRESS RAM table + * (macro slot -> IR instruction address; present-bitmask returned, entries + * copied into `entries`). The prelude buffer is capped at 512 words, so + * the SA table is recorded independently; the GR launch builder completes + * the SA RAM it replays and stubs every called-but-unknown slot with a + * harmless end-noop (the r1116 0x238c wall: SA[0x22] = 0 on a fresh + * context made CALL_MME_MACRO(0x22) jump to IR 0 and the slot-0 macro's + * emission trapped the lane). + */ +u64 gsp_sema_mme_sa_get(u32 *entries, u32 n); + +/* + * SESSION 153-CONTINUED-101: the client's own SET_MME_SHADOW_SCRATCH(0x18) + * write (method 0x3460) - mmeCodeOGL_C597_SETUP_FIXED_CONST (slot 0) reads + * shadow scratch 0x18/0xd18 via STATE and emits it as the fixed-function + * constant-buffer selector (SET_CONSTANT_BUFFER_SELECTOR_B/C). Returns + * true (and stores the value exactly as pushed: fixed-CB address >> 8) + * once any decoded client push carried the write. false until then. + */ +bool gsp_sema_mme_scratch18_get(u32 *val); + +/* + * SESSION 153-CONTINUED-115: the client's own SET_MME_SHADOW_SCRATCH(0x19) + * write (method 0x3464) - STATE 0xD19, the constant-buffer selector B/C + * (address>>8) the bug2548370 loadConstantBuffer macro's in-band update + * emits from once the fast path is refused. Companion of scratch 0x18 + * above; returns true (and stores the value exactly as pushed) once any + * decoded client push carried the write. false until then (never observed + * live so far; sema_mme_state_d19 can supply the seed value instead). + */ +bool gsp_sema_mme_scratch19_get(u32 *val); + +/* + * SESSION 153-CONTINUED-116: the client's own SET_MME_SHADOW_SCRATCH(0x24) + * write (method 0x3490) - STATE 0xD24, the next unseeded slow-path input the + * bug2548370 macro body reads beside the d19 selector. Companion of scratch + * 0x19 above; returns true (and stores the value exactly as pushed) once any + * decoded client push carried the write; sema_mme_state_d24 can supply the + * seed value instead. + */ +bool gsp_sema_mme_scratch24_get(u32 *val); + +/* + * SESSION 153-CONTINUED-116: the client's own SET_MME_SHADOW_SCRATCH(0x25) + * write (method 0x3494) - STATE 0xD25. Same first-write-wins capture and + * builder fallback (sema_mme_state_d25) as scratch 0x24 above. + */ +bool gsp_sema_mme_scratch25_get(u32 *val); + +/* + * SESSION 153-CONTINUED-87: the SPH VERSION of the live program images + * (sema_gl_vs/ps_hex blobs), so the GR launch builder emits a + * SET_SPH_VERSION window that MATCHES the images the engine will fetch + * (the client's own init push says 4/4, but the r950 gate proved the V3 + * images execute; a 4/4 window over a V3 SPH reproduces the + * SPH9|SPH11|SPH18 fault). 0 when no image is loaded. + */ +int gsp_sema_gl_sph_blob_version(void); + +/* + * SESSION 153-CONTINUED-68: furthest IR instruction covered by the CAPTURED + * 0x114 uploads (pc + ceil(code words / 3)). The GR launch builder places + * its end-noop macro stub at this address (aligned past the replayed + * program) so the stub cannot clobber a captured macro body; 0 until the + * first 0x114 upload enters the prelude buffer. + */ +u32 gsp_sema_mme_ir_end_get(void); + +/* + * SESSION 153-CONTINUED-70: IR code words captured so far (the maturity + * gate's measure; printed by the MME-PRE-DUMP for the offline sim). + */ +u32 gsp_sema_mme_code_words_get(void); + +/* + * SESSION 153-CONTINUED-68: is the IR instruction at address `pc` present + * in the replay buffer (i.e. was it uploaded by a captured run that was + * NOT dropped as a duplicate/over-cap)? The launch builder stubs every + * called macro slot whose SA entry sits outside this set, so a truncated + * or generation-mismatched capture becomes a no-op instead of a macro + * fetch into never-replayed IR (Graphics Exception: MISSING_MACRO_DATA). + */ +bool gsp_sema_mme_ir_covered(u32 pc); /* * r378 write guards. Both re-validate a guest-memory write target that was diff --git a/kernel/include/gsp_display_engine.h b/kernel/include/gsp_display_engine.h new file mode 100644 index 0000000..064c95c --- /dev/null +++ b/kernel/include/gsp_display_engine.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _GSP_DISPLAY_ENGINE_H_ +#define _GSP_DISPLAY_ENGINE_H_ + +/* + * Minimal host-side display (EVO) engine emulation for the emulated vGPU. + * + * The guest's nvidia-modeset programs the display channels through a small + * push buffer per channel and kicks it by writing the channel's PUT register + * in BAR0 (the UDISP channel control window). On real hardware the display + * engine consumes the methods, advances GET, and - when the push carries an + * UPDATE (NVC37D/NVC57D_UPDATE 0x200) with the completion notifier armed - + * writes an NV_DISP_NOTIFIER (status FINISHED) to the notifier surface the + * guest allocated for the channel. The driver then stops polling. + * + * This engine implements exactly that contract from the host side: + * - display channel registration from the RM control + * NV2080_CTRL_CMD_INTERNAL_DISPLAY_CHANNEL_PUSHBUFFER (0x20800a58), + * which carries the push-buffer guest physical address, size, class and + * instance; + * - the NVC57D method parser (METHOD/JUMP/NONINC/SET_SUBDEVICE_MASK), + * including the wrap JUMP the driver writes at the end of the ring; + * - GET emulation (write consumed offset into the channel control page); + * - SET_CONTEXT_DMA_NOTIFIER / SET_NOTIFIER_CONTROL tracking and the + * FINISHED notifier write on UPDATE; + * - a free-running raster line counter for NVC57E_GET_LINE so the driver's + * vblank/scanline reads make progress; + * - a vblank tick that posts the display event (best-effort; the mode set + * itself only needs the notifier). + * + * Everything here is host-side emulation of hardware behaviour; the guest + * driver is unmodified. + */ + +struct nv_vgpu_device; + +#define GSP_DISP_CLASS_CORE 0xc57d +#define GSP_DISP_CLASS_WINDOW 0xc57e +#define GSP_DISP_CLASS_WINDOW_IMM 0xc57b /* immediate-mode window */ +#define GSP_DISP_CLASS_CURSOR_PIO 0xc57a /* PIO cursor (valid=0) */ + +/* Channel was announced by NV2080_CTRL_CMD_INTERNAL_DISPLAY_CHANNEL_PUSHBUFFER. */ +void gsp_disp_engine_channel_add(struct nv_vgpu_device *vgpu, u32 hclass, + u32 instance, u64 pb_pa, u64 limit, + u32 address_space, bool valid); + +/* + * A display channel object was allocated (RM handle). The push buffer's + * SET_CONTEXT_DMA_NOTIFIER carries a context-DMA handle, and the guest's + * display instance memory hash table maps that handle to the notifier + * surface address; the channel's own hObject is needed to post the + * completion event the driver registered on it. + */ +void gsp_disp_engine_note_object(struct nv_vgpu_device *vgpu, u32 hclass, + u32 instance, u32 h_object); + +/* + * r1765 flip-event shield predicate: is @h_object a registered display + * channel object of @vgpu? The event-post fallback paths consult this to + * keep display events under the display engine's 1:1 posts while + * disp_events_strict is enabled. Returns false while the parameter is 0 + * (the legacy default), so the call sites are inert in legacy mode. + */ +bool gsp_disp_engine_owns_object(struct nv_vgpu_device *vgpu, u32 h_object); + +/* Run one pass for a guest (called from the sema service thread). */ +void gsp_disp_engine_pump(struct nv_vgpu_device *vgpu); + +/* + * Display instance memory location from + * NV2080_CTRL_CMD_INTERNAL_DISPLAY_WRITE_INST_MEM (0x20800a49): + * instMemPhysAddr / instMemSize / instMemAddrSpace. The notifier + * context-DMA handles resolve through this block's hash table. + */ +void gsp_disp_engine_inst_mem(struct nv_vgpu_device *vgpu, u64 pa, u64 size, + u32 addr_space); + +/* Drop all display channels of one guest (mdev cleanup). */ +void gsp_disp_engine_vgpu_cleanup(struct nv_vgpu_device *vgpu); + +int gsp_disp_engine_init(void); +void gsp_disp_engine_fini(void); + +#endif /* _GSP_DISPLAY_ENGINE_H_ */ diff --git a/kernel/main.c b/kernel/main.c index 0110372..cb296f6 100644 --- a/kernel/main.c +++ b/kernel/main.c @@ -849,25 +849,36 @@ static ssize_t nv_vgpu_vfio_dev_write(struct vfio_device *vdev, const char __use if (copy_from_user(tmp, buf, count)) return -EFAULT; - if ((u64)pos == 0x88704 || (gsp_sema_dbg && bar0_log < 64)) { + if ((u64)pos >= 0x88000 && (u64)pos < 0x89000) { + /* + * SESSION 153-CONTINUED-34: ANY write in the PCFG + * window is a submission doorbell, not only the + * virtual-function doorbell at 0x88704 the old code + * dispatched. Feed every 4-byte write to the doorbell + * path so the pump is woken for the client's channel + * kicks no matter which register the driver poked + * (the service thread also watches the mmap'd shadow + * for writes that never trap). Logging stays bounded. + */ static u32 db_log; u64 v = 0; memcpy(&v, tmp, count); - - if ((u64)pos == 0x88704) { - if (gsp_sema_dbg && db_log < 512) { - pr_info("vgsp: DOORBELL write pos=0x%llx size=%zu val=0x%llx\n", - (u64)pos, count, v); - db_log++; - } - if (count == 4) - gsp_sema_service_doorbell((u32)v); - } else { - pr_info("vgsp: BAR0 write pos=0x%llx size=%zu val=0x%llx\n", + if (gsp_sema_dbg && db_log < 512) { + pr_info("vgsp: PCFG write pos=0x%llx size=%zu val=0x%llx\n", (u64)pos, count, v); - bar0_log++; + db_log++; } + if (count == 4) + gsp_sema_service_doorbell((u32)v); + } else if ((u64)pos == 0x88704 || + (gsp_sema_dbg && bar0_log < 64)) { + u64 v = 0; + + memcpy(&v, tmp, count); + pr_info("vgsp: BAR0 write pos=0x%llx size=%zu val=0x%llx\n", + (u64)pos, count, v); + bar0_log++; } ret = nv_vgpu_bar_emul_write(vgpu, 0, (u64)pos, tmp, count); @@ -896,6 +907,28 @@ static ssize_t nv_vgpu_vfio_dev_write(struct vfio_device *vdev, const char __use if (copy_from_user(vgpu->bars[1].vaddr + bar1_off, buf, count)) return -EFAULT; + /* + * SESSION 154 r1506: the X screen (VA 0x120400000, 900 pages) + * is the BAR1 FB shadow at 0x200000. Guest CPU writes into + * that range are either the X server's own fills (expected) + * or another client's BAR1 upload that overlaps the screen + * allocation (the shader-cache collision). Name them so the + * owner of the pollution is measured, not inferred. + */ + { + static u32 bar1_scr_log; + + if (bar1_off >= 0x200000u && bar1_off < 0x600000u && + bar1_scr_log < 64u && count <= 8) { + u64 v = 0; + + memcpy(&v, vgpu->bars[1].vaddr + bar1_off, + count); + pr_info("vgsp: BAR1-SCREEN-WRITE off=0x%llx size=%zu val=0x%llx (screen range 0x200000-0x584000)\n", + bar1_off, count, v); + bar1_scr_log++; + } + } *ppos += count; return count; } @@ -2554,6 +2587,25 @@ module_param_cb(sema_probe, &nv_vgpu_sema_probe_ops, NULL, 0200); MODULE_PARM_DESC(sema_probe, "Dump sema-service map + slot values for a GPU VA"); /* + * ISSUE-0002 aid: force a client-session flush past the live-channel skip. + * Write anything (echo 1 > .../parameters/sema_force_flush). + */ +static int nv_vgpu_sema_force_flush_set(const char *val, + const struct kernel_param *kp) +{ + gsp_sema_service_sess_flush_force(); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_force_flush_ops = { + .set = nv_vgpu_sema_force_flush_set, +}; + +module_param_cb(sema_force_flush, &nv_vgpu_sema_force_flush_ops, NULL, 0200); +MODULE_PARM_DESC(sema_force_flush, + "Force a client-session flush even with live channels (ISSUE-0002 aid)"); + +/* * Register the guest PMA scrubber channel buffer words (finish payload PA, * host-semaphore PA) the guest scanner resolved: * echo 0x11a8ff004,0x11a8ff000 > .../parameters/sema_scrubbuf_add @@ -2585,6 +2637,87 @@ MODULE_PARM_DESC(sema_scrubbuf_add, "Register the PMA scrubber finish/host-sema PAs (hex fin,hex hsem)"); /* + * r1780 (V34): bounded page-directory scan for a content-verified pair: + * echo 0x<va>,0x<pa>[,0x<h_vas>[,0x<ram_lim>]] > .../parameters/sema_pdb_scan + * Walks every 4 KB candidate root in the FB shadow (then optional guest + * RAM) for `va` and registers the root that reproduces `pa` under `h_vas`. + */ +static int nv_vgpu_sema_pdb_scan_set(const char *val, + const struct kernel_param *kp) +{ + char buf[80]; + char *fields[4] = { NULL }; + u64 v[4] = { 0 }; + int nf = 0, i; + char *p; + + strscpy(buf, val, sizeof(buf)); + p = buf; + for (i = 0; i < 4 && p; i++) { + fields[nf++] = p; + p = strchr(p, ','); + if (p) + *p++ = '\0'; + } + for (i = 0; i < nf; i++) + if (kstrtoull(fields[i], 0, &v[i])) + return -EINVAL; + if (nf < 2) + return -EINVAL; + return gsp_sema_service_pdb_scan(v[0], v[1], + nf > 2 ? (u32)v[2] : 0xbeef0300u, + nf > 3 ? v[3] : 0); +} + +static const struct kernel_param_ops nv_vgpu_sema_pdb_scan_ops = { + .set = nv_vgpu_sema_pdb_scan_set, +}; + +module_param_cb(sema_pdb_scan, &nv_vgpu_sema_pdb_scan_ops, NULL, 0200); +MODULE_PARM_DESC(sema_pdb_scan, + "Scan FB/RAM roots for a content-verified VA->PA pair and register the VAS"); + +/* + * r1780 (V34): direct VAS registration: + * echo 0x<h_vas>,0x<pdb>[,<entries>[,<vidmem>]] > .../parameters/sema_vas_force + */ +static int nv_vgpu_sema_vas_force_set(const char *val, + const struct kernel_param *kp) +{ + char buf[80]; + char *fields[4] = { NULL }; + u64 v[4] = { 0 }; + int nf = 0, i; + char *p; + + strscpy(buf, val, sizeof(buf)); + p = buf; + for (i = 0; i < 4 && p; i++) { + fields[nf++] = p; + p = strchr(p, ','); + if (p) + *p++ = '\0'; + } + for (i = 0; i < nf; i++) + if (kstrtoull(fields[i], 0, &v[i])) + return -EINVAL; + if (nf < 2 || !v[0] || !v[1]) + return -EINVAL; + gsp_sema_service_vas_force((u32)v[0], v[1], + nf > 2 ? (u32)v[2] : 4u, + nf > 3 ? !!v[3] : false); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_sema_vas_force_ops = { + .set = nv_vgpu_sema_vas_force_set, +}; + +module_param_cb(sema_vas_force, &nv_vgpu_sema_vas_force_ops, NULL, 0200); +MODULE_PARM_DESC(sema_vas_force, + "Register a VAS directory out-of-band (h,pdb,entries,vidmem)"); + +/* * Debug: run the page-table walks for a GPU VA with per-level tracing, * bypassing the pin/lat/pair map (see gsp_sema_service_walk_probe): * echo 0x20400f000 > .../parameters/walk_probe @@ -2752,6 +2885,10 @@ MODULE_PARM_DESC(sema_copy_dst2, "Second destination window for sema_copy_dst (d * the release the client synchronizes on; the translation usually lands within * a service pass. */ +module_param_named(gsp_phys_gr_peek, gsp_phys_gr_peek, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_peek, + "Peek the ring slots past GP_PUT at each GR launch decision (diagnostic, default 0)"); + module_param_named(sema_ce_grace_ms, gsp_sema_ce_grace_ms, uint, 0644); MODULE_PARM_DESC(sema_ce_grace_ms, "CE pending-entry grace window in ms (default 25000)"); @@ -2764,6 +2901,17 @@ module_param_named(sema_push_hold_ms, gsp_sema_push_hold_ms, uint, 0644); MODULE_PARM_DESC(sema_push_hold_ms, "CE unreadable-entry work-before-release hold in ms (default 2500, 0 disables)"); /* + * SESSION 153-CONTINUED-54: GR setup-push (client MME-setup upload / any + * >= 12 KB GR push) pending hold (ms). An unreadable/all-zero setup push + * keeps its ring entry pending past the failed-rescue consume until the + * demanded 4K-page guest pin lands; 0 disables (demand rows still emitted). + * Default 0: the demand channel cannot yet resolve the GL pushbuffer VA + * families, so the hold only delays the ring (see gsp_sema_service.c). + */ +module_param_named(sema_gr_setup_hold_ms, gsp_sema_gr_setup_hold_ms, uint, 0644); +MODULE_PARM_DESC(sema_gr_setup_hold_ms, "GR setup-push hold in ms (default 0 = off; clamped <=2000): the unreadable 60216 B MME-setup entry stays pending and immature beef draw windows busy-defer instead of IMMATURE-SKIP until the pin/upload lands or the window expires"); + +/* * Grace window (ms) an untranslatable GR entry stays pending before the pump * consumes it as stale, so a late push-buffer pin can still make it readable. */ @@ -2788,6 +2936,43 @@ MODULE_PARM_DESC(gsp_phys_gr_hold_max_ms, "Absolute GR fill hold bound in ms, ke */ module_param_named(gsp_phys_gr_fill_hold, gsp_phys_gr_fill_hold, uint, 0644); MODULE_PARM_DESC(gsp_phys_gr_fill_hold, "GR fill hold on unresolved operand pages (default 0: log-only, never hold)"); + +/* + * r576 DIAG: per-boot witness budget for the GR launch-decision dump (see + * svc_gr_dec_dump in gsp_sema_service.c). Nonzero logs, for the first N + * launches whose fill saw dropped pages / a partial argument set or whose + * decision point carries a dstpend/copy-pending verdict: the decision + * verdicts, the launch's resolved data pages (VA -> guest PA), the live + * deferred-write records and the pending CE opdef entries. 0 (default) = + * off. + */ +module_param_named(gsp_phys_gr_dec_log, gsp_phys_gr_dec_log_max, uint, 0644); +MODULE_PARM_DESC(gsp_phys_gr_dec_log, "GR launch-decision dump budget per boot (default 0: off)"); + +/* + * r576 DIAG: per-boot witness budget for GSP event-post payloads (see + * gsp_rm_evlog_post in gsp_rm_emulation.c). Nonzero logs every + * EVENT_POST_EVENT record actually sent, with its full payload + * (hClient/hEvent/notifyIndex/data) and send result - the evidence needed + * when a client parks waiting for a completion wake-up that never arrives. + * 0 (default) = off. + */ +module_param_named(gsp_rm_ev_log, gsp_rm_ev_log_max, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_log, "GSP event-post payload witness budget per boot (default 0: off)"); + +/* + * r577 experiment knobs for the guest-RM event drop (runtime-writable so + * one boot can A/B). See gsp_rm_emulation.c. All default 0 = legacy. + */ +module_param_named(gsp_rm_ev_seq, gsp_rm_ev_seq, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_seq, "event records carry the status-queue sequence as RPC sequence (default 0: legacy 0)"); +module_param_named(gsp_rm_ev_nlist, gsp_rm_ev_nlist, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_nlist, "set bNotifyList on event records (default 0: specific-hEvent dispatch)"); +module_param_named(gsp_rm_ev_cl, gsp_rm_ev_cl, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_cl, "override event record hClient (default 0: the allocating client)"); +module_param_named(gsp_rm_ev_func, gsp_rm_ev_func, uint, 0644); +MODULE_PARM_DESC(gsp_rm_ev_func, "reachability probe: event records carry this RPC function instead of POST_EVENT 0x1003 (default 0)"); + module_param_named(gsp_sema_walk_trace, gsp_sema_walk_trace, uint, 0644); MODULE_PARM_DESC(gsp_sema_walk_trace, "Trace GMMU walks for unresolved high-VA operands (default 0: off)"); module_param_named(gsp_sema_winfill, gsp_sema_winfill, bool, 0644); @@ -2832,7 +3017,17 @@ module_param_named(sema_wait_gate, gsp_sema_wait_gate, bool, 0644); MODULE_PARM_DESC(sema_wait_gate, "honor C56F host-semaphore acquires before running a ring entry's work (default 1 since r408; 0 restores pass-through)"); module_param_named(sema_wait_hold_ms, gsp_sema_wait_hold_ms, uint, 0644); MODULE_PARM_DESC(sema_wait_hold_ms, "bounded acquire-wait hold before fail-open, ms (r401, default 4000; must exceed the emulated launch latency)"); +module_param_named(sema_push_raw, gsp_sema_push_raw, uint, 0644); +MODULE_PARM_DESC(sema_push_raw, "r481 diagnostic: dump the decoded method stream of the first N large GR pushes (default 0 off)"); MODULE_PARM_DESC(sema_copy_dst_pin_guard, "hold device-dst copies without a pin-resolved translation (default 1)"); +/* + * r609: after a device-dst guard window expires WITHOUT a pin, skip the + * window for further first-write device-dst pages of the same channel for + * this long (ms) - the r606 held-chain serialization fix. 0 restores the + * per-page windows (pre-r609). + */ +module_param_named(sema_copy_dst_age_ms, gsp_sema_copy_dst_age_ms, uint, 0644); +MODULE_PARM_DESC(sema_copy_dst_age_ms, "device-dst guard burst-ageing window after an un-pinned release, ms (r609, default 3000; 0 off)"); /* * r384 (guest boot-corruption lottery): fill-destination guards. A large @@ -2909,6 +3104,61 @@ module_param_cb(sema_map_add, &nv_vgpu_sema_map_add_ops, NULL, 0200); MODULE_PARM_DESC(sema_map_add, "Inject a GPU VA -> guest PA mapping (hex va,hex pa)"); /* + * Debug: retrigger the client-VAS page-directory discovery with a host + * content-verified {ring VA -> ring PA} pair (the ring page's GPFIFO + * entries are the fingerprint): + * echo 0x<va>,0x<pa> > .../parameters/sema_pdb_discover + */ +static int nv_vgpu_sema_pdb_discover_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 va = 0, pa = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &va) || kstrtoull(comma + 1, 0, &pa)) + return -EINVAL; + + return gsp_sema_service_pdb_discover(va, pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_pdb_discover_ops = { + .set = nv_vgpu_sema_pdb_discover_set, +}; + +module_param_cb(sema_pdb_discover, &nv_vgpu_sema_pdb_discover_ops, NULL, 0200); +MODULE_PARM_DESC(sema_pdb_discover, "Retrigger the VAS pdb discovery with a verified ring pair (hex va,hex pa)"); + +/* + * Debug: bounded channel rewind. Clears the consumed-slot bitmap and the + * per-entry rescue state for one channel (hex h_object) or every channel + * ("all"), so the pump re-processes its ring from slot 0 - used after the + * VAS pdb anchor lands so entries skipped before it re-decode (the release + * they carry would otherwise be lost for that put epoch). + */ +static int nv_vgpu_sema_chan_rewind_set(const char *val, + const struct kernel_param *kp) +{ + u32 h = 0xffffffffu; + + if (strncmp(val, "all", 3) && kstrtouint(val, 0, &h)) + return -EINVAL; + return gsp_sema_service_chan_rewind(h); +} + +static const struct kernel_param_ops nv_vgpu_sema_chan_rewind_ops = { + .set = nv_vgpu_sema_chan_rewind_set, +}; + +module_param_cb(sema_chan_rewind, &nv_vgpu_sema_chan_rewind_ops, NULL, 0200); +MODULE_PARM_DESC(sema_chan_rewind, "Rewind a channel's ring (hex h_object or 'all')"); + +/* * Debug: register a channel's GPU completion counter (guest PA of the u32 * tracking counter the GPU would write when a push completes): * echo <h_object>,<pa> > .../parameters/sema_counter @@ -2972,6 +3222,62 @@ module_param_cb(sema_vidmem_dump, &nv_vgpu_sema_vidmem_dump_ops, NULL, 0200); MODULE_PARM_DESC(sema_vidmem_dump, "Dump emulated vidmem (addr,len) to the kernel log"); /* + * X/GL frame-pacer mirror (SESSION 148 perf): mirror the X server's + * submitted-byte counter (guest PA) into its GPU-progress word (guest PA) + * so the driver's per-frame 10 s wait completes in milliseconds. + * echo 0x112345678,0x112345000 > .../parameters/sema_xw_mirror + * echo 0,0 > .../parameters/sema_xw_mirror (stop) + */ +static int nv_vgpu_sema_xw_mirror_set(const char *val, + const struct kernel_param *kp) +{ + char buf[64]; + char *comma; + u64 cnt = 0, mem = 0; + + strscpy(buf, val, sizeof(buf)); + comma = strchr(buf, ','); + if (!comma) + return -EINVAL; + *comma = '\0'; + if (kstrtoull(buf, 0, &cnt) || kstrtoull(comma + 1, 0, &mem)) + return -EINVAL; + + return gsp_sema_service_xw_mirror(cnt, mem); +} + +static const struct kernel_param_ops nv_vgpu_sema_xw_mirror_ops = { + .set = nv_vgpu_sema_xw_mirror_set, +}; + +module_param_cb(sema_xw_mirror, &nv_vgpu_sema_xw_mirror_ops, NULL, 0200); +MODULE_PARM_DESC(sema_xw_mirror, "Mirror X frame counter (cnt_pa,mem_pa; 0,0=stop)"); + +/* + * X/GL submit-kick completion clearer (SESSION 148 r873): PA of the u16 + * status word (member page offset 0xBE) that the X submit path arms to + * 0x8000 and polls until 0. Cleared host-side so the kick settles in ms. + * echo 0x1123456be > .../parameters/sema_xw_comp + * echo 0 > .../parameters/sema_xw_comp (stop) + */ +static int nv_vgpu_sema_xw_comp_set(const char *val, + const struct kernel_param *kp) +{ + u64 pa = 0; + + if (kstrtoull(val, 0, &pa)) + return -EINVAL; + return gsp_sema_service_xw_comp(pa); +} + +static const struct kernel_param_ops nv_vgpu_sema_xw_comp_ops = { + .set = nv_vgpu_sema_xw_comp_set, +}; + +module_param_cb(sema_xw_comp, &nv_vgpu_sema_xw_comp_ops, NULL, 0200); +MODULE_PARM_DESC(sema_xw_comp, "Clear X submit member status at PA (0=stop)"); + +/* * Debug: search the whole emulated vidmem for an 8-byte pattern: * echo 0x0000000400000001,16 > .../parameters/sema_vidmem_find */ @@ -3376,6 +3682,47 @@ static int nv_vgpu_dump_vidmem_set(const char *val, const struct kernel_param *k mutex_unlock(&nv_vgpu_driver.lock); return 0; } + if (!strncmp(buf, "map", 3)) { + /* + * Coarse non-zero map: one line per 1 MB block that holds + * any non-zero dword (SESSION 131: locate the live client's + * uploaded content - the image BO, the code heap, the rings - + * inside the 256 MB aperture without 65536 range dumps). + */ + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + u8 *base = vgpu->bars[1].vaddr; + size_t size = vgpu->bars[1].size; + size_t blk, nblk = (size + (1 << 20) - 1) >> 20; + size_t o; + + if (!base || !size) + continue; + for (blk = 0; blk < nblk; blk++) { + size_t start = blk << 20; + size_t end = start + (1 << 20); + u32 nz = 0, first = 0; + + if (end > size) + end = size; + for (o = start; o + 4 <= end; o += 4) { + u32 v; + + memcpy(&v, base + o, 4); + if (v) { + if (!nz) + first = (u32)(o & 0xFFFFF); + nz++; + } + } + if (nz) + pr_info("vgsp: vidmem map blk 0x%zx nz=%u first=0x%x\n", + start, nz, first); + } + } + mutex_unlock(&nv_vgpu_driver.lock); + return 0; + } comma = strchr(buf, ','); if (comma) @@ -3416,6 +3763,38 @@ module_param_cb(dump_vidmem, &nv_vgpu_dump_vidmem_ops, NULL, 0200); MODULE_PARM_DESC(dump_vidmem, "Dump emulated vidmem range (hex offset,hex len)"); /* + * Whole-vidmem page-hash bracket: snapshot every 4 KB page hash of the + * emulated vidmem, then re-hash and report the pages whose content changed. + * echo 1 > .../parameters/vidmem_hash_snap + * echo 1 > .../parameters/vidmem_hash_diff + */ +static int nv_vgpu_vidmem_hash_snap_set(const char *val, + const struct kernel_param *kp) +{ + return gsp_sema_service_vidmem_hash_snapshot(); +} + +static int nv_vgpu_vidmem_hash_diff_set(const char *val, + const struct kernel_param *kp) +{ + return gsp_sema_service_vidmem_hash_diff("manual", 64); +} + +static const struct kernel_param_ops nv_vgpu_vidmem_hash_snap_ops = { + .set = nv_vgpu_vidmem_hash_snap_set, +}; + +static const struct kernel_param_ops nv_vgpu_vidmem_hash_diff_ops = { + .set = nv_vgpu_vidmem_hash_diff_set, +}; + +module_param_cb(vidmem_hash_snap, &nv_vgpu_vidmem_hash_snap_ops, NULL, 0200); +MODULE_PARM_DESC(vidmem_hash_snap, "Snapshot whole-vidmem page hashes"); +module_param_cb(vidmem_hash_diff, &nv_vgpu_vidmem_hash_diff_ops, NULL, 0200); +MODULE_PARM_DESC(vidmem_hash_diff, + "Diff whole-vidmem page hashes (tag manual, first 64 pages)"); + +/* * Debug: hex-dump or scan the BAR0 register shadow. Guest MMIO writes that go * through the mapped shadow never reach the write callback, so this is the * only way to observe registers the guest writes directly (e.g. the @@ -3842,5 +4221,143 @@ static const struct kernel_param_ops nv_vgpu_rmpost_ops = { module_param_cb(rmpost, &nv_vgpu_rmpost_ops, NULL, 0200); MODULE_PARM_DESC(rmpost, "Post a GSP-RM event to the guest by event index (debug)"); +/* + * Debug: poke a channel's error notifier with an engine-exception shape and + * post its channel event (the relayed-fault delivery edge, manually): + * echo 0x5c000019 > .../parameters/grfaultpoke + * echo 0 > .../parameters/grfaultpoke (every valid channel) + * Experiment: a refused/degenerate GR launch ends its client's + * cudaDeviceSynchronize spin if a synthetic engine error completes the wait + * (r616; see gsp_sema_channel_fault_poke). + */ +struct nv_vgpu_faultpoke_req { + u32 h_object; + int rc; + struct completion done; +}; + +static int nv_vgpu_faultpoke_thread(void *arg) +{ + struct nv_vgpu_faultpoke_req *req = arg; + struct nv_vgpu_device *vgpu; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + if (atomic_read(&vgpu->vfio_dev_open_count) <= 0) + continue; + req->rc = gsp_sema_channel_fault_poke(vgpu, req->h_object); + pr_info("vgsp: grfaultpoke h=0x%x -> %d\n", + req->h_object, req->rc); + } + mutex_unlock(&nv_vgpu_driver.lock); + + complete(&req->done); + return 0; +} + +static int nv_vgpu_faultpoke_set(const char *val, const struct kernel_param *kp) +{ + static struct nv_vgpu_faultpoke_req req; + struct task_struct *task; + + memset(&req, 0, sizeof(req)); + req.rc = -ENODEV; + if (kstrtou32(val, 0, &req.h_object)) + return -EINVAL; + + init_completion(&req.done); + task = kthread_run(nv_vgpu_faultpoke_thread, &req, "vgpu-faultpoke"); + if (IS_ERR(task)) + return PTR_ERR(task); + wait_for_completion(&req.done); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_faultpoke_ops = { + .set = nv_vgpu_faultpoke_set, +}; + +module_param_cb(grfaultpoke, &nv_vgpu_faultpoke_ops, NULL, 0200); +MODULE_PARM_DESC(grfaultpoke, "Write a channel error notifier + post its event (r616 experiment)"); + +/* + * Debug r623: deliver a synthetic OS_ERROR_LOG (0x1006) engine-fault report + * to the guest's RM queue: + * echo 1 > .../parameters/oserrlog + * echo 0x1,0,0 > .../parameters/oserrlog (exceptType,runlistId,chid) + * Experiment: after a synth-fault, the next client sticks pre-submission; if + * the guest's fault handling completes only when the 0x1006 report arrives, + * this record is the missing recovery half. + */ +struct nv_vgpu_oserrlog_req { + u32 except_type, runlist_id, chid; + int rc; + struct completion done; +}; + +static int nv_vgpu_oserrlog_thread(void *arg) +{ + struct nv_vgpu_oserrlog_req *req = arg; + struct nv_vgpu_device *vgpu; + + mutex_lock(&nv_vgpu_driver.lock); + list_for_each_entry(vgpu, &nv_vgpu_driver.devices, list) { + if (atomic_read(&vgpu->vfio_dev_open_count) <= 0) + continue; + req->rc = gsp_rm_send_os_error_log(vgpu, req->except_type, + req->runlist_id, req->chid, + "GR"); + pr_info("vgsp: oserrlog except=0x%x rl=%u chid=%u -> %d\n", + req->except_type, req->runlist_id, req->chid, req->rc); + } + mutex_unlock(&nv_vgpu_driver.lock); + + complete(&req->done); + return 0; +} + +static int nv_vgpu_oserrlog_set(const char *val, const struct kernel_param *kp) +{ + static struct nv_vgpu_oserrlog_req req; + struct task_struct *task; + char buf[64]; + char *c1, *c2; + + memset(&req, 0, sizeof(req)); + req.rc = -ENODEV; + strscpy(buf, val, sizeof(buf)); + req.except_type = 1; /* generic engine exception, as the synth-fault */ + { + c1 = strchr(buf, ','); + if (c1) { + *c1++ = '\0'; + c2 = strchr(c1, ','); + if (c2) { + *c2++ = '\0'; + if (kstrtou32(c2, 0, &req.chid)) + return -EINVAL; + } + if (kstrtou32(c1, 0, &req.runlist_id)) + return -EINVAL; + } + if (kstrtou32(buf, 0, &req.except_type)) + return -EINVAL; + } + + init_completion(&req.done); + task = kthread_run(nv_vgpu_oserrlog_thread, &req, "vgpu-oserrlog"); + if (IS_ERR(task)) + return PTR_ERR(task); + wait_for_completion(&req.done); + return 0; +} + +static const struct kernel_param_ops nv_vgpu_oserrlog_ops = { + .set = nv_vgpu_oserrlog_set, +}; + +module_param_cb(oserrlog, &nv_vgpu_oserrlog_ops, NULL, 0200); +MODULE_PARM_DESC(oserrlog, "Send a synthetic OS_ERROR_LOG (0x1006) RM record to the guest (r623 experiment)"); + module_init(nv_vgpu_init); module_exit(nv_vgpu_exit); diff --git a/tools/cuda-ab/battery_resume.sh b/tools/cuda-ab/battery_resume.sh new file mode 100755 index 0000000..33bbe86 --- /dev/null +++ b/tools/cuda-ab/battery_resume.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# battery_resume2.sh - resume a battery after a partial stop, with a +# zombie-tolerant completion check: a client is DONE when its .bat.out holds +# a verdict line (STATUS OK|FAIL|CUDA_ERROR) even if a zombie with the same +# comm lingers (the pgrep-x poll in battery_smileylast falsely hangs on +# those). A client with an alive non-zombie process and no verdict after +# HANG_S is a real hang. +# Usage: battery_resume2.sh <outfile> <client...> +set -u +cd /home/rmcore/Git/integdev_gpu_drv/open-iov +OUT="${1:?usage: battery_resume2.sh <out> <client...>}"; shift +CLIENTS="$*" +HANG_S=600 +: >> "$OUT" +pass=0; fail=0; hang=0 +for c in $CLIENTS; do + echo "=== $c ===" | tee -a "$OUT" + timeout 40 scripts/ssh_guest.sh "test -x /tmp/$c || exit 9; cd /tmp && rm -f /tmp/$c.bat.out && setsid nohup ./$c > /tmp/$c.bat.out 2>&1 < /dev/null & echo LAUNCHED" >/dev/null 2>&1 + t0=$(date +%s); done_ok=0 + while :; do + sleep 5 + block=$(timeout 25 scripts/ssh_guest.sh "cat /tmp/$c.bat.out 2>/dev/null" 2>/dev/null | grep -v "Permanently added") + if printf '%s\n' "$block" | grep -qE "STATUS OK|STATUS FAIL|CUDA_ERROR"; then done_ok=1; break; fi + # alive non-zombie process with no verdict? + alive=$(printf '%s\n' "$block"; timeout 25 scripts/ssh_guest.sh "ps -eo stat=,comm= | awk -v n='$c' '\$2==n && \$1 !~ /^Z/' | wc -l" 2>/dev/null | tail -1) + now=$(date +%s) + if [ $((now - t0)) -ge $HANG_S ]; then + echo "HANG: $c no verdict after ${HANG_S}s" | tee -a "$OUT" + hang=$((hang+1)) + echo "BATTERY RESUME STOPPED at $c (hang; boot preserved, do not kill)" | tee -a "$OUT" + echo "BATTERY RESUME pass=$pass fail=$fail hang=$hang" | tee -a "$OUT" + exit 0 + fi + done + printf '%s\n' "$block" | tail -8 | tee -a "$OUT" + if printf '%s\n' "$block" | grep -q "STATUS OK"; then pass=$((pass+1)); else fail=$((fail+1)); fi + echo "--- done ---" | tee -a "$OUT" +done +echo "BATTERY RESUME pass=$pass fail=$fail hang=$hang" | tee -a "$OUT"
\ No newline at end of file diff --git a/tools/cuda-ab/evhold.cu b/tools/cuda-ab/evhold.cu new file mode 100644 index 0000000..3817ff3 --- /dev/null +++ b/tools/cuda-ab/evhold.cu @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * evhold.cu - deterministic event-holder / event-wait probe for the guest + * event-delivery class (r577). Opens a CUDA context (so the driver + * registers its RM events), runs one small op, records and synchronizes a + * CUDA event (which exercises the client's wait path), then holds the + * context open for N seconds so host-side experiments (rmpost rounds with + * the gsp_rm_ev_* knobs, ev_post kprobe) have a live event table to post + * into. + * + * Usage: evhold [hold_seconds] + * Output lines: + * EVHOLD ctx ok + * EVHOLD event sync ok + * EVHOLD hold done + */ +#include <cstdio> +#include <cstdlib> +#include <unistd.h> +#include <cuda_runtime.h> + +static void check(const char *what, cudaError_t e) +{ + if (e != cudaSuccess) { + printf("EVHOLD FAIL %s: %s\n", what, cudaGetErrorString(e)); + fflush(stdout); + exit(1); + } +} + +int main(int argc, char **argv) +{ + int hold = (argc > 1) ? atoi(argv[1]) : 120; + float *d = NULL; + cudaEvent_t ev; + + check("freectx", cudaFree(0)); + printf("EVHOLD ctx ok\n"); + fflush(stdout); + + check("malloc", cudaMalloc(&d, 4096)); + check("memset", cudaMemset(d, 0, 4096)); + check("evcreate", cudaEventCreate(&ev)); + check("evrecord", cudaEventRecord(ev, 0)); + check("evsync", cudaEventSynchronize(ev)); + printf("EVHOLD event sync ok\n"); + fflush(stdout); + + for (int i = 0; i < hold; i++) + sleep(1); + + cudaEventDestroy(ev); + cudaFree(d); + printf("EVHOLD hold done\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe.cu b/tools/cuda-ab/ip_probe.cu new file mode 100644 index 0000000..366fbc8 --- /dev/null +++ b/tools/cuda-ab/ip_probe.cu @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe.cu - how many CTAs actually executed for an in-place kernel? + * + * scale_mark marks progress[blockIdx.x]=0xCAFEBABE after the c store; D2H of + * progress counts executed CTAs exactly. c (the in-place buffer) is read back + * and verified. seed (argv[1]) makes the run content-unique. + * + * Build: nvcc -O2 --cudart=static -o ip_probe ip_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_mark(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g grid=%d\n", prop.name, (double)k, grid); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + /* in-place scale with CTA markers */ + scale_mark<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0, first_missing = -1; + + for (i = 0; i < grid; i++) { + if (h_prog[i] == 0xCAFEBABEu) + marked++; + else if (first_missing < 0) + first_missing = i; + } + printf("CTA-MARK marked=%d/%d first_missing=%d\n", marked, grid, + first_missing); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + if (first >= 0) + printf("SCALE firstbad got=%g want=%g\n", hc[first], + (ha[first] + hb[first]) * 2.0f); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe2.cu b/tools/cuda-ab/ip_probe2.cu new file mode 100644 index 0000000..a5fa2ec --- /dev/null +++ b/tools/cuda-ab/ip_probe2.cu @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe2.cu - why do in-place c stores vanish while prog stores land? + * + * After a vadd that fills c, three variants run on the SAME c: + * 1. scale_if : if (i<n) c[i] *= s; (control, like ip_probe) + * 2. scale_u : c[i] *= s; (unconditional, grid exact) + * 3. fill_c : c[i] = 7.0f; (store only, no load) + * 4. add_c : c[i] = c[i] + 1.0f; (load + store) + * Each stage reads c back and compares against the expected value at that + * stage. A separate prog buffer marks CTA execution. + * + * Build: nvcc -O2 --cudart=static -o ip_probe2 ip_probe2.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_if(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xCAFEBABEu; +} + +__global__ void scale_u(float *c, float s, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] * s; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xDEADBEEFu; +} + +__global__ void fill_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = 7.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0xFEEDFACu; +} + +__global__ void add_c(float *c, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + c[i] = c[i] + 1.0f; + if (threadIdx.x == 0) + prog[blockIdx.x] = 0x0DDF00Du; +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[128]; + int i, stage; + float mult; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 128 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + mult = 2.0f; + for (stage = 0; stage < 4; stage++) { + unsigned mark = 0; + + CHECK(cudaMemset(d_prog, 0xEE, 128 * sizeof(unsigned))); + switch (stage) { + case 0: + scale_if<<<grid, 256>>>(dc, 2.0f, n, d_prog); + mult = 2.0f; + mark = 0xCAFEBABEu; + break; + case 1: + scale_u<<<grid, 256>>>(dc, 2.0f, d_prog); + mult = 4.0f; + mark = 0xDEADBEEFu; + break; + case 2: + fill_c<<<grid, 256>>>(dc, d_prog); + mark = 0xFEEDFACu; + break; + default: + add_c<<<grid, 256>>>(dc, d_prog); + mult = -1.0f; + mark = 0x0DDF00Du; + break; + } + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 128 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int marked = 0; + + for (i = 0; i < grid; i++) + if (h_prog[i] == mark) + marked++; + printf("S%d CTA-MARK marked=%d/%d\n", stage + 1, + marked, grid); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + if (stage < 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) { + b++; + if (first < 0) + first = i; + } + printf("S%d readback bad=%d/%d first=%d\n", + stage + 1, b, n, first); + } else if (stage == 2) { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 7.0f) { + b++; + if (first < 0) + first = i; + } + printf("S3 readback bad=%d/%d first=%d (fill 7.0)\n", + b, n, first); + } else { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != 8.0f) { + b++; + if (first < 0) + first = i; + } + printf("S4 readback bad=%d/%d first=%d (add 1.0 to fill)\n", + b, n, first); + } + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/ip_probe3.cu b/tools/cuda-ab/ip_probe3.cu new file mode 100644 index 0000000..a50df0e --- /dev/null +++ b/tools/cuda-ab/ip_probe3.cu @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * ip_probe3.cu - what does the SM actually READ for the scale kernel? + * + * scale_report writes, per CTA: prog[2b] = bits of the float param s, + * prog[2b+1] = bits of c[b*256] (the load). + * The D2H of prog then shows exactly what the SM saw, independent of any + * store-visibility path for c. + * + * Build: nvcc -O2 --cudart=static -o ip_probe3 ip_probe3.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale_report(float *c, float s, int n, unsigned *prog) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; + if (threadIdx.x == 0) { + prog[2 * blockIdx.x] = __float_as_uint(s); + prog[2 * blockIdx.x + 1] = __float_as_uint(c[blockIdx.x * 256]); + } +} + +int main(int argc, char **argv) +{ + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float k = argc > 1 ? atof(argv[1]) : 0.0f; + float *ha, *hb, *hc; + float *da, *db, *dc; + unsigned *d_prog, h_prog[256]; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s seed=%g\n", prop.name, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&d_prog, 256 * sizeof(unsigned))); + printf("DPTRS da=%p db=%p dc=%p prog=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)d_prog); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemset(d_prog, 0xEE, 256 * sizeof(unsigned))); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("VADD readback bad=%d/%d\n", b, n); + } + + scale_report<<<grid, 256>>>(dc, 2.0f, n, d_prog); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_prog, d_prog, 256 * sizeof(unsigned), + cudaMemcpyDeviceToHost)); + { + int s_ok = 0, s_1 = 0, c_ok = 0, n_seen = 0; + unsigned first_s = 0, first_c = 0; + + for (i = 0; i < grid; i++) { + unsigned sb = h_prog[2 * i]; + unsigned cb = h_prog[2 * i + 1]; + + if (i == 0) { + first_s = sb; + first_c = cb; + } + if (sb == 0x40000000u) + s_ok++; + if (sb == 0x3F800000u) + s_1++; + { + float want = ha[i * 256] + hb[i * 256]; + + if (cb == *(unsigned *)&want) + c_ok++; + } + n_seen++; + } + printf("REPORT ctas=%d s_eq_2.0=%d s_eq_1.0=%d c_load_ok=%d first_s=0x%08x first_c=0x%08x\n", + n_seen, s_ok, s_1, c_ok, first_s, first_c); + } + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, first = -1; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * 2.0f) { + b++; + if (first < 0) + first = i; + } + printf("SCALE readback bad=%d/%d first=%d\n", b, n, first); + } + printf("STATUS DONE\n"); + return 0; +} diff --git a/tools/cuda-ab/memset2d_probe.cu b/tools/cuda-ab/memset2d_probe.cu new file mode 100644 index 0000000..acf43d4 --- /dev/null +++ b/tools/cuda-ab/memset2d_probe.cu @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * memset2d_probe.cu - per-phase diagnostics for victory_memset2d. + * + * victory_memset2d reports one aggregate bad count (~12.5M) that mixes all + * three phases. This probe runs the SAME three phases and reports each + * phase's bad count, the first mismatching byte (offset, got, want), and a + * per-phase 1MB-chunk histogram so the failing phase and the failing region + * are attributable in one run. + * + * Phases: + * P1 memset : cudaMemset(d_img, 0xA5, pitch*h); D2H 2D readback + * P2 roundtrip: H2D 2D pattern; D2H 2D readback + * P3 memset2d : cudaMemset2D quadrant 0x3C; D2H 2D readback + * + * Output lines: + * DEVICE ... + * PHASE p=1 bad=<n> first_off=<i> got=<g> want=<w> + * PHIST p=1 mb=<m> ok=<n> bad=<n> + * MEMSET2D2 bad_total=<n> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define IMG_W 4096 +#define IMG_H 1024 +#define FILL_BYTE 0xA5 +#define QUAD_BYTE 0x3C + +static int phase_check(const char *tag, const unsigned char *back, + const unsigned char *want, size_t n, int p) +{ + size_t i, first = (size_t)-1; + int bad = 0, got = 0, wantv = 0; + size_t mb; + + for (i = 0; i < n; i++) { + if (back[i] != want[i]) { + if (first == (size_t)-1) { + first = i; + got = back[i]; + wantv = want[i]; + } + bad++; + } + } + printf("PHASE p=%d bad=%d first_off=%zu got=0x%02x want=0x%02x\n", + p, bad, first, got, wantv); + if (bad) { + size_t nb = (n + (1u << 20) - 1) >> 20; + + for (mb = 0; mb < nb && mb < 8; mb++) { + size_t lo = mb << 20, hi = lo + (1u << 20); + size_t j; + int bok = 0, bbad = 0; + + if (hi > n) + hi = n; + for (j = lo; j < hi; j++) { + if (back[j] == want[j]) + bok++; + else + bbad++; + } + printf("PHIST p=%d mb=%zu ok=%d bad=%d\n", + p, mb, bok, bbad); + } + } + (void)tag; + return bad; +} + +int main(void) +{ + cudaDeviceProp prop; + const size_t w = IMG_W, h = IMG_H; + unsigned char *h_img, *h_back, *h_want; + unsigned char *d_img; + size_t pitch = 0; + int bad = 0, row, col; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + h_img = (unsigned char *)malloc(w * h); + h_back = (unsigned char *)malloc(w * h); + h_want = (unsigned char *)malloc(w * h); + if (!h_img || !h_back || !h_want) + return 1; + + CHECK(cudaMallocPitch(&d_img, &pitch, w, h)); + printf("PITCH pitch=%zu\n", pitch); + printf("DPTRS h_img=%p h_back=%p h_want=%p d_img=%p\n", (void *)h_img, + (void *)h_back, (void *)h_want, (void *)d_img); + + /* P1: full cudaMemset over the pitched allocation, 2D readback. */ + CHECK(cudaMemset(d_img, FILL_BYTE, pitch * h)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + memset(h_want, FILL_BYTE, w * h); + bad += phase_check("P1", h_back, h_want, w * h, 1); + + /* P2: pitched H2D pattern -> D2H round trip. */ + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) + h_img[row * w + col] = (unsigned char)((row + col) & 0xFF); + CHECK(cudaMemcpy2D(d_img, pitch, h_img, w, w, h, + cudaMemcpyHostToDevice)); + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P2", h_back, h_img, w * h, 2); + + /* P3: cudaMemset2D of the upper-left quadrant only. */ + CHECK(cudaMemset2D(d_img, pitch, QUAD_BYTE, w / 2, h / 2)); + for (row = 0; row < (int)h; row++) + for (col = 0; col < (int)w; col++) { + unsigned char expect = h_img[row * w + col]; + + if (row < (int)(h / 2) && col < (int)(w / 2)) + expect = QUAD_BYTE; + h_want[row * w + col] = expect; + } + memset(h_back, 0, w * h); + CHECK(cudaMemcpy2D(h_back, w, d_img, pitch, w, h, + cudaMemcpyDeviceToHost)); + bad += phase_check("P3", h_back, h_want, w * h, 3); + + printf("MEMSET2D2 bad_total=%d\n", bad); + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/push_battery.sh b/tools/cuda-ab/push_battery.sh index 644beba..7bd3df9 100755 --- a/tools/cuda-ab/push_battery.sh +++ b/tools/cuda-ab/push_battery.sh @@ -6,7 +6,7 @@ set -u HERE="$(cd "$(dirname "$0")" && pwd)" RUN="$HERE/run" GUEST_DIR="${1:-/tmp}" -CLIENTS="victory_mail smiley vecadd_loop streams2 memcpy_roundtrip multikernel victory_saxpy victory_reduce victory_matmul victory_strided victory_memset2d victory_atomics victory_streams_events victory_math vector_add" +CLIENTS="victory_mail smiley vecadd_loop streams2 memcpy_roundtrip multikernel victory_saxpy victory_reduce victory_matmul victory_strided victory_memset2d victory_atomics victory_streams_events victory_math vector_add inplace_scale streams2_probe streams_events_probe memset2d_probe tgemm_probe transformer transformer_stages ip_probe ip_probe2 ip_probe3 scale_probe rt_probe launch_count egl_frame frame_producer" export SSHPASS="${VGPU_GUEST_PASS:-ubuntu}" FILES="" for c in $CLIENTS; do [ -x "$RUN/$c" ] && FILES="$FILES $RUN/$c"; done diff --git a/tools/cuda-ab/r18_anim.cu b/tools/cuda-ab/r18_anim.cu new file mode 100644 index 0000000..89111be --- /dev/null +++ b/tools/cuda-ab/r18_anim.cu @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * r18_anim.cu - GPU-computed animation for R18 (v3: single-page kernel). + * + * Renders NFRAMES frames of an animated plasma with one launch per frame. + * The emulated forward path delivers the kernel through ONE 4 KB code page + * (the window shadow copies the program page; multi-page kernels execute off + * the end of the mapped page - the 904-instruction v2 kernel faulted with + * "Illegal Instruction Encoding" on the real engine). This version keeps the + * SASS text under 4096 bytes: 512x360 RGBA8 (power-of-two width for cheap + * index math), grid-stride loop, three MUFU.SIN under --use_fast_math. + * + * ONE 0.9 MB device frame buffer is reused for every frame (small-alloc + * regime, matching the proven CUDA path); after each launch the frame is + * copied back (DtoH) and APPENDED to /dev/shm with a magic header so the + * host can find it by scanning the guest RAM file. The file is + * + * "R18FRM01" u32 count u32 width u32 height u32 stride + * then count*height*stride bytes of RGBA8 + * + * The header is written first with the final count and bytes are appended + * (flushed) per frame, so a host-side harvest can recover every completed + * frame even if the client or guest wedges mid-run. + * + * Build (host): + * nvcc -O2 --use_fast_math --cudart=static -arch=sm_75 -o r18_anim r18_anim.cu + * Usage: r18_anim [frames] [outpath] [blocks] + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <ctime> +#include <cuda_runtime.h> + +#define WIDTH 512 +#define HEIGHT 360 +#define NFRAMES 60 +#define OUTPATH "/dev/shm/r18frames.bin" + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +/* 3 MUFU.SIN only - the text must stay under one 4 KB page. */ +__global__ void plasma_kernel(uchar4 *fb, int w, int h, int frame) +{ + unsigned stride = gridDim.x * blockDim.x; + unsigned xmask = (unsigned)w - 1u; /* w is a power of two */ + int idx; + + for (idx = blockIdx.x * blockDim.x + threadIdx.x; idx < w * h; + idx += stride) { + float u = (float)((unsigned)idx & xmask) * (1.0f / 512.0f); + float v = (float)((unsigned)idx >> 9) * (1.0f / 360.0f); + float t = (float)frame * 0.10f; + float p = sinf(u * 9.0f + t * 2.0f) + + sinf(v * 7.0f - t * 2.0f); + float s = sinf(p * 1.5f); + + p *= 0.5f; /* -1 .. 1 */ + fb[idx] = make_uchar4((unsigned char)(127.5f + 127.5f * s), + (unsigned char)(127.5f + 127.5f * p), + (unsigned char)(127.5f - 127.5f * s), + 255); + } +} + +static uint64_t fnv1a(const unsigned char *p, size_t n) +{ + uint64_t h = 1469598103934665603ull; + + while (n--) { + h ^= *p++; + h *= 1099511628211ull; + } + return h; +} + +static double now_ms(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (double)ts.tv_sec * 1e3 + (double)ts.tv_nsec / 1e6; +} + +int main(int argc, char **argv) +{ + int nframes = NFRAMES; + const char *outpath = OUTPATH; + size_t fbytes = (size_t)WIDTH * HEIGHT * 4; + uchar4 *d_fb = NULL; + unsigned char *h_fb = NULL; + double km = 0.0, cm = 0.0, t0, t1, wall0 = now_ms(); + int blocks = 720; + int f, dev = -1; + FILE *fp; + uint32_t hdr[4]; + + if (argc > 1) + nframes = atoi(argv[1]); + if (nframes <= 0 || nframes > 60000) + nframes = NFRAMES; + if (argc > 2) + outpath = argv[2]; + if (argc > 3) + blocks = atoi(argv[3]); + if (blocks <= 0 || blocks > 4096) + blocks = 720; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDevice(&dev)); + { + cudaDeviceProp prop; + + if (cudaGetDeviceProperties(&prop, dev) == cudaSuccess) + printf("DEVICE %s cc=%d.%d\n", prop.name, + prop.major, prop.minor); + } + printf("ANIM start frames=%d %dx%d blocks=%d out=%s\n", + nframes, WIDTH, HEIGHT, blocks, outpath); + + CHECK(cudaMalloc(&d_fb, fbytes)); + h_fb = (unsigned char *)malloc(fbytes); + if (!h_fb) { + printf("HOST_ALLOC_FAIL\nSTATUS FAIL\n"); + return 1; + } + + fp = fopen(outpath, "wb"); + if (!fp) { + printf("OPEN_FAIL %s\nSTATUS FAIL\n", outpath); + return 1; + } + hdr[0] = (uint32_t)nframes; + hdr[1] = WIDTH; + hdr[2] = HEIGHT; + hdr[3] = WIDTH * 4; + if (fwrite("R18FRM01", 1, 8, fp) != 8 || + fwrite(hdr, 1, sizeof(hdr), fp) != sizeof(hdr) || fflush(fp)) { + printf("WRITE_FAIL header\nSTATUS FAIL\n"); + fclose(fp); + return 1; + } + + for (f = 0; f < nframes; f++) { + uchar4 *dst = d_fb; + uint64_t sum = 0; + size_t i; + + t0 = now_ms(); + plasma_kernel<<<blocks, 256>>>(dst, WIDTH, HEIGHT, f); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + t1 = now_ms(); + km += t1 - t0; + CHECK(cudaMemcpy(h_fb, dst, fbytes, cudaMemcpyDeviceToHost)); + cm += now_ms() - t1; + + for (i = 0; i < fbytes; i++) + sum += h_fb[i]; + if (fwrite(h_fb, 1, fbytes, fp) != fbytes || fflush(fp)) { + printf("WRITE_FAIL frame %d\nSTATUS FAIL\n", f); + fclose(fp); + return 1; + } + printf("FRAME %03d fnv=0x%016llx mean=%u kms=%.1f cms=%.1f wall=%.1f\n", + f, (unsigned long long)fnv1a(h_fb, fbytes), + (unsigned)(sum / fbytes), km, cm, now_ms() - wall0); + } + + fclose(fp); + printf("KERNEL frames=%d %dx%d blocks=%d launches=%d total_ms=%.2f per_frame_ms=%.2f\n", + nframes, WIDTH, HEIGHT, blocks, nframes, km, km / nframes); + printf("COPY frames=%d total_ms=%.2f per_frame_ms=%.2f\n", nframes, + cm, cm / nframes); + printf("WRITE path=%s bytes=%zu magic=R18FRM01 count=%d %dx%d stride=%d wall_ms=%.1f\n", + outpath, 24 + fbytes * (size_t)nframes, nframes, WIDTH, HEIGHT, + WIDTH * 4, now_ms() - wall0); + printf("STATUS OK\n"); + + CHECK(cudaFree(d_fb)); + free(h_fb); + return 0; +} diff --git a/tools/cuda-ab/rt_probe.cu b/tools/cuda-ab/rt_probe.cu new file mode 100644 index 0000000..025301d --- /dev/null +++ b/tools/cuda-ab/rt_probe.cu @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * rt_probe.cu - instrumented memcpy_roundtrip variant isolating the + * memset-phase D2H readback failure. + * + * Phases (argv[1]): + * "full" : H2D -> kernel -> D2H -> memset -> D2H (the original order) + * "sync" : same but cudaDeviceSynchronize() before the second D2H + * "memset" : memset -> D2H only (memset_probe shape, 3-buffer alloc) + * "rev" : memset -> D2H first, then H2D -> kernel -> D2H + * + * Forensics on the memset readback: first bad offset, byte histogram + * (0x5A / 0x00 / other), per-1MB chunk histogram, and the first 32 bytes. + * + * Build: nvcc -O2 --cudart=static -o rt_probe rt_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define MemsetByte 0x5A + +__global__ void bump(const float *in, float *out, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + out[i] = in[i] + 0.5f; +} + +static void memset_forensics(const unsigned char *h, size_t bytes, + const char *tag) +{ + size_t i, first_bad = (size_t)-1; + int bad = 0, n5a = 0, n00 = 0, nother = 0; + size_t chunk; + + for (i = 0; i < bytes; i++) { + if (h[i] == 0x5A) + n5a++; + else if (h[i] == 0x00) + n00++; + else + nother++; + if (h[i] != 0x5A) { + if (first_bad == (size_t)-1) + first_bad = i; + bad++; + } + } + printf("MEMSET-READ %s bad=%d first_bad=%zu n5a=%d n00=%d nother=%d\n", + tag, bad, first_bad, n5a, n00, nother); + printf("MEMSET-HEAD %s", tag); + for (i = 0; i < 32; i++) + printf(" %02x", h[i]); + printf("\n"); + if (first_bad != (size_t)-1) { + printf("MEMSET-ATBAD %s off=%zu:", tag, first_bad); + for (i = first_bad; i < first_bad + 16 && i < bytes; i++) + printf(" %02x", h[i]); + printf("\n"); + } + for (chunk = 0; chunk < bytes / (1u << 20); chunk++) { + size_t lo = chunk << 20, hi = lo + (1u << 20); + int c5a = 0, c00 = 0, cother = 0; + + for (i = lo; i < hi; i++) { + if (h[i] == 0x5A) + c5a++; + else if (h[i] == 0x00) + c00++; + else + cother++; + } + printf("MEMSET-MB %s mb=%zu 5a=%d 00=%d other=%d\n", + tag, chunk, c5a, c00, cother); + } +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "full"; + cudaDeviceProp prop; + const size_t bytes = 8u << 20; + const int n = (int)(bytes / sizeof(float)); + unsigned char *h_m; + float *h_in, *h_out; + float *d_in, *d_out; + unsigned char *d_m; + double csum = 0.0; + int bad = 0, i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu mode=%s\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem, mode); + + h_in = (float *)malloc(bytes); + h_out = (float *)malloc(bytes); + h_m = (unsigned char *)malloc(bytes); + if (!h_in || !h_out || !h_m) + return 1; + + for (i = 0; i < n; i++) + h_in[i] = (float)(i % 1024) * 0.25f; + + CHECK(cudaMalloc(&d_in, bytes)); + CHECK(cudaMalloc(&d_out, bytes)); + CHECK(cudaMalloc(&d_m, bytes)); + printf("DPTRS d_in=%p d_out=%p d_m=%p\n", (void *)d_in, + (void *)d_out, (void *)d_m); + + if (!strcmp(mode, "memset") || !strcmp(mode, "rev")) { + /* memset phase first */ + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "early"); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync") || + !strcmp(mode, "rev")) { + CHECK(cudaMemcpy(d_in, h_in, bytes, cudaMemcpyHostToDevice)); + bump<<<grid, 256>>>(d_in, d_out, n); + CHECK(cudaGetLastError()); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, bytes, cudaMemcpyDeviceToHost)); + + for (i = 0; i < n; i++) { + if (h_out[i] != h_in[i] + 0.5f) + bad++; + csum += (double)h_out[i]; + } + printf("BUMP bad=%d csum=%.6f\n", bad, csum); + } + + if (!strcmp(mode, "full") || !strcmp(mode, "sync")) { + CHECK(cudaMemset(d_m, MemsetByte, bytes)); + if (!strcmp(mode, "sync")) + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_m, d_m, bytes, cudaMemcpyDeviceToHost)); + memset_forensics(h_m, bytes, "late"); + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/run/egl_frame b/tools/cuda-ab/run/egl_frame Binary files differnew file mode 100755 index 0000000..130f032 --- /dev/null +++ b/tools/cuda-ab/run/egl_frame diff --git a/tools/cuda-ab/run/evhold b/tools/cuda-ab/run/evhold Binary files differnew file mode 100755 index 0000000..de4448c --- /dev/null +++ b/tools/cuda-ab/run/evhold diff --git a/tools/cuda-ab/run/ip4 b/tools/cuda-ab/run/ip4 Binary files differnew file mode 100755 index 0000000..54937e6 --- /dev/null +++ b/tools/cuda-ab/run/ip4 diff --git a/tools/cuda-ab/run/ip5 b/tools/cuda-ab/run/ip5 Binary files differnew file mode 100755 index 0000000..c4b4a39 --- /dev/null +++ b/tools/cuda-ab/run/ip5 diff --git a/tools/cuda-ab/run/ip6 b/tools/cuda-ab/run/ip6 Binary files differnew file mode 100755 index 0000000..56815c2 --- /dev/null +++ b/tools/cuda-ab/run/ip6 diff --git a/tools/cuda-ab/run/ip_probe b/tools/cuda-ab/run/ip_probe Binary files differnew file mode 100755 index 0000000..08fb86e --- /dev/null +++ b/tools/cuda-ab/run/ip_probe diff --git a/tools/cuda-ab/run/ip_probe2 b/tools/cuda-ab/run/ip_probe2 Binary files differnew file mode 100755 index 0000000..dfe6ab9 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe2 diff --git a/tools/cuda-ab/run/ip_probe3 b/tools/cuda-ab/run/ip_probe3 Binary files differnew file mode 100755 index 0000000..8a34c05 --- /dev/null +++ b/tools/cuda-ab/run/ip_probe3 diff --git a/tools/cuda-ab/run/memset2d_probe b/tools/cuda-ab/run/memset2d_probe Binary files differnew file mode 100755 index 0000000..f5bd071 --- /dev/null +++ b/tools/cuda-ab/run/memset2d_probe diff --git a/tools/cuda-ab/run/r18_anim b/tools/cuda-ab/run/r18_anim Binary files differnew file mode 100755 index 0000000..6a944f9 --- /dev/null +++ b/tools/cuda-ab/run/r18_anim diff --git a/tools/cuda-ab/run/rt_probe b/tools/cuda-ab/run/rt_probe Binary files differnew file mode 100755 index 0000000..48d866f --- /dev/null +++ b/tools/cuda-ab/run/rt_probe diff --git a/tools/cuda-ab/run/scale_probe b/tools/cuda-ab/run/scale_probe Binary files differnew file mode 100755 index 0000000..ca3fc20 --- /dev/null +++ b/tools/cuda-ab/run/scale_probe diff --git a/tools/cuda-ab/run/streams2_probe b/tools/cuda-ab/run/streams2_probe Binary files differnew file mode 100755 index 0000000..206d6d6 --- /dev/null +++ b/tools/cuda-ab/run/streams2_probe diff --git a/tools/cuda-ab/run/streams_events_probe b/tools/cuda-ab/run/streams_events_probe Binary files differnew file mode 100755 index 0000000..00f3539 --- /dev/null +++ b/tools/cuda-ab/run/streams_events_probe diff --git a/tools/cuda-ab/run/tgemm_probe b/tools/cuda-ab/run/tgemm_probe Binary files differnew file mode 100755 index 0000000..d9c9aa3 --- /dev/null +++ b/tools/cuda-ab/run/tgemm_probe diff --git a/tools/cuda-ab/scale_probe.cu b/tools/cuda-ab/scale_probe.cu new file mode 100644 index 0000000..8a0df59 --- /dev/null +++ b/tools/cuda-ab/scale_probe.cu @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * scale_probe.cu - localize the deterministic "in-place scale writes only the + * first 4 KB page" failure. + * + * Modes (argv[1]): + * V : vadd(a,b->c); sync; D2H+verify; vadd again; D2H+verify + * I : vadd(a,b->c); sync; D2H+verify; scale(c,*2 in place); sync; + * D2H+verify (with page map); scale again; sync; D2H+verify; D2H again + * O : vadd(a,b->c); sync; D2H+verify; scale_out(c->d,*2); sync; + * D2H+verify (with page map) + * + * Every stage also runs a GPU-side verify kernel that counts matching + * elements reading the device buffers (engine's own view) and writes the + * counts to a small buffer, D2H'd afterwards: separates "device pages never + * written" (GPU count low) from "D2H readback stale" (GPU count high, host + * count low). + * + * Build: nvcc -O2 --cudart=static -o scale_probe scale_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +__global__ void scale_out(const float *c, float *d, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + d[i] = c[i] * s; +} + +/* verify kernel: out[0]=count(c==a+b), out[1]=count(c==2*(a+b)) */ +__global__ void vdigest(const float *a, const float *b, const float *c, int n, + unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c[i] == a[i] + b[i]) + atomicAdd(&out[0], 1ull); + if (c[i] == (a[i] + b[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "V"; + cudaDeviceProp prop; + const int n = 1 << 15; + const int grid = (n + 255) / 256; + size_t bytes = (size_t)n * sizeof(float); + float *ha, *hb, *hc, *hd; + float *da, *db, *dc, *dd; + unsigned long long *d_out, h_out[2]; + int i; + float k = argc > 2 ? atof(argv[2]) : 0.0f; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + { + FILE *f = fopen("/proc/uptime", "r"); + double up = -1.0; + + if (f) { if (fscanf(f, "%lf", &up) != 1) up = -1.0; fclose(f); } + printf("UPTIME %.3f\n", up); + } + printf("DEVICE %s mode=%s seed=%g\n", prop.name, mode, (double)k); + + ha = (float *)malloc(bytes); + hb = (float *)malloc(bytes); + hc = (float *)malloc(bytes); + hd = (float *)malloc(bytes); + for (i = 0; i < n; i++) { + ha[i] = (float)(i % 1000) * 0.5f + k; + hb[i] = (float)(i % 997) * 0.25f + 2.0f * k; + } + memset(hc, 0xCD, bytes); + memset(hd, 0xCD, bytes); + + CHECK(cudaMalloc(&da, bytes)); + CHECK(cudaMalloc(&db, bytes)); + CHECK(cudaMalloc(&dc, bytes)); + CHECK(cudaMalloc(&dd, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da=%p db=%p dc=%p dd=%p\n", (void *)da, (void *)db, + (void *)dc, (void *)dd); + + CHECK(cudaMemcpy(da, ha, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(db, hb, bytes, cudaMemcpyHostToDevice)); + + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S1 vadd readback bad=%d/%d\n", b, n); + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S1 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", h_out[0], + h_out[1]); + } + + if (!strcmp(mode, "V")) { + vadd<<<grid, 256>>>(da, db, dc, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0; + + for (i = 0; i < n; i++) + if (hc[i] != ha[i] + hb[i]) + b++; + printf("S2 vadd-again readback bad=%d/%d\n", b, n); + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "I")) { + int stage; + + for (stage = 0; stage < 2; stage++) { + float mult = stage == 0 ? 2.0f : 4.0f; + + scale<<<grid, 256>>>(dc, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hc, dc, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hc[i] != (ha[i] + hb[i]) * mult) + b++; + printf("S%d inplace-scale readback bad=%d/%d (mult=%g)\n", + 2 + stage, b, n, mult); + if (stage == 0) + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hc[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * mult) + pb++; + if (pb) + printf("S%d PAGE %d bad=%d/256 first=%g want=%g\n", + 2 + stage, page, + pb, hc[page * 256], + (ha[page * 256] + + hb[page * 256]) * mult); + } + } + if (stage == 0) { + CHECK(cudaMemset(d_out, 0, + 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dc, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, + 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST c_eq_ab=%llu c_eq_2ab=%llu\n", + h_out[0], h_out[1]); + } + } + printf("STATUS DONE\n"); + return 0; + } + + if (!strcmp(mode, "O")) { + scale_out<<<grid, 256>>>(dc, dd, 2.0f, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hd, dd, bytes, cudaMemcpyDeviceToHost)); + { + int b = 0, page; + + for (i = 0; i < n; i++) + if (hd[i] != (ha[i] + hb[i]) * 2.0f) + b++; + printf("S2 outofplace-scale readback bad=%d/%d\n", b, n); + for (page = 0; page < 32; page++) { + int pb = 0, k; + + for (k = 0; k < 256; k++) + if (hd[page * 256 + k] != + (ha[page * 256 + k] + + hb[page * 256 + k]) * 2.0f) + pb++; + if (pb) + printf("S2 PAGE %d bad=%d/256\n", page, + pb); + } + } + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + vdigest<<<grid, 256>>>(da, db, dd, n, d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("S2 GPU-DIGEST d_eq_ab=%llu d_eq_2ab=%llu\n", h_out[0], + h_out[1]); + printf("STATUS DONE\n"); + return 0; + } + return 1; +} diff --git a/tools/cuda-ab/soak.sh b/tools/cuda-ab/soak.sh index c150692..4f1430f 100755 --- a/tools/cuda-ab/soak.sh +++ b/tools/cuda-ab/soak.sh @@ -192,7 +192,7 @@ log "SOAK START mode=$MODE seconds=$SECS guest_dir=$GUEST_DIR timeout=${TMO}s lo # cannot silently shrink the battery. found="" for i in 1 2 3; do - found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done")" || found="" + found="$(gssh "for b in $CLIENTS; do [ -x $GUEST_DIR/\$b ] && echo \$b; done; true")" || found="" [ -n "$found" ] && break sleep 3 done diff --git a/tools/cuda-ab/streams2_probe.cu b/tools/cuda-ab/streams2_probe.cu new file mode 100644 index 0000000..5bebab4 --- /dev/null +++ b/tools/cuda-ab/streams2_probe.cu @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams2_probe.cu - instrumented streams2 to localize the deterministic + * failure: per-array mismatch counts, first-bad indices with values, and a + * GPU-side verify-kernel digest of the device buffers (to separate "device + * pages never written" from "D2H readback wrong"). + * + * Modes (argv[1]): + * A : exactly streams2's shape (control) + * B : like A but the two D2H copy-backs use synchronous cudaMemcpy + * C : like A but no events cross the streams; both streams are joined + * with cudaStreamSynchronize before any copy-back + * D : single stream (everything on s1) + * + * Build: nvcc -O2 --cudart=static -o streams2_probe streams2_probe.cu + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cstring> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void scale(float *c, float s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] * s; +} + +/* GPU-side digest: counts of matches for c1==a1+b1 and c2==(a2+b2)*2 */ +__global__ void digest(const float *a1, const float *b1, const float *c1, + const float *a2, const float *b2, const float *c2, + int n, unsigned long long *out) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i >= n) + return; + if (c1[i] == a1[i] + b1[i]) + atomicAdd(&out[0], 1ull); + if (c2[i] == (a2[i] + b2[i]) * 2.0f) + atomicAdd(&out[1], 1ull); +} + +static int report(const char *tag, const float *h_c, const float *h_a, + const float *h_b, const float *h_a2, const float *h_b2, + int n, float scale2) +{ + int bad = 0, i, shown = 0; + + for (i = 0; i < n; i++) { + float want = h_a[i] + h_b[i]; + + if (scale2 != 1.0f) + want = (h_a2[i] + h_b2[i]) * scale2; + if (h_c[i] != want) { + bad++; + if (shown < 4) { + printf("FIRSTBAD %s i=%d got=%g want=%g\n", + tag, i, h_c[i], want); + shown++; + } + } + } + printf("BAD %s n=%d bad=%d\n", tag, n, bad); + return bad; +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "A"; + cudaDeviceProp prop; + const int n = 1 << 15; + const float s2 = 2.0f; + size_t bytes = (size_t)n * sizeof(float); + float *h_a1, *h_b1, *h_c1, *h_a2, *h_b2, *h_c2; + float *d_a1, *d_b1, *d_c1, *d_a2, *d_b2, *d_c2; + unsigned long long *d_out, h_out[2]; + cudaStream_t s1, s2s; + cudaEvent_t ev_add1, ev_scale2, t0, t1; + float ms = 0.0f; + int i; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s mode=%s\n", prop.name, mode); + + h_a1 = (float *)malloc(bytes); + h_b1 = (float *)malloc(bytes); + h_c1 = (float *)malloc(bytes); + h_a2 = (float *)malloc(bytes); + h_b2 = (float *)malloc(bytes); + h_c2 = (float *)malloc(bytes); + + for (i = 0; i < n; i++) { + h_a1[i] = (float)(i % 1000) * 0.5f; + h_b1[i] = (float)(i % 997) * 0.25f; + h_a2[i] = (float)(i % 991) * 0.125f; + h_b2[i] = (float)(i % 983) * 0.0625f; + } + memset(h_c1, 0xCD, bytes); + memset(h_c2, 0xCD, bytes); + + CHECK(cudaMalloc(&d_a1, bytes)); + CHECK(cudaMalloc(&d_b1, bytes)); + CHECK(cudaMalloc(&d_c1, bytes)); + CHECK(cudaMalloc(&d_a2, bytes)); + CHECK(cudaMalloc(&d_b2, bytes)); + CHECK(cudaMalloc(&d_c2, bytes)); + CHECK(cudaMalloc(&d_out, 2 * sizeof(unsigned long long))); + printf("DPTRS da1=%p db1=%p dc1=%p da2=%p db2=%p dc2=%p\n", + (void *)d_a1, (void *)d_b1, (void *)d_c1, (void *)d_a2, + (void *)d_b2, (void *)d_c2); + + CHECK(cudaStreamCreate(&s1)); + CHECK(cudaStreamCreate(&s2s)); + CHECK(cudaEventCreate(&ev_add1)); + CHECK(cudaEventCreate(&ev_scale2)); + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + CHECK(cudaEventRecord(t0)); + + if (!strcmp(mode, "D")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s1)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s1>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + scale<<<grid, 256, 0, s1>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + goto verify; + } + + if (!strcmp(mode, "S")) { + /* staged verification: check device content after every step */ + float *hb1 = (float *)malloc(bytes); + float *hb2 = (float *)malloc(bytes); + float *hb3 = (float *)malloc(bytes); + + CHECK(cudaMemcpy(d_a1, h_a1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b1, h_b1, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_a2, h_a2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(d_b2, h_b2, bytes, cudaMemcpyHostToDevice)); + CHECK(cudaMemcpy(hb1, d_a1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a1[i]) b++; + printf("STAGE-IN da1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b1[i]) b++; + printf("STAGE-IN db1 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_a2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_a2[i]) b++; + printf("STAGE-IN da2 bad=%d\n", b); } + CHECK(cudaMemcpy(hb1, d_b2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0; for (i = 0; i < n; i++) if (hb1[i] != h_b2[i]) b++; + printf("STAGE-IN db2 bad=%d\n", b); } + + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; + if (first < 0) { first = i; + printf("S1-FIRSTBAD i=%d got=%g want=%g a2b2=%g\n", + i, hb1[i], h_a1[i]+h_b1[i], h_a2[i]+h_b2[i]); } } + printf("STAGE vadd1->c1 readback bad=%d first=%d\n", b, first); } + + CHECK(cudaDeviceSynchronize()); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb2, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb2[i] != h_a2[i] + h_b2[i]) { b++; if (first < 0) first = i; } + printf("STAGE vadd2->c2 readback bad=%d first=%d\n", b, first); } + + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(hb3, d_c2, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb3[i] != (h_a2[i] + h_b2[i]) * 2.0f) { b++; if (first < 0) first = i; } + printf("STAGE scale->c2 readback bad=%d first=%d\n", b, first); } + /* second readback of c1 AFTER everything: does it change? */ + CHECK(cudaMemcpy(hb1, d_c1, bytes, cudaMemcpyDeviceToHost)); + { int b = 0, first = -1; for (i = 0; i < n; i++) + if (hb1[i] != h_a1[i] + h_b1[i]) { b++; if (first < 0) first = i; } + printf("STAGE c1-late readback bad=%d first=%d\n", b, first); } + printf("STATUS OK\n"); + return 0; + } + + if (!strcmp(mode, "C")) { + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, + cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, + cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, + cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + goto verify; + } + + /* A and B: original shape */ + CHECK(cudaMemcpyAsync(d_a1, h_a1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_b1, h_b1, bytes, cudaMemcpyHostToDevice, s1)); + CHECK(cudaMemcpyAsync(d_a2, h_a2, bytes, cudaMemcpyHostToDevice, s2s)); + CHECK(cudaMemcpyAsync(d_b2, h_b2, bytes, cudaMemcpyHostToDevice, s2s)); + vadd<<<grid, 256, 0, s1>>>(d_a1, d_b1, d_c1, n); + vadd<<<grid, 256, 0, s2s>>>(d_a2, d_b2, d_c2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_add1, s1)); + CHECK(cudaStreamWaitEvent(s2s, ev_add1, 0)); + scale<<<grid, 256, 0, s2s>>>(d_c2, s2, n); + CHECK(cudaGetLastError()); + + CHECK(cudaEventRecord(ev_scale2, s2s)); + CHECK(cudaStreamWaitEvent(s1, ev_scale2, 0)); + if (!strcmp(mode, "B")) { + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaMemcpy(h_c1, d_c1, bytes, cudaMemcpyDeviceToHost)); + CHECK(cudaMemcpy(h_c2, d_c2, bytes, cudaMemcpyDeviceToHost)); + } else { + CHECK(cudaMemcpyAsync(h_c1, d_c1, bytes, + cudaMemcpyDeviceToHost, s1)); + CHECK(cudaMemcpyAsync(h_c2, d_c2, bytes, + cudaMemcpyDeviceToHost, s2s)); + } + CHECK(cudaEventRecord(t1, s1)); + + CHECK(cudaStreamSynchronize(s1)); + CHECK(cudaStreamSynchronize(s2s)); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + +verify: + { + int bad1, bad2; + + bad1 = report("C1", h_c1, h_a1, h_b1, h_a2, h_b2, n, 1.0f); + bad2 = report("C2", h_c2, h_a2, h_b2, h_a2, h_b2, n, s2); + printf("STREAMS n=%d bad=%d\n", n, bad1 + bad2); + printf("TIMING ms=%.3f\n", ms); + + /* GPU-side digest of the device buffers */ + CHECK(cudaMemset(d_out, 0, 2 * sizeof(unsigned long long))); + digest<<<grid, 256>>>(d_a1, d_b1, d_c1, d_a2, d_b2, d_c2, n, + d_out); + CHECK(cudaDeviceSynchronize()); + CHECK(cudaMemcpy(h_out, d_out, 2 * sizeof(unsigned long long), + cudaMemcpyDeviceToHost)); + printf("GPU-DIGEST c1_ok=%llu c2_ok=%llu\n", h_out[0], + h_out[1]); + printf("STATUS %s\n", (bad1 | bad2) ? "FAIL" : "OK"); + } + return 0; +} diff --git a/tools/cuda-ab/streams_events_probe.cu b/tools/cuda-ab/streams_events_probe.cu new file mode 100644 index 0000000..94aef69 --- /dev/null +++ b/tools/cuda-ab/streams_events_probe.cu @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * streams_events_probe.cu - instrumented victory_streams_events. + * + * Same 3-stream event chain (c0 = a0+b0; c1 = (a1+b1)+c0; c2 = (a2+b2)+c1) + * but with per-buffer diagnostics: device pointers, per-buffer bad counts, + * and the first 8 expected/actual values of every output so a stale-input + * vs stale-readback failure is distinguishable in one run. + * + * Output lines: + * DEVICE ... + * DPTRS da0..dc2 (device), ha0..hc2 (host) + * BUF s=<n> bad=<n> + * FIRST s=<n> exp=<v0..v7> got=<v0..v7> + * STREAMS3 n=<count> bad=<mismatches> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define CHECK(x) \ + do { \ + cudaError_t e_ = (x); \ + if (e_ != cudaSuccess) { \ + printf("CUDA_ERROR %s at line %d\n", \ + cudaGetErrorString(e_), __LINE__); \ + printf("STATUS FAIL\n"); \ + return 1; \ + } \ + } while (0) + +#define NSTREAMS 3 + +__global__ void vadd(const float *a, const float *b, float *c, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = a[i] + b[i]; +} + +__global__ void vadd_into(float *c, const float *s, int n) +{ + int i = blockIdx.x * blockDim.x + threadIdx.x; + + if (i < n) + c[i] = c[i] + s[i]; +} + +int main(void) +{ + cudaDeviceProp prop; + const int n = 1 << 16; + size_t bytes = (size_t)n * sizeof(float); + float *h_a[NSTREAMS], *h_b[NSTREAMS], *h_c[NSTREAMS]; + float *d_a[NSTREAMS], *d_b[NSTREAMS], *d_c[NSTREAMS]; + cudaStream_t st[NSTREAMS]; + cudaEvent_t ev[NSTREAMS], t0, t1; + float ms = 0.0f; + int bad = 0, i, s; + const int grid = (n + 255) / 256; + + setvbuf(stdout, NULL, _IONBF, 0); + + CHECK(cudaGetDeviceProperties(&prop, 0)); + printf("DEVICE %s cc=%d.%d sm=%d mem=%llu\n", prop.name, + prop.major, prop.minor, prop.multiProcessorCount, + (unsigned long long)prop.totalGlobalMem); + + for (s = 0; s < NSTREAMS; s++) { + h_a[s] = (float *)malloc(bytes); + h_b[s] = (float *)malloc(bytes); + h_c[s] = (float *)malloc(bytes); + if (!h_a[s] || !h_b[s] || !h_c[s]) + return 1; + for (i = 0; i < n; i++) { + h_a[s][i] = (float)((i + s) % 1000) * 0.5f; + h_b[s][i] = (float)((i + 2 * s) % 997) * 0.25f; + } + CHECK(cudaMalloc(&d_a[s], bytes)); + CHECK(cudaMalloc(&d_b[s], bytes)); + CHECK(cudaMalloc(&d_c[s], bytes)); + CHECK(cudaStreamCreate(&st[s])); + CHECK(cudaEventCreate(&ev[s])); + } + CHECK(cudaEventCreate(&t0)); + CHECK(cudaEventCreate(&t1)); + + printf("DPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" da%d=%p db%d=%p dc%d=%p", s, (void *)d_a[s], s, + (void *)d_b[s], s, (void *)d_c[s]); + printf("\n"); + printf("HPTRS"); + for (s = 0; s < NSTREAMS; s++) + printf(" ha%d=%p hb%d=%p hc%d=%p", s, (void *)h_a[s], s, + (void *)h_b[s], s, (void *)h_c[s]); + printf("\n"); + + CHECK(cudaEventRecord(t0)); + + for (s = 0; s < NSTREAMS; s++) { + CHECK(cudaMemcpyAsync(d_a[s], h_a[s], bytes, + cudaMemcpyHostToDevice, st[s])); + CHECK(cudaMemcpyAsync(d_b[s], h_b[s], bytes, + cudaMemcpyHostToDevice, st[s])); + vadd<<<grid, 256, 0, st[s]>>>(d_a[s], d_b[s], d_c[s], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[s], st[s])); + } + + CHECK(cudaStreamWaitEvent(st[1], ev[0], 0)); + vadd_into<<<grid, 256, 0, st[1]>>>(d_c[1], d_c[0], n); + CHECK(cudaGetLastError()); + CHECK(cudaEventRecord(ev[1], st[1])); + + CHECK(cudaStreamWaitEvent(st[2], ev[1], 0)); + vadd_into<<<grid, 256, 0, st[2]>>>(d_c[2], d_c[1], n); + CHECK(cudaGetLastError()); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaMemcpyAsync(h_c[s], d_c[s], bytes, + cudaMemcpyDeviceToHost, st[s])); + CHECK(cudaEventRecord(t1, st[2])); + + for (s = 0; s < NSTREAMS; s++) + CHECK(cudaStreamSynchronize(st[s])); + CHECK(cudaEventElapsedTime(&ms, t0, t1)); + + for (s = 0; s < NSTREAMS; s++) { + int b = 0; + float exp0[8], got0[8]; + + for (i = 0; i < n; i++) { + float c0 = h_a[0][i] + h_b[0][i]; + float c1 = h_a[1][i] + h_b[1][i] + c0; + float c2 = h_a[2][i] + h_b[2][i] + c1; + float want = (s == 0) ? c0 : (s == 1) ? c1 : c2; + + if (i < 8) { + exp0[i] = want; + got0[i] = h_c[s][i]; + } + if (h_c[s][i] != want) + b++; + } + bad += b; + printf("BUF s=%d bad=%d\n", s, b); + printf("FIRST s=%d exp=", s); + for (i = 0; i < 8; i++) + printf("%.4f ", exp0[i]); + printf(" got="); + for (i = 0; i < 8; i++) + printf("%.4f ", got0[i]); + printf("\n"); + } + + printf("STREAMS3 n=%d bad=%d\n", n, bad); + printf("TIMING ms=%.3f\n", ms); + + if (bad != 0) { + printf("STATUS FAIL\n"); + return 1; + } + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/cuda-ab/tgemm_probe.cu b/tools/cuda-ab/tgemm_probe.cu new file mode 100644 index 0000000..f0a1084 --- /dev/null +++ b/tools/cuda-ab/tgemm_probe.cu @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * tgemm_probe.cu - minimal isolation for the transformer K/V divergence. + * + * Reproduces exactly the transformer's Q/K/V GEMM shape (X[64x256] x + * W[256x256], tile 16) for three weight matrices uploaded back-to-back + * (wq, wk, wv), then verifies each against a CPU reference and prints + * per-GEMM bad counts plus a bad-position histogram (first bad index per + * 16-row band and per 16-col band) so a copy-side hole is distinguishable + * from a kernel-side error. + * + * Output: + * DEVICE ... + * GEMM name=<q|k|v> bad=<n> first=(row,col) got=.. ref=.. + * BADPOS name=.. rows=<r0,r1,..> cols=<c0,c1,..> + * STATUS OK|FAIL + */ + +#include <cstdio> +#include <cstdlib> +#include <cstdint> +#include <cuda_runtime.h> + +#define SEQ 64 +#define DH 256 +#define TILE 16 + +__global__ void gemm_nn(const float *A, const float *B, float *C, + int M, int N, int K) +{ + __shared__ float As[TILE][TILE]; + __shared__ float Bs[TILE][TILE]; + int row = blockIdx.y * TILE + threadIdx.y; + int col = blockIdx.x * TILE + threadIdx.x; + float acc = 0.0f; + + for (int t = 0; t < K / TILE; t++) { + As[threadIdx.y][threadIdx.x] = A[row * K + t * TILE + threadIdx.x]; + Bs[threadIdx.y][threadIdx.x] = + B[(t * TILE + threadIdx.y) * N + col]; + __syncthreads(); + for (int k = 0; k < TILE; k++) + acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; + __syncthreads(); + } + C[row * N + col] = acc; +} + +static float wval(int i, int seed) +{ + return (float)(((i * 13 + seed * 101 + 7) % 5) - 2) * 0.125f; +} + +static float xval(int i, int j) +{ + return (float)(((i * 7 + j * 3 + 1) % 9) - 4) * 0.25f; +} + +int main(void) +{ + cudaDeviceProp prop; + float *h_x, *h_w, *h_c, *h_ref; + float *d_x, *d_w[3], *d_c[3]; + const int seqdh = SEQ * DH, dh2 = DH * DH; + int g, i, j, k; + + setvbuf(stdout, NULL, _IONBF, 0); + if (cudaGetDeviceProperties(&prop, 0) != cudaSuccess) { + printf("STATUS FAIL\n"); + return 1; + } + printf("DEVICE %s cc=%d.%d\n", prop.name, prop.major, prop.minor); + + h_x = (float *)malloc(seqdh * 4); + h_w = (float *)malloc(dh2 * 4); + h_c = (float *)malloc(seqdh * 4); + h_ref = (float *)malloc(seqdh * 4); + + if (cudaMalloc(&d_x, seqdh * 4) != cudaSuccess) + return 1; + for (g = 0; g < 3; g++) { + if (cudaMalloc(&d_w[g], dh2 * 4) != cudaSuccess) + return 1; + if (cudaMalloc(&d_c[g], seqdh * 4) != cudaSuccess) + return 1; + } + + for (i = 0; i < seqdh; i++) + h_x[i] = xval(i / DH, i % DH); + + /* Upload X once (the transformer's shared input). */ + if (cudaMemcpy(d_x, h_x, seqdh * 4, cudaMemcpyHostToDevice) != + cudaSuccess) + return 1; + /* Verify the X upload itself (first row + full buffer). */ + { + float *h_xb = (float *)malloc(seqdh * 4); + int xbad = 0, xr0 = 0; + + if (!h_xb) + return 1; + if (cudaMemcpy(h_xb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (h_xb[i] != h_x[i]) { + xbad++; + if (i < DH) + xr0++; + } + printf("XVERIFY bad=%d row0_bad=%d\n", xbad, xr0); + free(h_xb); + } + + for (g = 0; g < 3; g++) { + int bad = 0, first_r = -1, first_c = -1; + float got = 0, ref = 0; + int rows[64] = {0}, cols[64] = {0}; + int nr = 0, nc = 0; + int pass; + + for (i = 0; i < dh2; i++) + h_w[i] = wval(i, g + 1); + if (cudaMemcpy(d_w[g], h_w, dh2 * 4, + cudaMemcpyHostToDevice) != cudaSuccess) + return 1; + + dim3 blk(TILE, TILE); + dim3 grd(DH / TILE, SEQ / TILE); + + for (pass = 0; pass < 3; pass++) { + float *hc = h_c, *hxb = h_ref; + int xbad = 0; + + gemm_nn<<<grd, blk>>>(d_x, d_w[g], d_c[g], SEQ, DH, DH); + if (cudaDeviceSynchronize() != cudaSuccess) { + printf("SYNC FAIL g=%d\n", g); + return 1; + } + if (cudaMemcpy(hc, d_c[g], seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + /* re-verify X after the launch */ + if (cudaMemcpy(hxb, d_x, seqdh * 4, + cudaMemcpyDeviceToHost) != cudaSuccess) + return 1; + for (i = 0; i < seqdh; i++) + if (hxb[i] != h_x[i]) { + printf("XDIFF pass=%d i=%d got=%.6f want=%.6f\n", + pass, i, hxb[i], h_x[i]); + xbad++; + } + bad = 0; + nr = 0; + nc = 0; + for (i = 0; i < SEQ; i++) + for (j = 0; j < DH; j++) { + double acc = 0.0; + float r, a, diff; + + for (k = 0; k < DH; k++) + acc += (double)h_x[i * DH + k] * + (double)h_w[k * DH + j]; + r = (float)acc; + a = hc[i * DH + j]; + diff = a - r; + if (diff < 0) + diff = -diff; + if (diff > 1e-4f) { + int rr = i, cc = j; + + if (bad == 0) { + first_r = rr; + first_c = cc; + got = a; + ref = r; + } + bad++; + if (nr < 64 && (bad == 1 || + rr != rows[nr - 1])) + rows[nr++] = rr; + if (nc < 64 && (bad == 1 || + cc != cols[nc - 1])) + cols[nc++] = cc; + } + } + printf("PASS name=%c pass=%d bad=%d xbad=%d first=(%d,%d) got=%.6f ref=%.6f\n", + "qkv"[g], pass, bad, xbad, first_r, first_c, + got, ref); + if (bad) { + printf("ROW0 name=%c got=", "qkv"[g]); + for (i = 0; i < 8; i++) + printf("%.6f ", hc[i]); + printf(" ref="); + for (i = 0; i < 8; i++) { + double acc = 0.0; + + for (k = 0; k < DH; k++) + acc += (double)h_x[k] * + (double)h_w[k * DH + i]; + printf("%.6f ", (float)acc); + } + printf("\n"); + } + } + (void)got; + (void)ref; + (void)first_r; + (void)first_c; + (void)nr; + (void)nc; + } + + printf("STATUS OK\n"); + return 0; +} diff --git a/tools/debug/pdb_scan.c b/tools/debug/pdb_scan.c new file mode 100644 index 0000000..da7227e --- /dev/null +++ b/tools/debug/pdb_scan.c @@ -0,0 +1,234 @@ +/* + * pdb_scan.c - read-only reverse scan of a QEMU pc.ram backing file for the + * 4-level page-table chain that maps a given VA to a given PA. + * + * Build: + * cc -O2 -Wall -Wextra -o pdb_scan pdb_scan.c + * + * Usage: + * sudo ./pdb_scan /dev/shm/vgpu-guest-ram 0xafef8000 0x4100000 + * + * Arguments: <ramfile> <target_pa> [va] + * - target_pa : physical address the leaf PT entry must point at (page aligned) + * - va : optional VA used to derive the four table indices + * (default 0x4100000: PML4[0], PDPT[0], PD[8], PT[0x100]) + * + * The scan is strictly read-only: the file is mmap()ed PROT_READ/MAP_PRIVATE + * and no writes are issued. It builds two reverse indexes: + * entry[0x000] -> pages (used for PML4[0] and PDPT[0]) + * entry[0x040] -> pages (used for PD[8]) + * then joins them against pages whose entry[0x800] points at target_pa. + */ +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <errno.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/mman.h> +#include <sys/stat.h> + +#define PAGE_SHIFT 12ULL +#define PAGE_SIZE (1ULL << PAGE_SHIFT) +#define PA_MASK 0x000FFFFFFFFFF000ULL + +static uint64_t target_pa; +static uint64_t va; +static uint64_t n_pages; +static const uint8_t *ram; + +static unsigned idx_pml4, idx_pdpt, idx_pd, idx_pt; +static const char *idx_names[4] = {"PML4", "PDPT", "PD", "PT"}; + +static inline uint64_t rd64(uint64_t pa) +{ + uint64_t v; + memcpy(&v, ram + pa, sizeof(v)); + return v; +} + +static inline uint64_t entry_mask(uint64_t e) { return e & PA_MASK; } + +static const char *bits(uint64_t e) +{ + static char b[128]; + snprintf(b, sizeof(b), "P=%d RW=%d US=%d PWT=%d PCD=%d A=%d D=%d PS=%d", + (int)(e & 1), (int)((e >> 1) & 1), (int)((e >> 2) & 1), + (int)((e >> 3) & 1), (int)((e >> 4) & 1), (int)((e >> 5) & 1), + (int)((e >> 6) & 1), (int)((e >> 7) & 1)); + return b; +} + +/* Generic 4-level walk; returns final PA (masked) or UINT64_MAX on break. */ +static uint64_t walk4(uint64_t pml4_pa, uint64_t *raw_out) +{ + uint64_t e[4], pa = pml4_pa; + unsigned idx[4] = {idx_pml4, idx_pdpt, idx_pd, idx_pt}; + int l; + for (l = 0; l < 4; l++) { + if (pa + PAGE_SIZE > (n_pages << PAGE_SHIFT)) + return UINT64_MAX; + e[l] = rd64(pa + (uint64_t)idx[l] * 8); + if (raw_out) raw_out[l] = e[l]; + if (!(e[l] & 1)) + return UINT64_MAX; + if (l == 2 && (e[l] & 0x80)) { /* PS=1: 2 MiB page at PD level */ + uint64_t base = entry_mask(e[l]) & ~((1ULL << 21) - 1); + return base | (va & ((1ULL << 21) - 1)); + } + pa = entry_mask(e[l]); + } + return pa; +} + +int main(int argc, char **argv) +{ + const char *path; + int fd; + struct stat st; + uint64_t i; + int32_t *head0, *next0, *head40, *next40, *pt_list; + uint64_t n_pt = 0; + int have_va = 0; + + if (argc < 3) { + fprintf(stderr, "usage: %s <ramfile> <target_pa> [va]\n", argv[0]); + return 2; + } + path = argv[1]; + target_pa = strtoull(argv[2], NULL, 0) & PA_MASK; + if (argc >= 4) { + va = strtoull(argv[3], NULL, 0); + have_va = 1; + } else { + va = 0x4100000ULL; + } + /* indices: canonical 4-level split */ + idx_pml4 = (unsigned)((va >> 39) & 0x1ff); + idx_pdpt = (unsigned)((va >> 30) & 0x1ff); + idx_pd = (unsigned)((va >> 21) & 0x1ff); + idx_pt = (unsigned)((va >> 12) & 0x1ff); + + fd = open(path, O_RDONLY); + if (fd < 0) { perror("open"); return 1; } + if (fstat(fd, &st) < 0) { perror("fstat"); return 1; } + if (st.st_size == 0) { fprintf(stderr, "empty file\n"); return 1; } + n_pages = (uint64_t)st.st_size >> PAGE_SHIFT; + + ram = mmap(NULL, (size_t)st.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (ram == MAP_FAILED) { perror("mmap"); return 1; } + madvise((void *)ram, (size_t)st.st_size, MADV_SEQUENTIAL); + + fprintf(stderr, "ramfile=%s size=%llu bytes pages=%llu\n", + path, (unsigned long long)st.st_size, (unsigned long long)n_pages); + fprintf(stderr, "target_pa=0x%llx va=0x%llx idx: PML4[%u] PDPT[%u] PD[%u] PT[%u]\n", + (unsigned long long)target_pa, (unsigned long long)va, + idx_pml4, idx_pdpt, idx_pd, idx_pt); + fprintf(stderr, "PT entry offset=0x%x PD entry offset=0x%x\n", + idx_pt * 8, idx_pd * 8); + + head0 = malloc(n_pages * sizeof(int32_t)); + next0 = malloc(n_pages * sizeof(int32_t)); + head40 = malloc(n_pages * sizeof(int32_t)); + next40 = malloc(n_pages * sizeof(int32_t)); + pt_list = malloc(n_pages * sizeof(int32_t)); + if (!head0 || !next0 || !head40 || !next40 || !pt_list) { + fprintf(stderr, "out of memory\n"); return 1; + } + for (i = 0; i < n_pages; i++) { head0[i] = -1; head40[i] = -1; } + + for (i = 0; i < n_pages; i++) { + const uint8_t *p = ram + (i << PAGE_SHIFT); + uint64_t e0, e40, ept, a; + memcpy(&e0, p + 0x000, 8); + memcpy(&e40, p + 0x040, 8); + memcpy(&ept, p + (uint64_t)idx_pt * 8, 8); + + if (entry_mask(ept) == target_pa) + pt_list[n_pt++] = (int32_t)i; + + a = entry_mask(e0); + if (a < (n_pages << PAGE_SHIFT) && (a & (PAGE_SIZE - 1)) == 0) { + uint64_t j = a >> PAGE_SHIFT; + next0[i] = head0[j]; + head0[j] = (int32_t)i; + } + a = entry_mask(e40); + if (a < (n_pages << PAGE_SHIFT) && (a & (PAGE_SIZE - 1)) == 0) { + uint64_t j = a >> PAGE_SHIFT; + next40[i] = head40[j]; + head40[j] = (int32_t)i; + } + } + madvise((void *)ram, (size_t)st.st_size, MADV_RANDOM); + + fprintf(stderr, "pages whose entry[0x%x] == target: %llu\n", + idx_pt * 8, (unsigned long long)n_pt); + + uint64_t n_chain = 0; + for (uint64_t t = 0; t < n_pt; t++) { + int32_t pt = pt_list[t]; + int32_t pd; + for (pd = head40[pt]; pd >= 0; pd = next40[pd]) { + int32_t pdpt; + for (pdpt = head0[pd]; pdpt >= 0; pdpt = next0[pdpt]) { + int32_t pml4; + for (pml4 = head0[pdpt]; pml4 >= 0; pml4 = next0[pml4]) { + uint64_t pa_pml4 = (uint64_t)pml4 << PAGE_SHIFT; + uint64_t pa_pdpt = (uint64_t)pdpt << PAGE_SHIFT; + uint64_t pa_pd = (uint64_t)pd << PAGE_SHIFT; + uint64_t pa_pt = (uint64_t)pt << PAGE_SHIFT; + uint64_t raw[4] = {0}, fin; + fin = walk4(pa_pml4, raw); + n_chain++; + printf("\n=== candidate #%llu ===\n", (unsigned long long)n_chain); + printf("pdb (PML4 PA) = 0x%llx\n", (unsigned long long)pa_pml4); + printf("PML4[%u] @0x%llx+0x%03x = 0x%016llx -> PDPT 0x%llx [%s]\n", + idx_pml4, (unsigned long long)pa_pml4, idx_pml4 * 8, + (unsigned long long)raw[0], (unsigned long long)pa_pdpt, + bits(raw[0])); + printf("PDPT[%u] @0x%llx+0x%03x = 0x%016llx -> PD 0x%llx [%s]\n", + idx_pdpt, (unsigned long long)pa_pdpt, idx_pdpt * 8, + (unsigned long long)raw[1], (unsigned long long)pa_pd, + bits(raw[1])); + printf("PD[%u] @0x%llx+0x%03x = 0x%016llx -> PT 0x%llx [%s]\n", + idx_pd, (unsigned long long)pa_pd, idx_pd * 8, + (unsigned long long)raw[2], (unsigned long long)pa_pt, + bits(raw[2])); + printf("PT[%u] @0x%llx+0x%03x = 0x%016llx -> PA 0x%llx [%s]\n", + idx_pt, (unsigned long long)pa_pt, idx_pt * 8, + (unsigned long long)raw[3], + (unsigned long long)entry_mask(raw[3]), bits(raw[3])); + printf("walk_result = 0x%llx %s\n", + (unsigned long long)fin, + fin == target_pa ? "MATCH" : "MISMATCH"); + } + } + } + } + + /* Reference: the caller-supplied known pdb, if present in the file. */ + if (have_va) { + uint64_t known = 0x11092a000ULL; + if (known < (n_pages << PAGE_SHIFT)) { + uint64_t raw[4] = {0}, fin = walk4(known, raw); + printf("\n=== reference walk from known pdb 0x%llx ===\n", + (unsigned long long)known); + printf("PML4[%u]=0x%016llx PDPT[%u]=0x%016llx PD[%u]=0x%016llx PT[%u]=0x%016llx\n", + idx_pml4, (unsigned long long)raw[0], + idx_pdpt, (unsigned long long)raw[1], + idx_pd, (unsigned long long)raw[2], + idx_pt, (unsigned long long)raw[3]); + printf("walk_result = 0x%llx\n", (unsigned long long)fin); + } + } + + printf("\ntotal candidate chains: %llu\n", (unsigned long long)n_chain); + + munmap((void *)ram, (size_t)st.st_size); + close(fd); + free(head0); free(next0); free(head40); free(next40); free(pt_list); + return 0; +} diff --git a/tools/debug/pdb_scan_gmmu.c b/tools/debug/pdb_scan_gmmu.c new file mode 100644 index 0000000..f0221f6 --- /dev/null +++ b/tools/debug/pdb_scan_gmmu.c @@ -0,0 +1,330 @@ +/* + * pdb_scan_gmmu.c - read-only scanner for the GP10X/VER2 GMMU page-table + * chain that maps a GPU VA to a target PA inside a live QEMU pc.ram file. + * + * IMPORTANT: the GPU page tables are NOT x86-64. The format (gp10x family, + * Pascal..Ampere) is: + * + * PD3[48:47] -> PD2[46:38] -> PD1[37:29] -> PD0[28:21] + * PD0 is a 16-byte dual entry: big half at +0, small at +8 + * PT[20:12] (4 KiB leaf) + * + * Entry address field: (entry >> 8) << 12 (aperture-dependent mask). + * Directory entries: ap bits 2:1 = 0 invalid, 1 vidmem, 2 sys-coherent, + * 3 sys-noncoherent. Leaf PTEs: 0 vidmem, 1 peer (invalid here), 2/3 sys. + * A directory entry has bit0 = IS_PTE = 0; a leaf has bit0 = VALID = 1. + * + * Also important: for this QEMU q35 guest the RAMBlock file offset is NOT + * equal to GPA above 4 GiB. Low RAM occupies GPA [0, low_ram) at file + * [0, low_ram); high RAM starts at GPA 4 GiB and maps to file offset + * low_ram (e820: low usable ends 0x7ffdf000, so low_ram = 0x80000000). + * Hence: file_off = GPA if GPA < low_ram, else GPA - 4G + low_ram. + * + * Build: + * cc -O2 -Wall -Wextra -o pdb_scan_gmmu pdb_scan_gmmu.c + * Run: + * sudo ./pdb_scan_gmmu /dev/shm/vgpu-guest-ram 0xafef8000 0x4100000 + * sudo ./pdb_scan_gmmu /dev/shm/vgpu-guest-ram 0xa6c000 0x12006c000 + * + * The tool is strictly read-only (PROT_READ/MAP_PRIVATE, no writes). + */ +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/mman.h> +#include <sys/stat.h> + +#define SYS_MASK 0x00003FFFFFFFFFFFULL +#define VID_MASK 0x0000000001FFFFFFULL +#define LOW_RAM 0x80000000ULL /* below-4G RAM block (override arg 4) */ +#define FOUR_G 0x100000000ULL + +static uint64_t va, target, low_ram = LOW_RAM; +static uint64_t fs, np; +static const uint8_t *ram; +static unsigned i_pd3, i_pd2, i_pd1, i_pd0, i_pt; + +static inline uint64_t gpa_to_off(uint64_t gpa) +{ + if (gpa < low_ram) return gpa; + if (gpa >= FOUR_G && gpa - FOUR_G < fs - low_ram) + return gpa - FOUR_G + low_ram; + return UINT64_MAX; +} +static inline uint64_t off_to_gpa(uint64_t off) +{ + if (off < low_ram) return off; + return off + (FOUR_G - low_ram); +} +static inline uint64_t rd64(uint64_t gpa) +{ + uint64_t o = gpa_to_off(gpa), v; + if (o == UINT64_MAX || o + 8 > fs) return UINT64_MAX; + memcpy(&v, ram + o, 8); + return v; +} +static inline unsigned apof(uint64_t e) { return (unsigned)((e >> 1) & 3); } +static inline uint64_t ent_addr(uint64_t e, unsigned ap) +{ + return ((e >> 8) & (ap == 1 ? VID_MASK : SYS_MASK)) << 12; +} +/* Directory entry -> child table. ap 0 invalid; ap 1 vid; ap 2/3 sys. */ +static int dir_child(uint64_t e, uint64_t *child) +{ + unsigned ap; + if (e == UINT64_MAX || (e & 1)) return -1; + ap = apof(e); + if (ap == 0) return -1; + *child = ent_addr(e, ap); + return 0; +} +/* Leaf -> backing PA. bit0 valid; ap 1 (peer) is not backed here. */ +static int leaf_pa(uint64_t e, uint64_t *pa) +{ + unsigned ap; + if (e == UINT64_MAX || !(e & 1)) return -1; + ap = apof(e); + if (ap == 1) return -1; + *pa = ent_addr(e, ap); + return 0; +} +static const char *apname(unsigned a) +{ + static const char *n[] = {"invalid", "vid", "sys-coh", "sys-nc"}; + return a < 4 ? n[a] : "?"; +} +static void dump_entry(const char *lvl, uint64_t table, unsigned idx, uint64_t e) +{ + printf(" %-4s table=0x%-10llx idx=0x%-3x e=0x%016llx V=%d ap=%u(%s)\n", + lvl, (unsigned long long)table, idx, (unsigned long long)e, + (int)(e & 1), apof(e), apname(apof(e))); +} + +/* Full walk. raw[0..4]=PD3,PD2,PD1,PD0-small,PT; big64/64k handled. */ +static uint64_t walk(uint64_t pdb, uint64_t *raw, int *kind) +{ + uint64_t table = pdb, e, child; + unsigned shifts[3] = {47, 38, 29}, idx; + int l; + + *kind = 0; /* 0=4K, 1=2M big, 2=64K */ + for (l = 0; l < 3; l++) { + idx = (unsigned)((va >> shifts[l]) & 0x1ff); + if (l == 0) idx &= 3; /* 4 root entries */ + e = rd64(table + 8 * idx); + raw[l] = e; + if (dir_child(e, &child)) return UINT64_MAX; + table = child; + if (gpa_to_off(table) == UINT64_MAX) return UINT64_MAX; + } + idx = (unsigned)((va >> 21) & 0xff); /* PD0 is 8 bits */ + { + uint64_t big = rd64(table + 16 * idx); + uint64_t small = rd64(table + 16 * idx + 8); + uint64_t pa, t64; + raw[3] = small; + if (big == UINT64_MAX || small == UINT64_MAX) return UINT64_MAX; + /* big half: 2 MiB leaf (bit0=1) or 64 KiB table pointer (bit0=0) */ + if ((big & 1) && !leaf_pa(big, &pa)) { + *kind = 1; + return pa | (va & ((1ULL << 21) - 1)); + } + if (!dir_child(big, &t64) && gpa_to_off(t64) != UINT64_MAX) { + uint64_t e64 = rd64(t64 + 8 * ((va >> 16) & 0x1f)); + if (!leaf_pa(e64, &pa)) { *kind = 2; return pa | (va & 0xffff); } + } + /* small half: 4 KiB page table */ + if (dir_child(small, &child)) return UINT64_MAX; + table = child; + if (gpa_to_off(table) == UINT64_MAX) return UINT64_MAX; + } + idx = (unsigned)((va >> 12) & 0x1ff); + { + uint64_t pa; + e = rd64(table + 8 * idx); + raw[4] = e; + if (leaf_pa(e, &pa)) return UINT64_MAX; + return pa | (va & 0xfff); + } +} + +/* Reverse chain: PT -> PD0 -> PD1 -> PD2 -> PD3, keyed by GPA page. */ +static void reverse_scan(void) +{ + uint64_t max_gpa_page = (off_to_gpa(fs) + 0xfff) >> 12; + int32_t *head0 = malloc((max_gpa_page + 1) * 4); + int32_t *next0 = malloc(np * 4); + int32_t *headpd = malloc((max_gpa_page + 1) * 4); + int32_t *nextpd = malloc(np * 4); + int32_t *pt_list = malloc(np * 4); + uint64_t i, n_pt = 0, n_chain = 0; + + if (!head0 || !next0 || !headpd || !nextpd || !pt_list) { + fprintf(stderr, "reverse scan: out of memory\n"); + return; + } + for (i = 0; i <= max_gpa_page; i++) { head0[i] = -1; headpd[i] = -1; } + + for (i = 0; i < np; i++) { + const uint8_t *p = ram + (i << 12); + uint64_t e0, epd, ept, child, pa; + memcpy(&e0, p + 0, 8); + memcpy(&epd, p + 16 * i_pd0 + 8, 8); + memcpy(&ept, p + 8 * i_pt, 8); + + if (!leaf_pa(ept, &pa) && pa == target) + pt_list[n_pt++] = (int32_t)i; + + if (!dir_child(e0, &child) && (child >> 12) <= max_gpa_page) { + next0[i] = head0[child >> 12]; + head0[child >> 12] = (int32_t)i; + } + if (!dir_child(epd, &child) && (child >> 12) <= max_gpa_page) { + nextpd[i] = headpd[child >> 12]; + headpd[child >> 12] = (int32_t)i; + } + } + printf("\n[reverse scan] PT pages with leaf[0x%x] -> 0x%llx : %llu\n", + i_pt, (unsigned long long)target, (unsigned long long)n_pt); + + for (uint64_t t = 0; t < n_pt; t++) { + int32_t pt = pt_list[t], pd0, pd1, pd2, pd3; + uint64_t pt_gp = off_to_gpa((uint64_t)pt << 12) >> 12; + for (pd0 = headpd[pt_gp]; pd0 >= 0; pd0 = nextpd[pd0]) { + uint64_t pd0_gp = off_to_gpa((uint64_t)pd0 << 12) >> 12; + for (pd1 = head0[pd0_gp]; pd1 >= 0; pd1 = next0[pd1]) { + uint64_t pd1_gp = off_to_gpa((uint64_t)pd1 << 12) >> 12; + for (pd2 = head0[pd1_gp]; pd2 >= 0; pd2 = next0[pd2]) { + uint64_t pd2_gp = off_to_gpa((uint64_t)pd2 << 12) >> 12; + for (pd3 = head0[pd2_gp]; pd3 >= 0; pd3 = next0[pd3]) { + uint64_t raw[5] = {0}, fin; + int kind; + fin = walk(off_to_gpa((uint64_t)pd3 << 12), raw, &kind); + if (fin != target) continue; + n_chain++; + printf("\n=== reverse chain #%llu (pdb GPA 0x%llx) ===\n", + (unsigned long long)n_chain, + (unsigned long long)off_to_gpa((uint64_t)pd3 << 12)); + dump_entry("PD3", off_to_gpa((uint64_t)pd3 << 12), i_pd3, raw[0]); + dump_entry("PD2", ent_addr(raw[0], apof(raw[0])), i_pd2, raw[1]); + dump_entry("PD1", ent_addr(raw[1], apof(raw[1])), i_pd1, raw[2]); + dump_entry("PD0", ent_addr(raw[2], apof(raw[2])), i_pd0, raw[3]); + dump_entry("PT", ent_addr(raw[3], apof(raw[3])), i_pt, raw[4]); + printf(" walk result = 0x%llx (%s)\n", + (unsigned long long)fin, + kind == 1 ? "2M big" : kind == 2 ? "64K" : "4K"); + } + } + } + } + } + printf("[reverse scan] matching chains: %llu\n", (unsigned long long)n_chain); + free(head0); free(next0); free(headpd); free(nextpd); free(pt_list); +} + +typedef struct { uint64_t pa, pdb; int kind; } Hit; +static int hit_cmp(const void *a, const void *b) +{ + const Hit *x = a, *y = b; + return x->pa < y->pa ? -1 : x->pa > y->pa ? 1 : 0; +} + +/* Brute force: treat every 4K page as a candidate pdb, walk the VA, tally. */ +static void brute_force(void) +{ + Hit *h = malloc(np * sizeof(Hit)); + uint64_t i, nh = 0, plausible = 0, matches = 0; + if (!h) { fprintf(stderr, "brute force: oom\n"); return; } + + for (i = 0; i < np; i++) { + uint64_t pdb = off_to_gpa(i << 12), e0, child, fin; + int kind; + uint64_t raw[5] = {0}; + e0 = rd64(pdb); + if (dir_child(e0, &child) || gpa_to_off(child) == UINT64_MAX) continue; + plausible++; + fin = walk(pdb, raw, &kind); + if (fin == UINT64_MAX) continue; + h[nh].pa = fin; h[nh].pdb = pdb; h[nh].kind = kind; nh++; + if (fin == target) { + matches++; + printf("\n=== brute-force MATCH pdb GPA 0x%llx -> 0x%llx (%s) ===\n", + (unsigned long long)pdb, (unsigned long long)fin, + kind == 1 ? "2M big" : kind == 2 ? "64K" : "4K"); + dump_entry("PD3", pdb, i_pd3, raw[0]); + dump_entry("PD2", ent_addr(raw[0], apof(raw[0])), i_pd2, raw[1]); + dump_entry("PD1", ent_addr(raw[1], apof(raw[1])), i_pd1, raw[2]); + dump_entry("PD0", ent_addr(raw[2], apof(raw[2])), i_pd0, raw[3]); + dump_entry("PT", ent_addr(raw[3], apof(raw[3])), i_pt, raw[4]); + } + } + qsort(h, nh, sizeof(Hit), hit_cmp); + printf("\n[brute force] plausible pdbs=%llu valid walks=%llu target matches=%llu\n", + (unsigned long long)plausible, (unsigned long long)nh, + (unsigned long long)matches); + i = 0; + while (i < nh) { + uint64_t j = i; + while (j < nh && h[j].pa == h[i].pa) j++; + printf(" VA 0x%llx -> PA 0x%llx : %llu pdb(s), %s\n", + (unsigned long long)va, (unsigned long long)h[i].pa, + (unsigned long long)(j - i), + h[i].kind == 1 ? "2M big" : h[i].kind == 2 ? "64K" : "4K"); + for (uint64_t k = i; k < j && k < i + 8; k++) + printf(" pdb 0x%llx\n", (unsigned long long)h[k].pdb); + if (j - i > 8) printf(" ...\n"); + i = j; + } + free(h); +} + +int main(int argc, char **argv) +{ + const char *path; + int fd; + struct stat st; + + if (argc < 3) { + fprintf(stderr, + "usage: %s <ramfile> <target_pa> [va] [low_ram_bytes]\n" + " default va=0x4100000 low_ram=0x80000000\n", argv[0]); + return 2; + } + path = argv[1]; + target = strtoull(argv[2], NULL, 0); + va = (argc >= 4) ? strtoull(argv[3], NULL, 0) : 0x4100000ULL; + if (argc >= 5) low_ram = strtoull(argv[4], NULL, 0); + + i_pd3 = (unsigned)((va >> 47) & 3); + i_pd2 = (unsigned)((va >> 38) & 0x1ff); + i_pd1 = (unsigned)((va >> 29) & 0x1ff); + i_pd0 = (unsigned)((va >> 21) & 0xff); + i_pt = (unsigned)((va >> 12) & 0x1ff); + + fd = open(path, O_RDONLY); + if (fd < 0) { perror("open"); return 1; } + if (fstat(fd, &st) < 0) { perror("fstat"); return 1; } + fs = (uint64_t)st.st_size; + np = fs >> 12; + ram = mmap(NULL, (size_t)fs, PROT_READ, MAP_PRIVATE, fd, 0); + if (ram == MAP_FAILED) { perror("mmap"); return 1; } + + printf("file=%s size=%llu pages=%llu\n", path, + (unsigned long long)fs, (unsigned long long)np); + printf("target_pa=0x%llx va=0x%llx low_ram=0x%llx\n", + (unsigned long long)target, (unsigned long long)va, + (unsigned long long)low_ram); + printf("GMMU idx: PD3[%u] PD2[%u] PD1[%u] PD0[%u] PT[%u]\n", + i_pd3, i_pd2, i_pd1, i_pd0, i_pt); + + reverse_scan(); + brute_force(); + + munmap((void *)ram, (size_t)fs); + close(fd); + return 0; +} diff --git a/tools/guest-diag/cycle-r1780.sh b/tools/guest-diag/cycle-r1780.sh new file mode 100755 index 0000000..23cf0d1 --- /dev/null +++ b/tools/guest-diag/cycle-r1780.sh @@ -0,0 +1,142 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2026 Open-IOV contributors +# cycle-r1780.sh - durable vGPU cycle launcher for the r1780 campaign +# (12 GiB file-backed guest RAM, kvmfr/kvmsr ivshmem, V31-class module). +# +# Usage: tools/guest-diag/cycle-r1780.sh [cycle-tag] [module.ko] +# Env: INSOMOD_PARAMS override args file (default params-r1780.txt) +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$HERE/../.." && pwd)" +TAG="${1:-c27}" +MODULE="${2:-$REPO/kernel/vgsp.ko}" +ARGV_FILE="${INSMOD_PARAMS:-$HERE/params-r1780.txt}" +BACKUP="$REPO/backups/evidence-20261002-victory/r1780" +DISK="${GUEST_DISK:-/home/rmcore/.local/libvf.io/kernel/linux.qcow2}" +MDEV_PARENT="0000:01:00.0" +MDEV_TYPE="nvidia-nvidia-vgpu-turing" +RUNDIR="$HERE/run" +QEMU_SCRIPT="$HERE/launch_qemu_r1780.run.sh" +mkdir -p "$RUNDIR" "$BACKUP" + +echo "[cycle $TAG] stop QEMU" +# Flush the guest before the forced teardown: a killed GL/X workload has +# corrupted the ext4 root twice (c38 panic, c43 EIO), costing an offline fsck +# cycle each time. +if [ -S /tmp/qemu-guest-agent.sock ]; then + timeout 8 python3 - <<'PYEOF' >/dev/null 2>&1 || true +import json, socket +s = socket.socket(socket.AF_UNIX) +s.settimeout(6) +s.connect('/tmp/qemu-guest-agent.sock') +for cmd in (["sync"], ["/bin/sh", "-c", "sync; echo 1 > /proc/sys/kernel/sysrq; echo o > /proc/sysrq-trigger"]): + try: + s.sendall(json.dumps({"execute": "guest-exec", "arguments": {"path": cmd[0], "arg": cmd[1:], "capture-output": True}}).encode() + b'\n') + s.recv(65536) + except Exception: + break +s.close() +PYEOF + sleep 2 +fi +sudo pkill -f qemu-system-x86_64 2>/dev/null || true +for i in $(seq 1 20); do pgrep -f qemu-system-x86_64 >/dev/null || break; sleep 0.5; done +sudo pkill -9 -f qemu-system-x86_64 2>/dev/null || true +sleep 1 + +# Always repair the guest root after a forced teardown (cheap when clean, +# prevents the recurring "guest-agent exec EIO / kill init" corruption class). +DISK="/home/rmcore/.local/libvf.io/kernel/linux.qcow2" +if [ -f "$DISK" ]; then + sudo modprobe nbd max_part=8 2>/dev/null || true + sudo qemu-nbd --disconnect /dev/nbd0 >/dev/null 2>&1 || true + if sudo qemu-nbd --connect=/dev/nbd0 "$DISK" -f qcow2 2>/dev/null; then + sleep 1 + sudo e2fsck -fy /dev/nbd0p1 > "$RUNDIR/fsck-$TAG.log" 2>&1 || true + sudo qemu-nbd --disconnect /dev/nbd0 >/dev/null 2>&1 || true + fi +fi + +echo "[cycle $TAG] remove mdevs" +for m in /sys/bus/mdev/devices/*; do + [ -d "$m" ] && echo 1 | sudo tee "$m/remove" >/dev/null || true +done + +echo "[cycle $TAG] unload vgsp" +sudo rmmod vgsp 2>/dev/null || true +for i in $(seq 1 20); do [ -d /sys/module/vgsp ] || break; sleep 0.5; done +if [ -d /sys/module/vgsp ]; then + echo "FATAL: vgsp stuck loaded (see /proc/modules)"; exit 1 +fi + +echo "[cycle $TAG] load VFIO/mdev prerequisites" +sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev 2>/dev/null || true +sudo modprobe vfio_iommu_type1 2>/dev/null || true + +echo "[cycle $TAG] stop nvidia-persistenced, power/control=on" +sudo systemctl stop nvidia-persistenced 2>/dev/null || true +echo on | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control >/dev/null +echo on | sudo tee /sys/bus/pci/devices/0000:00:01.0/power/control >/dev/null 2>&1 || true + +echo "[cycle $TAG] insmod $(basename "$MODULE")" +# shellcheck disable=SC2046 +sudo insmod "$MODULE" $(cat "$ARGV_FILE") + +U=$(uuidgen) +echo "$U" | sudo tee "/sys/class/mdev_bus/$MDEV_PARENT/mdev_supported_types/$MDEV_TYPE/create" >/dev/null +if [ ! -d "/sys/bus/mdev/devices/$U" ]; then echo "FATAL: mdev create failed"; exit 1; fi + +mkdir -p "/tmp/sockets/$U" +sudo rm -f /tmp/qemu-qmp.sock /tmp/qemu-guest-agent.sock /tmp/qemu-harness.pid +rm -f /dev/shm/vgpu-guest-ram 2>/dev/null || true +touch /dev/shm/kvmfr-$U /dev/shm/kvmsr-$U 2>/dev/null || true +chmod 660 /dev/shm/kvmfr-$U /dev/shm/kvmsr-$U 2>/dev/null || true + +QLOG="$BACKUP/${TAG}-qemu-session.txt" +SL="$BACKUP/${TAG}-serial.log" +cat > "$QEMU_SCRIPT" <<EOF +#!/bin/bash +exec /bin/qemu-system-x86_64 -D $QLOG -no-hpet \ + -qmp unix:/tmp/qemu-qmp.sock,server,nowait \ + -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 \ + -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \ + -serial file:$SL -gdb tcp::1234 -display none \ + -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr \ + -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-$U,size=128M,share=yes \ + -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr \ + -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-$U,size=2M,share=yes \ + -object memory-backend-file,id=pc.ram,size=12288M,mem-path=/dev/shm/vgpu-guest-ram,share=yes,prealloc=on \ + -uuid $U -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \ + -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on \ + -rtc clock=host,base=localtime -m 12288 -smp cores=4,threads=1,sockets=1 \ + -hda $DISK --enable-kvm \ + -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 \ + -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/$U,display=off \ + -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 \ + -net user -net nic -device vhost-vsock-pci,guest-cid=85770 \ + -daemonize -pidfile /tmp/qemu-harness.pid +EOF +chmod +x "$QEMU_SCRIPT" + +echo "[cycle $TAG] launch QEMU (mdev $U)" +sudo bash "$QEMU_SCRIPT" > "$RUNDIR/qemu-$TAG.log" 2>&1 +sleep 3 +if ! pgrep -f qemu-system-x86_64 >/dev/null; then + echo "FATAL: QEMU failed"; tail -5 "$RUNDIR/qemu-$TAG.log"; exit 1 +fi + +echo "[cycle $TAG] mdev $U" +echo "$U" > "$RUNDIR/current-mdev" +echo "$TAG" > "$RUNDIR/current-tag" + +for i in $(seq 1 90); do + if timeout 5 sudo python3 "$REPO/scripts/vgpu_vm.py" exec "echo CYCLE_UP" 2>/dev/null | grep -q CYCLE_UP; then + echo "[cycle $TAG] guest agent up" + exit 0 + fi + sleep 2 +done +echo "WARN: guest agent not reachable yet (serial: $SL)" +exit 0 diff --git a/tools/guest-diag/egl_retry.c b/tools/guest-diag/egl_retry.c new file mode 100644 index 0000000..ce45d45 --- /dev/null +++ b/tools/guest-diag/egl_retry.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * egl_retry.c - EGL/GLES2 offscreen probe with an explicit eglMakeCurrent + * retry loop. The r581-era trace showed the glsi engine-init returning + * 0xee00001 ("busy") exactly once and setting the device state bit 0x20; + * eglcore does not retry. This probe retries the make-current (and the whole + * display/context creation) up to N times, printing the precise EGL error of + * every attempt, so we can tell "one-shot busy, retry succeeds" from + * "permanent failure". + * + * Build (guest): gcc -O2 -o egl_retry egl_retry.c -lEGL -lGLESv2 + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <EGL/egl.h> +#include <EGL/eglext.h> +#include <GLES2/gl2.h> + +static EGLDisplay get_device_display(void) +{ + PFNEGLGETPLATFORMDISPLAYEXTPROC gp = + (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress( + "eglGetPlatformDisplayEXT"); + PFNEGLQUERYDEVICESEXTPROC qd = + (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress( + "eglQueryDevicesEXT"); + EGLDeviceEXT devs[8]; + EGLint n = 0; + + if (!gp || !qd || !qd(8, devs, &n) || n <= 0) + return EGL_NO_DISPLAY; + printf("EGL_DEVICES n=%d\n", n); + return gp(EGL_PLATFORM_DEVICE_EXT, devs[0], NULL); +} + +int main(int argc, char **argv) +{ + int tries = argc > 1 ? atoi(argv[1]) : 10; + EGLDisplay dpy; + EGLint major = 0, minor = 0; + EGLConfig cfg; + EGLint ncfg = 0; + EGLSurface surf; + EGLContext ctx = EGL_NO_CONTEXT; + EGLint attrs[] = { + EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, + EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT, + EGL_RED_SIZE, 8, EGL_GREEN_SIZE, 8, EGL_BLUE_SIZE, 8, + EGL_ALPHA_SIZE, 8, EGL_NONE + }; + EGLint pb[] = { EGL_WIDTH, 64, EGL_HEIGHT, 64, EGL_NONE }; + EGLint cattr[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE }; + int i; + + setvbuf(stdout, NULL, _IONBF, 0); + + dpy = get_device_display(); + if (dpy == EGL_NO_DISPLAY) { + printf("NO_DEVICE_DISPLAY\n"); + return 1; + } + if (!eglInitialize(dpy, &major, &minor)) { + printf("EGL_INIT_FAIL 0x%x\n", eglGetError()); + return 1; + } + printf("EGL_INIT ok %d.%d vendor=%s\n", major, minor, + eglQueryString(dpy, EGL_VENDOR)); + if (!eglBindAPI(EGL_OPENGL_ES_API)) { + printf("BIND_API_FAIL 0x%x\n", eglGetError()); + return 1; + } + if (!eglChooseConfig(dpy, attrs, &cfg, 1, &ncfg) || ncfg < 1) { + printf("CHOOSE_CONFIG_FAIL 0x%x n=%d\n", eglGetError(), ncfg); + return 1; + } + for (i = 0; i < tries; i++) { + surf = eglCreatePbufferSurface(dpy, cfg, pb); + if (surf == EGL_NO_SURFACE) { + printf("TRY %d CREATE_SURFACE_FAIL 0x%x\n", i + 1, + eglGetError()); + usleep(200000); + continue; + } + if (ctx == EGL_NO_CONTEXT) { + ctx = eglCreateContext(dpy, cfg, EGL_NO_CONTEXT, + cattr); + if (ctx == EGL_NO_CONTEXT) { + printf("TRY %d CREATE_CTX_FAIL 0x%x\n", i + 1, + eglGetError()); + eglDestroySurface(dpy, surf); + usleep(200000); + continue; + } + } + if (eglMakeCurrent(dpy, surf, surf, ctx)) { + const char *rend = (const char *)glGetString( + GL_RENDERER); + const char *ver = (const char *)glGetString( + GL_VERSION); + printf("TRY %d CURRENT OK renderer=%s version=%s\n", + i + 1, rend ? rend : "?", ver ? ver : "?"); + /* draw one clear + readback so the pipeline runs */ + glClearColor(0.1f, 0.4f, 0.9f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + { + unsigned char px[4]; + glReadPixels(0, 0, 1, 1, GL_RGBA, + GL_UNSIGNED_BYTE, px); + printf("PIXEL %02x%02x%02x%02x\n", px[0], + px[1], px[2], px[3]); + } + printf("EGL_RETRY_OK on try %d\n", i + 1); + return 0; + } + printf("TRY %d MAKE_CURRENT_FAIL 0x%x\n", i + 1, + eglGetError()); + eglDestroySurface(dpy, surf); + usleep(200000); + } + printf("EGL_RETRY_EXHAUSTED tries=%d\n", tries); + return 1; +}
\ No newline at end of file diff --git a/tools/guest-diag/fbcopy_pub.c b/tools/guest-diag/fbcopy_pub.c new file mode 100644 index 0000000..a064882 --- /dev/null +++ b/tools/guest-diag/fbcopy_pub.c @@ -0,0 +1,718 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * fbcopy_pub.c - capture the vGPU framebuffer WITHOUT X and publish frames + * into the kvmfr ivshmem BAR. + * + * The X server's screen has 1280x720 pixels in the emulated VRAM, which the + * guest maps as the vGPU's BAR1 (resource1). The X SHM readback path can + * stall behind the X server's own push wait; this tool reads the framebuffer + * pixels straight out of the BAR1 mapping instead (host RAM backing, no GPU + * or X round-trip), swizzles BGRA -> RGBA and writes the kvmfr frame + * protocol the host reader already decodes. + * + * Usage (guest, as root): + * ./fbcopy_pub test one solid red frame (pipeline check) + * ./fbcopy_pub find [passes] [ms] motion-scan first 64 MB of BAR1 + * ./fbcopy_pub findp [off] [sz] [p] [ms] + * pread motion-scan of any BAR1 range + * (mmap only reaches the first 64 MB) + * ./fbcopy_pub pub <off> [n] [fps] [w] [h] [pitch] + * copy w*h from BAR1@off (pitch), + * mmap path (off < 64 MB) + * ./fbcopy_pub pubp <off> [n] [fps] [w] [h] [pitch] + * same but pread path (any offset) + * + * Find/build on the host (same glibc as the guest): gcc -O2 -o fbcopy_pub + * fbcopy_pub.c + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <dirent.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <time.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" */ +#define DEF_W 1280 +#define DEF_H 720 +#define DEF_PITCH 5120 /* xorg-vgpu.conf: 1280*4 rounded */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +static const char *find_vgpu_resource1(void) +{ + static char path[320]; + DIR *d = opendir("/sys/bus/pci/devices"); + struct dirent *de; + + if (!d) + return NULL; + while ((de = readdir(d))) { + char p[340], buf[32]; + FILE *f; + int n; + + if (de->d_name[0] == '.') + continue; + snprintf(p, sizeof(p), "/sys/bus/pci/devices/%s/vendor", + de->d_name); + f = fopen(p, "r"); + if (!f) + continue; + n = fread(buf, 1, sizeof(buf) - 1, f); + fclose(f); + if (n <= 0) + continue; + buf[n] = 0; + if (strtoul(buf, NULL, 16) != 0x10de) + continue; + snprintf(path, sizeof(path), + "/sys/bus/pci/devices/%s/resource1", de->d_name); + closedir(d); + return path; + } + closedir(d); + return NULL; +} + +/* + * The vfio mdev region mmap is limited: only file offset 0 with len <= 64 MB + * works (nonzero offsets and 256 MB both EINVAL). Map the first 64 MB once at + * offset 0 and index into the mapping for every access. + */ +#define CHUNK_MAX (64UL << 20) + +static unsigned char *map_vram(int fd, size_t want, size_t *got) +{ + size_t len = want > CHUNK_MAX ? CHUNK_MAX : want; + unsigned char *p; + + p = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0); + if (p == MAP_FAILED) + return NULL; + *got = len; + return p; +} + +static double t_ms(const struct timespec *a, const struct timespec *b) +{ + return (double)(b->tv_sec - a->tv_sec) * 1e3 + + (double)(b->tv_nsec - a->tv_nsec) / 1e6; +} + +/* + * Find the largest contiguous run of "dense" pages (>= 64 of 256 sampled + * bytes nonzero) - an image in VRAM looks like one long dense run. Small + * gaps (<= 16 pages) are tolerated. + */ +static void report_cluster(const uint32_t *nz, size_t pages, + unsigned long long base_off, const char *tag) +{ + size_t p, run_start = 0, last_dense = 0, best_start = 0, best_len = 0; + int in_run = 0; + + for (p = 0; p <= pages; p++) { + int dense = p < pages && nz[p] >= 64; + + if (dense) { + if (!in_run) { + run_start = p; + in_run = 1; + } + last_dense = p; + } else if (in_run && p - last_dense > 16) { + size_t len = last_dense - run_start + 1; + + if (len > best_len) { + best_len = len; + best_start = run_start; + } + in_run = 0; + } + } + if (best_len >= 8) + printf("%s_CLUSTER off=0x%llx pages=%zu bytes=%zu\n", tag, + base_off + ((unsigned long long)best_start << 12), + best_len, best_len << 12); +} + +int main(int argc, char **argv) +{ + const char *mode = argc > 1 ? argv[1] : "find"; + const char *bar_path = getenv("FBCOPY_BAR") ? getenv("FBCOPY_BAR") : + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + const char *res1 = getenv("FBCOPY_RES1") ? getenv("FBCOPY_RES1") : + find_vgpu_resource1(); + struct stat st; + size_t map_size, i; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + int fd, vram_fd; + + setvbuf(stdout, NULL, _IONBF, 0); + if (!res1) { + printf("NO_VGPU\n"); + return 1; + } + printf("VGPU_RES1 %s\n", res1); + if (stat(res1, &st) || !st.st_size) { + printf("STAT_FAIL\n"); + return 1; + } + map_size = (size_t)st.st_size; + printf("VRAM_SIZE 0x%zx (%zu MB)\n", map_size, map_size >> 20); + vram_fd = open(res1, O_RDONLY | O_SYNC); + if (vram_fd < 0) { + perror(res1); + return 1; + } + + /* kvmfr BAR next: every mode needs it except find. */ + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + perror(bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + close(fd); + if (bar == MAP_FAILED) { + perror("mmap kvmfr"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + if (!strcmp(mode, "zero")) { + /* Reset the kvmfr header seq so a reader seeding from it will + * accept the next pub run's frame 1. */ + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = DEF_W; + hdr->height = DEF_H; + hdr->stride = DEF_W * 4; + hdr->format = 0; + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = 0; + printf("SEQ_ZEROED\n"); + return 0; + } + + if (!strcmp(mode, "test")) { + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + size_t n = (size_t)DEF_W * DEF_H * 4; + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = DEF_W; + hdr->height = DEF_H; + hdr->stride = DEF_W * 4; + hdr->format = 0; + for (i = 0; i < n; i += 4) { + dst[i + 0] = (unsigned char)(argc > 2 ? atoi(argv[2]) + : 255); + dst[i + 1] = (unsigned char)(argc > 3 ? atoi(argv[3]) + : 0); + dst[i + 2] = (unsigned char)(argc > 4 ? atoi(argv[4]) + : 0); + dst[i + 3] = 0xff; + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = 1; + printf("TEST_FRAME_PUBLISHED\n"); + return 0; + } + + if (!strcmp(mode, "find")) { + /* + * Motion + content scan of the first 64 MB of BAR1 (the only + * window the vfio mdev mmap accepts). Sample 256 B of every + * 4 KB page over `passes` passes `interval_ms` apart; score + * pages by total differing bytes (motion) and by nonzero + * density (content). Rendering can be slower than one frame + * per second, so the interval defaults to 3 s and is + * configurable: find [passes] [interval_ms] + */ + int passes = argc > 2 ? atoi(argv[2]) : 3; + int interval_ms = argc > 3 ? atoi(argv[3]) : 3000; + size_t scan_size = map_size > CHUNK_MAX ? CHUNK_MAX : map_size; + size_t pages = scan_size >> 12; + unsigned char *s0, *s1, *tmp, *w; + uint32_t *motion, *nz; + size_t p, wlen; + int top, pass; + + if (passes < 2) + passes = 2; + w = map_vram(vram_fd, scan_size, &wlen); + if (!w) { + printf("CHUNK_MAP_FAIL\n"); + return 1; + } + printf("SCAN_WINDOW 0x%zx pages=%zu passes=%d interval=%dms\n", + wlen, pages, passes, interval_ms); + s0 = malloc(pages * 256); + s1 = malloc(pages * 256); + motion = calloc(pages, sizeof(uint32_t)); + nz = calloc(pages, sizeof(uint32_t)); + if (!s0 || !s1 || !motion || !nz) + return 1; + for (p = 0; p < pages; p++) + memcpy(s0 + p * 256, w + (p << 12), 256); + for (pass = 1; pass < passes; pass++) { + struct timespec a, b; + uint64_t moved = 0; + + usleep((useconds_t)interval_ms * 1000); + clock_gettime(CLOCK_MONOTONIC, &a); + for (p = 0; p < pages; p++) + memcpy(s1 + p * 256, w + (p << 12), 256); + clock_gettime(CLOCK_MONOTONIC, &b); + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + unsigned char *y = s1 + p * 256; + uint32_t d = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j] != y[j]) + d++; + motion[p] += d; + moved += d; + } + printf("PASS %d moved_bytes=%llu dt=%.0fms\n", + pass, (unsigned long long)moved, t_ms(&a, &b)); + tmp = s0; s0 = s1; s1 = tmp; + } + /* nonzero density of the last sample */ + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + uint32_t c = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j]) + c++; + nz[p] = c; + } + report_cluster(nz, pages, 0, "FIND"); + /* per-16MB summary */ + for (i = 0; i + 4096 <= pages; i += 4096) { + uint64_t s = 0, c = 0; + size_t j; + + for (j = i; j < i + 4096; j++) { + s += motion[j]; + c += nz[j]; + } + if (s || c) + printf("BLOCK off=0x%zx moving_bytes=%llu nonzero_bytes=%llu\n", + i << 12, (unsigned long long)s, + (unsigned long long)c); + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (motion[p] > bs) { + bs = motion[p]; + best = p; + } + if (!bs) + break; + printf("HOT off=0x%zx score=%u\n", best << 12, bs); + motion[best] = 0; + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (nz[p] > bs) { + bs = nz[p]; + best = p; + } + if (!bs) + break; + printf("CONTENT off=0x%zx nonzero=%u\n", + best << 12, bs); + nz[best] = 0; + } + printf("FIND_DONE\n"); + return 0; + } + + if (!strcmp(mode, "probe")) { + /* + * Ground truth on what BAR1 access actually works: try mmap + * windows at several offsets/sizes and pread at several + * offsets, report each result. Read-only. + */ + struct { + unsigned long long off, len; + } mm[] = { + { 0, 64UL << 20 }, + { 0, 256UL << 20 }, + { 0, 4UL << 20 }, + { 4UL << 20, 4UL << 20 }, + { 64UL << 20, 64UL << 20 }, + { 128UL << 20, 4UL << 20 }, + { 192UL << 20, 4UL << 20 }, + { 60UL << 20, 8UL << 20 }, + }; + unsigned long long po[] = { + 0, 0x100000, 4UL << 20, 60UL << 20, + 64UL << 20, 128UL << 20, 252UL << 20, + }; + unsigned char buf[64]; + size_t k; + + for (k = 0; k < sizeof(mm) / sizeof(mm[0]); k++) { + unsigned char *p; + + if (mm[k].off + mm[k].len > map_size) { + printf("MMAP off=0x%llx len=0x%llx SKIP(>bar)\n", + mm[k].off, mm[k].len); + continue; + } + errno = 0; + p = mmap(NULL, (size_t)mm[k].len, PROT_READ, + MAP_SHARED, vram_fd, (off_t)mm[k].off); + if (p == MAP_FAILED) + printf("MMAP off=0x%llx len=0x%llx FAIL errno=%d\n", + mm[k].off, mm[k].len, errno); + else { + printf("MMAP off=0x%llx len=0x%llx OK first=%02x%02x%02x%02x\n", + mm[k].off, mm[k].len, p[0], p[1], p[2], p[3]); + munmap(p, (size_t)mm[k].len); + } + } + for (k = 0; k < sizeof(po) / sizeof(po[0]); k++) { + ssize_t r; + + errno = 0; + r = pread(vram_fd, buf, 16, (off_t)po[k]); + if (r != 16) + printf("PREAD off=0x%llx FAIL r=%zd errno=%d\n", + po[k], r, errno); + else + printf("PREAD off=0x%llx OK %02x%02x%02x%02x\n", + po[k], buf[0], buf[1], buf[2], buf[3]); + } + printf("PROBE_DONE\n"); + return 0; + } + + if (!strcmp(mode, "findp")) { + /* + * Same motion + content scan as `find`, but via pread so the + * whole 256 MB BAR is reachable (mmap stops at 64 MB). + * findp [start_off] [size] [passes] [interval_ms] + */ + unsigned long long start = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + unsigned long long size = argc > 3 ? + strtoull(argv[3], NULL, 0) : map_size - start; + int passes = argc > 4 ? atoi(argv[4]) : 3; + int interval_ms = argc > 5 ? atoi(argv[5]) : 3000; + size_t pages = size >> 12; + unsigned char *s0, *s1, *tmp; + uint32_t *motion, *nz; + size_t p; + int top, pass; + + if (passes < 2) + passes = 2; + if (start + size > map_size) { + printf("RANGE_FAIL start=0x%llx size=0x%llx bar=0x%zx\n", + start, size, map_size); + return 1; + } + printf("SCANP start=0x%llx size=0x%llx pages=%zu passes=%d interval=%dms\n", + start, size, pages, passes, interval_ms); + s0 = malloc(pages * 256); + s1 = malloc(pages * 256); + motion = calloc(pages, sizeof(uint32_t)); + nz = calloc(pages, sizeof(uint32_t)); + if (!s0 || !s1 || !motion || !nz) + return 1; + for (pass = 0; pass < passes; pass++) { + struct timespec a, b; + uint64_t moved = 0; + + if (pass) + usleep((useconds_t)interval_ms * 1000); + clock_gettime(CLOCK_MONOTONIC, &a); + for (p = 0; p < pages; p++) { + unsigned char *dst = pass ? s1 : s0; + ssize_t r = pread(vram_fd, dst + p * 256, 256, + (off_t)(start + (p << 12))); + + if (r != 256) { + printf("PREAD_FAIL off=0x%llx r=%zd errno=%d\n", + start + (p << 12), r, errno); + return 1; + } + } + clock_gettime(CLOCK_MONOTONIC, &b); + if (pass) { + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + unsigned char *y = s1 + p * 256; + uint32_t d = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j] != y[j]) + d++; + motion[p] += d; + moved += d; + } + printf("PASS %d moved_bytes=%llu dt=%.0fms\n", + pass, (unsigned long long)moved, + t_ms(&a, &b)); + tmp = s0; s0 = s1; s1 = tmp; + } else { + printf("PASS 0 sampled dt=%.0fms\n", + t_ms(&a, &b)); + } + } + for (p = 0; p < pages; p++) { + unsigned char *x = s0 + p * 256; + uint32_t c = 0; + int j; + + for (j = 0; j < 256; j++) + if (x[j]) + c++; + nz[p] = c; + } + report_cluster(nz, pages, start, "FINDP"); + for (i = 0; i + 4096 <= pages; i += 4096) { + uint64_t s = 0, c = 0; + size_t j; + + for (j = i; j < i + 4096; j++) { + s += motion[j]; + c += nz[j]; + } + if (s || c) + printf("BLOCK off=0x%llx moving_bytes=%llu nonzero_bytes=%llu\n", + start + ((unsigned long long)i << 12), + (unsigned long long)s, + (unsigned long long)c); + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (motion[p] > bs) { + bs = motion[p]; + best = p; + } + if (!bs) + break; + printf("HOT off=0x%llx score=%u\n", + start + ((unsigned long long)best << 12), bs); + motion[best] = 0; + } + for (top = 0; top < 16; top++) { + size_t best = 0; + uint32_t bs = 0; + + for (p = 0; p < pages; p++) + if (nz[p] > bs) { + bs = nz[p]; + best = p; + } + if (!bs) + break; + printf("CONTENT off=0x%llx nonzero=%u\n", + start + ((unsigned long long)best << 12), bs); + nz[best] = 0; + } + printf("FINDP_DONE\n"); + return 0; + } + + if (!strcmp(mode, "pub")) { + unsigned long long off = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + int frames = argc > 3 ? atoi(argv[3]) : 30; + int fps = argc > 4 ? atoi(argv[4]) : 5; + int w = argc > 5 ? atoi(argv[5]) : DEF_W; + int h = argc > 6 ? atoi(argv[6]) : DEF_H; + int pitch = argc > 7 ? atoi(argv[7]) : DEF_PITCH; + unsigned long long span = (unsigned long long)(h - 1) * pitch + + (unsigned long long)w * 4; + unsigned long long base = off & ~0xfffULL; + size_t wlen = (size_t)((off - base) + span); + unsigned char *wmap; + int f, y, x; + + if (off + span > map_size) { + printf("OFF_RANGE_FAIL need=0x%llx bar=0x%zx\n", + off + span, map_size); + return 1; + } + /* 2026-10-28 (session 149): nonzero-offset mmaps of BAR1 work + * on this mdev (earlier builds assumed only offset 0 <= 64 MB + * was mappable). Map directly at the requested offset; fall + * back to the old offset-0 64 MB window if that ever fails. */ + wlen = (wlen + 0xfff) & ~0xfffUL; + wmap = mmap(NULL, wlen, PROT_READ, MAP_SHARED, vram_fd, + (off_t)base); + if (wmap == MAP_FAILED) { + size_t got = 0; + wmap = map_vram(vram_fd, map_size, &got); + if (wmap) { + base = 0; + wlen = got; + } + if (!wmap || off + span > wlen) { + printf("WIN_MAP_FAIL off=0x%llx len=0x%zx errno=%d\n", + base, wlen, errno); + return 1; + } + } + printf("PUB off=0x%llx base=0x%llx len=0x%zx frames=%d fps=%d %dx%d pitch=%d\n", + off, base, wlen, frames, fps, w, h, pitch); + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + for (f = 0; f < frames; f++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + unsigned char *src = wmap + (size_t)(off - base); + + for (y = 0; y < h; y++) { + unsigned char *s = src + (size_t)y * pitch; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + d[4 * x + 0] = s[4 * x + 2]; + d[4 * x + 1] = s[4 * x + 1]; + d[4 * x + 2] = s[4 * x + 0]; + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(f + 1); + __sync_synchronize(); + printf("FRAME seq=%d ms=%.1f\n", f + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d\n", f); + return 0; + } + + if (!strcmp(mode, "pubp")) { + unsigned long long off = argc > 2 ? + strtoull(argv[2], NULL, 0) : 0; + int frames = argc > 3 ? atoi(argv[3]) : 30; + int fps = argc > 4 ? atoi(argv[4]) : 5; + int w = argc > 5 ? atoi(argv[5]) : DEF_W; + int h = argc > 6 ? atoi(argv[6]) : DEF_H; + int pitch = argc > 7 ? atoi(argv[7]) : DEF_PITCH; + unsigned long long span = (unsigned long long)(h - 1) * pitch + + (unsigned long long)w * 4; + unsigned long long base = off & ~0xfffULL; + size_t len = (size_t)((off - base) + span); + unsigned char *buf; + int f, y, x; + + len = (len + 0xfff) & ~0xfffUL; + if (off + span > map_size) { + printf("OFF_RANGE_FAIL need=0x%llx bar=0x%zx\n", + off + span, map_size); + return 1; + } + buf = malloc(len); + if (!buf) + return 1; + printf("PUBP off=0x%llx frames=%d fps=%d %dx%d pitch=%d len=0x%zx\n", + off, frames, fps, w, h, pitch, len); + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + for (f = 0; f < frames; f++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + unsigned char *src; + ssize_t r = pread(vram_fd, buf, len, (off_t)base); + + if (r != (ssize_t)len) { + printf("PREAD_FAIL off=0x%llx r=%zd errno=%d\n", + base, r, errno); + return 1; + } + src = buf + (size_t)(off - base); + for (y = 0; y < h; y++) { + unsigned char *s = src + (size_t)y * pitch; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + d[4 * x + 0] = s[4 * x + 2]; + d[4 * x + 1] = s[4 * x + 1]; + d[4 * x + 2] = s[4 * x + 0]; + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(f + 1); + __sync_synchronize(); + printf("FRAME seq=%d ms=%.1f\n", f + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d\n", f); + return 0; + } + + printf("USAGE: fbcopy_pub test|find|findp|pub|pubp " + "<off> [n] [fps] [w] [h] [pitch]\n"); + return 1; +} diff --git a/tools/guest-diag/fenceword_assist.py b/tools/guest-diag/fenceword_assist.py new file mode 100644 index 0000000..9e207e2 --- /dev/null +++ b/tools/guest-diag/fenceword_assist.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# fenceword_assist.py - keep the Vulkan fence wait word of a live vk_frame120 +# client released (session 125). +# +# SUPERSEDED (SESSION 126, commit 95e7f57e): the service now authors these +# releases itself. The `sema_gr_ahead` fence fold sweeps the channel's +# pre-built ring continuation beyond GP_PUT through the ring's validated +# GPFIFO translation, decodes the INC-5 RELEASE groups it finds (addr 0x5c +# count>=5) and queues them in the standard pending ledger; r717 evidence: +# `sema-ahead: h=0xbeec0100 put=11 swept=10 rels=4` followed 30 ms later by +# `sema-svc: deferred release va=0x45ba090 payload=1 (was cur=0)`, with +# VKFRAME seq/DONE and NO assist running (see +# backups/evidence-20261002-victory/r717-fencefold/). Keep this tool only as +# the A/B fallback for boots that predate the fold (or with +# sema_gr_ahead=0). +# +# The client's fence wait polls a word in a 16 KB /dev/nvidiactl BO mapped +# read-write into the client: the init wait word lives at +0x80 (the t122 +# "BW" chaser writes it once) and the per-submit fence word at +0x90 of the +# SAME BO (GPU VA <base>+0x90, delivered by the submit push's INC-5 semaphore +# group). While the emulation does not apply that group, this host assist +# writes 1 to the +0x90 slot whenever it reads 0 - the direct predecessor of +# the service-side fold (see Handoff SESSION 125/126). +# +# Usage (host, while a t122-driven client runs): +# tools/guest-diag/fenceword_assist.py [seconds] [guest-pid] +import os +import subprocess +import sys +import time + +VMEXEC = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), + "..", "..", "scripts", "vgpu_vm.py")) +PY = r''' +import struct, sys, time +pids = [p for p in sys.argv[1:] if p] +maps_of = {} +def vma(pid, addr): + for line in open("/proc/%d/maps" % pid): + rng, perms = line.split(None, 2)[:2] + a, b = [int(x, 16) for x in rng.split("-")] + if a <= addr < b: + return line.strip() + return None +found = {} +for pid in pids: + try: + maps = [l for l in open("/proc/%d/maps" % pid) if "nvidia" in l and "rw-s" in l] + except OSError: + continue + for l in maps: + rng, perms, off = l.split(None, 3)[:3] + a, b = [int(x, 16) for x in rng.split("-")] + if b - a != 0x4000: + continue + try: + mem = open("/proc/%d/mem" % pid, "r+b", 0) + mem.seek(a + 0x80); init = mem.read(4) + mem.seek(a + 0x90); fw = mem.read(4) + except Exception: + continue + if init == b"\x01\x00\x00\x00" and fw == b"\x00\x00\x00\x00": + found[pid] = (a + 0x90, mem) + print("found pid=%s word=%#x" % (pid, a + 0x90), flush=True) +end = time.time() + float(sys.argv[1]) +while time.time() < end: + for pid, (w, mem) in list(found.items()): + try: + mem.seek(w) + if mem.read(4) == b"\x00\x00\x00\x00": + mem.seek(w); mem.write(struct.pack("<I", 1)); mem.flush() + print("released pid=%s word=%#x" % (pid, w), flush=True) + except Exception: + del found[pid] + time.sleep(0.5) +print("assist done", flush=True) +''' + + +def main(): + dur = sys.argv[1] if len(sys.argv) > 1 else "300" + pid = sys.argv[2] if len(sys.argv) > 2 else "" + b64 = __import__("base64").b64encode(PY.encode()).decode() + subprocess.run([VMEXEC, "exec", "--timeout", str(int(dur) + 60), + "echo %s | base64 -d > /tmp/fwa.py; sudo python3 /tmp/fwa.py %s %s" + % (b64, dur, pid)]) + + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/gl_frames.c b/tools/guest-diag/gl_frames.c new file mode 100644 index 0000000..d442d66 --- /dev/null +++ b/tools/guest-diag/gl_frames.c @@ -0,0 +1,237 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * gl_frames.c - render animated frames with direct GL (GLX pbuffer) on the + * guest's NVIDIA vGPU and publish them into the ivshmem (kvmfr) BAR so the + * host can see what the GL client actually rendered. + * + * This is the GL sibling of tools/cuda-ab/frame_producer.cu: the transport + * (guest GPU -> guest sysmem -> ivshmem -> host frame_reader.py) is identical, + * but the pixels come from the direct-GL client channel instead of CUDA. + * The pbuffer avoids the X window draw path that wedges the X server. + * + * Layout at BAR offset 0x100000 (same as frame_producer.cu): + * struct frame_hdr { magic, version, width, height, stride, format, seq } + * then width*height*4 bytes of RGBA8. + * + * Usage: sudo ./gl_frames [frames] [width] [height] [fps] + * Prints: FRAME seq=<n> ms=<render+readback ms> + */ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <stdint.h> +#include <time.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <glob.h> +#include <math.h> +#include <X11/Xlib.h> +#include <GL/glx.h> +#include <GL/gl.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" little-endian-ish */ +#define BAR_BYTES (128UL << 20) +#define BAR_PATH_DEF "/sys/bus/pci/devices/0000:00:04.0/resource2" + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +/* + * The ivshmem kvmfr device moves with the QEMU topology (it was 00:03.0 in an + * older launch, 00:04.0 in the current one). Discover the 128 MiB BAR instead + * of hardcoding a slot; GL_FRAMES_BAR overrides for debugging. + */ +static const char *find_bar(char *buf, size_t len) +{ + glob_t g; + size_t i; + const char *hit = NULL; + + if (glob("/sys/bus/pci/devices/*/resource2", 0, NULL, &g) != 0) + return NULL; + for (i = 0; i < g.gl_pathc; i++) { + struct stat st; + + if (stat(g.gl_pathv[i], &st) == 0 && + (unsigned long long)st.st_size == BAR_BYTES) { + snprintf(buf, len, "%s", g.gl_pathv[i]); + hit = buf; + break; + } + } + globfree(&g); + return hit; +} + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +static void draw_frame(int f, int w, int h) +{ + float a = (float)f * 0.15f; + int i; + + glViewport(0, 0, w, h); + glClearColor(0.05f, 0.05f, 0.10f, 1.0f); + glClear(GL_COLOR_BUFFER_BIT); + + glMatrixMode(GL_PROJECTION); + glLoadIdentity(); + glOrtho(-1.0, 1.0, -1.0, 1.0, -1.0, 1.0); + glMatrixMode(GL_MODELVIEW); + glLoadIdentity(); + + for (i = 0; i < 8; i++) { + float t = a + (float)i * 0.78539816339f; + + glBegin(GL_TRIANGLES); + glColor3f(1.0f, 0.2f + 0.09f * i, 0.1f); + glVertex2f(0.0f, 0.0f); + glColor3f(0.1f, 0.9f, 0.3f); + glVertex2f(0.75f * cosf(t), 0.75f * sinf(t)); + glColor3f(0.2f, 0.3f, 1.0f); + glVertex2f(0.55f * cosf(t + 0.5f), + 0.55f * sinf(t + 0.5f)); + glEnd(); + } + glFinish(); +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 30; + int w = argc > 2 ? atoi(argv[2]) : 640; + int h = argc > 3 ? atoi(argv[3]) : 480; + int fps = argc > 4 ? atoi(argv[4]) : 5; + static int attribs[] = { + GLX_DRAWABLE_TYPE, GLX_PBUFFER_BIT, + GLX_RENDER_TYPE, GLX_RGBA_BIT, + GLX_RED_SIZE, 8, GLX_GREEN_SIZE, 8, GLX_BLUE_SIZE, 8, + GLX_ALPHA_SIZE, 8, + None + }; + GLXFBConfig *fb; + GLXPbuffer pb; + XVisualInfo *vi; + GLXContext ctx; + Display *d; + int scr, n = 0, fd, i; + unsigned char *bar, *px; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + + char found[256]; + const char *bar_path = getenv("GL_FRAMES_BAR"); + + setvbuf(stdout, NULL, _IONBF, 0); + if (!bar_path) + bar_path = find_bar(found, sizeof(found)); + if (!bar_path) + bar_path = BAR_PATH_DEF; + printf("BAR %s\n", bar_path); + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, BAR_BYTES, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + d = XOpenDisplay(NULL); + if (!d) { + printf("NODISPLAY\n"); + return 2; + } + scr = DefaultScreen(d); + fb = glXChooseFBConfig(d, scr, attribs, &n); + if (!fb || n < 1) { + printf("NOFBCONFIG\n"); + return 3; + } + pb = glXCreatePbuffer(d, fb[0], (int[]){ + GLX_PBUFFER_WIDTH, w, GLX_PBUFFER_HEIGHT, h, None }); + if (!pb) { + printf("NOPBUFFER\n"); + return 4; + } + vi = glXGetVisualFromFBConfig(d, fb[0]); + ctx = glXCreateContext(d, vi, NULL, True); + if (!ctx) { + printf("NOCTX\n"); + return 5; + } + if (!glXMakeCurrent(d, pb, ctx)) { + printf("MAKECURRENT_FAIL\n"); + return 6; + } + printf("GL %s | %s\n", (const char *)glGetString(GL_VENDOR), + (const char *)glGetString(GL_RENDERER)); + + px = (unsigned char *)malloc((size_t)w * h * 4); + if (!px) { + printf("HOST_ALLOC_FAIL\n"); + return 7; + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + uint32_t seq; + + draw_frame(i, w, h); + glReadPixels(0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, px); + if (glGetError() != GL_NO_ERROR) + printf("GL_ERROR frame=%d\n", i); + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), px, + (size_t)w * h * 4); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + seq = (uint32_t)(i + 1); + *seqp = seq; + __sync_synchronize(); + printf("FRAME seq=%u ms=%.1f px0=%02x%02x%02x%02x\n", seq, + (double)(now_ns() - t0) / 1e6, + px[0], px[1], px[2], px[3]); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + printf("FRAME_DONE frames=%d %dx%d\n", frames, w, h); + + glXMakeCurrent(d, None, NULL); + glXDestroyContext(d, ctx); + glXDestroyPbuffer(d, pb); + XCloseDisplay(d); + free(px); + munmap(bar, BAR_BYTES); + close(fd); + return 0; +} diff --git a/tools/guest-diag/guest_pin_stream.py b/tools/guest-diag/guest_pin_stream.py index bb253a2..96c1e43 100644 --- a/tools/guest-diag/guest_pin_stream.py +++ b/tools/guest-diag/guest_pin_stream.py @@ -211,7 +211,11 @@ def read_u32(kc, va): return struct.unpack("<I", d)[0] if d else 0xFFFFFFFF -DBG_BUDGET = int(os.environ.get("STREAM_DBG_BUDGET", "600")) +DBG_BUDGET = int(os.environ.get("STREAM_DBG_BUDGET", "2400")) +# SESSION 153-CONTINUED-65: minimum seconds between full-offset sweeps of +# one strict anchor (the demand file re-sends every ~2 s; the sweep can cost +# tens of seconds - see the throttle comment at the strict processing loop). +ANCHOR_RETRY_S = float(os.environ.get("STREAM_ANCHOR_RETRY_S", "20")) _dbg_count = [0] @@ -306,6 +310,23 @@ def _nv_vmap_areas(kc): # instead of walking the whole +/-16 neighbourhood. _area_off = {} +# SESSION 153-CONTINUED-55: per-GL-family learned {family_base: (area_lo, +# adj)}. Once one page of a GL push-buffer family is content-verified the +# whole family resolves through the learned area/offset (continuation and +# zero pages included); cleared when the area disappears (context rebuild). +_gl_fam = {} + +# SESSION 153-r1397b: strict-validated window pins are re-emitted every few +# seconds. The one-shot emission loses pages: a burst can be dropped under +# host load, and the `done` latch then means the lost pages are never +# retried - the 60216 B GL setup push needs all 15 pages, so one missing +# page keeps the whole entry unreadable (r1397: pages 0x4007000..0x4009000 +# re-landed via the loose path but 0x400a000..0x400e000 never did; every +# 60216 B read answered src=255 unreadable=1 and the entry was skipped). +# The pairs are content-validated, so a repeat is idempotent; the host +# re-asserts GL-family pairs on receipt. epage -> [pairs, next_ts]. +_strict_reemit = {} + def resolve_va_by_content(kc, va, out): """Demand pin: find the page carrying GPU VA 'va' by content match. @@ -320,6 +341,12 @@ def resolve_va_by_content(kc, va, out): lottery. Only the requested VA is pinned, so a wrong candidate cannot poison other VAs. + SESSION 153-CONTINUED-55: the same rule now covers the GL push-buffer + families (PB_GL_FAMILIES). The setup-push holds (va=0x4001xxx) and the + draw pushes (0x402xxxx) are nv_vm_map_pages allocations exactly like the + CE window; without this they were rejected here silently and the held + setup push was never pinned, so its 4K pages read zeros at the host. + 2026-10-16b: the offset search window was +/-1 page. Pushes of a client whose pushbuffer vmap carries a larger per-allocation header (observed with multi-MB CE staging: pushes at 0x120000f58+ never resolved and the @@ -328,11 +355,47 @@ def resolve_va_by_content(kc, va, out): vmap area, because consecutive pushes advance sequentially inside the same allocation and share the offset. """ - if not (PB_CE_LO <= va < PB_CE_HI): - return False - off = (va - PB_CE_LO) & 0xFFFFFF - k = off >> 12 + if PB_CE_LO <= va < PB_CE_HI: + k = ((va - PB_CE_LO) & 0xFFFFFF) >> 12 + gl_base = None + else: + gl_base = next((lo for lo, hi in PB_GL_FAMILIES if lo <= va < hi), + None) + if gl_base is None: + return False + k = (va - gl_base) >> 12 areas = _nv_vmap_areas(kc) + # SESSION 153-CONTINUED-55: a GL push-buffer family is one allocation + # whose vmap offset is IDENTITY with the family offset (verified: the + # setup push's bytes at va 0x4001004 read back byte-exact at the vmap + # page's +0x1004). Once one page of the family resolves, every page of + # the family is known: continuation pages of a stream do NOT start with + # a method header (the 60216-byte setup push crosses page boundaries + # mid-run, so pages 2/3/5/7 fail the prefix test) and unwritten pages + # read all zero (pages 8..15 of the same push) yet are real translations + # the host read still needs. Use the learned (area, adj) for the whole + # family; only the FIRST learn pays the content test. + learned_fam = _gl_fam.get(gl_base) if gl_base else None + if learned_fam is not None: + lo, adj = learned_fam + if any(a_lo == lo for (a_lo, _h, _p) in areas): + idx = k + adj + pages = next(p for (a_lo, _h, p) in areas if a_lo == lo) + if 0 <= idx < pages: + cpu = lo + idx * 0x1000 + pa = kc.va_to_pa(cpu) + d = kc.read(cpu, 64) if pa is not None else None + if pa is not None and d is not None: + if gl_base is not None or (_push_prefix_ok(d) and + any(d[:32])): + out.append("0x%x,0x%x,1\n" % (va & ~0xFFF, pa)) + dbg("demand-va: 0x%x -> pa=0x%x (gl family learned" + " area=0x%x adj=%+d)" + % (va & ~0xFFF, pa, lo, adj)) + return True + # The learned area is gone (context rebuilt): fall through to a + # fresh content match and re-learn. + _gl_fam.pop(gl_base, None) for (lo, _hi, pages) in areas: learned = _area_off.get(lo) if learned is not None: @@ -345,24 +408,640 @@ def resolve_va_by_content(kc, va, out): if idx < 0 or idx >= pages: continue cpu = lo + idx * 0x1000 - d = kc.read(cpu, 64) - if not d or not any(d[:32]): - continue - if not _push_prefix_ok(d): - continue pa = kc.va_to_pa(cpu) if pa is None: continue + # 0x200 (not 64): a GL family's page 0 can carry its first + # method header at offset ~0x68 behind a zero-filled prefix + # (observed on the 60216 B setup push), invisible to a 64-byte + # read. + d = kc.read(cpu, 0x200) + if not d or not any(d): + continue + if not _push_prefix_ok(d): + continue _area_off[lo] = adj + if gl_base is not None: + # First content-verified page of this GL family: the + # identity mapping (area, adj) now serves every page of + # the family, including continuation and zero pages. + _gl_fam[gl_base] = (lo, adj) out.append("0x%x,0x%x,1\n" % (va & ~0xFFF, pa)) - dbg("demand-va: 0x%x -> pa=0x%x (area off=%d adj=%+d)" - % (va & ~0xFFF, pa, idx, adj)) + dbg("demand-va: 0x%x -> pa=0x%x (area off=%d adj=%+d%s)" + % (va & ~0xFFF, pa, idx, adj, + " gl-learn" if gl_base is not None else "")) return True dbg("demand-va: 0x%x not found by content (%d areas)" % (va & ~0xFFF, len(areas))) return False +# SESSION 153-CONTINUED-64: candidate adjacencies for the strict window scan. +_STRICT_ADJ = (0, 1, -1, 2, -2, 3, -3, 4, -4, 6, -6, 8, -8, 14, -14, 16, -16) + + +def _stream_parse_full(buf): + """Exact port of the host's svc_stream_parse (gsp_sema_service.c), plus + the MME-setup content requirements the bare parse lacks. + + Zero words are NOP padding (skipped anywhere in the stream), ops 1/3/5 + are INC/NON_INC/ONE_INC runs with a 12-bit count (1..0xfff) and count + data words, ops 0/2/4/7 are header-only, anything else fails. The buffer + parses only when at least one method was seen and the walk lands exactly + on the end (trailing zero padding allowed). + + SESSION 153-CONTINUED-64b: the bare exact-end parse accepted a GMMU + page-table window - PTEs are {pfn<<12, flags} u32 pairs whose values sit + below 0x20000000, so every word reads as op 0 (header-only) and the walk + lands exactly on the end (r1122 pinned pa=0x11dbb0000: all 32 words of + every page op 0, zero data runs, host read `sema-bigrun hdr=0x1dbb7000`). + A window is now only accepted when it carries at least one data run AND + at least one run addressing an MME-setup method (0x114..0x124 / 0x234 / + 0x550..0x568 / 0x3400..0x34fc - the same windows the host's + svc_mme_setup_run_addr classifies with). A page table can never satisfy + either; the held setup push always carries both (the r1116 capture's + first run `20020047` = INC run @0x11c). + """ + off = 0 + methods = 0 + data_runs = 0 + mme_hits = 0 + code_words = 0 + n = len(buf) + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count or off + 4 + count * 4 > n: + return False + data_runs += 1 + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + mme_hits += 1 + if addr == 0x114 and count >= 1: + code_words += count + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + return False + methods += 1 + if methods > 8192: + return False + while off < n and buf[off] == 0: + off += 1 + return (methods > 0 and off == n and data_runs > 0 and mme_hits > 0 + and code_words >= 8) + + +def _walk_measure(buf): + """Method-shape walk returning (end_ok, data_runs, mme_hits, code_words, + pair_runs, parsed_bytes). 'end_ok' mirrors _stream_parse_full's + exact-end + zero padding landing; 'pair_runs' counts runs addressing the + SA-pair method 0x11c (the client push signature); 'parsed_bytes' is how + far the walk got before a shape failure (or the window end).""" + off = 0 + methods = 0 + data_runs = 0 + mme_hits = 0 + code_words = 0 + pair_runs = 0 + n = len(buf) + end_ok = False + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count or off + 4 + count * 4 > n: + break + data_runs += 1 + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + mme_hits += 1 + if addr == 0x114: + code_words += count + if addr == 0x11c and count >= 2: + pair_runs += 1 + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + break + methods += 1 + if methods > 8192: + break + else: + o2 = off + while o2 < n and buf[o2] == 0: + o2 += 1 + end_ok = (methods > 0 and o2 == n) + return end_ok, data_runs, mme_hits, code_words, pair_runs, off + + +def _window_accept(buf, ln): + """Acceptance for a client-mapping candidate window. + + SESSION 153-CONTINUED-65: the full-offset sweep of the client-mem + resolver needs a false-positive gate that survives pointer/event data. + The probe of the r1125 boot found a 60216-byte window of pointer junk + that walked to an exact end with one misinterpreted 0x114 run + (mme=1): the old accept (data>0, mme>0, code>=8) would have pinned it. + Large windows therefore require the client push signature - at least + two MME-setup runs or two SA-pair (0x11c) runs - in addition to the + counts; small windows keep the historical strict rule. The relaxed + path additionally accepts a window whose tail is unmapped/unwritten + (walk parses >= 0x1000) when it carries >= 64 code words (the macro + upload signature). + + Returns (ok, mode). + """ + if ln < 0x8000: + return _stream_parse_full(buf), "strict" + end_ok, data, mme, code, pairs, parsed = _walk_measure(buf) + # The client-push signature: SA-pair (0x11c) runs or 0x114 code runs. + # The r1125 probe of the live client mappings measured 10k+ pointer/float + # windows with data/mme hits but pairs=0, code=0 - MME-setup-family hits + # alone (0x124/0x34xx) are NOT a discriminator; the pair/code runs are. + sig = (pairs >= 2 or code >= 8) + if end_ok and data >= 2 and sig: + return True, "strict" + if data >= 2 and sig and parsed >= 0x1000: + return True, "relaxed" + return False, None + + +def _prefix_walk_ok(buf, off): + """Cheap first-page pre-filter for a strict window candidate. + + Walks the method shapes from `off` to the end of `buf` WITHOUT requiring + the end to land on a boundary (runs may cross into the next page and a + run header near the page end declares data we cannot see yet). Fails on + the first header no method shape can produce; this kills arbitrary data + pages after a few words so the full parse only runs on survivors. + """ + n = len(buf) + while off + 4 <= n: + hdr = struct.unpack_from('<I', buf, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count: + return False + off += 4 + count * 4 + elif op in (0, 2, 4, 7): + off += 4 + else: + return False + return True + + +def _read_window(kc, cpu, off, ln): + """Read [cpu+off, cpu+off+ln) page-by-page through kcore. + + A vmap area's pages are contiguous in the kernel VA but arbitrary in + physical memory, so the window is assembled one page at a time. + """ + out = b"" + a = cpu + off + left = ln + while left > 0: + n = min(left, 0x1000 - (a & 0xFFF)) + d = kc.read(a, n) + if not d: + return None + out += d + a += n + left -= n + return out + + +def _mme_run_in_page(d): + """True when a 4K page contains a run addressing an MME-setup method. + + The strict-window prefilter for client-mapping candidates (cheap: one + page read rejects the ~511/512 wrong shifts before the full parse). + """ + n = len(d) + off = 0 + while off + 4 <= n: + hdr = struct.unpack_from('<I', d, off)[0] + if hdr == 0: + off += 4 + continue + op = (hdr >> 29) & 7 + count = (hdr >> 16) & 0xFFF + if op in (1, 3, 5): + if not count: + off += 4 + continue + addr = (hdr & 0x1FFF) << 2 + if (0x114 <= addr <= 0x124 or addr == 0x234 or + 0x550 <= addr <= 0x568 or 0x3400 <= addr <= 0x34FC): + return True + off += 4 + count * 4 + else: + off += 4 + return False + + +def resolve_va_client_mem(pid, ent_va, off, ln, out): + """Strict pin for a setup push whose pages live in CLIENT sysmem. + + SESSION 153-CONTINUED-64c: when the setup push's allocation is UVM + sysmem it has NO kernel vmap (r1122: no nv_vm_map_pages area held the + `a0100045` 0x114 upload runs anywhere on the boot - the content exists + only in the client's `rw-s` /dev/nvidia mappings, and the host's + GPU-page-table walk answered an unrelated page-table buffer for the + held VA). The client's CPU mappings + /proc/<pid>/pagemap are the only + ground truth for those pages. Scan the client's readable+writable + mappings: for each candidate page shift, read the exact [va, va+len) + window via /proc/<pid>/mem, validate with the host's own full parse, + and emit pins for every page of the validated window from the + pagemap. The cheap per-page MME-run prefilter rejects wrong shifts. + """ + if not pid: + return False + try: + pfd = os.open('/proc/%d/pagemap' % pid, os.O_RDONLY) + mfd = os.open('/proc/%d/mem' % pid, os.O_RDONLY) + except OSError: + return False + try: + gl_base = next((lo for lo, hi in PB_GL_FAMILIES + if lo <= (ent_va & ~0xFFF) < hi), None) + k = ((ent_va & ~0xFFF) - gl_base) >> 12 if gl_base is not None else None + if ln <= 0 or ln > 0x40000: + return False + need = (off + ln + 0xFFF) // 0x1000 + if need > 64: + return False + try: + maps = open('/proc/%d/maps' % pid).read().splitlines() + except OSError: + return False + # SESSION 153-r1398: collect the candidate maps first and run a + # GLOBAL two-pass candidate order - the neighbourhood offsets for + # EVERY map, then the full sweep. The r1398 client (glxgears) had + # 74 qualifying rw mappings ahead of the nvidiactl mapping that + # holds the MME 0x114 pack; their full sweeps burned 37 808 of the + # 8 192-candidate budget before the target map was reached, so the + # strict client-mem parse answered False while a manual parse of + # the same map accepted at idx 1 (near-first PER MAP is not enough + # when the budget is global; near-first ACROSS maps is). The + # content gates are unchanged: only _window_accept passes emit. + near_def = [0, 1, -1, 2, -2, 4, -4, 8, -8, 16, -16, 32, -32, + 64, -64, 128, -128, 256, -256] + near_set = set(near_def) + cand_maps = [] + for line in maps: + parts = line.split() + if len(parts) < 5 or 'rw' not in parts[1]: + continue + # SESSION 153-CONTINUED-64e: any rw mapping may carry the + # sysmem push (observed as pathless rw-s /dev/nvidia mappings; + # memfd-backed buffers cannot be excluded either - the strict + # parse + MME prefilter are the safety gates, not the path). + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + except ValueError: + continue + if hi - lo < (need + 4) * 0x1000 or hi - lo > (64 << 20): + continue + cand_maps.append((lo, (hi - lo) >> 12)) + tried = 0 + if k is None or k == 0: + # Unknown family, or the FAMILY-BASE page (k == 0): neither has + # an identity page index to key on, so sweep every page start + # of the mapping (bounded). SESSION 153-CONTINUED-65: the + # 0x3200000 macro-upload push's window sits at an arbitrary + # page offset inside the client's mapping (the r1124 + # power-of-two neighbourhood never tried it; the window content + # measures code=712, exact-end, under the strict walk). + phases = (0,) + else: + phases = (0, 1) + for phase in phases: + tried = 0 + for (lo, pages) in cand_maps: + if k is None or k == 0: + adjs = range(0, max(1, pages - need + 1)) + capped = True + elif phase == 0: + # SESSION 153-CONTINUED-65: neighbourhood first, then + # the FULL page sweep. The r1125 setup push (family + # pages k=1 and k=33, 60216 B) failed every power-of-two + # offset while a matching window could sit at an + # arbitrary mapping page; every candidate still passes + # _window_accept before any pin is emitted, so the + # widened sweep cannot pin a window the parse rejects. + # The near pass is cheap (<=19 windows per map) and is + # not charged against the r1398 global budget. + adjs = near_def + capped = False + else: + adjs = [a for a in range(-k, pages - need + 1 - k) + if a not in near_set] + capped = True + for adj in adjs: + idx = (k + adj) if k is not None else adj + if idx < 0 or idx + need > pages: + continue + win = lo + idx * 0x1000 + off + tried += 1 + if capped and tried > 8192: + break + try: + first = os.pread(mfd, 0x1000 - off, win) + except OSError: + continue + if not first: + continue + # An all-zero first page is a legitimate push head (the + # method stream may start later); only a page with non-zero + # content and no MME-setup run is rejected outright. + if not _mme_run_in_page(first) and any(first): + continue + buf = b'' + left = ln + a = win + ok = True + while left > 0: + n = min(left, 0x1000 - (a & 0xFFF)) + try: + d = os.pread(mfd, n, a) + except OSError: + ok = False + break + if not d or len(d) != n: + ok = False + break + buf += d + a += n + left -= n + if not ok: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpg = (ent_va & ~0xFFF) + j * 0x1000 + cpg = lo + (idx + j) * 0x1000 + os.lseek(pfd, (cpg >> 12) * 8, 0) + e = int.from_bytes(os.read(pfd, 8), 'little') + if not (e >> 63) & 1: + pas = None + break + pas.append(((e & ((1 << 55) - 1)) << 12)) + if not pas: + continue + for j, pa in enumerate(pas): + out.append("0x%x,0x%x,1\n" + % ((ent_va & ~0xFFF) + j * 0x1000, pa)) + dbg("demand-va: 0x%x -> pa=0x%x (%s client-mem parse" + " %u B ok pages=%u map=0x%x idx=%d)" + % (ent_va & ~0xFFF, pas[0], mode, ln, need, lo, idx)) + return True + # The near pass (phase 0, family anchors) is cheap per map and + # completes for every map; only the full-sweep pass keeps the + # historical 8192-candidate bound (r1398: prior maps burned + # 37 808 candidates before the content map, so the bound must + # never apply to the near pass or it dies before the target). + if (k is None or k == 0 or phase == 1) and tried > 8192: + break + return False + finally: + os.close(pfd) + os.close(mfd) + + +def _strict_kvm_sweep(kc, ent_va, off, ln, out): + """Kernel-vmap sweep for an anchor with no known-family page index. + + SESSION 153-CONTINUED-74: mirrors the k == 0 known-family path (the + resolver that found the r1126 0x3200000 macro window) but works for ANY + anchor outside PB_GL_FAMILIES. Every candidate window passes the host's + own full parse (`_window_accept`) before a single pin is emitted, so a + false positive is impossible by construction; the cheap per-page prefix + walk rejects arbitrary data after a few words. Bounded: at most + 8192 candidate windows and at most 64 pages of a window (the same caps + as resolve_va_client_mem). + + Areas swept: the nv_vm_map_pages vmap set (the historical resolver) plus + the nvidia os_alloc_mem vmalloc areas (`vpages`), which the r1156 probe + showed can be large (571 pages in one area) and are completely outside + _nv_vmap_areas' filter. A window is only emitted for an area whose pages + are all present in the CPU alias. + """ + import re as _re + + page = ent_va & ~0xFFF + need = (off + ln + 0xFFF) // 0x1000 + if ln <= 0 or ln > 0x40000 or need > 64: + return False + areas = list(_nv_vmap_areas(kc)) + try: + for line in open("/proc/vmallocinfo"): + if "[nvidia]" not in line or "vmalloc" not in line: + continue + if "nv_vm_map_pages" in line or "phys=" in line: + continue + if "os_map_kernel_space" in line or "ioremap" in line: + continue + m = _re.match(r'0x([0-9a-f]+)-0x([0-9a-f]+)\s+(\d+)', line) + if not m: + continue + lo = int(m.group(1), 16) + hi = int(m.group(2), 16) + pages = (hi - lo) >> 12 + if pages >= need and lo < hi: + areas.append((lo, hi, pages)) + except OSError: + pass + tried = 0 + for (lo, _hi, pages) in areas: + for adj in range(0, max(1, pages - need + 1)): + tried += 1 + if tried > 8192: + return False + cpu = lo + adj * 0x1000 + first = kc.read(cpu, 0x1000) + if not first: + continue + if any(first) and not _prefix_walk_ok(first, off): + continue + buf = _read_window(kc, cpu, off, ln) + if buf is None: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpa = kc.va_to_pa(cpu + j * 0x1000) + if vpa is None: + pas = None + break + pas.append(vpa) + if not pas: + continue + for j, vpa in enumerate(pas): + out.append("0x%x,0x%x,1\n" % (page + j * 0x1000, vpa)) + dbg("demand-va: 0x%x -> pa=0x%x (%s vmap parse %u B ok" + " pages=%u area=0x%x adj=%+d)" + % (page, pas[0], mode, ln, need, lo, adj)) + return True + dbg("demand-va: 0x%x strict vmap sweep failed (%u B, %u windows)" + % (page, ln, tried)) + return False + + +def resolve_va_strict(kc, ent_va, off, ln, out, pid=None): + """Demand pin validated by the host's exact full-parse of [ent_va, +ln). + + SESSION 153-CONTINUED-64: the content heuristic learned a GL family from + ONE page's weak 0x200-byte prefix test and then served every family page + from that (area, adj) with NO content check at all; r1120 pinned + 0x4001000 -> a page whose bytes (`00040000 00001000...`) failed the + host's stream parse for the whole boot (`plausible=0 unreadable=1`), so + the setup push was never captured and the prelude gate deferred every + launch. This path runs the HOST'S OWN full parse on the exact bytes the + host will read before any pin is emitted, then pins EVERY page of the + validated window and learns the family from it. A family is never + served from an unvalidated learn. + """ + page = ent_va & ~0xFFF + gl_base = next((lo for lo, hi in PB_GL_FAMILIES if lo <= page < hi), None) + if ln <= 0 or ln > 0x40000: + return False + if gl_base is None: + # SESSION 153-CONTINUED-64e: an anchor outside the known families + # (r1124: the hold moved to a 153600-byte push at 0x3200000) has no + # identity page index; only the client-mapping sweep can resolve it. + # + # SESSION 153-CONTINUED-74: the unknown-family branch skipped the + # kernel-vmap sweep entirely - the known-family path has always tried + # it, and it is the resolver that found the r1126 0x3200000 macro + # window (a KERNEL-mapped allocation no process pagemap covers). The + # r1156 held push (va=0x3e4ccccc, 123916 B) failed here after only + # the client-mem attempt ("strict unknown-family client-mem failed"), + # so a kernel-resident window could never be found for any anchor + # outside PB_GL_FAMILIES. Try the vmap sweep first (same full-parse + # gate; strictly additive - a failed sweep costs the bounded reads + # and falls through to the client-mem path unchanged). + try: + if _strict_kvm_sweep(kc, ent_va, off, ln, out): + return True + except Exception: + pass + try: + if resolve_va_client_mem(pid, ent_va, off, ln, out): + return True + except Exception: + pass + dbg("demand-va: 0x%x strict unknown-family (vmap+client-mem) failed" + % page) + return False + gl_hi = next(hi for lo, hi in PB_GL_FAMILIES if lo <= page < hi) + k = (page - gl_base) >> 12 + need = (off + ln + 0xFFF) // 0x1000 + if need > 64 or page + need * 0x1000 > gl_hi: + return False + areas = _nv_vmap_areas(kc) + cands = [] + fam = _gl_fam.get(gl_base) + if fam is not None and any(a_lo == fam[0] for (a_lo, _h, _p) in areas): + cands.append(fam) + if k == 0: + # SESSION 153-CONTINUED-65: a family-BASE anchor (the 0x3200000 + # macro-upload push) has no identity page index to key on. The + # r1126 macro window lives in a KERNEL-mapped allocation (no + # process pagemap maps it: the whole-process reverse scan found + # none), so the kcore vmap sweep is the only resolver - and the + # power-of-two neighborhood missed the window's arbitrary page + # index. Sweep every page of every nv_vm_map_pages area; the + # _window_accept parse gates each candidate. + for (lo, _hi, pages) in areas: + for adj in range(0, max(1, pages - need + 1)): + c = (lo, adj) + if c not in cands: + cands.append(c) + else: + for (lo, _hi, pages) in areas: + for adj in _STRICT_ADJ: + c = (lo, adj) + if c not in cands: + cands.append(c) + tried = 0 + for (lo, adj) in cands: + pages = next(p for (a_lo, _h, p) in areas if a_lo == lo) + idx = k + adj + if idx < 0 or idx + need > pages: + continue + tried += 1 + if tried > (8192 if k == 0 else 128): + break + cpu = lo + idx * 0x1000 + first = kc.read(cpu, 0x1000) + if not first: + continue + # SESSION 153-CONTINUED-65: an all-zero first page is a legitimate + # push head - the r1126 macro-upload window (0x3200000, 153600 B) + # starts with a fully zero page (the method stream lands later in + # the window); the old `not any(first)` skip threw exactly that + # candidate away (the r1128 sweep tried 166 kernel-vmap windows and + # rejected the true one on this check). The prefix walk trivially + # passes an all-zero page; the full-window accept is the gate. + if any(first) and not _prefix_walk_ok(first, off): + continue + buf = _read_window(kc, cpu, off, ln) + if buf is None: + continue + acc, mode = _window_accept(buf, ln) + if not acc: + continue + pas = [] + for j in range(need): + vpa = kc.va_to_pa(cpu + j * 0x1000) + if vpa is None: + pas = None + break + pas.append(vpa) + if not pas: + continue + for j, vpa in enumerate(pas): + out.append("0x%x,0x%x,1\n" % (page + j * 0x1000, vpa)) + _area_off[lo] = adj + _gl_fam[gl_base] = (lo, adj) + dbg("demand-va: 0x%x -> pa=0x%x (%s parse %u B ok pages=%u" + " area=0x%x adj=%+d)" % (page, pas[0], mode, ln, need, lo, adj)) + return True + dbg("demand-va: 0x%x strict parse failed (%u B, %u windows tried)" + % (page, ln, tried)) + # SESSION 153-CONTINUED-64c: no kernel vmap held the content - the + # allocation is UVM sysmem visible only in the client's own mappings. + try: + if resolve_va_client_mem(pid, ent_va, off, ln, out): + return True + except Exception: + pass + dbg("demand-va: 0x%x strict client-mem parse failed too" % page) + return False + + def resolve_mem_pins(kc, mem, out, default_pages=16, va_lo=0, va_hi=0, page_off=0): """uvm_rm_mem kptr -> GPU-VA/GPA runs (direct map or page-table walk). @@ -810,6 +1489,32 @@ def resolve_pb_pins(kc, pb, out): # hold the ring-entry pushes is chosen (content anchor, _mem_content_ok). PB_CE_LO = 0x120000000 PB_CE_HI = 0x121000000 +# SESSION 153-CONTINUED-55: GL push-buffer VA families. The GL client's +# setup/MME-prelude push buffers (observed hold va=0x4001004 len=60216) and +# the draw pushes (observed va=0x4021004) are separate nv_vm_map_pages +# allocations in two fixed windows; the same family-relative page index +# rule applies: k = (va - base) >> 12. +# SESSION 153-CONTINUED-63: the r1117 boot's setup push lives at +# va=0x4200110 len=12288 - a THIRD window (0x4200000..0x4210000) the two +# original families did not cover, so the content resolver answered +# nothing and the read lottery could never be won for it (the +# 0x4020000-family hold re-armed for its whole 20 s window). +# SESSION 153-R1394: the r1390/r1393 boots' setup push moved to +# va=0x4250110 len=12288 (context push 0x4217014), past the 0x4210000 +# end of the r1117 window, so the same silent family miss left entry 9 +# unreadable (~337 `skipping stale entry` repeats). Widen the third +# family to 0x4200000..0x4300000. Base stays 0x4200000, so the identity +# k=(va-base)>>12 for the moved setup push is 0x50 and the search +# neighbourhood is unchanged (same rule as the r1117 push at k=0). +# RISK (bounded, noted): the widened window spans several allocations and +# _gl_fam is keyed on the shared base, so a loose resolver hit for a +# second allocation (e.g. 0x4217014) may be served from the first +# allocation's learned (area, adj) with no content check. The strict +# anchors (full host parse via resolve_va_strict) and the loose-deferral +# in the want-file walk remain the gate for those VAs. +PB_GL_FAMILIES = ((0x3200000, 0x3300000), (0x4000000, 0x4100000), + (0x4200000, 0x4300000), + (0x42000000, 0x43000000)) # m165 (D-F1a): UVM semaphore-pool page range. The pool is a 4 KB sysmem # uvm_rm_mem placed by the UVM arena right after the 16 MB CE push buffer; # the client's tracker words (0x121000000 + 4*idx) live in its first page. @@ -1428,11 +2133,19 @@ def pid_line(tag, pid, start): def cdt_pid(): prefixes = tuple(p for p in os.environ.get("CLIENT_PREFIXES", - "cdt,va,victory").split(",") if p) + "cdt,va,victory,Xorg,X,glxgears," + "gzdoom").split(",") if p) # Processes that hold a /dev/nvidia* fd but are not CUDA clients; the # scanner must never commit to them (a transient nvidia-smi at boot was # picked once and its exited pagemap read back EOF forever). - non_clients = ("nvidia-smi", "nvidia-persistenced", "Xorg", "X") + # + # SESSION 146: Xorg/X used to be excluded here too, but the GL/GZDOOM + # path needs the X server's pages pinned - Xorg screen init fails with + # "Failed to initialize DMA / Failed to allocate push buffer" when the + # nvidia-push buffers cannot be resolved (observed r816). Xorg is a + # long-lived process (unlike the nvidia-smi transient the exclusion + # guarded against), so it is a valid, stable pin target. + non_clients = ("nvidia-smi", "nvidia-persistenced") cands = [] for d in os.listdir('/proc'): if not d.isdigit(): @@ -1527,6 +2240,42 @@ def scan_range(fd, lo, hi, out): va += got * 0x1000 +def _pagemap_window_pin(pid, ev, off, ln, out): + """Direct client-pagemap pin for [ev, ev+ln) (window-wide r586 fallback). + + SESSION 153-r1397: a strict anchor whose full-window content parse fails + (both the kernel-vmap candidates and the client-mem scan) used to set + strict_pending and thereby stall EVERY loose demand VA of its family via + the pend_bases deferral - the 60 216 B setup push's pages were demanded + individually by the host but suppressed, so the entry stayed unreadable + for the whole generation (r1396: entry 8 unreadable=1 on all gens while + its sibling entries read fine). The anchor VA is the host's own GPFIFO + entry VA: when the client maps the pushbuffer at the same VA (rw-s + /dev/nvidia mappings - the same premise as the r586 single-page pagemap + fallback proven for the non-family draws), a present page IS the + translation the host needs; no content gate and no family learn (so no + k/(area,adj) risk, no r1120 poisoning class). Partial presence is fine: + the anchor resend (~2 s) re-tries and the missing pages heal late. + """ + pfd = os.open('/proc/%d/pagemap' % pid, os.O_RDONLY) + try: + sub = {} + vpg = ev & ~0xFFF + end = vpg + ((off + ln + 0xFFF) // 0x1000) * 0x1000 + scan_range(pfd, vpg, end, sub) + if not sub: + dbg('strict-pagemap: 0x%x no present page (%u pages)' + % (vpg, (end - vpg) >> 12)) + return False + for v, p in sorted(sub.items()): + out.append('0x%x,0x%x,1\n' % (v, p)) + dbg('strict-pagemap: 0x%x -> %u/%u pages pinned' + % (vpg, len(sub), (end - vpg) >> 12)) + return True + finally: + os.close(pfd) + + def scan_vmas(fd, pid, out): """Present pages of the client's high-VA nvidia//dev/zero mappings. @@ -1544,6 +2293,18 @@ def scan_vmas(fd, pid, out): parts = line.split() if len(parts) < 5: continue + # Skip mappings with no read permission: a CUDA device buffer + # is a PROT_NONE reservation (e.g. `---p 00000000 00:00 0`), + # potentially hundreds of MB. Its pages are never present in + # the pagemap, so scanning it produces no pairs - but it + # consumed the VMA_MAX_PAGES budget and the scan never reached + # the client's present staging buffers that sit ABOVE the + # device reservation (observed: a client whose device range + # started below its staging buffer got no pins at all and its + # H2D copies failed with rc=-2 forever). + perms = parts[1] if len(parts) >= 2 else '' + if 'r' not in perms: + continue # Anonymous mappings (no path field) are the client's UVM sysmem # buffers when the driver maps them without a /dev/zero backing: # the second client's cudaMalloc'd data buffers show up exactly @@ -1654,7 +2415,8 @@ def main(): # deinit), so the set deliberately persists across client switches. pool_pages = {} pool_reemit_next = 0.0 - _want_state = {"mtime": None, "done": set()} + _want_state = {"mtime": None, "done": set(), "strict_pending": {}, + "strict_try": {}} loop = 0 while True: pid = cdt_pid() @@ -1760,12 +2522,160 @@ def main(): st_d = os.stat("/tmp/guest_want_vas.txt") if st_d.st_mtime != _want_state.get("mtime"): with open("/tmp/guest_want_vas.txt") as fw: - wvas = [int(t, 16) for t in fw.read().split() - if t.startswith("0x")] + # r582: tolerate malformed historical lines. The + # r581 boot's scanner died here: a transport-escaped + # append merged VAs into one token containing + # literal backslash-n ('0x20400f000\\n0x121000000\\n + # ...') and int(t, 16) raised ValueError out of + # main(), stopping ALL new pins (the observed + # pool-pin latency). Extract every 0x token instead; + # a bad token is skipped, never fatal. + text = fw.read() _want_state["mtime"] = st_d.st_mtime - for wv in wvas[-64:]: + # SESSION 153-CONTINUED-64: strict demand anchors first. + # A line `S 0x<entry_va> 0x<off> <len>` (written by + # pin_gr_stream from the service's anchored debugfs + # demand record) asks for the exact [entry_va, +len) + # read window to be validated with the host's own full + # stream parse before a pin is injected. Loose lines + # (plain `0x<va>`) keep the content/pagemap paths, but + # a GL family with a pending anchor is deferred so its + # learn can only come from the strict validation. + strict = [] + loose_tokens = [] + for line in text.splitlines(): + ms = re.match( + r"\s*S\s+0x([0-9a-fA-F]+)\s+0x([0-9a-fA-F]+)" + r"\s+(\d+)\s*$", line) + if ms: + try: + strict.append((int(ms.group(1), 16), + int(ms.group(2), 16), + int(ms.group(3)))) + except ValueError: + pass + continue + loose_tokens += re.findall(r"0x[0-9a-fA-F]+", line) + now_d = time.time() + for epage in list(_want_state["strict_pending"]): + if now_d - _want_state["strict_pending"][epage] > 300.0: + del _want_state["strict_pending"][epage] + for (ev, eoff, eln) in reversed(strict): + epage = ev & ~0xFFF + if epage in _want_state["done"]: + continue + # SESSION 153-CONTINUED-65: throttle anchor + # re-resolution. The full-offset sweep can read + # thousands of windows (tens of seconds of CPU); + # the demand file re-sends the anchor every ~2 s, + # so without this throttle the scanner starves the + # client (both measured at ~91% CPU at r1125). One + # attempt per anchor per window is enough - content + # that lands late is picked up on the next window. + lt = _want_state["strict_try"].get(epage, 0.0) + if now_d - lt < ANCHOR_RETRY_S: + continue + _want_state["strict_try"][epage] = now_d + got = [] + try: + ok = resolve_va_strict(kc, ev, eoff, eln, got, + pid) + except Exception: + ok = False + got = [] + if not ok and eoff: + # The demand's off is the entry va's in-page + # offset; the push content can be page-aligned + # instead (r1123 vs r1125 parse behaviour). A + # second try at page alignment is cheap next to + # the completed sweep and the same pins result. + got2 = [] + try: + ok = resolve_va_strict(kc, ev & ~0xFFF, 0, + eln, got2, pid) + except Exception: + ok = False + if ok: + got = got2 + if not ok and pid and eln: + # SESSION 153-r1397: a failed content parse must + # not make the anchor's pages (and, via + # pend_bases, every loose demand VA of its + # family) unpinnable. Fall back to the direct + # client-pagemap window pin - the VA is the + # host's own entry VA, so a present page is by + # definition its translation (see + # _pagemap_window_pin). This un-deadlocks the + # family: on success the anchor counts as + # resolved and the deferral lifts; on failure + # the 'strict-pagemap:' line says the pages are + # not client-pagemap-present (kernel-resident). + got2 = [] + try: + ok = _pagemap_window_pin(pid, ev, eoff, eln, + got2) + except Exception: + ok = False + if ok: + got = got2 + if ok and got: + lines.extend(got) + # SESSION 153-r1397b: keep re-emitting this + # content-validated window so a page lost in + # burst delivery heals without waiting for a + # new resolution (see _strict_reemit). + _strict_reemit[epage] = [list(got), + time.time() + 4.0] + for g in got: + try: + _want_state["done"].add( + int(g.split(",", 1)[0], 16)) + except ValueError: + pass + _want_state["done"].add(epage) + _want_state["strict_pending"].pop(epage, None) + else: + _want_state["strict_pending"].setdefault( + epage, now_d) + _rr_now = time.time() + for _epg in list(_strict_reemit): + _pairs, _nxt = _strict_reemit[_epg] + if _rr_now >= _nxt: + lines.extend(_pairs) + _strict_reemit[_epg][1] = _rr_now + 4.0 + # Loose VAs in a family whose strict anchor is still + # pending are deferred to the strict path only. + pend_bases = set() + for epage in _want_state["strict_pending"]: + for lo, hi in PB_GL_FAMILIES: + if lo <= epage < hi: + pend_bases.add((lo, hi)) + # SESSION 153-CONTINUED-55: process the NEWEST VAs first, + # not the deduped head. The file is append-only and grows + # to hundreds of lines (every stalled-VA row is appended + # each demand cycle, ~48/4 s under load), so both the + # literal tail (`wvas[-64:]`) and a first-occurrence + # dedupe sorted older pending VAs out of the window (the + # 16-page GL setup-push demand fell out between its + # appends and the next mtime change). Walk the raw tail + # backwards, newest instruction first, skipping resolved + # VAs, up to a bounded batch. + wvas = [] + for t in reversed(loose_tokens): + try: + v = int(t, 16) + except ValueError: + continue + if v not in _want_state["done"] and v not in wvas: + wvas.append(v) + if len(wvas) >= 256: + break + for wv in wvas: if wv in _want_state["done"]: continue + if any(lo <= (wv & ~0xFFF) < hi + for lo, hi in pend_bases): + continue got = [] if POOL_SEM_LO <= (wv & ~0xFFF) < POOL_SEM_HI: # m165 (D-F1a): pool/waiter VAs have no method @@ -1794,6 +2704,32 @@ def main(): except Exception: ok = False got = [] + # r586: a demanded VA outside the CE + # pushbuffer window (the GRFILL staging pages, + # high client VAs) is a plain client mapping + # page: resolve it through the client's own + # pagemap. Without this the demand for those + # VAs was a silent no-op (resolve_va_by_content + # rejects anything outside its windows) and the + # fills kept dropping them. + if not ok and not got and pid: + try: + pfd = os.open('/proc/%d/pagemap' % pid, + os.O_RDONLY) + try: + sub = {} + vpg = wv & ~0xFFF + scan_range(pfd, vpg, vpg + 0x1000, + sub) + finally: + os.close(pfd) + if sub: + for v, p in sub.items(): + got.append('0x%x,0x%x,1\n' + % (v, p)) + ok = True + except OSError: + pass if ok and got: lines.extend(got) _want_state["done"].add(wv) diff --git a/tools/guest-diag/gzf2_video.py b/tools/guest-diag/gzf2_video.py new file mode 100644 index 0000000..aa88aa0 --- /dev/null +++ b/tools/guest-diag/gzf2_video.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# gzf2_video.py - turn a GZCAPV01/GZF2 guest capture into a constant-rate mp4. +# +# Reads the append-only frame stream written by sdl_cap.c (LD_PRELOAD +# SDL_GL_SwapWindow shim) or xgrab.c, resamples by CLOCK_MONOTONIC timestamps +# (nearest frame), streams raw pixels straight into ffmpeg (no PPM +# intermediates), and validates the encoded file with ffprobe. +# +# Format: [0..8) = "GZCAPV01", zero pad to 4096; then records: +# 4B "GZF2" | u32 w | u32 h | u32 stride | u32 fmt | u64 ts_ns | u64 seq | +# stride*h bytes, top-down. fmt 32 = RGBA8 (GL swap path), 33 = BGRA8 +# (SDL_RenderReadPixels ARGB8888 / X surface). A trailing +# "GZF2" + u64 0xffffffffffffffff marks a clean stop. +# +# Usage: +# gzf2_video.py CAPTURE OUT.mp4 [--fps 60] [--skip 5] [--duration 20] +# [--encoder libx264|nvenc] [--crf 18] +# [--pix-fmt rgba|bgra] [--info-only] +import argparse +import bisect +import hashlib +import json +import os +import struct +import subprocess +import sys + +MAGIC = b"GZCAPV01" +SENTINEL = b"\xff" * 8 + + +def die(msg): + print(f"GZF2-ERROR {msg}", file=sys.stderr) + sys.exit(2) + + +def index_capture(path): + size = os.path.getsize(path) + recs, clean, corrupt = [], False, 0 + with open(path, "rb") as f: + hdr = f.read(4096) + if len(hdr) < 4096 or hdr[:8] != MAGIC: + die(f"{path}: not a GZCAPV01 file") + off = 4096 + while off + 12 <= size: + f.seek(off) + head = f.read(36) + if len(head) < 12 or head[:4] != b"GZF2": + break + if head[4:12] == SENTINEL: + clean = True + break + if len(head) < 36: + break + w, h, stride, fmt = struct.unpack_from("<IIII", head, 4) + ts, seq = struct.unpack_from("<QQ", head, 20) + if not (0 < w <= 8192 and 0 < h <= 8192 + and 0 < stride <= 4 * 8192) or seq == 0xFFFFFFFFFFFFFFFF: + corrupt += 1 + break + n = stride * h + if off + 36 + n > size: + corrupt += 1 + break + f.seek(off + 36) + px = f.read(n) + if len(px) != n: + corrupt += 1 + break + recs.append({ + "off": off + 36, "ts": ts, "w": w, "h": h, + "stride": stride, "fmt": fmt, "size": n, + "md5": hashlib.md5(px).hexdigest(), + }) + off += 36 + n + if not recs: + die(f"{path}: header only (no frames captured)") + return recs, clean, corrupt + + +def sampled_black(px): + s = px[:65536] + return sum(s[0::4]) + sum(s[1::4]) + sum(s[2::4]) == 0 + + +def pick_frames(recs, fps, skip_s, duration_s): + t0 = recs[0]["ts"] + int(skip_s * 1e9) + cand = [r for r in recs if r["ts"] >= t0] or recs + ts = [r["ts"] for r in cand] + span = (ts[-1] - ts[0]) / 1e9 + n = int(duration_s * fps) if duration_s else max(1, int(span * fps)) + picks, held = [], 0 + for i in range(n): + want = ts[0] + int(i * 1e9 / fps) + j = bisect.bisect_left(ts, want) + if j == 0: + k = 0 + elif j >= len(cand): + k, held = len(cand) - 1, held + 1 + else: + k = j if ts[j] - want < want - ts[j - 1] else j - 1 + picks.append(cand[k]) + return picks, span, held + + +def encoder_args(name, crf): + if name == "nvenc": + return ["-c:v", "h264_nvenc", "-preset", "p6", "-rc", "vbr", + "-cq", str(crf), "-b:v", "12M"] + return ["-c:v", "libx264", "-preset", "slow", "-crf", str(crf)] + + +def encode(cap, picks, args, encoder): + first = picks[0] + w, h, fmt = first["w"], first["h"], args.pix_fmt + if any(p["w"] != w or p["h"] != h for p in picks): + die("geometry changes inside the capture; resize before recording") + if args.pix_fmt == "auto": + fmt = "bgra" if first["fmt"] == 33 else "rgba" + cmd = ["ffmpeg", "-hide_banner", "-v", "error", "-y", + "-f", "rawvideo", "-pixel_format", fmt, + "-video_size", f"{w}x{h}", "-framerate", str(args.fps), + "-i", "pipe:0", "-an", *encoder_args(encoder, args.crf), + "-frames:v", str(len(picks)), "-pix_fmt", "yuv420p", + "-movflags", "+faststart", args.out] + print(f"GZF2-ENCODE {encoder} {w}x{h} {fmt} {len(picks)}f @ {args.fps}fps") + proc = subprocess.Popen(cmd, stdin=subprocess.PIPE) + last, last_px = None, None + try: + with open(cap, "rb") as f: + for p in picks: + if p["off"] != last: + f.seek(p["off"]) + last_px = f.read(p["size"]) + if len(last_px) != p["size"]: + raise IOError(f"short read at {p['off']}") + last = p["off"] + if p["stride"] == w * 4: + proc.stdin.write(last_px) + else: + row = w * 4 + proc.stdin.write(b"".join( + last_px[y * p["stride"]:y * p["stride"] + row] + for y in range(h))) + except (BrokenPipeError, IOError) as e: + print(f"GZF2-ENCODE write failed: {e}", file=sys.stderr) + finally: + if proc.stdin: + try: + proc.stdin.close() + except BrokenPipeError: + pass + return proc.wait() + + +def ffprobe(path): + r = subprocess.run( + ["ffprobe", "-v", "error", "-count_frames", "-select_streams", "v:0", + "-show_entries", + "stream=codec_name,width,height,r_frame_rate,nb_read_frames", + "-show_entries", "format=duration,size", + "-of", "json", path], capture_output=True, text=True) + if r.returncode: + die(f"ffprobe failed: {r.stderr.strip()}") + return json.loads(r.stdout) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("capture") + ap.add_argument("out") + ap.add_argument("--fps", type=int, default=60) + ap.add_argument("--skip", type=float, default=0.0, + help="seconds of menu/load to drop from the start") + ap.add_argument("--duration", type=float, default=0.0, + help="output seconds (0 = whole remaining span)") + ap.add_argument("--encoder", choices=["libx264", "nvenc"], + default="libx264") + ap.add_argument("--crf", type=int, default=18) + ap.add_argument("--pix-fmt", choices=["auto", "rgba", "bgra"], + default="auto", help="auto uses record fmt (33=BGRA)") + ap.add_argument("--info-only", action="store_true") + args = ap.parse_args() + + recs, clean, corrupt = index_capture(args.capture) + uniq = len({r["md5"] for r in recs}) + span = (recs[-1]["ts"] - recs[0]["ts"]) / 1e9 + rate = len(recs) / span if span else 0.0 + blacks = 0 + with open(args.capture, "rb") as f: + for i, r in enumerate(recs): + if i % 60 == 0: + f.seek(r["off"]) + blacks += sampled_black(f.read(min(r["size"], 65536))) + print(f"GZF2-CAPTURE frames={len(recs)} span={span:.2f}s rate={rate:.1f}/s " + f"unique={uniq} ({100.0 * uniq / len(recs):.1f}%) " + f"black_sampled={blacks} clean_end={clean} corrupt={corrupt} " + f"size={os.path.getsize(args.capture)}B " + f"geom={recs[0]['w']}x{recs[0]['h']} fmt={recs[0]['fmt']}") + if args.info_only: + return + picks, span, held = pick_frames(recs, args.fps, args.skip, + args.duration or None) + md5_by_off = {r["off"]: r["md5"] for r in recs} + pu = len({md5_by_off[p["off"]] for p in picks}) + print(f"GZF2-RESAMPLE out={len(picks)}f span={span:.2f}s " + f"unique={pu}/{len(picks)} ({100.0 * pu / len(picks):.1f}%) " + f"held_last={held}") + if pu < len(picks) // 10: + print("GZF2-WARN mostly static/black output", file=sys.stderr) + + rc = encode(args.capture, picks, args, args.encoder) + if rc != 0 and args.encoder == "nvenc": + print("GZF2-WARN nvenc failed; retrying libx264", file=sys.stderr) + rc = encode(args.capture, picks, args, "libx264") + if rc != 0: + die(f"ffmpeg exited {rc}") + info = ffprobe(args.out) + st = info["streams"][0] + print("GZF2-OUTPUT " + " ".join( + f"{k}={st.get(k)}" for k in + ("codec_name", "width", "height", "r_frame_rate", "nb_read_frames"))) + print(f"GZF2-OUTPUT duration={info['format'].get('duration')} " + f"size={info['format'].get('size')}") + if int(st.get("nb_read_frames", 0)) != len(picks): + die("encoded frame count mismatch") + print("GZF2-OK", args.out) + + +if __name__ == "__main__": + main()
\ No newline at end of file diff --git a/tools/guest-diag/launch_qemu_rootport.sh b/tools/guest-diag/launch_qemu_rootport.sh index 444bdfc..d38321b 100755 --- a/tools/guest-diag/launch_qemu_rootport.sh +++ b/tools/guest-diag/launch_qemu_rootport.sh @@ -1,4 +1,4 @@ #!/bin/bash # SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2026 Open-IOV contributors -exec /bin/qemu-system-x86_64 -D /home/rmcore/.local/libvf.io/logs/qemu/e726ba0b-204f-4ec9-91da-983ed7f41d06-session.txt -no-hpet -mem-prealloc -qmp unix:/tmp/qemu-qmp.sock,server,nowait -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -serial file:/tmp/vgpu-guest-serial.log -gdb tcp::1234 -display none -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=128M,share=yes -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=2M,share=yes -object memory-backend-file,id=pc.ram,size=4096M,mem-path=/dev/shm/vgpu-guest-ram,share=yes,prealloc=on -uuid e726ba0b-204f-4ec9-91da-983ed7f41d06 -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on -rtc clock=host,base=localtime -m 4096 -smp cores=4,threads=1,sockets=1 -hda /home/rmcore/.local/libvf.io/kernel/linux.qcow2 --enable-kvm -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/<MDEV-UUID>,display=off -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 -device vhost-vsock-pci,guest-cid=50718 -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/main.sock,server,nowait -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/master.sock,server,nowait +exec /bin/qemu-system-x86_64 -D /home/rmcore/.local/libvf.io/logs/qemu/e726ba0b-204f-4ec9-91da-983ed7f41d06-session.txt -no-hpet -mem-prealloc -qmp unix:/tmp/qemu-qmp.sock,server,nowait -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -serial file:/tmp/vgpu-guest-serial.log -gdb tcp::1234 -display none -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=128M,share=yes -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=2M,share=yes -object memory-backend-file,id=pc.ram,size=8192M,mem-path=/dev/shm/vgpu-guest-ram,share=yes,prealloc=on -uuid e726ba0b-204f-4ec9-91da-983ed7f41d06 -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on -rtc clock=host,base=localtime -m 8192 -smp cores=4,threads=1,sockets=1 -hda /home/rmcore/.local/libvf.io/kernel/linux.qcow2 --enable-kvm -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/<MDEV-UUID>,display=off -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 -device vhost-vsock-pci,guest-cid=50718 -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/main.sock,server,nowait -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/master.sock,server,nowait diff --git a/tools/guest-diag/nvkmstrace.c b/tools/guest-diag/nvkmstrace.c new file mode 100644 index 0000000..ca72e46 --- /dev/null +++ b/tools/guest-diag/nvkmstrace.c @@ -0,0 +1,62 @@ +// nvkmstrace.c - LD_PRELOAD shim: log NVKMS_IOCTL_IOWR commands +#define _GNU_SOURCE +#include <stdio.h> +#include <stdarg.h> +#include <stdint.h> +#include <string.h> +#include <dlfcn.h> +#include <sys/ioctl.h> + +struct nvkms_ioctl_params { + uint32_t cmd; + uint32_t size; + uint64_t address; +}; + +#define NVKMS_REQ 0xC0106D00UL + +static int (*real_ioctl)(int, unsigned long, ...); + +int ioctl(int fd, unsigned long req, ...) +{ + va_list ap; + void *arg; + + va_start(ap, req); + arg = va_arg(ap, void *); + va_end(ap); + if (!real_ioctl) + real_ioctl = dlsym(RTLD_NEXT, "ioctl"); + if (req == NVKMS_REQ && arg) { + struct nvkms_ioctl_params *p = arg; + int rc; + + rc = real_ioctl(fd, req, arg); + fprintf(stderr, "NVKMS_IOCTL cmd=%u size=%u addr=%p rc=%d\n", + p->cmd, p->size, (void *)(uintptr_t)p->address, rc); + if (p->address && p->size) { + unsigned char *q = (unsigned char *)(uintptr_t)p->address; + unsigned n = p->size < 64 ? p->size : 64, i; + + fprintf(stderr, " data:"); + for (i = 0; i < n; i++) + fprintf(stderr, " %02x", q[i]); + fprintf(stderr, "\n"); + if (p->size >= 700) { + /* reply.status candidates: request = 32 + 4 + + * 4*1 (NvBool=NvU8) + 16*(32+4) = 616 bytes */ + unsigned o; + + for (o = 600; o <= 700; o += 4) { + unsigned v; + memcpy(&v, q + o, 4); + if (v <= 16) + fprintf(stderr, " small@%u=%u\n", + o, v); + } + } + } + return rc; + } + return real_ioctl(fd, req, arg); +} diff --git a/tools/guest-diag/params-r1780.txt b/tools/guest-diag/params-r1780.txt new file mode 100644 index 0000000..70985c7 --- /dev/null +++ b/tools/guest-diag/params-r1780.txt @@ -0,0 +1,2 @@ +gsp_phys_gpu=1 gsp_phys_hazard_ack=1 gsp_phys_boot=1 gsp_phys_nocat=1 gsp_phys_fw=nvidia/tu102/gsp/gsp-570.144.bin gsp_phys_guest_mem=1 gsp_phys_gr_forward=1 cuda_bridge=0 gsp_phys_rmcall_engine=1 gsp_phys_rmcall_golden=1 gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F gsp_phys_rmcall_vas_shared=1 gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 +sema_dbg=1 sema_fake_rel=0 diff --git a/tools/guest-diag/pin_gr_stream.py b/tools/guest-diag/pin_gr_stream.py index 6ba51ee..eb48aa5 100644 --- a/tools/guest-diag/pin_gr_stream.py +++ b/tools/guest-diag/pin_gr_stream.py @@ -100,26 +100,70 @@ def _scanner_alive(): return "ALIVE" in (out or "") +def _demand_write_cmd(vas): + """Build a transport-proof append command for the guest demand file. + + r582: the old `printf '%s' '<lines with real LF>'` form was mangled by + the guest-agent transport under load - the newlines arrived as LITERAL + backslash-n text and merged VAs onto one line, and the guest scanner's + parser then crashed on `int('0x20400f000\\\\n0x121000000...', 16)` + (ValueError in main; r581 boot scanner log). The scanner's death stopped + ALL new pins while the host only re-asserted known pairs - the observed + pool-pin-latency (0x121000000 never arrived while other injections + flowed). One `echo <va> >> file` per VA carries no newlines and no + backslashes at all, so no transport layer can escape them.""" + return "; ".join("echo 0x%x >> /tmp/guest_want_vas.txt" % va + for va in vas) + + def _demand_send(vas): """Append VA hints to the guest scanner's demand file (best-effort). Same channel pin_gr_stream._demand_loop uses for host-dmesg stalls; the stall watchdog uses it to ask the scanner to re-resolve the fixed pre-capture VAs (counter page / UVM pool) by content/pagemap.""" - lines = "".join("0x%x\n" % va for va in vas) + cmd = _demand_write_cmd(vas) + if not cmd: + return try: - vexec("printf '%%s' '%s' >> /tmp/guest_want_vas.txt" % lines, 30) + vexec(cmd, 30) except Exception: pass -def _start_cmd(): +def _push_guest_script(): + """Push the scanner into the guest in argv-safe chunks. + + SESSION 153-CONTINUED-64: the one-shot `echo <b64> | base64 -d >` form + put the whole ~135 KB base64 of the (grown) scanner on vgpu_vm.py's + argv, and the host's MAX_ARG_STRLEN (128 KB) rejected the exec before + it ever reached the guest ('OSError: Argument list too long'), so no + scanner started and the boot streamed zero pins. Push 60 KB chunks and + print the guest-side md5 for verification. + """ with open(GUEST_SCRIPT) as f: b64 = subprocess.run(["base64", "-w0"], stdin=f, capture_output=True, text=True, - timeout=10).stdout - prefixes = os.environ.get("CLIENT_PREFIXES", "cdt,va,victory") + timeout=10).stdout.strip() + vexec("rm -f /tmp/gps.b64", 30) + for off in range(0, len(b64), 60000): + vexec("printf %%s '%s' >> /tmp/gps.b64" % b64[off:off + 60000], 30) + return vexec("base64 -d /tmp/gps.b64 > /tmp/guest_pin_stream.py && " + "rm -f /tmp/gps.b64 && md5sum /tmp/guest_pin_stream.py " + "&& echo PUSH_OK", 60) or "" + + +def _start_cmd(): + prefixes = os.environ.get("CLIENT_PREFIXES", + "cdt,va,victory,Xorg,X,glxgears,gzdoom") vma_every = os.environ.get("STREAM_VMA_EVERY", "1") + # r1410: the strict-anchor retry throttle paces the setup-push pin at + # +23.5 s (one 20 s ANCHOR_RETRY_S window) while the GL client's setup + # generation lives ~6-12 s, so the 60216-byte init push is consumed + # unreadable before its pins land. Thread the guest scanner's throttle + # so a cycle can shorten it (STREAM_ANCHOR_RETRY_S=2) without editing + # the guest script. + anchor_retry = os.environ.get("STREAM_ANCHOR_RETRY_S", "20") # Kill EVERY existing scanner, not just the pid-file one: a retry after # a partially-failed first attempt (pid file not written yet) used to # leave the first scanner alive, and multiple scanners each re-assert @@ -132,14 +176,14 @@ def _start_cmd(): # failed silently (r149). return ("P=$(printf '^python3 /tmp/guest_pin_%%s.py' stream); " "sudo pkill -f \"$P\" 2>/dev/null; sleep 1; " - "echo '%s' | base64 -d > /tmp/guest_pin_stream.py; " "sudo rm -f /tmp/guest_pin_stream.pid; " "sudo setsid nohup env CLIENT_PREFIXES='%s' " "STREAM_VMA_EVERY='%s' " + "STREAM_ANCHOR_RETRY_S='%s' " "python3 /tmp/guest_pin_stream.py %d %d %s %s %s " ">/tmp/guest_pin_stream.log 2>&1 & sleep 2; " "sudo cat /tmp/guest_pin_stream.pid 2>/dev/null" - % (b64, prefixes, vma_every, 2, PORT, LO, HI, PAMAX)) + % (prefixes, vma_every, anchor_retry, 2, PORT, LO, HI, PAMAX)) def install_and_start(): @@ -148,6 +192,14 @@ def install_and_start(): # a boot without the scanner streams no pins and the client then hangs. for attempt in range(5): _heartbeat() + try: + out_push = _push_guest_script() + if "PUSH_OK" not in (out_push or ""): + log("WARN: scanner push attempt %d incomplete: %s" + % (attempt + 1, (out_push or "")[-120:])) + except Exception as e: + log("WARN: scanner push attempt %d failed: %s" + % (attempt + 1, e)) out = vexec(_start_cmd(), 60) if any(ch.isdigit() for ch in out): return True @@ -201,35 +253,141 @@ def _demand_loop(): that was not yet written when first asked is picked up on a retry. This closes the stall class where the one-shot #sem kprobe event was lost and the chain resolver never learns the client. + + r908: the GR ring VA itself ('GR ring h=0x... ... (va=0x...)', the + path-B "RAM content scan found no candidate" and "not at fb" rows) is + harvested too - the ring page is a kernel allocation the userspace + pagemap cannot cover, so the guest's content resolver is the only + source, and the pump needs the ring readable to decode its entries. + Same bounded/deduped tracking (distinct VAs, 20 s resend, 300 s + expiry) as the stalled-VA class. """ track = {} # va -> [first_seen, last_sent] + # SESSION 153-CONTINUED-64: strict demand anchors (entry va -> [first, + # last_sent]) from the anchored debugfs lines. The guest validates the + # exact [entry_va, +len) read window with the host's full stream parse + # before pinning, so the setup-push pin can never land on a page whose + # bytes the host's read will reject. Resent fast (2 s) while the hold + # is up; the loose cadence (20 s) is too slow for the client's churn. + strict_track = {} + it = 0 while True: - time.sleep(3) - try: - r = subprocess.run(["sudo", "-n", "dmesg"], - capture_output=True, text=True, timeout=20) - except Exception: - continue + # SESSION 153-CONTINUED-57: the 3 s pass was the losing half of the + # r1096 race - the beef setup-hold's demanded pages resolved in the + # guest ~2-4 s after the arm, while the pump re-read the entry + # every ~110 ms and moved on. Poll fast (0.75 s) so a newly armed + # demand goes to the guest within a second; the expensive dmesg + # harvest stays on a slower cadence (every 4th pass) because the + # debugfs setup-demand set is the time-critical source. + time.sleep(0.75) + it += 1 + r = None + if it % 4 == 1: + try: + r = subprocess.run(["sudo", "-n", "dmesg"], + capture_output=True, text=True, timeout=20) + except Exception: + continue now = time.time() - for m in re.finditer(r"(?:stalled VA #\d+|pending release starved va)=(0x[0-9a-f]+)", - (r.stdout or "")[-200000:]): - va = int(m.group(1), 16) + for m in re.finditer( + r"(?:stalled VA #\d+[:=]\s*" + r"|pending release starved va=" + r"|GR ring h=0x[0-9a-f]+ rank=\d+[^\n]*?\(va=)" + r"(0x[0-9a-f]+)" + # r914: an unreadable GR push names its pushbuffer VA - + # the GL client's push page is a kernel allocation the + # userspace pagemap cannot cover, so the guest's content + # resolver is the only source (same class as the ring VA + # demand above). Harvest it so the guest resolves and + # injects the page. + r"|sema-pushdec: h=0x[0-9a-f]+ va=(0x[0-9a-f]+)" + r"[^\n]*unreadable=1" + # SESSION 153-CONTINUED-46: a big GL push whose trusted + # chain answered an all-zero page is the beef draw that + # never captures ("sema-pushzero: ... rescued=0"). The + # promote rescue inside the read may heal it; when it + # cannot, the guest content resolver is the last source - + # harvest the VA so it resolves and injects the real page. + r"|sema-pushzero: h=0x[0-9a-f]+ va=(0x[0-9a-f]+)", + (r.stdout or "")[-2000000:] if r else ""): + va = int(m.group(1) or m.group(2) or m.group(3), 16) track.setdefault(va, [now, 0.0]) + # SESSION 153-CONTINUED-51: the pushzero line now carries the + # push length; a zero-head WINDOW push spans 1-2 pages, and the + # demand of a single page leaves the capture zero - expand the + # range and demand every page so the guest pagemap resolves the + # whole push at once (bulk demand instead of one page per cycle). + for m in re.finditer( + r"sema-pushzero: h=0x[0-9a-f]+ va=(0x[0-9a-f]+) len=(\d+)", + (r.stdout or "")[-2000000:] if r else ""): + zva = int(m.group(1), 16) + zlen = int(m.group(2)) + for zoff in range(0, min(zlen, 0x8000), 0x1000): + track.setdefault(zva + zoff, [now, 0.0]) + # SESSION 153-CONTINUED-56: the armed setup-hold's demand VA set is + # exposed by the service as STATE (debugfs), not as a dmesg row, so + # it cannot race the ring rotation under vgsp-sema spam (r1090 lost + # the setup-push pages that way). Read it every pass and feed the + # same track/resend machinery: new VAs are sent this pass (last=0), + # existing ones keep the 20 s resend / 300 s expiry cadence. + try: + rr = subprocess.run( + ["sudo", "-n", "cat", + "/sys/kernel/debug/vgsp/sema_setup_demand"], + capture_output=True, text=True, timeout=10) + if rr.returncode == 0: + for line in (rr.stdout or "").splitlines(): + ma = re.match( + r"\s*0x([0-9a-f]+)\s+0x([0-9a-f]+)\s+(\d+)\s*$", + line) + if ma: + ev = int(ma.group(1), 16) + off = int(ma.group(2), 16) + ln = int(ma.group(3)) + if ev and 0 < ln <= 0x40000: + strict_track.setdefault(ev, [now, 0.0, + off, ln]) + continue + for m in re.finditer(r"0x[0-9a-f]+", line): + sva = int(m.group(0), 16) & ~0xfff + if sva: + track.setdefault(sva, [now, 0.0]) + except Exception: + pass for va in list(track): first, last = track[va] if now - first > 300.0: del track[va] + for ev in list(strict_track): + first, last, off, ln = strict_track[ev] + if now - first > 300.0: + del strict_track[ev] + cmds = [] todo = [va for va, (f, l) in track.items() if now - l > 20.0] - if not todo: + if todo: + todo = todo[:48] + cmds.append(_demand_write_cmd(todo)) + stodo = [ev for ev, (f, l, o, ln) in strict_track.items() + if now - l > 2.0] + if stodo: + stodo = stodo[:16] + cmds.append("; ".join( + "echo S 0x%x 0x%x %d >> /tmp/guest_want_vas.txt" + % (ev, strict_track[ev][2], strict_track[ev][3]) + for ev in stodo)) + if not cmds: continue - lines = "".join("0x%x\n" % va for va in todo) try: - vexec("printf '%%s' '%s' >> /tmp/guest_want_vas.txt" % lines, 30) + vexec("; ".join(cmds), 30) except Exception: continue for va in todo: track[va][1] = now - log("demand: sent %d stalled VA(s) to guest" % len(todo)) + for ev in stodo: + strict_track[ev][1] = now + log("demand: sent %d stalled VA(s)%s to guest" % + (len(todo), + " + %d strict anchor(s)" % len(stodo) if stodo else "")) _DEMAND_STARTED = False @@ -525,7 +683,19 @@ def _flush(runs, seen, full=False): # r145 stall: sema-probe "NO ENTRY" for the live push pages) and the # pool/tracker page (pending release starved, every client's first # seconds). - ce_win = 0x120000000 <= va < 0x122000000 + # + # SESSION 153-r1397b: the GL setup-push families get the same + # treatment. The guest re-emits a strict-validated 60 216 B window + # every ~4 s because a burst can be partially dropped under host + # load and the `done` latch otherwise never retries the lost pages + # (r1397: pages 0x400a000..0x400e000 never landed; every 60216 B + # entry read answered src=255 and the push was skipped). Without + # this exception the unchanged re-emits were dropped here. + gl_win = ((0x4000000 <= va < 0x4160000) or + (0x4200000 <= va < 0x4300000) or + (0xcc80000 <= va < 0xcc90000) or + (0x6f00000 <= va < 0x6f10000)) + ce_win = 0x120000000 <= va < 0x122000000 or gl_win if not full and not ce_win and seen.get(va) == pa and n == 1: continue todo[(va, pa, n)] = True diff --git a/tools/guest-diag/pin_supervisor.sh b/tools/guest-diag/pin_supervisor.sh index b87d5c1..72f588f 100755 --- a/tools/guest-diag/pin_supervisor.sh +++ b/tools/guest-diag/pin_supervisor.sh @@ -70,7 +70,14 @@ kill_tool() { start_pin() { if tool_alive "$PIN"; then return 0; fi touch "$PIN_HB" 2>/dev/null || true - ( setsid nohup env SEMFIX_COUNTER_RETRY=10 \ + # CLIENT_PREFIXES threads through pin_gr_stream.py into the guest + # scanner launch (its _start_cmd). GL clients (glxgears, gzdoom) and + # the X server must be covered or the GL client's ring/mappings are + # never pinned (SESSION 149: the GR ring scan then finds no candidate + # for h=0xbfef0002, va=0x1203cfff0). + ( setsid nohup env \ + CLIENT_PREFIXES="${CLIENT_PREFIXES:-cdt,va,victory,Xorg,X,glxgears,gzdoom}" \ + SEMFIX_COUNTER_RETRY=10 \ python3 "$PIN" >> "$PIN_LOG" 2>&1 & ) pin_started=$(date +%s) log "START pin_gr_stream" diff --git a/tools/guest-diag/pool_page.py b/tools/guest-diag/pool_page.py new file mode 100755 index 0000000..a9f7b15 --- /dev/null +++ b/tools/guest-diag/pool_page.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +# pool_page.py <pa-hex> - dump the UVM semaphore-pool page's u32 slots via +# /proc/kcore (guest root). The pool VA 0x121000000 maps to the PA the +# scanner+host pinned; nonzero slots are the applied release values. Compare +# against the tracker's expected payload (tracker_walk.py) and the driver's +# uvsv wait value to find write-target divergence / starvation. +import sys +import struct + +pa = int(sys.argv[1], 16) if len(sys.argv) > 1 else 0x107fd8000 +sys.argv = ["pool_page"] +sys.path.insert(0, "/tmp") +import guest_pin_stream as g # noqa: E402 + +kc = g.Kcore() +if not kc.load(): + print("KCORE LOAD FAILED") + sys.exit(1) + +nz = [] +for off in range(0, 0x1000, 4): + d = kc.read(kc.page_offset_base + pa + off, 4) + v = struct.unpack("<I", d)[0] if d else 0 + if v: + nz.append((off, v)) +print("pa=0x%x nonzero slots=%d" % (pa, len(nz))) +for off, v in nz[:64]: + print("off=0x%x val=%u" % (off, v)) diff --git a/tools/guest-diag/qga_fetch.py b/tools/guest-diag/qga_fetch.py new file mode 100755 index 0000000..2fd53fc --- /dev/null +++ b/tools/guest-diag/qga_fetch.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# Fetch a file from the guest via the QEMU guest agent protocol +# (/tmp/qemu-guest-agent.sock, the virtio-serial QGA chardev - the same +# socket vgpu_vm.py exec uses). Usage: qga_fetch.py <guest_path> <host_path> [chunk] +# +# Robustness (r1490): the r1489 evidence capture fetched nothing because the +# guest-side frame was still being written / the agent answered late. Retry +# the open a few times, tolerate error replies with a clear diagnostic, use a +# long socket timeout, and write to a temporary file so a partial transfer is +# never mistaken for a complete capture. +import base64 +import json +import os +import socket +import sys +import time + +if len(sys.argv) < 3: + print("usage: qga_fetch.py <guest_path> <host_path> [chunk]", file=sys.stderr) + sys.exit(2) + +GUEST = sys.argv[1] +HOST = sys.argv[2] +CHUNK = int(sys.argv[3]) if len(sys.argv) > 3 else 1 << 20 +SOCK = "/tmp/qemu-guest-agent.sock" +TRIES = 6 +TMP = HOST + ".part" + + +def recv_line(s): + buf = b"" + while b"\n" not in buf: + chunk = s.recv(65536) + if not chunk: + break + buf += chunk + return buf.split(b"\n", 1)[0] + + +def cmd(s, obj): + s.sendall((json.dumps(obj) + "\r\n").encode()) + line = recv_line(s) + return json.loads(line) + + +def open_guest(): + last = None + for i in range(TRIES): + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.settimeout(120.0) + s.connect(SOCK) + r = cmd(s, {"execute": "guest-file-open", + "arguments": {"path": GUEST, "mode": "r"}}) + if "return" in r: + return s, r["return"], None + last = f"guest-file-open error: {r.get('error')}" + except Exception as e: # noqa: BLE001 - diagnostic surface + last = f"guest-file-open exception: {e}" + try: + s.close() + except Exception: + pass + time.sleep(1.5) + if i + 1 < TRIES: + print(f"retry {i + 1}/{TRIES}: {last}", file=sys.stderr) + return None, None, last + + +s, h, err = open_guest() +if s is None: + print(f"FETCH-FAIL {GUEST} -> {HOST}: {err}", file=sys.stderr) + sys.exit(1) + +print(f"handle={h}") +total = 0 +try: + with open(TMP, "wb") as out: + while True: + r = cmd(s, {"execute": "guest-file-read", + "arguments": {"handle": h, "count": CHUNK}}) + if "return" not in r: + raise RuntimeError(f"guest-file-read error: {r.get('error')}") + d = r["return"] + buf = base64.b64decode(d.get("buf-b64", "")) + if buf: + out.write(buf) + total += len(buf) + print(f"\r{total}", end="", flush=True) + if d.get("eof"): + break + cmd(s, {"execute": "guest-file-close", "arguments": {"handle": h}}) +except Exception as e: # noqa: BLE001 + print(f"\nFETCH-FAIL {GUEST} -> {HOST}: {e}", file=sys.stderr) + try: + if os.path.exists(TMP): + os.unlink(TMP) + except OSError: + pass + sys.exit(1) +finally: + try: + s.close() + except Exception: + pass + +os.replace(TMP, HOST) +print(f"\nFETCHED {total} bytes -> {HOST}") +sys.exit(0) diff --git a/tools/guest-diag/qga_snip.py b/tools/guest-diag/qga_snip.py new file mode 100644 index 0000000..cd3b992 --- /dev/null +++ b/tools/guest-diag/qga_snip.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +# Fetch a byte range from a guest file via the QEMU guest agent. +# Usage: qga_snip.py <guest_path> <offset> <length> <host_out> +import base64 +import json +import socket +import sys + +GUEST, OFF, LEN, HOST = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4] +import os +SOCK = os.environ.get("QGA_SOCK", "/tmp/qemu-guest-agent.sock") + + +def recv_line(s): + buf = b"" + while b"\n" not in buf: + c = s.recv(65536) + if not c: + break + buf += c + return buf.split(b"\n", 1)[0] + + +def cmd(s, obj): + s.sendall((json.dumps(obj) + "\r\n").encode()) + return json.loads(recv_line(s)) + + +s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) +s.settimeout(30.0) +s.connect(SOCK) +h = cmd(s, {"execute": "guest-file-open", + "arguments": {"path": GUEST, "mode": "r"}})["return"] +cmd(s, {"execute": "guest-file-seek", + "arguments": {"handle": h, "offset": OFF, "whence": 0}}) +r = cmd(s, {"execute": "guest-file-read", + "arguments": {"handle": h, "count": LEN}}) +data = base64.b64decode(r["return"].get("buf-b64", "")) +open(HOST, "wb").write(data) +cmd(s, {"execute": "guest-file-close", "arguments": {"handle": h}}) +print(f"GOT {len(data)} bytes @ {OFF} -> {HOST}") diff --git a/tools/guest-diag/ram_content_pin.py b/tools/guest-diag/ram_content_pin.py new file mode 100755 index 0000000..d521d91 --- /dev/null +++ b/tools/guest-diag/ram_content_pin.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Host-side CONTENT-based resolver for demanded GR setup-push windows. + +SESSION 153-CONTINUED-66. The client's GL setup pushes (and its 0x3200000 / +0x6f00000 macro-upload windows) live in allocations that have NO kernel vmap +and NO client CPU mapping on the failing boots: the guest scanner's area+idx +identity sweep and its client-mem sweep both fail (`strict unknown-family +client-mem failed`, `strict parse failed (N B, M windows tried)`), so the +armed GR setup hold never lands its pin and the MME prelude stays 0. + +The bytes ARE in guest RAM - a RAM-wide pattern scan finds the MME/SA +prelude runs (`0xa0100045` = INC 0x114 x16 args, `0x20020047` = INC 0x11c x2 +args). This tool therefore resolves a demanded window from CONTENT instead of +identity: + + 1. read the demand set the service publishes (debugfs sema_setup_demand: + "0xVA 0xOFF LEN" per line); + 2. locate every physical page in the guest RAM image that carries an + MME/SA prelude run signature; + 3. for each candidate physical page P and each window page index j, read + the EXACT window bytes the host would read - [P - j*0x1000 + OFF, +LEN) + - straight out of the RAM image and run the host's own method-stream + acceptance on them; + 4. inject the validated page run into the service through the + sema_map_add module parameter (0xVA,0xPA,LEN,1), so the host read + resolves the demanded window from real content. + +Nothing is injected unless the host's parser accepts the exact bytes, so a +wrong candidate cannot mis-resolve a demand VA (the r1120 lesson). + +Usage: ram_content_pin.py [ram-image] [seconds] +""" +import importlib.util +import mmap +import os +import re +import subprocess +import sys +import time + +import numpy as np + +RAM = sys.argv[1] if len(sys.argv) > 1 else '/dev/shm/vgpu-guest-ram' +SECS = float(sys.argv[2]) if len(sys.argv) > 2 else 120.0 +DEMAND = '/sys/kernel/debug/vgsp/sema_setup_demand' +MAPADD = '/sys/module/vgsp/parameters/sema_map_add' +GPS = ('/home/rmcore/Git/integdev_gpu_drv/open-iov/tools/guest-diag/' + 'guest_pin_stream.py') +SIGS = (0xa0100045, 0x20020047) # INC 0x114 x16, INC 0x11c x2 +PAGE = 0x1000 +MAXHITS = 4096 + + +def log(msg): + print('[%s] %s' % (time.strftime('%H:%M:%S'), msg), flush=True) + + +def load_accept(): + spec = importlib.util.spec_from_file_location('gps', GPS) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod._window_accept + + +def read_demand(): + out = [] + try: + with open(DEMAND) as f: + for ln in f: + m = re.match(r'0x([0-9a-fA-F]+)\s+0x([0-9a-fA-F]+)\s+(\d+)', + ln.strip()) + if not m: + continue + out.append((int(m.group(1), 16), int(m.group(2), 16), + int(m.group(3)))) + except OSError: + pass + return out + + +def inject(va_page, pa_page, ln): + cmd = "printf '0x%x,0x%x,0x%x,1' > %s" % (va_page, pa_page, ln, MAPADD) + try: + r = subprocess.run(['sudo', '-n', 'sh', '-c', cmd], + capture_output=True, text=True, timeout=10) + except subprocess.TimeoutExpired: + return False + return r.returncode == 0 + + +def main(): + accept = load_accept() + f = open(RAM, 'rb') + mm = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) + size = len(mm) + a32 = np.frombuffer(mm, dtype=np.uint32) + hits = [] + for sig in SIGS: + idx = np.flatnonzero(a32 == sig) + log('sig 0x%08x: %d hit(s)' % (sig, idx.size)) + for i in idx[:MAXHITS]: + hits.append(int(i) * 4) + del a32 + hits = sorted(set(h >> 12 for h in hits)) + log('candidate physical pages: %d (ram image %d MB)' + % (len(hits), size >> 20)) + if not hits: + return 1 + seen = set() + t0 = time.time() + while time.time() - t0 < SECS: + dem = read_demand() + for (va, off, ln) in dem: + key = (va & ~(PAGE - 1), ln) + if key in seen: + continue + if ln <= 0 or ln > 0x40000: + continue + need = (off + ln + PAGE - 1) // PAGE + if need > 64: + continue + va_page = va & ~(PAGE - 1) + found = None + for P in hits: + for j in range(need): + P0 = P - j * PAGE + if P0 < 0 or P0 + off + ln > size: + continue + buf = mm[P0 + off:P0 + off + ln] + try: + ok, mode = accept(buf, ln) + except Exception: + ok = False + if ok: + found = (P0, mode) + break + if found: + break + if not found: + log('demand va=0x%x off=0x%x len=%d: no content candidate' + % (va, off, ln)) + seen.add(key) + continue + P0, mode = found + if inject(va_page, P0, need * PAGE): + log('demand va=0x%x off=0x%x len=%d -> pa=0x%x (%s parse, ' + 'pages=%d) INJECTED' + % (va, off, ln, P0, mode, need)) + seen.add(key) + else: + log('demand va=0x%x: injection refused by the module param' + % va) + seen.add(key) + time.sleep(1.0) + log('done') + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/guest-diag/read_proc_va.py b/tools/guest-diag/read_proc_va.py new file mode 100644 index 0000000..c035840 --- /dev/null +++ b/tools/guest-diag/read_proc_va.py @@ -0,0 +1,16 @@ +import os, sys +pat = sys.argv[1] +vas = [int(a, 16) for a in sys.argv[2:]] +pids = os.popen("pgrep -x " + pat).read().split() +if not pids: + print("NO-PID"); sys.exit(0) +pid = int(pids[0]) +print("pid", pid) +fd = os.open("/proc/%d/mem" % pid, os.O_RDONLY) +for a in vas: + try: + os.lseek(fd, a, 0) + d = os.read(fd, 4) + print(hex(a), d.hex()) + except OSError as e: + print(hex(a), "ERR", e) diff --git a/tools/guest-diag/redboot-triage.sh b/tools/guest-diag/redboot-triage.sh new file mode 100755 index 0000000..041e9a6 --- /dev/null +++ b/tools/guest-diag/redboot-triage.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# redboot-triage.sh <live-log> [base-va] - pull the r613b/r613d wedge-class +# evidence from a boot's dmesg capture in one pass. +# usage: redboot-triage.sh /tmp/opencode/rNNN-live.log [0x7fc716c00000] +set -u +LOG="${1:?usage: redboot-triage.sh <log> [base-va]}" +VA="${2:-}" +echo "=== $LOG ===" +echo "--- sema-shadow-dst (r614 witness; the red-boot entry point)" +grep -h "sema-shadow-dst" "$LOG" | head -12 +echo "--- GRFILL-DROP + held-for-retry + degenerate refusal" +grep -hE "GRFILL-DROP|launch held for a bounded retry|GR submit refused|GR forward failed|GR-STALL-fwd-fail|GR-COMPLETE-SW" "$LOG" | tail -24 +echo "--- arg-miss walk lists (r614: full, untruncated)" +grep -h "arg-miss" "$LOG" | tail -6 +echo "--- cand[0] arg base probes (first resolution of each client)" +grep -h "cand\[0\] arg base" "$LOG" | tail -12 +echo "--- EPOST drop signature (status-ring starvation rounds)" +grep -h "no postable event" "$LOG" | tail -6 +if [ -n "$VA" ]; then + echo "--- all lines for base $VA" + grep -h "$VA" "$LOG" | grep -vE "ldatrace|dmatrace" | head -30 +fi +echo "=== done ==="
\ No newline at end of file diff --git a/tools/guest-diag/scrub_pin_qmp.py b/tools/guest-diag/scrub_pin_qmp.py new file mode 100644 index 0000000..653bb5d --- /dev/null +++ b/tools/guest-diag/scrub_pin_qmp.py @@ -0,0 +1,371 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2026 Open-IOV contributors +# scrub_pin_qmp.py - host-side channel-buffer (PMA scrubber / X DDX CE) +# semaphore+finish page scanner. +# +# The guest RM polls the channel-buffer host semaphore (channelWaitForFreeEntry) +# and finish payload (channelWaitForFinishPayload) through the *vmap* page of +# the buffer memdesc. After the guest RM re-creates its kernel VAS the host's +# GPU-VAS walk can keep resolving a stale page (r1712: pass authored +# 0x126b60000 while the guest polled 0x124fec000), so the waits never +# completed. This tool resolves the guest's own vmap pages and registers them +# through sema_scrubbuf_add; the service then authors the proven all-ones pair +# into the pages the guest actually reads (CONT.409). +# +# Usage (host, root for the QMP socket): +# sudo python3 tools/guest-diag/scrub_pin_qmp.py [--duration 0] [--interval 2] +# +# Every registered page must look like a channel buffer: the ring page +# (base+0x64000) is all-zero or 906F entries whose GET names a push inside +# the channel's own GPU window. A page that fails the shape test is skipped. +import argparse +import importlib.util +import os +import re +import subprocess +import sys +import time + +HERE = os.path.dirname(os.path.abspath(__file__)) +OIV = os.path.normpath(os.path.join(HERE, "..", "..")) +VMEXEC = os.path.join(OIV, "scripts", "vgpu_vm.py") +PARAM = "/sys/module/vgsp/parameters/sema_scrubbuf_add" + +# nv_vm_map_pages areas seen for channel buffers (pb 0x64000 + gpfifo + sema +# + finish, with per-size variants). The fwimage boots observed 0x6e000 and +# neighbours; keep a small band and rely on the ring shape test. +SCRUB_SIZES = (0x6b000, 0x6c000, 0x6d000, 0x6e000, 0x6f000, 0x70000, + 0x71000, 0x72000) +PAGE = 0x1000 + + +def log(msg): + print("[%s] %s" % (time.strftime("%H:%M:%S"), msg), flush=True) + + +QMP_PATH = "/tmp/qemu-qmp.sock" + + +def load_qmp(wait=0.0): + """Connect to the QEMU monitor, retrying until it exists. + + The systemd unit starts at host boot, before the VM exists, and cycle.sh + removes and re-creates the socket on every VM cycle. Wait for the socket + instead of exiting, so one long-lived scanner covers the whole session + (it must already be scanning when Xorg creates its DDX channel buffer - + that is the point of running it from boot).""" + spec = importlib.util.spec_from_file_location( + "semfix_watch", os.path.join(HERE, "semfix_watch.py")) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + deadline = time.time() + wait if wait else None + last = "socket missing" + while True: + if os.path.exists(QMP_PATH): + try: + return mod.Qmp(QMP_PATH) + except Exception as e: + last = e + if deadline is not None and time.time() >= deadline: + raise SystemExit("no QMP socket: %s" % last) + time.sleep(2) + + +def vmexec(cmd, timeout=25): + try: + out = subprocess.run( + ["python3", VMEXEC, "exec", "--timeout", str(timeout), cmd], + capture_output=True, text=True, timeout=timeout + 15).stdout + except Exception: + return "" + return out + + +def guest_vmaps(): + out = vmexec( + "sudo -n grep -E 'nv_vm_map_pages' /proc/vmallocinfo 2>/dev/null | " + "awk '{split($1,a,\"-\"); " + "sz=strtonum(\"0x\" substr(a[2],3))-strtonum(\"0x\" substr(a[1],3)); " + "print a[1], sz}'", timeout=20) + found = [] + for line in out.splitlines(): + m = re.match(r"^(0x[0-9a-fA-F]+)\s+(\d+)$", line.strip()) + if m: + base, size = int(m.group(1), 16), int(m.group(2)) + if size in SCRUB_SIZES: + found.append((base, size)) + return found + + +def phys_read64(q, pa, n=2): + """Read n qwords of guest physical memory through the QEMU monitor. + + HMP prints each output line as "<addr>: <values>"; only the part after + the colon is data (the address tokens are not). + """ + out = q.hmp("xp/%dgx 0x%x" % (n, pa)) + vals = [] + for line in out.splitlines(): + if ":" not in line: + continue + vals += [int(x, 16) for x in re.findall(r"0x([0-9a-fA-F]+)", + line.split(":", 1)[1])] + return vals[:n] + + +def ring_ok(q, ring_pa): + """906F GPFIFO ring page: all-zero or entries whose GET names a push + inside the channel's own GPU window (the guest scanner's shape test, + guest_pin_stream.py _scrub_ring_ok).""" + vals = phys_read64(q, ring_pa, n=16) + if not vals: + return False + for i in range(0, len(vals), 2): + e0 = vals[i] & 0xffffffff + if e0 == 0: + continue + # Low 32 bits of the push GPU VA: the CeUtils scrubber window is + # 0x120000000..0x120080000, the DDX channel buffer pushes reach + # 0x1200a0000+, so accept through 0x120200000. + if not (0x20000000 <= e0 < 0x20200000) or (e0 & 3): + return False + return True + + +def wait_shape(q, hsem): + """A channel-buffer wait signature: the host semaphore reads a + plausible consumed count and the finish word is still zero. + + r1749: the DDX's own channel buffer does not expose the CeUtils ring + layout at +0x64000 (its ring reads the semaphore word 0xfff), so the + ring test rejected exactly the page X was blocked on + (_nv015102rm timed wait on vmap+0x6c004). The consumed value with a + zero finish word is the wait the release pair exists to end.""" + hv = phys_read64(q, hsem, n=4) + if len(hv) < 2: + return False + return 0 < hv[0] <= 0x20000 and hv[1] == 0 and hv[2] == 0 \ + and hv[3] == 0 + + +def register(fin_pa, hsem_pa): + req = "0x%x,0x%x" % (fin_pa, hsem_pa) + r = subprocess.run(["sudo", "tee", PARAM], input=req + "\n", + capture_output=True, text=True) + return r.returncode == 0 + + +def clear(): + register(0, 0) + + +def scan(q): + pages = [] + for base, size in guest_vmaps(): + try: + hsem = q.gva2gpa(base + 0x6c000) + ring = q.gva2gpa(base + 0x64000) + except Exception: + continue + if not hsem or not ring: + continue + # r1749/r1752: the ring-shape test rejected exactly the buffers + # the guest was blocked on, and the QMP physical reads are not + # stable enough to shape-test a moving page. The candidate set is + # already narrow (nv_vm_map_pages area of a channel-buffer size), + # so register every stable candidate; the service only authors a + # record the guest itself polls (and only while !submitted). + # + # r1760: in the single green cycle the strict "two identical reads" + # gate skipped the live DDX wait page (hsem_pa=0x103789000) for the + # WHOLE cycle -> no registration -> the Xorg DDX wedge the release + # exists to end. Retry the read a few times (the flakiness is often + # a single bad sample), and accept a page that reads wait-shaped + # even when the samples disagree: consumed count in (0, 0x20000] + # with a zero finish word is the exact signature of the wait this + # registration releases (wait_shape below). + stable = None + prev = None + last = [] + for _ in range(6): + hv = phys_read64(q, hsem, n=2) + if len(hv) < 2: + continue + last = hv + if prev is not None and hv == prev: + stable = hv + break + prev = hv + if stable is None: + # r1760: the live page read hsem=0x20000064/fin=0x6001 while the + # DDX spun on it and every sample pair disagreed. The page the + # guest waits on IS a live channel buffer; a nonzero sema word + # at the channel-buffer sema offset is that signature. Register + # with a loud log; the service only authors records the guest + # polls and (for all-ones) only while no USERD submissions are + # observed, so a false registration is bounded. + if wait_shape(q, hsem) or (last and last[0]): + log("register vmap %#x (unstable but live: last=%s) " + "hsem_pa=%#x" % (base, [hex(v) for v in last], hsem)) + else: + log("skip vmap %#x (unstable read) hsem_pa=%#x" % + (base, hsem)) + continue + pages.append((hsem + 4, hsem)) + wp = walk_page_cached(SCRUB_SEMA_VA) + if wp: + pages.append((wp + 4, wp)) + return pages + + +# r1762: the DDX scrub wait polls the CeUtils scrubber channel's sema/fin +# page at the GPU VA 0x12006c000 (+4 = fin, the mem_scrub finish word), which +# the service's kernel-VAS walk resolves (walk_probe) - NOT the guest-vmap +# candidate pages this scanner used to register (0x128013000 was already +# all-ones and the guest still spun; registering the walk-resolved +# 0x12801b000 released the wait in ~23 ms). Register both: the vmap path +# stays the fallback for boots where the walk is stale (r1712). +SCRUB_SEMA_VA = 0x12006c000 +WALK_PARAM = "/sys/module/vgsp/parameters/walk_probe" +_walk_cache = {"va": 0, "pa": 0, "at": 0.0} + + +def host_walk_page(va): + """Resolve a GPU VA to a guest PA through the service's kernel-VAS walk. + + walk_probe is a module parameter; the result lands in dmesg as the last + 'sema-walk-probe: ... discovered rc=0 pa=0x...' block.""" + try: + r = subprocess.run(["sudo", "tee", WALK_PARAM], + input="0x%x\n" % va, + capture_output=True, text=True) + if r.returncode != 0: + return 0 + out = subprocess.run(["sudo", "dmesg"], capture_output=True, + text=True).stdout + except Exception: + return 0 + pa = 0 + kern_pa = 0 + vas_h = None + for line in out.splitlines(): + if "sema-walk-probe: va=0x" in line: + pa = 0 # a newer block begins; only its result counts + kern_pa = 0 + vas_h = None + elif "sema-walk-probe: vas[" in line: + try: + vas_h = int(line.split("h=0x", 1)[1].split()[0], 16) + except Exception: + vas_h = None + elif "sema-walk-probe:" in line and "rc=0 pa=0x" in line: + try: + p = int(line.rsplit("pa=0x", 1)[1].split()[0], 16) + except Exception: + p = 0 + if "discovered" in line: + pa = p + elif vas_h == 0xc and p: + # r1764: the summary 'discovered' path can fail (rc=-2) + # while the per-VAS kernel row (h=0xc) still resolves the + # polled scrubber sema page (r1667-authoritative). + kern_pa = p + return pa or kern_pa + + +def walk_page_cached(va): + now = time.time() + if _walk_cache["va"] == va and now - _walk_cache["at"] < 15: + return _walk_cache["pa"] + pa = host_walk_page(va) + if pa != _walk_cache["pa"] or not _walk_cache["at"]: + log("scrubber walk page va=%#x -> pa=%#x (fin=%#x)" % + (va, pa, pa + 4)) + _walk_cache.update({"va": va, "pa": pa, "at": now}) + return pa + + +def module_generation(): + """Identity of the current vgsp module + QEMU instance. + + scrub_pin_qmp is long-lived: cycle.sh rmmod/insmods vgsp and relaunches + QEMU between rounds. Both reset the registration state the scanner + tracks implicitly, so a changed generation must force a full re-register + even when the guest's vmap addresses happen to repeat.""" + mod = "?" + try: + with open("/proc/modules") as f: + for line in f: + if line.startswith("vgsp "): + f = line.split() + mod = f[0] + "@" + (f[5] if len(f) > 5 else "?") + break + except Exception: + pass + pid = "?" + try: + with open("/tmp/qemu-harness.pid") as f: + pid = f.read().strip() + except Exception: + pass + return (mod, pid) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--duration", type=float, default=0, help="seconds (0 = forever)") + ap.add_argument("--interval", type=float, default=2.0) + args = ap.parse_args() + + q = load_qmp() + cur = None + gen = None + last_refresh = 0.0 + start = time.time() + while True: + if args.duration and time.time() - start >= args.duration: + break + g = module_generation() + if g != gen: + log("generation %s -> %s, re-registering" % (gen, g)) + gen = g + cur = None + try: + pages = scan(q) + except Exception as e: + log("scan failed: %s" % e) + pages = [] + key = tuple(sorted(pages)) + now = time.time() + if key != cur: + if cur is not None and not key: + # r1743: a transiently-empty scan (the live buffer was only + # visible through a rejected alias for a pass) used to + # clear() and the release stopped, wedging the DDX wait the + # registration exists to release. Keep the live records; + # only a *different non-empty* set replaces them. + log("scan empty; keeping %d registration(s)" % len(cur)) + else: + if cur is not None: + clear() + for fin, hsem in pages: + if register(fin, hsem): + log("registered fin=%#x hsem=%#x" % (fin, hsem)) + else: + log("register failed fin=%#x hsem=%#x" % (fin, hsem)) + cur = key + last_refresh = now + elif key and now - last_refresh > 30: + # Additive re-assert: the service reaps records on its own + # (bounded registry); re-registering is idempotent because + # records are keyed by hsem_pa. + for fin, hsem in pages: + register(fin, hsem) + last_refresh = now + time.sleep(args.interval) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/sdl_cap.c b/tools/guest-diag/sdl_cap.c new file mode 100644 index 0000000..23c83ab --- /dev/null +++ b/tools/guest-diag/sdl_cap.c @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: BSD-3-Clause +// sdl_cap.c - LD_PRELOAD capture shim: records every frame GZDoom presents +// through SDL into an append-only file, so the host can harvest the guest's +// real rendered frames without any X/driver readback path. +// +// Hooks: +// SDL_RenderPresent(SDL_Renderer*) -> SDL_RenderReadPixels(ARGB8888) +// SDL_UpdateWindowSurface(SDL_Window*) -> SDL_GetWindowSurface pixels +// Both forward to the real SDL after capturing. +// +// File layout (little endian): +// [0] 8B magic "GZCAPV01" (+ 4088 B reserved zero) +// [4096] frame records: +// 4B "GZF2" | u32 w | u32 h | u32 stride | u32 bpp +// u64 ts_ns (CLOCK_MONOTONIC) | u64 seq | stride*h bytes (top-down) +// A trailing record with seq = 0xFFFFFFFFFFFFFFFF marks clean stop. +// +// Env: GZCAP (path, default /dev/shm/gzframes.bin), GZCAP_MAXF (0=unlimited) +#define _GNU_SOURCE +#include <dlfcn.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> + +#define ARGB8888 0x16362004u + +typedef struct SDL_Renderer SDL_Renderer; +typedef struct SDL_Window SDL_Window; + +typedef struct SDL_Surface { + uint32_t flags; + void *format; + int w, h; + int pitch; + void *pixels; + void *userdata; + int locked; + void *list_blitmap; + struct { int x, y, w, h; } clip_rect; + void *map; + int refcount; +} SDL_Surface; + +static int (*real_RenderReadPixels)(SDL_Renderer *, const void *, uint32_t, + void *, int); +static int (*real_RenderPresent)(SDL_Renderer *); +static SDL_Surface *(*real_GetWindowSurface)(SDL_Window *); +static int (*real_UpdateWindowSurface)(SDL_Window *); +static int (*real_GetRendererOutputSize)(SDL_Renderer *, int *, int *); +static void (*real_GL_SwapWindow)(SDL_Window *); +static int (*real_GL_GetDrawableSize)(SDL_Window *, int *, int *); +static void (*real_glReadPixels)(int, int, int, int, unsigned, unsigned, void *); + +static FILE *cap_fp; +static unsigned long long cap_count, cap_max = 0; +static int cap_stop; + +__attribute__((constructor)) static void cap_init(void) +{ + const char *p = getenv("GZCAP"); + const char *m = getenv("GZCAP_MAXF"); + char hdr[4096]; + + if (m) + cap_max = strtoull(m, NULL, 0); + if (!p) + p = "/dev/shm/gzframes.bin"; + real_RenderReadPixels = dlsym(RTLD_NEXT, "SDL_RenderReadPixels"); + real_RenderPresent = dlsym(RTLD_NEXT, "SDL_RenderPresent"); + real_GetWindowSurface = dlsym(RTLD_NEXT, "SDL_GetWindowSurface"); + real_UpdateWindowSurface = dlsym(RTLD_NEXT, "SDL_UpdateWindowSurface"); + real_GetRendererOutputSize = + dlsym(RTLD_NEXT, "SDL_GetRendererOutputSize"); + real_GL_SwapWindow = dlsym(RTLD_NEXT, "SDL_GL_SwapWindow"); + real_GL_GetDrawableSize = dlsym(RTLD_NEXT, "SDL_GL_GetDrawableSize"); + real_glReadPixels = dlsym(RTLD_NEXT, "glReadPixels"); + if (!real_glReadPixels) + real_glReadPixels = dlsym(RTLD_DEFAULT, "glReadPixels"); + if (!real_glReadPixels) { + void *h = dlopen("libGL.so.1", RTLD_NOW | RTLD_GLOBAL); + + if (!h) + h = dlopen("libGLESv2.so.2", RTLD_NOW | RTLD_GLOBAL); + if (h) + real_glReadPixels = dlsym(h, "glReadPixels"); + } + fprintf(stderr, "[gzcap] init path=%s present=%p updwin=%p swap=%p read=%p gros=%p\n", + p, (void *)real_RenderPresent, (void *)real_UpdateWindowSurface, + (void *)real_GL_SwapWindow, (void *)real_RenderReadPixels, + (void *)real_GetRendererOutputSize, (void *)real_glReadPixels); + cap_fp = fopen(p, "wb"); + if (!cap_fp) + return; + memset(hdr, 0, sizeof(hdr)); + memcpy(hdr, "GZCAPV01", 8); + fwrite(hdr, 1, sizeof(hdr), cap_fp); + fflush(cap_fp); +} + +static void cap_record(const void *px, int w, int h, int stride, int bpp) +{ + struct timespec ts; + uint64_t tns, seq; + uint32_t rh[4]; + char m[4] = { 'G', 'Z', 'F', '2' }; + + if (!cap_fp || cap_stop) + return; + if (cap_max && cap_count >= cap_max) { + uint64_t endseq = 0xFFFFFFFFFFFFFFFFull; + + fwrite(m, 1, 4, cap_fp); + fwrite(&endseq, 8, 1, cap_fp); + fflush(cap_fp); + cap_stop = 1; + return; + } + clock_gettime(CLOCK_MONOTONIC, &ts); + tns = (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; + seq = cap_count; + rh[0] = (uint32_t)w; + rh[1] = (uint32_t)h; + rh[2] = (uint32_t)stride; + rh[3] = (uint32_t)bpp; + fwrite(m, 1, 4, cap_fp); + fwrite(rh, 4, 4, cap_fp); + fwrite(&tns, 8, 1, cap_fp); + fwrite(&seq, 8, 1, cap_fp); + fwrite(px, 1, (size_t)stride * h, cap_fp); + fflush(cap_fp); + cap_count++; + if ((cap_count & 31) == 0) + fprintf(stderr, "[gzcap] frames=%llu\n", cap_count); +} + +int SDL_RenderPresent(SDL_Renderer *r) +{ + static int hit; + + if (!hit++) + fprintf(stderr, "[gzcap] hook RenderPresent\n"); + if (cap_fp && !cap_stop && real_RenderReadPixels && + real_GetRendererOutputSize) { + int w = 0, h = 0; + + if (real_GetRendererOutputSize(r, &w, &h) == 0 && w > 0 && h > 0 && + w <= 4096 && h <= 4096) { + static unsigned char *buf; + static size_t bufsz; + size_t need = (size_t)w * 4 * h; + + if (bufsz < need) { + free(buf); + buf = malloc(need); + bufsz = buf ? need : 0; + } + if (buf && real_RenderReadPixels(r, NULL, ARGB8888, buf, + w * 4) == 0) + cap_record(buf, w, h, w * 4, 32); + } + } + if (real_RenderPresent) + return real_RenderPresent(r); + return 0; +} + +int SDL_UpdateWindowSurface(SDL_Window *win) +{ + static int hit; + + if (!hit++) + fprintf(stderr, "[gzcap] hook UpdateWindowSurface\n"); + if (cap_fp && !cap_stop && real_GetWindowSurface) { + SDL_Surface *s = real_GetWindowSurface(win); + + if (s && s->pixels && s->w > 0 && s->h > 0 && s->pitch > 0 && + s->w <= 4096 && s->h <= 4096) + cap_record(s->pixels, s->w, s->h, s->pitch, 32); + } + if (real_UpdateWindowSurface) + return real_UpdateWindowSurface(win); + return 0; +} + +void SDL_GL_SwapWindow(SDL_Window *win) +{ + static int hit; + static unsigned char *gbuf; + static size_t gbufsz; + + if (!hit++) + fprintf(stderr, "[gzcap] hook GLSwapWindow\n"); + if (cap_fp && !cap_stop && real_glReadPixels && real_GL_GetDrawableSize) { + int w = 0, h = 0; + + if (real_GL_GetDrawableSize(win, &w, &h) == 0 && w > 0 && h > 0 && + w <= 4096 && h <= 4096) { + size_t need = (size_t)w * 4 * h; + + if (gbufsz < need) { + free(gbuf); + gbuf = malloc(need); + gbufsz = gbuf ? need : 0; + } + if (gbuf) { + real_glReadPixels(0, 0, w, h, 0x1908, 0x1401, gbuf); + /* GL rows are bottom-up: flip into a scratch record */ + { + unsigned char *tmp = malloc(need); + int y; + + if (tmp) { + for (y = 0; y < h; y++) + memcpy(tmp + (size_t)y * w * 4, + gbuf + (size_t)(h - 1 - y) * w * 4, + (size_t)w * 4); + cap_record(tmp, w, h, w * 4, 32); + free(tmp); + } + } + } + } + } + if (real_GL_SwapWindow) + real_GL_SwapWindow(win); +} diff --git a/tools/guest-diag/stuck_client_capture.sh b/tools/guest-diag/stuck_client_capture.sh new file mode 100755 index 0000000..6acd8b5 --- /dev/null +++ b/tools/guest-diag/stuck_client_capture.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# stuck_client_capture.sh - one-shot capture for a stuck CUDA client in the +# vGPU guest (run on the HOST; talks to the guest over ssh_guest.sh). +# +# The in-guest pin scanner consumes trace_pipe continuously, so kprobe reads +# return entries=0 while it runs. This tool: +# 1. pauses the scanner (kill -STOP), +# 2. reads the retained kprobe trace tail for the client's UVM waits +# (uvtw/uvsv/uvmt) - the tracker semaphore kptr and the value waited, +# 3. walks the tracker object (tracker_walk.py) for its expected payload, +# 4. dumps the semaphore-pool page (pool_page.py) for applied values, +# 5. resumes the scanner. +# Usage: stuck_client_capture.sh <client-comm> [pool-pa-hex] +# e.g. stuck_client_capture.sh victory_matmul 0x107fd8000 +set -u +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +SSH="$REPO/scripts/ssh_guest.sh" +CLIENT="${1:?usage: stuck_client_capture.sh <client-comm> [pool-pa]}" +POOL="${2:-0x107fd8000}" + +echo "=== pushing walk helpers to the guest ===" +"$SSH" "cat > /tmp/tracker_walk.py" < "$REPO/tools/guest-diag/tracker_walk.py" +"$SSH" "cat > /tmp/pool_page.py" < "$REPO/tools/guest-diag/pool_page.py" + +echo "=== pausing scanner ===" +timeout 30 "$SSH" "echo ubuntu | sudo -S sh -c 'pkill -STOP -f guest_pin_stream.py; echo STOPPED'" 2>/dev/null | tail -1 + +echo "=== kprobe trace tail (client $CLIENT) ===" +timeout 60 "$SSH" "echo ubuntu | sudo -S sh -c 'tail -c 300000 /sys/kernel/tracing/trace | grep -a \"\$CLIENT\" | grep -aE \"uvtw|uvsv|uvmt\" | tail -8'" 2>/dev/null | tail -10 + +echo "=== tracker walk (kptr from the uvsv line above) ===" +echo " (run: sudo python3 /tmp/tracker_walk.py 0x<sem-kptr>)" + +echo "=== pool page $POOL ===" +timeout 90 "$SSH" "echo ubuntu | sudo -S python3 /tmp/poolpage.py $POOL" 2>/dev/null | tail -12 + +echo "=== resuming scanner ===" +timeout 30 "$SSH" "echo ubuntu | sudo -S sh -c 'pkill -CONT -f guest_pin_stream.py; echo RESUMED'" 2>/dev/null | tail -1 +echo "=== done ===" diff --git a/tools/guest-diag/tracker_walk.py b/tools/guest-diag/tracker_walk.py new file mode 100755 index 0000000..3846fa1 --- /dev/null +++ b/tools/guest-diag/tracker_walk.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +# tracker_walk.py - read a UVM tracking semaphore's expected payload and walk +# its semaphore object via /proc/kcore (run in the guest as root). +# +# Use with a live stuck client: the udvt/uvtw kprobes or the scanner's #sem +# lines give the tracker kptr (e.g. 0xffff899a885b5480); this prints +# tracker+0x0 = semaphore kptr +# tracker+0x28 = expected payload (the value the client waits to observe) +# semaphore+0.. = its pointers (rm_mem/page) and candidate value words +# Pair it with the host's pool-wr/deferred-release logs and a pool page dump +# (pool_page.py) to compare what the driver expects vs what the host wrote. +import sys +import struct + +sys.argv = ["tracker_walk"] +sys.path.insert(0, "/tmp") +import guest_pin_stream as g # noqa: E402 + +kc = g.Kcore() +if not kc.load(): + print("KCORE LOAD FAILED") + sys.exit(1) + + +def r64(a): + d = kc.read(a, 8) + return struct.unpack("<Q", d)[0] if d else None + + +def r32(a): + d = kc.read(a, 4) + return struct.unpack("<I", d)[0] if d else None + + +for arg in sys.argv[1:] or ["0xffff899a885b5480"]: + SEM = int(arg, 16) + print("tracker 0x%x:" % SEM) + for off in range(0, 0x40, 8): + print(" +0x%x = 0x%x" % (off, r64(SEM + off) or 0)) + sp = r64(SEM) + print("semaphore=0x%x" % (sp or 0)) + if sp and g.is_kern_ptr(sp): + for off in range(0, 0x80, 8): + v = r64(sp + off) + print(" sem+0x%x = 0x%x" % (off, v or 0)) + for off in range(0, 0x80, 8): + v = r64(sp + off) + if v and g.is_kern_ptr(v): + w = r32(v) + print(" *sem+0x%x -> dword=0x%x (%u)" % (off, w or 0, w or 0)) diff --git a/tools/guest-diag/wall2/README-session74.md b/tools/guest-diag/wall2/README-session74.md new file mode 100644 index 0000000..58587eb --- /dev/null +++ b/tools/guest-diag/wall2/README-session74.md @@ -0,0 +1,66 @@ +# WALL #2 / device-init completion — session 74 findings (r675) + +## The client's "11 s stalls" fully root-caused (this is NOT the channel-init release) +strace (ioctl-only) shows THREE exact 11.00 s gaps with NO ioctl in flight; a +full strace shows the client doing a pure `sched_yield` spin (~57K/s) + +`clock_gettime` (perf) inside **libnvidia-eglcore.so.570.144** — i.e. a +userspace timeout loop. gdb backtraces place the spin inside the Vulkan ICD's +EGL/GL init (`vkCreateInstance` -> libGLX_nvidia vk_icdNegotiate... -> +libEGL -> eglcore wait loop). + +## The wait target: completion slots in /dev/nvidiactl mappings +- The wait loop (eglcore file offsets ~0xb284b0 delay, ~0xb33d00 wait loop, + ~0xb1c750 slot check) polls a slot table on the client's heap object: + table at object+0x1428, count at object+0x22efc. +- Slot check: `esi` is offset by 0x43; for esi-0x43 > 4 it reads a **u16 at + slot+0xe** and treats `0xffff` as "empty/not complete". +- gdb sampling during the stall: 13164 checks with `u16@slot+0xe = 0xffff`, + 1 with `0` — the client spins until the flag changes. +- The slot pointer = a **/dev/nvidiactl rw-s mapping** (page-sized; base+0x20 + = slot). These are the client's mappings of RM memory (the channel + notifier pages: service logs `notif=0x11627f060` per channel). + +## Filling the slots (guest-side) WORKS mechanically but is not the -4 cause +- `slotfill.sh` / `slotfill2.sh`: while vk_frame runs, write 0 to + `base+0x2e` of every small /dev/nvidiactl rw-s mapping when it reads 0xffff. +- Result: the 11 s gaps vanish (5 fills within ~2 s; the client proceeds + immediately) — **but vkCreateDevice still fails -4**. So the 11 s waits are + not the failure; they are an init-side slowdown the client tolerates. + +## The next wall: the polled RM control 0x2080a097 +- With the slots filled, the client settles into **NVOS54 (ioctl nr=0x2a) + polling every ~1.25 s** (strace), and the host dmesg shows + `GSP_RM_CONTROL cmd=0x2080a097 hClient=0xc1d000bb hObject=0xbeef0004 + paramsSize=1168 flags=0x0` -> `unknown cmd=0x2080a097 -> generic SUCCESS` + (all-zero reply), 36 polls in 45 s. +- The NVOS54 struct (gdb dump): {hClient=0xc1d000c0, hObject=0xbeef0004, + cmd=0x2080a097, flags=0, params=ptr, paramsSize=0x490, status=ptr, ...}. +- The params buffer = 1168 bytes of 32-byte records with a `0x25` field at + +0x10 (dump in /tmp/opencode evidence; gdbparam.sh captures it live). +- The client polls until the control's status/reply signals completion; our + generic all-zero reply never does. **This is the wall that must be solved to + get vkCreateDevice past -4** (either a real reply for 0x2080a097 or making + the polled status field change). + +## Other session-74 results +- The GR ring-entry writer was NOT found by disassembly-guessing: nvidia.ko + `channelFillGpFifo` and nvidia-modeset `nvWriteGpEntry` never fire for this + channel's kick (gdb hbreaks); nvidia/modeset kprobes are `notrace`; QEMU gdb + hbreak works. +- `sema_push_raw` dumps (module param) captured the service's reads of the + push: entry0 decodes only 2 methods then zeros (`consumed=0x288/1432`), + entry2 starts `SET_OBJECT 0xc597` then zeros; **no release in the push** + (`have_rel=0` for all three entries) — the channel-init release theory for + these waits is dead. +- `find_ring_va.py` / `vmap_watch2.py`: the ring entries are NOT in any + vmallocinfo area during the run; the nv_vm_map_pages areas hold the CE + pushes; the GR push content is not found by content searches. + +## Files +- gdbpoll6.gdb, gdbu16.gdb, gdbfill.gdb, gdbwatch.gdb, gdbparam.sh, + gdbgap3.sh, gdbspin.sh — the gdb probes (slot check, u16 flag, fill, + NVOS54 param dump, backtraces). +- slotfill.sh / slotfill2.sh — the guest-side slot-fill watcher. +- stracevk.sh, fastrun.sh, fastrun2.sh — the strace/perf recipes. +- perfstall5.sh — perf sampling with map-relative offsets. +- find_ring_va.py, vmap_watch2.py — the vmap/kcore watchers. diff --git a/tools/guest-diag/wall2/README-session76.md b/tools/guest-diag/wall2/README-session76.md new file mode 100644 index 0000000..828b3b9 --- /dev/null +++ b/tools/guest-diag/wall2/README-session76.md @@ -0,0 +1,134 @@ +# session 76 notes (r680) — the 0x2080a097 engine-monitor re-examined + +## What the firmware says (ground truth for the a097 family) + +The shipped GSP-RM 570.144 image embeds the host-side control +descriptor table (file offset 0x41df8..0x42450, entry = 0x20 bytes +{u32 cmd, u32 params_size, u32 0x4049580, u32 0, u32 handler_ptr, +u32 0, u32 out_size, u32 0}). Entries seen: + + 0x2080a07f size 0x408 handler vaddr 0x551b2c0 + 0x2080a079 size 0x14804 + 0x2080a097 size 0x490 handler vaddr 0x55166f4 + 0x2080a0d1 size 0x7e8 + ... + +The real 0x2080a097 handler (0x55166f4) writes up to FOUR {u32,u32} +mode pairs into params+4..+0x23 (params[0] low byte = the request's +count flag): for i=1..4 it calls an engine-status helper +(0x5516334 / 0x55d7f04 / 0x55d7c1c / 0x55d7d60) with mode=i and emits +{pair} = {engine value, 1000 * u16(engine+0x1bc0)}; the loop stops at +the first helper error. So the real reply overwrites the client's +request record {1,0,0,0}. + +## What the client does with the reply + +libnvidia-eglcore's local handler (file offset 0xecd2e0, the userspace +implementation of the internal cmd 0x2080a040) copies back: + + result+0 = reply+0 (count) + result+0x10 = reply+0x204 (flag) + result+0x20 = reply+0x488 + result+0x24 = reply+0x48c + count*16 bytes from reply+4 -> *(result+8) + +For the engine tick (file offset 0xd5e3e0, called from the wait loop +0xd6ef40 per iteration) the result struct IS the caller's request at +rsp+0x10, and *(result+8) points at a 16-byte "inner" buffer +(rsp+0). The tick then reads inner+4 and stores it to session+0xe104 +(the per-engine "progress" cache; poll interval ~1.25 s from the same +function). A second consumer of the tick's return value compares it +against per-boot config at lib+0x2019320: + + 0xfc8 = 1000, 0xffc = 95 (progress threshold), + 0xfec = 10, 0xfed = 20 (bit-history cutoffs), + 0x1000 = 500 + +and ORs the comparison bit into a round-robin bitfield (+0xb4) of the +state machine object. + +## Live evidence on r680 (fresh boot, full-param recipe) + +* The 0x2080a097 polls (NVOS54 0x2a, 1.25 s cadence) come from a + WORKER thread (gdb bt: start_thread -> eglcore a11b2a -> wait loop + 0xd6ef40 -> tick 0xd5e3e0). The engine object E (0x604101365e90) + sits at E+0x150 = 0x00200000 (bit 21 only), E+0x1b038 = 1, + E+0x1b039 = 1, *(E+0x1b030)+8 != 0 — all static for 5+ minutes. +* MAIN thread (vkCreateDevice) is in a poll(fd=/dev/nvidia0, events + POLLIN|POLLPRI, timeout=10s) loop inside eglcore (bt: poll <- + eglcore+0xa12de4 <- 0xd69159 <- 0xd6a02b <- 0xdcebc6 <- ... <- + libvulkan vkCreateDevice). No -4 is ever printed in this state: the + client waits indefinitely (killed by its own timeout in runs). +* GR channel kicks continue concurrently (sema-entry lines for + h=0xbeec0100 va=0x4ca0000 len=1432 etc.) and end as + "skipping stale entry ... after N failed rescues" (gr_wedge>0). +* Reply sweep on r680 (module params) — flag=0/2/1/1..0xffffffff, + v488, v48c AND the new record matrix count/rec0..rec7 = + 0/{1,0,0,0}/100/1000/100000/1000000/0xdeadbeef — produced ZERO + observable change: same a097 cadence, no writers to + E+0x150/E+0x1b038/E+0x1b039 ever caught, no client progress. The + hw watchpoints on those words fired zero times over ~5 min. + +## Tools + +* t75g.py — in-guest gdb python: dispatcher-type log, a097 issuer + dump, tick (d5e449) state + cfg dump, E/S capture, hw watchpoints on + E+0x150 / E+0x1b038 / E+0x1b039 / S+0x120, POST a040 consumption + dump. +* diag76.py — in-guest gdb python: thread-apply bt + map-relative + address resolution (used to find the main-thread poll wait). +* slots_probe.sh — read the u16 completion-slot words of every small + /dev/nvidiactl rw-s mapping (SESSION 74's slotfill ABI). +* run75g.sh — trace + record-word sweep driver. + +## Verdict + +The a097 monitor is a background worker, not an obviously blocking +gate: every knob the client reads from it can be varied with no +observable state change, while the main thread blocks elsewhere +(/dev/nvidia0 poll loop) and the GR channel-init pushes still cannot +be read by the sema service. Next lever is the main thread's wait +(what would make the /dev/nvidia0 poll return readable) or the GR +channel-init completion — not the a097 reply words. + +## ADDENDUM (r681) — THE MAIN-THREAD vkCreateDevice WAIT DECODED + +Live gdb on the main thread (r681, fresh boot) resolved the loop at +eglcore 0xdceb80 (frame chain: poll <- a12de4 <- d69159 <- d6a02b <- +dcebc6 <- d3838a <- ... <- libvulkan vkCreateDevice): + +* rbx (waitable, vt 0x...3f58f8): vt+0x18 = d69ee0 = "should wait" + (returns `(*0x48(*(rbx+0x70)) != 1)`); vt+0x28 = d69f50 = "wait" + (builds a pollfd array from the object's fd holder and poll()s up to + 16 fds with the 10 s timeout). +* rbp (loop object, vt 0x...3f71f8): vt+0x38 = dcee20 = the give-up + predicate. dcee20 returns nonzero (-> return -4 = the client's + VKFAIL -4 origin) when either object at rbp+0x110 / rbp+0x120 shows + an error (b29d60(...) != 0 or *(obj+0x70cc) != 0); otherwise 0 and + the loop repeats. It also throttles itself at 1e6 (units) via + rbp+0x1b8. +* The readiness value: sub = *(rbx+0x70) (0x...f7f0c90); sub+0x58 -> + obj2; obj2's vt+0x10 = 0xce71f0 = `read u32 at (*(obj2+0x58)) + + *(u32*)(obj2+0x60)` — i.e. a u32 word at holder_base + 0x80 (the + live object had obj2+0x60 = 0x80, obj2+0x64 = 0x1e). Observed: 0x0 + every poll (39/39) -> should-wait = 1 forever. +* The word lives in a **/dev/nvidiactl rw-s mapping** + (0x7623d0480000-0x7623d0484000) — a VM_IO driver page, not plain + guest RAM: /proc/PID/mem reads of it return zeros, and writing the + guest-RAM file at the pagemap-derived PAs (0x1036dd080 and both + hole-shifted candidates) did NOT change the client's behaviour. + The writer is therefore the guest kernel driver/RM when the awaited + operation completes (the same class as the SESSION-74 completion + slots) — not a host RAM poke. +* All 20 sampled polls of the wait helper returned 0 (timeout): the + fd (/dev/nvidia0) never becomes readable (nv_poll reports + POLLPRI|POLLIN only when an nv_event is pending on the file). + +So vkCreateDevice waits for (a) the readiness word == 1, and the wait +ends only when the driver/RM signals that word (or an nv_event arrives +on /dev/nvidia0). The give-up path (-4) fires only if the RM error +flags appear. Next: identify the operation whose completion writes the +word (the object chain sub -> obj2 -> holder is a "signal/semaphore" +object; dump the holder object and its creating control), then make +the emulation drive the guest kernel to complete it (the engine/ +channel-init completion the sema service already tries to deliver). diff --git a/tools/guest-diag/wall2/README-session77.md b/tools/guest-diag/wall2/README-session77.md new file mode 100644 index 0000000..d830bef --- /dev/null +++ b/tools/guest-diag/wall2/README-session77.md @@ -0,0 +1,1195 @@ +# session 77 (r683/r684) — vkCreateDevice PASSES: the 0x7E event validation and the readiness-word completion + +## The two-part unblock + +1. **r682 fix validated** (`e3c1cdc5`, loaded on r683/r684): the + nvkms/Vulkan path allocates its completion event as + NV01_EVENT_KERNEL_CALLBACK_EX (class 0x7E); the emulation now + registers it alongside 0x79. Live proof (r683, sema_dbg=1): + + GSP_RM_ALLOC class=0x7e paramsSize=24 hClient=0xc1e00011 + hParent=0xcaf00001 hObject=0xcaf00003 + event registered hEvent=0xcaf00003 hSrc=0x0 + notifyIndex=1006632972 (off 8, slot 13) + params=1100e0c1000000007e0000000c00003c0000000000000000 + EVPOST all ev[13] cl=0xc1e00011 ev=0xcaf00003 nidx=0x3c00000c rc=0 + + (notifyIndex 0x3c00000c = subdevice 0x3c, index 0xc = + NV2080_NOTIFIERS_GRAPHICS). The guest consumes the records + (`guest_read == our_write` in the rmstate dump), but the guest's + event dispatch does NOT write the event's notification word — the + residual delivery gap. + +2. **The completion word found and driven (the actual unblock)**. The + main-thread vkCreateDevice wait (eglcore 0xdceb80, session 76) reads + a u32 readiness word: `should-wait = (word != 1)`; the word lives at + `+0x80` of a **16 KB rw-s /dev/nvidiactl mapping** (the event + notification buffer; holder chain sub -> obj2 -> holder, resolved + live by t79.py). The host can write it directly in the guest RAM + file: + + PA = pagemap PFN << 12 (+0x80); file offset = PA - 0x80000000 + (above-4G RAM; below-4G would be PA itself per the cycle's + guest-RAM registration: gpa 0x0/0x100000000 -> off 0x0/0x80000000) + + Writing 1 flipped the wait and **vkCreateDevice returned success** + (`VKDEVICE ok family=0`) — reproduced three times: + * r683: manual write after a t79 probe; + * r684: nudger (all 16 KB mapping words) + slotfill restart; + * r684d: probe -> single-word write (the reliable recipe). + +## The reliable recipe (per run) + +1. r682+ module; `sema_dbg=1 gsp_rm_ev_log=300` optional. +2. Launch `vk_frame` + `slotfill2.sh` (EGL completion-slot fills; the + slots re-arm, so re-run slotfill if the client is stuck in the + sched_yield spin before the word-wait). +3. Wait for the main thread in `do_poll` (the word-wait). +4. `t79.py` (gdb) -> `WAIT word=0x...`; guest pagemap -> PFN -> PA; + host write `01 00 00 00` at `PA-0x80000000`. +5. `VKDEVICE ok` follows within ~10 s. The client then proceeds to + image/memory/pipeline setup; its next wait is NOT the same word + (the word stays 1) — next session: probe the follow-on state + (attach fails while the client spins in the post-device phase; + use the /proc-only checks + a longer observation window). + +## Cautions learned + +* Writing ALL 16 KB mapping words (the multi-page nudger) can trip the + -4 give-up path (an RM error flag word gets written): restrict nudges + to the single probed word. +* Attaching gdb to the client while it is in the tight post-device spin + can hang the attach (leave a `timeout`-wrapped gdb, then kill that + gdb by PID and `kill -CONT` the client — the pending SIGSTOP from a + dead tracer otherwise wedges it). +* zsh truncates 64-bit hex literals in `$(( ))`: compute PFN/PA in + python, not zsh. + +## Tools + +* `t79.py` — resolves the wait chain (sub -> obj2 -> holder -> word VA) + and prints the client's nvidiactl rw-s mapping sizes. +* `guest_words.sh` — prints VA/PA of the +0x80 word for every 16 KB + nvidiactl mapping (pagemap; run in the guest as root). +* `nudge.sh` — host-side loop: poll guest_words.sh and write 1 to any + zero word (multi-page variant; prefer the single-word recipe). + +## ADDENDUM (r685, session 78) — POST-DEVICE WAIT IDENTIFIED: the NvOgl (GL ucode) engine init; push windows now READABLE (resolver6) + +* The post-device spin is the **GL module bring-up**: the main thread's + A8-layer retry loop (eglcore 0xa85d24) works on a work item whose + embedded name is ASCII **"NvOgl_m0"** (+0xf04: "NvOgl_m40"); the D6 + submit layer (0xd624e0) ORs **0x200000** into its pending word (the + same engine-pending bit SESSION 75/76 watched on E+0x150), then the + A8 loop yields via the B284B0 spin helper (sched_yield) waiting for + the GL ucode init to complete on the GR engine. +* **The push windows ARE now readable** (first time): the resolver6 + recipe (window-record pair {push_va,0x20000}+{ring_va,0x40000} -> + same-page 0x20000 memdesc PA -> `sema_map_add`) injected + `0x4480000 -> 0x120cfb000` (content `2001c00d 70000000 2001c00d + 68000000` = the 2-method semaphore-like push) and `0x4ca0000 -> + 0x12e51e000` (the 1432-byte GR init stream: SET_OBJECT 0xc5c0/0xc5b5, + `0x200121ed`, `0x200120a9`, bulk methods, tail `2001c00d + 70/68`). The service now decodes the streams (`sema-push ... tail=0dc0 + 012000000070...`) where SESSION 73-74 saw "unread". + CAUTION: the resolver injects every candidate; a zero-page candidate + injected last replaces the good mapping for that VA (re-inject the + good page after the scan). +* **Remaining gate**: the service still reports `have_rel=0` for every + GR push (the CE-shaped release matcher) and — decisively — the + GR-forward machinery is NOT engaging: `drain held=4 starved=149 + forwarded=0 ... gr_nocap=4`. The GL ucode init work is never + forwarded/executed, so the completion (and the 0x200000 bit clear) + never arrives. Next: the GR forward/capture path for these channels + (operand capture + release decode for the GR shape), then the client + proceeds to VKFRAME. + +## ADDENDUM (r686, session 79) — the low-VA release fix and the corrected injection recipe + +* The GR/GL init releases decode only with the low-aperture acceptance + (`SEMA_VA_LO` bound relaxed to >= 0x4000000 in both release forms, + commit cb8b75c5): live `rel=0x4b19b10 pay=1` (REPORT_SEM group + `200426c0 00000000 04b19b10 00000001 10000000`), `rel=0x4b17bb0 + pay=3` and `rel=0x4b17e30 pay=1` (0x5c-form `2005c017 04923060 ... + 02000001`); `have_rel=1`, `applied` rising, un-translated releases + queue in svc_pending (417 observed). +* Corrected pair-specific push injections for r686: + `0x4480000 -> 0x125692000` (record page 0x1163c9000, live desc), + `0x4ca0000 -> 0x131649000` (page 0x122f20000), plus + `0x4b16000 -> 0x12a76a000` (the {0x4b16000,0x1000} window). + CAUTION: resolver7 injects each page's descs for BOTH pairs (union + loop) - fix it to keep per-pair pages or inject manually from the + per-pair scan. The window-record layout is {tag, VA, SIZE, 0, + "tail", ...} stride 0x40 (search the VA u64 with size at +8 and + "tail" at +0x10). +* Remaining: the semaphore VAs (0x4b17bb0/0x4b17e30/0x4b19b10) have no + window records in the known record pages; the service cannot resolve + them (targeted rm_mem miss) so the releases stay pending and the + NvOgl wait spins. Find their PAs (client mapping pages / RM + VA-tracker / live poll-watch), inject, and the pending releases + should apply. + +## ADDENDUM (r686, session 80) — ALL low-aperture window records found (the "liat" tag fix); the semaphore windows identified and their releases applying + +* **The record scan fix**: the RM window record is 0x40 bytes + {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c; u32 0}. The tag is + stored as the u32 0x7461696c - on disk the BYTES are "liat", NOT + "tail" (the hex display `7461696c` reads as "tail" only as a number). + Searching for b'tail' finds nothing; `wall2_recscan.py` searches for + the byte pattern 6c 69 61 74 and decodes {tag,VA,SIZE} at + -0x20/-0x18/-0x10. It found **all 75 records** in one pass. +* **The GR init semaphore windows** (r686): + `va=0x4b17000 size=0x2000 -> pa=0x12a76f000` (verified by content: + +0xbb0=3, +0xe30=1 - exactly the rel=0x4b17bb0 pay=3 / + rel=0x4b17e30 pay=1 payloads) and + `va=0x4b19000 size=0x1000 -> pa=0x122f9b000` (after injection the + release applied there: +0xb10 became 2). Injections: + `echo 0x4b17000,0x12a76f000,0x2000,1 > sema_map_add` and + `echo 0x4b19000,0x122f9b000,0x1000,1 > sema_map_add`. +* **State after the injections**: the low-VA releases decode + (`have_rel=1`) and apply to the semaphore pages; `pending` hovers at + ~418 (the client re-kicks continuously); the NvOgl wait still spins + (the A8 loop compares per-entry counter tables r14+0x5e8 vs the + wanted set - the completion is not just the semaphore value). Next: + the A8 counter tables' source (what fills r14+0x5e8 / the "wanted" + table - likely the GR channel's own progress counters) or the GR + forward/ucode execution (`forwarded=0 gr_nocap=7` still). + +## ADDENDUM (r686, session 81) — the counter-autoreg gap analysed: the counter page 0x20400f000 is unresolvable; the manual `sema_counter` registration is the bypass + +* `svc_counter_autoreg` (per-pass for eng==0 channels, n_gr=20) needs a + translation for the rank slot VA `0x20400fec0 + (19-r)*0x10` + (r = gr_index = rank by ascending handle among the guest's GR + channels): either an ipin pin (`svc_ipin_lookup`, 9275 entries live + from the guest pin stream) or `svc_counter_derive` (the + vaspaces/discovered/targeted walk chain, `sema_counter_derive=1` + default). Both miss for the counter page -> no "counter auto" log, + `pub_calls=0` -> the A8 loop's per-entry counter tables never fill and + the NvOgl wait spins. +* The client does NOT map the counter pool: `/proc/<pid>/pagemap` for + 0x20400f000/0x20400ffd0/0x20400ffc0/0x20400ffb0 is all-zero (not + present) and no VMA covers it - it is a UVM kernel-side object, so the + pin stream cannot resolve it through the client's page tables. The + stream's log shows `sem=0` (zero semaphore resolutions) even though + the want-list (`/tmp/guest_want_vas.txt`, host-written) contains 44 + `20400f` entries; the stream's "channel-anchored expansion" for the + UVM tracking sems (guest_pin_stream.py ~line 950-1010) is the + machinery that must find them (it is written for the CUDA second-client + stall - "the only live uvmt event is the spinning sem"). +* Bypass available now: the `sema_counter` parameter registers a channel + slot manually - `echo <h_object>,<pa> > .../parameters/sema_counter`. + Once the counter page's PA is known (via the guest kcore/UVM walk), + this wires the publish path without a kernel rebuild. +* The rank slots for the GL client's channels (handles 0xbeec0100, + 0xbeec0104, 0xbeee0100 -> ranks 0,1,2): 0x20400ffd0 / 0x20400ffc0 / + 0x20400ffb0. + +## ADDENDUM (r686, session 82) — the counter-page PA hunt: pagemap route confirmed dead, vidmem-window hypothesis tested and not supported + +* The client's own pagemap does NOT contain the counter pool: the + semfix_uvmt tool's resolver (`guest_pmap_win.py <pid> 0x20400f000 + 0x204010000 64`) returns NOTHING, as does the UVM pool range + (`0x121000000 0x121100000`). The tool's `pin_counter_page` has + therefore never succeeded for these clients (the STALL WATCHDOG's + "re-pin counter" attempts across sessions produced no "counter: + pinned" line). +* VIDMEM-window hypothesis tested: the counter VA 0x20400f000 is in the + emulated vidmem window (>= SVC_VIDMEM_VA_BASE 0x200080000), so its + window offset is 0x3F87000 (0x20400f000 - 0x200080000). The shadow + there is reachable and all-zero (`sema_vidmem_dump 0x3f87000,64`); + poking the rank slots (+0xfd0/+0xfc0/+0xfb0 = ranks 0/1/2) with 6 and + then 0x40 via `poke_vidmem` changed the shadow (verified by dump) but + did NOT move the client's wait. The vidmem route is not (or not only) + the completion path; the A8 loop's tables have another source. +* gdb-attach hazard (again): attaches to the client in its post-device + spin hang with no output; two gdb chains had to be killed by PID + (never a pattern) and the client SIGCONT'd. Probe only while the + client is in do_poll. + +## ADDENDUM (r687, session 83) — THE A8 COUNTER SOURCE DECODED: a client readback window the host can write (first live read-back confirmed) + +* The A8 loop calls `a82f30(obj)` per entry; a82f30 calls the object's + method pointer at ***(obj+0x19b8)** = eglcore+0xb250f0, which returns + `(hi<<32)|count`. The count comes from **b1bf00**: + - fast path (obj+0x224 != 0): `*(u32*)(obj+0x1078) - 1`; + - slow path (observed: obj+0x224 == 0): **b27c90(obj, 0x800000bb/0xbc)** + which reads `*(u32*)(*(*(obj+0x1480)+0x58) + idx*16)` - a **client + READBACK WINDOW** (idx = 0xbb..0xbb+n, n = *(u32*)(obj+0x22efc)); + the loop takes the MIN of those reads. +* Live (r687, object handle 0xbeec0100): the window = VA + 0x7827e6c12000 (a /dev/nvidiactl mapping), all zeros; obj+0x224=0 + (slow path active), obj+0x1078=7 (fast-path field unused). +* **The host CAN write it**: resolving the window's PA via the client's + pagemap and writing 0x100 into the window (offsets 0xbb0..0x1ff0, via + the RAM file) made the client read it - **obj+0x1070 = 0x100** (the + method stores the returned count there) and the A8 "actual" table + entry became `0xffffffff00000100`. FIRST LIVE PROOF that the client's + completion counter can be driven from the host. +* The wait still spins: the A8 loop's first scan (compare + a85e1d: actual vs want, want observed 0) passes with our value, but + the loop continues into a second scan (compare a85ec1, driven by + b29dc0) and the `*(rsp+0x38)` branch (a85e60). NEXT: decode those + (the second scan's actual source and the rsp+0x38 flag), and whether + the other objects (entries 0-2 of the A8 table) need the same window + writes. +* The other objects' windows were not observed (only 0xbeec0100 called + a82f30 in the sampled window); the nvidia0-side window (obj+0x1478) + is NOT present in the pagemap (VM_IO; host cannot write it via the + RAM file). + +## ADDENDUM (r687, session 84) — the a86038 want-scan mapped: ALL WANTS PASS with the driven window value; the residual gate is the A8 loop's internal callback/retry machinery + +* Live dump of the a86038 scan (BP at 0xa8608e, per-index): + i=0 want=0x0 actual=0x0 pend=no + i=1 want=0x0 actual=0x0 pend=no + i=2 want=0x0 actual=0x0 pend=no + i=3 want=0x6 actual=0xffffffff00000100 pend=no + (want from a82fd0(job_obj, i) = the job object's +0x98 array maxed + against global+0x600; actual from r14+0x5e8 = our driven counter). + NO pending entries -> the scan completes. +* Writing 0x100 to the ENTIRE readback window (offsets 0x0..0x1ff0 of VA + 0x7827e6c12000) did not clear the wait either. +* The loop still cycles: the bt stays at the a85f08-a85f1a yield call + (b284b0(*(rsp+0x10), 0xb) - a bounded pause/yield spin of ~0x14 + units, hence the slow cadence), and the exit-attempt path a85f30 + decrements *(rbp) (w0=1) while firing the global callbacks + (0x206f790/0x206f7a0 vtables: *(%rax), *0x10, *0x18) with args from + rbp+0x30/0x38/0x40. The remaining gate is this callback/retry + machinery (or another wait object), not the readback values. +* NEXT: resolve the global callback targets (0x206f790/0x206f7a0 + vtables at runtime) and trace the a85f30 exit path (what the + callbacks do and what makes *(rbp) reach 0); or find how the real GPU + writes the readback window (the op-4 methods 0x800000bb/0xbc the + service skips) and emulate that write so the client's own machinery + completes. + +## ADDENDUM (r687, session 85) — **THE NvOgl WAIT CLEARED** (the session-count gate) and the client advanced to vkAllocateMemory (new wall: -2 OUT_OF_DEVICE_MEMORY) + +* The A8 loop's exit requires the session count `*(u32*)(r14+0x5bc)` + (= lib_base + 0x2070b7c, the global at +0x20705c0+0x5bc) to reach 0; + the loop head increments `*(rbp)` and the entry scans pass with the + driven window value, but with count=4 the exit checks + (a85b48/a85e78: `if (r12d == 0)`) are never taken - the loop waits for + the engine sessions to complete (count -> 0). +* **DIRECT TEST (succeeded)**: `set {unsigned int}0x<base+0x2070b7c> = 0` + via gdb -> the loop exited, the NvOgl init completed, and the client + ADVANCED past the post-device wait for the first time: + + VKDEVICE ok family=0 + VKFAIL vkAllocateMemory(dev, &ma, NULL, &img_mem) -> -2 (line 204) + +* **NEW WALL**: `vkAllocateMemory` for the image (DEVICE_LOCAL) returns + -2 (VK_ERROR_OUT_OF_DEVICE_MEMORY) at vk_frame.c:204 - the guest + driver's heap/type selection fails before any alloc RPC (no + GSP_RM_ALLOC in dmesg). Next: the memory-heap reporting + (FB_GET_INFO_V2 0x20801303 is handled; check the heap-size/type + queries the driver uses for the DEVICE_LOCAL type, and the emulated + VRAM size the driver sees). +* Recipe for the next run: slotfill + word nudge -> VKDEVICE ok; then + write 0 to lib_base+0x2070b7c (gdb; the harness) once the NvOgl wait + starts (the A8 loop's count gate); then the memory-alloc wall. + +## ADDENDUM (r689, session 86) — **THE DEVICE_LOCAL -2 WALL CRACKED**: the 512 MiB emulated VRAM was the gate; the "nvidia-nvidia-vgpu-turing-2g" mdev type (2 GiB) lets the allocation proceed + +* Evidence chain: the ICD hooks (t91.py) captured the client's allocation + `ALLOC sType=5 size=0x4000 typeIndex=1` (a valid 16 KB DEVICE_LOCAL + request); the standalone memory-properties dump (vk_memprops.py) + showed device-local heap[0] = 512 MiB with a separate 2936 MiB + non-local heap. With 512 MiB the driver failed the allocation + (VK_ERROR_OUT_OF_DEVICE_MEMORY, no RM RPC). +* **FIX TESTED**: the cycle with `VGPU_MDEV_TYPE=nvidia-nvidia-vgpu-turing-2g` + (start_vm_keep.sh honours the env var) -> `heap[0] size=0x80000000 + (2048 MiB)` (vk_memprops.py) -> the SAME client with the SAME recipe + (slotfill + word nudge + count auto-clear) reaches `VKDEVICE ok` and + the allocation NO LONGER fails -2: the client proceeds into + `os_schedule` (a deeper wait). +* REMAINING in the new state: the client's later pushes (va=0x44852d4 / + 0x4ca52d4, then +0x586c) extend beyond the first window page and read + `pa=0x0`/zeros (the sema injections covered page 1 only; the window's + page list is scattered - the memdesc carries only the FIRST page). + NEXT: extend the window injections to the full scattered page list + (from the memdesc's page array) or accept 1-page pushes; then the + allocation completes and the client proceeds to the buffer/pipeline. +* Tools: vk_memprops.py (standalone memory-properties dump; safe + alongside other clients), vk_alloc_test.py (standalone allocation + reproducer - NOTE: it hits vkCreateDevice -4 without the harness + (word nudge + count clear), so it cannot replace the vk_frame run), + t91.py (auto-clear + ICD memprops/alloc hooks). + +## ADDENDUM (r689, session 87) — the window page-list hunt: the pages are scattered; content matches found but the VA<->page relation is unresolved + +* The 2 GiB cycle's window records (recscan89): `0x4480000` descs = + {0x107c47000 (live init stream), 0x103c06000, 0x12ecc2000}; + `0x4ca0000` descs = {0x11dbfa000, 0x12f398000 (live init stream)}. + The window's later offsets (va+0x52d4, +0x586c, +0x9000) at + first_PA+offset are ALL ZEROS -> the windows are NOT contiguous + (scattered pages). +* The memdesc record (page 0x106128000) is {size=0x20000, flags + 0x03008000, PA=0x107c47000, kernel pointers 0xffff8d2e5e21ec50 / + 0x4fe7ec50 / 0x461287e8 ... , "head"/"tail"} — the page list hangs off + the kernel pointers. A guest kcore walk of those pointers returned + "unreadable" (stale addresses by the time of the walk; the walk must + happen on FRESH pointers in the same pass as the dump). +* Content search for the distinctive init-stream prefix (`00 80 01 20 + b5 c5 00 00 04 00 04 20`) found the window pages: + 0x107c47000+0x0 (the injected init page), 0x103c06000+0x858, + 0x11dbfa000+0x858, 0x126d69000+0x59c — the later pages hold the same + engine-init streams at arbitrary offsets; the VA<->page relation is + NOT linear (the +0x59c push appears in pages whose +0x0 is zero), so + the mapping cannot be reconstructed by offset alone. +* NEXT: (a) dump the memdesc + kcore-walk its page-list pointers in ONE + guest pass (fresh addresses); or (b) inject the content-matched pages + at trial VAs and watch the service's reads; or (c) the service-side + decode of the window page lists. The client (2632) blocks in + os_schedule at the allocation — the unread later pushes are the + suspected gate. + +## Session 88 addendum - the push-page fix, the gdb-kill wedge, the ring-mapping gap + +**1. PATH-5 RULE (the push pages).** A GR window's pushes do NOT live at +the window base page: the r689-family windows carry their method streams in +WINDOW PAGE 5 (va offsets 0x52d4/0x559c/0x586c/0x596c all live in +0x5000-0x5fff). Standing injections at the base page resolve pa=0x0 for the +pushes (the service's `sema-pushall ... pa=0x0` lines) and the channel +holds/skips entries as "GR no-capture". The fix: page PA = +match_filoff - (va & 0xfff) + 0x80000000 for a content match of the push's +first8 bytes constrained to the same in-page offset; then +`echo 0x<va&~0xfff>,0x<pa>,0x1000,1 > .../sema_map_add`. Proven on r689: +channel beec0104's pushes at va=0x4d00000 (page -> PA 0x12f398000) went +from first8=00000000 to first8=20012000 0000c5c0, and `fast sweep` +applied 3281 -> 3809 (+30/sweep sustained). + +**2. host_resolve.py** automates 1: it polls dmesg for sema-pushall lines +with pa=0x0 and non-zero first8, searches /dev/shm/vgpu-guest-ram +(offset-constrained), and injects the page. The service's 6 s no-capture +hold is the race window - run it BEFORE the client and keep it polling. + +**3. NEVER SIGKILL AN ATTACHED GDB.** On r689, killing the stale t91.py +gdb (kill -9 on PID 5334) killed its tracees (client threads +2883/2884/2885/3048); one held the guest RM API rwsem write lock; the lock +leaked and every later RM call blocked (`[vkps] Update blocked for more +than 1228 seconds` in rmapiLockAcquire -> RmGssLegacyRpcCmd -> +_nv04ControlWithSecInfo). The t91 harness design is correct: `timeout 500` +SIGTERMs the gdb and it detaches cleanly. If a gdb must go, use SIGTERM +and wait; SIGKILL only when the guest is about to be cycled anyway. A +leaked kernel rwsem cannot be repaired live. + +**4. The r690 gap: the GR channel ring mapping.** Fresh cycle r690 (mdev +f5a796b0): every RM RPC of the client init is answered (GR ctx commit, +channel allocs class 0x9067 hObject 0xcef00000/0xcef00080, EVPOSTs), but +the service reports `gpfifo done=0 nomap=938 pushes=0` - it cannot read +any GPFIFO ring, so no entry decodes, no pushall line appears, and the +guest's completion poll loops on RM control 0x20800a70 until the ~10 s RM +timeout -> vkCreateDevice returns -4 (VK_ERROR_DEVICE_LOST). r689's +service had done=19108 (rings mapped) - so the ring mapping is the +missing ingredient for a fresh client in r690; candidate paths: the FIFO +alloc params (12 bytes, GPFIFO VA inside) are not logged; the scanner saw +the CE window (ipins 4592) but no GR ring this boot. Next: log the 0x9067 +alloc params (or scan the guest for the ring VA) and inject the ring page. + +**5. gr_walk.py / gr_walk2.py** are the guest-side kcore walkers (fresh-pass +memdesc discovery; 12 records found, kptrs readable in-pass) kept as the +alternate grounding path. + +## Session 89 addendum - THE PT WALK WORKS (live); the completion-word gap + +**1. GMMU PT walk resolved LIVE (the mission's PT-binding step: DONE).** +The service's own walker math (svc_gmmu_walk: root_bits=ilog2(entries), +shift=49-root_bits; L1 shift38, L2 shift29, PD0 idx=(va>>21)&0xff with 16-byte +dual big/small entries, PT idx=(va>>12)&0x1ff; PDE bit0=0, PTE bit0=1; +pte_addr=(e>>8)<<12) was replicated host-side over /dev/shm/vgpu-guest-ram +(walk.py / live_walk.py). The LIVE published pdb (dmesg: "page directory +... pdb=0x... entries=4" - a NEW pdb per client, e.g. 0x105f0a000 / +0x12a359000 / 0x107012000) resolves every client VA while the client runs: +ring 0x44a0000 -> PA, windows 0x4480000/0x4485000/0x4ca0000..., releases +0x4213000/0x45a0bb0, and the injection via sema_map_add (ap=1 sysmem) makes +the service read them. VALIDATED BY CONTENT: the ring page held exactly the +GPFIFO entries the service logged (entry[1] raw=8000040004480598 va=0x4480598 +len=4), and the push page at va=0x4480598 held the method stream +(...0x20020047...). Post-mortem reads fail: the guest frees/zeros the +structures and pdb pages when the client dies - ALL resolution must run +LIVE during the client's ~11 s life (live_walk.py polls dmesg every 0.4 s, +parses pdb + VAs, walks, injects; static family list pre-seeds the +deterministic VAs). + +**2. Progress with the walk+inject live: gpfifo done=0 -> 6-9.** +With the live injections the service decoded and CONSUMED the whole GR ring +(entries raw logged, releases decoded: rel=0x4213000 pay=1, rel=0x45a0bb0 +pay=3, have_rel=1) and submissions reached 6. The client STILL ends +vkCreateDevice -4 (~11 s, RM 10 s poll of control 0x20800a70) because the +releases are decoded but NOT APPLIED and the client's completion counter +page is unresolvable. + +**3. Why releases do not apply (two gates found).** +(a) svc_fwd_defer_arm(): with gsp_phys_gr_forward=1 a GR ring candidate that +is valid but not "offered" keeps defer armed - every GR release goes to the +forwarded-hardware path (which never runs) instead of the inline software +apply. Cycling with gsp_phys_gr_forward=0 removes the arm, but the release +still does not apply - the decode's apply path is additionally gated by +svc.op_tr_pending (a global "operand targeted scan in progress" state) via +svc_opdef_release: while op_tr_pending=1 every release is recorded into an +opdef slot and withheld, silently when the slot already contains it. The +live symptom: `counter page 0x20400f000 unreadable via +vaspaces/discovered/targeted; slot stays pending` repeated at the client's +poll rate plus repeated `targeted resolve va=0x20400f000: full RAM scan +found nothing`. + +**4. The counter/notifier page record tables (host-side resolve).** +Searching the RAM for the counter-page VA as u64 finds 16-byte {pa,va} +record arrays, e.g. {0x5FEDC000,0x20400D800},{0x5FEDD000,0x20400E000}, +{0x5FEDE000,0x20400E800},{0x5FEDF000,0x20400F000},... (VA step 0x800, +PA step 0x1000) and a second variant ({0x5FEEE000,0x20400E000}, +{0x5FEEF000,0x20400F000}). These are how the counter page (and probably +the notifier, notif=0x1xxxxxxx0 per registration) can be resolved WITHOUT +the VA-space walk: search the VA u64, parse the neighbouring u64 as PA, +inject. Must be done live (records vanish post-mortem). NOTE: the service's +own targeted scan reports "found nothing" for this VA - the record layout +the host parses is not what that scan looks for. + +**5. Next steps (ranked) for session 90.** +1. Extend live_walk.py with a record-resolve fallback: for any VA the + service logs as unreadable/pending/stalled (regex set already widens), + search the RAM for its u64, parse {pa,va} records (both orders), inject + candidates; feed it counter-page (0x20400f000 family), notifier + (notif=... per registration) and release VAs BEFORE the decode (static + list already covers the deterministic ones: add 0x20400f000 with the + record-resolve because the pdb walk cannot reach the UVM/counter space). +2. Or fix the two release gates kernel-side: (a) don't arm the GR deferral + when the forwarding engine has no proven channel (offered flag); (b) + treat op_tr_pending as per-request state (clear when no scan is actually + in progress for the entry) so a stuck global does not withhold every + release. Both are small, testable changes (gsp_sema_service.c). +3. Watch for the first live sign: `rel-apply ... written` / `pub-adv` / + `counter auto/derived` lines on the client's channel; then VKDEVICE + should pass and the flow proceeds to vkAllocateMemory. + +**6. Counter-page record resolve PROVEN LIVE (session 89 end).** +`echo 0x20400f000,0x5FEDF000,0x1000,1 > sema_map_add` (the {pa,va} record +found by the u64 search) made the service learn the slot: +`sema-chan: counter auto h=0xbeef0100 rank=0 pa=0x5fedfff0 (slot +va=0x20400fff0)` and the completion publish went from ZERO to 172 calls +(`counter pub_calls=72 -> 172` over the client's life; previously 0 on every +boot of this session). The client still ends vkCreateDevice -4, so the +remaining unmet wait is the GR ctx-init release word (rel=0x4213000 pay=1, +rel=0x45a0bb0 pay=3 - decoded, have_rel=1, never applied: no rel-apply / +pub-adv line for either). With the counter now advancing, the next +instrumented step is naming the exact GR release-handling branch that +swallows the apply (rebuild with a bounded print at each branch of +svc_rel_apply_906f / svc_opdef_release / svc_rel_apply_launch), then fix +that gate. The gr_forward defer-arm gate is already understood (see 3a) and +gr_forward=0 is the correct test configuration; the remaining suspect is +the service-global svc.op_tr_pending path in svc_opdef_release (it records +the release into an opdef slot and returns true silently when the slot +already contains it). + +**7. Session 90: the CURRENT-boot wait loop site (the old recipe no longer matches).** +The client's long spin on this driver/boot sits INSIDE vkCreateInstance (bt: +libEGL -> eglcore -> 0xB2852D helper), a bounded yield loop at +eglcore+0xB2852D: `mov (%r11),%r12d / test $1,%r12b / jne <exit> / +mov 0x4(%r11),%eax / cmp $1,%eax / jne <retry>` - the loop waits for an +IN-PROCESS word pair at r11 (e.g. 0x7ffc2a1c5080, stack/heap; guest-written +only, the host RAM file cannot feed it because it is not a GPU buffer). +Evidence the old site moved: t91's breakpoint at eglcore+0xa85d24 never +fires this boot (`clears=0`) and a read-watchpoint (t80r.py) on the +0x80 +words of the two 16 KB rw-s nvidiactl mappings does not fire either (the +16K event-buffer wait of sessions 77/78 is not the live target now; the +spin stack is +0xB2852D <- +0xA5325A <- +0xA539AD <- +0xA47491 <- +0x9FF515 +<- +0xA20D04 <- libEGL, NOT the 0xA85F1A/A8 path). A widened multi-word +nudge (nudge2.sh/guest_words2.sh: all rw-s nvidia mappings <=256K, +0x80) +DID fire on 20+ words but the client still returned -4 - the session-77 +caution stands: multi-word writes trip the error path; single-word only, +after identifying the true target. +NEXT: (a) rwatch WRITE on r11+4 to name the writer of the in-process word +(the component that must run; if nothing ever writes it, disassemble the +exit branch 0x...ebc9 vs retry 0x...eb93 to see the pass/fail contract); +(b) the same for the 0xA53xxx caller chain - one of these callers is the +A8/NvOgl GL-ucode wait of sessions 78/79 (its completion comes from the +emulated GR engine path); (c) only then re-test the count-clear + single +word nudge recipes. + +**8. Session 91: THE vkCreateInstance WALL BREACHED (the marker-clear recipe).** +The client's retry loop (eglcore+0xB33D30) waits until NO event-slot entry has +the pending marker 0xffff: per entry ptr = *(u64*)(obj+0x1428+i*8), +cnt = *(u32*)(obj+0x22efc), the checked slot = ptr + n*0x10 (n = the call's +esi), marker = u16@+0xe == 0xffff (or u32@+0 when (esi-0x43)<=4) via the +check at +0xB1C750 (OR of all pending markers). The guest's event dispatch +NEVER clears it, so the client burns its 10 s -4 timeout. PROVEN FIX: +clear {u16@slot+0xe, u16@slot+0xc, u32@slot+0} to 0 at every hit of ++0xB33D30 (t93 single-shot; t94 auto-clearer, 4000 hits) -> the client +passes vkCreateInstance AND device enumeration FOR THE FIRST TIME +(`VKINSTANCE ok / VKDEVS 2 / VKPICK`, previously the 10 s -4 fired during +this phase). t94 cleared 5+ distinct module objects (obj=0x5d67063a84a0, +0x5d670661ad80, 0x5d670673eed0, 0x5d67067ebb50, ...) over the client's life. +The client then sat in vkCreateDevice's poll(/dev/nvidia0, nfds=1, +timeout=10 ms) loop at eglcore+0xA12DDF..+0xA12DE4 WITHOUT any -4 for 5+ +minutes (vs the old ~11 s death) - the longest the client has ever run. + +**9. The remaining gap (single): the event-fd signal.** The device-phase wait +is a poll() on /dev/nvidia0 (fd 27) - nvidia_poll wakes only when the guest +RM signals the client's event state; the guest consumes our POST_EVENT +records but never performs the event signaling (session 78's "residual +delivery gap"). Writing to /proc/PID/fd/<nvidia0> is not a signal. The +session-84 count write (base+0x2070b7c, read 4, wrote 0, t97) did not flip +this wait alone. NEXT: fix the host POST_EVENT -> guest dispatch association +(gsp_rm_emulation.c r579 area: the payload fields hClient/hEvent/notifyIndex/ +data/info/status; compare against the guest's _kgspRpcPostEvent -> +gpuNotifySubDeviceEvent expectations) so the guest signals the event and +nvidia_poll returns POLLIN; then the same t94-pattern (or a service-side +equivalent) durably unblocks both waits per boot. + +**10. Session 92: THE AUTOMATED PIPELINE - fresh boot to VKDEVICE with ZERO manual steps.** +On a fresh r692 boot (cycle params unchanged, gr_forward=0) a single gdb +script (t99.py) carrying two breakpoint handlers drove the client end-to-end: +(a) the instance-phase marker cleaner at eglcore+0xB33D30 (t94 pattern: clear +u16@slot+0xe / u16@slot+0xc / u32@slot to 0 per entry at ptr+n*0x10 for +entries *(obj+0x1428+i*8), cnt *(obj+0x22efc)) - fired CLEAR#1..#5 across +five module objects; (b) the device-phase wait-word chaser at +eglcore+0xDCEBD9 (t79 chain: rbx -> *(rbx+0x70) -> *(sub+0x58)=obj2 -> + off=*(u32)(obj2+0x60), holder=*(obj2+0x58), base=*(holder+0x58), +word=base+off) - found WORD addr=0x734781a86080 val=0x0 and wrote 1. +Result: `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok family=0` with no +manual intervention (r692 client 1725). The client then sat in the +POST-DEVICE wait (the next phase - allocation/engine-init), which t99 does +not cover. The live_walk PT-walk injector ran alongside and injected the +window/ring VAs (0x4d20000, 0x4923000...; note 0x4b19b10 resolved ap=FB - +the GL init release is vidmem-mapped and skipped by the SYS-only injector). + +**11. HARD LESSON (attach discipline, again).** A gdb attached to a client +that never hits its breakpoints wedges: `continue` never returns, the +timeout's SIGTERM does not tear it down, and the ptrace slot blocks all +later attaches (hit 3x this session: 146084, the t91-on-1725 attach, both +left holding). RULE: only attach with breakpoints that are KNOWN to fire +for the client's current phase, keep windows <= ~60 s, and prefer +self-terminating handlers (hit caps). A wedged attach leaves the client +RUNNING (state R) but unattachable; cycle the VM for a clean environment +when more attaches are needed. + +**12. Continuing the chain (next session).** (1) Identify the post-device +wait of the VKDEVICE-ok client (bt sample once attachable; likely the +allocation/UVM phase or the A8/NvOgl engine-init with its count gate at +lib_base+0x2070b7c=4 -> 0 and/or the 0xdceb80 wait loop - add the t91 +count-clear BP to the t99 bundle if the A8 path is live). (2) Extend t99 +into the "durable harness bundle": marker-clean + word-chase + count-clear ++ (once VKFRAME) the release/alloc phases. (3) The bNotifyList=1 +(gsp_rm_ev_nlist) path stays armed for the event-signal experiment; +post/EVPOST logging re-budgeted (gsp_rm_ev_log=2000 set). + +**13. Session 93: THE ZERO-PTRACE PIPELINE (r693) - VKDEVICE with no gdb-wedge risk.** +Split the harness by risk: (a) ONE gdb (t101.py) for the INSTANCE-phase +marker clears only (base+0xB33D30 handler, clears u16@+0xe/+0xc and u32@+0 +of each entry; self-disables after 24 clears; NO other BPs so it cannot +fire constantly); (b) the WAIT WORDS handled by /tmp/unwait.py - a guest +python poller with ZERO ptrace that scans the newest vk_frame's rw-s +nvidia mappings of exactly 0x4000 bytes and writes 1 to any zero u32 at +mapping+0x80 via /proc/PID/mem (the session-77 geometry, confirmed by t79 +chains: word = the +0x80 of a 16 KB rw-s nvidiactl mapping). Result on +fresh r693 (mdev 261980af): t101 CLEAR#1..#5 then idle; unwait wrote FIVE +wait words (0x7876ead57080, 0x7876f1816080, 0x7876ea47f080, +0x7876ea81a080, 0x7876ead51080); client reached `VKINSTANCE ok / VKDEVS 2 +/ VKPICK / VKDEVICE ok family=0` with NO wedge exposure. live_walk (PT-walk +injector) ran alongside. The client then sat in the post-device wait that +the +0x80 poller does not cover (t102.py = the one-shot safe diagnostic: +attach -> print PC/base/A8-count(base+0x2070b7c)/bt -> detach, NO continue +and NO breakpoints - use it when the slot is free). + +**14. THE RM-LOCK WEDGE ROOT CAUSE + RULE.** A gdb killed (SIGTERM) while +the tracee sits AT a breakpoint leaves the tracee STOPPED; if that stop is +inside an RM api critical section (the rwlock held across an RM call!), the +RM write lock stays held and EVERY other client blocks in +os_acquire_rwlock_write (state D) - observed directly: old client 1725 +stopped by its wedged t99 gdb -> new client 7967 hung `state=D wchan= +os_acquire_rwlock_write` on ioctl 0xc020462b; killing the gdb + `kill -CONT` +resumed 1725 out of the ptrace stop but it resumed INSIDE the same RM wait, +so the lock stayed held until the VM was cycled. RULES: never let a gdb +timeout fire while a tracee is at a frequent-firing BP; prefer BPs that +fire once or self-disable; prefer no-ptrace pollers for anything with a +repeating pattern; the safe diagnostic is attach -> read -> detach without +continue. + +**15. Session 94: THE INSTANCE CLEARS ARE REQUIRED (zero-gdb disproved); t104 is the safe form.** +Pure zero-gdb run (fresh r694, client + unwait.py ONLY, no gdb): unwait +wrote three +0x80 words and the client still ended `VKFAIL vkCreateDevice +... -4` - WITHOUT the instance-phase marker clears the device gate fails +even with the words. With t104.py (single BP at +0xB33D30; clears +u16@+0xe/+0xc/u32@+0; self-exits on cleared>=8 OR chits>=40 OR 25 s since +the last clear -> returns True so the script tail runs and batch gdb exits +cleanly, no outer-timeout dependence) + unwait + live_walk: CLEAR#1..#5, +two words written, client reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / +VKDEVICE ok family=0` (r694 client 3725). The client then sat in the +post-device wait again; the t102 diagnostic could not attach because the +t104 gdb still held the slot (its 25 s-idle exit never fired - no +post-instance B33D30 hits - and its 150 s timeout likely fired at a stop += the known wedge risk). NEXT FORM: make t104 detach ITSELF from inside +the handler once the clears are done (`gdb.execute("detach")` from the +stop handler; the tracee resumes; the script ends and gdb exits) so NO +gdb is ever held while the client runs - then t102 always has the slot +and the wedge class disappears end-to-end. + +**16. The allocation/post-device phase is the remaining gate.** With +VKDEVICE green the client waits in a site that is neither the +0x80 word +pattern nor (likely) the A8 count path; the next session should (a) get +the detach-first t104 live, (b) run t102 immediately after VKDEVICE to +name the wait (read PC/bt/A8 count = base+0x2070b7c), and (c) inspect the +service for this client (pushes decoded / releases applied / counter +pub_calls) to see whether the allocation needs the live injections only +or a service-side gap. + +**17. Session 95: THE GDB THREAD-STOP INTERFERENCE + the SIGKILL nuance.** +The t105 attach (t=1.5 s, before the client's worker threads spawned) left +four worker threads in `ptrace_stop.part.0` (futex) - the client's main +thread span waiting on them, so the client froze at the post-device wait +with the gdb holding the trace. TERM/INT/HUP on the wedged gdb were all +ignored (batch gdb stuck in continue). SIGKILL on the gdb (1584) was tried +with the reasoning that all stopped threads sat at futex (NO RM lock held): +result - the tracer died, the tracees DETACHED+RESUMED (client 1518 +SURVIVED with its workers back in futex_wait_queue; contrast session 88 +where SIGKILLed-tracer tracees were KILLED mid-syscall). So the SIGKILL +outcome depends on the tracee's stop context; the mission's never-SIGKILL +rule stands, but this datapoint matters when judging recoverability. +After the gdb died, 1518 still held the RM write lock from its own +post-device wait (its wait word was no longer polled because unwait.py +tracks the NEWEST vk_frame and a fresh test client 6555 was launched and +blocked D-state on os_acquire_rwlock_write) - the unwait-newest coupling +must be replaced by ALL-clients polling in the next form. + +**18. HARNESS DESIGN RULES (accumulated).** (1) A gdb attached across +thread-spawn windows stops worker threads and freezes the client - attach +only AFTER all threads exist, or ensure the gdb explicitly resumes new +threads, or do the work with NO gdb. (2) Any gdb that must run through +thread-spawn windows MUST self-detach quickly (the instance phase spans +~10-20 s and the clears arrive in ~5 bursts) and NEVER rely on outer +timeouts. (3) unwait.py should poll EVERY live vk_frame (not just the +newest). (4) The durable answer for all of this is the service-side +folding: the instance markers and the +0x80 words are guest-RAM writes a +kernel-side service can perform on the same triggers (the marker semantics +and the word semantics are understood) - that removes gdb and /proc +helpers from the loop entirely. + +**19. Session 96: THE DETACH-FIRST CLEARER PROVEN (t106) + the orphan-tracer plague.** +t106.py (single +0xB33D30 BP; clears markers; detaches FROM THE HANDLER as +soon as cleared>=5 or chits>=14: `gdb.execute("detach")` + return True so +the batch gdb exits) on fresh r696 (mdev 241af1c1): CLEAR#1..#5, +`BC-DETACHED cleaned=5 chits=5`, `T106-DONE chits=5 cleared=5`, +`TracerPid: 0` - THE SLOT IS LEFT FREE. unwait.py wrote five words and the +client (1549) reached `VKINSTANCE ok / VKDEVS 2 / VKPICK / VKDEVICE ok +family=0`. THE CLEAN FORM OF THE CLEARER EXISTS NOW. + +**20. THE ORPHAN-TRACER PLAGUE (root cause of every t102 failure).** Every +INTERRUPTED gdb attempt leaves an attached orphan: (a) piping gdb to +`head` closes the pipe early and the gdb stays attached (hit twice); (b) an +ssh command dying/timing out mid-gdb-add leaves the gdb running (hit +again). Symptoms: later attaches fail with `Could not attach ... already +traced by process N` / `ptrace: Inappropriate ioctl for device` and the +t102 log stays empty. RECOVERY (session-95 evidence): TERM is usually +ignored (wedged in continue); SIGKILL the orphan + `kill -CONT <client>`; +the tracee SURVIVES when it was running/at-futex and detaches cleanly +(TracerPid 0). STRICT DISCIPLINE: never pipe gdb output (redirect to a +file, read it after); never let an ssh command end while a gdb may be +attached (generous timeouts); after every gdb run verify TracerPid 0. + +**21. THE NEXT FORM (t107, one gdb, one window).** Combine the proven +clearer and the diagnostic in ONE batch gdb: attach -> clears -> detach -> +`python time.sleep(...)` idle -> RE-ATTACH (gdb can attach again after +detach) -> read PC / bt / A8 count (t102 content) -> detach -> print - +one ssh session, one window, everything self-contained, no orphan risk +(every path ends in an explicit detach; wrap each phase in try/except with +a final detach). Expected: names the post-device wait (allocation/UVM vs +A8 count base+0x2070b7c) so the allocation can be passed and the first +VKFRAME reached. + +**22. Session 97: THE DEEP SUBMISSION STATE + THE COUNTER-SLOT WEB.** +The r696 client (1549) reached the deepest state yet: the service reported +`gpfifo done=11576 pushes=10932 counter pub_calls=1356` and climbing +(gpfifo done was 0 in the pre-pipeline boots) - the client submits the full +work stream and the service decodes/processes it (pairs=14395 map=14318, +ipins=9240 from live_walk). The client then sits in the KNOWN userspace +post-device spin (main thread `running` in all samples; one tid in +os_acquire_rwlock_read). NEW: a counter-slot collision spam +(`counter auto ... pa=0x5fedfff0 rejected (slot owned by another channel; +not installed)`, 232+ callbacks suppressed) - the 0x20400f000 injection +resolved the WHOLE counter VA family to ONE page, so both live channels +collided; the manual bypass `echo <h>,<pa> > .../parameters/sema_counter` +(h_object,pa syntax, main.c:2965) was applied for both handles +(0xbeee0100,0xbeec0100 -> 0x5fedfff0) and pub_calls jumped 1356 -> 3739. +The client gate (VKDEVICE -> frames) persists: its remaining wait is the +userspace spin word NOT covered by the +0x80/16K poller (candidate: the +tracker/counter the service now advances - watch whether pub traffic +alone resolves the spin, else t107-GDB the spin site). ENVIRONMENT: one +orphan gdb (10092 on 1549) + stale channels (beec0100) from prior +attempts; the orphan-kill discipline applies (SIGKILL by PID + CONT + +verify TracerPid). + +**23. THE IOCTL-ARG SAMPLING AXIS (no ptrace).** /proc/<tid>/syscall shows +the running ioctl's ARGS (fd, cmd, arg) when caught - sampling the main +thread repeatedly during an ioctl-loop phase names the exact stuck RM +control without any gdb. Use it before ptrace-based diagnostics when the +thread does syscalls at a catchable rate. + +**24. Session 98: THE POST-DEVICE SPIN NAMED - THE SEC2 SCRUBBER AUTH-TAG WAIT.** +perf (no ptrace!) on the spun client: top symbol `channelReadChannelMemdesc` +24.33% (then scheduler internals) - the guest's nvidia driver hot-loops +reading the CHANNEL MEMDESC. Source decode (nvidia-570.144-reference, +sec2_utils.c): the only reference caller is +`_sec2utilsUpdateGetPtr` -> `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` inside `_sec2utilsGetNextAuthTagSlot`, whose loop +waits for the scrub auth-tag slots to free; `authTagBufSemaOffset = +finishPayloadOffset + CHANNEL_ENGINE_SEMAPHORE_SIZE` (4). So the spin = +the SEC2 MEMORY-SCRUBBER channel waiting for its auth-tag semaphore (a u32 +inside the channel memdesc buffer) to advance - the GPU never delivers it +in our emulation, so the client waits forever (each poll = one full memdesc +transfer = the 24% cost). THE FIX DIRECTION: the service must complete the +scrubber's semaphore updates - the scrubber channel registers in the +service (`scrubber channel 0x... detected (gpfifo=... userd=...)`) and its +submits carry semaphore releases targeting the auth-tag VA +(pbGpuVA + authTagBufSemaOffset) in the channel buffer; on decode, write +the release value to that VA's page (the channel buffer is a guest-RAM +allocation the service can resolve/pin; the pending queue holding 17 +entries is the suspect pool for the unresolvable ones - inspect +svc_pending[]). Service-side counter to watch after the fix: the scrub +loop exits, the client proceeds to the allocation/pipeline. Tool: perf +record/report on the guest is the no-ptrace spin-pointer (`perf record -p +<pid> -- sleep 3; perf report --stdio --sort=symbol,dso`). + +**25. Session 99: THE SCRUBBER GATE CONNECTED TO THE CE CHANNELS' ok=0.** +chan_dump on r696 (client 1549 spun): six channels - +`h=0xbeee0100 eng=0 gpfifo=0x44a0000 userd=0xfee00 ok=1 get=6 sub=6354 rel=0x4213020 crel=0x5fedfff0 notif=0x11cd51060` +`h=0xbeec0100 eng=0 gpfifo=0x4cc0000 userd=0xfee00 ok=1 rel=0x4923020 crel=0x5fedfff0` +`h=0xbeec0104 eng=0 gpfifo=0x4d20000 userd=0xfee00 ok=1 rel=0x4b17e30 crel=0x5fedffe0` +and the CE-class channels `beee0101 (eng=11, gpfifo=0x4500000 userd=0x8f600 as=2 ok=0)`, +`beee0102 (eng=12, gpfifo=0x4560000 userd=0x8f400 ok=0)`, `beed0101 (eng=12, +gpfifo=0x49c0000 userd=0x8f200 ok=0)` - ALL ok=0: the CE userd discovery +(failed probe: EAGAIN 0/0 or EINVAL) means no CE pump, no CE sema applies, +so the SEC2 scrubber channel (a CE-class channel) never sees its auth-tag +semaphore advance and the client spins in _sec2utilsGetNextAuthTagSlot. +The service's candidates (registered PA, ring+0x10000 in both apertures, +8-slot list) all miss for the CE channels. `find_userd` (FB scan for a +live put@0x8c/get@0x88 pair) found ZERO candidates - the CE userds are not +in FB with live puts (either the client's PUT writes go to a page the +service's candidates miss, or the CE userds genuinely idle). A raw host-RAM +shape scan (put in 1..65535, get<=put) is too noisy (24 hits in the first +28KB of BIOS-era low RAM) - needs tighter validation. NEXT APPROACH: find +the REAL userd page by DIFF-WATCH - sample the candidate pages (and the +CE ring pages) twice around a scrub submit and diff for the page whose ++0x88/+0x8c pair changes; the guest kernel writes GP_PUT there, so the +changing page is the true userd. Then fix the discovery (or add a +sema_userd-style injection param) so the CE pump engages and the scrubber +completes. NOTE: the GR channels' userd=0xfee00 (as=2, FB) and ok=1 with +get=6/sub=6354 - the GR machinery runs fine; only the CE/scrubber path is +dark. + +**26. Session 100: THE CE USERD HUNT EXHAUSTED THE STANDARD LAYOUTS; THE CIRCULAR WAIT.** +Candidates examined for the CE userds this session: (a) the registered FB +offsets (0x8f600/0x8f400/0x8f200 as=2) - ALL ZERO in the FB shadow +(sema_vidmem_dump 0x8f100,0x600 twice); (b) the ring-PA + 0x10000 linear +guess - random/zero/table garbage (the allocations are NON-CONTIGUOUS); +(c) the PDB walks of gpfifo+0x10000 (0x4501000->pa 0x125e91000, +0x4561000->pa 0x132032000, 0x49c1000->pa 0x11bddd000 via pdb 0x1152f4000) +- ALL THREE PAGES ALL ZERO. The CE userds were never written - and the +deeper reading: the SEC2 scrub flow is a CIRCULAR WAIT: the scrubber waits +for its previous scrub pushes to be CONSUMED (the auth-tag sema advances +only when the GPU runs the scrub) -> the consumption needs the CE pump -> +the CE pump needs a live userd -> the userd is written BY SUBMITS -> the +submits are held by the wait. In a working boot the guest's first scrub +push happens BEFORE the wait loop (the loop only waits when the slots are +full) - so in our boots the FIRST scrub push must have gone into the CE +ring, the service failed to consume it (ok=0/userd dark), and the client +now waits for a completion that will never come. + +**27. THE DIRECT BYPASS (next session, ranked).** (1) Write the auth-tag +semaphore in the CHANNEL BUFFER (memdesc) page directly: the scrub loop's +`_sec2utilsUpdateGetPtr` reads `channelReadChannelMemdesc(pChannel, +authTagBufSemaOffset)` = a u32 at memdesc+finishPayloadOffset+4; advancing +that value frees the slots and unblocks the client WITHOUT the CE pump. +The memdesc = the channel buffer allocation (a VA in the client's space); +resolve it via the PDB (the guest's channel struct gives the buffer VA; +the service's channel registration/alloc params carry it) and write the +incremented value from the service (or the host RAM file). (2) OR decode +the scrub pushes the service already sees (check sema-push for the SCRUBBER +channel - its submits carry the auth-tag VA; the release decode should +target it; the service's release applies would write the sema if the VA +resolves). (3) The registered FB userd offsets being zero suggests the +client's kernel maps the userd through a VA whose PA the discovery misses; +the diff-watch (two samples around activity) remains the fallback proof. + +**28. Session 101: THE LOOP SEMANTICS + perf trace (NO-PTRACE SYSCALL TRACING).** +The wait loop decoded exactly (sec2_utils.c:420-441): +`while (nextPut == pSec2Utils->authTagGetIndex) { gpuCheckTimeout(); +authTagGetIndex = _sec2utilsUpdateGetPtr(); osSpinLoop(); }` - it exits +when the memdesc sema (u32 at pChannelBufferMemdesc + +authTagBufSemaOffset; authTagBufSemaOffset = finishPayloadOffset + 4) +advances past nextPut; it also has a GPU timeout that would error out. +perf trace on the spun client (no ptrace!) revealed the live pattern: +tid 1793 = a 10ms-timed futex loop (WAKE val=1 then +WAIT_BITSET|CLOCK_REALTIME 10ms - the paced retry); tid 1788 = repeated +`openat -> read(fd, 299 bytes) -> close -> futex-wait` - a transient-fd +POLLING READER of a small blob (the fd is opened and closed each cycle, so +fd 28 no longer exists; the client's persistent fds: /dev/nvidia-modeset +(10), /dev/nvidia0 (11/12/14/15), GLCache .toc/.bin (16/17), memfd +\"allocation fd\" (18)). The 299-byte read is the likely userland face of +the memdesc read path (the 24% channelReadChannelMemdesc hot loop). +NEXT: (a) catch the transient fd path by sampling `ls -l /proc/<pid>/fd` +every ~0.2 s in a loop around the burst (or perf trace -s during a burst); +(b) if it is a driver read() of the memdesc, resolve its buffer and write +the auth-tag sema value (advance past nextPut) via the RAM file; (c) the +loop timeout (gpuCheckTimeout) may eventually error the client - the write +must land first. + +**29. Session 102: fd-sampler miss, no nvidia debugfs; the memdesc hunt narrows.** +The 0.15 s /proc/1549/fd sampler over 20 s caught NO transient path (the +open/read(299)/close reader is bursty or dormant between bursts; the static +fds are /dev/nvidiactl, /dev/nvidia0, /dev/nvidia-modeset, a socket, the +GLCache .toc/.bin pair, and a memfd "allocation fd"). No nvidia debugfs +tree exists in the guest. Remaining ranked paths to the auth-tag sema +(memdesc + finishPayloadOffset + 4): (1) content-locate the scrubber's +channel buffer by its method stream (the C5B5/C56F shapes: SET_OBJECT +0xc5c0/0xc5b5, headers 0x200121ed/0x200120a9 observed in the session-78 +window dumps) and read/write the sema inside; (2) the scrubber channel's +own sema-push decodes (its pushes carry the auth-tag VA = pbGpuVA + +authTagBufSemaOffset; grep the boot's sema-pushall for the scrubber +channel id and read the rel= VA - then PDB-walk it); (3) sample the fd +burst narrower (0.02 s) right after a client restart when the burst is +hot. The service-side ideal: on detecting the scrubber channel, resolve +its channel buffer and pre-advance the auth-tag sema (a durable fold). + +**30. Session 103: THE SCRUB STREAM FOUND; ONE RELEASE DELIVERED; THE SEMA PAGE STILL AMBIGUOUS.** +Content-located the CE/scrub init method stream (headers 0x200121ed/ +0x200120a9 searched): full stream at `pa=0x114e98000` = SET_OBJECT +0xC5C0/0xC5B5 (0x20012000/0x20012080 headers), engine-init methods +0x200121ed(3)/0x200120a9(1)/0x2001209c(0x0c000001), bulk methods +(0x8000xxxx), then TWO REPORT_SEM releases in one buffer: +`0x200426c0 00000000 04b19b10 ...` (va=0x4b19b10) and +`0x200426c0 00000000 04b17e30 ...` (va=0x4b17e30), tail 0x2001c00d +70/68. Twins at pa=0x1026ad000+0x87c and pa=0x12afba000+0x87c (identical +content at identical offsets - multi-mapped copies). Release resolution: +`0x4b17e30 -> pa=0x114efbe30 (SYS) cur=1` - THAT release WAS delivered (the +service applied it, cur=1). `0x4b19b10` resolves via the walker's big-half +2MB branch to FB 0x660b10 holding garbage (0xD9F26F94) - poke_vidmem +tests ([0x660b10]=1 then =0x40000000) did NOT flip the client, and the +linear-block guesses (buffer VA 0x4b00000 <-> stream pa 0x114e98000, sema +candidate 0x114eb1b10) read all zero. NEXT: the auth-tag sema page must be +picked by VALIDATION (the existing value must be a small plausible +GetPtr): walk 0x4b19000 returning ALL chains (small/64K/2MB both +apertures), pick the page whose u32 at +0x1b10 (or the offset the memdesc +actually uses) is a small number; or take the buffer aperture from the +channel's alloc params (the service's registration) and walk with that +aperture exclusively. The fix remains: advance that u32 (write via the +host RAM file if SYS, poke_vidmem if FB). + +**31. Session 104: THE SEMA VA 0x4b19b10 IS NOT THE LOOP'S PAGE; THE KPROBE IS THE ANSWER.** +The validated multi-chain walk (small-4KB and big chains BOTH) resolves +VA 0x4b19b10 -> FB 0x660b10 (both chains converge; the guest's own PTs say +the object lives in vidmem). poke_vidmem writes at 0x660b10 (0x1, +0x40000000) and 0x660b14 (0x1, 0x40000000) LANDED (verified via +sema_vidmem_dump: [0x660b10]=0x11112222) yet the client does not flip and +perf still shows channelReadChannelMemdesc at 24.52%. CONCLUSION: the +REPORT_SEM VA 0x4b19b10 (and 0x4b17e30) are NOT the auth-tag sema the loop +reads - they are the GL-init release objects; the real auth-tag sema is at +pChannelBufferMemdesc + finishPayloadOffset + 4 (a VA and page we do not +yet know; finishPayloadOffset is NOT 0x19b10). THE DEFINITIVE INSTRUMENT: +a guest kprobe on channelReadChannelMemdesc (the offset argument = %si = +the authTagBufSemaOffset VALUE; and the pChannel pointer lets us read the +channel struct -> pChannelBufferMemdesc): `echo 'p:crcd +channelReadChannelMemdesc offset=%si' >> /sys/kernel/tracing/kprobe_events` ++ enable the event + read tracing pipe WHILE the loop runs. This names the +offset and the memdesc in one shot, then the sema page = walk(memdesc+off) +and the advance (RAM file if SYS; FB + poke if the converged mapping). + +**32. Session 105: KPROBE BLOCKED (notrace); THE BAR1-PATH ANALYSIS; THE CHANNEL-FIELD OFFSETS.** +(1) The kprobe on channelReadChannelMemdesc is IMPOSSIBLE: the guest's +dmesg says `trace_kprobe: Could not probe notrace function +channelReadChannelMemdesc [nvidia]` - the function is annotated notrace +(ftrace/perf-probe/BPF-kprobe all blocked). The tracefs filesystem had to +be mounted first (mount -t tracefs tracefs /sys/kernel/tracing) and it +already carries UVM kprobes from earlier sessions (uvmt/uvch/uvbp/uvtw/ +uvsv...). The static sec2utils functions (the callers) are absent from +kallsyms (inlined) - no probe target on that path. +(2) BAR1-path analysis: the guest driver's memdesc read uses +TRANSFER_FLAGS_USE_BAR1|SHADOW_ALLOC|SHADOW_INIT_MEM - a CPU read of the +GUEST'S BAR1 mapping, which the mdev mmap maps onto the HOST's vidmem +shadow (the same buffer poke_vidmem writes). The pokes at FB 0x660b10/ +0x660b14 LANDED (verified) yet the loop did not advance, so the memdesc +page is NOT the FB 0x660b10 the PT walk converged on (VA 0x4b19b10 belongs +to the GL-init release object, not the channel-buffer memdesc). +(3) The channel registration parse exposes the guest-visible channel +fields and their alloc-param offsets (gsp_rm_emulation.c +gsp_rm_register_channel: GSPRM_CHAN_GPFIFO_OFFSET/ENTRIES, ENGINE_TYPE, +CID, INSTANCE_MEM, USERD_MEM(+APERTURE), NOTIFIER_MEM(+APERTURE)) - the +channel-buffer/memdesc field (pChannelBufferMemdesc) is not among them; +the next session should dump a CE channel's FULL alloc params (the +service's rate-limited alloc dumps or a fresh boot) and look for the +channel-buffer VA/PA field, then sema = buffer + finishPayloadOffset + 4 +(per-channel offset, e.g. the 0x19b10-style for the GL channel). +(4) Fallback if the field is not in the params: per-VA content hunt for +the auth-tag sema VALUE (the retprobe would have given it; without +kprobes: the sema value = a small slot counter - scan the CE channel's +allocation candidates for a plausible counter that CHANGES across two +samples). + +**33. Session 106: THE CHANNEL BUFFER IS NOT A CLIENT PARAM; THE TAIL SWEEP AND HASH SCAN NEGATIVE.** +NV_CHANNEL_ALLOC_PARAMS (alloc_channel.h:300-341) layout mapped to the +service offsets: gpFifoOffset@8/entries@16, engineType@128, cid@132, +instanceMem@144, userdMem@168, ramfcMem@192, mthdbufMem@216, +hPhysChannelGroup/internalFlags, errorNotifierMem@248... - THE CHANNEL +BUFFER (pChannelBufferMemdesc) IS NOT IN THE PARAMS: it is a +driver-internal memdesc created by channelAlloc, so the alloc-params path +cannot name it (the kprobe/debuginfo path was the right one; it is +notrace-blocked). The finish-payload tail of the stream page +(0x114e98000 +0xd4..+0x118) was swept with 0x40000000 (18 u32s; the +pre-sweep values: +0xd8=0x68000000 finish marker, +0xe0=0x901025a6, ++0xe4..+0xec=0x8000xxxx methods) - no flip. A 16-page high-entropy scan of +0x114e98000..0x114ea8000 for the auth-tag SHA-256 array: ZERO high-entropy +32-byte blocks (the hash array is elsewhere / another allocation). NEXT: +(1) the memmgrMemDescBeginTransfer SHADOW insight - the read happens from +a FRESH kmalloc'd shadow copied from the memdesc each call; the shadow's +source = the memdesc's backing page; the memdesc page must be found by +TRANSFER-TIME properties (not the VA walk); (2) or a guest perf hardware +watchpoint on the driver's BAR1 ioremap VA + the (unknown) memdesc offset; +(3) or the service-side broad assist: increment a RANGE of plausible FB +sema cells periodically (blind but bounded), watching for the flip. + +**34. Session 107: FB ASSIST SWEEPS NEGATIVE - THE READ IS NOT FB-SOURCED; THE TWINS ARE THE NEXT TARGET.** +Two bounded FB "sema assist" sweeps (distinct values per cell, +0x40000000|idx): (a) the walked page 0x660b00..0x661000 (320 cells) - no +flip; (b) the CE-channel allocation cluster 0x8f000..0x90000 (1024 cells, +where the registered userds live) - no flip; perf re-confirmed +channelReadChannelMemdesc 23.13% (the loop still runs). CONCLUSION: the +memdesc read does NOT come from the FB shadow the CPU can poke - the +transfer's SHADOW_INIT must read the memdesc through a path that bypasses +these bytes, or the memdesc backing is SYSTEM memory via an unknown VA. +NEXT (one-minute test): the STREAM TWINS - pa 0x1026ad000 and +0x12afba000 hold BYTE-IDENTICAL copies of the CE method stream (verified at ++0x7c0/+0x880: same 0x68000000 finish marker area, same 0x200120a9/9c +methods) - one of them is very likely the memdesc BACKING in sysmem (the +driver copies the same stream per channel/boot). Sweep BOTH twins' +stream-area pages (reuse the assist code; 1 page each) and watch for the +flip; if it flips, bisect the winning page to the sema cell and make it +durable (host RAM file writes = zero-cost, no poke_vidmem needed for +sysmem). Failing that: content-diff the twins across a driver transition +(the memdesc copy updates during submits; the copy that CHANGES is the +backing). + +**35. Session 108: THE CONTENT-DIFF - 9 ACTIVE PAGES + THE CONSTANT-READ DEDUCTION.** +Page-hash diff over three +/-256KB windows (twins 0x1026ad000/0x12afba000, +stream 0x114e98000) across 15 s: NINE pages change: +0x102690000, 0x102698000 (440 word diffs - a buffer written continuously = +the push copy!), 0x1026a7000 (+0xe8: 934 -> 940 - a slow clean counter), +0x1026b3000 (+0x48: 32768 -> 1 transition), 0x1026b5000 (+0x244/+0x24c: +24987 -> 32765, two u16s at ~650/s), 0x1026c2000 (+0xe8: 232 -> 3, a +wrapping counter mod ~233), 0x1026e6000 (+0x18: 2764 -> 1930, +0x88/+0x90 +decreasing - countdowns), 0x114e6b000, 0x114e6e000. THE DEDUCTION: the +stuck loop's read value must be CONSTANT (its page shows no writes; the +GPU never advances the sema), so the auth-tag sema is a static cell - +likely zero - and the fix is to make the read NONZERO (advance it) at the +right page. None of the 9 changing pages is it (their cells move on their +own). NEXT: (1) restrict the search to CONSTANT pages among the CE +channel's candidate backing (the twins region pages that DID NOT change +but sit adjacent to the active set); (2) target the sema cell: the cell +that equals nextPut (small constant, e.g. 0) at the memdesc+off; sweep the +CONSTANT pages' zero u32s in a bounded ring around the twins/active set +with 1 (the sysmem write is free); (3) the durable service fold once +found. + +**36. Session 109: THE BURST vs HEARTBEAT DISCOVERY + THE 200ms READER.** +perf trace of the whole process over 10 s with -e ioctl: ZERO ioctls from +ANY thread; meanwhile the process shows a strict 200 ms pattern on thread +1788: `openat(filename ptr 0x5eaec740) -> read(fd 28, count 299) -> close` +(the transient fd the fd-sampler missed!) and thread 1793's 10 ms +futex re-arms. The perf-reported 24% channelReadChannelMemdesc (a kernel +function only reachable via RM calls) CONTRADICTS the zero-ioctl window: +the hot kernel work is BURSTY (periodic rounds between the 200 ms +heartbeats), not continuous. The 299-byte blob read every 200 ms = the +client's health poll (unnamed path; candidate: /proc/driver/nvidia/... +informational node - find it by reading /proc/PID/io or by narrowing the +sampler to the 200 ms phase). THE MODEL NOW: the client idles (futex +waits) + wakes every 200 ms to poll a node + periodically runs RM bursts +(the burst = a full retry round hitting channelReadChannelMemdesc). NEXT: +(1) name the 299-byte node (it may BE the wait's condition - a status the +service must change); (2) capture the BURST with perf (a long record + +per-thread breakdown) to see which thread runs the memdesc reads (the RM +burst thread) and correlate with the host-side RPC traffic in the same +window (the host dmesg RPC lines during a burst = the exact RM call!). + +**37. Session 110: THE LOOP IS CONTINUOUS INSIDE ONE LONG IOCTL (THE LOCK MODEL CLOSED).** +(1) The 299-byte/200ms reader: the filename pointer 0x5eaec740 is NEVER in +the client's maps (30 Hz sampler over 15 s, zero hits) yet openat returns +28 every cycle - a decode artifact/transience, and at 5 reads/s it is +irrelevant to the 24% blocker. Treat it as a RED HERRING. (2) A 20-second +perf record (80K samples): channelReadChannelMemdesc 24.65% + a scheduler +family (pick_next_task_fair/update_curr/pick_eevdf/__schedule/ +schedule_timeout/os_schedule ~30% total) = the SEC2 wait loop running +CONTINUOUSLY. The earlier zero-ioctl windows are explained: THE MAIN +THREAD SITS INSIDE ONE LONG-LIVED ioctl (entered hours ago; the loop runs +in the kernel inside it) - and that ioctl HOLDS THE RM WRITE LOCK, which +closes the lock model: tid 1791 waits on the RM read lock, new clients +block D-state (os_acquire_rwlock_write), all consistent. (3) PEBS/ +mem-loads events are unavailable in the guest perf (no memory-address +sampling) - the read address cannot be sampled. Instruments for the sema +address are now exhausted short of the notrace blocker or the blind +service-side assist. +NEXT: (1) the blind assist, refined: the loop's READ is CONTINUOUS and +the sema is CONSTANT - sweep ONLY constant small-value cells in the CE +candidate pages (see session 108's active set for the region), one value +at a time, 2 s per batch, watching the flip; (2) or a graft: since the +loop runs inside the client's blocked ioctl, unloading the CLIENT (kill) +releases the RM lock and the fresh client restarts the same phase - use +fresh-client runs for any A/B of service-side fixes; (3) the durable +service-side fold remains the endgame once the cell is known. + +**38. Session 111: THE FB IS FULLY DISPROVED (WHOLE-FB FILL, NO FLIP); THE SYS-FILL WEDGED THE GUEST (CYCLE NEXT).** +(1) scan_vidmem for the stream signatures (0x200121ed, 0x200120a9) and even +0x68: ZERO hits across 256MB - THE FB SHADOW WAS ALL ZEROS (before any +pokes): the GPU-side objects are never written in our emulation (the +driver->GPU DMA never runs), so the driver's memdesc SHADOW_INIT copies +zeros and the sec2 loop reads 0 forever (nextPut==authTagGetIndex==0 -> +infinite wait; the complete model). (2) The refined assist: swept FB +0x80000..0x90000 (16384 cells, distinct values; 60s/32KB via poke_vidmem) +- no flip. (3) THE DEFINITIVE TEST: `fill_vidmem 0x40000000` filled the +WHOLE 256MB FB shadow in-kernel (`vgsp: fill_vidmem value=0x40000000 +size=0x10000000 done`) - STILL NO FLIP: THE LOOP'S READ DOES NOT COME FROM +THE FB SHADOW AT ALL; the channel-buffer memdesc is SYS-BACKED (or another +backing) in our emulation, not the FB. (4) The follow-up SYS fill (128 +candidate pages near the active set overwritten with 0x40000000 via the +RAM file) CLOBBERED LIVE DRIVER DATA and WEDGED THE GUEST (SSH banner +timeout; QEMU alive) - the next session MUST CYCLE (fresh boot); lesson: +never bulk-overwrite pages adjacent to the identified active set (0x102698000 +was a hot staging buffer). +NEXT: (1) fresh cycle; (2) the sema is a SYS page - apply the full-page +marker fill ONLY to genuinely-constant, non-active pages (re-run the +content-diff first, then fill the constant set in small batches with a +client check after each, NOT the active neighborhoods); (3) or the +service-side durable route: have the SERVICE write the marker into the +channel-buffer page once it can resolve it (the memdesc VA = the CE +channel's buffer; the service's channel registration fields + the PT walk +give the page candidates). + +**39. Sessions 112b-114: THE SCRUB GATE FELL, THE COMPR WALL WAS DECODED AND FLIPPED, AND THE FIRST VULKAN SUBMISSION LANDED (VKFENCES IS THE NEW WALL).** +Context first (sessions 112b/112c, commits `97489187`/`cb7d8ce7`/`752e2b30`/ +`d6ccdaf3`/`8700f65a`): the CeUtils scrubber's finish payload is authored +in-service (bound to the EXACT 4096-entry channel), the client then leaves +`scrubCheckAndWaitForSize` permanently and runs +vkCreateImage -> vkGetImageMemoryRequirements -> vkAllocateMemory; the +remaining `-2` was decomposed to the guest-internal VidHeap `#149`-shape +call (attr COMPR_ANY + 2MB page + owner 0x474C0000) returning 0x51 +NV_ERR_NO_MEMORY; session 113 read that as a 64KB comptag/CBM alloc. This +session REFINED it: the COMPR+2MB class is not always 64KB - the r705 +runs show the *image data* alloc itself (size 0x140000 = 1.25 MB for +640x480x4 + pad, attr 0x1002000, flags 0xc10d) in exactly the same shape. +The "separate 64KB CBM" reading was too narrow; every alloc of the class +{attr&0xF000 == 0x2000, 2MB page, owner 0x474C0000} fails 0x51. + +**A. THE ZERO-TOUCH CREATE RECIPE (NEW, SCRIPTED).** Two live assists are +required per client, both proven again this session: (1) the +instance-phase marker clears at eglcore+0xB33D30 (t106/t99 `BC`, 5 +clears); (2) THE DEVICE-PHASE WAIT-WORD CHASER at eglcore+0xDCEBD9 +(t99 `BW`: word = *(holder+0x58)+*(u32)(obj2+0x60) with +obj2 = *(*(rbx+0x70)+0x58); write 1 when 0). Without (2) the client +reaches `VKDEVICE`'s poll(`/dev/nvidia0`, 10 ms) retry loop and sits +there INDEFINITELY (session-76 state; the a097 engine-monitor worker +keeps polling 0x2080a097 every 1.25 s regardless - its reply words are +still a red herring for the main-thread gate). With (1)+(2) the run is +`VKINSTANCE ok / VKDEVS 2 / VKDEVICE ok family=0` then straight to the +image path. `t99.py`-style attach works too: attaching `BW` to an +already-stuck client writes the word and the client immediately +proceeds (proven on a stuck pid 15797: WORDWROTE x2 -> VKDEVICE ok -> +VKFAIL -2 at the alloc). +NEW TOOLS: `t120.py` (BC+BW+single-shot COMPR->non-COMPR flip), +`t121.py` (BC+BW+full VidHeap trace from #120), `t122.py` (BC+BW+flip +for ANY COMPR+2MB internal alloc, size >= 0x10000), `a8clear.py` (the +no-ptrace count-gate clearer for base+0x2070b7c, reconstructed). + +**B. THE FLIP (THE DECISIVE EXPERIMENT).** With `t122.py`: at the +vidheap ioctl whose params match (attr&0xF000)==0x2000, owner +0x474C0000, size>=0x10000, the script clears attr bits 13:12 +(COMPR_ANY -> NONE) in the client's own params before the kernel sees +them. Result (r705c, reproducible twice on the same boot): +`HIT149 size=0x140000 attr=0x1002000 -> 0x1000000` (image data alloc +SUCCEEDS), a second `HIT149 size=0x140000 attr=0x1022004 -> 0x1020004` +(buffer-class COMPR alloc) also succeeds, and the client runs the WHOLE +pipeline for the first time in project history: +vkCreateImageView -> buffer -> shader module -> descriptor set +layout/pool -> pipeline layout -> compute pipeline -> command pool -> +command buffer -> fence -> vkResetFences -> **vkQueueSubmit OK** -> +`VKFAIL vkWaitForFences(dev, 1, &fence, VK_TRUE, 30000000000ull) -> 2 +(line 459)` - THE FIRST VULKAN SUBMISSION, then the FIRST FENCE TIMEOUT +(30 s, the client's own bound; the run exits rc=1 cleanly). + +**C. THE NEW WALL = THE FIRST FENCE RELEASE.** Host-side evidence in the +submit window: the client's GR channel registers +(`sema-chan: h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ent=32768 +userd=0xfee00 put=11 get=2 ring rc=0 pa=0x1217e3000 ap=1`) - the guest +driver published 11 GPFIFO entries, the emulation's reader consumed 2; +the service still reports the channel-init pushes unread +(`sema-entry: unread h=0xbeec0100 idx=2 va=0x4ca059c len=976 want=0 +exp=0 unread=1`) and `gr_wedge` grew 3 -> 9 over the runs. So the fence's +release is never authored after the submit - this is the session-76/93 +"wall #2" class (push-page readability/completion), now the ONLY gate +between the flipped client and the first `VKFRAME seq/DONE` + PPM. + +**D. DURABLE-FIX LEADS (for the COMPR class, no guest patching).** +(1) memmgrAllocHal_GM107 (570.144) downgrades COMPR_ANY -> NONE when +`pMemorySystemConfig->bDisableCompbitBacking` is set - the guest's +MEMORY_SYSTEM_STATIC_CONFIG comes from the GSP (the MEMSYS_GET_STATIC_ +CONFIG control 0x20800a1c / the GSP static info path); answering it (or +the same knob wherever the guest's config actually comes from) is the +host-side durable equivalent of the flip. 0x20800a1c was NEVER observed +in the RM_CONTROL log, so first find where the guest's config is fed +(control cache? GSP init args?) before coding the handler. +(2) Alternatively locate glcore's COMPR decision (attr construction at +eglcore+0x2c83fb/0x2c8465 and +0x50e038/0x50e894/0x50e8bd per session +113) and answer the capability that drives it. (3) The session-85 +device-local -2 story (2g mdev type) remains a prerequisite context. + +NEXT (ranked): (1) author the fence release for the first submit (the +push-page-5 content-match / pinning machinery; or have the service +complete the fence when it consumes the submit's GPFIFO entries); (2) +then the first VKFRAME seq/DONE + PPM; (3) fold the COMPR flip into a +host-side answer (see D); (4) the mission series (Xorg/GZDOOM, KVMFR, +CUDA 15/15 + soak, test series, video, Windows). + +**40. Session 115: THE FENCE WALL DECOMPOSED - THE CLIENT VAS PAGE DIRECTORY IS UNDISCOVERABLE AND THE PUSHES ARE NONCONTIGUOUS; RING PAGES ARE CONTENT-LOCATABLE AND INJECTABLE, BUT THE RING PAIR ALONE DOES NOT UNLOCK THE PUSH DECODE.** +(1) The fence timeout is upstream of everything else: the sema service cannot +decode the submit's GR pushes (`sema-entry: unread ... va=0x4ca0000 +len=1432` -> `skipping stale entry ... after 185-194 failed rescues`; +`work_gr submitted=0` - no GR forward ever starts). (2) The pump's push +reads resolve ONLY through the client VAS page directory (or pins). The +service has NO pdb for this client: `walk_probe` of the ring and push VAs +prints `discovered pdb=0x0 found=0` and no registered VASes; the r496f +vidmem pdb discovery never ran (it is only called from the RAM ring-scan +latch), and the r496j COPY_SERVER_PDES candidate (0x109b9c000) is garbage +(its content is not a page table). (3) THE RAM RING SCAN MISSES BY +CONSTRUCTION for this client: it sweeps [0, 6 GB) in 12 slices and gives +up (`found no candidate in 12 slice(s)`), but the sweep runs BEFORE the +client's pushes exist and never re-runs - while the ring page content is +provably in RAM (content-located below). (4) CONTENT LOCATION WORKS: the +GR ring page (ring VA 0x4cc0000, first entries {va=0x4ca0000 len=1432, +va=0x4ca0598 len=4}) was found in the guest RAM file by its first two GPFIFO +entries (at GPA 0x12C600000 then 0x12898c000 on later runs - per-run +allocations), and `sema_map_add` injection of the TRUE ring pair is +accepted (`INJECT va=0x4d20000 pa=0x13619d000 : ok`). BUT the pushes do not +become readable from the ring pair alone: the pushbuffer is NONCONTIGUOUS +(the window guess `ring_pa - 0x20000` reads zeros for both the GR and the +CE channels), the pushes are not CPU-mapped in the client process (the +ring GPA is absent from /proc/PID/pagemap of every rw-s nvidia mapping), +and a bottom-up search for the level-3 table (a 4 KB page whose index-2 +big/small entry maps into the ring's 2 MB window) found ZERO candidates in +RAM - the page tables are almost certainly in VIDMEM (r496f's note) or use +a shape the walker does not model. (5) The service's OWN successful read of +one CE init push (`sema-push: h=0xbeec0104 va=0x4d00000 ... rel=0x4b19b10 +pay=1 have_rel=1`, `ops_runs=0`) did NOT apply its release either - +whatever resolved that one read is not reproducible for the GR pushes. +NEW TOOLS: `fence_assist.py` (host: watches a `dmesg -w` capture for the +pump's entry decodes, content-locates the ring pages by their first two +entries in the RAM file, injects the pairs via sema_map_add - proven to +find and inject, but the ring pair alone is insufficient) and +`find_push.py` (guest: tests whether the client CPU-maps the ring GPA). +NEXT (ranked, concrete): (1) find the client VAS page directory: extract the +pdb from the GUEST kernel's RM objects (kcore walk of the VAS/GPU object +for this channel - the GPFIFO ring page content anchors the search), then +register it via the service's VAS table (a new sysfs/debugfs entry point +or the existing candidate path) - this is the "PDB injection" the mission +names; (2) or scan VIDMEM for the pdb with the ring pair as the verifier +(extend `svc_xvas_pdb_discover`'s trigger: call it from the ring-injection +path, not only from the RAM scan latch); (3) make the RAM ring scan re-arm +per channel with the pushes present (its 12-slice give-up is the reason +the auto-discovery never latches on this client); (4) then the submit +decodes, the release lands and `vkWaitForFences` returns -> VKFRAME. diff --git a/tools/guest-diag/wall2/README.md b/tools/guest-diag/wall2/README.md new file mode 100644 index 0000000..1ef9558 --- /dev/null +++ b/tools/guest-diag/wall2/README.md @@ -0,0 +1,85 @@ +# WALL #2 (GR channel-init release) — session 73 findings + tools + +## Goal +The vk_frame Vulkan client fails `vkCreateDevice -> VKFAIL -4` because the +kernel RM kicks its GR channel (h=0xbeee0100, ring VA 0x44a0000, push VA +0x4480000) and waits ~11 s per attempt for a completion the sema service +cannot apply: it cannot translate the push VA ("false translation for push +va=0x4480000 ... rescuing"), so no release is decoded and the wait times out. + +## Findings (r674/r675, 8+ client runs) +1. The push is RM-built (nvidia.ko), NOT UVM: a kprobe on + `uvm_hal_turing_host_semaphore_release` never fired during a client run. +2. nvidia.ko / nvidia-modeset.ko functions are `notrace` -> `kprobe_events` + refuses them ("Could not probe notrace function ... [nvidia]"). UVM module + probes DO work. QEMU gdbstub hardware breakpoints (hbreak) work fine. +3. The service's rescue matcher is CE-shaped only (hdr 0x20038090, hi==1, + lo&0xfff==slot, w5/v5 shape) -> it can never match a GR push. Entries 0/1/2 + are consumed as "skipping stale entry ... after ~150 failed rescues"; + gr_wedge=1. +4. Host-side content searches of /dev/shm/vgpu-guest-ram during the stall found + NO push data: neither the CE release group (0x20038090) nor the host-class + release header (0x20050017) appears outside driver code pages. +5. Ring entry encoding (from `channelFillGpFifo` in nvidia.ko and + `nvWriteGpEntry` in nvidia-modeset.ko): [0] = va & ~3, [1] = + ((va>>32)&0xff) | ((len<<8)&0x7ffffc00) | 0x80000000 (flag bit seen on + entry[1]). +6. Neither `channelFillGpFifo` (nvidia.ko) nor `nvWriteGpEntry` + (nvidia-modeset.ko) is called during the client run (gdb hbreaks, 0 hits + for the GR channel) -> the GR channel's ring entries are written by an + unidentified writer (possibly the emulation/GSP path); the push content + was never located in guest RAM. +7. RM window records + memdescs (host-readable, stable layout): + - a page holds the client channel's window records + {VA=0x4480000, SIZE=0x20000} (push) and {VA=0x44a0000, SIZE=0x40000} + (ring), each inside head/tail list nodes; + - memdescs `{SIZE, 0x03008000, FIRST_PA, ptr}` live in the same page: + the ring memdesc's FIRST_PA == the service's validated ring pa + (verified twice: 0x1158ce000, 0x129415000) -> memdesc PA == the PA of the + window's first VA page; + - the push memdesc `{0x20000, 0x03008000, PA}` was injected as + 0x4480000 -> PA (resolver4, r675: PA 0x11595c000; "injected mapping + va=0x4480000 ... (sysmem)" in dmesg) and the service then read the page + WITHOUT the "false translation" line - but decoded no release (the page + content does not look like the method stream; later candidates read as + zero). +8. The kernel-side stall shows no GSP RPCs and no ioctls; the wait function + `channelWaitForFinishPayload` (polls `channelReadChannelMemdesc(channel, + channel->0x50)`, which reads `*(channel+0xe8)+off`) was NEVER called during + a client run (gdb hbreak) -> the RM's wait path for this kick is elsewhere. + +## Tools (this directory) +- wall2_resolver2.py — v2: CE release-group content search + tuple arrays + (pa&~0xfff, strides 0x20/0x40/0x60). No candidates live. +- wall2_resolver3.py — v3: ring memdesc -> same-page push memdesc recipe. +- wall2_resolver4.py — v4: window-record pair + ring memdesc cross-check + (this one injected successfully; rdesc[0] bug fixed in v5). +- wall2_resolver5.py — v5: continuous scan (no ring-line gate), injects every + 0x20000 memdesc PA found in pages holding both window records. +- wall2_livecap{,2,3}.py — live stall-time scanners for the CE group and the + host release header (both negative; useful for re-checks). +- scan_r673*.py — RAM structure analysis (ring-PA refs, record pages, memdesc + scans). +- trace_push674.py — VA-reference tracer with direct-map pointer following. +- gdb675*.gdb — QEMU gdbstub scripts (channelFillGpFifo / nvWriteGpEntry + probes, channel struct dumps). + +## Next steps (ranked) +1. Find the GR ring-entry writer (gdb watchpoint on the ring page's CPU VA; + get the VA from the channel struct; find the channel struct via the ring + memdesc page's head/tail pointers). The writer's caller holds the push's + CPU mapping -> PA -> inject -> service decodes the release. +2. Or: gdb-break at the stall on the RM's wait (sample the sleeping task's + kernel stack; the current candidates never fired) and read the semaphore + value it polls; bump it host-side via the ring-page PA (the finish payload + may live at ring_cpu + channel->0x50). +3. Or: fix the service's rescue matcher to accept GR-shaped releases once the + push content is captured (see item 1). + +## Ops +- pkill -f self-matches: the bracket trick fails when the pattern text appears + elsewhere in the agent's own command line; kill by explicit PID from + `ps -eo pid,comm,args | awk '$2=="python3" && /<script>/ {print $1}'`. +- r674 wedged after a gdb-interrupted run; fresh cycle r675 used (no host + reboot). gdb674d/gdb675d sessions left attached (never kill gdbs). +- Never reload the guest nvidia modules; never kill clients mid-flight. diff --git a/tools/guest-diag/wall2/a8clear.py b/tools/guest-diag/wall2/a8clear.py new file mode 100644 index 0000000..261312a --- /dev/null +++ b/tools/guest-diag/wall2/a8clear.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +# a8clear.py - no-ptrace A8/NvOgl count clearer: finds each live vk_frame's +# libnvidia-eglcore base (the off==0 mapping) and writes 0 to the u32 at +# base+0x2070b7c whenever it is nonzero (the count gate t91 cleared via gdb). +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +log = set() +end = time.time() + DUR +while time.time() < end: + for pid in pids(): + base = None + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "libnvidia-eglcore" not in p[-1]: + continue + try: + off = int(p[2], 16) + except Exception: + continue + if off == 0: + base = int(p[0].split("-")[0], 16) + break + except Exception: + continue + if not base: + continue + w = base + 0x2070b7c + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, w, 0) + v = os.read(fd, 4) + if len(v) == 4: + n = int.from_bytes(v, "little") + if n: + os.lseek(fd, w, 0) + os.write(fd, b"\x00\x00\x00\x00") + k = (pid, n) + if len(log) < 40: + log.add(k) + print("A8CLEAR pid=%d count=%u -> 0" % (pid, n), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.5) +print("A8CLEAR-DONE", flush=True) diff --git a/tools/guest-diag/wall2/diag76.py b/tools/guest-diag/wall2/diag76.py new file mode 100644 index 0000000..989abd3 --- /dev/null +++ b/tools/guest-diag/wall2/diag76.py @@ -0,0 +1,47 @@ +import gdb + +inf = gdb.selected_inferior() +pid = inf.pid +maps = [] +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + maps.append((lo, hi, off, parts[5] if len(parts) > 5 else "")) + except Exception: + pass + +def resolve(addr): + for lo, hi, off, name in maps: + if lo <= addr < hi: + return "%s+0x%x" % (name.split('/')[-1], off + (addr - lo)) + return "?" + +gdb.execute("set pagination off") +print("=== THREADS ===") +for t in inf.threads(): + try: + t.switch() + except Exception as e: + print("switch err", e) + continue + pc = int(gdb.parse_and_eval("$pc")) + print("TID %s pc=%s" % (t.ptid, resolve(pc))) + try: + bt = gdb.execute("bt 12", to_string=True) + except Exception as e: + bt = "bt err %s" % e + for ln in bt.splitlines(): + ln = ln.strip() + if ln.startswith("#"): + try: + a = int(ln.split()[1], 16) + print(" ", ln, "=>", resolve(a)) + except Exception: + print(" ", ln) + else: + print(" ", ln) +print("DIAG-DONE") +gdb.execute("detach") +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/fastrun.sh b/tools/guest-diag/wall2/fastrun.sh new file mode 100644 index 0000000..73b9d80 --- /dev/null +++ b/tools/guest-diag/wall2/fastrun.sh @@ -0,0 +1,10 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /tmp/slotfill2.log 2>&1 & ) +sleep 0.5 +sudo timeout 60 strace -f -T -tt -e trace=ioctl -o /tmp/vk_strace3.out /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk77.out 2>&1 +echo "=== client ===" +cat /tmp/vk77.out +echo "=== last ioctls ===" +tail -12 /tmp/vk_strace3.out diff --git a/tools/guest-diag/wall2/fastrun2.sh b/tools/guest-diag/wall2/fastrun2.sh new file mode 100644 index 0000000..3c7b943 --- /dev/null +++ b/tools/guest-diag/wall2/fastrun2.sh @@ -0,0 +1,12 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /dev/null 2>&1 & ) +sleep 0.5 +sudo timeout 45 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk78.out 2>&1 +echo "=== client ===" +cat /tmp/vk78.out +echo "=== repeated controls (top) ===" +sudo dmesg -T | grep -oE "GSP_RM_CONTROL cmd=0x[0-9a-f]+" | sort | uniq -c | sort -rn | head -12 +echo "=== last controls ===" +sudo dmesg -T | grep "GSP_RM_CONTROL cmd" | tail -8 diff --git a/tools/guest-diag/wall2/fasttick.py b/tools/guest-diag/wall2/fasttick.py new file mode 100644 index 0000000..413d3ba --- /dev/null +++ b/tools/guest-diag/wall2/fasttick.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +# fasttick.py - watch the service log for the GR0 channel's ring resolution +# (sema-chan: h=0xbeec0100 ... ring rc=0 pa=0x...) and immediately hand the +# verified {ring VA -> ring PA} pair to the service (sema_pdb_discover), so +# the parked VAS pdb candidate can be verified before/at age-out. +import re, subprocess, sys, time + +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +RING_VA = 0x4cc0000 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = time.time() + dur + fired = set() + seen_pdbs = set() + print("# fasttick start (%.0fs)" % dur, flush=True) + while time.time() < end: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + tail = out.splitlines()[-120:] + for ln in tail: + m = re.search(r"sema-chan: h=0xbeec0100 eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if m: + ring_va = int(m.group(1), 16) + ring_pa = int(m.group(2), 16) + key = (ring_va, ring_pa) + if key not in fired: + fired.add(key) + cmd = "echo 0x%x,0x%x > %s" % (ring_va, ring_pa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("[%s] TRIGGER va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), ring_va, ring_pa, r.returncode), flush=True) + m2 = re.search(r"page directory (?:registered UNVERIFIED after \d+ ms parked \()?pdb=0x([0-9a-f]+) entries=(\d+)", ln) + if not m2: + m2 = re.search(r"VA space 0x[0-9a-f]+ page directory 0x([0-9a-f]+) entries=(\d+)", ln) + if m2: + key = int(m2.group(1), 16) + if key not in seen_pdbs: + seen_pdbs.add(key) + print("[%s] PDB candidate 0x%x entries=%s" % ( + time.strftime("%H:%M:%S"), key, m2.group(2)), flush=True) + time.sleep(0.3) + print("# fasttick end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/fence_assist.py b/tools/guest-diag/wall2/fence_assist.py new file mode 100644 index 0000000..4ea0b0c --- /dev/null +++ b/tools/guest-diag/wall2/fence_assist.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# fence_assist.py - content-locate the client's ring pages in guest RAM from the +# service's own GPFIFO entry decodes (dmesg) and inject the TRUE VA->PA pairs +# via sema_map_add, so the pump's push reads (and its pushbuffer pairing) have +# a correct anchor. +import mmap, re, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" + +def gpa_of(fo): + return fo if fo < 0x80000000 else fo - 0x80000000 + 0x100000000 + +def inject(va, pa, ln=0x1000, flag=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va & ~0xfff, pa & ~0xfff, ln, flag, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("INJECT va=0x%x pa=0x%x : %s" % (va & ~0xfff, pa & ~0xfff, + "ok" if r.returncode == 0 else "FAIL"), flush=True) + +def search(mm, pats, results): + CH = 8 << 20 + size = len(mm) + off = 0 + while off < size: + d = mm[off:off+CH] + for (name, pat) in pats.items(): + if name in results: + continue + i = d.find(pat) + if i >= 0: + results[name] = gpa_of(off + i) + off += CH - 32 + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + end = time.time() + dur + pats = {} # name -> pattern bytes + ring_va = {} # name -> ring VA + seen = set() # raw entry lines processed + done = set() # injected names + print("# fence_assist start (%.0fs)" % dur, flush=True) + while time.time() < end: + try: + with open("/tmp/opencode/dmsg_w.txt", "r", errors="ignore") as df: + tail = df.read().splitlines()[-6000:] + out = "\n".join(tail) + except OSError: + out = "" + tail = [] + tail = [ln for ln in tail if "sema-chan:" in ln or "sema-push" in ln] + cur = {} + for ln in tail: + m = re.search(r"sema-chan: entry\[(\d+)\] h=0x([0-9a-f]+) eng=\d+ raw=([0-9a-f]+) va=0x([0-9a-f]+)", ln) + if m: + idx, h, raw, va = int(m.group(1)), m.group(2), int(m.group(3), 16), int(m.group(4), 16) + cur.setdefault(h, {})[idx] = raw + if idx == 0: + ring_va[h] = va # entry[0] va is the push VA (ring_va - 0x20000) + for h, ents in cur.items(): + key = (h, tuple(sorted(ents.items()))) + if key in seen or len(ents) < 2: + continue + seen.add(key) + name = h + pat = b"".join(struct.pack("<Q", ents[i]) for i in sorted(ents)[:2]) + pats[name] = pat + print("# channel h=0x%s entries=%s pattern ready" % (h, sorted(ents)), flush=True) + if pats: + results = {} + threads = [] + # split the map among 4 threads + LO = 0x100000000; HI = 0x180000000 + size = len(mm) + step = size // 4 + class Sl(shim := object): + def __init__(self, a, b): + self.a, self.b = a, b + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + if off < 0x80000000 and b <= 0x80000000: pass + d = mm[off:off+CH] + for (nm, p) in list(pats.items()): + if nm in results: continue + i = d.find(p) + if i >= 0: + results[nm] = gpa_of(off + i) + off += CH - 32 + for k in range(4): + a = 0x80000000 + k * (0x80000000 // 4) + b = 0x80000000 + (k + 1) * (0x80000000 // 4) + t = threading.Thread(target=worker, args=(a, b)) + t.start(); threads.append(t) + for t in threads: t.join() + for nm, gpa in results.items(): + if nm in done: continue + done.add(nm) + # ring page content starts with entry[0]; the ring VA is the + # channel's gpfifo VA = we reconstruct: entry0 va is the push, + # ring = first "gpfifo=0x..." dmesg line for this h + m = re.search(r"sema-chan: h=0x%s eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+)" % nm, out) + rva = int(m.group(1), 16) if m else None + if rva: + print("RING FOUND h=0x%s gpfifo_va=0x%x -> gpa=0x%x" % (nm, rva, gpa), flush=True) + inject(rva, gpa, 0x1000) + time.sleep(0.6) + print("# fence_assist done, injected %d" % len(done), flush=True) + +main() diff --git a/tools/guest-diag/wall2/find_push.py b/tools/guest-diag/wall2/find_push.py new file mode 100644 index 0000000..4bf8ccc --- /dev/null +++ b/tools/guest-diag/wall2/find_push.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +# find_push.py - locate the client's CPU mapping that owns the GR ring page +# (GPA 0x12C600000 as content-located on the host) and print the CPU VA<-> +# GPU VA correspondence for the low GPU VAs (0x4ca0000 etc). +import os, struct, sys, subprocess, mmap, time + +RING_GPA = 0x12C600000 +PUSH_GVA = 0x4ca0000 +RING_GVA = 0x4cc0000 + +def pids(): + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + +def read_pagemap(pid, va): + try: + with open("/proc/%d/pagemap" % pid, "rb") as f: + f.seek((va // 4096) * 8) + d = f.read(8) + if len(d) < 8: + return None + e = int.from_bytes(d, "little") + if not (e >> 63) & 1: + return None + return (e & ((1 << 55) - 1)) << 12 + except OSError: + return None + +def main(): + for pid in pids(): + print("pid=%d" % pid) + try: + maps = open("/proc/%d/maps" % pid).read().splitlines() + except OSError: + continue + for line in maps: + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x1000000: + continue + # scan pages; look for RING_GPA + for va in range(lo, hi, 0x10000): + for off in range(0, 0x10000, 0x1000): + pa = read_pagemap(pid, va + off) + if pa == RING_GPA: + print("FOUND ring page: cpu_va=0x%x (map 0x%x-0x%x %s)" % (va+off, lo, hi, p[-1])) + delta_cpu = (va + off) - RING_GVA + push_cpu = PUSH_GVA + delta_cpu + print(" delta_cpu = 0x%x" % delta_cpu) + for gva in (PUSH_GVA, PUSH_GVA+0x1000, 0x4ca059c, 0x4ca0980, 0x4ca5870, 0x4903000): + cva = gva + delta_cpu + pa2 = read_pagemap(pid, cva & ~0xfff) + print(" gva=0x%x -> cpu=0x%x -> pa=%s" % (gva, cva, hex(pa2) if pa2 else None)) + return + print(" ring GPA 0x%x not found in this pid's mappings" % RING_GPA) + +main() diff --git a/tools/guest-diag/wall2/find_ring_va.py b/tools/guest-diag/wall2/find_ring_va.py new file mode 100644 index 0000000..c02ea40 --- /dev/null +++ b/tools/guest-diag/wall2/find_ring_va.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +# Guest-side: read the nv_vm_map_pages vmap areas via /proc/kcore and find the +# GPFIFO entry pattern (entry0 raw 0x0005980004480000 for va=0x4480000 len=1432). +import struct, sys +pat = bytes.fromhex("0000480400980500") # 0x0005980004480000 LE +class Kcore: + def __init__(self): + self.fd = open("/proc/kcore", "rb") + hdr = self.fd.read(64) + e_phoff = struct.unpack_from("<Q", hdr, 32)[0] + e_phentsize, e_phnum = struct.unpack_from("<HH", hdr, 54) + self.segs = [] + for i in range(e_phnum): + self.fd.seek(e_phoff + i * e_phentsize) + ph = self.fd.read(56) + p_type, _fl, p_off, p_va, _pa, p_sz = struct.unpack_from("<IIQQQQ", ph, 0) + if p_type == 1: + self.segs.append((p_va, p_va + p_sz, p_off)) + def read(self, va, n): + for lo, hi, off in self.segs: + if lo <= va and va + n <= hi: + self.fd.seek(off + (va - lo)) + return self.fd.read(n) + return None +kc = Kcore() +areas = [] +for line in open("/proc/vmallocinfo"): + p = line.split() + if len(p) < 2: continue + if "nv_vm_map_pages" not in line: continue + lo, hi = p[0].split("-") + areas.append((int(lo, 16), int(hi, 16))) +print("nv_vm_map_pages areas:", len(areas)) +for lo, hi in areas: + size = hi - lo + d = kc.read(lo, size) + if not d: + print(" 0x%x-0x%x unreadable" % (lo, hi)); continue + idx = d.find(pat) + print(" 0x%x-0x%x size=0x%x hit=%s" % (lo, hi, size, ("+0x%x -> VA 0x%x" % (idx, lo + idx)) if idx >= 0 else "no")) + if idx >= 0: + # dump the page around the hit + base = lo + (idx & ~0xfff) + page = kc.read(base, 0x100) + print(" page head: %s" % page.hex()) diff --git a/tools/guest-diag/wall2/gdb675.gdb b/tools/guest-diag/wall2/gdb675.gdb new file mode 100644 index 0000000..e4b73ce --- /dev/null +++ b/tools/guest-diag/wall2/gdb675.gdb @@ -0,0 +1,21 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc0d0b9f0 +condition 1 *(unsigned int*)($rdi+0x110) == 0xc46f +commands 1 +silent +printf "GRFILL ch=%p idx=%d len=%d\n", $rdi, $esi, $edx +printf " +0x048=%p +0x0b8=%p +0x0e8=%p\n", *(unsigned long*)($rdi+0x48), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0xe8) +printf " +0x060=%p +0x0c0=%p +0x0c8=%p +0x0d0=%p +0x0d8=%p\n", *(unsigned long*)($rdi+0x60), *(unsigned long*)($rdi+0xc0), *(unsigned long*)($rdi+0xc8), *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8) +printf " +0x0e0=%p +0x0f0=%p +0x0f8=%p +0x108=%p +0x140=%p +0x148=%p\n", *(unsigned long*)($rdi+0xe0), *(unsigned long*)($rdi+0xf0), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x108), *(unsigned long*)($rdi+0x140), *(unsigned long*)($rdi+0x148) +x/8gx $rdi+0x40 +x/8gx $rdi+0x80 +x/8gx $rdi+0xc0 +x/8gx $rdi+0xe0 +x/8gx $rdi+0x100 +x/8gx $rdi+0x140 +continue +end +continue diff --git a/tools/guest-diag/wall2/gdb675c.gdb b/tools/guest-diag/wall2/gdb675c.gdb new file mode 100644 index 0000000..25b5ca0 --- /dev/null +++ b/tools/guest-diag/wall2/gdb675c.gdb @@ -0,0 +1,12 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc0d0b9f0 +commands 1 +silent +printf "FILL ch=%p cls100=0x%x b8=0x%x stride48=0x%x idx=%d len=%d\n", $rdi, *(unsigned int*)($rdi+0x100), *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x48), $esi, $edx +printf " d0=%p d8=%p e8=%p f8=%p 60=%p\n", *(unsigned long*)($rdi+0xd0), *(unsigned long*)($rdi+0xd8), *(unsigned long*)($rdi+0xe8), *(unsigned long*)($rdi+0xf8), *(unsigned long*)($rdi+0x60) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdb675d.gdb b/tools/guest-diag/wall2/gdb675d.gdb new file mode 100644 index 0000000..f9dc460 --- /dev/null +++ b/tools/guest-diag/wall2/gdb675d.gdb @@ -0,0 +1,11 @@ +set pagination off +set confirm off +set print elements 0 +target remote :1234 +hbreak *0xffffffffc1605fd0 +commands 1 +silent +printf "NVWRITE ch=%p idx=%u len=%u b8=0x%x f68=%p f50=0x%x fb0=0x%x fc0=0x%x f198=%p\n", $rdi, $esi, $edx, *(unsigned long*)($rdi+0xb8), *(unsigned long*)($rdi+0x68), *(unsigned int*)($rdi+0x50), *(unsigned int*)($rdi+0xb0), *(unsigned int*)($rdi+0xc0), *(unsigned long*)($rdi+0x198) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbfill.gdb b/tools/guest-diag/wall2/gdbfill.gdb new file mode 100644 index 0000000..41d1ea3 --- /dev/null +++ b/tools/guest-diag/wall2/gdbfill.gdb @@ -0,0 +1,13 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +set $e = *(unsigned long*)($rdi+0x1428) +if *(unsigned short*)($e+0x2e) == 0xffff + set {unsigned short}($e+0x2e) = 0 + printf "FILLED slot at %p+0x2e -> 0\n", $e +end +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbgap3.sh b/tools/guest-diag/wall2/gdbgap3.sh new file mode 100644 index 0000000..bf19370 --- /dev/null +++ b/tools/guest-diag/wall2/gdbgap3.sh @@ -0,0 +1,28 @@ +#!/bin/bash +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk65.out 2>&1 &) +sleep 5 +PID=$(pgrep -x vk_frame | head -1) +cp /proc/$PID/maps /tmp/maps_d.txt +gdb -p $PID -batch -ex "set pagination off" -ex "bt 12" 2>&1 | grep -E "^#" > /tmp/bt_gap3.txt +python3 - <<'PY' +import re +maps=open('/tmp/maps_d.txt').read().splitlines() +segs=[] +for m in maps: + p=m.split() + if len(p)<6: continue + lo,hi=[int(x,16) for x in p[0].split('-')] + off=int(p[2],16) + segs.append((lo,hi,off,p[5])) +for line in open('/tmp/bt_gap3.txt'): + m=re.match(r'#(\d+)\s+0x([0-9a-f]+)', line) + if not m: continue + a=int(m.group(2),16) + for lo,hi,off,name in segs: + if lo<=a<hi: + print('#%s %s + 0x%x' % (m.group(1), name.split('/')[-1], off+(a-lo))) + break + else: + print('#%s 0x%x unmapped' % (m.group(1), a)) +PY diff --git a/tools/guest-diag/wall2/gdbparam.sh b/tools/guest-diag/wall2/gdbparam.sh new file mode 100644 index 0000000..3f30d49 --- /dev/null +++ b/tools/guest-diag/wall2/gdbparam.sh @@ -0,0 +1,16 @@ +#!/bin/bash +cd /tmp +sudo dmesg -C +( sudo bash /tmp/slotfill2.sh > /dev/null 2>&1 & ) +sleep 0.5 +(timeout 60 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk79.out 2>&1 &) +sleep 5 +PID=$(pgrep -x vk_frame | head -1) +gdb -p $PID -batch \ + -ex "set pagination off" \ + -ex "catch syscall ioctl" \ + -ex "condition 1 \$rsi == 0xc020462a" \ + -ex "continue" \ + -ex "printf \"ioctl nr=0x2a arg=%p\n\", \$rdx" \ + -ex "x/40wx \$rdx" \ + -ex "detach" 2>/dev/null | tail -45 diff --git a/tools/guest-diag/wall2/gdbpoll6.gdb b/tools/guest-diag/wall2/gdbpoll6.gdb new file mode 100644 index 0000000..d7fccfd --- /dev/null +++ b/tools/guest-diag/wall2/gdbpoll6.gdb @@ -0,0 +1,9 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +printf "CHK rsi=%d cnt=%d e0=%p v=%x\n", $rsi, *(int*)($rdi+0x22efc), *(unsigned long*)($rdi+0x1428), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)+($rsi<<4)) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbspin.sh b/tools/guest-diag/wall2/gdbspin.sh new file mode 100644 index 0000000..989e584 --- /dev/null +++ b/tools/guest-diag/wall2/gdbspin.sh @@ -0,0 +1,19 @@ +#!/bin/bash +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk57.out 2>&1 &) +sleep 25 +PID=$(pgrep -x vk_frame | head -1) +echo "pid=$PID" +BASE=$(grep 'eglcore' /proc/$PID/maps | head -1 | cut -d- -f1) +echo "base=$BASE" +SPIN=$(printf '0x%x' $((0x$BASE + 0xa10e90))) +echo "spin=$SPIN" +gdb -p $PID -batch \ + -ex "set pagination off" \ + -ex "break *$SPIN" \ + -ex "continue" \ + -ex "printf \"HIT rip=%p rdi=%p rsi=%p rdx=%p rbx=%p rbp=%p\n\", \$rip, \$rdi, \$rsi, \$rdx, \$rbx, \$rbp" \ + -ex "bt" \ + -ex "info registers r12 r13 r14 r15" \ + -ex "x/12i \$rip-24" \ + -ex "detach" 2>&1 | tail -40 diff --git a/tools/guest-diag/wall2/gdbu16.gdb b/tools/guest-diag/wall2/gdbu16.gdb new file mode 100644 index 0000000..61f335b --- /dev/null +++ b/tools/guest-diag/wall2/gdbu16.gdb @@ -0,0 +1,9 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +printf "CHK rsi=%d cnt=%d e0=%p u16@2e=%x u32@20=%x u32@0=%x\n", $rsi, *(int*)($rdi+0x22efc), *(unsigned long*)($rdi+0x1428), *(unsigned short*)(*(unsigned long*)($rdi+0x1428)+0x2e), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)+0x20), *(unsigned int*)(*(unsigned long*)($rdi+0x1428)) +continue +end +continue diff --git a/tools/guest-diag/wall2/gdbwatch.gdb b/tools/guest-diag/wall2/gdbwatch.gdb new file mode 100644 index 0000000..36b3339 --- /dev/null +++ b/tools/guest-diag/wall2/gdbwatch.gdb @@ -0,0 +1,18 @@ +set pagination off +set confirm off +break *POLLADDR +commands 1 +silent +set $e = *(unsigned long*)($rdi+0x1428) +watch *(unsigned short*)($e+0x2e) +printf "WATCH set at %p+0x2e\n", $e +disable 1 +continue +end +commands 2 +silent +printf "WRITE to slot+0x2e: old=%x new=%x by pc=%p\n", *(unsigned short*)($e+0x2e), $eax, $pc +bt 3 +continue +end +continue diff --git a/tools/guest-diag/wall2/gr_walk.py b/tools/guest-diag/wall2/gr_walk.py new file mode 100644 index 0000000..cbdb02d --- /dev/null +++ b/tools/guest-diag/wall2/gr_walk.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# gr_walk.py - find the RM window memdesc records in the guest's own RAM +# (via /proc/kcore direct map) and walk their kernel page-list pointers in +# the SAME pass (addresses go stale otherwise). Prints VA->PA candidates. +import sys, struct +sys.path.insert(0, "/tmp") +import guest_pin_stream as gps + +PAT_SIZE = struct.pack("<Q", 0x20000) +PAT_FLAGS = struct.pack("<Q", 0x03008000) + +def main(): + kc = gps.Kcore() + if not kc.load(): + print("# kcore load failed"); return 1 + pov = kc.page_offset_base + print("# page_offset_base=0x%x" % pov) + found = [] + CH = 1 << 20 + pa = 0x100000000 + end = 0x180000000 + while pa < end and len(found) < 12: + va = pov + pa + d = kc.read(va, CH) + if d: + st = 0 + while True: + i = d.find(PAT_SIZE, st) + if i < 0: + break + st = i + 1 + if i + 16 <= len(d) and d[i+8:i+16] == PAT_FLAGS: + found.append((pa + i, d[i:i+0x60])) + pa += CH + print("# memdesc records: %d" % len(found)) + for (gpa, rec) in found: + size, flags = struct.unpack_from("<QQ", rec, 0) + wpa = struct.unpack_from("<Q", rec, 0x10)[0] + kptrs = [struct.unpack_from("<Q", rec, o)[0] for o in (0x18, 0x20, 0x28, 0x30, 0x38)] + print("rec@0x%x size=0x%x pa=0x%x kptrs=%s" % + (gpa, size, wpa, ["0x%x" % k for k in kptrs])) + # walk each kptr for PA-like lists + for kp in kptrs: + if not gps.is_kern_ptr(kp): + continue + dd = kc.read(kp, 0x80) + if not dd: + print(" kptr 0x%x: unreadable" % kp); continue + words = struct.unpack_from("<%dQ" % (len(dd)//8), dd, 0) + pas = [w for w in words if 0x100000000 <= w < 0x180000000] + npas = [w for w in words if 0x100000000 <= (w & ~0xfff) < 0x180000000] + print(" kptr 0x%x: %s" % (kp, " ".join("0x%x" % w for w in words[:10]))) + if len(npas) >= 2: + print(" PA-like: %s" % ["0x%x" % w for w in npas[:16]]) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/gr_walk2.py b/tools/guest-diag/wall2/gr_walk2.py new file mode 100644 index 0000000..7a50bc8 --- /dev/null +++ b/tools/guest-diag/wall2/gr_walk2.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# gr_walk2.py - deeper walk: from the window memdesc records, follow the +# kernel pointers 2-3 levels and search for page-array records +# {vmap_cpu, flags, PA, 0x1000} (2+ consecutive) to recover the page lists. +import sys, struct +sys.path.insert(0, "/tmp") +import guest_pin_stream as gps + +PAT_SIZE = struct.pack("<Q", 0x20000) +PAT_FLAGS = struct.pack("<Q", 0x03008000) + +def is_pa(v): + return 0x100000000 <= v < 0x180000000 + +def page_array_at(kc, va, n=0x400): + """Return [(off, [PA,...])] for runs of {ptr,flags,PA,0x1000} records.""" + d = kc.read(va, n) + if not d: + return [] + out = [] + i = 0 + while i + 32 <= len(d): + p, fl, pa, z = struct.unpack_from("<4Q", d, i) + if z == 0x1000 and is_pa(pa) and gps.is_kern_ptr(p): + run = [pa] + j = i + 32 + while j + 32 <= len(d): + p2, fl2, pa2, z2 = struct.unpack_from("<4Q", d, j) + if z2 == 0x1000 and is_pa(pa2): + run.append(pa2) + j += 32 + else: + break + if len(run) >= 2: + out.append((i, run)) + i = j + else: + i += 8 + return out + +def main(): + kc = gps.Kcore() + if not kc.load(): + print("# kcore load failed"); return 1 + pov = kc.page_offset_base + # find memdesc records (fast direct-map scan) + found = [] + CH = 1 << 20 + pa = 0x100000000 + while pa < 0x180000000 and len(found) < 12: + d = kc.read(pov + pa, CH) + if d: + st = 0 + while True: + i = d.find(PAT_SIZE, st) + if i < 0: + break + st = i + 1 + if i + 16 <= len(d) and d[i+8:i+16] == PAT_FLAGS: + found.append((pa + i, d[i:i+0x40])) + pa += CH + print("# memdesc records: %d" % len(found)) + for (gpa, rec) in found: + size, flags = struct.unpack_from("<QQ", rec, 0) + wpa = struct.unpack_from("<Q", rec, 0x10)[0] + print("rec@0x%x size=0x%x pa=0x%x" % (gpa, size, wpa)) + kptrs = [struct.unpack_from("<Q", rec, o)[0] for o in range(0x18, 0x40, 8)] + for kp in kptrs: + if not gps.is_kern_ptr(kp): + continue + # level 1: read the target, look for page arrays directly + for arr in page_array_at(kc, kp): + print(" L1 kptr 0x%x: page-array run at +0x%x: %s" % + (kp, arr[0], ["0x%x" % x for x in arr[1][:12]])) + # level 2: follow pointers in the target + d = kc.read(kp, 0x100) + if not d: + continue + for o in range(0, 0x100, 8): + p2 = struct.unpack_from("<Q", d, o)[0] + if not gps.is_kern_ptr(p2): + continue + for arr in page_array_at(kc, p2): + print(" L2 0x%x -> 0x%x: page-array at +0x%x: %s" % + (kp, p2, arr[0], ["0x%x" % x for x in arr[1][:12]])) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/guest_words.sh b/tools/guest-diag/wall2/guest_words.sh new file mode 100755 index 0000000..6ac200c --- /dev/null +++ b/tools/guest-diag/wall2/guest_words.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# guest_words.sh - print "VA PA" for the +0x80 word of every 16 KB rw-s +# /dev/nvidiactl mapping of the vk_frame client (pagemap PFN -> PA). +PID=$(pgrep -x vk_frame | head -1) +[ -z "$PID" ] && exit 0 +sudo grep -E 'rw-s' /proc/$PID/maps | grep nvidiactl | while read -r line; do + RNG=$(echo "$line" | awk '{print $1}') + LO=${RNG%-*}; HI=${RNG#*-} + SZ=$((0x$HI - 0x$LO)) + [ "$SZ" -ne 16384 ] && continue + VP=$(( (0x$LO + 0x80) / 4096 )) + ENT=$(sudo dd if=/proc/$PID/pagemap bs=8 skip=$VP count=1 2>/dev/null | xxd -e -g8 | awk '{print $2}') + [ -z "$ENT" ] && continue + PFN=$(( 0x$ENT & 0x7fffffffffffff )) + PA=$(( PFN * 4096 + 0x80 )) + echo "0x$LO 0x$(printf %x $PA)" +done diff --git a/tools/guest-diag/wall2/host_resolve.py b/tools/guest-diag/wall2/host_resolve.py new file mode 100644 index 0000000..258d994 --- /dev/null +++ b/tools/guest-diag/wall2/host_resolve.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +# host_resolve.py - host-side push resolver: read the service's sema-pushall +# lines (pa=0x0, non-zero first8), find the content page in +# /dev/shm/vgpu-guest-ram by the push's first8 bytes, and inject the page +# mapping via sema_map_add. Replaces the guest-side kcore walker: the +# service already tells us push VA + content prefix, the host RAM file has +# the bytes. +import mmap, os, re, subprocess, sys, time + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +BASE = 0x80000000 # file offset = pa - BASE + +def inject(va_page, pa): + cmd = "echo 0x%x,0x%x,0x1000,1 > %s" % (va_page, pa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + ok = r.returncode == 0 + print("inject 0x%x -> 0x%x : %s" % (va_page, pa, "ok" if ok else r.stderr.decode().strip())) + return ok + +def main(): + seen = set() + f = open(RAM, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + injected = 0 + deadline = time.time() + (float(sys.argv[1]) if len(sys.argv) > 1 else 600) + while time.time() < deadline and injected < 40: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + lines = [l for l in out.splitlines() if "sema-pushall" in l] + for l in lines[-800:]: + m = re.search(r"va=(0x[0-9a-f]+) pa=0x0 len=\d+ allow=\d trusted=\d first8=([0-9a-f]{8}) ([0-9a-f]{8})", l) + if not m: + continue + va = int(m.group(1), 16) + w0, w1 = int(m.group(2), 16), int(m.group(3), 16) + if (w0, w1) == (0, 0) and (w0 == 0 or w1 == 0): + pass + if w0 == 0 and w1 == 0: + continue + key = (va & ~0xFFF, w0, w1) + if key in seen: + continue + pat = w0.to_bytes(4, "little") + w1.to_bytes(4, "little") + inpage = va & 0xFFF + # search with the in-page offset constraint + start = 0 + hit = None + while True: + i = mm.find(pat, start) + if i < 0: + break + if (i & 0xFFF) == inpage: + hit = i + break + start = i + 1 + if hit is None: + # fallback: any hit within the same page offset modulo + start = 0 + while True: + i = mm.find(pat, start) + if i < 0: + break + hit = i + break + # proceed only if the constraint is loose? no - skip + hit = None + if hit is None: + seen.add(key) + print("no match for va=0x%x first8=%08x %08x" % (va, w0, w1)) + continue + pa = (hit & ~0xFFF) + BASE + seen.add(key) + if inject(va & ~0xFFF, pa): + injected += 1 + time.sleep(0.5) + mm.close() + f.close() + print("done: injected=%d" % injected) + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/wall2/kc_dump_fn.py b/tools/guest-diag/wall2/kc_dump_fn.py new file mode 100644 index 0000000..e481d24 --- /dev/null +++ b/tools/guest-diag/wall2/kc_dump_fn.py @@ -0,0 +1,21 @@ +import struct, sys +VA = int(sys.argv[1], 16) +N = int(sys.argv[2], 16) if len(sys.argv) > 2 else 0x800 +f = open("/proc/kcore","rb") +hdr = f.read(64) +e_phoff = struct.unpack_from("<Q", hdr, 0x20)[0] +e_phentsize = struct.unpack_from("<H", hdr, 0x36)[0] +e_phnum = struct.unpack_from("<H", hdr, 0x38)[0] +segs = [] +for i in range(e_phnum): + f.seek(e_phoff + i*e_phentsize) + ph = f.read(e_phentsize) + p_type, p_flags, p_offset, p_vaddr, p_paddr, p_filesz, p_memsz = struct.unpack_from("<IIQQQQQ", ph, 0) + if p_type == 1: + segs.append((p_vaddr, p_memsz, p_offset)) +for v, sz, off in segs: + if v <= VA < v+sz: + f.seek(off + (VA - v)) + d = f.read(min(N, v+sz-VA)) + sys.stdout.write(d.hex()) + break diff --git a/tools/guest-diag/wall2/live_walk.py b/tools/guest-diag/wall2/live_walk.py new file mode 100644 index 0000000..48f131f --- /dev/null +++ b/tools/guest-diag/wall2/live_walk.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +# live_walk.py - live PT-walk + inject loop for the running client. +# Polls dmesg for the published page directory (pdb) and for the VAs the +# service is failing to read (sema-entry: unread / entry[] lines / registered +# gpfifo), walks each VA through the pdb with the service's own GMMU math, +# and injects resolved pairs via sema_map_add (ap=1 sysmem). +import mmap, re, subprocess, sys, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +SEMA_PA_MAX = 0x180000000 + +WINDOW_VAS = [0x44a0000, 0x4500000, 0x4560000, # rings + 0x4480000, 0x4485000, 0x4ca0000, 0x4ca5000, + 0x4d00000, 0x4b17000, 0x4020000, 0x4100000, 0x4160000, + 0x4213000, 0x45a0bb0, # ctx-init releases (deterministic) + 0x124935060, 0x102e34060, 0x108d4f060, 0x12a3a4060, 0x116c0b060, # notif families + 0x20400f000, 0x20400ffc0, 0x20480fef0, 0x204827f7c] # counter pages / tracker VAs + +mm = None +def off(pa): return pa if pa < 0x80000000 else pa - 0x80000000 +def rd64(pa): + if pa + 8 > SEMA_PA_MAX: return None + d = mm[off(pa):off(pa)+8] + return int.from_bytes(d, "little") if len(d) == 8 else None +def pde_ap(ap): return "FB" if ap == 1 else ("SYS" if ap in (2, 3) else None) +def pte_ap(ap): return "FB" if ap == 0 else (None if ap == 1 else "SYS") +def pte_addr(e, fb): return ((e >> 8) & 0x1FFFFFF) << 12 if fb else ((e >> 8) & 0x3FFFFFFFFFFF) << 12 +def pte_leaf(table, idx): + e = rd64(table + 8 * idx) + if e is None or not (e & 1): return None + ap = pte_ap((e >> 1) & 3) + if ap is None: return None + a = pte_addr(e, ap == "FB") + return (a, ap) if a + 8 <= SEMA_PA_MAX else None + +def walk(pdb, va, entries=4): + import math + root_bits = max(1, min(12, int(math.log2(entries or 1)))) + table = pdb + for level in range(3): + if level == 0: + bits, shift = root_bits, 49 - root_bits + else: + bits, shift = 9, 38 if level == 1 else 29 + idx = (va >> shift) & ((1 << bits) - 1) + e = rd64(table + 8 * idx) + if e is None or (e & 1): return None + ap = pde_ap((e >> 1) & 3) + if ap is None: return None + table = pte_addr(e, ap == "FB") + idx = (va >> 21) & 0xff + big = rd64(table + 16 * idx); small = rd64(table + 16 * idx + 8) + if big is None or small is None: return None + if not (small & 1): + sap = pde_ap((small >> 1) & 3) + if sap is not None: + r = pte_leaf(pte_addr(small, sap == "FB"), (va >> 12) & 0x1ff) + if r: return r[0] + (va & 0xfff), r[1] + if big & 1: + bap = pte_ap((big >> 1) & 3) + if bap is not None: + a = pte_addr(big, bap == "FB") + (va & ((1 << 21) - 1)) + if a + 8 <= SEMA_PA_MAX: return a, bap + else: + bap = pde_ap((big >> 1) & 3) + if bap is not None: + r = pte_leaf(pte_addr(big, bap == "FB"), (va >> 16) & 0x1f) + if r: return r[0] + (va & 0xffff), r[1] + return None + +def inject(va, pa, ln=0x1000): + cmd = "echo 0x%x,0x%x,0x%x,1 > %s" % (va & ~0xfff, pa & ~0xfff, ln, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print(" INJECT 0x%x -> 0x%x len=0x%x : %s" % + (va & ~0xfff, pa & ~0xfff, ln, "ok" if r.returncode == 0 else "FAIL"), flush=True) + return r.returncode == 0 + +def main(): + global mm + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 300 + end = time.time() + dur + pdbs = [] # (pdb_pa, entries) + done = set() # (pdb, va_page) + seen_va = set() + last_lines = 0 + print("# live_walk start (%.0fs)" % dur, flush=True) + while time.time() < end: + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + lines = out.splitlines()[-500:] + out = "\n".join(lines) + # pdb offers/registrations + for m in re.finditer(r"page directory (?:offered as candidate )?pdb=0x([0-9a-f]+) entries=(\d+)", out): + pdb, ent = int(m.group(1), 16), int(m.group(2)) + if (pdb, ent) not in pdbs: + pdbs.append((pdb, ent)) + print("# pdb 0x%x entries=%u" % (pdb, ent), flush=True) + # VAs the service is failing on + vas = [] + for m in re.finditer(r"sema-entry: unread h=0x[0-9a-f]+ idx=\d+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"sema-chan: entry\[\d+\] h=0x[0-9a-f]+ eng=\d+ raw=[0-9a-f]+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"registered gpfifo=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"GR ring h=0x[0-9a-f]+ rank=\d+ not at fb 0x[0-9a-f]+ yet \(e0=0x[0-9a-f]+ e1=0x[0-9a-f]+ va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"rel=0x([0-9a-f]+) ", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"notif=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"counter page 0x([0-9a-f]+) unreadable", out): + vas.append(int(m.group(1), 16)) + for m in re.finditer(r"release (?:learned|derived) h=0x[0-9a-f]+ pa=0x[0-9a-f]+ \(from release va=0x([0-9a-f]+)", out): + vas.append(int(m.group(1), 16)) + vas = [v for v in vas if 0x4000000 <= v < 0x2000000000] + for va in WINDOW_VAS + vas: + seen_va.add(va) + # resolve + inject through every pdb + for (pdb, ent) in pdbs: + for va in list(seen_va): + key = (pdb, va & ~0xfff) + if key in done: + continue + r = walk(pdb, va, ent) + if not r: + continue + done.add(key) + pa, ap = r + if ap != "SYS": + print("# va=0x%x -> pa=0x%x ap=%s (skip non-sys)" % (va, pa, ap), flush=True) + continue + print("# walk ok pdb=0x%x va=0x%x -> pa=0x%x" % (pdb, va, pa), flush=True) + inject(va, pa) + time.sleep(0.4) + mm.close(); f.close() + print("# live_walk done", flush=True) + +if __name__ == "__main__": + main() diff --git a/tools/guest-diag/wall2/nudge.sh b/tools/guest-diag/wall2/nudge.sh new file mode 100755 index 0000000..143b1cd --- /dev/null +++ b/tools/guest-diag/wall2/nudge.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# nudge.sh - host-side readiness-word nudger for the vk_frame wait. +# Polls the guest for the 16 KB nvidiactl mapping words (guest_words.sh), +# and writes 1 to any that are still 0 in the guest RAM file. +# Usage: nudge.sh <duration_seconds> +DUR=${1:-300} +F=/dev/shm/vgpu-guest-ram +END=$(( $(date +%s) + DUR )) +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/guest_words.sh ubuntu@127.0.0.1:/tmp/guest_words.sh >/dev/null 2>&1 +while [ "$(date +%s)" -lt "$END" ]; do + OUT=$(timeout 20 sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 "sudo bash /tmp/guest_words.sh" 2>/dev/null) + echo "[$(date +%H:%M:%S)] words:"; echo "$OUT" + echo "$OUT" | while read -r VA PA; do + [ -z "$PA" ] && continue + PAD=$((PA)); FOD=$((PAD - 0x80000000)) + [ $FOD -lt 0 ] && continue + CUR=$(sudo dd if=$F bs=4 skip=$((FOD/4)) count=1 2>/dev/null | xxd -p) + if [ "$CUR" = "00000000" ]; then + printf '\x01\x00\x00\x00' | sudo dd of=$F bs=1 seek=$FOD count=4 conv=notrunc 2>/dev/null + echo " NUDGE $PA (va=$VA) -> 1" + else + echo " skip $PA (va=$VA) = $CUR" + fi + done + sleep 2 +done +echo NUDGE-DONE diff --git a/tools/guest-diag/wall2/nudge2.sh b/tools/guest-diag/wall2/nudge2.sh new file mode 100755 index 0000000..16ebcd8 --- /dev/null +++ b/tools/guest-diag/wall2/nudge2.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# nudge2.sh - widened readiness-word nudger: ALL rw-s nvidia mappings <=256K, +# writes 1 to any +0x80 u32 that is 0. +DUR=${1:-300} +F=/dev/shm/vgpu-guest-ram +END=$(( $(date +%s) + DUR )) +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/guest_words2.sh ubuntu@127.0.0.1:/tmp/guest_words2.sh >/dev/null 2>&1 +while [ "$(date +%s)" -lt "$END" ]; do + OUT=$(timeout 12 sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 "sudo bash /tmp/guest_words2.sh" 2>/dev/null) + [ -n "$OUT" ] && echo "[$(date +%H:%M:%S)] words:" && echo "$OUT" + echo "$OUT" | while read -r VA PA SZ; do + [ -z "$PA" ] && continue + PAD=$((PA)); FOD=$((PAD - 0x80000000)); [ $FOD -lt 0 ] && FOD=$((PAD)) + CUR=$(sudo dd if=$F bs=4 skip=$((FOD/4)) count=1 2>/dev/null | xxd -p) + if [ "$CUR" = "00000000" ]; then + printf '\x01\x00\x00\x00' | sudo dd of=$F bs=1 seek=$FOD count=4 conv=notrunc 2>/dev/null + echo " NUDGE va=$VA pa=$PA -> 1" + fi + done + sleep 1.5 +done +echo NUDGE2-DONE diff --git a/tools/guest-diag/wall2/pdbtick.py b/tools/guest-diag/wall2/pdbtick.py new file mode 100644 index 0000000..d65b79d --- /dev/null +++ b/tools/guest-diag/wall2/pdbtick.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# pdbtick.py - content-locate the GR0 ring page (deterministic first three +# GPFIFO entries) in the guest RAM file, then hand the verified +# {ring VA -> ring PA} pair to the service (sema_pdb_discover), which +# verifies the parked COPY_SERVER_PDES candidates and falls back to the +# vidmem walk-root sweep. Runs while the client is alive. +import mmap, struct, subprocess, sys, threading, time + +F = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_pdb_discover" +PAT = struct.pack("<QQQ", 0x0005980004ca0000, 0x8000040004ca0598, + 0x0003d00004ca059c) +RING_VA = 0x4cc0000 + +def gpa_of(x): + return x if x < 0x80000000 else x - 0x80000000 + 0x100000000 + +def main(): + fobj = open(F, "rb") + mm = mmap.mmap(fobj.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + nthr = 8 + step = size // nthr + res = [] + lock = threading.Lock() + def worker(a, b): + CH = 8 << 20 + off = a + while off < b: + d = mm[off:off + CH] + i = d.find(PAT) + while i >= 0: + with lock: + res.append(off + i) + i = d.find(PAT, i + 1) + off += CH - 64 + ts = [] + for k in range(nthr): + a = k * step + b = (k + 1) * step if k < nthr - 1 else size + t = threading.Thread(target=worker, args=(a, b)); t.start(); ts.append(t) + for t in ts: t.join() + if not res: + print("RING NOT FOUND") + return None + hits = [gpa_of(x) for x in res] + print("ring candidates:", [hex(h) for h in hits], flush=True) + for gpa in hits: + cmd = "echo 0x%x,0x%x > %s" % (RING_VA, gpa, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True) + print("sema_pdb_discover 0x%x,0x%x rc=%d" % (RING_VA, gpa, r.returncode), flush=True) + return hits + + +def loop(): + import time as _t + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 120 + end = _t.time() + dur + fired = set() + while _t.time() < end: + hits = main() + if hits: + new = [h for h in hits if h not in fired] + if new: + fired.update(new) + _t.sleep(1.0) + print("# pdbtick end", flush=True) + +loop() diff --git a/tools/guest-diag/wall2/perfstall5.sh b/tools/guest-diag/wall2/perfstall5.sh new file mode 100644 index 0000000..ae1c161 --- /dev/null +++ b/tools/guest-diag/wall2/perfstall5.sh @@ -0,0 +1,9 @@ +#!/bin/sh +cd /tmp +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk55.out 2>&1 &) +sleep 25 +PID=$(pgrep -x vk_frame | head -1) +cp /proc/$PID/maps /tmp/vk_maps2.txt +sudo perf record -p $PID -g -o /tmp/perf6.data -- sleep 3 2>/dev/null +sudo perf script -i /tmp/perf6.data -F ip,dso 2>/dev/null | grep eglcore | awk '{print $1}' | sort | uniq -c | sort -rn | head -8 > /tmp/spin_addrs.txt +cat /tmp/spin_addrs.txt diff --git a/tools/guest-diag/wall2/pt_bind.py b/tools/guest-diag/wall2/pt_bind.py new file mode 100644 index 0000000..126ed01 --- /dev/null +++ b/tools/guest-diag/wall2/pt_bind.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# pt_bind.py - bind GMMU PT pages to VA bases and read out VA->PA for the +# client's ring/window family. Strategy: +# 1. ring-page candidates: pages whose 8B-aligned u64 pairs look like +# GPFIFO entries (e0 = push VA & ~3 in the 0x44x-0x4ex families with +# va<4G; e1 top byte 0; length bits sane). +# 2. anchor: for each candidate PA and each family VA, check every 4KB +# page for entry idx=(VA>>12)&0x1ff whose PA part == candidate PA. +# A hit binds (PT page, VA, PA). +# 3. read out: from bound PT pages, decode entries for the whole VA list. +import mmap, struct, sys + +F = "/dev/shm/vgpu-guest-ram" +FAMILY = [ + 0x4020000, 0x4100000, 0x4160000, # r689 eng rings + 0x44a0000, 0x4500000, 0x4560000, # r690 ring VAs + 0x4480000, 0x4485000, 0x4480980, 0x4480f1c, + 0x4ca0000, 0x4ca5000, 0x4d00000, 0x4b17000, +] + +def gpa_to_off(pa): + return pa if pa < 0x80000000 else pa - 0x80000000 + +def off_to_gpa(off): + return off if off < 0x80000000 else off + 0x80000000 + +def pte_pa(v): + return (v >> 8) << 12 + +def main(): + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + print("# file size 0x%x" % size) + + # ---- pass 1: ring-page candidates by content ---- + cands = [] + pos = 0 + step = 4096 + while pos + step <= size: + d = mm[pos:pos+step] + hits = [] + for i in range(0, step, 8): + e0 = int.from_bytes(d[i:i+4], "little") + e1 = int.from_bytes(d[i+4:i+8], "little") + if 0x4400000 <= e0 <= 0x4e00000 and (e0 & 3) == 0 and (e1 & 0xff) == 0: + lw = (e1 >> 10) & 0x1fffff + if 0x40 <= lw <= 0x100000: + hits.append((i, e0, lw << 2)) + if len(hits) >= 3: + cands.append((off_to_gpa(pos), hits)) + if len(cands) >= 12: + break + pos += step + print("# ring-page candidates: %d" % len(cands)) + for pa, hits in cands: + print("cand pa=0x%x hits=%d first=%s" % + (pa, len(hits), ["0x%x/0x%x" % (h[1], h[2]) for h in hits[:4]])) + if not cands: + return 1 + + # ---- pass 2: anchor PT pages for candidate PAs x family VAs ---- + idxs = {} + for va in FAMILY: + idxs.setdefault((va >> 12) & 0x1ff, []).append(va) + cand_pas = set(pa for pa, _ in cands) + bound = [] + pos = 0 + while pos + step <= size: + pg = off_to_gpa(pos) + # skip scanning the candidate pages themselves + if pg not in cand_pas: + d = mm[pos:pos+step] + for idx, vas in idxs.items(): + o = idx * 8 + v = int.from_bytes(d[o:o+8], "little") + if v & 1: + pa = pte_pa(v) + if pa in cand_pas: + bound.append((pg, idx, vas, pa, v & 0xff)) + pos += step + print("# PT anchors found: %d" % len(bound)) + for pg, idx, vas, pa, fl in bound[:20]: + print("PT pa=0x%x entry[0x%x] flags=0x%x -> PA=0x%x (VA cands %s)" % + (pg, idx, fl, pa, ["0x%x" % x for x in vas])) + + # ---- pass 3: read out PT entries for the whole family from each bound PT page ---- + seen_pt = set() + for pg, idx, vas, pa, fl in bound: + if pg in seen_pt: + continue + seen_pt.add(pg) + off = gpa_to_off(pg) + d = mm[off:off+4096] + print("== PT page pa=0x%x family readout:" % pg) + for va in FAMILY: + i = (va >> 12) & 0x1ff + v = int.from_bytes(d[i*8:i*8+8], "little") + if v & 1: + print(" va=0x%x idx=0x%x -> pa=0x%x (raw=0x%x fl=0x%x)" % + (va, i, pte_pa(v), v, v & 0xff)) + else: + print(" va=0x%x idx=0x%x -> absent (raw=0x%x)" % (va, i, v)) + mm.close(); f.close() + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/run75g.sh b/tools/guest-diag/wall2/run75g.sh new file mode 100755 index 0000000..001a80f --- /dev/null +++ b/tools/guest-diag/wall2/run75g.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# run75g.sh - trace + record-word sweep on r680. +cd /tmp/opencode +rm -f /tmp/opencode/t75g-sweep.log +sshpass -p ubuntu scp -o StrictHostKeyChecking=no -P 2222 /tmp/opencode/t75g.py ubuntu@127.0.0.1:/tmp/t75g.py >/dev/null 2>&1 +( + sshpass -p ubuntu ssh -o StrictHostKeyChecking=no -p 2222 ubuntu@127.0.0.1 \ + 'sudo bash /tmp/slotfill2.sh > /tmp/slotfill75g.log 2>&1 & sleep 0.5; timeout 400 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk_t75g.out 2>&1 & sleep 8; PID=$(pgrep -x vk_frame | head -1); echo "PID=$PID"; sudo timeout 380 gdb -q -batch -p $PID -x /tmp/t75g.py > /tmp/t75g.out 2>&1; echo GDB_DONE rc=$?' +) > /tmp/opencode/t75g-ssh.log 2>&1 & +GPID=$! +sleep 35 +setvar() { echo "$2" | sudo tee /sys/module/vgsp/parameters/$1 >/dev/null; } +mark() { echo "[$(date +%s.%N)] $*" >> /tmp/opencode/t75g-sweep.log; } +# phase 1: baseline echo (client already polling) +mark "baseline echo" +sleep 20 +# phase 2: all-zero records +setvar wall2_a097_rec0 0; setvar wall2_a097_rec1 0; setvar wall2_a097_rec2 0; setvar wall2_a097_rec3 0 +setvar wall2_a097_rec4 0; setvar wall2_a097_rec5 0; setvar wall2_a097_rec6 0; setvar wall2_a097_rec7 0 +mark "records all zero"; sleep 20 +# phase 3: progress=100 +setvar wall2_a097_rec1 100 +mark "rec1=100"; sleep 20 +# phase 4: progress=1000 +setvar wall2_a097_rec1 1000 +mark "rec1=1000"; sleep 20 +# phase 5: progress=100000 +setvar wall2_a097_rec1 100000 +mark "rec1=100000"; sleep 20 +# phase 6: progress=1000000 + rec0=0 +setvar wall2_a097_rec1 1000000 +setvar wall2_a097_rec0 0 +mark "rec1=1000000 rec0=0"; sleep 20 +# phase 7: all ones +setvar wall2_a097_rec0 3735928559; setvar wall2_a097_rec1 3735928559; setvar wall2_a097_rec2 3735928559; setvar wall2_a097_rec3 3735928559 +mark "rec0-3=0xdeadbeef"; sleep 20 +# phase 8: back to echo +setvar wall2_a097_rec0 4294967295; setvar wall2_a097_rec1 4294967295; setvar wall2_a097_rec2 4294967295; setvar wall2_a097_rec3 4294967295 +mark "baseline echo again" +wait $GPID +echo SWEEP-DONE diff --git a/tools/guest-diag/wall2/runbook.py b/tools/guest-diag/wall2/runbook.py new file mode 100644 index 0000000..256d255 --- /dev/null +++ b/tools/guest-diag/wall2/runbook.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +# runbook.py - watch for the GR0 channel ring registration, hand the pair to +# sema_pdb_discover, then (after the pdb anchor) issue bounded +# sema_chan_rewind calls so entries skipped before the anchor re-decode. +import re, subprocess, sys, time + +PDB = "/sys/module/vgsp/parameters/sema_pdb_discover" +REW = "/sys/module/vgsp/parameters/sema_chan_rewind" + +def dmesg_tail(n=150): + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + return out.splitlines()[-n:] + +def main(): + dur = float(sys.argv[1]) if len(sys.argv) > 1 else 180 + end = time.time() + dur + fired = set() + rewinds = {} + print("# runbook start (%.0fs)" % dur, flush=True) + while time.time() < end: + for ln in dmesg_tail(): + m = re.search(r"sema-chan: h=0x([0-9a-f]+) eng=\d+ cid=\d+ gpfifo=0x([0-9a-f]+) ent=\d+ userd=0x[0-9a-f]+ as=\d+ put=\d+ get=\d+ ring rc=0 pa=0x([0-9a-f]+)", ln) + if not m: + continue + h = int(m.group(1), 16) + gv = int(m.group(2), 16) + rp = int(m.group(3), 16) + key = (h, gv, rp) + if key in fired: + # after the anchor has had time to land, rewind this channel + if h == 0xbeec0100: + t0 = rewinds.setdefault(key, time.time()) + n = int((time.time() - t0) // 8) + done = rewinds.get(("done", key), 0) + if n > done: + subprocess.run(["sudo", "sh", "-c", + "echo all > %s" % REW], capture_output=True) + rewinds[("done", key)] = n + print("[%s] REWIND all (h=0x%x, n=%d)" % ( + time.strftime("%H:%M:%S"), h, n), flush=True) + continue + fired.add(key) + r = subprocess.run(["sudo", "sh", "-c", + "echo 0x%x,0x%x > %s" % (gv, rp, PDB)], + capture_output=True) + print("[%s] TRIGGER h=0x%x va=0x%x pa=0x%x rc=%d" % ( + time.strftime("%H:%M:%S"), h, gv, rp, r.returncode), flush=True) + # report decodes/releases + out = subprocess.run(["sudo", "dmesg"], capture_output=True).stdout.decode(errors="ignore") + for ln in out.splitlines()[-8:]: + if ("sema-push: h=0xbeec0100" in ln and "have_rel=1" in ln) or \ + "registered after verified" in ln or "rewind" in ln or \ + ("sema-push: h=0xbeec0100" in ln and "len=1432" in ln): + print("LOG %s" % ln.strip()[-170:], flush=True) + time.sleep(0.4) + print("# runbook end", flush=True) + +main() diff --git a/tools/guest-diag/wall2/scan_r673.py b/tools/guest-diag/wall2/scan_r673.py new file mode 100644 index 0000000..71a0128 --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +# Analyze the live r673 guest RAM: locate every reference to the validated ring +# PA (0x12a783000) including flag-bit variants, and characterize the record +# arrays around each hit. Also dump the two pages named in the handoff. +import numpy as np, struct, sys, time + +RAM="/dev/shm/vgpu-guest-ram" +RING=0x12a783000 +MASK=~0xfff & ((1<<64)-1) + +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) + +img=open(RAM,"rb") +import os +fsz=os.path.getsize(RAM) +print("ram size 0x%x" % fsz) + +t0=time.time() +hits=[] +CH=64<<20 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0] + for i in idx: + off=pos+int(i)*8 + val=int(a[i]) + hits.append((off,val)) +print("hits=%d scan=%.1fs" % (len(hits), time.time()-t0)) +for off,val in hits: + print(" off=0x%x gpa=0x%x val=0x%x (low=0x%x)" % (off,off2gpa(off),val,val&0xfff)) + +# characterize each hit page +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +def plausible_pa(v): return 0x100000000 <= (v & MASK) < 0x180000000 +def ptrish(v): + return (0xffff800000000000 <= v < 0xffffffffffffffff) + +for off,val in hits: + gpa=off2gpa(off) + page=gpa & MASK + rec_off=gpa-page + d=rd(page,0x1000) + print("=== hit gpa=0x%x page=0x%x rec_off=0x%x ===" % (gpa,page,rec_off)) + for stride in (0x20,0x40,0x60,0x30,0x50,0x28,0x48): + # record start such that the hit qword falls inside the record + for rel in range(0,stride,8): + start=rec_off-rel + if start<0: continue + # walk records; count plausible PA fields anywhere in each record + n_pa=0; n_ptr=0; n_1000=0 + for o in range(start,0x1000-stride,stride): + rec=d[o:o+stride] + qs=struct.unpack_from("<%dQ"%(stride//8),rec,0) + if any(plausible_pa(q) for q in qs): n_pa+=1 + if any(ptrish(q) for q in qs): n_ptr+=1 + if 0x1000 in qs: n_1000+=1 + if n_pa>=8: + print(" stride=0x%x start=0x%x: PA-records=%d ptr-records=%d 0x1000=%d" % (stride,start,n_pa,n_ptr,n_1000)) + # dump the first 0x100 bytes of the page and around the hit + print(" page head: %s" % d[:0x60].hex()) + s=max(0,rec_off-0x40); e=min(0x1000,rec_off+0x80) + print(" around hit [0x%x..0x%x]: %s" % (s,e,d[s:e].hex())) + +# dump the two named pages +for pa in (0x101499000,0x127cf2000): + d=rd(pa,0x1000) + print("=== page 0x%x ===" % pa) + if d is None: print(" unreadable"); continue + for o in range(0,0x120,0x20): + qs=struct.unpack_from("<4Q",d,o) + print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+qs)) diff --git a/tools/guest-diag/wall2/scan_r673b.py b/tools/guest-diag/wall2/scan_r673b.py new file mode 100644 index 0000000..11613b1 --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673b.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# r673: dump the full record page 0x101499000 and search RAM for the ring/push +# VA structures to identify which record is the 16-page push (VA 0x4500000). +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +MASK=~0xfff & ((1<<64)-1) +img=open(RAM,"rb") +fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +print("== full record page 0x101499000 ==") +d=rd(0x101499000,0x1000) +nrec=0 +for o in range(0,0x1000,32): + p,f,pa,sz=struct.unpack_from("<QQQQ",d,o) + if p==0 and f==0 and pa==0 and sz==0: continue + nrec+=1 + print(" rec[%3d] vmap=%016x flags=%016x pa=%012x size=0x%x" % (o//32,p,f,pa,sz)) +print("nonempty records:",nrec) + +print() +print("== search RAM for qword 0x4520000 (ring VA) ==") +t0=time.time(); CH=64<<20 +for pat_name,pat in (("0x4520000",0x4520000),("0x4500000",0x4500000)): + print("-- pattern %s --" % pat_name) + hits=[] + for pos in range(0,fsz,CH): + img.seek(pos); dd=img.read(min(CH,fsz-pos)) + n=len(dd)//8 + a=np.frombuffer(dd[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(pat))[0] + for i in idx: + hits.append(pos+int(i)*8) + print(" hits=%d (%.1fs)" % (len(hits),time.time()-t0)) + for off in hits[:40]: + gpa=off2gpa(off); page=gpa&MASK + # show 32 bytes before/after + dd=rd(page,0x1000) + if not dd: continue + ro=gpa-page + s=max(0,ro-32); e=min(0x1000,ro+64) + print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,dd[s:e].hex())) + if len(hits)>40: print(" ... %d more" % (len(hits)-40)) diff --git a/tools/guest-diag/wall2/scan_r673c.py b/tools/guest-diag/wall2/scan_r673c.py new file mode 100644 index 0000000..db69b5d --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673c.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +# r673: find the push page. Patterns: ring entry raw qwords, the sem VA, and +# method-stream signatures. Dump contexts. +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +pats=[ + ("entry0raw", 0x0005980004500000), + ("entry1raw", 0x0000040004500598), + ("entry2raw", 0x0003d0000450059c), + ("sem_va", 0x4500598), + ("sem_va2", 0x450059c), + ("m20038090", 0x20038090), + ("push_va", 0x4500000), +] +CH=64<<20 +for name,pat in pats: + hits=[] + t0=time.time() + for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(pat))[0] + for i in idx: hits.append(pos+int(i)*8) + print("== %s pat=0x%x hits=%d (%.1fs)" % (name,pat,len(hits),time.time()-t0)) + for off in hits[:12]: + gpa=off2gpa(off); page=gpa&~0xfff + d=rd(page,0x1000) + ro=gpa-page + s=max(0,ro-24); e=min(0x1000,ro+40) + print(" gpa=0x%x page=0x%x off=0x%x ctx=%s" % (gpa,page,ro,d[s:e].hex() if d else "?")) + if len(hits)>12: print(" ... %d more" % (len(hits)-12)) diff --git a/tools/guest-diag/wall2/scan_r673d.py b/tools/guest-diag/wall2/scan_r673d.py new file mode 100644 index 0000000..1fa89ca --- /dev/null +++ b/tools/guest-diag/wall2/scan_r673d.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# r673 deep-dive: all ring-PA refs with low bits, ring page dump, RM record +# pages, and a RAM-wide scan for 0xa3bf20 VA-table records mentioning 0x4500000. +import numpy as np, struct, os, time + +RAM="/dev/shm/vgpu-guest-ram" +RING=0x12a783000; TAG=0xa3bf20 +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) + +MASK=~0xfff & ((1<<64)-1) +print("== all qwords (v & ~0xfff)==ring_pa ==") +CH=64<<20 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero((a & np.uint64(MASK))==np.uint64(RING))[0] + for i in idx: + off=pos+int(i)*8 + gpa=off2gpa(off); page=gpa&~0xfff + dd=rd(page,0x1000); ro=gpa-page + s=max(0,ro-32); e=min(0x1000,ro+48) + print(" gpa=0x%x low=0x%x ctx=%s" % (gpa,int(a[i])&0xfff,dd[s:e].hex() if dd else "?")) + +print() +print("== ring page 0x%x first 0x120 ==" % RING) +d=rd(RING,0x120) +for o in range(0,0x120,0x20): + print(" +0x%03x: %s" % (o,d[o:o+0x20].hex())) + +for pa in (0x1013a6000,0x104acb000,0x114760000,0x1202cd000,0x13e2c1000,0x13e62e000): + print() + print("== page 0x%x first 0x140 ==" % pa) + d=rd(pa,0x140) + for o in range(0,0x140,0x20): + q=struct.unpack_from("<4Q",d,o) + print(" +0x%03x: %016x %016x %016x %016x" % ((o,)+q)) + +print() +print("== RAM-wide: 32-byte records {va, cpu, 0xa3bf20, pages} with va==0x4500000 ==") +t0=time.time() +found=0 +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(TAG))[0] + for i in idx: + off=pos+int(i)*8 + gpa=off2gpa(off); page=gpa&~0xfff + dd=rd(page,0x1000) + ro=gpa-page + if ro>=16 and ro+8<=0x1000: + va,cpu=struct.unpack_from("<QQ",dd,ro-16) + pgs=struct.unpack_from("<Q",dd,ro+8)[0] + if va==0x4500000 or (0x4400000<=va<=0x4600000): + print(" rec gpa=0x%x va=0x%x cpu=0x%x pages=0x%x" % (gpa,va,cpu,pgs)) + found+=1 +print("total", found, "(%.1fs)" % (time.time()-t0)) diff --git a/tools/guest-diag/wall2/session73-notes.md b/tools/guest-diag/wall2/session73-notes.md new file mode 100644 index 0000000..d2079d3 --- /dev/null +++ b/tools/guest-diag/wall2/session73-notes.md @@ -0,0 +1,51 @@ +# SESSION 73 notes (wall #2 live validation, r674 + r675) + +## r674 cycle results (8 client runs, all VKFAIL -4) +- Client vk_frame reproducibly fails at vkCreateDevice (-4) as before. +- wall2_resolver2 (content-first CE group 0x20038090 + tuple arrays): no candidates found + during the stall; tuple arrays contain the ring PA but not the push. +- livecap/livecap2/livecap3 (0x20038090 shape / 0x20050017 host-release shape): NO data + hits in guest RAM during the stall; only driver CODE pages match. +- Conclusion: the GR channel-init push content is NOT findable in /dev/shm/vgpu-guest-ram + by these patterns, even live. (The ring page content IS there and matches the service's + decode.) + +## Structural findings +- The push is RM-built (nvidia.ko), NOT UVM: kprobe on + uvm_hal_turing_host_semaphore_release never fired during the client run. +- nvidia.ko functions are `notrace` -> kprobe_events refuses them + ("Could not probe notrace function ... [nvidia]"). UVM module probes DO work. + QEMU gdbstub hardware breakpoints work fine (hbreak *addr). +- Service rescue matcher is CE-shaped only (hdr 0x20038090, hi==1, lo&0xfff==slot, + w5/v5 shape) -> can never match a GR push. Entries 0/1/2 are consumed as "stale" + after ~150 failed rescues each; gr_wedge=1. +- channelFillGpFifo(channel, idx, len) disasm: rdi=channel; +0x178=gpu; + +0xe8=cached ring CPU VA; +0xb8=push GPU VA base; +0x48=stride; + +0x110=class (0xc5b5 seen for a CE channel; GR = 0xc46f). + Entry encoding: [0]=va&~3 (u32), [1]=(va>>32 &0xff)|((len<<8)&0x7ffffc00). +- r674 ring PAs per run: 0x12a783000(?) no: run1 0x1050b6000, run2 0x10297d000, + later 0x102317000 (ring line format: "h=0x... gpfifo=0x44a0000 ent=32768 userd=0xffe00 as=2 put=3 get=0 ring rc=0 pa=0x... ap=1"). +- Push VA = 0x4480000, ring VA = 0x44a0000 (delta 0x20000); entry0 len=1432, + entry1 {va=0x4480598,len=4}, entry2 {va=0x448059c,len=976}. +- The pushbuffer (UVM) = 0x20000 (32 pages) aligned; UVM_PUSHBUFFER_SIZE=0x20000 + (UVM_MAX_PUSH_SIZE=0x400). UVM_RM_MEM_TYPE_SYS with CPU map. +- RM memdesc records {size=0x20000, flags=0x03008000, first_PA, ptr} exist in RAM + (11 found post-run) but their PAs did not contain the release group. +- RM window record {0x4480000, 0x20000} + head/tail list nodes exist in RAM. +- The r673 VA-table record {0x10, 0x4500000, 0x786fd9598ab0, 0xa3bf20} has a + "cpu" field that may be a user VA of the client (0x64..-0x78.. range). + +## Ops +- r674 wedged after the gdb-interrupted vk31 run (SSH + guest agent dead, vCPUs idle). + Fresh cycle r675 started. gdb674d still attached to the dead QEMU (never kill gdbs). +- /dev/shm 53%. + +## r675 plan +1. gdb675.gdb: hbreak channelFillGpFifo if class(+0x110)==0xc46f; on hit dump the + channel struct pointer fields (see gdb675.gdb) -> identify the push's CPU mapping + (direct-map or vmalloc) by content. +2. Compute PA (direct-map: VA - 0xffff8916c0000000 style base) and read the push + content from the RAM file; find the release group. +3. Inject 0x4480000 -> PA via sema_map_add BEFORE/DURING the stall; watch the service + apply the release and vkCreateDevice pass. +4. Durability: integrate into pin_gr_stream.py demand path or a service hook. diff --git a/tools/guest-diag/wall2/slotfill.sh b/tools/guest-diag/wall2/slotfill.sh new file mode 100644 index 0000000..a3416a9 --- /dev/null +++ b/tools/guest-diag/wall2/slotfill.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Watch the vk_frame client and fill the completion-slot u16 (base+0x2e) in +# every small /dev/nvidiactl rw-s mapping while it waits. +cd /tmp +rm -f /tmp/slotfill.log +(timeout 120 /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk76.out 2>&1 &) +PID="" +for i in $(seq 1 100); do PID=$(pgrep -x vk_frame | head -1); [ -n "$PID" ] && break; sleep 0.1; done +echo "pid=$PID" >> /tmp/slotfill.log +END=$((SECONDS+60)) +while [ $SECONDS -lt $END ]; do + [ -d /proc/$PID ] || break + for m in $(grep 'nvidiactl' /proc/$PID/maps | awk '$2 ~ /rw-s/ {print $1}'); do + lo=${m%-*}; hi=${m#*-} + size=$((0x$hi - 0x$lo)) + [ $size -gt 65536 ] && continue + off=$((0x$lo + 0x2e)) + v=$(dd if=/proc/$PID/mem bs=1 skip=$off count=2 2>/dev/null | xxd -p) + if [ "$v" = "ffff" ]; then + printf '\x00\x00' | dd of=/proc/$PID/mem bs=1 seek=$off count=2 conv=notrunc 2>/dev/null + echo "$(date +%s.%N) FILL $m+0x2e was ffff" >> /tmp/slotfill.log + fi + done + sleep 0.2 +done +echo "done" >> /tmp/slotfill.log +echo "=== client ===" +cat /tmp/vk76.out +echo "=== fills ===" +grep -c FILL /tmp/slotfill.log diff --git a/tools/guest-diag/wall2/slotfill2.sh b/tools/guest-diag/wall2/slotfill2.sh new file mode 100644 index 0000000..33ad702 --- /dev/null +++ b/tools/guest-diag/wall2/slotfill2.sh @@ -0,0 +1,19 @@ +#!/bin/bash +cd /tmp +PID="" +for i in $(seq 1 200); do PID=$(pgrep -x vk_frame | head -1); [ -n "$PID" ] && break; sleep 0.05; done +END=$((SECONDS+45)) +while [ $SECONDS -lt $END ]; do + [ -d /proc/$PID ] || break + for m in $(grep 'nvidiactl' /proc/$PID/maps 2>/dev/null | awk '$2 ~ /rw-s/ {print $1}'); do + lo=${m%-*}; hi=${m#*-} + size=$((0x$hi - 0x$lo)) + [ $size -gt 65536 ] && continue + off=$((0x$lo + 0x2e)) + v=$(dd if=/proc/$PID/mem bs=1 skip=$off count=2 2>/dev/null | xxd -p) + if [ "$v" = "ffff" ]; then + printf '\x00\x00' | dd of=/proc/$PID/mem bs=1 seek=$off count=2 conv=notrunc 2>/dev/null + fi + done + sleep 0.1 +done diff --git a/tools/guest-diag/wall2/slots_probe.sh b/tools/guest-diag/wall2/slots_probe.sh new file mode 100644 index 0000000..71e6fb7 --- /dev/null +++ b/tools/guest-diag/wall2/slots_probe.sh @@ -0,0 +1,11 @@ +#!/bin/bash +PID=$1 +sudo grep -E 'rw-s' /proc/$PID/maps | grep nvidiactl | while read -r line; do + RNG=$(echo "$line" | awk '{print $1}') + LO=${RNG%-*}; HI=${RNG#*-} + SZ=$((0x$HI-0x$LO)) + [ $SZ -gt 262144 ] && continue + V2E=$(sudo dd if=/proc/$PID/mem bs=1 skip=$((0x$LO+0x2e)) count=2 2>/dev/null | xxd -p) + V0E=$(sudo dd if=/proc/$PID/mem bs=1 skip=$((0x$LO+0xe)) count=2 2>/dev/null | xxd -p) + echo "$RNG sz=$SZ w2e=$V2E w0e=$V0E" +done diff --git a/tools/guest-diag/wall2/stracevk.sh b/tools/guest-diag/wall2/stracevk.sh new file mode 100644 index 0000000..b2752c4 --- /dev/null +++ b/tools/guest-diag/wall2/stracevk.sh @@ -0,0 +1,7 @@ +#!/bin/sh +cd /tmp +sudo timeout 90 strace -f -T -tt -e trace=ioctl -o /tmp/vk_strace.out /tmp/vk_frame 1 64 64 5 /tmp 0 > /tmp/vk58.out 2>&1 +echo "=== client out ===" +cat /tmp/vk58.out +echo "=== ioctl tail ===" +tail -30 /tmp/vk_strace.out diff --git a/tools/guest-diag/wall2/t101.py b/tools/guest-diag/wall2/t101.py new file mode 100644 index 0000000..436c609 --- /dev/null +++ b/tools/guest-diag/wall2/t101.py @@ -0,0 +1,55 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception as e: + if state["chits"] <= 4: + print("chitfail %s" % e, flush=True) + if state["cleared"] >= 24: + self.enabled = False + print("BC-DISABLED", flush=True) + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T101-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t102.py b/tools/guest-diag/wall2/t102.py new file mode 100644 index 0000000..759d974 --- /dev/null +++ b/tools/guest-diag/wall2/t102.py @@ -0,0 +1,31 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +try: + pc = int(gdb.parse_and_eval("$pc")) + print("PC 0x%x off=0x%x" % (pc, pc - base if base else 0)) +except Exception as e: + print("pcfail %s" % e) +try: + C = base + 0x2070B7C + print("A8COUNT %u" % int(gdb.parse_and_eval("*(unsigned int*)0x%x" % C))) +except Exception as e: + print("countfail %s" % e) +try: + print(gr("bt 6", to_string=True)) +except Exception as e: + print("btfail %s" % e) +gr("detach") +gr("quit") +print("T102-DONE") diff --git a/tools/guest-diag/wall2/t104.py b/tools/guest-diag/wall2/t104.py new file mode 100644 index 0000000..5a4c059 --- /dev/null +++ b/tools/guest-diag/wall2/t104.py @@ -0,0 +1,56 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "t0":time.time(), "last":time.time()} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + state["last"] = time.time() + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 8 or state["chits"] >= 40 or (time.time() - state["last"]) > 25.0: + self.enabled = False + print("BC-EXIT cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("bpfail %s" % e, flush=True) +print("T104-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t106.py b/tools/guest-diag/wall2/t106.py new file mode 100644 index 0000000..ceaac44 --- /dev/null +++ b/tools/guest-diag/wall2/t106.py @@ -0,0 +1,61 @@ +import gdb, time +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid), flush=True) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "done":False} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + if state["done"]: + return False + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + if state["cleared"] >= 5 or state["chits"] >= 14: + state["done"] = True + try: + gr("detach") + print("BC-DETACHED cleaned=%d chits=%d" % (state["cleared"], state["chits"]), flush=True) + except Exception as e: + print("detachfail %s" % e, flush=True) + return True + return False +try: + BC(); print("BPS-SET", flush=True) + gr("continue") +except Exception as e: + print("contend %s" % e, flush=True) +print("T106-DONE chits=%d cleared=%d" % (state["chits"], state["cleared"]), flush=True) diff --git a/tools/guest-diag/wall2/t112c-gdbstub.py b/tools/guest-diag/wall2/t112c-gdbstub.py new file mode 100644 index 0000000..27d0b9d --- /dev/null +++ b/tools/guest-diag/wall2/t112c-gdbstub.py @@ -0,0 +1,42 @@ +import gdb + +gr = gdb.execute + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +state = {"hits": {}} + +class HB(gdb.Breakpoint): + def __init__(self, pa, tag, quiet_after=8): + super().__init__("*0x%x" % pa, type=gdb.BP_BREAKPOINT, internal=False) + self.tag = tag + self.quiet_after = quiet_after + self.silent = True + def stop(self): + n = state["hits"].get(self.tag, 0) + 1 + state["hits"][self.tag] = n + try: + rdi = int(gdb.parse_and_eval("$rdi")) + rsi = int(gdb.parse_and_eval("$rsi")) + rdx = int(gdb.parse_and_eval("$rdx")) + rcx = int(gdb.parse_and_eval("$rcx")) + r8 = int(gdb.parse_and_eval("$r8")) + r9 = int(gdb.parse_and_eval("$r9")) + if n <= self.quiet_after or self.tag in ("PRED", "COMPR"): + print("HIT %s #%d rdi=0x%x rsi=0x%x rdx=0x%x rcx=0x%x r8=0x%x r9=0x%x" % + (self.tag, n, rdi, rsi, rdx, rcx, r8, r9), flush=True) + except Exception as e: + print("err", e, flush=True) + return False + +HB(0xffffffffc0c403b0, "VIDH") +HB(0xffffffffc0d667a0, "PMA", 4) +HB(0xffffffffc0d6ae30, "PRED") +HB(0xffffffffc0d57d60, "COMPR", 16) +HB(0xffffffffc0d607f0, "SCRUB", 4) +print("HB-SET", flush=True) +gr("continue") +print("T112B-DONE %s" % state["hits"], flush=True) diff --git a/tools/guest-diag/wall2/t120.py b/tools/guest-diag/wall2/t120.py new file mode 100644 index 0000000..68a1b8b --- /dev/null +++ b/tools/guest-diag/wall2/t120.py @@ -0,0 +1,148 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2 and size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + state["n"] += 1 + print("HIT149 n=%d attr=0x%x flags=0x%x owner=0x%x align=0x%x" % ( + state["n"], attr, flags, owner, u64(arg + 0x60)), flush=True) + if state["n"] == 1: + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t121.py b/tools/guest-diag/wall2/t121.py new file mode 100644 index 0000000..1ae25ed --- /dev/null +++ b/tools/guest-diag/wall2/t121.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size == 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 attr=0x%x flags=0x%x" % (attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t122.py b/tools/guest-diag/wall2/t122.py new file mode 100644 index 0000000..8c9c6fd --- /dev/null +++ b/tools/guest-diag/wall2/t122.py @@ -0,0 +1,151 @@ +import gdb +gr = gdb.execute +state = {"n": 0, "any": 0, "pend": None, "base": None, "chits": 0, "cleared": 0, + "whits": 0, "wrote": 0} + +def u32(a): + return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +def u64(a): + return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) + +def ioc(d, t, nr, size): + return (d << 30) | (size << 16) | (t << 8) | nr + +VIDHEAP = ioc(3, 0x46, 0x4a, 0xb8) + +def glbase(): + try: + pid = gdb.selected_inferior().pid + for line in open("/proc/%d/maps" % pid): + parts = line.split() + if "libnvidia-eglcore" in line and int(parts[2], 16) == 0: + return int(parts[0].split("-")[0], 16) + except Exception: + pass + return None + +class BC(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i * 8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt), flush=True) + except Exception: + pass + except Exception: + pass + return False + +class BW(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*0x%x" % addr, internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val), flush=True) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + except Exception: + pass + return False + +class IOC(gdb.Breakpoint): + def __init__(self): + super().__init__("ioctl", internal=False) + self.silent = True + def stop(self): + if state["base"] is None: + b = glbase() + if b: + state["base"] = b + print("GLBASE 0x%x" % b, flush=True) + try: + BC(b + 0xB33D30) + BW(b + 0xDCEBD9) + print("BC-BW-SET", flush=True) + except Exception as e: + print("bpset-fail %s" % e, flush=True) + try: + req = int(gdb.parse_and_eval("$rsi")) & 0xFFFFFFFF + arg = int(gdb.parse_and_eval("$rdx")) + except Exception: + return False + if state["pend"] is not None: + try: + st = u32(state["pend"] + 0x14) + at = u32(state["pend"] + 0x38) + off = u64(state["pend"] + 0x68) + print("PREV149-OUT st=0x%x attr=0x%x off=0x%x" % (st, at, off), flush=True) + except Exception as e: + print("PREV149-STATUS unreadable %s" % e, flush=True) + state["pend"] = None + if req != VIDHEAP: + return False + state["any"] += 1 + try: + fn = u32(arg + 0x08) + size = u64(arg + 0x58) + attr = u32(arg + 0x38) + owner = u32(arg + 0x28) + flags = u32(arg + 0x34) + except Exception: + return False + if fn == 2: + state["n"] += 1 + if state["n"] >= 120: + print("VH#%d size=0x%x align=0x%x attr=0x%x attr2=0x%x flags=0x%x owner=0x%x range=0x%x-0x%x" % ( + state["n"], size, u64(arg + 0x60), attr, u32(arg + 0x90), flags, owner, + u64(arg + 0x80), u64(arg + 0x88)), flush=True) + if size >= 0x10000 and (attr & 0xf000) == 0x2000 and owner == 0x474C0000: + print("HIT149 size=0x%x attr=0x%x flags=0x%x" % (size, attr, flags), flush=True) + gr("bt 12") + newattr = attr & 0xFFFFCFFF + gr("set {unsigned int}0x%x = 0x%x" % (arg + 0x38, newattr)) + print("FLIP149 attr 0x%x -> 0x%x" % (attr, newattr), flush=True) + state["pend"] = arg + return False + +try: + IOC() + print("BPS-SET", flush=True) +except Exception as e: + print("bpset-fail %s" % e, flush=True) + +gr("set pagination off") +gr("set confirm off") +try: + gr("run") +except Exception as e: + print("runend %s" % e, flush=True) +print("T120-DONE n=%d any=%d chits=%d cleared=%d whits=%d wrote=%d" % ( + state["n"], state["any"], state["chits"], state["cleared"], + state["whits"], state["wrote"]), flush=True) +gr("quit") diff --git a/tools/guest-diag/wall2/t75g.py b/tools/guest-diag/wall2/t75g.py new file mode 100644 index 0000000..2a0881a --- /dev/null +++ b/tools/guest-diag/wall2/t75g.py @@ -0,0 +1,170 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid + +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found") + gdb.execute("detach") + sys.exit(1) +print("EGLBASE 0x%x PID %d" % (base, pid)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"out": 0, "E": 0, "S": 0, "armed150": False, "armedS": False, + "bt_done": False, "post_done": False} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def a97(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + r14 = int(gdb.parse_and_eval("$r14")) + out = u64(rbp+0x10) + state["out"] = out + q = rd(out, 0x28) + rec_ptr = int.from_bytes(q[8:16], 'little') + print("A97 out=0x%x count=%d f10=%08x f20=%08x f24=%08x recp=0x%x" % + (out, int.from_bytes(q[0:4], 'little'), int.from_bytes(q[0x10:0x14], 'little'), + int.from_bytes(q[0x20:0x24], 'little'), int.from_bytes(q[0x24:0x28], 'little'), rec_ptr)) + if rec_ptr and rec_ptr > 0x1000: + try: + rec = rd(rec_ptr, 16) + print("A97 rec=%08x %08x %08x %08x" % + tuple(int.from_bytes(rec[i*4:i*4+4], 'little') for i in range(4))) + except Exception as e: + print("A97 rec-unreadable", e) + return False + +def post(bp): + if state["post_done"]: + return False + state["post_done"] = True + rsp = int(gdb.parse_and_eval("$rsp")) + q = rd(rsp, 0x40) + print("POST rsp=0x%x out=0x%x" % (rsp, state["out"])) + for i in range(0, 0x40, 8): + v = int.from_bytes(q[i:i+8], 'little') + print("POST rsp+%02x = 0x%016x" % (i, v)) + if state["out"]: + oq = rd(state["out"], 0x40) + for i in range(0, 0x40, 8): + v = int.from_bytes(oq[i:i+8], 'little') + print("POST out+%02x = 0x%016x" % (i, v)) + return False + +def t150(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + if not state.get("vt_done"): + state["vt_done"] = True + rdi = u64(rbp+0x148) + vt = u64(rdi) if rdi else 0 + m118 = u64(vt+0x118) if vt else 0 + m170 = u64(vt+0x170) if vt else 0 + print("VT E148=0x%x vt=0x%x m118=0x%x m170=0x%x" % (rdi, vt, m118, m170)) + if not state.get("eng_done"): + state["eng_done"] = True + n = u32(rbp+8) + ids = [u32(rbp+0x54+i*4) for i in range(min(n,8))] + print("ENG n=%d ids=%s" % (n, ",".join("0x%x" % x for x in ids))) + + # cfg area pointer is rip-relative 0x2019320 at runtime = base + (0x2019320 - 0) + # eglcore base == min load addr; file offset == vaddr for the .text; data seg present? + try: + vals = rd(base + 0x2019320 + 0xf80, 0xe0) + for i in range(0, 0xe0, 0x10): + v4 = [int.from_bytes(vals[i+j:i+j+4],'little') for j in (0,4,8,12)] + print("CFG +0x%03x = %s" % (0xf80+i, ["0x%x"%x for x in v4])) + except Exception as e: + print("CFG dump err", e) + v = u32(rbp+0x150) + b38 = rd(rbp+0x1b038, 1)[0]; b39 = rd(rbp+0x1b039, 1)[0] + p = u64(rbp+0x1b030); s8 = u64(p+8) if p else 0 + print("T150 E=0x%x v=0x%08x b038=%d b039=%d p=0x%x p8=0x%x" % + (rbp, v, b38, b39, p, s8)) + if not state["bt_done"]: + state["bt_done"] = True + print(gdb.execute("bt 8", to_string=True)) + if not state["armed150"]: + state["armed150"] = True + state["E"] = rbp + try: + gdb.execute("frame 1") + S = int(gdb.parse_and_eval("$rbp")) + gdb.execute("frame 0") + print("S=0x%x S120=%08x S128=%08x" % (S, u32(S+0x120), u32(S+0x128))) + state["S"] = S + except Exception as e: + print("S capture err", e) + S = 0 + W("*(unsigned int *)0x%x" % (rbp+0x150), rbp, 0x150, 4, "W150") + W("*(unsigned char *)0x%x" % (rbp+0x1b038), rbp, 0x1b038, 1, "W038") + W("*(unsigned char *)0x%x" % (rbp+0x1b039), rbp, 0x1b039, 1, "W039") + if S: + W("*(unsigned int *)0x%x" % (S+0x120), S, 0x120, 4, "W120") + print("WATCHES armed E=0x%x S=0x%x" % (rbp, S)) + return False + +def sl(bp): + rbp = int(gdb.parse_and_eval("$rbp")) + print("SLOOP S=0x%x S120=%08x" % (rbp, u32(rbp+0x120))) + if not state["armedS"]: + state["armedS"] = True + W("*(unsigned int *)0x%x" % (rbp+0x120), rbp, 0x120, 4, "W120") + print("W120 armed on S+0x120") + return False + +class W(gdb.Breakpoint): + def __init__(self, spec, E, off, size, tag): + super().__init__(spec, type=gdb.BP_WATCHPOINT) + self.silent = True; self.E = E; self.off = off; self.size = size + self.tag = tag; self.hits = 0 + def stop(self): + self.hits += 1 + cur = int.from_bytes(rd(self.E+self.off, self.size), 'little') + pc = int(gdb.parse_and_eval("$pc")) + print("%s v=0x%x pc=0x%x (rel 0x%x)" % (self.tag, cur, pc, pc-base)) + if self.hits <= 8: + print(gdb.execute("bt 10", to_string=True)) + return self.hits >= 400 + +B("*0x%x" % (base+0xecd43d), a97, 220) +B("*0x%x" % (base+0xd5e4d3), post, 100000) +B("*0x%x" % (base+0xd5e449), t150, 220) +B("*0x%x" % (base+0xd6ef76), sl, 240) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +print("TRACE-END") +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t79.py b/tools/guest-diag/wall2/t79.py new file mode 100644 index 0000000..c2fe381 --- /dev/null +++ b/tools/guest-diag/wall2/t79.py @@ -0,0 +1,99 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +print("EGLBASE 0x%x PID %d" % (base, pid)) + +maps = [] +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x, 16) for x in parts[0].split('-')] + maps.append((lo, hi, parts[1], parts[-1] if len(parts) > 5 else "anon")) + except Exception: + pass + +def resolve(addr): + for lo, hi, perm, name in maps: + if lo <= addr < hi: + return "0x%x-0x%x %s %s" % (lo, hi, perm, name) + return "?" + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"done": False} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def loophead(bp): + if state["done"]: + return False + state["done"] = True + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx+0x70) + print("WAIT rbx=0x%x sub=0x%x" % (rbx, sub)) + try: + obj2 = u64(sub+0x58) + off = u32(obj2+0x60) + holder = u64(obj2+0x58) + base2 = u64(holder+0x58) + word_addr = base2 + off + val = u32(word_addr) + print("WAIT obj2=0x%x off=0x%x holder=0x%x base=0x%x word=0x%x val=0x%x" % + (obj2, off, holder, base2, word_addr, val)) + print("WAIT word map:", resolve(word_addr)) + print("WAIT holder obj2+0x50=0x%x obj2+0x64=0x%x" % (u64(obj2+0x50), u32(obj2+0x64))) + try: + hd = rd(holder, 0x30) + print("WAIT holder dump:", " ".join("%02x" % b for b in hd)) + except Exception as e: + print("WAIT holder dump err", e) + except Exception as e: + print("WAIT chain err", e) + # dump the client's nvidiactl rw-s mappings + print("WAIT nvidiactl maps:") + for lo, hi, perm, name in maps: + if "nvidiactl" in name and "rw-s" in perm: + print(" 0x%x-0x%x size=0x%x" % (lo, hi, hi-lo)) + return True + +B("*0x%x" % (base+0xdcebd9), loophead, 1) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +print("TRACE-END") +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t80r.py b/tools/guest-diag/wall2/t80r.py new file mode 100644 index 0000000..ac2263b --- /dev/null +++ b/tools/guest-diag/wall2/t80r.py @@ -0,0 +1,32 @@ +import gdb, sys +inf = gdb.selected_inferior() +pid = inf.pid +cands = [] +for line in open("/proc/%d/maps" % pid): + if "nvidiactl" in line and "rw-s" in line: + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if hi - lo == 0x4000: + cands.append(lo + 0x80) +print("CANDIDATES %s" % ["0x%x" % c for c in cands]) +class W(gdb.Breakpoint): + def __init__(self, addr): + super().__init__("*(unsigned int*)0x%x" % addr, gdb.BP_WATCHPOINT, gdb.WP_READ, internal=False) + self.addr = addr + self.silent = True + def stop(self): + print("READWORD 0x%x" % self.addr) + return True +armed = 0 +for c in cands: + try: + W(c); armed += 1 + except Exception as e: + print("watchfail 0x%x: %s" % (c, e)) +print("ARMED %d" % armed) +if armed: + gdb.execute("continue") +print("T80R-DONE") diff --git a/tools/guest-diag/wall2/t83.py b/tools/guest-diag/wall2/t83.py new file mode 100644 index 0000000..5e3cce2 --- /dev/null +++ b/tools/guest-diag/wall2/t83.py @@ -0,0 +1,25 @@ +import gdb +from collections import Counter +gr = gdb.execute +pcs = Counter() +inf = gdb.selected_inferior() +for i in range(300): + try: + gr("stepi", to_string=True) + except Exception as e: + print("stepfail %s" % e); break + try: + pcs[int(gdb.parse_and_eval("$pc"))] += 1 + except Exception: + break +print("HOTPCS") +top = pcs.most_common(4) +for pc, n in top: + print("PC 0x%x hits=%d" % (pc, n)) +for pc, n in top[:1]: + try: + print(gr("x/10i 0x%x" % pc, to_string=True)) + print(gr("info registers rax rbx rcx rdx rsi rdi rbp r8 r9 r10 r11 r12 r13 r14 r15", to_string=True)) + except Exception as e: + print("fail %s" % e) +print("T83-DONE") diff --git a/tools/guest-diag/wall2/t89.py b/tools/guest-diag/wall2/t89.py new file mode 100644 index 0000000..92c1de3 --- /dev/null +++ b/tools/guest-diag/wall2/t89.py @@ -0,0 +1,17 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +print(gr("x/70i 0x%x" % (base + 0xB28519), to_string=True)) +print(gr("x/40i 0x%x" % (base + 0xB28560), to_string=True)) +print("T89-DONE") diff --git a/tools/guest-diag/wall2/t90.py b/tools/guest-diag/wall2/t90.py new file mode 100644 index 0000000..8518235 --- /dev/null +++ b/tools/guest-diag/wall2/t90.py @@ -0,0 +1,57 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +COUNT = base + 0x2070b7c +print("EGLBASE 0x%x PID %d COUNT_ADDR 0x%x" % (base, pid, COUNT)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') + +state = {"n": 0} + +class B(gdb.Breakpoint): + def __init__(self, spec, limit=100000): + super().__init__(spec, internal=False) + self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + v = u32(COUNT) + if v: + state["n"] += 1 + if state["n"] <= 8: + print("CLEAR count %u -> 0 (hit %d)" % (v, self.hits)) + gdb.execute("set {unsigned int}0x%x = 0" % COUNT) + except Exception as e: + print("ERR", e) + return self.hits >= 400 + +B("*0x%x" % (base+0xa85d24), 400) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +except KeyboardInterrupt: + pass +print("TRACE-END clears=%d" % state["n"]) +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t91.py b/tools/guest-diag/wall2/t91.py new file mode 100644 index 0000000..5c84ba9 --- /dev/null +++ b/tools/guest-diag/wall2/t91.py @@ -0,0 +1,95 @@ +import gdb, sys + +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + if "libnvidia-eglcore.so.570.144" in line: + parts = line.split() + lo, hi = [int(x, 16) for x in parts[0].split('-')] + off = int(parts[2], 16) + if off == 0: + base = lo + break +if base is None: + print("EGLBASE not found"); gdb.execute("detach"); sys.exit(1) +COUNT = base + 0x2070b7c +print("EGLBASE 0x%x PID %d COUNT 0x%x" % (base, pid, COUNT)) + +def rd(a, n): + return inf.read_memory(a, n).tobytes() +def u32(a): + return int.from_bytes(rd(a, 4), 'little') +def u64(a): + return int.from_bytes(rd(a, 8), 'little') + +state = {"clears": 0, "mp": 0, "am": 0} + +class B(gdb.Breakpoint): + def __init__(self, spec, fn, limit=100000): + super().__init__(spec, internal=False) + self.fn = fn; self.limit = limit; self.hits = 0 + self.silent = True + def stop(self): + self.hits += 1 + try: + stop = bool(self.fn(self)) + except Exception as e: + print("ERR@0x%x %s" % (int(gdb.parse_and_eval("$pc")), e)) + stop = False + return stop or self.hits >= self.limit + +def clear(bp): + try: + v = u32(COUNT) + if v: + state["clears"] += 1 + if state["clears"] <= 5: + print("CLEAR count %u -> 0" % v) + gdb.execute("set {unsigned int}0x%x = 0" % COUNT) + except Exception as e: + print("CLEAR err", e) + return False + +def mp(bp): + rsi = int(gdb.parse_and_eval("$rsi")) + state["mp"] += 1 + if state["mp"] <= 3: + print("MEMPROPS ptr=0x%x" % rsi) + return False + +def am(bp): + rsi = int(gdb.parse_and_eval("$rsi")) + state["am"] += 1 + if state["am"] <= 10: + try: + st = u32(rsi) + size = u64(rsi+0x10) + tidx = u32(rsi+0x18) + print("ALLOC sType=%d size=0x%x typeIndex=%d" % (st, size, tidx)) + except Exception as e: + print("ALLOC err", e) + return False + +B("*0x%x" % (base+0xa85d24), clear, 100000) +try: + B("vkGetPhysicalDeviceMemoryProperties", mp, 5) +except Exception as e: + print("MP BP err", e) +try: + B("vkAllocateMemory", am, 20) +except Exception as e: + print("AM BP err", e) + +try: + gdb.execute("continue") +except gdb.error as e: + print("CONT-ERR %s" % e) +except KeyboardInterrupt: + pass +print("TRACE-END clears=%d" % state["clears"]) +try: + gdb.execute("detach") +except gdb.error as e: + print("DETACH-ERR %s" % e) +gdb.execute("quit") diff --git a/tools/guest-diag/wall2/t93.py b/tools/guest-diag/wall2/t93.py new file mode 100644 index 0000000..6a0c23d --- /dev/null +++ b/tools/guest-diag/wall2/t93.py @@ -0,0 +1,57 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True; self.done = False + def stop(self): + if self.done: + return True + self.done = True + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + print("RETRYFN obj=0x%x n=0x%x" % (obj, n)) + cnt = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % (obj + 0x22efc))) + print("CNT %d" % cnt) + wrote = 0 + for i in range(0, min(cnt, 32)): + p = int(gdb.parse_and_eval("*(unsigned long*)0x%x" % (obj + 0x1428 + i*8))) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + v0 = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % tgt)) + print("E%d ptr=0x%x tgt=0x%x w0=0x%x w16e=0x%x" % (i, p, tgt, v0, v16)) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + wrote += 1 + except Exception as e: + print("E%d fail %s" % (i, e)) + print("CLEARED %d" % wrote) + gr("detach") + except Exception as e: + print("fail %s" % e) + return True +try: + B() + print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T92-DONE") diff --git a/tools/guest-diag/wall2/t94.py b/tools/guest-diag/wall2/t94.py new file mode 100644 index 0000000..0d2aa27 --- /dev/null +++ b/tools/guest-diag/wall2/t94.py @@ -0,0 +1,54 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +state = {"hits":0, "cleared":0} +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["hits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % (obj + 0x22efc))) + if state["hits"] <= 6: + print("HIT %d obj=0x%x n=0x%x cnt=%d" % (state["hits"], obj, n, cnt)) + for i in range(0, min(cnt, 32)): + p = int(gdb.parse_and_eval("*(unsigned long*)0x%x" % (obj + 0x1428 + i*8))) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + if state["cleared"] <= 12: + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt)) + except Exception as e: + if state["hits"] <= 6: + print("E%d fail %s" % (i, e)) + except Exception as e: + if state["hits"] <= 6: + print("hitfail %s" % e) + return state["hits"] >= 4000 +try: + B(); print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T94-DONE hits=%d cleared=%d" % (state["hits"], state["cleared"])) diff --git a/tools/guest-diag/wall2/t95.py b/tools/guest-diag/wall2/t95.py new file mode 100644 index 0000000..8daf6ca --- /dev/null +++ b/tools/guest-diag/wall2/t95.py @@ -0,0 +1,19 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +print("--- wait site +0xA12D90..:") +print(gr("x/60i 0x%x" % (base + 0xA12D90), to_string=True)) +print("--- caller +0xD690D0..:") +print(gr("x/40i 0x%x" % (base + 0xD690D0), to_string=True)) +print("T95-DONE") diff --git a/tools/guest-diag/wall2/t96b.py b/tools/guest-diag/wall2/t96b.py new file mode 100644 index 0000000..cd3b7ab --- /dev/null +++ b/tools/guest-diag/wall2/t96b.py @@ -0,0 +1,40 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +class B(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xA12DDF), internal=False) + self.silent = True; self.done = False + def stop(self): + if self.done: + return True + self.done = True + try: + rdi = int(gdb.parse_and_eval("$rdi")) + rsi = int(gdb.parse_and_eval("$rsi")) + rdx = int(gdb.parse_and_eval("$rdx")) + print("POLL nfds=%d timeout=%d arr=0x%x" % (rsi, rdx, rdi)) + fd = int(gdb.parse_and_eval("*(int*)0x%x" % rdi)) + ev = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (rdi + 4))) + rv = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (rdi + 6))) + print("FDPOLL fd=%d events=0x%x revents=0x%x" % (fd, ev, rv)) + except Exception as e: + print("fail %s" % e) + return True +try: + B(); print("BP-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T96-DONE") diff --git a/tools/guest-diag/wall2/t97.py b/tools/guest-diag/wall2/t97.py new file mode 100644 index 0000000..5a12348 --- /dev/null +++ b/tools/guest-diag/wall2/t97.py @@ -0,0 +1,22 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x" % (base or 0)) +C = base + 0x2070b7c +v = int(gdb.parse_and_eval("*(unsigned int*)0x%x" % C)) +print("COUNT %u" % v) +print(gr("x/8wx 0x%x" % (C - 0x10), to_string=True)) +if v: + gr("set {unsigned int}0x%x = 0" % C) + print("WROTE 0") +print("T97-DONE") diff --git a/tools/guest-diag/wall2/t99.py b/tools/guest-diag/wall2/t99.py new file mode 100644 index 0000000..ed665fb --- /dev/null +++ b/tools/guest-diag/wall2/t99.py @@ -0,0 +1,79 @@ +import gdb +gr = gdb.execute +inf = gdb.selected_inferior() +pid = inf.pid +base = None +for line in open("/proc/%d/maps" % pid): + parts = line.split() + try: + lo, hi = [int(x,16) for x in parts[0].split('-')] + except Exception: + continue + if base is None and "libnvidia-eglcore" in line and int(parts[2],16) == 0: + base = lo +print("BASE 0x%x PID %d" % (base or 0, pid)) +def u64(a): return int(gdb.parse_and_eval("*(unsigned long*)0x%x" % a)) +def u32(a): return int(gdb.parse_and_eval("*(unsigned int*)0x%x" % a)) +state = {"chits":0, "cleared":0, "whits":0, "wrote":0} +class BC(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xB33D30), internal=False) + self.silent = True + def stop(self): + state["chits"] += 1 + try: + obj = int(gdb.parse_and_eval("$rdi")) + n = int(gdb.parse_and_eval("$rsi")) + cnt = u32(obj + 0x22efc) + for i in range(0, min(cnt, 32)): + p = u64(obj + 0x1428 + i*8) + if not p: + continue + tgt = p + n * 0x10 + try: + v16 = int(gdb.parse_and_eval("*(unsigned short*)0x%x" % (tgt + 0xe))) + if v16 == 0xffff: + gr("set {unsigned short}0x%x = 0" % (tgt + 0xe)) + gr("set {unsigned int}0x%x = 0" % tgt) + gr("set {unsigned short}0x%x = 0" % (tgt + 0xc)) + state["cleared"] += 1 + if state["cleared"] <= 10: + print("CLEAR#%d obj=0x%x n=0x%x tgt=0x%x" % (state["cleared"], obj, n, tgt)) + except Exception: + pass + except Exception as e: + if state["chits"] <= 4: + print("chitfail %s" % e) + return state["chits"] >= 6000 +class BW(gdb.Breakpoint): + def __init__(self): + super().__init__("*0x%x" % (base + 0xDCEBD9), internal=False) + self.silent = True + def stop(self): + state["whits"] += 1 + try: + rbx = int(gdb.parse_and_eval("$rbx")) + sub = u64(rbx + 0x70) + obj2 = u64(sub + 0x58) + off = u32(obj2 + 0x60) + holder = u64(obj2 + 0x58) + base2 = u64(holder + 0x58) + word = base2 + off + val = u32(word) + if state["whits"] <= 6 or val == 0: + print("WORD#%d addr=0x%x val=0x%x" % (state["whits"], word, val)) + if val == 0: + gr("set {unsigned int}0x%x = 1" % word) + state["wrote"] += 1 + if state["wrote"] <= 10: + print("WORDWROTE 0x%x" % word) + except Exception as e: + if state["whits"] <= 4: + print("whitfail %s" % e) + return False +try: + BC(); BW(); print("BPS-SET") + gr("continue") +except Exception as e: + print("bpfail %s" % e) +print("T99-DONE chits=%d cleared=%d whits=%d wrote=%d" % (state["chits"], state["cleared"], state["whits"], state["wrote"])) diff --git a/tools/guest-diag/wall2/trace_push674.py b/tools/guest-diag/wall2/trace_push674.py new file mode 100644 index 0000000..ced4168 --- /dev/null +++ b/tools/guest-diag/wall2/trace_push674.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +# r674: find every reference to the push VA 0x4480000 and follow RM structures +# (direct-map pointers: PA = ptr - page_offset_base) toward a page list. +import numpy as np, struct, os, time, sys + +RAM="/dev/shm/vgpu-guest-ram" +VA=0x4480000 +BASE=0xffff8916c0000000 # r674 page_offset_base (from dmesg rebuild #15) +img=open(RAM,"rb"); fsz=os.path.getsize(RAM) +def gpa2off(pa): + if pa<0x80000000: return pa + if pa<0x180000000: return 0x80000000+(pa-0x100000000) + return None +def off2gpa(off): + return off if off<0x80000000 else 0x100000000+(off-0x80000000) +def rd(pa,n): + off=gpa2off(pa) + if off is None or off+n>fsz: return None + img.seek(off); return img.read(n) +def ptr2pa(v): + if BASE<=v<BASE+0x180000000: + return v-BASE + return None + +CH=64<<20 +hits=[] +t0=time.time() +for pos in range(0,fsz,CH): + img.seek(pos); d=img.read(min(CH,fsz-pos)) + n=len(d)//8 + a=np.frombuffer(d[:n*8],dtype=np.uint64) + idx=np.nonzero(a==np.uint64(VA))[0] + for i in idx: hits.append(off2gpa(pos+int(i)*8)) +print("VA hits:", len(hits), "(%.1fs)" % (time.time()-t0)) + +def tag(v): + bs=struct.pack("<Q",v) + return bs.decode() if all(32<=c<127 for c in bs) else "" + +for gpa in hits: + page=gpa&~0xfff + d=rd(page,0x1000) + if not d: continue + ro=gpa-page + print("== hit gpa=0x%x page=0x%x off=0x%x ==" % (gpa,page,ro)) + # print 0x60 before, 0x80 after + s=max(0,ro-0x60); e=min(0x1000,ro+0x80) + for o in range(s,e,0x10): + a,b=struct.unpack_from("<QQ",d,o) + print(" +0x%03x: %016x %016x %s %s" % (o,a,b,tag(a),tag(b))) + # follow direct-map pointers in this page + for o in range(0,0x1000-8,8): + v=struct.unpack_from("<Q",d,o)[0] + pa=ptr2pa(v) + if pa is None: continue + dd=rd(pa&~0xfff,0x1000) + if not dd: continue + # search that page for PA runs (>=8 page-aligned RAM PAs ascending/descending) + runs=[]; cur=[] + for oo in range(0,0xff8,8): + vv=struct.unpack_from("<Q",dd,oo)[0] + if 0x100000000<=vv<0x180000000 and (vv&0xfff)==0: + cur.append((oo,vv)) + else: + if len(cur)>=8: runs.append(cur) + cur=[] + if len(cur)>=8: runs.append(cur) + if runs: + print(" ptr +0x%03x -> pa=0x%x has %d PA-run(s):" % (o,pa&~0xfff,len(runs))) + for r in runs[:4]: + print(" run at +0x%x: %d pages first=0x%x last=0x%x" % (r[0][0],len(r),r[0][1],r[-1][1])) diff --git a/tools/guest-diag/wall2/unwait.py b/tools/guest-diag/wall2/unwait.py new file mode 100644 index 0000000..94e677c --- /dev/null +++ b/tools/guest-diag/wall2/unwait.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +# unwait.py - guest-side, zero-ptrace wait-word poller: writes 1 to the +0x80 +# u32 of every zero 16 KB rw-s nvidiactl mapping of the newest vk_frame. +import os, re, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def newest_pid(): + try: + out = subprocess.run(["pgrep", "-n", "-x", "vk_frame"], capture_output=True).stdout + return int(out.decode().strip()) + except Exception: + return None +def maps(pid): + try: + return open("/proc/%d/maps" % pid).read() + except Exception: + return "" +logged = set() +end = time.time() + DUR +while time.time() < end: + pid = newest_pid() + if pid: + for line in maps(pid).splitlines(): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + try: + lo, hi = [int(x, 16) for x in p[0].split("-")] + except Exception: + continue + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + os.lseek(fd, wa, 0) + v = os.read(fd, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + os.lseek(fd, wa, 0) + os.write(fd, b"\x01\x00\x00\x00") + key = (pid, wa) + if key not in logged: + logged.add(key) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + os.close(fd) + except Exception: + pass + time.sleep(0.4) +print("UNWAIT-DONE", flush=True) diff --git a/tools/guest-diag/wall2/unwait2.py b/tools/guest-diag/wall2/unwait2.py new file mode 100644 index 0000000..cd4169b --- /dev/null +++ b/tools/guest-diag/wall2/unwait2.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +# unwait2.py - no-ptrace guest-side harness: +# (a) writes 1 to any zero u32 at +0x80 of every 0x4000-byte rw-s nvidia +# mapping of EVERY live vk_frame; +# (b) scans each live vk_frame's anonymous/heap rw-p mappings (16-byte +# aligned) for marker slots {u32@0==0, u16@+0xe==0xffff} and clears +# u16@+0xe to 0 (the instance-phase pending markers the t105 gdb used +# to clear). No ptrace anywhere. +import os, subprocess, sys, time +DUR = float(sys.argv[1]) if len(sys.argv) > 1 else 600 +def pids(): + try: + out = subprocess.run(["pgrep", "-x", "vk_frame"], capture_output=True).stdout + return [int(x) for x in out.split()] + except Exception: + return [] +def rd(fd, off, n): + try: + os.lseek(fd, off, 0) + return os.read(fd, n) + except Exception: + return b"" +def wr(fd, off, data): + try: + os.lseek(fd, off, 0) + os.write(fd, data) + return True + except Exception: + return False +wlog = set(); mlog = set() +end = time.time() + DUR +last_scan = 0.0 +while time.time() < end: + for pid in pids(): + try: + fd = os.open("/proc/%d/mem" % pid, os.O_RDWR) + except Exception: + continue + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or "nvidia" not in p[-1] or "rw-s" not in p[1]: + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo != 0x4000: + continue + wa = lo + 0x80 + v = rd(fd, wa, 4) + if len(v) == 4 and v == b"\x00\x00\x00\x00": + if wr(fd, wa, b"\x01\x00\x00\x00"): + k = (pid, wa) + if k not in wlog: + wlog.add(k) + print("WORDWROTE pid=%d va=0x%x" % (pid, wa), flush=True) + except Exception: + pass + if time.time() - last_scan > 1.5: + budget = 64 + try: + for line in open("/proc/%d/maps" % pid): + p = line.split() + if len(p) < 6 or p[1] != "rw-p": + continue + path = p[-1] + if path.startswith("/") and "nvidia" not in path and path != "[heap]": + continue + lo, hi = [int(x, 16) for x in p[0].split("-")] + if hi - lo > 0x10000000: + hi = lo + 0x10000000 + off = lo + while off < hi: + chunk = rd(fd, off, 0x100000) + if len(chunk) < 32: + break + j = 0 + while j + 16 <= len(chunk): + if chunk[j:j+4] == b"\x00\x00\x00\x00" and chunk[j+0xe:j+0x10] == b"\xff\xff": + wa = off + j + 0xe + if (pid, wa) not in mlog: + if wr(fd, wa, b"\x00\x00"): + mlog.add((pid, wa)) + print("MARKCLR pid=%d va=0x%x" % (pid, wa), flush=True) + j += 16 + off += 0x100000 + budget -= 1 + if budget <= 0: + break + if budget <= 0: + break + except Exception: + pass + last_scan = time.time() + os.close(fd) + time.sleep(0.4) +print("UNWAIT2-DONE", flush=True) diff --git a/tools/guest-diag/wall2/vk_alloc_test.py b/tools/guest-diag/wall2/vk_alloc_test.py new file mode 100644 index 0000000..0bcf35a --- /dev/null +++ b/tools/guest-diag/wall2/vk_alloc_test.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# vk_alloc_test.py - reproduce vk_frame's DEVICE_LOCAL image allocation: +# instance -> device -> OPTIMAL R8G8B8A8 storage image -> mem reqs -> +# pick first DEVICE_LOCAL type -> vkAllocateMemory -> report rc/type/size. +import ctypes, sys + +vk = ctypes.CDLL("libvulkan.so.1") + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32)] + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("pApplicationInfo", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p)] + +class VkMemoryType(ctypes.Structure): + _fields_ = [("propertyFlags", ctypes.c_uint32), ("heapIndex", ctypes.c_uint32)] +class VkMemoryHeap(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("flags", ctypes.c_uint32)] +class VkPhysicalDeviceMemoryProperties(ctypes.Structure): + _fields_ = [("memoryTypeCount", ctypes.c_uint32), + ("memoryTypes", VkMemoryType * 32), + ("memoryHeapCount", ctypes.c_uint32), + ("memoryHeaps", VkMemoryHeap * 16)] + +class VkDeviceQueueCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("queueFamilyIndex", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), ("pQueuePriorities", ctypes.c_void_p)] +class VkDeviceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("queueCreateInfoCount", ctypes.c_uint32), + ("pQueueCreateInfos", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p), + ("pEnabledFeatures", ctypes.c_void_p)] + +class VkExtent3D(ctypes.Structure): + _fields_ = [("width", ctypes.c_uint32), ("height", ctypes.c_uint32), ("depth", ctypes.c_uint32)] +class VkImageCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), ("flags", ctypes.c_uint32), + ("imageType", ctypes.c_uint32), ("format", ctypes.c_uint32), + ("extent", VkExtent3D), ("mipLevels", ctypes.c_uint32), ("arrayLayers", ctypes.c_uint32), + ("samples", ctypes.c_uint32), ("tiling", ctypes.c_uint32), ("usage", ctypes.c_uint32), + ("sharingMode", ctypes.c_uint32), ("queueFamilyIndexCount", ctypes.c_uint32), + ("pQueueFamilyIndices", ctypes.c_void_p), ("initialLayout", ctypes.c_uint32)] +class VkMemoryRequirements(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("alignment", ctypes.c_uint64), ("memoryTypeBits", ctypes.c_uint32)] +class VkMemoryAllocateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("allocationSize", ctypes.c_uint64), ("memoryTypeIndex", ctypes.c_uint32)] + +app = VkApplicationInfo(0, None, b"alloctest", 1, b"alloctest", 1, 0x401000) +ci = VkInstanceCreateInfo(1, None, 0, ctypes.cast(ctypes.pointer(app), ctypes.c_void_p), 0, None, 0, None) +inst = ctypes.c_void_p() +print("vkCreateInstance rc=%d" % vk.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst))) + +n = ctypes.c_uint32(0) +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +devs = (ctypes.c_void_p * n.value)() +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) +phys = devs[0] +print("phys=0x%x" % (phys or 0)) + +prio = ctypes.c_float(1.0) +qci = VkDeviceQueueCreateInfo(2, None, 0, 0, 1, ctypes.cast(ctypes.pointer(prio), ctypes.c_void_p)) +dci = VkDeviceCreateInfo(3, None, 0, 1, ctypes.cast(ctypes.pointer(qci), ctypes.c_void_p), 0, None, 0, None, None) +dev = ctypes.c_void_p() +rc = vk.vkCreateDevice(phys, ctypes.byref(dci), None, ctypes.byref(dev)) +print("vkCreateDevice rc=%d dev=0x%x" % (rc, dev or 0)) +if rc != 0: + sys.exit(1) + +mp = VkPhysicalDeviceMemoryProperties() +vk.vkGetPhysicalDeviceMemoryProperties(phys, ctypes.byref(mp)) + +img_ci = VkImageCreateInfo(14, None, 0, 1, 37, # VK_IMAGE_TYPE_2D, R8G8B8A8_UNORM + VkExtent3D(64, 64, 1), 1, 1, 1, 0, # OPTIMAL + 0x2 | 0x1, 0, 0, None, 0) # STORAGE | TRANSFER_SRC, UNDEFINED +img = ctypes.c_void_p() +rc = vk.vkCreateImage(dev, ctypes.byref(img_ci), None, ctypes.byref(img)) +print("vkCreateImage rc=%d img=0x%x" % (rc, img or 0)) +if rc != 0: + sys.exit(1) + +mr = VkMemoryRequirements() +vk.vkGetImageMemoryRequirements(dev, img, ctypes.byref(mr)) +print("memreqs size=0x%x align=0x%x typeBits=0x%x" % (mr.size, mr.alignment, mr.memoryTypeBits)) + +tidx = 0 +found = False +for k in range(mp.memoryTypeCount): + if (mr.memoryTypeBits & (1 << k)) and (mp.memoryTypes[k].propertyFlags & 0x1): + tidx = k + found = True + break +print("selected type=%d found_device_local=%s (types: %s)" % + (tidx, found, ", ".join("%d:0x%x" % (k, mp.memoryTypes[k].propertyFlags) for k in range(mp.memoryTypeCount)))) + +ma = VkMemoryAllocateInfo(5, None, mr.size, tidx) +mem = ctypes.c_void_p() +rc = vk.vkAllocateMemory(dev, ctypes.byref(ma), None, ctypes.byref(mem)) +print("vkAllocateMemory size=0x%x type=%d rc=%d" % (mr.size, tidx, rc)) diff --git a/tools/guest-diag/wall2/vk_memprops.py b/tools/guest-diag/wall2/vk_memprops.py new file mode 100644 index 0000000..3cf6ba2 --- /dev/null +++ b/tools/guest-diag/wall2/vk_memprops.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +# vk_memprops.py - enumerate Vulkan devices and dump their memory properties +# via ctypes (no vkCreateDevice needed; safe alongside other clients). +import ctypes, ctypes.util, sys + +vk = ctypes.CDLL("libvulkan.so.1") + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32)] + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [("sType", ctypes.c_uint32), ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), ("pApplicationInfo", ctypes.c_void_p), + ("enabledLayerCount", ctypes.c_uint32), ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), ("ppEnabledExtensionNames", ctypes.c_void_p)] + +class VkMemoryType(ctypes.Structure): + _fields_ = [("propertyFlags", ctypes.c_uint32), ("heapIndex", ctypes.c_uint32)] + +class VkMemoryHeap(ctypes.Structure): + _fields_ = [("size", ctypes.c_uint64), ("flags", ctypes.c_uint32)] + +class VkPhysicalDeviceMemoryProperties(ctypes.Structure): + _fields_ = [("memoryTypeCount", ctypes.c_uint32), + ("memoryTypes", VkMemoryType * 32), + ("memoryHeapCount", ctypes.c_uint32), + ("memoryHeaps", VkMemoryHeap * 16)] + +app = VkApplicationInfo(0, None, b"memprops", 1, b"memprops", 1, 0x401000) +ci = VkInstanceCreateInfo(1, None, 0, ctypes.cast(ctypes.pointer(app), ctypes.c_void_p), 0, None, 0, None) +inst = ctypes.c_void_p() +rc = vk.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("vkCreateInstance rc=%d" % rc) +if rc != 0: + sys.exit(1) + +n = ctypes.c_uint32(0) +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +print("devices=%d" % n.value) +devs = (ctypes.c_void_p * n.value)() +vk.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) + +for i in range(n.value): + props = VkPhysicalDeviceMemoryProperties() + vk.vkGetPhysicalDeviceMemoryProperties(devs[i], ctypes.byref(props)) + print("=== DEV %d phys=0x%x ===" % (i, devs[i] or 0)) + print(" memoryTypeCount=%d memoryHeapCount=%d" % (props.memoryTypeCount, props.memoryHeapCount)) + for t in range(props.memoryTypeCount): + mt = props.memoryTypes[t] + print(" type[%d] flags=0x%08x heap=%d" % (t, mt.propertyFlags, mt.heapIndex)) + for h in range(props.memoryHeapCount): + mh = props.memoryHeaps[h] + print(" heap[%d] size=0x%x (%u MiB) flags=0x%x" % (h, mh.size, mh.size >> 20, mh.flags)) diff --git a/tools/guest-diag/wall2/vmap_watch2.py b/tools/guest-diag/wall2/vmap_watch2.py new file mode 100644 index 0000000..194b0d6 --- /dev/null +++ b/tools/guest-diag/wall2/vmap_watch2.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +# Guest-side live watcher v2: scan ALL vmallocinfo areas via /proc/kcore for +# (a) GR ring entry patterns, (b) pages starting with a small INC_METHOD header. +import struct, time +ENTRIES = [bytes.fromhex("0000480400980500"), bytes.fromhex("8000040004480598"), bytes.fromhex("0003d0000448059c")] +class Kcore: + def __init__(self): + self.fd = open("/proc/kcore", "rb") + hdr = self.fd.read(64) + e_phoff = struct.unpack_from("<Q", hdr, 32)[0] + e_phentsize, e_phnum = struct.unpack_from("<HH", hdr, 54) + self.segs = [] + for i in range(e_phnum): + self.fd.seek(e_phoff + i * e_phentsize) + ph = self.fd.read(56) + p_type, _fl, p_off, p_va, _pa, p_sz = struct.unpack_from("<IIQQQQ", ph, 0) + if p_type == 1: + self.segs.append((p_va, p_va + p_sz, p_off)) + def read(self, va, n): + for lo, hi, off in self.segs: + if lo <= va and va + n <= hi: + self.fd.seek(off + (va - lo)) + return self.fd.read(n) + return None +kc = Kcore() +def areas(): + out = [] + for line in open("/proc/vmallocinfo"): + p = line.split() + if len(p) < 2 or "-" not in p[0]: continue + try: + lo, hi = p[0].split("-") + lo, hi = int(lo,16), int(hi,16) + except Exception: + continue + out.append((lo, hi, line.strip()[-60:])) + return out +def hdr_ok(page): + v = struct.unpack_from("<I", page, 0)[0] + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return op == 1 and 1 <= cnt <= 8 +print("vmap_watch2 start", flush=True) +found = set() +t0 = time.time() +it = 0 +while time.time() - t0 < 300: + it += 1 + nareas = 0 + for lo, hi, tag in areas(): + size = hi - lo + if size > 8 << 20: continue + nareas += 1 + d = kc.read(lo, size) + if not d: continue + for pat in ENTRIES: + i = d.find(pat) + if i >= 0: + key = (lo, i & ~0xfff, pat.hex()) + if key not in found: + found.add(key) + print("%.2f RING %s VA 0x%x (+0x%x) %s" % (time.time()-t0, pat.hex(), lo+(i&~0xfff), i&~0xfff, tag), flush=True) + for off in range(0, size - 0x1000, 0x1000): + page = d[off:off+0x1000] + if hdr_ok(page): + key = (lo, off, "h") + if key not in found: + found.add(key) + print("%.2f HDR VA 0x%x (+0x%x) %s head=%s" % (time.time()-t0, lo+off, off, tag, page[:0x30].hex()), flush=True) + if it % 10 == 0: + print("%.1f iter %d areas %d" % (time.time()-t0, it, nareas), flush=True) + time.sleep(0.2) +print("vmap_watch2 done", flush=True) diff --git a/tools/guest-diag/wall2/walk.py b/tools/guest-diag/wall2/walk.py new file mode 100644 index 0000000..3624007 --- /dev/null +++ b/tools/guest-diag/wall2/walk.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +# walk.py - host-side replica of the service's svc_gmmu_walk + pd_discover, +# run against /dev/shm/vgpu-guest-ram. Discovers the page directory that +# reproduces the service's map[] sample pairs, then walks the ring/window +# VAs through it. +import mmap, struct, sys + +F = "/dev/shm/vgpu-guest-ram" +SEMA_PA_MAX = 0x180000000 + +# samples from the r690 dmesg map[] lines +SAMPLES = [ + (0x100000000, 0x100326000), + (0x100e2f000, 0x10096e000), + (0x100e33000, 0x10096b000), + (0x100e34000, 0x10096a000), + (0x100e35000, 0x100969000), + (0x100e37000, 0x102dcb000), + (0x100e38000, 0x102dca000), + (0x100e39000, 0x102dc9000), +] + +TARGET_VAS = [0x44a0000, 0x4500000, 0x4560000, 0x4480000, 0x4485000, + 0x4ca0000, 0x4ca5000, 0x4d00000, 0x4b17000, 0x4020000, + 0x4100000, 0x4160000] + +mm = None + +def off(pa): + return pa if pa < 0x80000000 else pa - 0x80000000 + +def rd(pa, n=8): + if pa + n > SEMA_PA_MAX: + return None + try: + d = mm[off(pa):off(pa)+n] + return d if len(d) == n else None + except Exception: + return None + +def rd64(pa): + d = rd(pa, 8) + return None if d is None else int.from_bytes(d, "little") + +def pde_aperture(ap): + if ap == 1: return "FB" + if ap in (2, 3): return "SYS" + return None + +def pte_aperture(ap): + if ap == 0: return "FB" + if ap == 1: return None + return "SYS" + +def pte_addr(e, fb): + if fb: + return ((e >> 8) & 0x1FFFFFF) << 12 + return ((e >> 8) & 0x3FFFFFFFFFFF) << 12 + +def pte_leaf(table, idx): + e = rd64(table + 8 * idx) + if e is None or not (e & 1): + return None + ap = pte_aperture((e >> 1) & 0x3) + if ap is None: + return None + a = pte_addr(e, ap == "FB") + if not fb_ok(a): + return None + return a, ap + +def fb_ok(a): + return a + 8 <= SEMA_PA_MAX + +def walk(va): + table = None + # root_bits = ilog2(entries=4) = 2; shift = 49-2 = 47 + e = rd64(table) if False else None + # level 0 + idx = (va >> 47) & 0x3 + # root table is the candidate itself (passed separately) + return None + +def walk_from(pdb, va, trace=False): + table = pdb + for level in range(3): + if level == 0: + bits, shift = 2, 47 + else: + bits, shift = 9, 38 if level == 1 else 29 + idx = (va >> shift) & ((1 << bits) - 1) + e = rd64(table + 8 * idx) + if e is None: + return None + if trace: + print(" L%d table=0x%x idx=%u e=0x%016x" % (level, table, idx, e)) + if e & 1: + return None + ap = pde_aperture((e >> 1) & 0x3) + if ap is None: + return None + table = pte_addr(e, ap == "FB") + if not fb_ok(table): + return None + idx = (va >> 21) & 0xff + big = rd64(table + 16 * idx) + small = rd64(table + 16 * idx + 8) + if big is None or small is None: + return None + if trace: + print(" PD0 table=0x%x idx=%u big=0x%016x small=0x%016x" % + (table, idx, big, small)) + if not (small & 1): + sap = pde_aperture((small >> 1) & 0x3) + if sap is not None: + t = pte_addr(small, sap == "FB") + r = pte_leaf(t, (va >> 12) & 0x1ff) + if r: + return r[0] + (va & 0xfff), r[1] + if big & 1: + bap = pte_aperture((big >> 1) & 0x3) + if bap is not None: + a = pte_addr(big, bap == "FB") + (va & ((1 << 21) - 1)) + if fb_ok(a): + return a, bap + else: + bap = pde_aperture((big >> 1) & 0x3) + if bap is not None: + t = pte_addr(big, bap == "FB") + r = pte_leaf(t, (va >> 16) & 0x1f) + if r: + return r[0] + (va & 0xffff), r[1] + return None + +def main(): + global mm + f = open(F, "rb") + mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + size = len(mm) + print("# scanning for directory candidates...") + cands = [] + for fo in range(0, size, 4096): + e = int.from_bytes(mm[fo:fo+8], "little") + if e and not (e & 1): + ap = (e >> 1) & 0x3 + if ap in (1, 2, 3): + pa = fo if fo < 0x80000000 else fo + 0x80000000 + if pa < SEMA_PA_MAX: + cands.append(pa) + print("# candidates: %d" % len(cands)) + best = [] + for c in cands: + hits = 0 + for (sva, spa) in SAMPLES: + r = walk_from(c, sva) + if r and (r[0] & ~0xfff) == (spa & ~0xfff): + hits += 1 + if hits >= 2: + best.append((hits, c)) + best.sort(reverse=True) + print("# candidates with >=2 hits: %d" % len(best)) + for hits, c in best[:10]: + print(" pdb=0x%x hits=%d/8" % (c, hits)) + if not best: + return 1 + pdb = best[0][1] + print("== walking targets through pdb=0x%x:" % pdb) + for va in TARGET_VAS: + r = walk_from(pdb, va) + if r: + print(" va=0x%x -> pa=0x%x ap=%s" % (va, r[0], r[1])) + else: + print(" va=0x%x -> no translation" % va) + mm.close(); f.close() + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap.py b/tools/guest-diag/wall2/wall2_livecap.py new file mode 100644 index 0000000..af68c03 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# wall2_livecap.py - live capture of the GR channel-init push page. +# +# The release group in the push is {0x20038090, sem_va_hi, sem_va_lo, payload, +# continuation}: sem_va = (hi<<32)|lo, and for the channel-init entry the sem +# sits at push_va + entry_len (entry0: len=1432=0x598, sem at +0x598), with the +# group itself at push_va + entry_len - 24 = +0x580. +# +# This watcher scans guest RAM for 0x20038090 groups during the client's stall, +# classifies them, and injects push_va -> page_pa via sema_map_add when the +# group's sem VA and offset identify the push page. Prints full evidence. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap.log" +MASK = ~0xfff & ((1 << 64) - 1) + +ENTRY_LEN = int(os.environ.get("W2_LEN", "1432")) +WATCH_S = int(os.environ.get("W2_WATCH", "120")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def cont_ok(v): + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return (op in (1, 3, 5) and cnt != 0) or op in (4, 7) + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + g = d[i:i+24] + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, hi, lo, pay, w5, v5, g)) + pos += CH + return out + +def main(): + log("=== livecap start len=%d watch=%ds ===" % (ENTRY_LEN, WATCH_S)) + injected = {} + deadline = time.time() + WATCH_S + while time.time() < deadline: + hits = scan() + cands = [] + for (page, off, hi, lo, pay, w5, v5, g) in hits: + sem = (hi << 32) | lo + # classify + pool = (hi == 1 and lo < 0x10000 and 0 < pay < (1 << 24) + and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4 + and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0) + if pool: + continue # CE pool release family, not the push + if not (0x4000000 <= sem < 0x100000000): + continue + if not (0 < pay < (1 << 24)): + continue + if not cont_ok(w5): + continue + push_va = sem - ENTRY_LEN + cands.append((page, off, sem, pay, push_va, w5, v5, g)) + if cands: + # prefer the canonical group offset 0x580 / sem-consistent push VA + cands.sort(key=lambda c: (c[1] != (ENTRY_LEN - 24), -c[3])) + page, off, sem, pay, push_va, w5, v5, g = cands[0] + d = rd(page, 0x100) + log("livecap: %d cand(s); page=0x%x off=0x%x sem=0x%x pay=%u push_va=0x%x w5=0x%08x v5=0x%08x" + % (len(cands), page, off, sem, pay, push_va, w5, v5)) + log(" page head: %s" % d[:0x40].hex()) + if (push_va & 0xfff) == 0 and push_va not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (push_va, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x (off=0x%x)" % (push_va, page, off)) + injected[push_va] = page + except OSError as e: + log(" inject FAIL", e) + time.sleep(1) + log("=== livecap done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap2.py b/tools/guest-diag/wall2/wall2_livecap2.py new file mode 100644 index 0000000..a33b25b --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap2.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# wall2_livecap2.py - capture the GR channel-init push page by its release +# group {0x20038090, hi, lo, payload, continuation} at page offset 0x598. +# The push VA comes from the service's own entry[0] decode in dmesg. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap2.log" +MASK = ~0xfff & ((1 << 64) - 1) +WATCH_S = int(os.environ.get("W2_WATCH", "180")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def prefix_ok(d): + off = 0 + while off + 4 <= len(d): + hdr = struct.unpack_from("<I", d, off)[0] + if hdr: break + off += 4 + if off + 4 > len(d): return False + op = (hdr >> 29) & 7 + if op in (4, 7): return True + if op not in (1, 3, 5): return False + return ((hdr >> 16) & 0xfff) != 0 + +def cont_ok(v): + op = (v >> 29) & 7 + cnt = (v >> 16) & 0xfff + return (op in (1, 3, 5) and cnt != 0) or op in (4, 7) + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + g = d[i:i+24] + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, hi, lo, pay, w5, v5)) + pos += CH + return out + +def main(): + log("=== livecap2 start watch=%ds ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + seen_pages = {} + while time.time() < deadline: + hits = scan() + for (page, off, hi, lo, pay, w5, v5) in hits: + if not (0 < pay < (1 << 24)): + continue + if not cont_ok(w5): + continue + d = rd(page, 0x1000) + if not d: continue + pok = prefix_ok(d[:0x100]) + key = (page, off) + if key in seen_pages: + continue + seen_pages[key] = 1 + log("hit page=0x%x off=0x%x sem=0x%x pay=%u w5=0x%08x v5=0x%08x prefix_ok=%s" + % (page, off, (hi << 32) | lo, pay, w5, v5, pok)) + log(" head: %s" % d[:0x30].hex()) + log(" @0x590: %s" % d[0x590:0x5c0].hex()) + if pok and off in (0x598, 0x590, 0x580, 0x588): + va = 0x4480000 + if va not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (va, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x" % (va, page)) + injected.add(va) + except OSError as e: + log(" inject FAIL", e) + time.sleep(0.5) + log("=== livecap2 done: %d injections ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_livecap3.py b/tools/guest-diag/wall2/wall2_livecap3.py new file mode 100644 index 0000000..db28bc2 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_livecap3.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# wall2_livecap3.py - capture the UVM GR channel-init push page via the host +# (C46F) semaphore release group emitted by uvm_hal_turing_host_semaphore_release: +# {0x20050017 (INC_METHOD count=5 addr=0x5c), sem_lo, sem_hi, payload, 0, exec} +# The group sits at push_va + entry_len - 24 = push_va + 0x580 (entry0 len 1432), +# with the header at +0x598 (entry1's 4-byte push). The push VA comes from the +# service's own entry[0] decode in dmesg. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_livecap3.log" +MASK = ~0xfff & ((1 << 64) - 1) +WATCH_S = int(os.environ.get("W2_WATCH", "240")) +PUSH_VA_DEFAULT = 0x4480000 + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def push_va_now(): + # newest entry[0] line for the Vulkan client's channel + va = None + for ln in dmesg().splitlines(): + m = re.search(r"entry\[0\] h=0x([0-9a-f]+) eng=0 raw=\S+ va=0x([0-9a-f]+) len=(\d+)", ln) + if m: + va = int(m.group(2), 16) + return va + +def scan(): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<I", 0x20050017) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + lo, hi, pay, w4, w5 = struct.unpack_from("<5I", d, i + 4) + # data-word sanity: payload_hi==0, RELEASE op, sane payload + if w4 != 0: continue + if (w5 & 0x7) != 1: continue + if not (0 < pay < (1 << 24)): continue + sem = ((hi & 0xffffffff) << 32) | ((lo & 0x3fffffff) << 2) + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, sem, pay, w5)) + pos += CH + return out + +def main(): + log("=== livecap3 start watch=%ds ===" % WATCH_S) + injected = {} + seen = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + hits = scan() + if hits: + for (page, off, sem, pay, w5) in hits: + key = (page, off) + if key in seen: continue + seen.add(key) + d = rd(page, 0x1000) + head = d[:0x20].hex() if d else "?" + log("hit page=0x%x off=0x%x sem=0x%x pay=%u exec=0x%08x head=%s" + % (page, off, sem, pay, w5, head)) + log(" @0x580: %s" % (d[0x580:0x5b0].hex() if d else "?")) + if off in (0x598, 0x580, 0x590): + pv = push_va_now() or PUSH_VA_DEFAULT + if pv not in injected: + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,1\n" % (pv, page, 0x1000)) + log(" INJECT 0x%x -> 0x%x" % (pv, page)) + injected[pv] = page + except OSError as e: + log(" inject FAIL", e) + time.sleep(0.5) + log("=== livecap3 done: %d injections ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_recscan.py b/tools/guest-diag/wall2/wall2_recscan.py new file mode 100644 index 0000000..7bc4e2c --- /dev/null +++ b/tools/guest-diag/wall2/wall2_recscan.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +# wall2_recscan.py - scan the guest RAM for the RM's low-aperture window +# records and their memdesc PAs. +# +# Record layout (verified r685/r686): 0x40-byte stride +# {u64 tag; u64 VA; u64 SIZE; u64 0; u32 0x7461696c ("tail" as a u32 - +# on disk the BYTES are "liat"); u32 0} +# The tag is NOT the byte string "tail" - searching for b'tail' finds +# nothing; search for the 4-byte pattern 6c 69 61 74 (struct '<I' +# 0x7461696c) at record+0x20 and decode {tag,VA,SIZE} at -0x20/-0x18/-0x10. +# +# Usage: wall2_recscan.py [VA_prefix_or_full_VA] +# Prints every record and the size-matching memdesc {SIZE,0x03008000,PA} +# found in the same page, and dumps the first words of each memdesc page so +# semaphore windows can be identified by the release payload values. +import struct, sys + +RAM = "/dev/shm/vgpu-guest-ram" +TAG = struct.pack("<I", 0x7461696c) + +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) + +def gpa2off(pa): + if pa < 0x80000000: + return pa + if pa < 0x180000000: + return 0x80000000 + (pa - 0x100000000) + return None + +def main(): + want = None + if len(sys.argv) > 1: + want = int(sys.argv[1], 0) + img = open(RAM, "rb", buffering=0) + img.seek(0, 2); fsz = img.tell() + recs = [] + pos = 0; CH = 64 << 20 + while pos < fsz: + img.seek(pos); buf = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = buf.find(TAG, st) + if i < 0: + break + st = i + 1 + if i >= 0x20: + tag, va, sz, z = struct.unpack_from("<QQQQ", buf, i - 0x20) + if 0x4000000 <= va < 0x100000000 and \ + 0x1000 <= sz <= 0x1000000 and (sz & 0xfff) == 0 and z == 0: + if want is None or va == want or (va & ~0xfffff) == (want & ~0xfffff): + recs.append((off2gpa(pos + i - 0x20), tag, va, sz)) + pos += CH + for gpa, tag, va, sz in sorted(recs, key=lambda r: r[2]): + page = gpa & ~0xfff + img.seek(gpa2off(page)); d = img.read(0x1000) + descs = [] + for o in range(0, 0x1000 - 24, 8): + s2, f2 = struct.unpack_from("<QQ", d, o) + pa = struct.unpack_from("<Q", d, o + 16)[0] + if f2 == 0x03008000 and s2 == sz and 0x100000000 <= pa < 0x180000000: + descs.append(pa) + print("rec gpa=0x%x tag=0x%x va=0x%x size=0x%x" % (gpa, tag, va, sz)) + for pa in descs: + img.seek(gpa2off(pa)); pd = img.read(0x20) + print(" desc pa=0x%x head=%s" % (pa, pd.hex(" "))) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver2.py b/tools/guest-diag/wall2/wall2_resolver2.py new file mode 100644 index 0000000..785e5e2 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver2.py @@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +# wall2_resolver2.py - HOST-SIDE wall#2 resolver v2 (r674+). +# +# Why: the vk_frame client's channel-init push (va=0x4500000 len=1432) is +# kernel-built and kernel-only; the sema service cannot translate it +# ("false translation for push ... rescuing"), the client times out (~11 s) +# per attempt, retries, then VKFAIL -4. Guest RAM is host-readable, so the +# push page can be found and injected via sema_map_add. +# +# v2 recipe (per the r672/r673 structure analysis): +# 1. wait for the channel ring line in dmesg (validated RING_PA + handle) +# 2. scan guest RAM for the push page: +# a) CONTENT: the 24-byte release group the service's own rescue matches +# (hdr=0x20038090, hi=1, payload sane, w5/v5 shape). The channel-init +# entry (va=0x4500000 len=1432) puts that group at page offset 0x580. +# b) TUPLES: pages of 32-byte records {vmap_ptr; flags; PA; size} (both +# observed layouts) matched on (PA & ~0xfff) == RING_PA with strides +# 0x20/0x40/0x60; entries below the ring entry in VA order map +# 0x4500000.. (push VA = ring VA 0x4520000 - 0x20000 = 32 pages). +# 3. inject (va -> pa, len) through /sys/module/vgsp/parameters/sema_map_add +# 4. keep scanning/re-injecting for the watch window (retries allocate new +# pages) and log everything. +# +# Run this FIRST (foreground), then launch the client in the guest. +import subprocess, struct, time, sys, re, os, signal + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver2.log" +RING_VA = 0x4520000 +PUSH_VA = 0x4500000 +DELTA = (RING_VA - PUSH_VA) // 0x1000 # 32 pages +REL_OFF = 0x580 # group offset in the push page +MASK = ~0xfff & ((1 << 64) - 1) + +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "150")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap)) + log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln)) + return True + except OSError as e: + log("inject FAIL", e) + return False + +def prefix_ok(d): + off = 0 + while off + 4 <= len(d): + hdr = struct.unpack_from("<I", d, off)[0] + if hdr: + break + off += 4 + if off + 4 > len(d): + return False + op = (hdr >> 29) & 7 + if op in (4, 7): + return True + if op not in (1, 3, 5): + return False + return ((hdr >> 16) & 0xfff) != 0 + +def rel_shape(g): + hdr, hi, lo, pay, w5, v5 = struct.unpack("<6I", g) + return (hdr == 0x20038090 and hi == 1 and 0 < pay < (1 << 24) + and (w5 >> 29) == 1 and ((w5 >> 13) & 7) == 4 + and ((w5 & 0x1fff) << 2) == 0x300 and (v5 & 3) == 0), lo & 0xfff, pay + +img = None +def open_ram(): + global img + if img is None: + img = open(RAM, "rb", buffering=0) + return img + +def gpa2off(pa): + if pa < 0x80000000: + return pa + if pa < 0x180000000: + return 0x80000000 + (pa - 0x100000000) + return None + +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) + +def rd(pa, n): + off = gpa2off(pa) + if off is None: + return None + f = open_ram() + try: + f.seek(off) + return f.read(n) + except OSError: + return None + +def find_content_candidates(): + """Pages containing the release group; returns [(page_pa, off, slot, pay)].""" + f = open_ram() + f.seek(0, 2) + fsz = f.tell() + pat = b"\x90\x80\x03\x20" + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + f.seek(pos) + d = f.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: + break + st = i + 1 + if i + 24 > len(d): + continue + ok, slot, pay = rel_shape(d[i:i + 24]) + if not ok: + continue + gpa = off2gpa(pos + i) + page = gpa & MASK + off = gpa - page + out.append((page, off, slot, pay)) + pos += CH + return out + +def find_tuple_arrays(ring_pa): + """Find record pages containing the ring PA; returns list of + (page_gpa, stride, ring_index, entries[list of pa]).""" + f = open_ram() + f.seek(0, 2) + fsz = f.tell() + pat = struct.pack("<Q", ring_pa) + arrays = [] + CH = 64 << 20 + pos = 0 + hits = [] + while pos < fsz: + f.seek(pos) + d = f.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: + break + hits.append(off2gpa(pos + i)) + st = i + 1 + pos += CH + seen = set() + for gpa in hits: + page = gpa & MASK + if page in seen: + continue + seen.add(page) + d = rd(page, 0x1000) + if not d: + continue + rec_off = gpa - page + for stride in (0x20, 0x40, 0x60): + for rel in range(0, min(stride, rec_off + 1), 8): + start = rec_off - rel + ents = [] + for o in range(start, 0x1000 - stride + 1, stride): + # PA field anywhere in the record: try +0x10 first, then any qword + vals = struct.unpack_from("<%dQ" % (stride // 8), d, o) + pa = None + for cand_off in (0x10, 0x8, 0x18, 0x0, 0x20, 0x28): + if cand_off + 8 <= stride: + v = vals[cand_off // 8] + if (v & MASK) == ring_pa: + pa = v & MASK + break + if pa is None: + continue + ents.append((o, pa)) + if len(ents) >= 4: + idx = [e[1] for e in ents].index(ring_pa) + arrays.append((page, stride, start, idx, [e[1] for e in ents])) + break + if arrays and arrays[-1][0] == page: + break + return arrays + +def main(): + log("=== wall2_resolver2 start (wait_ring=%ss watch=%ss) ===" % (WAIT_RING_S, WATCH_S)) + ring_pa = None + ring_h = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + for ln in dmesg().splitlines(): + m = re.search(r"h=(0x[0-9a-f]+) .*ring rc=0 pa=0x([0-9a-f]+)", ln) + if m: + ring_h = m.group(1) + ring_pa = int(m.group(2), 16) + break + if ring_pa: + break + time.sleep(1) + if not ring_pa: + log("no ring line seen in %ss; exiting" % WAIT_RING_S) + return 1 + log("RING %s PA=0x%x" % (ring_h, ring_pa)) + + injected = set() + watch_deadline = time.time() + WATCH_S + last_pass = 0 + while time.time() < watch_deadline: + # ---- content pass + try: + cands = find_content_candidates() + except Exception as e: + log("content scan error:", e) + cands = [] + # prefer the canonical group offset 0x580, then highest payload + cands.sort(key=lambda c: (c[1] != REL_OFF, -c[3])) + if cands: + page, off, slot, pay = cands[0] + d = rd(page, 0x1000) + pok = bool(d) and prefix_ok(d[:0x100]) + log("content: %d candidate(s); best page=0x%x off=0x%x slot=0x%x pay=%u prefix_ok=%s" + % (len(cands), page, off, slot, pay, pok)) + key = ("C", PUSH_VA, page) + if key not in injected and pok: + if inject(PUSH_VA, page, 0x1000): + injected.add(key) + # also cover the following page (entry2 tail) if non-contiguous unknown; skip + else: + # ---- tuple fallback (only every ~10 s to save time) + if time.time() - last_pass > 10: + last_pass = time.time() + try: + arrs = find_tuple_arrays(ring_pa) + except Exception as e: + log("tuple scan error:", e) + arrs = [] + if arrs: + page, stride, start, idx, ents = arrs[0] + log("tuple: array page=0x%x stride=0x%x ring_index=%d entries=%d" + % (page, stride, idx, len(ents))) + for k in range(idx - DELTA, idx): + if k < 0 or k >= len(ents): + continue + va = PUSH_VA + (k - (idx - DELTA)) * 0x1000 + pa = ents[k] + key = ("T", va, pa) + if key in injected: + continue + d = rd(pa, 0x40) + if d and any(d): + inject(va, pa, 0x1000) + injected.add(key) + else: + log("tuple: no array with ring PA found") + # stop early when the service confirms our mapping + if len(injected) and "injected mapping va=0x%x" % PUSH_VA in dmesg(): + log("service confirmed the injected mapping; done") + break + time.sleep(1) + log("=== done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver3.py b/tools/guest-diag/wall2/wall2_resolver3.py new file mode 100644 index 0000000..4b16686 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver3.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +# wall2_resolver3.py - HOST-SIDE wall#2 resolver v3: the RM window-memdesc recipe. +# +# Proven structure (r674/r675): the RM keeps per-window records +# {..., VA, SIZE, ...} (head/tail list nodes) +# and per-window memdescs in the same page: +# {SIZE, 0x03008000, FIRST_PA, ptr} +# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring +# window {VA=0x44a0000, SIZE=0x40000} -> memdesc PA == the service's validated +# ring pa (0x1158ce000 r675). Therefore the push window +# {VA=0x4480000, SIZE=0x20000} -> memdesc PA == PA of VA 0x4480000 (the push +# page the sema service cannot translate). +# +# Recipe: wait for the ring line -> find the ring window memdesc +# {0x40000, 0x03008000, RING_PA} -> in the same page find the push window +# memdesc {0x20000, 0x03008000, PUSH_PA} -> inject 0x4480000 -> PUSH_PA. +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver3.log" +PUSH_VA = 0x4480000 +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "150")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + try: + with open(PARAM, "w") as f: + f.write("0x%x,0x%x,0x%x,%d\n" % (va, pa, ln, ap)) + log("INJECT va=0x%x -> pa=0x%x len=0x%x" % (va, pa, ln)) + return True + except OSError as e: + log("inject FAIL", e) + return False + +def find_memdescs(size): + """Find {size, 0x03008000, PA} patterns; returns [(page, off, pa)].""" + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", size, 0x03008000) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + if i + 24 > len(d): continue + pa = struct.unpack_from("<Q", d, i + 16)[0] + gpa = off2gpa(pos + i) + if 0x100000000 <= pa < 0x180000000: + out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff), pa)) + pos += CH + return out + +def main(): + log("=== resolver3 start ===") + # count pre-existing ring lines; wait for a NEW one (fresh client kick) + def ring_lines(): + out = [] + for ln in dmesg().splitlines(): + m = re.search(r"ring rc=0 pa=0x([0-9a-f]+) ap=", ln) + if m: + out.append(int(m.group(1), 16)) + return out + base = ring_lines() + log("pre-existing ring lines: %d" % len(base)) + ring_pa = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + cur = ring_lines() + if len(cur) > len(base): + ring_pa = cur[-1] + break + time.sleep(1) + if not ring_pa: + log("no NEW ring line; exit"); return 1 + log("NEW RING PA = 0x%x" % ring_pa) + + injected = set() + watch = time.time() + WATCH_S + while time.time() < watch: + # ring window memdesc {0x40000, 0x03008000, ring_pa} + rd_hits = [x for x in find_memdescs(0x40000) if x[2] == ring_pa] + if not rd_hits: + time.sleep(0.5); continue + for (page, off, _pa) in rd_hits: + d = rd(page, 0x1000) + if not d: continue + # find sibling push memdesc in the same page + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if not (0x100000000 <= pa < 0x180000000): + continue + key = (page, pa) + if key in injected: continue + log("memdesc page=0x%x ring_desc@0x%x push_desc@0x%x push_pa=0x%x" + % (page, off, o, pa)) + # content sanity: read the push page + pd = rd(pa, 0x40) + log(" push page head: %s" % (pd.hex() if pd else "?")) + if inject(PUSH_VA, pa, 0x1000): + injected.add(key) + if injected and ("injected mapping va=0x%x" % PUSH_VA) in dmesg(): + log("service confirmed; keep watching") + time.sleep(0.5) + log("=== resolver3 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver4.py b/tools/guest-diag/wall2/wall2_resolver4.py new file mode 100644 index 0000000..140aaa7 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver4.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# wall2_resolver4.py - HOST-SIDE wall#2 resolver v4 (final recipe). +# +# Association proven on r675: the RM keeps, in ONE page, the client's channel +# window records and their memdescs: +# push window record {VA=0x4480000, SIZE=0x20000} +# ring window record {VA=0x44a0000, SIZE=0x40000} +# push memdesc {0x20000, 0x03008000, FIRST_PA, ptr} +# ring memdesc {0x40000, 0x03008000, FIRST_PA, ptr} +# where FIRST_PA is the PA of the window's FIRST VA page. Verified: the ring +# memdesc's PA == the service's validated ring pa. So the push memdesc's PA == +# the PA of VA 0x4480000 (the push page the service cannot translate). +# +# Recipe: wait for a NEW ring line -> find the page containing the ring window +# record {0x44a0000, 0x40000} -> require the sibling push window record +# {0x4480000, 0x20000} and the memdescs in the same page -> cross-check the +# ring memdesc PA == ring_pa -> inject 0x4480000 -> push memdesc PA (sudo). +import subprocess, struct, time, sys, re, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver4.log" +PUSH_VA = 0x4480000 +RING_VA = 0x44a0000 +WAIT_RING_S = int(os.environ.get("W2_WAIT_RING", "420")) +WATCH_S = int(os.environ.get("W2_WATCH", "180")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def dmesg(): + return subprocess.run(["sudo", "dmesg"], capture_output=True, text=True).stdout + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x len=0x%x OK" % (va, pa, ln)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + """Find qword pairs {a,b}; returns [(page, off)].""" + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + gpa = off2gpa(pos + i) + out.append((gpa & ~0xfff, gpa - (gpa & ~0xfff))) + pos += CH + return out + +def scan_page_descs(page): + d = rd(page, 0x1000) + if not d: return [] + out = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if fl != 0x03008000: continue + if sz not in (0x20000, 0x40000): continue + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + out.append((sz, o, pa)) + return out + +def main(): + log("=== resolver4 start ===") + def ring_lines(): + return re.findall(r"ring rc=0 pa=0x([0-9a-f]+) ap=", dmesg()) + base = ring_lines() + log("pre-existing ring lines: %d" % len(base)) + ring_pa = None + deadline = time.time() + WAIT_RING_S + while time.time() < deadline: + cur = ring_lines() + if len(cur) > len(base): + ring_pa = int(cur[-1], 16) + break + time.sleep(1) + if not ring_pa: + log("no NEW ring line; exit"); return 1 + log("NEW RING PA = 0x%x" % ring_pa) + + injected = set() + LEARNED = [0x101e39000, 0x102a62000] # RM window-record pages (stable across runs) + def try_page(page): + descs = scan_page_descs(page) + rdesc = [x for x in descs if x[0] == 0x40000] + pdesc = [x for x in descs if x[0] == 0x20000] + if not rdesc or not pdesc: + return False + if rdesc[0][2] != ring_pa: + return False + ok = False + for (_sz, o, pa) in pdesc: + key = (page, o) + if key in injected: continue + pd = rd(pa, 0x40) + log("page=0x%x ring_desc@0x%x(pa=0x%x OK) push_desc@0x%x pa=0x%x head=%s" + % (page, rdesc[0][1], rdesc[0][2], o, pa, + pd.hex() if pd else "?")) + if inject(PUSH_VA, pa, 0x1000): + injected.add(key); ok = True + return ok + log("learned pages: %s" % [hex(p) for p in LEARNED]) + watch = time.time() + WATCH_S + last_full = 0 + while time.time() < watch: + hit = False + for page in LEARNED: + if try_page(page): + hit = True + if not hit and time.time() - last_full > 5: + last_full = time.time() + for (page, _off) in find_qpair(RING_VA, 0x40000): + if try_page(page): + if page not in LEARNED: + LEARNED.append(page) + hit = True + time.sleep(0.25) + log("=== resolver4 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver5.py b/tools/guest-diag/wall2/wall2_resolver5.py new file mode 100644 index 0000000..faec902 --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver5.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +# wall2_resolver5.py - final: continuous scan, all-candidate injection. +# +# Structure (r675, verified): an RM page holds the client channel's window +# records and memdescs: +# {0x4480000, 0x20000} push window record +# {0x44a0000, 0x40000} ring window record +# {0x20000, 0x03008000, PUSH_PA} push window memdesc +# {0x40000, 0x03008000, RING_PA} ring window memdesc (PA == validated ring pa) +# The resolver scans CONTINUOUSLY (no ring-line gate), finds pages holding the +# pair of window records, and injects 0x4480000 -> every 0x20000 memdesc PA in +# the same page (last write wins in the service; content check arbitrates). +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver5.log" +PUSH_VA = 0x4480000 +RING_VA = 0x44a0000 +WATCH_S = int(os.environ.get("W2_WATCH", "600")) + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + gpa = off2gpa(pos + i) + out.append(gpa & ~0xfff) + pos += CH + return out + +def main(): + log("=== resolver5 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + try: + push_pages = set(find_qpair(PUSH_VA, 0x20000)) + ring_pages = set(find_qpair(RING_VA, 0x40000)) + except Exception as e: + log("scan error:", e); time.sleep(2); continue + for page in sorted(push_pages & ring_pages): + d = rd(page, 0x1000) + if not d: continue + # collect 0x20000 descs in this page + cands = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + cands.append((o, pa)) + for (o, pa) in cands: + key = (page, o) + if key in injected: continue + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("page=0x%x desc@0x%x pa=0x%x live_nz=%d head=%s" + % (page, o, pa, nz, pd.hex()[:64] if pd else "?")) + if inject(PUSH_VA, pa): + injected.add(key) + time.sleep(1) + log("=== resolver5 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver6.py b/tools/guest-diag/wall2/wall2_resolver6.py new file mode 100644 index 0000000..4653efa --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver6.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +# wall2_resolver6.py - resolver5 logic for BOTH current push windows: +# pair A: push 0x4480000 (0x20000) + ring 0x44a0000 (0x40000) +# pair B: push 0x4ca0000 (0x20000) + ring 0x4cc0000 (0x40000) +# Scans guest RAM continuously; injects PUSH_VA -> every 0x20000 memdesc PA +# found in a page holding both window records. +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver6.log" +WATCH_S = int(os.environ.get("W2_WATCH", "420")) +PAIRS = [(0x4480000, 0x44a0000), (0x4ca0000, 0x4cc0000)] + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def off2gpa(off): + return off if off < 0x80000000 else 0x100000000 + (off - 0x80000000) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + out.append(off2gpa(pos + i) & ~0xfff) + pos += CH + return out + +def main(): + log("=== resolver6 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + for (pva, rva) in PAIRS: + try: + pages = set(find_qpair(pva, 0x20000)) & set(find_qpair(rva, 0x40000)) + except Exception as e: + log("scan error:", e); continue + for page in sorted(pages): + d = rd(page, 0x1000) + if not d: continue + cands = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + if sz == 0x20000 and fl == 0x03008000: + pa = struct.unpack_from("<Q", d, o + 16)[0] + if 0x100000000 <= pa < 0x180000000: + cands.append((o, pa)) + for (o, pa) in cands: + key = (pva, page, o) + if key in injected: continue + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("pair=0x%x page=0x%x desc@0x%x pa=0x%x live_nz=%d head=%s" + % (pva, page, o, pa, nz, pd.hex()[:64] if pd else "?")) + if inject(pva, pa): + injected.add(key) + time.sleep(1) + log("=== resolver6 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/wall2/wall2_resolver7.py b/tools/guest-diag/wall2/wall2_resolver7.py new file mode 100644 index 0000000..5d9e58e --- /dev/null +++ b/tools/guest-diag/wall2/wall2_resolver7.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +# wall2_resolver7.py - resolver6 + release-window injection + live-first ordering. +# Fixes the resolver6 zero-page overwrite: candidates are injected good-first +# (live_nz>0) and zero pages last, so the readable mapping wins by order if the +# service keeps the newest write. +# New: the GR init release windows (0x4b16000 size 0x1000 from the record page, +# and any {VA,0x1000} record alongside) are injected from the size-0x1000 +# memdescs of the same page. +import subprocess, struct, time, sys, os + +RAM = "/dev/shm/vgpu-guest-ram" +PARAM = "/sys/module/vgsp/parameters/sema_map_add" +LOG = "/tmp/opencode/wall2_resolver7.log" +WATCH_S = int(os.environ.get("W2_WATCH", "420")) +PAIRS = [(0x4480000, 0x44a0000), (0x4ca0000, 0x4cc0000)] +REL_WINS = [0x4b16000] + +logf = open(LOG, "a", buffering=1) +def log(*a): + s = " ".join(str(x) for x in a) + print(s, flush=True) + logf.write("[%s] %s\n" % (time.strftime("%H:%M:%S"), s)) + +img = open(RAM, "rb", buffering=0) +def off2gpa(off): return off if off < 0x80000000 else 0x100000000+(off-0x80000000) +def gpa2off(pa): + if pa < 0x80000000: return pa + if pa < 0x180000000: return 0x80000000 + (pa - 0x100000000) + return None +def rd(pa, n): + off = gpa2off(pa) + if off is None: return None + img.seek(off); return img.read(n) + +def inject(va, pa, ln=0x1000, ap=1): + cmd = "echo 0x%x,0x%x,0x%x,%d > %s" % (va, pa, ln, ap, PARAM) + r = subprocess.run(["sudo", "sh", "-c", cmd], capture_output=True, text=True) + if r.returncode == 0: + log("INJECT va=0x%x -> pa=0x%x" % (va, pa)) + return True + log("inject FAIL rc=%d %s" % (r.returncode, r.stderr.strip())) + return False + +def find_qpair(a, b): + img.seek(0, 2); fsz = img.tell() + pat = struct.pack("<QQ", a, b) + out = [] + CH = 64 << 20 + pos = 0 + while pos < fsz: + img.seek(pos); d = img.read(min(CH, fsz - pos)) + st = 0 + while True: + i = d.find(pat, st) + if i < 0: break + st = i + 1 + out.append(off2gpa(pos + i) & ~0xfff) + pos += CH + return out + +def collect(page): + d = rd(page, 0x1000) + if not d: return None + descs = [] + wins = [] + for o in range(0, 0x1000 - 24, 8): + sz, fl = struct.unpack_from("<QQ", d, o) + pa = struct.unpack_from("<Q", d, o + 16)[0] + if fl == 0x03008000 and 0x1000 <= sz <= 0x400000 and \ + 0x100000000 <= pa < 0x180000000: + descs.append((o, sz, pa)) + va2 = sz + if 0x4000000 <= va2 < 0x100000000 and (va2 & 0xfff) == 0: + sz2 = fl + if 0x1000 <= sz2 <= 0x1000000 and (sz2 & 0xfff) == 0: + wins.append((va2, sz2)) + return descs, wins + +def main(): + log("=== resolver7 start (watch=%ds) ===" % WATCH_S) + injected = set() + deadline = time.time() + WATCH_S + while time.time() < deadline: + pages = set() + for (pva, rva) in PAIRS: + try: + pages |= set(find_qpair(pva, 0x20000)) & set(find_qpair(rva, 0x40000)) + except Exception as e: + log("scan error:", e) + for page in sorted(pages): + got = collect(page) + if not got: continue + descs, wins = got + # push windows: inject live-first (nz>0 first, zeros last) + for (pva, _rva) in PAIRS: + cands = [(o, sz, pa) for (o, sz, pa) in descs if sz == 0x20000] + order = [] + for (o, sz, pa) in cands: + pd = rd(pa, 0x40) + nz = sum(1 for b in pd if b) if pd else -1 + log("pair=0x%x page=0x%x desc@0x%x pa=0x%x live_nz=%d" % (pva, page, o, pa, nz)) + order.append((nz, o, pa)) + order.sort(key=lambda t: t[0]) # zeros first, live last + for (nz, o, pa) in order: + key = (pva, page, o) + if key in injected: continue + if inject(pva, pa): + injected.add(key) + # release windows: any {VA, 0x1000} record on the page -> size-0x1000 descs + for (va, sz) in wins: + if va not in REL_WINS and not (0x4b00000 <= va <= 0x4b20000): + continue + for (o, dsz, pa) in descs: + if dsz != sz: + continue + key = (va, page, o) + if key in injected: continue + if inject(va, pa): + injected.add(key) + time.sleep(1) + log("=== resolver7 done: %d injection(s) ===" % len(injected)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/guest-diag/xfocus.c b/tools/guest-diag/xfocus.c new file mode 100644 index 0000000..e0ec05d --- /dev/null +++ b/tools/guest-diag/xfocus.c @@ -0,0 +1,69 @@ +// xfocus.c - print the current X input focus and the gzdoom window, and try +// to focus the gzdoom window; then report again. Diagnostic for WM-less X. +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <string.h> + +static Window find_named(Display *d, Window root, const char *sub, + char *title, size_t tl) +{ + Window r, parent, *kids = NULL; + unsigned int nk = 0; + + title[0] = 0; + if (!XQueryTree(d, root, &r, &parent, &kids, &nk)) + return 0; + for (unsigned int i = 0; i < nk; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + if (strstr(name, sub)) { + snprintf(title, tl, "%s", name); + Window w = kids[i]; + + XFree(name); + if (kids) + XFree(kids); + return w; + } + XFree(name); + } + } + if (kids) + XFree(kids); + return 0; +} + +int main(void) +{ + Display *d = XOpenDisplay(NULL); + Window root, focus, w, croot; + int rx, ry, wx, wy; + unsigned int mask; + char title[256]; + + if (!d) { + printf("no display\n"); + return 1; + } + root = DefaultRootWindow(d); + w = find_named(d, root, "Freedoom", title, sizeof(title)); + XGetInputFocus(d, &focus, &rx); + printf("gzdoom window: 0x%lx '%s'\n", w, title); + printf("input focus: 0x%lx\n", focus); + printf("pointer window: "); + if (XQueryPointer(d, root, &croot, &rx /*reuse*/, &wx, &wy, &rx, &ry, + &mask)) + printf("0x%lx at %d,%d mask=0x%x\n", rx, wx, wy, mask); + else + printf("query failed\n"); + if (w) { + XSetInputFocus(d, w, RevertToParent, CurrentTime); + XFlush(d); + XGetInputFocus(d, &focus, &rx); + printf("after set: 0x%lx\n", focus); + } + XCloseDisplay(d); + return 0; +} diff --git a/tools/guest-diag/xgrab.c b/tools/guest-diag/xgrab.c new file mode 100644 index 0000000..eb605d6 --- /dev/null +++ b/tools/guest-diag/xgrab.c @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xgrab.c - capture the X screen (root window) into an append-only frame file. +// Pure client-side XGetImage on a software X server (Xvfb): no GL, no driver, +// no present-path hooks. Same record format as sdl_cap.c: +// [0] 8B "GZCAPV01" + zero pad to 4096 +// [4096..] records: 4B "GZF2" | u32 w | u32 h | u32 stride | u32 bpp | +// u64 ts_ns | u64 seq | stride*h bytes (top-down BGRA) +// Trailing record seq=0xFFFFFFFFFFFFFFFF on clean stop. +// Usage: xgrab <display> <out> <maxframes|0> <pace_us|0> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> + +int main(int argc, char **argv) +{ + const char *dpy = argc > 1 ? argv[1] : ":1"; + const char *out = argc > 2 ? argv[2] : "/dev/shm/gzframes.bin"; + long maxf = argc > 3 ? atol(argv[3]) : 0; + long pace = argc > 4 ? atol(argv[4]) : 0; + int r_x = argc > 5 ? atoi(argv[5]) : -1; + int r_y = argc > 6 ? atoi(argv[6]) : 0; + int r_w = argc > 7 ? atoi(argv[7]) : 0; + int r_h = argc > 8 ? atoi(argv[8]) : 0; + Display *d; + Window root; + XWindowAttributes wa; + FILE *f; + char hdr[4096]; + uint64_t seq = 0; + + d = XOpenDisplay(dpy); + if (!d) { + fprintf(stderr, "xgrab: cannot open display %s\n", dpy); + return 1; + } + root = DefaultRootWindow(d); + if (!XGetWindowAttributes(d, root, &wa)) { + fprintf(stderr, "xgrab: no root attrs\n"); + return 1; + } + if (r_x < 0 || r_w <= 0 || r_h <= 0 || r_x + r_w > wa.width || + r_y + r_h > wa.height) { + r_x = 0; r_y = 0; r_w = wa.width; r_h = wa.height; + } + fprintf(stderr, "xgrab: root %dx%d depth=%d region %d,%d %dx%d\n", + wa.width, wa.height, wa.depth, r_x, r_y, r_w, r_h); + f = fopen(out, "wb"); + if (!f) { + perror("xgrab: open out"); + return 1; + } + memset(hdr, 0, sizeof(hdr)); + memcpy(hdr, "GZCAPV01", 8); + fwrite(hdr, 1, sizeof(hdr), f); + fflush(f); + + while (!maxf || (long)seq < maxf) { + XImage *im = XGetImage(d, root, r_x, r_y, r_w, r_h, AllPlanes, + ZPixmap); + struct timespec ts; + uint64_t tns; + uint32_t rh[4]; + char m[4] = { 'G', 'Z', 'F', '2' }; + int y; + + if (!im) { + usleep(10000); + continue; + } + if (im->bits_per_pixel != 32) { + fprintf(stderr, "xgrab: unexpected bpp %d\n", + im->bits_per_pixel); + XDestroyImage(im); + break; + } + clock_gettime(CLOCK_MONOTONIC, &ts); + tns = (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; + rh[0] = (uint32_t)r_w; + rh[1] = (uint32_t)r_h; + rh[2] = (uint32_t)r_w * 4; + rh[3] = 32; + fwrite(m, 1, 4, f); + fwrite(rh, 4, 4, f); + fwrite(&tns, 8, 1, f); + fwrite(&seq, 8, 1, f); + for (y = 0; y < r_h; y++) + fwrite(im->data + (size_t)y * im->bytes_per_line, 1, + (size_t)r_w * 4, f); + fflush(f); + seq++; + XDestroyImage(im); + if (pace) + usleep(pace); + if ((seq & 63) == 0) + fprintf(stderr, "xgrab: frames=%llu\n", + (unsigned long long)seq); + } + { + char m[4] = { 'G', 'Z', 'F', '2' }; + uint64_t end = 0xFFFFFFFFFFFFFFFFull; + + fwrite(m, 1, 4, f); + fwrite(&end, 8, 1, f); + fflush(f); + } + fclose(f); + XCloseDisplay(d); + fprintf(stderr, "xgrab: done frames=%llu\n", (unsigned long long)seq); + return 0; +} diff --git a/tools/guest-diag/xorg-vgpu.conf b/tools/guest-diag/xorg-vgpu.conf new file mode 100644 index 0000000..4d4c07d --- /dev/null +++ b/tools/guest-diag/xorg-vgpu.conf @@ -0,0 +1,31 @@ +# Minimal Xorg config for the emulated vGPU (nvidia driver), headless. +# AllowEmptyInitialConfiguration lets the X server start with no connected +# display; UseDisplayDevice "None" keeps the server from demanding one. +Section "ServerLayout" + Identifier "Layout0" + Screen 0 "Screen0" +EndSection + +Section "Device" + Identifier "Device0" + Driver "nvidia" + BusID "PCI:1:0:0" + Option "AllowEmptyInitialConfiguration" "True" + Option "UseDisplayDevice" "None" + Option "ConnectToAcpid" "False" +EndSection + +Section "Screen" + Identifier "Screen0" + Device "Device0" + DefaultDepth 24 + SubSection "Display" + Depth 24 + Virtual 1280 720 + EndSubSection +EndSection + +Section "ServerFlags" + Option "AutoAddDevices" "False" + Option "AutoEnableDevices" "False" +EndSection
\ No newline at end of file diff --git a/tools/guest-diag/xresize.c b/tools/guest-diag/xresize.c new file mode 100644 index 0000000..877f0e9 --- /dev/null +++ b/tools/guest-diag/xresize.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xresize.c - find a window by name substring and resize/move it (Xvfb has +// no WM, so a client-side XMoveResizeWindow is the only way to force the +// game window onto the screen). +// Usage: xresize <display> <name-substr> <w> <h> <x> <y> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <string.h> +#include <stdlib.h> + +static Window find_named(Display *d, Window root, const char *sub) +{ + Window r, parent, *kids = NULL; + unsigned int nkids = 0; + + if (!XQueryTree(d, root, &r, &parent, &kids, &nkids)) + return 0; + for (unsigned int i = 0; i < nkids; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + int hit = strstr(name, sub) != NULL; + + XFree(name); + if (hit) { + Window w = kids[i]; + + if (kids) + XFree(kids); + return w; + } + } + } + if (kids) + XFree(kids); + return 0; +} + +int main(int argc, char **argv) +{ + Display *d; + Window w; + + if (argc < 6) { + fprintf(stderr, "usage: xresize <display> <name> <w> <h> <x> <y>\n"); + return 2; + } + d = XOpenDisplay(argv[1]); + if (!d) { + fprintf(stderr, "no display\n"); + return 1; + } + w = find_named(d, DefaultRootWindow(d), argv[2]); + if (!w) { + fprintf(stderr, "window '%s' not found\n", argv[2]); + return 1; + } + XMoveResizeWindow(d, w, atoi(argv[5]), atoi(argv[6]), atoi(argv[3]), + atoi(argv[4])); + XFlush(d); + printf("resized 0x%lx to %sx%s at %s,%s\n", w, argv[3], argv[4], + argv[5], argv[6]); + XCloseDisplay(d); + return 0; +} diff --git a/tools/guest-diag/xshm_pub.c b/tools/guest-diag/xshm_pub.c new file mode 100644 index 0000000..98fb539 --- /dev/null +++ b/tools/guest-diag/xshm_pub.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * xshm_pub.c - capture an X11 screen and publish frames into the kvmfr + * ivshmem BAR so the host can display them (the Looking-Glass-style path: + * the guest renders with its real graphics stack - X on the passed GPU / + * GLX / Vulkan surface - and this tool copies the framebuffer into the + * shared BAR; the host reader decodes it). + * + * Layout in the BAR (same as tools/cuda-ab/frame_producer.cu): + * offset 0x100000: struct frame_hdr { magic, version, width, height, + * stride, format, seq } then width*height*4 RGBA8 bytes. + * + * Usage (guest, with DISPLAY set): + * ./xshm_pub [frames] [fps] [x] [y] [w] [h] + * Prints FRAME lines (first 5 and every 30th) then FRAME_DONE. + * Build (host, same glibc): gcc -O2 -o xshm_pub xshm_pub.c -lX11 -lXext + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <sys/mman.h> +#include <time.h> +#include <X11/Xlib.h> +#include <X11/Xutil.h> +#include <X11/extensions/XShm.h> +#include <sys/ipc.h> +#include <sys/shm.h> + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u /* "FMVK" */ + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; /* 0 = RGBA8 */ + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +int main(int argc, char **argv) +{ + int frames = argc > 1 ? atoi(argv[1]) : 120; + int fps = argc > 2 ? atoi(argv[2]) : 10; + int gx = argc > 3 ? atoi(argv[3]) : 0; + int gy = argc > 4 ? atoi(argv[4]) : 0; + int gw = argc > 5 ? atoi(argv[5]) : 0; + int gh = argc > 6 ? atoi(argv[6]) : 0; + const char *bar_path = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + Display *dpy; + Window root; + XImage *img = NULL; + XShmSegmentInfo shm; + int use_shm = 1, fd, i, w, h; + unsigned char *bar; + struct frame_hdr *hdr; + volatile uint32_t *seqp; + uint64_t t_first = 0; + uint64_t t_last = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + + dpy = XOpenDisplay(NULL); + if (!dpy) { + printf("XOPEN_FAIL DISPLAY=%s\n", + getenv("DISPLAY") ? getenv("DISPLAY") : "(unset)"); + return 1; + } + root = DefaultRootWindow(dpy); + { + XWindowAttributes wa; + + XGetWindowAttributes(dpy, root, &wa); + w = gw > 0 ? gw : wa.width; + h = gh > 0 ? gh : wa.height; + printf("XSCREEN %dx%d root=%dx%d\n", w, h, wa.width, + wa.height); + } + fd = open(bar_path, O_RDWR | O_SYNC); + if (fd < 0) { + printf("OPEN_FAIL %s\n", bar_path); + return 1; + } + bar = (unsigned char *)mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (bar == MAP_FAILED) { + printf("MMAP_FAIL\n"); + return 1; + } + hdr = (struct frame_hdr *)(bar + FRAME_OFFSET); + seqp = &hdr->seq; + + memset(&shm, 0, sizeof(shm)); + if (use_shm) { + img = XShmCreateImage(dpy, DefaultVisual(dpy, + DefaultScreen(dpy)), + DefaultDepth(dpy, DefaultScreen(dpy)), + ZPixmap, NULL, &shm, w, h); + if (!img) { + printf("XSHM_CREATE_FAIL\n"); + return 1; + } + shm.shmid = shmget(IPC_PRIVATE, + (size_t)img->bytes_per_line * h, + IPC_CREAT | 0600); + if (shm.shmid < 0) { + printf("SHMGET_FAIL\n"); + return 1; + } + shm.shmaddr = img->data = shmat(shm.shmid, NULL, 0); + if (shm.shmaddr == (char *)-1) { + printf("SHMAT_FAIL\n"); + return 1; + } + shm.readOnly = False; + if (!XShmAttach(dpy, &shm)) { + printf("XSHM_ATTACH_FAIL (falling back)\n"); + XDestroyImage(img); + img = NULL; + } else { + XSync(dpy, False); + printf("XSHM OK %dx%d bpl=%d\n", w, h, + img->bytes_per_line); + } + } + if (!img) { + img = XGetImage(dpy, root, gx, gy, w, h, AllPlanes, ZPixmap); + if (!img) { + printf("XGETIMAGE_FAIL\n"); + return 1; + } + printf("XGETIMAGE OK %dx%d bpl=%d\n", w, h, + img->bytes_per_line); + } + + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = (uint32_t)w; + hdr->height = (uint32_t)h; + hdr->stride = (uint32_t)w * 4; + hdr->format = 0; + + for (i = 0; i < frames; i++) { + uint64_t t0 = now_ns(); + unsigned char *dst = bar + FRAME_OFFSET + + sizeof(struct frame_hdr); + int x, y; + + if (shm.shmaddr) { + XShmGetImage(dpy, root, img, gx, gy, AllPlanes); + } else { + XImage *n = XGetImage(dpy, root, gx, gy, w, h, + AllPlanes, ZPixmap); + + if (!n) + break; + XDestroyImage(img); + img = n; + } + /* X ZPixmap: 0x00RRGGBB little-endian -> RGBA bytes */ + for (y = 0; y < h; y++) { + unsigned char *s = (unsigned char *)img->data + + (size_t)y * img->bytes_per_line; + unsigned char *d = dst + (size_t)y * w * 4; + + for (x = 0; x < w; x++) { + uint32_t p; + + memcpy(&p, s + 4 * x, 4); + d[4 * x + 0] = (unsigned char)(p >> 16); + d[4 * x + 1] = (unsigned char)(p >> 8); + d[4 * x + 2] = (unsigned char)(p); + d[4 * x + 3] = 0xff; + } + } + hdr->timestamp_ns = now_ns(); + if (!t_first) + t_first = t0; + t_last = t0; + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + if (i < 5 || (i % 30) == 29) + printf("FRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + if (t_first && t_last > t_first) + printf("XPUB_RATE fps=%.2f frames=%d\n", + (double)(i - 1) * 1e9 / + (double)(t_last - t_first), i); + printf("FRAME_DONE frames=%d %dx%d\n", i, w, h); + if (shm.shmaddr) + XShmDetach(dpy, &shm); + XCloseDisplay(dpy); + return 0; +}
\ No newline at end of file diff --git a/tools/guest-diag/xtest_move.c b/tools/guest-diag/xtest_move.c new file mode 100644 index 0000000..8331487 --- /dev/null +++ b/tools/guest-diag/xtest_move.c @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: BSD-3-Clause +// xtest_move.c - drive the game with XSendEvent-synthetic keys (SDL2 on this +// Xorg ignores XTEST-faked keys; it accepts send_event KeyPress/KeyRelease). +// Finds the game window by name, then holds forward+fire and alternates +// turning left/right so the captured footage has constant motion. +// Usage: xtest_move <display> <seconds> +#include <X11/Xlib.h> +#include <X11/keysym.h> +#include <X11/Xutil.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> + +static Display *d; +static Window game; + +static Window find_named(const char *sub) +{ + Window root = DefaultRootWindow(d), r, parent, *kids = NULL; + unsigned int nk = 0; + + if (!XQueryTree(d, root, &r, &parent, &kids, &nk)) + return 0; + for (unsigned int i = 0; i < nk; i++) { + char *name = NULL; + + if (XFetchName(d, kids[i], &name) && name) { + if (strstr(name, sub)) { + Window w = kids[i]; + + XFree(name); + if (kids) + XFree(kids); + return w; + } + XFree(name); + } + } + if (kids) + XFree(kids); + return 0; +} + +static void send_key(KeySym ks, int down) +{ + XEvent e; + Window root = DefaultRootWindow(d); + int rx, ry, wx, wy; + unsigned int mask; + Window cr, pr; + + if (!game) + return; + memset(&e, 0, sizeof(e)); + e.xkey.type = down ? KeyPress : KeyRelease; + e.xkey.display = d; + e.xkey.window = game; + e.xkey.root = root; + e.xkey.subwindow = None; + e.xkey.time = CurrentTime; + XQueryPointer(d, root, &cr, &pr, &rx, &ry, &wx, &wy, &mask); + e.xkey.x = wx; + e.xkey.y = wy; + e.xkey.x_root = rx; + e.xkey.y_root = ry; + e.xkey.same_screen = True; + e.xkey.keycode = XKeysymToKeycode(d, ks); + e.xkey.state = 0; + XSendEvent(d, game, True, down ? KeyPressMask : KeyReleaseMask, &e); + XFlush(d); +} + +int main(int argc, char **argv) +{ + int secs = argc > 2 ? atoi(argv[2]) : 12; + int t; + + d = XOpenDisplay(argc > 1 ? argv[1] : ":1"); + if (!d) { + fprintf(stderr, "xtest_move: no display\n"); + return 1; + } + game = find_named("Freedoom"); + if (!game) + game = find_named("Outer Prison"); + if (!game) + game = find_named("Doom"); + fprintf(stderr, "xtest_move: game window 0x%lx\n", game); + if (!game) + return 1; + XSetInputFocus(d, game, RevertToParent, CurrentTime); + XFlush(d); + + /* Hold forward only (no fire: fire pulls enemies and the player can + * die mid-capture; with +noclip from the command line this is a smooth + * fly-through). */ + send_key(XK_Up, 1); + + for (t = 0; t < secs * 10; t++) { + static int dir; /* 0 = left held, 1 = right held */ + + XSetInputFocus(d, game, RevertToParent, CurrentTime); /* keep focus */ + /* Re-assert the held forward key: SDL's held state drops between + * synthetic events, so forward motion stalls unless re-sent. */ + if ((t % 4) == 0) + send_key(XK_Up, 1); + if ((t % 9) == 8) { + KeySym off = dir ? XK_Right : XK_Left; + KeySym on = dir ? XK_Left : XK_Right; + + send_key(off, 0); + usleep(30000); + send_key(on, 1); + dir = !dir; + } + XFlush(d); + usleep(100000); + } + send_key(XK_Left, 0); + send_key(XK_Right, 0); + send_key(XK_Up, 0); + XCloseDisplay(d); + fprintf(stderr, "xtest_move: done\n"); + return 0; +} diff --git a/tools/lg/build_video.sh b/tools/lg/build_video.sh new file mode 100755 index 0000000..4f80be8 --- /dev/null +++ b/tools/lg/build_video.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# build_video.sh - build the final open-iov 1.0.0 demo video: a drawtext +# title card, the split-screen main segment (two GZDOOM VMs, one per pane, +# captured through the KVMFR ivshmem path), and an optional end card, +# concatenated into one 1280x720@30 libx264 yuv420p mp4. +# +# Usage: +# build_video.sh <title-text> <main.mp4> [end-text] +# build_video.sh <title-text> <dirA> <dirB> [end-text] +# title-text text for the title card (required, non-empty) +# main.mp4 pre-built split-screen video (e.g. from split_screen.sh) +# dirA dirB two frame-%04d.ppm dirs; split_screen.sh (next to this +# script) is invoked first at FPS to build the main segment +# end-text optional text for the end card (omitted = no end card) +# +# Output: $OUT if set, else ./open-iov-1.0.0-demo.mp4 in the current dir. +# Env knobs: FPS (default 30), CARD_SECS (default 5), FONTSIZE (default 48). +# +# Approach: title/end cards are generated with ffmpeg drawtext (white on +# dark 0x101010, 1280x720); the main segment is normalized to 1280x720@30 +# (aspect-preserving fit + letterbox pad, audio dropped - the demo video is +# video-only); the three segments are concatenated with the concat demuxer +# followed by ONE re-encode. Stream-copy concat is brittle across +# separately-generated mp4s (timestamp/param mismatches); the re-encode is +# cheap at these lengths and guarantees a clean single-stream output. +# The title font is auto-detected (DejaVuSans/NotoSans); the script fails +# with a clear message if no TTF font exists. +# +# Requires: ffmpeg with libx264 + drawtext (libfreetype); split_screen.sh +# next to this script for the two-dirs form. +set -euo pipefail + +usage() { + sed -n '3,31p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "build_video.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" +ffmpeg_has drawtext -filters || die "ffmpeg has no drawtext filter (needs libfreetype)" + +# --- locate a usable TTF font ------------------------------------------- +FONT="" +for f in \ + /usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf \ + /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf \ + /usr/share/fonts/truetype/noto/NotoSans-Regular.ttf \ + /usr/share/fonts/dejavu/DejaVuSans.ttf; do + if [ -f "$f" ]; then FONT=$f; break; fi +done +if [ -z "$FONT" ] && command -v fc-match >/dev/null 2>&1; then + f=$(fc-match -f '%{file}' sans 2>/dev/null || true) + [ -n "$f" ] && [ -f "$f" ] && FONT=$f +fi +[ -n "$FONT" ] || die "no usable TTF font found (looked for DejaVuSans, NotoSans; \ +install fonts-dejavu-core or set up fontconfig)" +echo "build_video.sh: font=$FONT" >&2 + +# --- args ---------------------------------------------------------------- +[ "$#" -ge 1 ] || { usage; exit 2; } +TITLE=$1; shift +[ -n "$TITLE" ] || die "title text must be non-empty" +MAIN="" DIR_A="" DIR_B="" END="" +if [ "$#" -ge 1 ] && [ -d "$1" ]; then + DIR_A=$1 + [ "$#" -ge 2 ] && [ -d "$2" ] || die "two-dirs form needs two directories: <dirA> <dirB> [end-text]" + DIR_B=$2 + END=${3:-} +elif [ "$#" -ge 1 ]; then + [ -f "$1" ] || die "main input not found: $1" + MAIN=$1 + END=${2:-} +else + die "missing main input (mp4 file or two frame dirs)" +fi + +FPS=${FPS:-30} +CARD_SECS=${CARD_SECS:-5} +FONTSIZE=${FONTSIZE:-48} +OUT=${OUT:-$PWD/open-iov-1.0.0-demo.mp4} +HERE=$(cd "$(dirname "$0")" && pwd) + +TMP=$(mktemp -d "${TMPDIR:-/tmp}/build_video.XXXXXX") +trap 'rm -rf "$TMP"' EXIT + +# --- cards (drawtext; text via textfile so ':' ',' '%' etc. are literal) -- +make_card() { + local text=$1 out=$2 tf="$TMP/card-$$.txt" + printf '%s' "$text" > "$tf" + ffmpeg -hide_banner -loglevel warning -y \ + -f lavfi -i "color=c=0x101010:s=1280x720:r=$FPS:d=$CARD_SECS" \ + -vf "drawtext=fontfile=$FONT:textfile=$tf:fontcolor=white:fontsize=$FONTSIZE:expansion=none:x=(w-text_w)/2:y=(h-text_h)/2" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$out" +} + +echo "build_video.sh: title card (${CARD_SECS}s)..." >&2 +make_card "$TITLE" "$TMP/title.mp4" + +# --- main segment -------------------------------------------------------- +if [ -n "$DIR_A" ]; then + [ -x "$HERE/split_screen.sh" ] || die "split_screen.sh not found/executable next to build_video.sh" + echo "build_video.sh: building split-screen main from $DIR_A + $DIR_B..." >&2 + "$HERE/split_screen.sh" "$DIR_A" "$DIR_B" "$TMP/split.mp4" "$FPS" + MAIN=$TMP/split.mp4 +fi +echo "build_video.sh: normalizing main segment to 1280x720@$FPS..." >&2 +ffmpeg -hide_banner -loglevel warning -y -i "$MAIN" \ + -vf "scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=$FPS" \ + -an -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + "$TMP/main.mp4" + +# --- concat [title, main, end] ------------------------------------------ +printf "file '%s'\n" "$TMP/title.mp4" "$TMP/main.mp4" > "$TMP/list.txt" +if [ -n "$END" ]; then + echo "build_video.sh: end card (${CARD_SECS}s)..." >&2 + make_card "$END" "$TMP/end.mp4" + printf "file '%s'\n" "$TMP/end.mp4" >> "$TMP/list.txt" +fi + +echo "build_video.sh: concatenating..." >&2 +ffmpeg -hide_banner -loglevel warning -y -f concat -safe 0 -i "$TMP/list.txt" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -select_streams v:0 \ + -show_entries stream=width,height,r_frame_rate,duration \ + -show_entries format=duration,size \ + -of default=nw=1 "$OUT" | sed 's/^/build_video.sh: /' >&2 +fi +echo "VIDEO_DONE $OUT" >&2 diff --git a/tools/lg/frame_reader.py b/tools/lg/frame_reader.py index f8bab40..783fcaf 100644 --- a/tools/lg/frame_reader.py +++ b/tools/lg/frame_reader.py @@ -46,7 +46,11 @@ def main(): deadline = time.time() + args.timeout saved = 0 - last_seq = 0 + mm.seek(FRAME_OFFSET) + magic, _v, _w, _h, _s, _f, last_seq, _r, _t = HDR.unpack(mm.read(HDR.size)) + if magic != MAGIC: + last_seq = 0 + print("SEED last_seq=%d" % last_seq) while saved < args.frames and time.time() < deadline: mm.seek(FRAME_OFFSET) magic, ver, w, h, stride, fmt, seq, _res, ts = HDR.unpack( diff --git a/tools/lg/split_screen.sh b/tools/lg/split_screen.sh new file mode 100755 index 0000000..55e11ee --- /dev/null +++ b/tools/lg/split_screen.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# SPDX-License-Identifier: BSD-3-Clause +# split_screen.sh - compose a side-by-side (split-screen) mp4 from two PPM +# frame sequences captured through the KVMFR ivshmem path (guest publishers: +# tools/cuda-ab/frame_producer.cu or tools/guest-diag/xshm_pub.c; host reader: +# tools/lg/frame_reader.py, which writes frame-%04d.ppm into a directory). +# +# Usage: split_screen.sh <dirA> <dirB> <out.mp4> [fps] [left_w] [left_h] +# dirA left-side frame-%04d.ppm directory (any starting index) +# dirB right-side frame-%04d.ppm directory +# out.mp4 output file (libx264, yuv420p) +# fps output frame rate (default 30; pass the measured capture rate, +# e.g. 10, for real-time playback of ~10 fps captures) +# left_w optional left-pane width; the left pane is fitted into +# left_w x H with letterbox padding (aspect preserved) +# left_h optional pane height H; BOTH panes are scaled to height H +# (default: native height of dirA's first frame). Widths follow +# each input's aspect ratio (forced even for yuv420p). +# +# Unequal sequence lengths: the shorter side is padded by repeating its last +# frame. hstack is a framesync filter whose repeatlast=1 default holds the +# shorter input's last frame until the longer input ends; shortest=0 is +# passed explicitly to make that the contract (verified on ffmpeg 6.1.1 for +# both input orders). No symlink/tpad workaround is needed. +# +# Fails loudly (exit 1) on: missing ffmpeg/libx264, missing input dirs, or an +# input dir with no frame-*.ppm files. Requires: ffmpeg (libx264); ffprobe is +# used for the final summary when present. +set -euo pipefail + +usage() { + sed -n '3,28p' "$0" | sed 's/^# \{0,1\}//' +} + +die() { echo "split_screen.sh: ERROR: $*" >&2; exit 1; } + +# grep -q would SIGPIPE the ffmpeg producer under pipefail; capture first. +ffmpeg_has() { # $1 = encoder/filter token + local out + out=$(ffmpeg -hide_banner "$2" 2>/dev/null) || true + case "$out" in *" $1 "*) return 0 ;; *) return 1 ;; esac +} + +# Read width/height from the PPM P6 header (line 2: "<w> <h>"). +ppm_dims() { + local line w h + line=$(sed -n '2p' "$1") + read -r w h _ <<< "$line" + [ -n "${w:-}" ] && [ -n "${h:-}" ] || return 1 + echo "$w $h" +} + +# Scan a frame dir: sets PD_COUNT PD_FIRST PD_LAST PD_W PD_H PD_FILE +probe_dir() { + local dir="$1" label="$2" files f dim + [ -d "$dir" ] || die "$label directory not found: $dir" + mapfile -t files < <(find "$dir" -maxdepth 1 -name 'frame-*.ppm' -type f \ + -printf '%f\n' | sort) + PD_COUNT=${#files[@]} + [ "$PD_COUNT" -gt 0 ] || die "$label directory has no frame-*.ppm files: $dir" + f=${files[0]} + PD_FIRST=${f#frame-}; PD_FIRST=${PD_FIRST%.ppm}; PD_FIRST=$((10#$PD_FIRST)) + f=${files[$((PD_COUNT - 1))]} + PD_LAST=${f#frame-}; PD_LAST=${PD_LAST%.ppm}; PD_LAST=$((10#$PD_LAST)) + if [ $((PD_LAST - PD_FIRST + 1)) -ne "$PD_COUNT" ]; then + echo "split_screen.sh: WARN: $label sequence has gaps ($PD_COUNT files," \ + "indices $PD_FIRST..$PD_LAST); ffmpeg stops at the first gap" >&2 + fi + PD_FILE=$(printf '%s/frame-%04d.ppm' "$dir" "$PD_FIRST") + dim=$(ppm_dims "$PD_FILE") || die "$label: cannot read PPM header of $PD_FILE" + PD_W=${dim% *}; PD_H=${dim#* } +} + +[ "$#" -ge 3 ] || { usage; exit 2; } +DIR_A=$1 DIR_B=$2 OUT=$3 FPS=${4:-30} LW=${5:-} LH=${6:-} + +command -v ffmpeg >/dev/null 2>&1 || die "ffmpeg not found in PATH" +ffmpeg_has libx264 -encoders || die "ffmpeg has no libx264 encoder" + +probe_dir "$DIR_A" left; A_COUNT=$PD_COUNT A_FIRST=$PD_FIRST A_H=$PD_H +probe_dir "$DIR_B" right; B_COUNT=$PD_COUNT B_FIRST=$PD_FIRST + +# Pane height H: left_h if given, else dirA's native height. Keep even. +H=${LH:-$A_H} +if [ $((H % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: pane height $H rounded down to even for yuv420p" >&2 + H=$((H / 2 * 2)) +fi +[ "$H" -ge 2 ] || die "pane height must be >= 2" + +# Left pane: fit into left_w x H (letterbox) when left_w given, else scale +# to height H with aspect preserved. Right pane: always scale to height H. +if [ -n "$LW" ]; then + if [ $((LW % 2)) -ne 0 ]; then + echo "split_screen.sh: WARN: left_w $LW rounded down to even for yuv420p" >&2 + LW=$((LW / 2 * 2)) + fi + [ "$LW" -ge 2 ] || die "left_w must be >= 2" + LEFT="[0:v]scale=${LW}:${H}:force_original_aspect_ratio=decrease,"\ +"pad=${LW}:${H}:(ow-iw)/2:(oh-ih)/2,setsar=1[l]" +else + LEFT="[0:v]scale=-2:${H},setsar=1[l]" +fi +RIGHT="[1:v]scale=-2:${H},setsar=1[r]" +# shortest=0 + framesync repeatlast=1 (default): shorter side's last frame is +# repeated until the longer side ends (see header comment). +FILTER="${LEFT};${RIGHT};[l][r]hstack=shortest=0" + +echo "split_screen.sh: left=$DIR_A ($A_COUNT frames from $A_FIRST)" \ + "right=$DIR_B ($B_COUNT frames from $B_FIRST) pane_h=$H fps=$FPS" >&2 + +ffmpeg -hide_banner -loglevel warning -y \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$A_FIRST" -i "$DIR_A/frame-%04d.ppm" \ + -thread_queue_size 512 -framerate "$FPS" -start_number "$B_FIRST" -i "$DIR_B/frame-%04d.ppm" \ + -filter_complex "$FILTER" \ + -c:v libx264 -preset medium -crf 20 -pix_fmt yuv420p \ + -r "$FPS" -movflags +faststart \ + "$OUT" + +[ -s "$OUT" ] || die "ffmpeg produced no output: $OUT" +if command -v ffprobe >/dev/null 2>&1; then + ffprobe -v error -count_frames -select_streams v:0 \ + -show_entries stream=width,height,nb_read_frames \ + -of default=nw=1 "$OUT" | sed 's/^/split_screen.sh: /' >&2 +fi +echo "SPLIT_DONE $OUT" >&2 diff --git a/tools/mme-audit/mme_audit.py b/tools/mme-audit/mme_audit.py new file mode 100644 index 0000000..8031f3c --- /dev/null +++ b/tools/mme-audit/mme_audit.py @@ -0,0 +1,570 @@ +#!/usr/bin/env python3 +"""mme-audit: offline audit of the vgsp MME golden overlay and replay streams. + +Pure Python 3.10 stdlib. Inputs (all local, all public): + * kernel/golden-mme-tu104.h (golden slot library) + * dmesg_live_*.log captures (3D win / MME-WINRUN / MME-PRE) + +What it checks (r1448-strategy 3.1): + 1. overlay simulation: golden_slot() accept rules, old (!gs->entry) vs new + (entry 0 valid) under force=1 -> 54 before, 55 after + 2. grammar walk of every 3D win[...] / MME-WINRUN / MME-PRE[...] block: + op 1/3/5 span, 0x114 IR loads, SA pairs; walk-end == len assertion + 3. call census per window (slot, off, payload) + 4. operand map over the mission slots: decode MME64 groups, flag + LOAD-sourced method emissions and suspect method immediates + 5. cross-boot pair-emitted/elided/MISS/first-fault table + 6. slot-57 focus decode around the r1448b/c stall (IR 868..930) +""" +import argparse +import os +import re +import sys + +# --------------------------------------------------------------------- golden + +GOLDEN_SLOT_RE = re.compile( + r"\[\s*(\d+)\]\s*=\s*\{\s*0x([0-9a-fA-F]+)u,\s*(\d+)u,\s*(\d+)u,\s*0x([0-9a-fA-F]+)u\s*\}") +GOLDEN_WORDS_RE = re.compile( + r"golden_mme_tu104_words\[GOLDEN_MME_TU104_WORDS\]\s*=\s*\{(.*?)\n\};", re.S) + +MISSION_SLOTS = [0, 2, 34, 39, 40, 48, 51, 57, 58] +SUSPECT_METHODS = [0x1614, 0x1618, 0x2054, 0x2058, 0x2094, 0x2098, + 0x2154, 0x2158, 0x0d74, 0x0d78] + +# MME64 layout (3-word groups), extracted from the observed disassembly +# convention: bits[95:64]=w0, [63:32]=w1, [31:0]=w2. +REG = ["R%d" % i for i in range(24)] + ["ZERO", "IMMED", "IMMEDPAIR", + "IMMED32", "LOAD0", "LOAD1"] +OP = ["ADD", "ADDC", "SUB", "SUBB", "MUL", "MULH", "MULU", "EXTENDED", "CLZ", + "SLL", "SRL", "SRA", "AND", "NAND", "OR", "XOR", "MERGE", "SLT", "SLTU", + "SLE", "SLEU", "SEQ", "STATE", "LOOP", "JAL", "BLT", "BLTU", "BLE", + "BLEU", "BEQ", "DREAD", "DWRITE"] +OUT = ["NONE", "ALU0", "ALU1", "LOAD0", "LOAD1", "IMMED0", "IMMED1", + "RESERVED", "IMMEDHIGH0", "IMMEDHIGH1", "IMMED32_0"] + + +def regname(v): + return REG[v] if v < 30 else "V%d" % (v - 32) + + +def mme64_decode(w0, w1, w2): + val = (w0 << 64) | (w1 << 32) | w2 + + def bits(hi, lo): + return (val >> lo) & ((1 << (hi - lo + 1)) - 1) + + alus = [] + for base in (10, 46): + alus.append((bits(base + 4, base), bits(base + 9, base + 5), + bits(base + 14, base + 10), bits(base + 19, base + 15), + bits(base + 35, base + 20))) + outs = [] + for base in (82, 89): + outs.append((bits(base + 2, base), bits(base + 5, base + 3))) + return { + "endNext": bits(0, 0), "predMode": bits(4, 1), "pred": bits(9, 5), + "alu": alus, "out": outs, + } + + +def mme64_text(d): + parts = [] + for op, dst, s0, s1, im in d["alu"]: + if (op, dst, s0, s1, im) == (0, 0, 0, 0, 0): + continue + parts.append("%s %s, %s, %s, 0x%04x" % + (OP[op], regname(dst), regname(s0), regname(s1), im)) + for meth, emit in d["out"]: + if meth or emit: + parts.append("OUT method-src=%s data-src=%s" % + (OUT[meth], OUT[emit])) + if d["endNext"]: + parts.append("endNext") + if d["predMode"]: + parts.append("pred=%s,%s" % (d["predMode"], regname(d["pred"]))) + return " | ".join(parts) if parts else "NOP" + + +def parse_golden(path): + text = open(path, encoding="utf-8").read() + pool = [] + m = GOLDEN_WORDS_RE.search(text) + if m: + pool = [int(t, 16) for t in re.findall(r"0x[0-9a-fA-F]+", m.group(1))] + entries = {} + for m in GOLDEN_SLOT_RE.finditer(text): + entries[int(m.group(1))] = (int(m.group(2), 16), int(m.group(3)), + int(m.group(4)), int(m.group(5), 16)) + return entries, pool + + +def golden_slot(entries, j, entry0_fix): + """Model of gsp_phys_gr_golden_slot() (knob presumed on).""" + if j >= 64 or j not in entries: + return None + entry, words, off, flags = entries[j] + if not (flags & 0x1) or not words: + return None + if not entry0_fix and not entry: + return None + return entry, words, off, flags + + +PRESENT = 0x1 +GMME_FLAGS = {0x2: "DRIVER", 0x4: "LIVE_VARIANT", 0x8: "GEFORCE"} + + +def overlay_sim(entries, pool): + out = {} + for fix in (False, True): + overlaid, rejected = [], [] + for j in range(64): + gs = golden_slot(entries, j, fix) + (overlaid if gs else rejected).append(j) + out[fix] = (sorted(overlaid), sorted(rejected)) + return out + + +# ------------------------------------------------------------------ walkers + +def walk_stream(words): + """Booter grammar walk: returns (ok, runs, image, calls, sa_pairs).""" + runs, image, calls, sa_pairs = [], {}, [], {} + off, n = 0, len(words) + ok = True + while off < n: + h = words[off] + op = (h >> 29) & 7 + cnt = (h >> 16) & 0xfff + mthd = (h & 0x1fff) << 2 + span = 1 + if op in (1, 3, 5) and cnt: + span = 1 + cnt + if off + span > n: + ok = False + span = n - off + runs.append((off, op, cnt, mthd, span)) + if mthd == 0x114 and span >= 3: + pc = words[off + 1] + code = words[off + 2:off + span] + for i in range(len(code) // 3): + image[pc + i] = tuple(code[3 * i:3 * i + 3]) + if 0x3800 <= mthd <= 0x39fc: + j = (mthd - 0x3800) // 8 + kind = "DATA" if (mthd & 4) else "MACRO" + payload = list(words[off + 1:off + span]) + calls.append((j, off, kind, mthd, payload)) + if mthd == 0x11c and span >= 3: + sa_pairs[words[off + 1]] = words[off + 2] + off += span + return ok, runs, image, calls, sa_pairs + + +CRLF = re.compile(r"\[(\d+\.\d+)\]\s+vgsp-gr:\s+(.*)$") +WIN_RE = re.compile(r"3D win\[\s*(\d+)\]:\s+((?:[0-9a-fA-F]+ ?)+)") +WINRUN_DUMP_RE = re.compile(r"MME-WINRUN dump h=0x([0-9a-fA-F]+) push=0x([0-9a-fA-F]+) window=(\d+)") +WINRUN_RUN_RE = re.compile( + r"MME-WINRUN off=([0-9a-fA-F]+) op=(\d+) cnt=(\d+) mthd=([0-9a-fA-F]+) d0=([0-9a-fA-F]+) d1=([0-9a-fA-F]+)") +PRE_RE = re.compile(r"MME-PRE\[\s*(\d+)\]:?\s+((?:[0-9a-fA-F]{8} ?)+)") + + +def parse_windows(path): + """Yield dicts for each 3D win[], MME-WINRUN and MME-PRE[] block.""" + cur = None + win_words, wr_runs, wr_meta = [], [], None + pre_idx, pre_words = -1, [] + with open(path, encoding="utf-8", errors="replace") as fh: + for line in fh: + m = CRLF.search(line) + if not m: + continue + ts, msg = m.group(1), m.group(2) + m = WIN_RE.search(msg) + if m: + off = int(m.group(1)) + ws = [int(t, 16) for t in m.group(2).split()] + if win_words and off != len(win_words) and off != 0: + yield {"kind": "win", "ts": ts, "words": win_words} + win_words = [] + if off == 0 and win_words: + yield {"kind": "win", "ts": ts, "words": win_words} + win_words = [] + win_words.extend(ws) + continue + m = WINRUN_DUMP_RE.search(msg) + if m: + if wr_meta and wr_runs: + yield {"kind": "winrun", "ts": ts, **wr_meta, "runs": wr_runs} + wr_meta = {"h": int(m.group(1), 16), "push": int(m.group(2), 16), + "window": int(m.group(3))} + wr_runs = [] + continue + m = WINRUN_RUN_RE.search(msg) + if m and wr_meta is not None: + wr_runs.append((int(m.group(1), 16), int(m.group(2)), + int(m.group(3)), int(m.group(4), 16), + int(m.group(5), 16), int(m.group(6), 16))) + continue + m = PRE_RE.search(msg) + if m: + idx = int(m.group(1)) + ws = [int(t, 16) for t in m.group(2).split()] + if pre_words and idx != pre_idx + len(pre_words): + yield {"kind": "pre", "ts": ts, "words": pre_words} + pre_words = [] + pre_words.extend(ws) + pre_idx = idx + continue + if win_words: + yield {"kind": "win", "ts": ts, "words": win_words} + win_words = [] + if wr_meta and wr_runs is not None: + # runs may legitimately be empty (dump with no decoded runs) + yield {"kind": "winrun", "ts": ts, **wr_meta, "runs": wr_runs} + wr_meta, wr_runs = None, [] + if pre_words: + yield {"kind": "pre", "ts": ts, "words": pre_words} + pre_words = [] + if win_words: + yield {"kind": "win", "ts": ts, "words": win_words} + if wr_meta: + yield {"kind": "winrun", "ts": ts, **wr_meta, "runs": wr_runs} + if pre_words: + yield {"kind": "pre", "ts": ts, "words": pre_words} + + +def walk_winrun(window, runs): + """Walk a decoded MME-WINRUN listing; returns (ok, calls, end).""" + seen, calls, end = set(), [], 0 + for off, op, cnt, mthd, d0, d1 in runs: + span = 1 + cnt if op in (1, 3, 5) and cnt else 1 + seen.add(off) + if 0x3800 <= mthd <= 0x39fc: + j = (mthd - 0x3800) // 8 + kind = "DATA" if (mthd & 4) else "MACRO" + calls.append((j, off, kind, mthd, [d0, d1])) + end = max(end, off + span) + ok = end == window and len(seen) == len(runs) + return ok, calls, end + + +# ------------------------------------------------------------ MME64 scanning + +def scan_slot(entries, pool, slot): + entry, words, off, flags = entries[slot] + rows = [] + for i in range(words // 3): + w = tuple(pool[off + 3 * i:off + 3 * i + 3]) + d = mme64_decode(*w) + rows.append((entry + i, i, w, d)) + return rows + + +def operand_map(entries, pool): + """For mission slots: flag emissions + suspect method constants.""" + sus2 = {m >> 2: m for m in SUSPECT_METHODS} + out = [] + for slot in MISSION_SLOTS: + if slot not in entries or not entries[slot][1]: + continue + for ir, idx, w, d in scan_slot(entries, pool, slot): + tags = [] + for meth, emit in d["out"]: + if emit in (3, 4): # LOAD0/LOAD1 data source + tags.append("LOAD-sourced emission (method-src=%s)" % OUT[meth]) + for word in w: + low = word & 0xfff + if low in sus2 and sus2[low] != 0x0d74 and sus2[low] != 0x0d78: + tags.append("suspect method %#x (field 11:0 of %08x)" % (sus2[low], word)) + for op, dst, s0, s1, im in d["alu"]: + if im in SUSPECT_METHODS: + tags.append("suspect imm %#x (ALU immed)" % im) + if s0 == 24 and im == 0: # ZERO + 0 immediate + tags.append("zero-operand ALU (src0=ZERO immed=0)") + for t in tags: + out.append((slot, ir, w, t, mme64_text(d))) + return out + + +# ----------------------------------------------------------- log-level facts + +FAULT_RE = re.compile(r'errString="([^"]+)"') +ESR_RE = re.compile(r"esr 80000000->badf1201") + + +def log_facts(path): + f = {"path": path} + counts = {"PAIR-DROP": 0, "PAIR-DROP-words": 0, "MISS-2054": 0, + "GOLDEN": [], "esr": 0, "macros": {}, "faults": [], + "trap2090": 0} + with open(path, encoding="utf-8", errors="replace") as fh: + for line in fh: + if "MME-PAIR-DROP" in line: + counts["PAIR-DROP"] += 1 + m = re.search(r"MME-PAIR-DROP (\d+) word", line) + if m: + counts["PAIR-DROP-words"] += int(m.group(1)) + if "pair addr=0x2054" in line and "-> MISS" in line: + counts["MISS-2054"] += 1 + m = re.search(r"MME-GOLDEN overlay (\d+) slot", line) + if m: + counts["GOLDEN"].append(int(m.group(1))) + if ESR_RE.search(line): + counts["esr"] += 1 + for m in re.finditer(r"macro-status=([0-9a-fA-F]{8})\(mpc=(\d+) mac=0x([0-9a-fA-F]+) run=(\d)\)", line): + counts["macros"][int(m.group(2))] = counts["macros"].get(int(m.group(2)), 0) + 1 + if "trap=00002090" in line or "trap=80002090" in line: + counts["trap2090"] += 1 + m = FAULT_RE.search(line) + if m: + counts["faults"].append(m.group(1).strip()) + f["counts"] = counts + return f + + +# ------------------------------------------------------------------- report + +def main(): + ap = argparse.ArgumentParser() + here = os.path.dirname(os.path.abspath(__file__)) + repo = os.path.abspath(os.path.join(here, "..", "..")) + ap.add_argument("--golden", default=os.path.join(repo, "kernel", "golden-mme-tu104.h")) + ap.add_argument("--logs", nargs="*", default=[]) + ap.add_argument("--out", default="/tmp/opencode/r1449-audit.md") + args = ap.parse_args() + + entries, pool = parse_golden(args.golden) + present = sorted(j for j in entries if entries[j][3] & PRESENT and entries[j][1]) + absent = sorted(set(range(64)) - set(present)) + lines = [] + A = lines.append + + A("# r1449 MME offline audit (mme-audit)") + A("") + A("- golden: `%s` (%d words, %d present slots)" % (args.golden, len(pool), len(present))) + A("- present: %s" % present) + A("- absent : %s" % absent) + A("") + + A("## 1. Overlay simulation (gsp_phys_gr_golden_slot + force overlay)") + sim = overlay_sim(entries, pool) + old_over, old_rej = sim[False] + new_over, new_rej = sim[True] + A("") + A("| mode | overlaid | rejected (no golden) |") + A("|---|---|---|") + A("| pre-r1448 (`!gs->entry` rejects slot 0) | **%d** | %s |" % (len(old_over), old_rej)) + A("| r1448 (entry 0 valid) | **%d** | %s |" % (len(new_over), new_rej)) + A("") + A("slot 0: %s" % ("golden entry=0x0 words=%d flags=%s (ACCEPTED after fix)" % + (entries[0][1], "+".join(n for b, n in GMME_FLAGS.items() if entries[0][3] & b)) + if 0 in entries else "absent")) + assert len(old_over) == 54 and len(new_over) == 55, "overlay sim mismatch" + A("") + A("ASSERT: 54 -> 55 after the entry-0 fix: OK") + A("") + + A("## 2. Grammar walks (walk-end == len)") + A("") + A("| log | blocks | win | winrun | pre | end-mismatch | calls | 0x2054 | 0x2094 | 0x2154 |") + A("|---|---|---|---|---|---|---|---|---|---|") + wins = {} + for path in args.logs: + if not os.path.exists(path): + A("| %s | MISSING | | | | | | | | |" % path) + continue + nw = nwr = npre = nmis = ncall = 0 + pair_counts = {0x2054: 0, 0x2094: 0, 0x2154: 0} + win_calls = [] + for blk in parse_windows(path): + if blk["kind"] == "win": + ok, runs, image, calls, sa = walk_stream(blk["words"]) + nw += 1 + nmis += 0 if ok else 1 + ncall += len(calls) + for j, off, kind, mthd, payload in calls: + win_calls.append((blk["ts"], j, off, kind, payload)) + elif blk["kind"] == "winrun": + ok, calls, end = walk_winrun(blk["window"], blk["runs"]) + nwr += 1 + nmis += 0 if ok else 1 + ncall += len(calls) + for off, op, cnt, mthd, d0, d1 in blk["runs"]: + if mthd in pair_counts: + pair_counts[mthd] += 1 + for j, off, kind, mthd, payload in calls: + win_calls.append((blk["ts"], j, off, kind, payload)) + elif blk["kind"] == "pre": + ok, runs, image, calls, sa = walk_stream(blk["words"]) + npre += 1 + nmis += 0 if ok else 1 + ncall += len(calls) + wins[path] = win_calls + base = os.path.basename(path) + A("| %s | %d | %d | %d | %d | %d | %d | %d | %d | %d |" % + (base, nw + nwr + npre, nw, nwr, npre, nmis, ncall, + pair_counts[0x2054], pair_counts[0x2094], pair_counts[0x2154])) + A("") + A("Note: dmesg_live_r1448.log carries both r1448b and r1448c boots " + "(148098..148521 and 148520/148627 sentinels); the per-boot logs named " + "r1448b/r1448c were not present at audit time.") + A("") + + A("## 3. Call census (windows that called MME macros)") + for path, calls in wins.items(): + if not calls: + continue + A("") + A("### %s" % os.path.basename(path)) + A("") + A("| ts | slot | off | kind | payload |") + A("|---|---|---|---|---|") + for ts, j, off, kind, payload in calls[:200]: + A("| %s | %d | 0x%x | %s | %s |" % + (ts, j, off, kind, " ".join("%08x" % p for p in payload[:8]))) + if len(calls) > 200: + A("| ... | (%d more) | | | |" % (len(calls) - 200)) + A("") + + A("## 4. Operand map (mission slots; LOAD-sourced emissions + suspects)") + A("") + om = operand_map(entries, pool) + if not om: + A("no flagged instructions") + else: + A("| slot | IR | words | flag | decode |") + A("|---|---|---|---|---|") + for slot, ir, w, t, txt in om[:120]: + A("| %d | %d | %08x %08x %08x | %s | %s |" % (slot, ir, w[0], w[1], w[2], t, txt)) + if len(om) > 120: + A("| ... | | | (%d more) | |" % (len(om) - 120)) + A("") + + A("## 5. Cross-boot pair/fault table") + A("") + A("| log | PAIR-DROP n | words | 0x2054 MISS | overlay | esr badf1201 | trap 0x2090 | first faults |") + A("|---|---|---|---|---|---|---|---|") + for path in args.logs: + if not os.path.exists(path): + continue + f = log_facts(path) + c = f["counts"] + A("| %s | %d | %d | %d | %s | %d | %d | %s |" % + (os.path.basename(path), c["PAIR-DROP"], c["PAIR-DROP-words"], + c["MISS-2054"], sorted(set(c["GOLDEN"])), c["esr"], c["trap2090"], + "; ".join(c["faults"][:4]) if c["faults"] else "-")) + A("") + + A("## 6. Slot-57 focus: golden body around the r1448b/c stall") + A("") + if 57 in entries and entries[57][1]: + rows = scan_slot(entries, pool, 57) + A("entry=0x%x words=%d instructions=%d" % (entries[57][0], entries[57][1], len(rows))) + A("") + A("| IR | idx | words | decode |") + A("|---|---|---|---|") + for ir, idx, w, d in rows: + mark = "" + if ir == 899: + mark = " **<- r1448b stall (mpc=899)**" + elif ir == 900: + mark = " **<- r1448c stall (mpc=900)**" + if idx < 14 or idx >= 20: + A("| %d | %d | %08x %08x %08x | %s%s |" % (ir, idx, w[0], w[1], w[2], mme64_text(d), mark)) + A("") + else: + A("slot 57 absent from golden library") + + A("## 7. Method-name map for the pipeline family (0x2000 + j*64)") + A("") + names = {0x00: "SET_PIPELINE", 0x08: "PIPELINE_RESERVED_A", + 0x0c: "PIPELINE_REGISTER_COUNT", 0x10: "PIPELINE_BINDING", + 0x14: "PROGRAM_ADDRESS_A", 0x18: "PROGRAM_ADDRESS_B", + 0x1c: "PIPELINE_RESERVED_D", 0x20: "PIPELINE_RESERVED_E"} + for m in (0x2054, 0x2090, 0x2094, 0x2098, 0x2154, 0x2158): + j = (m - 0x2000) // 64 + off = (m - 0x2000) % 64 + A("- 0x%04x = SET_PIPELINE(%d) + 0x%02x = %s" % (m, j, off, names.get(off, "?"))) + A("") + + A("## 8. Stall-diagnosis evidence (r1448b/c residual)") + A("") + A("The MME-FAST `trap=<mthd> data=<d>` telemetry carries the method the FE") + A("last processed. Calibration against the window runs:") + A("") + TRAP_RE = re.compile(r"trap=([0-9a-fA-F]{8}) data=([0-9a-fA-F]{8})") + for path in args.logs: + if not os.path.exists(path): + continue + wr_pairs, trap_pairs, stall = set(), set(), [] + with open(path, encoding="utf-8", errors="replace") as fh: + for line in fh: + m = WINRUN_RUN_RE.search(line) + if m: + wr_pairs.add((int(m.group(4), 16), int(m.group(5), 16))) + m = TRAP_RE.search(line) + if m: + trap_pairs.add((int(m.group(1), 16) & 0x7fffffff, + int(m.group(2), 16))) + if "macro-status=" in line and ("mpc=899" in line or "mpc=900" in line): + stall.append(line.rstrip()) + hits = sorted(wr_pairs & trap_pairs) + n2090_wr = sum(1 for line in open(path, encoding="utf-8", errors="replace") + if WINRUN_RUN_RE.search(line) and "mthd=2090" in line) + n2090_trap = sum(1 for line in open(path, encoding="utf-8", errors="replace") + if "trap=00002090" in line or "trap=80002090" in line) + A("### %s" % os.path.basename(path)) + A("") + A("- window-run `(mthd,d0)` vs telemetry `(trap,data)` intersections: %s" % + (" ".join("0x%x/0x%x" % p for p in hits[:8]) if hits else "none")) + A("- `mthd=2090` window runs: **%d**; `trap=...2090` telemetry lines: **%d**" % + (n2090_wr, n2090_trap)) + for s in stall[:6]: + A("- stall: `%s`" % s) + A("") + + A("## 9. Findings (synthesis)") + A("") + A("1. **Overlay completion:** the entry-0 fix takes the force overlay from") + A(" 54 to 55 slots (slot 0 SETUP_FIXED_CONST, entry 0x0, 15 words is the") + A(" only slot the old `!gs->entry` reject skipped).") + A("2. **r1448b/c residual:** the slots carried the program pairs (PAIR-DROP=0;") + A(" `mthd=2054/2094/2154 d1=0x19b0900/0x19b0200/0x19b0000`); the MME was") + A(" sampled at mpc=899 (r1448b, 3x then poison) / 900 (r1448c, 2x then") + A(" trap) inside slot 57. NOTE: the clean r1443b boot passes through the") + A(" same mpc=899/900 samples (and advances to mac=0x33), so the PC alone") + A(" is not the anomaly - the differentiator is `trap=0x2090 data=0`") + A(" (0 occurrences in r1443b/r1447/r1447c, 4 in r1448) followed by") + A(" Class Error Offset 00002090 ErrorCode 4.") + A("3. **0x2090 is macro-emitted, not client-emitted:** no window in any log") + A(" contains `mthd=2090`; the trap field is calibrated against window runs") + A(" (`mthd=22c d0=800` <-> `trap=0000022c data=00000800`). 0x2090 =") + A(" SET_PIPELINE(2)+0x10 = PIPELINE_BINDING(2), the stage-2 binding in the") + A(" same family as the carried pairs. It is emitted by the MME and") + A(" rejected only in the pair-carried boots.") + A("4. **Stall-site decode:** slot 57 IR 899 (idx 31) is an emission group") + A(" `ADD R3,R3,IMMEDPAIR,0x08e4 | OUT method-src=IMMED0 data-src=LOAD0`;") + A(" IR 903 repeats it; IR 900 (idx 32) is `JAL +0x1d` to the macro tail.") + A(" LOAD0 is never written inside the slot-57 body (first read at IR 868") + A(" as macro input word 0): the emitted binding payload is call-input") + A(" dependent and null-capable - the rejected `data=0` snapshot is") + A(" consistent with a zero call-input word on the stalling call.") + A("5. **Next lever:** the carried-pair launch must satisfy the stage-2") + A(" binding method (0x2090) the macro emits, or the emission must be") + A(" bracketed by a valid stage-2 selector/binding state (the window writes") + A(" `mthd=2080 SET_PIPELINE(2) d0=0x21 d1=0x20010850` but no 0x2090).") + A(" Alternative: keep the r1447 pairs-elided shape for the calls-kept") + A(" launch (it reached mpc=930/22) and carry the pairs only on held-call") + A(" launches.") + A("") + + text = "\n".join(lines) + "\n" + with open(args.out, "w", encoding="utf-8") as fh: + fh.write(text) + print(text) + print("wrote", args.out, file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/tools/site/pages/Documentation.wiki b/tools/site/pages/Documentation.wiki index 79f2cec..9d3a9dc 100644 --- a/tools/site/pages/Documentation.wiki +++ b/tools/site/pages/Documentation.wiki @@ -3,10 +3,12 @@ This is the index of the Open-IOV wiki. If you are new, read '''[[Project Status]]''' first, then '''[[System Overview]]'''. The wiki is a curated public view of the project's maintained documentation; each page names its canonical document in the [https://git.vgpu.io/open-iov.git/ repository]. == Start here == +* [[Project Goals]] — what Open-IOV is for, and the status of each goal +* [[Project Plan]] — the phases, gates, workstreams and what is next * [[Project Status]] — what works today, what is in progress, what is not started * [[Install and First Run]] — build, load, and a stock guest driver in about half an hour * [[System Overview]] — the architecture in one page -* [[FAQ]] — common questions, including "why is nvidia-smi working but CUDA not?" +* [[FAQ]] — common questions, including the current state of CUDA and 3D == Architecture == * [[System Overview]] — principle, layer map, data flows @@ -19,10 +21,18 @@ This is the index of the Open-IOV wiki. If you are new, read '''[[Project Status * [[GMMU Multi-Tenant Protection]] — isolation design for multiple VMs and tenants * [[Physical GPU Backend and Guest Wiring]] — how guest work reaches real hardware * [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA, with real control IDs and log signatures +* [[Performance]] — the cost model, tracing, gaps and improvement programme +* [[Interfaces and Stability]] — which surfaces are contracts, which may change == CUDA == * [[CUDA Userspace Driver]] — the <code>libcuda.so</code> deep dive -* [[The Execution Frontier]] — the GR compute-launch chain and its current wall +* [[The Execution Frontier]] — the GR compute-launch chain and its leading edge + +== Graphics == +* [[Graphics Execution]] — how OpenGL/Vulkan draw commands reach the physical engines +* [[OpenGL]] — the stock guest GL stack on the emulated vGPU +* [[Vulkan]] — the guest Vulkan pipeline on the emulated vGPU +* [[DirectX]] — the Windows/DirectX target == Firmware interoperability == * [[GSP 570.144W Firmware Analysis]] — how the 28.5 MB image was decompiled and mapped diff --git a/tools/site/pages/Main-Page.wiki b/tools/site/pages/Main-Page.wiki index f91fc17..f3fb52a 100644 --- a/tools/site/pages/Main-Page.wiki +++ b/tools/site/pages/Main-Page.wiki @@ -12,13 +12,15 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i </div> == Start here == +* '''[[Project Goals]]''' — what Open-IOV is for, and the status of each goal. +* '''[[Project Plan]]''' — how the work is sequenced, gated, and what is next. * '''[[Project Status]]''' — the honest, dated state of the project. * '''[[Install and First Run]]''' — fresh host to a stock guest driver in about half an hour. * '''[[System Overview]]''' — how Open-IOV works, in one page. * '''[[Documentation]]''' — the full index of this wiki. * '''[[FAQ]]''' — the questions we get asked most. -== Project status (2026-10-20) == +== Project status (2026-11-09) == {| class="wikitable" ! Area !! State |- @@ -30,15 +32,26 @@ The only new component is one host kernel module (<code>vgsp.ko</code>), which i |- | CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working |- -| Guest CUDA kernel execution on real hardware || Working — vector-add, matrix-multiply and transformer kernels run green on fresh boots; hardening and breadth in progress +| Guest CUDA kernel execution on real hardware || Working — vector-add, matrix-multiply and transformer kernels run green on fresh boots; the battery's staged-transfer class and per-guest isolation are in progress |- -| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet — multiple mediated devices (different emulated VRAM sizes) and multiple guests already coexist; per-guest execution isolation is in progress +| Display and KMS (nvidia-modeset/NvKms, /dev/dri, X) || Working — the emulated display engine serves the modeset push buffers and notifier completion; NvKms registers, X settles and glxinfo reports OpenGL 4.6.0 with direct rendering +|- +| 3D graphics: Xorg + OpenGL, Vulkan || In progress — the display/KMS and GL stacks run, the 2D path is green, the DDX 3D replay executes, and the first real GPU readback is proven (pbuffer red pixel). The direct-GL client channel still emits only setup/notify kicks and a window draw wedges X, so direct-GL draw capture and the window/screen present are the current frontier — see [[Graphics Execution]], [[OpenGL]], [[Vulkan]] +|- +| Windows guests, multi-tenancy, migration, performance || Not yet — multiple mediated devices (different emulated VRAM sizes) and multiple guests already coexist; per-guest execution isolation is in progress |} The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]]. +== Graphics == +* [[Graphics Execution]] — how guest 3D work reaches the physical engines +* [[OpenGL]] — the stock guest GL stack on the emulated vGPU +* [[Vulkan]] — the guest Vulkan pipeline on the emulated vGPU +* [[DirectX]] — the Windows/DirectX target + == Deep dives == * [[System Overview]] — the architecture before any subsystem +* [[Project Goals]] — mission, scope and goal-by-goal status * [[GSP and GSP-RM]] — the firmware interface we emulate * [[The vGSP Execution Plane (vEP)]] — how guest work becomes execution and completion * [[The Host Module (vgsp.ko)]] — the module internals @@ -48,6 +61,8 @@ The full story, with evidence and the current wall: [[Project Status]] and [[The * [[UVM in the Guest]] — unified memory as a correctness oracle * [[Physical GPU Backend and Guest Wiring]] — how work reaches real hardware * [[The Execution Frontier]] — the leading edge, evidence-led +* [[Performance]] — the cost model, tracing and improvement programme +* [[Interfaces and Stability]] — which surfaces are contracts, which may change * [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA with real IDs and log signatures == Firmware interoperability == diff --git a/tools/site/pages/MediaWiki-Sidebar.wiki b/tools/site/pages/MediaWiki-Sidebar.wiki index 4e6ab70..0085b42 100644 --- a/tools/site/pages/MediaWiki-Sidebar.wiki +++ b/tools/site/pages/MediaWiki-Sidebar.wiki @@ -1,5 +1,7 @@ * navigation ** mainpage|mainpage-description +** Project Goals +** Project Plan ** Project Status ** Documentation ** FAQ @@ -15,6 +17,13 @@ ** UVM in the Guest ** Physical GPU Backend and Guest Wiring ** The Execution Frontier +** Performance +** Interfaces and Stability +* Graphics +** Graphics Execution +** OpenGL +** Vulkan +** DirectX * Project ** Open-IOV:About|About Open-IOV ** Install and First Run diff --git a/tools/site/pages/Open-IOV-About.wiki b/tools/site/pages/Open-IOV-About.wiki index 0195c91..5fcad03 100644 --- a/tools/site/pages/Open-IOV-About.wiki +++ b/tools/site/pages/Open-IOV-About.wiki @@ -1,6 +1,6 @@ {{About message}} -'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs. It runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with GPU compute (CUDA), and eventually 3D graphics and video, on hardware you already own: without a proprietary vGPU license, without patching the guest driver, the host driver, the hypervisor or the GPU firmware. +'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs. It runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with GPU compute (CUDA) working today, 3D graphics in active development, and video on the roadmap, on hardware you already own: without a proprietary vGPU license, without patching the guest driver, the host driver, the hypervisor or the GPU firmware. == Mission == Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without modifying NVIDIA software, firmware or hardware. @@ -10,7 +10,7 @@ The only component the project adds is a host kernel module (<code>vgsp.ko</code ::'''''Emulate the interface. Pass through the work. Never patch anyone.''''' -See [[System Overview]] for the architecture and [[Project Status]] for the honest current state. +See [[System Overview]] for the architecture, [[Project Goals]] for the mission and the status of each goal, and [[Project Status]] for the honest current state. == Principles == * '''No patches anywhere.''' Guest drivers, host drivers, firmware and hypervisors are all used unmodified; interoperability is the goal, not counterfeiting. diff --git a/tools/site/titles.json b/tools/site/titles.json index b193342..ed2c588 100644 --- a/tools/site/titles.json +++ b/tools/site/titles.json @@ -16,6 +16,10 @@ "vGPU-IO-Integration-Guide.md": "VGPU-IO Integration Guide", "WPR-Management.md": "WPR Management", "Project-Status.md": "Project Status", + "Project-Goals.md": "Project Goals", + "Project-Plan.md": "Project Plan", + "Performance.md": "Performance", + "Interfaces-and-Stability.md": "Interfaces and Stability", "Install-and-First-Run.md": "Install and First Run", "FAQ.md": "FAQ", "System-Overview.md": "System Overview", @@ -23,6 +27,10 @@ "Glossary.md": "Glossary", "Reference-Glossary.md": "Glossary", "Execution-Frontier.md": "The Execution Frontier", + "Graphics-Execution.md": "Graphics Execution", + "OpenGL.md": "OpenGL", + "Vulkan.md": "Vulkan", + "DirectX.md": "DirectX", "Host-Module.md": "The Host Module (vgsp.ko)", "Guest-Stack-and-CUDASW.md": "The Guest Stack and CUDASW", "UVM-in-the-Guest.md": "UVM in the Guest", @@ -33,7 +41,9 @@ "HAAEEP-and-Collaboration.md": "HAAEEP and Collaboration" }, "mermaid_images": { - "Execution-Plane.md": "[[File:The-vGSP-Execution-Plane-diagram.png|frameless|center|upright=2.2|The vGSP execution plane (vEP): one CUDA launch, end to end.]]" + "Execution-Plane.md": "[[File:The-vGSP-Execution-Plane-diagram.png|frameless|center|upright=2.2|The vGSP execution plane (vEP): one CUDA launch, end to end.]]", + "Graphics-Execution.md": "[[File:Open-IOV-graphics-execution-paths.png|frameless|center|upright=2.2|The graphics execution paths: guest OpenGL/Vulkan/CUDA commands to the physical engines, and completions back.]]", + "Vulkan.md": "[[File:Open-IOV-vulkan-submit-path.png|frameless|center|upright=2.2|One Vulkan queue submit end to end on the emulated vGPU.]]" }, "doc_pages": { "docs/README.md": "Documentation", "docs/03-architecture/01-system-overview.md": "System Overview", diff --git a/tools/vk/vk_frame.c b/tools/vk/vk_frame.c new file mode 100644 index 0000000..62548be --- /dev/null +++ b/tools/vk/vk_frame.c @@ -0,0 +1,510 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* + * vk_frame.c - headless Vulkan render probe for the vGPU guest. + * + * Creates a real Vulkan device on the NVIDIA ICD, runs a compute shader that + * paints an animated plasma into an RGBA8 storage image, copies the image to + * a host-visible buffer, saves a PPM, and (optionally) publishes each frame + * into the kvmfr ivshmem BAR in the frame_producer layout so the host can + * display it. No window system involved (pure headless), so it works even + * while the GL/GLX walls are being fixed. + * + * Usage: ./vk_frame [frames] [width] [height] [fps] [outdir] [publish] + * Build: gcc -O2 -o vk_frame vk_frame.c -lvulkan (same glibc as the guest) + */ +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <fcntl.h> +#include <time.h> +#include <sys/mman.h> +#include <vulkan/vulkan.h> + +#include "vk_plasma_spv.h" + +#define FRAME_OFFSET 0x100000UL +#define HDR_MAGIC 0x4b564d46u + +static VkImageTiling g_tiling = VK_IMAGE_TILING_OPTIMAL; + +struct frame_hdr { + uint32_t magic; + uint32_t version; + uint32_t width; + uint32_t height; + uint32_t stride; + uint32_t format; + uint32_t seq; + uint32_t reserved; + uint64_t timestamp_ns; +}; + +static uint64_t now_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +#define VKC(x) \ + do { \ + VkResult r_ = (x); \ + if (r_ != VK_SUCCESS) { \ + printf("VKFAIL %s -> %d (line %d)\n", #x, r_, \ + __LINE__); \ + return 1; \ + } \ + } while (0) + +int main(int argc, char **argv) +{ + if (argc > 7 && atoi(argv[7]) == 1) { + g_tiling = VK_IMAGE_TILING_LINEAR; + printf("TILING linear\n"); + } + + int frames = argc > 1 ? atoi(argv[1]) : 60; + uint32_t W = argc > 2 ? (uint32_t)atoi(argv[2]) : 640; + uint32_t H = argc > 3 ? (uint32_t)atoi(argv[3]) : 480; + int fps = argc > 4 ? atoi(argv[4]) : 10; + const char *outdir = argc > 5 ? argv[5] : NULL; + int publish = argc > 6 ? atoi(argv[6]) : 1; + VkApplicationInfo app = {0}; + VkInstanceCreateInfo ici = {0}; + VkInstance inst; + uint32_t ndev = 0; + VkPhysicalDevice devs[16]; + VkPhysicalDevice phys = VK_NULL_HANDLE; + char picked[256] = ""; + uint32_t nfam = 0, fam = ~0u; + VkQueueFamilyProperties fams[16]; + VkDevice dev; + VkQueue queue; + VkImage image; + VkDeviceMemory img_mem; + VkImageView view; + VkBuffer buf; + VkDeviceMemory buf_mem; + VkShaderModule shader; + VkDescriptorSetLayout dsl; + VkPipelineLayout pl; + VkPipeline pipe; + VkDescriptorPool pool; + VkDescriptorSet dset; + VkCommandPool cpool; + VkCommandBuffer cb; + VkFence fence; + uint32_t i; + float prio = 1.0f; + VkDeviceSize imgsz = (VkDeviceSize)W * H * 4; + unsigned char *pix; + struct frame_hdr *hdr = NULL; + volatile uint32_t *seqp = NULL; + unsigned char *bar = NULL; + int bfd = -1; + int saved = 0; + uint64_t t_first = 0, t_last = 0; + + setvbuf(stdout, NULL, _IONBF, 0); + + app.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO; + app.pApplicationName = "vk_frame"; + app.pEngineName = "vk_frame"; + app.apiVersion = VK_API_VERSION_1_2; + ici.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO; + ici.pApplicationInfo = &app; + VKC(vkCreateInstance(&ici, NULL, &inst)); + printf("VKINSTANCE ok\n"); + + VKC(vkEnumeratePhysicalDevices(inst, &ndev, NULL)); + if (ndev > 16) + ndev = 16; + VKC(vkEnumeratePhysicalDevices(inst, &ndev, devs)); + printf("VKDEVS %u\n", ndev); + for (i = 0; i < ndev; i++) { + VkPhysicalDeviceProperties pp; + + vkGetPhysicalDeviceProperties(devs[i], &pp); + printf("VKDEV %u: %s vendor=0x%04x\n", i, pp.deviceName, + pp.vendorID); + if (pp.vendorID == 0x10de || strstr(pp.deviceName, "NVIDIA")) { + phys = devs[i]; + snprintf(picked, sizeof(picked), "%s", + pp.deviceName); + } + } + if (!phys) { + printf("VK_NO_NVIDIA\n"); + return 2; + } + printf("VKPICK %s\n", picked); + vkGetPhysicalDeviceQueueFamilyProperties(phys, &nfam, NULL); + if (nfam > 16) + nfam = 16; + vkGetPhysicalDeviceQueueFamilyProperties(phys, &nfam, fams); + for (i = 0; i < nfam; i++) + if ((fams[i].queueFlags & VK_QUEUE_COMPUTE_BIT) && + fams[i].queueCount > 0) { + fam = i; + break; + } + if (fam == ~0u) { + printf("VK_NO_COMPUTE_QUEUE\n"); + return 3; + } + { + VkDeviceQueueCreateInfo qci = {0}; + VkDeviceCreateInfo dci = {0}; + + qci.sType = VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO; + qci.queueFamilyIndex = fam; + qci.queueCount = 1; + qci.pQueuePriorities = &prio; + dci.sType = VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO; + dci.queueCreateInfoCount = 1; + dci.pQueueCreateInfos = &qci; + VKC(vkCreateDevice(phys, &dci, NULL, &dev)); + } + printf("VKDEVICE ok family=%u\n", fam); + vkGetDeviceQueue(dev, fam, 0, &queue); + + /* storage image */ + { + VkImageCreateInfo ci = {0}; + VkMemoryRequirements mr; + VkMemoryAllocateInfo ma = {0}; + + ci.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO; + ci.imageType = VK_IMAGE_TYPE_2D; + ci.format = VK_FORMAT_R8G8B8A8_UNORM; + ci.extent.width = W; + ci.extent.height = H; + ci.extent.depth = 1; + ci.mipLevels = 1; + ci.arrayLayers = 1; + ci.samples = VK_SAMPLE_COUNT_1_BIT; + ci.tiling = g_tiling; + ci.usage = VK_IMAGE_USAGE_STORAGE_BIT | + VK_IMAGE_USAGE_TRANSFER_SRC_BIT; + ci.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; + VKC(vkCreateImage(dev, &ci, NULL, &image)); + vkGetImageMemoryRequirements(dev, image, &mr); + ma.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; + ma.allocationSize = mr.size; + /* find a device-local memory type */ + { + VkPhysicalDeviceMemoryProperties mp; + uint32_t k; + + vkGetPhysicalDeviceMemoryProperties(phys, &mp); + for (k = 0; k < mp.memoryTypeCount; k++) + if ((mr.memoryTypeBits & (1u << k)) && + (mp.memoryTypes[k].propertyFlags & + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)) { + ma.memoryTypeIndex = k; + break; + } + } + VKC(vkAllocateMemory(dev, &ma, NULL, &img_mem)); + VKC(vkBindImageMemory(dev, image, img_mem, 0)); + } + { + VkImageViewCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO; + ci.image = image; + ci.viewType = VK_IMAGE_VIEW_TYPE_2D; + ci.format = VK_FORMAT_R8G8B8A8_UNORM; + ci.subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + ci.subresourceRange.levelCount = 1; + ci.subresourceRange.layerCount = 1; + VKC(vkCreateImageView(dev, &ci, NULL, &view)); + } + /* host-visible readback buffer */ + { + VkBufferCreateInfo ci = {0}; + VkMemoryRequirements mr; + VkMemoryAllocateInfo ma = {0}; + + ci.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO; + ci.size = imgsz; + ci.usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT; + ci.sharingMode = VK_SHARING_MODE_EXCLUSIVE; + VKC(vkCreateBuffer(dev, &ci, NULL, &buf)); + vkGetBufferMemoryRequirements(dev, buf, &mr); + ma.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; + ma.allocationSize = mr.size; + { + VkPhysicalDeviceMemoryProperties mp; + uint32_t k; + + vkGetPhysicalDeviceMemoryProperties(phys, &mp); + for (k = 0; k < mp.memoryTypeCount; k++) + if ((mr.memoryTypeBits & (1u << k)) && + (mp.memoryTypes[k].propertyFlags & + (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | + VK_MEMORY_PROPERTY_HOST_COHERENT_BIT))) { + ma.memoryTypeIndex = k; + break; + } + } + { + int rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + if (rc_ != VK_SUCCESS) { + VkPhysicalDeviceMemoryProperties mp2; + uint32_t k2; + vkGetPhysicalDeviceMemoryProperties(phys, &mp2); + for (k2 = 0; k2 < mp2.memoryTypeCount && rc_ != VK_SUCCESS; k2++) { + if (!(mr.memoryTypeBits & (1u << k2))) + continue; + ma.memoryTypeIndex = k2; + rc_ = vkAllocateMemory(dev, &ma, NULL, &buf_mem); + printf("BUF retry type=%u rc=%d\n", k2, rc_); + } + } + if (rc_ != VK_SUCCESS) { + printf("VKFAIL vkAllocateMemory(buf) -> %d (line %d)\n", rc_, __LINE__); + return 1; + } + } + VKC(vkBindBufferMemory(dev, buf, buf_mem, 0)); + } + /* shader + pipeline */ + { + VkShaderModuleCreateInfo ci = {0}; + VkDescriptorSetLayoutBinding b = {0}; + VkDescriptorSetLayoutCreateInfo dci = {0}; + VkPushConstantRange pcr = {0}; + VkPipelineLayoutCreateInfo plci = {0}; + VkComputePipelineCreateInfo pci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO; + ci.codeSize = vk_plasma_spv_len; + ci.pCode = (const uint32_t *)vk_plasma_spv; + VKC(vkCreateShaderModule(dev, &ci, NULL, &shader)); + b.binding = 0; + b.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + b.descriptorCount = 1; + b.stageFlags = VK_SHADER_STAGE_COMPUTE_BIT; + dci.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO; + dci.bindingCount = 1; + dci.pBindings = &b; + VKC(vkCreateDescriptorSetLayout(dev, &dci, NULL, &dsl)); + pcr.stageFlags = VK_SHADER_STAGE_COMPUTE_BIT; + pcr.size = 16; + plci.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO; + plci.setLayoutCount = 1; + plci.pSetLayouts = &dsl; + plci.pushConstantRangeCount = 1; + plci.pPushConstantRanges = &pcr; + VKC(vkCreatePipelineLayout(dev, &plci, NULL, &pl)); + pci.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO; + pci.stage.sType = + VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO; + pci.stage.stage = VK_SHADER_STAGE_COMPUTE_BIT; + pci.stage.module = shader; + pci.stage.pName = "main"; + pci.layout = pl; + VKC(vkCreateComputePipelines(dev, VK_NULL_HANDLE, 1, &pci, + NULL, &pipe)); + } + { + VkDescriptorPoolSize ps = {0}; + VkDescriptorPoolCreateInfo pci = {0}; + VkDescriptorSetAllocateInfo ai = {0}; + VkDescriptorImageInfo ii = {0}; + VkWriteDescriptorSet w = {0}; + + ps.type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + ps.descriptorCount = 1; + pci.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO; + pci.maxSets = 1; + pci.poolSizeCount = 1; + pci.pPoolSizes = &ps; + VKC(vkCreateDescriptorPool(dev, &pci, NULL, &pool)); + ai.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO; + ai.descriptorPool = pool; + ai.descriptorSetCount = 1; + ai.pSetLayouts = &dsl; + VKC(vkAllocateDescriptorSets(dev, &ai, &dset)); + ii.imageView = view; + ii.imageLayout = VK_IMAGE_LAYOUT_GENERAL; + w.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET; + w.dstSet = dset; + w.dstBinding = 0; + w.descriptorCount = 1; + w.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE; + w.pImageInfo = ⅈ + vkUpdateDescriptorSets(dev, 1, &w, 0, NULL); + } + { + VkCommandPoolCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO; + ci.queueFamilyIndex = fam; + VKC(vkCreateCommandPool(dev, &ci, NULL, &cpool)); + VkCommandBufferAllocateInfo ai = {0}; + + ai.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO; + ai.commandPool = cpool; + ai.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY; + ai.commandBufferCount = 1; + VKC(vkAllocateCommandBuffers(dev, &ai, &cb)); + } + { + VkFenceCreateInfo ci = {0}; + + ci.sType = VK_STRUCTURE_TYPE_FENCE_CREATE_INFO; + VKC(vkCreateFence(dev, &ci, NULL, &fence)); + } + if (publish) { + const char *bp = + "/sys/bus/pci/devices/0000:00:03.0/resource2"; + + bfd = open(bp, O_RDWR | O_SYNC); + if (bfd >= 0) { + bar = mmap(NULL, 128UL << 20, PROT_READ | PROT_WRITE, + MAP_SHARED, bfd, 0); + if (bar != MAP_FAILED) { + hdr = (struct frame_hdr *)(bar + + FRAME_OFFSET); + seqp = &hdr->seq; + hdr->magic = HDR_MAGIC; + hdr->version = 1; + hdr->width = W; + hdr->height = H; + hdr->stride = W * 4; + hdr->format = 0; + printf("PUBLISH on\n"); + } else { + bar = NULL; + } + } + if (!bar) + printf("PUBLISH off (BAR open/mmap failed)\n"); + } + VKC(vkMapMemory(dev, buf_mem, 0, imgsz, 0, (void **)&pix)); + + for (i = 0; i < (uint32_t)frames; i++) { + VkCommandBufferBeginInfo bi = {0}; + VkImageMemoryBarrier ib = {0}; + VkBufferImageCopy bc = {0}; + VkBufferMemoryBarrier bb = {0}; + float pc[4]; + uint64_t t0 = now_ns(); + VkSubmitInfo si = {0}; + uint32_t gw = (W + 15) / 16, gh = (H + 15) / 16; + + VKC(vkResetCommandBuffer(cb, 0)); + bi.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO; + VKC(vkBeginCommandBuffer(cb, &bi)); + ib.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER; + ib.srcAccessMask = 0; + ib.dstAccessMask = VK_ACCESS_SHADER_WRITE_BIT; + ib.oldLayout = VK_IMAGE_LAYOUT_UNDEFINED; + ib.newLayout = VK_IMAGE_LAYOUT_GENERAL; + ib.image = image; + ib.subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + ib.subresourceRange.levelCount = 1; + ib.subresourceRange.layerCount = 1; + vkCmdPipelineBarrier(cb, + VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 0, + NULL, 0, NULL, 1, &ib); + vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, pipe); + vkCmdBindDescriptorSets(cb, VK_PIPELINE_BIND_POINT_COMPUTE, pl, + 0, 1, &dset, 0, NULL); + pc[0] = (float)i * 0.1f; + pc[1] = (float)W; + pc[2] = (float)H; + pc[3] = 0.0f; + vkCmdPushConstants(cb, pl, VK_SHADER_STAGE_COMPUTE_BIT, 0, 16, + pc); + vkCmdDispatch(cb, gw, gh, 1); + ib.srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT; + ib.dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT; + ib.oldLayout = VK_IMAGE_LAYOUT_GENERAL; + ib.newLayout = VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL; + vkCmdPipelineBarrier(cb, + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, NULL, + 0, NULL, 1, &ib); + bc.imageSubresource.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT; + bc.imageSubresource.layerCount = 1; + bc.imageExtent.width = W; + bc.imageExtent.height = H; + bc.imageExtent.depth = 1; + vkCmdCopyImageToBuffer(cb, image, + VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, + buf, 1, &bc); + bb.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER; + bb.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT; + bb.dstAccessMask = VK_ACCESS_HOST_READ_BIT; + bb.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + bb.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + bb.buffer = buf; + bb.size = VK_WHOLE_SIZE; + vkCmdPipelineBarrier(cb, VK_PIPELINE_STAGE_TRANSFER_BIT, + VK_PIPELINE_STAGE_HOST_BIT, 0, 0, NULL, 1, + &bb, 0, NULL); + VKC(vkEndCommandBuffer(cb)); + si.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO; + si.commandBufferCount = 1; + si.pCommandBuffers = &cb; + VKC(vkResetFences(dev, 1, &fence)); + VKC(vkQueueSubmit(queue, 1, &si, fence)); + VKC(vkWaitForFences(dev, 1, &fence, VK_TRUE, 30000000000ull)); + vkUnmapMemory(dev, buf_mem); + vkMapMemory(dev, buf_mem, 0, imgsz, 0, (void **)&pix); + + if (outdir) { + char path[512]; + FILE *f; + + snprintf(path, sizeof(path), "%s/vk-%04d.ppm", + outdir, i + 1); + f = fopen(path, "wb"); + if (f) { + unsigned char *d = malloc((size_t)W * H * 3); + uint32_t p; + + for (p = 0; p < W * H; p++) { + d[p * 3 + 0] = pix[p * 4 + 0]; + d[p * 3 + 1] = pix[p * 4 + 1]; + d[p * 3 + 2] = pix[p * 4 + 2]; + } + fprintf(f, "P6\n%u %u\n255\n", W, H); + fwrite(d, 1, (size_t)W * H * 3, f); + fclose(f); + free(d); + saved++; + } + } + if (bar) { + memcpy(bar + FRAME_OFFSET + sizeof(struct frame_hdr), + pix, imgsz); + hdr->timestamp_ns = now_ns(); + __sync_synchronize(); + *seqp = (uint32_t)(i + 1); + __sync_synchronize(); + } + if (!t_first) + t_first = t0; + t_last = t0; + if (i < 5 || (i % 30) == 29) + printf("VKFRAME seq=%d ms=%.1f\n", i + 1, + (double)(now_ns() - t0) / 1e6); + if (fps > 0) + usleep((useconds_t)(1000000 / fps)); + } + if (t_first && t_last > t_first) + printf("VKFRAME_RATE fps=%.2f frames=%d\n", + (double)(i - 1) * 1e9 / (double)(t_last - t_first), i); + printf("VKFRAME_DONE frames=%d %ux%u saved=%d\n", i, W, H, saved); + vkDeviceWaitIdle(dev); + printf("VKFRAME_IDLE ok\n"); + return 0; +}
\ No newline at end of file diff --git a/tools/vk/vk_plasma.comp b/tools/vk/vk_plasma.comp new file mode 100644 index 0000000..fe6b3ac --- /dev/null +++ b/tools/vk/vk_plasma.comp @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: BSD-3-Clause +// vk_plasma.comp - compute-shader frame painter for the Vulkan guest test. +#version 450 +layout(local_size_x = 16, local_size_y = 16) in; +layout(set = 0, binding = 0, rgba8) uniform writeonly image2D fb; +layout(push_constant) uniform PC { float t; float w; float h; float pad; } pc; +void main() { + ivec2 p = ivec2(gl_GlobalInvocationID.xy); + if (p.x >= int(pc.w) || p.y >= int(pc.h)) + return; + vec2 uv = vec2(p) / vec2(pc.w, pc.h); + float v = sin(uv.x * 12.0 + pc.t * 2.0) * cos(uv.y * 9.0 - pc.t * 1.7); + float r = 0.5 + 0.5 * sin(pc.t + uv.x * 6.2831); + float g = 0.5 + 0.5 * sin(pc.t * 1.3 + uv.y * 6.2831); + float b = 0.5 + 0.5 * v; + imageStore(fb, p, vec4(r, g, b, 1.0)); +} diff --git a/tools/vk/vk_plasma.spv b/tools/vk/vk_plasma.spv Binary files differnew file mode 100644 index 0000000..7c12b51 --- /dev/null +++ b/tools/vk/vk_plasma.spv diff --git a/tools/vk/vk_plasma_spv.h b/tools/vk/vk_plasma_spv.h new file mode 100644 index 0000000..49c16f1 --- /dev/null +++ b/tools/vk/vk_plasma_spv.h @@ -0,0 +1,237 @@ +unsigned char vk_plasma_spv[] = { + 0x03, 0x02, 0x23, 0x07, 0x00, 0x00, 0x01, 0x00, 0x0b, 0x00, 0x08, 0x00, + 0x7c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x11, 0x00, 0x02, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x06, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x47, 0x4c, 0x53, 0x4c, 0x2e, 0x73, 0x74, 0x64, 0x2e, 0x34, 0x35, 0x30, + 0x00, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x06, 0x00, 0x05, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x6d, 0x61, 0x69, 0x6e, 0x00, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x10, 0x00, 0x06, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x11, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x03, 0x00, 0x03, 0x00, 0x02, 0x00, 0x00, 0x00, + 0xc2, 0x01, 0x00, 0x00, 0x05, 0x00, 0x04, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x6d, 0x61, 0x69, 0x6e, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x70, 0x00, 0x00, 0x00, 0x05, 0x00, 0x08, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x67, 0x6c, 0x5f, 0x47, 0x6c, 0x6f, 0x62, 0x61, + 0x6c, 0x49, 0x6e, 0x76, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x49, + 0x44, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x50, 0x43, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x74, 0x00, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, + 0x18, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x00, + 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x68, 0x00, 0x00, 0x00, 0x06, 0x00, 0x04, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x03, 0x00, 0x00, 0x00, 0x70, 0x61, 0x64, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x1a, 0x00, 0x00, 0x00, 0x70, 0x63, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x75, 0x76, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x3c, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x53, 0x00, 0x00, 0x00, 0x72, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x67, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x6b, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, 0x05, 0x00, 0x03, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x66, 0x62, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x03, 0x00, 0x18, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x23, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x48, 0x00, 0x05, 0x00, + 0x18, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, + 0x48, 0x00, 0x05, 0x00, 0x18, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x23, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x47, 0x00, 0x03, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x04, 0x00, 0x71, 0x00, 0x00, 0x00, 0x22, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x47, 0x00, 0x04, 0x00, 0x7b, 0x00, 0x00, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x13, 0x00, 0x02, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x21, 0x00, 0x03, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x15, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x08, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x15, 0x00, 0x04, 0x00, 0x0a, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x0c, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, + 0x0e, 0x00, 0x00, 0x00, 0x0a, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, + 0x14, 0x00, 0x02, 0x00, 0x12, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x0a, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x14, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x16, 0x00, 0x03, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x1e, 0x00, 0x06, 0x00, 0x18, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, 0x19, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x19, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x1b, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x0a, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x27, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x17, 0x00, 0x04, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x20, 0x00, 0x04, 0x00, + 0x31, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x3f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x41, 0x2b, 0x00, 0x04, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x44, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x40, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x4a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x41, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x4e, 0x00, 0x00, 0x00, 0x9a, 0x99, 0xd9, 0x3f, + 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x3f, 0x2b, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x59, 0x00, 0x00, 0x00, 0x28, 0x0f, 0xc9, 0x40, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, 0x66, 0x66, 0xa6, 0x3f, + 0x19, 0x00, 0x09, 0x00, 0x6f, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x04, 0x00, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x6f, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x70, 0x00, 0x00, 0x00, + 0x71, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x3f, + 0x17, 0x00, 0x04, 0x00, 0x78, 0x00, 0x00, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x04, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x04, 0x00, 0x0a, 0x00, 0x00, 0x00, + 0x7a, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x2c, 0x00, 0x06, 0x00, + 0x0b, 0x00, 0x00, 0x00, 0x7b, 0x00, 0x00, 0x00, 0x7a, 0x00, 0x00, 0x00, + 0x7a, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x36, 0x00, 0x05, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x03, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, + 0x3b, 0x00, 0x04, 0x00, 0x08, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x31, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x3b, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x53, 0x00, 0x00, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, 0x3b, 0x00, 0x00, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, 0x3b, 0x00, 0x04, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x0b, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, + 0x0d, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x07, 0x00, 0x0e, 0x00, 0x00, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, 0x0f, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x7c, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x11, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x03, 0x00, 0x09, 0x00, 0x00, 0x00, 0x11, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x14, 0x00, 0x00, 0x00, 0x15, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x06, 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x15, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x1d, 0x00, 0x00, 0x00, + 0x1a, 0x00, 0x00, 0x00, 0x1b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x1d, 0x00, 0x00, 0x00, + 0x6e, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x1f, 0x00, 0x00, 0x00, + 0x1e, 0x00, 0x00, 0x00, 0xaf, 0x00, 0x05, 0x00, 0x12, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x16, 0x00, 0x00, 0x00, 0x1f, 0x00, 0x00, 0x00, + 0xa8, 0x00, 0x04, 0x00, 0x12, 0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0xf7, 0x00, 0x03, 0x00, 0x23, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0xfa, 0x00, 0x04, 0x00, 0x21, 0x00, 0x00, 0x00, + 0x22, 0x00, 0x00, 0x00, 0x23, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, + 0x22, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x14, 0x00, 0x00, 0x00, + 0x25, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x26, 0x00, 0x00, 0x00, + 0x25, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x28, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x27, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x29, 0x00, 0x00, 0x00, + 0x28, 0x00, 0x00, 0x00, 0x6e, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, + 0x2a, 0x00, 0x00, 0x00, 0x29, 0x00, 0x00, 0x00, 0xaf, 0x00, 0x05, 0x00, + 0x12, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x00, 0x00, 0x26, 0x00, 0x00, 0x00, + 0x2a, 0x00, 0x00, 0x00, 0xf9, 0x00, 0x02, 0x00, 0x23, 0x00, 0x00, 0x00, + 0xf8, 0x00, 0x02, 0x00, 0x23, 0x00, 0x00, 0x00, 0xf5, 0x00, 0x07, 0x00, + 0x12, 0x00, 0x00, 0x00, 0x2c, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, + 0x05, 0x00, 0x00, 0x00, 0x2b, 0x00, 0x00, 0x00, 0x22, 0x00, 0x00, 0x00, + 0xf7, 0x00, 0x03, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0xfa, 0x00, 0x04, 0x00, 0x2c, 0x00, 0x00, 0x00, 0x2d, 0x00, 0x00, 0x00, + 0x2e, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x2d, 0x00, 0x00, 0x00, + 0xfd, 0x00, 0x01, 0x00, 0xf8, 0x00, 0x02, 0x00, 0x2e, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x07, 0x00, 0x00, 0x00, 0x33, 0x00, 0x00, 0x00, + 0x09, 0x00, 0x00, 0x00, 0x6f, 0x00, 0x04, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x34, 0x00, 0x00, 0x00, 0x33, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x35, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x1b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x36, 0x00, 0x00, 0x00, 0x35, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x27, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x38, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x50, 0x00, 0x05, 0x00, + 0x30, 0x00, 0x00, 0x00, 0x39, 0x00, 0x00, 0x00, 0x36, 0x00, 0x00, 0x00, + 0x38, 0x00, 0x00, 0x00, 0x88, 0x00, 0x05, 0x00, 0x30, 0x00, 0x00, 0x00, + 0x3a, 0x00, 0x00, 0x00, 0x34, 0x00, 0x00, 0x00, 0x39, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x03, 0x00, 0x32, 0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, + 0x3e, 0x00, 0x00, 0x00, 0x3f, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x43, 0x00, 0x00, 0x00, 0x42, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, 0x43, 0x00, 0x00, 0x00, + 0x44, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x46, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x45, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x47, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, 0x46, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x49, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x4b, 0x00, 0x00, 0x00, + 0x49, 0x00, 0x00, 0x00, 0x4a, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x4c, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x4d, 0x00, 0x00, 0x00, 0x4c, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x4d, 0x00, 0x00, 0x00, + 0x4e, 0x00, 0x00, 0x00, 0x83, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x50, 0x00, 0x00, 0x00, 0x4b, 0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x50, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x52, 0x00, 0x00, 0x00, + 0x47, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, + 0x3c, 0x00, 0x00, 0x00, 0x52, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x1c, 0x00, 0x00, 0x00, 0x55, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x56, 0x00, 0x00, 0x00, 0x55, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, + 0x3b, 0x00, 0x00, 0x00, 0x57, 0x00, 0x00, 0x00, 0x32, 0x00, 0x00, 0x00, + 0x13, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x58, 0x00, 0x00, 0x00, 0x57, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x5a, 0x00, 0x00, 0x00, 0x58, 0x00, 0x00, 0x00, + 0x59, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x5b, 0x00, 0x00, 0x00, 0x56, 0x00, 0x00, 0x00, 0x5a, 0x00, 0x00, 0x00, + 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, 0x5c, 0x00, 0x00, 0x00, + 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, 0x5b, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x5d, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x5c, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x5e, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x5d, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, 0x53, 0x00, 0x00, 0x00, + 0x5e, 0x00, 0x00, 0x00, 0x41, 0x00, 0x05, 0x00, 0x1c, 0x00, 0x00, 0x00, + 0x60, 0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x61, 0x00, 0x00, 0x00, + 0x60, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x63, 0x00, 0x00, 0x00, 0x61, 0x00, 0x00, 0x00, 0x62, 0x00, 0x00, 0x00, + 0x41, 0x00, 0x05, 0x00, 0x3b, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, + 0x32, 0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x65, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x66, 0x00, 0x00, 0x00, + 0x65, 0x00, 0x00, 0x00, 0x59, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x67, 0x00, 0x00, 0x00, 0x63, 0x00, 0x00, 0x00, + 0x66, 0x00, 0x00, 0x00, 0x0c, 0x00, 0x06, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x68, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x0d, 0x00, 0x00, 0x00, + 0x67, 0x00, 0x00, 0x00, 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x69, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, 0x68, 0x00, 0x00, 0x00, + 0x81, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x6a, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x69, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, + 0x5f, 0x00, 0x00, 0x00, 0x6a, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x6c, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, 0x00, + 0x85, 0x00, 0x05, 0x00, 0x17, 0x00, 0x00, 0x00, 0x6d, 0x00, 0x00, 0x00, + 0x54, 0x00, 0x00, 0x00, 0x6c, 0x00, 0x00, 0x00, 0x81, 0x00, 0x05, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x6e, 0x00, 0x00, 0x00, 0x54, 0x00, 0x00, 0x00, + 0x6d, 0x00, 0x00, 0x00, 0x3e, 0x00, 0x03, 0x00, 0x6b, 0x00, 0x00, 0x00, + 0x6e, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x6f, 0x00, 0x00, 0x00, + 0x72, 0x00, 0x00, 0x00, 0x71, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x07, 0x00, 0x00, 0x00, 0x73, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, + 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, 0x74, 0x00, 0x00, 0x00, + 0x53, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, 0x17, 0x00, 0x00, 0x00, + 0x75, 0x00, 0x00, 0x00, 0x5f, 0x00, 0x00, 0x00, 0x3d, 0x00, 0x04, 0x00, + 0x17, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, 0x6b, 0x00, 0x00, 0x00, + 0x50, 0x00, 0x07, 0x00, 0x78, 0x00, 0x00, 0x00, 0x79, 0x00, 0x00, 0x00, + 0x74, 0x00, 0x00, 0x00, 0x75, 0x00, 0x00, 0x00, 0x76, 0x00, 0x00, 0x00, + 0x77, 0x00, 0x00, 0x00, 0x63, 0x00, 0x04, 0x00, 0x72, 0x00, 0x00, 0x00, + 0x73, 0x00, 0x00, 0x00, 0x79, 0x00, 0x00, 0x00, 0xfd, 0x00, 0x01, 0x00, + 0x38, 0x00, 0x01, 0x00 +}; +unsigned int vk_plasma_spv_len = 2800; diff --git a/tools/vk/vk_probe.py b/tools/vk/vk_probe.py new file mode 100644 index 0000000..3113c3f --- /dev/null +++ b/tools/vk/vk_probe.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +# vk_probe.py - enumerate real Vulkan physical devices via the loader (ctypes, +# no dev headers needed). Prints one DEV line per physical device. +import ctypes +import sys + +VK_SUCCESS = 0 +VK_STRUCTURE_TYPE_APPLICATION_INFO = 0 +VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO = 1 +VK_API_VERSION_1_2 = (1 << 22) | (2 << 12) + +try: + vulkan = ctypes.CDLL("libvulkan.so.1") +except OSError as e: + print("LOAD_FAIL %s" % e) + sys.exit(1) + + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), + ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), + ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32), + ] + + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("pApplicationInfo", ctypes.POINTER(VkApplicationInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ] + + +app = VkApplicationInfo( + VK_STRUCTURE_TYPE_APPLICATION_INFO, None, b"vk_probe", 1, b"vk_probe", 1, + VK_API_VERSION_1_2) +ci = VkInstanceCreateInfo( + VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, None, 0, ctypes.pointer(app), 0, + None, 0, None) +inst = ctypes.c_void_p() +r = vulkan.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("vkCreateInstance rc=%d (0x%x)" % (r, r & 0xfffffffff if r < 0 else r)) +if r != VK_SUCCESS: + sys.exit(1) + +n = ctypes.c_uint32(0) +r = vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +print("vkEnumeratePhysicalDevices rc=%d count=%d" % (r, n.value)) +if r != VK_SUCCESS or n.value == 0: + sys.exit(2) + +devs = (ctypes.c_void_p * n.value)() +r = vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) +ok = 0 +for i in range(n.value): + buf = ctypes.create_string_buffer(4096) + vulkan.vkGetPhysicalDeviceProperties(devs[i], buf) + raw = buf.raw + api = int.from_bytes(raw[0:4], "little") + drv = int.from_bytes(raw[4:8], "little") + vendor = int.from_bytes(raw[8:12], "little") + devid = int.from_bytes(raw[12:16], "little") + dtype = int.from_bytes(raw[16:20], "little") + name = raw[20:20 + 256].split(b"\x00")[0].decode("utf-8", "replace") + print("DEV %d: %s vendor=0x%04x device=0x%04x type=%d api=%d.%d.%d drv=%d" + % (i, name, vendor, devid, dtype, api >> 22, + (api >> 12) & 0x3ff, api & 0xfff, drv)) + if vendor == 0x10de: + ok = 1 +print("VK_NVIDIA=%d" % ok) +sys.exit(0 if ok else 3)
\ No newline at end of file diff --git a/tools/vk/vk_probe2.py b/tools/vk/vk_probe2.py new file mode 100644 index 0000000..557b888 --- /dev/null +++ b/tools/vk/vk_probe2.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +# vk_probe2.py - Vulkan probe stage 2: create a logical device + queue on the +# NVIDIA physical device. Prints the queue-family table and the create +# result. ctypes only, no dev headers. +import ctypes +import sys + +VK_SUCCESS = 0 +VK_STRUCTURE_TYPE_APPLICATION_INFO = 0 +VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO = 1 +VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO = 2 +VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO = 3 +VK_API_VERSION_1_2 = (1 << 22) | (2 << 12) + +vulkan = ctypes.CDLL("libvulkan.so.1") + + +class VkApplicationInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("pApplicationName", ctypes.c_char_p), + ("applicationVersion", ctypes.c_uint32), + ("pEngineName", ctypes.c_char_p), + ("engineVersion", ctypes.c_uint32), + ("apiVersion", ctypes.c_uint32), + ] + + +class VkInstanceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("pApplicationInfo", ctypes.POINTER(VkApplicationInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ] + + +class VkQueueFamilyProperties(ctypes.Structure): + _fields_ = [ + ("queueFlags", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), + ("timestampValidBits", ctypes.c_uint32), + ("minImageTransferGranularity", ctypes.c_uint8 * 12), + ] + + +class VkDeviceQueueCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("queueFamilyIndex", ctypes.c_uint32), + ("queueCount", ctypes.c_uint32), + ("pQueuePriorities", ctypes.POINTER(ctypes.c_float)), + ] + + +class VkDeviceCreateInfo(ctypes.Structure): + _fields_ = [ + ("sType", ctypes.c_int), + ("pNext", ctypes.c_void_p), + ("flags", ctypes.c_uint32), + ("queueCreateInfoCount", ctypes.c_uint32), + ("pQueueCreateInfos", ctypes.POINTER(VkDeviceQueueCreateInfo)), + ("enabledLayerCount", ctypes.c_uint32), + ("ppEnabledLayerNames", ctypes.c_void_p), + ("enabledExtensionCount", ctypes.c_uint32), + ("ppEnabledExtensionNames", ctypes.c_void_p), + ("pEnabledFeatures", ctypes.c_void_p), + ] + + +app = VkApplicationInfo(VK_STRUCTURE_TYPE_APPLICATION_INFO, None, b"vk_probe2", + 1, b"vk_probe2", 1, VK_API_VERSION_1_2) +ci = VkInstanceCreateInfo(VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO, None, 0, + ctypes.pointer(app), 0, None, 0, None) +inst = ctypes.c_void_p() +r = vulkan.vkCreateInstance(ctypes.byref(ci), None, ctypes.byref(inst)) +print("INSTANCE rc=%d" % r) +if r != 0: + sys.exit(1) + +n = ctypes.c_uint32(0) +vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), None) +devs = (ctypes.c_void_p * n.value)() +vulkan.vkEnumeratePhysicalDevices(inst, ctypes.byref(n), devs) + +# pick the NVIDIA device by name +best = None +for i in range(n.value): + buf = ctypes.create_string_buffer(4096) + vulkan.vkGetPhysicalDeviceProperties(devs[i], buf) + raw = buf.raw + name = raw[20:276].split(b"\x00")[0].decode("utf-8", "replace") + print("PROPS dev=%d head=%s name=%r" % (i, raw[:40].hex(), name)) + if "NVIDIA" in name or "llvmpipe" in name: + if "NVIDIA" in name: + best = devs[i] + print("PICK NVIDIA dev=%d name=%s" % (i, name)) + break +if best is None: + print("NO_NVIDIA") + sys.exit(2) + +nfam = ctypes.c_uint32(0) +vulkan.vkGetPhysicalDeviceQueueFamilyProperties(best, ctypes.byref(nfam), + None) +fams = (VkQueueFamilyProperties * nfam.value)() +vulkan.vkGetPhysicalDeviceQueueFamilyProperties(best, ctypes.byref(nfam), + fams) +gfam = None +for j in range(nfam.value): + print("QFAM %d flags=0x%x count=%d" % (j, fams[j].queueFlags, + fams[j].queueCount)) + if (fams[j].queueFlags & 0x1) and fams[j].queueCount > 0 and gfam is None: + gfam = j +if gfam is None: + print("NO_GRAPHICS_QUEUE") + sys.exit(3) +prio = ctypes.c_float(1.0) +qci = VkDeviceQueueCreateInfo(VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO, + None, 0, gfam, 1, ctypes.pointer(prio)) +dci = VkDeviceCreateInfo(VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, None, 0, 1, + ctypes.pointer(qci), 0, None, 0, None, None) +dev = ctypes.c_void_p() +r = vulkan.vkCreateDevice(best, ctypes.byref(dci), None, ctypes.byref(dev)) +print("vkCreateDevice rc=%d (0x%x)" % (r, r if r < 0 else 0)) +if r != 0: + sys.exit(4) +queue = ctypes.c_void_p() +vulkan.vkGetDeviceQueue(dev, gfam, 0, ctypes.byref(queue)) +print("QUEUE device=%s queue=%s" % (bool(dev), bool(queue))) +# optional: wait-idle (exercises the queue) +vulkan.vkDeviceWaitIdle(dev) +print("VK2_DEVICE_OK") +sys.exit(0)
\ No newline at end of file diff --git a/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md b/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md new file mode 100644 index 0000000..48e9f35 --- /dev/null +++ b/userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md @@ -0,0 +1,156 @@ +# ANALYSIS - EGL 3D-engine selection wall (570.144 guest userspace) + +Status: OPEN (2026-10-23). The EGL-device path initialises (devices enumerate, +`eglInitialize` succeeds, `vendor=NVIDIA`), but `eglMakeCurrent` fails with +`EGL_BAD_ALLOC (0x3003)` because the per-device EGL-core client creator +returns NULL after a lower-layer engine-init call fails. + +This document records the *shipped-binary* evidence only (disassembly + +live gdb traces). Method: run the probe under gdb with ASLR disabled +(gdb's default) so library offsets are stable, set breakpoints at the +recorded offsets, and read the arguments/returns. + +## 1. The failing chain (libnvidia-eglcore.so.570.144) + +``` +eglMakeCurrent + -> NvGlEglApiInit family: per-device EGL-core client creator (+0xa246f0) + -> +0xa06ed0 (device/engine selection wrapper) + -> +0x9fe790 (engine-list walk) + list head: *( *(device + 0x28738) + 8 ) + filters per engine: engine[0x59] & 0x40 + engine[0x70] == 0 + per-engine callback: *0x70( *(*(engine+0x28738)+0x10) ) + -> +0xa22000 (callback) + rbx = *(reqs + 0xa0) (device object) + mode = *(reqs + 0x20) + mode 6 -> +0xa09830 + mode 7 -> +0xa0a4d0 + else -> +0x9fdfd0(device, engine, obj, reqs) + NULL -> EGL_BAD_ALLOC +``` + +`+0x9fdfd0` ends in the rejection site: + +``` +9fe146: rax = *(r14 + 0x10) ; r14 = *(engine + 0x28738) +9fe14d: rdi = *(rbx + 0x20) ; rbx = *(reqs + 0xa0) (device object) +9fe151: call *0x30(rax) ; vtable slot 0x30 +9fe154: test eax,eax +9fe156: je 9fe170 ; 0 == continue (success) +9fe158: free(tmp); return 0 ; non-zero == reject +``` + +## 2. The rejecting method (libnvidia-glsi.so.570.144) + +The vtable at `*(*(engine+0x28738)+0x10)` is a *glsi* device-engine vtable; +its slot 0x30 resolves (live gdb: vtable `0x7ffff71ddfe0`, method +`+0xa29440` in eglcore terms = glsi `+0x5f480` base-adjusted) to the glsi +engine-init entry: + +``` +glsi +0x5f480(device, reqs): + 5f4c0: fd = *(*(*(dev+0x108)+0x58)) ; int fd (nvidia device) + 5f4c8: rsi = *(dev+0x118) + 5f4dc: call +0x1aba0(fd, rsi, reqs) ; FIRST call + 5f52e: call +0x1af20(...) ; second call +``` + +Live trace (probe `egl_frame` under gdb, breakpoint at the callback's call +site): the **first** glsi call returns `0xee00001`; the caller propagates it +and the engine is rejected. `0xee00001` is a glsi-internal status produced by +two sites (`glsi +0x31f84`, `+0x33910`) that set flag bit `0x20` in the +device's `+0xb84` word and return "busy"-style; the engine-init state machine +(`glsi +0x301b0` and callees `+0x2fbe0` -> `+0x29760` -> `+0x32110`/`+0x2dc70`) +is what fails. + +## 3. What is ruled out + +* **Not an RM-control error.** During the failing probe the emulated GSP + answers every `GSP_RM_CONTROL` with `status=0x0` (including the + `0x20810107` BINAPI query that was previously suspected; that one is a red + herring - a real class-0x2081 object on the live session answers + `NV_OK`/`{0,0}` exactly like the emulation). +* **Not the engine-list shape alone.** The single candidate engine passes the + walk's own filters (`flag59 & 0x40` set, `f70 == 0`), so the list contains + a usable engine; the failure is inside the per-engine init. +* **Not ASLR-dependent.** All offsets above are stable under gdb + (`set disable-randomization on`). + +## 4. Entry-trace update (2026-10-23, r543 boot) + +A batch-gdb entry trace (breakpoints on the glsi engine-init entry and its +inner functions, breakpoint set after `eglInitialize` so the vendor libraries +are mapped; ASLR disabled reproduces glsi base `0x7ffff7e2a000`) captured the +failing call order: + +``` +glsi+0x5f480 (engine-init) + glsi+0x1aba0 (first inner call) + glsi+0x22f30 device lookup by GPU id (miss -> 0xee00002) + glsi+0x2c360 object lookup (miss -> 0xee00003) + glsi+0x33b80 device-flags + state <-- the failing layer + glsi+0x301b0 (state machine, entered twice) + glsi+0x3a6f0 (entered ~7x) + glsi+0x2ac10 (entered twice) + -> EGL_CURRENT_FAIL err=0x3003 (eglcore rejects the engine) +``` + +`+0x5f480` returns as soon as `+0x1aba0` reports non-zero; `+0x1aba0` +propagates the first failure of `+0x22f30` / `+0x2c360` / `+0x33b80`. In the +trace all three were entered, and the failure lands inside `+0x33b80`'s +state-machine call. `+0x33b80` reads the device flags word at `+0xb84`; on a +state-machine failure with the retry bit set it marks the device busy +(`or 0x20` into the state record at `device + idx*0xdc8 + 0x291b4`) and +returns `0xee00001` once (`glsi+0x30ef2` inside its callee `+0x30b20`); +the higher layer reads `+0xb84` bit 0x20 and rejects the engine for this +attempt without retry. `+0x22f30` is called ~1300 times during the probe +(per-device/per-engine enumeration) - the failure is not a lookup miss. + +The state machine (`+0x301b0`) consumes per-engine context records; its +inputs are the fields the caller passes in (`+0x95` mode byte, `+0x38` +record array, `+0x74` id, `+0x98` index) plus the device flags. The next +step is a return-value trace of `+0x3a6f0`/`+0x2ac10` (their failure +conditions) or a differential state dump against a physical-GPU run. + +Note (same day): a control experiment was attempted - Xorg (GLX path) on +the same boot died in `Failed to initialize DMA` / `Failed to allocate push +buffer`, but that boot had a stalled CUDA client holding the device, so the +result is inconclusive; retry on a clean boot. + +## 5. Next steps (for the successor) + +1. Return-value trace of `glsi +0x33b80`'s callees (`+0x301b0`'s + `+0x3a6f0` / `+0x2ac10` results): the first non-zero return there is the + true failing layer. +2. Compare the glsi device-engine state with the same probe on a physical + Turing GPU (the host's own 570 driver runs the identical userspace; a + host-side `egl_frame` run with the real GPU succeeds, so a differential + dump of the engine records / the `+0xb84` flags word will show what the + emulated device is missing). +3. The engine records glsi consumes come from the kernel driver's device + data (engine table + class list). The CUDA path builds the analogous + records from `GET_ENGINES_V2`; the CUDA engine list on this guest is + known to be complete for GR + CE (see the private analysis area). The + glsi path may require an entry (e.g. a specific engine kind or a caps + bit) that the emulated device does not report. + +## 5. Reproduction + +``` +# in the guest (570.144 stock driver): +gcc -O2 -o /tmp/egl_frame /tmp/egl_frame.c -lEGL -lGLESv2 # tools/guest-diag/egl_frame.c +timeout 90 /tmp/egl_frame # EGL_CURRENT_FAIL err=0x3003 + +# under gdb (offsets stable, ASLR disabled): +gdb -q -batch -x egl-gdb11.txt ./egl_frame # catches the vtable call site +``` + +Breakpoints used (no-ASLR addresses with eglcore base 0x7ffff5200000, +glsi base 0x7ffff7e2a000): + +| address | meaning | +|---|---| +| `0x7ffff5bfe151` | eglcore rejection call `*0x30(vtable)` | +| `0x7ffff7e89480` | glsi engine-init entry (+0x5f480) | +| `0x7ffff7e894e1` | after glsi's first inner call (returns 0xee00001) | diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json index 4fe571b..8083321 100644 --- a/wiki/.publish-manifest.json +++ b/wiki/.publish-manifest.json @@ -19,11 +19,11 @@ }, "CUDA Userspace Driver": { "file": "CUDA-Userspace-Driver.wiki", - "new_revid": 23998, - "old_revid": 23963, - "source_sha": "d467dbcd9f323768", - "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", - "timestamp": "2026-10-20T09:19:03Z", + "new_revid": 24014, + "old_revid": 23998, + "source_sha": "8703a12fa11674f0", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:25Z", "title": "CUDA Userspace Driver" }, "Community": { @@ -35,13 +35,22 @@ "timestamp": "2026-10-12T04:40:27Z", "title": "Community" }, + "DirectX": { + "file": "DirectX.wiki", + "new_revid": 24118, + "old_revid": 24007, + "source_sha": "8701ac31d99d4619", + "summary": "Remove virtio-gpu/virgl references; they are not part of this project's path", + "timestamp": "2026-11-11T16:22:39Z", + "title": "DirectX" + }, "Documentation": { "file": "Documentation.wiki", - "new_revid": 23974, - "old_revid": 23930, - "source_sha": "28a5145b53d94c28", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:25Z", + "new_revid": 24031, + "old_revid": 24026, + "source_sha": "04de62980311a5e4", + "summary": "2026-10-30 wiki refresh: add Project Plan page (phases, gates, workstreams, next steps)", + "timestamp": "2026-10-30T17:12:05Z", "title": "Documentation" }, "Error Codes": { @@ -55,11 +64,11 @@ }, "FAQ": { "file": "FAQ.wiki", - "new_revid": 23968, - "old_revid": 23968, - "source_sha": "02b90d88b6c94c84", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:08Z", + "new_revid": 24109, + "old_revid": 24097, + "source_sha": "cf9501f4109622b6", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:54Z", "title": "FAQ" }, "GMMU Multi-Tenant Protection": { @@ -100,11 +109,11 @@ }, "GSP and GSP-RM": { "file": "GSP-and-GSP-RM.wiki", - "new_revid": 23977, - "old_revid": 23943, - "source_sha": "e9bd4bdc8de69c62", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:55Z", + "new_revid": 24010, + "old_revid": 23977, + "source_sha": "0019a098ff128830", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:19Z", "title": "GSP and GSP-RM" }, "Glossary": { @@ -116,31 +125,40 @@ "timestamp": "2026-10-12T04:41:01Z", "title": "Glossary" }, + "Graphics Execution": { + "file": "Graphics-Execution.wiki", + "new_revid": 24119, + "old_revid": 24110, + "source_sha": "284779126afc0832", + "summary": "Remove virtio-gpu/virgl references; they are not part of this project's path", + "timestamp": "2026-11-11T16:22:40Z", + "title": "Graphics Execution" + }, "Guest Driver Lifecycle: A Field Guide": { "file": "Driver-Lifecycle-Field-Guide.wiki", - "new_revid": 23964, + "new_revid": 24019, "old_revid": 23964, - "source_sha": "518c07ce20d3b840", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "source_sha": "f24d66bd99dce76f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:32Z", "title": "Guest Driver Lifecycle: A Field Guide" }, "HAAEEP and Collaboration": { "file": "HAAEEP-and-Collaboration.wiki", - "new_revid": 23979, - "old_revid": 23945, - "source_sha": "cd1d9dfd41b1e6b6", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:56Z", + "new_revid": 24023, + "old_revid": 23979, + "source_sha": "20990217e179badc", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:38Z", "title": "HAAEEP and Collaboration" }, "Hardware and Software Matrix": { "file": "Hardware-Software-Matrix.wiki", - "new_revid": 23980, - "old_revid": 23946, - "source_sha": "899c86d3ae612537", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:57Z", + "new_revid": 24022, + "old_revid": 23980, + "source_sha": "fc7ff074320feafb", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:37Z", "title": "Hardware and Software Matrix" }, "Install and First Run": { @@ -152,6 +170,15 @@ "timestamp": "2026-10-12T04:40:58Z", "title": "Install and First Run" }, + "Interfaces and Stability": { + "file": "Interfaces-and-Stability.wiki", + "new_revid": 24009, + "old_revid": null, + "source_sha": "5e16ac6f098129ce", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:17Z", + "title": "Interfaces and Stability" + }, "Introduction to GMMU": { "file": "Introduction-to-GMMU.wiki", "new_revid": 23983, @@ -163,20 +190,20 @@ }, "Main Page": { "file": "Main-Page.wiki", - "new_revid": 24000, - "old_revid": 23972, - "source_sha": "2e8171cab56c394c", - "summary": "Status table refresh 2026-10-20: guest CUDA kernel execution working", - "timestamp": "2026-10-20T09:19:09Z", + "new_revid": 24106, + "old_revid": 24094, + "source_sha": "255b09c45c397709", + "summary": "2026-11-10 afternoon refresh: first real GPU readback; direct-GL draw capture and window/screen present are the frontier", + "timestamp": "2026-11-10T21:41:07Z", "title": "Main Page" }, "MediaWiki:Sidebar": { "file": "MediaWiki-Sidebar.wiki", - "new_revid": 23976, - "old_revid": 23906, - "source_sha": "73777ab1026c6dc6", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:28Z", + "new_revid": 24032, + "old_revid": 24027, + "source_sha": "290bc8a24b8ab97c", + "summary": "2026-10-30 wiki refresh: add Project Plan to sidebar", + "timestamp": "2026-10-30T17:12:10Z", "title": "MediaWiki:Sidebar" }, "Migration Protocol": { @@ -190,29 +217,65 @@ }, "Open-IOV:About": { "file": "Open-IOV-About.wiki", - "new_revid": 23973, - "old_revid": 23929, - "source_sha": "2cba9bbf3ab93ecf", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:24Z", + "new_revid": 24028, + "old_revid": 23973, + "source_sha": "78307e495e4241ab", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:46Z", "title": "Open-IOV:About" }, + "OpenGL": { + "file": "OpenGL.wiki", + "new_revid": 24120, + "old_revid": 24112, + "source_sha": "d3c63c152bd202ce", + "summary": "Remove virtio-gpu/virgl references; they are not part of this project's path", + "timestamp": "2026-11-11T16:22:40Z", + "title": "OpenGL" + }, + "Performance": { + "file": "Performance.wiki", + "new_revid": 24008, + "old_revid": null, + "source_sha": "f2b44ad5e7ccaeb0", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:16Z", + "title": "Performance" + }, "Physical GPU Backend and Guest Wiring": { "file": "Physical-GPU-Backend.wiki", - "new_revid": 23999, - "old_revid": 23985, - "source_sha": "d762abd50cef636d", - "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", - "timestamp": "2026-10-20T09:19:05Z", + "new_revid": 24113, + "old_revid": 24101, + "source_sha": "045d4371c63d7f57", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:56Z", "title": "Physical GPU Backend and Guest Wiring" }, + "Project Goals": { + "file": "Project-Goals.wiki", + "new_revid": 24121, + "old_revid": 24114, + "source_sha": "4bc1d42ce7317922", + "summary": "Remove virtio-gpu/virgl references; they are not part of this project's path", + "timestamp": "2026-11-11T16:22:41Z", + "title": "Project Goals" + }, + "Project Plan": { + "file": "Project-Plan.wiki", + "new_revid": 24115, + "old_revid": 24103, + "source_sha": "5413cd31633f0781", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:57Z", + "title": "Project Plan" + }, "Project Status": { "file": "Project-Status.wiki", - "new_revid": 23997, - "old_revid": 23986, - "source_sha": "cb714adeebcfdc33", - "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", - "timestamp": "2026-10-20T09:19:01Z", + "new_revid": 24116, + "old_revid": 24104, + "source_sha": "cdbe587f95371ca5", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:58Z", "title": "Project Status" }, "RPC Message Format": { @@ -226,11 +289,11 @@ }, "System Overview": { "file": "System-Overview.wiki", - "new_revid": 23990, - "old_revid": 23956, - "source_sha": "f2658a051178f477", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:02Z", + "new_revid": 24117, + "old_revid": 24105, + "source_sha": "6d4d08fffb5ff6c0", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:58Z", "title": "System Overview" }, "Template:About message": { @@ -262,47 +325,47 @@ }, "The Execution Frontier": { "file": "Execution-Frontier.wiki", - "new_revid": 23996, - "old_revid": 23966, - "source_sha": "8c0007ec287387b9", - "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", - "timestamp": "2026-10-20T09:19:00Z", + "new_revid": 24107, + "old_revid": 24095, + "source_sha": "0a952f3fe56873b1", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:53Z", "title": "The Execution Frontier" }, "The Guest Stack and CUDASW": { "file": "Guest-Stack-and-CUDASW.wiki", - "new_revid": 23978, - "old_revid": 23944, - "source_sha": "40480b8806fd7de5", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:56Z", + "new_revid": 24018, + "old_revid": 23978, + "source_sha": "2683360ca15b4995", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:31Z", "title": "The Guest Stack and CUDASW" }, "The Host Module (vgsp.ko)": { "file": "Host-Module.wiki", - "new_revid": 23981, - "old_revid": 23947, - "source_sha": "ede82920d8920b8f", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:57Z", + "new_revid": 24111, + "old_revid": 24099, + "source_sha": "a92d0397166fb875", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:55Z", "title": "The Host Module (vgsp.ko)" }, "The vGSP Execution Plane (vEP)": { "file": "Execution-Plane.wiki", - "new_revid": 23967, - "old_revid": 23967, - "source_sha": "f2b7e219c1b3b18a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:08Z", + "new_revid": 24108, + "old_revid": 24096, + "source_sha": "758b224f3c94f02a", + "summary": "2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls", + "timestamp": "2026-11-11T15:48:54Z", "title": "The vGSP Execution Plane (vEP)" }, "UVM in the Guest": { "file": "UVM-in-the-Guest.wiki", - "new_revid": 23991, - "old_revid": 23957, - "source_sha": "46ea1e4d85c342e5", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:02Z", + "new_revid": 24020, + "old_revid": 23991, + "source_sha": "70774393afbe8e1f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:34Z", "title": "UVM in the Guest" }, "VGPU-IO Integration Guide": { @@ -314,6 +377,15 @@ "timestamp": "2026-10-12T04:41:03Z", "title": "VGPU-IO Integration Guide" }, + "Vulkan": { + "file": "Vulkan.wiki", + "new_revid": 24006, + "old_revid": null, + "source_sha": "d94fa2d2b85e4057", + "summary": "New page (2026-10-30 wiki refresh): Vulkan", + "timestamp": "2026-10-30T17:05:01Z", + "title": "Vulkan" + }, "WPR Management": { "file": "WPR-Management.wiki", "new_revid": 23992, @@ -325,11 +397,11 @@ }, "risc-v-gsp-rm-debugger": { "file": "Riscv-GSP-RM-Debugger.wiki", - "new_revid": 23989, - "old_revid": 23955, - "source_sha": "5482a72b81b0e4e6", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:01Z", + "new_revid": 24024, + "old_revid": 23989, + "source_sha": "fcd6dc67e49c8d1f", + "summary": "2026-10-30 wiki refresh: goals, GSP, graphics, performance, interfaces", + "timestamp": "2026-10-30T17:06:40Z", "title": "risc-v-gsp-rm-debugger" } }
\ No newline at end of file diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md index 6975a1d..c18a6ab 100644 --- a/wiki/CUDA-Userspace-Driver.md +++ b/wiki/CUDA-Userspace-Driver.md @@ -1,20 +1,23 @@ # CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive -> **Status (2026-10-20, honest).** The analysis covers the whole library for +> **Status (2026-10-30, honest).** The analysis covers the whole library for > driver 570.144 and the tooling is version-agnostic. The *control plane* gets > the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate` > and the full driver-API smoke test — allocation, copies, free and destroy > complete end to end (`ALL OK`; the 64-byte payload round-trips exactly). > The *data plane* decodes guest push buffers and executes the memory and > semaphore work it understands; CUDA kernel launches are forwarded to the -> physical execution path and now execute on the physical GPU — the +> physical execution path and execute on the physical GPU — the > engine-authored completions relay to the guest and its synchronisation > waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply > (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, -> `FFN4 layers=4 bad=0`) clients verify green on fresh boots. The -> large-buffer staged-transfer class (multi-megabyte transfers between guest -> memory and device memory) is under repair, and reproducibility across fresh -> boots is being re-measured after the boot-poisoning fix. Canonical detail: +> `FFN4 layers=4 bad=0`) clients verify green on fresh boots, and a +> ring-block ownership defect that could stall the first client after a +> reboot was root-caused and fixed. The staged-transfer class (multi-megabyte +> transfers between guest memory and device memory) is under repair with its +> failure modes proven. Graphics (OpenGL/Vulkan) shares this driver's object +> machinery on a parallel frontier +> ([`Graphics-Execution.md`](Graphics-Execution.md)). Canonical detail: > [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) > through `08`; reading order > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); @@ -178,11 +181,10 @@ the data plane (`docs/08`). smoke test (allocation, copies, free and destroy). * **Current frontier:** guest CUDA kernel launches execute on the physical GPU with engine-authored completions relayed to the guest; the verified - clients are green on fresh boots. The leading edge is the large-buffer - staged-transfer class (some large multi-chunk clients still mis-verify), - re-measurement of reproducibility across fresh boots after the - boot-poisoning fix, and per-guest execution isolation. The chain, the - evidence and the live state are on + clients are green on fresh boots. The leading edge is the staged-transfer + class (some large multi-chunk clients still mis-verify, with the failure + modes proven), the every-boot acceptance campaign, and per-guest execution + isolation. The chain, the evidence and the live state are on [`Execution-Frontier.md`](Execution-Frontier.md). * **Data plane today:** the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands diff --git a/wiki/DirectX.md b/wiki/DirectX.md new file mode 100644 index 0000000..85faca6 --- /dev/null +++ b/wiki/DirectX.md @@ -0,0 +1,100 @@ +# DirectX + +> **Scope.** The Windows guest path — Direct3D and the rest of the DirectX +> family — as a **project target**: how the Windows NVIDIA stack relates to +> the interface Open-IOV already emulates for Linux, what a bring-up would +> require, and what has *not* happened yet (no Windows guest has been booted +> on the emulated vGPU). Linux graphics is covered on +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and +> [`Graphics-Execution.md`](Graphics-Execution.md). Canonical sources: +> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) +> (guest stack) and +> [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) +> (support ladder). The docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[target]` designed for, +bring-up not started; `[planned]` roadmap; no Windows-specific code exists in +the project today. + +## 1. The Windows graphics stack + +```text +Direct3D application (D3D12 / D3D11 / D3D9) or CUDA application + -> DXGI (swap chains, adapters, presentation) + -> user-mode display driver (UMD): nvd3dumx / nvwgf2umx [D3D] + -> kernel-mode driver (KMD): nvlddmkm.sys, under WDDM + -> GPU +Windows kernel graphics subsystem (dxgkrnl) + -> GPU scheduler, video-memory manager (VidMm), residency, TDR +``` + +Two structural differences from Linux matter for this project: + +* **WDDM owns more of the pipeline.** The OS schedules command buffers + submitted by the UMD, manages video-memory residency, and runs the GPU + scheduler in the kernel; the driver's KMD programs hardware contexts and + fences beneath that. (D3D12 exposes queues and fences with an explicit + model that is closer to Vulkan than older APIs were.) +* **The device bring-up is different.** A Windows display adapter starts + through the OS's display/PDO path (adapter enumeration, ROM/VBIOS views, + VIDPN topology, power transitions), not the Linux DRM/modeset path. On + Turing and later, however, the driver still talks to the **same GSP + firmware resource manager** — the interface `vgsp.ko` emulates. + +CUDA on Windows rides the same KMD with its own userspace driver, so the +device-facing requirements overlap almost entirely with the Linux CUDA work +already proven ([`Execution-Frontier.md`](Execution-Frontier.md)). + +## 2. What this means for Open-IOV + +The project's architecture does not change for Windows: **emulate the +firmware interface, pass the work through, patch nobody.** The Windows KMD +would be stock (signed, as shipped); the emulation must satisfy its +device-view and its GSP-RM usage. The measured bring-up items, from +[`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md): + +* WDDM device start on the emulated device (PDO/display path, MSI-X and + interrupt semantics). +* ROM/VBIOS provisioning in the device profile (Windows adapters expect one). +* Power-management transitions (Dx states) and reset behaviour. +* D3D/DXGI device creation, then compute (CUDA on Windows), then rendering + through the same execution lanes as Linux. + +## 3. Status + +| Item | State | +|---|---| +| Windows guest boots on the emulated vGPU | `[planned]` — not started | +| KMD (`nvlddmkm.sys`) initialises on the emulated device | `[target]` | +| CUDA on Windows (driver-API smoke) | `[target]` | +| Direct3D device creation / rendering | `[target]` | +| Multi-VM Windows guests | `[target]` | + +The support matrix lists **Windows x86_64 (WDDM driver, CUDA, 3D)** as +**Target**: the architecture is designed for it; the bring-up is ahead. The +proposed ladder mirrors the Linux milestones: KMD initialises → adapter +enumeration healthy → CUDA-on-Windows driver-API smoke → D3D device creation +→ first compute execution → first rendered pixels → multiple VMs. + +## 4. What the project will not do + +* No Windows-specific code inside the guest — no patched `nvlddmkm`, no + shims in the guest kernel, no guest agent. Any Windows-only quirk belongs + in the host module's device model. +* No reliance on the vendor vGPU host stack; the same mediator answers both + guest OS families. +* The two-VM GPU demo videos published so far used **Linux** guests; neither + is DirectX. + +## 5. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the execution lanes + Direct3D would use. +* [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) — the Linux API paths, + further along. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the support + ladder, including the Windows target row. +* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md) — §7 covers + the Windows kernel-mode driver and the no-guest-code rule. +* [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) — + the canonical matrix and what each support level means. diff --git a/wiki/Driver-Lifecycle-Field-Guide.md b/wiki/Driver-Lifecycle-Field-Guide.md index 1115be9..9d3fde8 100644 --- a/wiki/Driver-Lifecycle-Field-Guide.md +++ b/wiki/Driver-Lifecycle-Field-Guide.md @@ -5,7 +5,7 @@ > unmodified 570.144 guest. Canonical source: > [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md); if this page and the docs tree disagree, the docs tree wins. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival. +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival. ## 1. How to read this page @@ -107,7 +107,7 @@ vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject **Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. **Emulation today:** the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet. -**Progress:** context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted `CTXNOTVALID`, and a following `GPU_PROMOTE_CTX` wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [`Execution-Frontier.md`](Execution-Frontier.md). The roadmap from here is: +**Progress:** context creation returns (the driver-API smoke completes end to end) and guest CUDA kernel launches now execute on the physical engine with engine-authored completions relayed to the guest — verified green clients on fresh boots. Earlier walls are closed: the host GR ladder's `CTXNOTVALID` fault and the `GPU_PROMOTE_CTX` wedge led to the commit-chain work, and the forwarded guest launch that once reached the front-end without a dispatched grid now executes end to end. The current leading edge is the staged-transfer class, the every-boot acceptance campaign, per-guest isolation, and the graphics pixel frontier. See [`Execution-Frontier.md`](Execution-Frontier.md) and [`Graphics-Execution.md`](Graphics-Execution.md). The roadmap from here is: ```text UVM registers → cuInit returns → context creation → memory allocation diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index 2d9d80c..fc076f1 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -4,13 +4,17 @@ > must fire for a stock guest CUDA kernel to run on the physical GPU, what has > been proven about it, how we know, and where the leading edge is now. The > chain fires end to end: verified guest CUDA clients execute on the physical -> GPU and wake up correctly; the leading edge is now breadth and hardening. It +> GPU and wake up correctly; the leading edge is now breadth and hardening. A +> **parallel graphics frontier** (OpenGL/Vulkan execution) shares the same +> submission machinery at a different execution layer — +> [`Graphics-Execution.md`](Graphics-Execution.md), +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md). It > is evidence-led: every statement carries a live observation, a > decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-20, tree `master @ 16291c72`. +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -72,21 +76,45 @@ synchronisation waits resolve. The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it: -* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers - between guest memory and device memory are under repair: some large - multi-chunk clients still mis-verify. -* **Reproducibility across boots.** A boot-poisoning defect was root-caused - and fixed: under the driver's status-queue mode the emulation read the wrong - queue cursor, so under load the reply ring could desynchronise and the first - CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). - Re-measurement across fresh boots is in progress. +* **The staged-transfer class.** Multi-megabyte transfers between guest memory + and device memory are under repair: some large multi-chunk clients still + mis-verify, with the failure modes proven (a release applied before the op + it owes, a ledger-vs-writer page divergence, and a readback racing a + writeback). +* **Reproducibility across boots.** Two defects landed and were fixed: the + boot-poisoning defect (under the driver's status-queue mode the emulation + read the wrong queue cursor, so under load the reply ring could + desynchronise and the first CUDA client would fail) and a ring-block + ownership defect that could stall the first client after a reboot. Green + clients reproduce across fresh boots; a formal every-boot campaign remains. * **Per-guest execution isolation.** Multiple mediated devices with different - emulated VRAM sizes and multiple guests already coexist at the device level; - per-guest execution isolation for multiple simultaneous CUDA guests is in - progress. - -No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration -and performance numbers remain planned. + emulated VRAM sizes and multiple guests already coexist at the device level + (two guests demonstrated); per-guest execution isolation for multiple + simultaneous CUDA guests is in progress. + +### A parallel frontier: 3D graphics + +Graphics execution shares this page's submission and completion machinery but +runs a different engine workload. The display and GL stacks now run end to +end in the guest: the emulated display engine serves the modeset push buffers, +nvidia-modeset/NvKms registers (`/dev/dri`), X settles, `glxinfo` reports +OpenGL 4.6.0 with direct rendering, and the X DDX waits are fixed +(class-aware display arm, `0x8000 -> 0` notification completion, decoded GL +release applied); the direct GL client's GR and CE channels submit and retire, +and the X DDX kick channel is decoded. The **first non-black GPU-rendered +output** landed: a pbuffer probe returns `pixel=ff0000ff`, the off-screen +path produced 500/500 animated 640x480 frames, and a 45.4 s video was +captured. The direct-GL **GR ring `0xbeef0100` now resolves** and its releases +advance; GZDoom reaches `V_Init: allocate screen` with its GL 4.6.0 context +whenever the shared nvClip `modeNum` is even. What is missing is **window/ +screen pixels through the NVIDIA path**: glxgears' window is black, the +direct-GL **CE ring `0xbeef0101` never resolves** (the guest wedges ~90 s +after `V_Init`), and X window creation wedges on some boots. The guest Vulkan +pipeline's inline-to-memory upload arm is its gap. The full picture lives on +[`Graphics-Execution.md`](Graphics-Execution.md). + +No breadth beyond these is claimed: Windows guests, migration and performance +numbers remain planned. ## 3. How we know (observables and their limits) @@ -125,31 +153,47 @@ boots. ## 5. The clear issues (ranked) -**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged -transfers between guest memory and device memory are under repair: some large -multi-chunk clients still mis-verify. +**Issue A — the staged-transfer class.** Multi-megabyte transfers between +guest memory and device memory are under repair: some large multi-chunk +clients still mis-verify, with the failure modes proven and named. -**Issue B — reproducibility across boots (re-measurement).** The -boot-poisoning defect (wrong status-queue consumption cursor under the -driver's status-queue mode; the reply ring could desynchronise under load, the -first client would fail) was root-caused and fixed. Re-measurement across -fresh boots is in progress; no "every boot" claim is made until it completes. +**Issue B — every-boot reproducibility (acceptance campaign).** The +boot-poisoning defect and the post-reboot ring-block ownership defect were +root-caused and fixed, and green clients reproduce on fresh boots; the formal +every-boot acceptance campaign is the remaining work. **Issue C — per-guest execution isolation.** Multiple mediated devices with -different emulated VRAM sizes and multiple guests coexist at the device level; -per-guest execution isolation for simultaneous CUDA guests is in progress. +different emulated VRAM sizes and multiple guests coexist at the device level +(two guests demonstrated); per-guest execution isolation for simultaneous +CUDA guests is in progress. + +**Issue D — the graphics pixel frontier.** The display/KMS stack and the GL +client now run end to end (NvKms registers, X settles, GL 4.6, the DDX 3D +replay executes, the first non-black GPU-rendered frames and a 45.4 s video +are captured off-screen, and the direct-GL GR ring resolves with advancing +releases); but the direct-GL CE ring `0xbeef0101` does not resolve, a +window draw wedges X and glxgears' window is black, and no non-black window/ +screen frame has been produced. The CE ring, the window/screen present and X +stability are the named blockers +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 6. Lines of investigation (live) -1. **Repair the large-buffer staged-transfer class** — the three green clients +1. **Repair the staged-transfer class** — the green clients stay the regression battery and must remain green throughout. -2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot - path across fresh boots before claiming boot-to-boot reproducibility. +2. **Complete the every-boot acceptance campaign** — demonstrate the fixed boot + path across a defined run of fresh boots before claiming boot-to-boot + reproducibility. 3. **Per-guest execution isolation** — build from the device-level coexistence to isolation for simultaneous CUDA guests. 4. **Hold the acceptance battery** — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (`bad=0` + `STATUS OK`). +5. **The graphics pixel frontier** — resolve the direct-GL CE ring + `0xbeef0101`, make the window/screen present land, and stabilise X across + boots so the proven off-screen frames become window pixels; the lanes, the + walls and the milestone ladder are on + [`Graphics-Execution.md`](Graphics-Execution.md). ## 7. Theories, resolved @@ -157,8 +201,9 @@ The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is -engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest -isolation), each with an acceptance signal named before the work. +engineering (staged transfers, an every-boot acceptance campaign, per-guest +isolation, the graphics pixel frontier), each with an acceptance signal named +before the work. ## 8. Claim discipline @@ -183,6 +228,8 @@ isolation), each with an acceptance signal named before the work. client path. * [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work is forwarded to. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the parallel graphics + frontier (OpenGL/Vulkan), sharing this machinery. * [`Execution-Plane.md`](Execution-Plane.md) — the execution layer's design. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page tables, QMDs and the engines. diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md index 5d1052f..d88a1f3 100644 --- a/wiki/Execution-Plane.md +++ b/wiki/Execution-Plane.md @@ -8,7 +8,7 @@ > when the two disagree, the docs tree wins. Front door: > [`../docs/README.md`](../docs/README.md). -Status: 2026-10-08, tree `master @ 1f571dda`. Tags: `[live]` proven with the +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or parameter-gated; `[wip]` partial; `[wall]` current blocker; `[target]` named success latch; `[planned]` designed, not built. @@ -37,7 +37,7 @@ would collide with existing project vocabulary. | Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request | | Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow | | Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix | -| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned | +| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; two guests with different emulated VRAM sizes coexist at the device level, per-guest execution isolation is in progress | ## 2. The shape of the plane @@ -93,18 +93,19 @@ guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to - the ladder channel's GPFIFO and rings the real doorbell. `[gated]` - default-off; latched on a proven real GR submission. -5. **Engine.** The front-end fetches the entry and executes methods; then the - **inline-QMD install** must materialise the descriptor for the engine to - read, currently **`[wall]`** — not observed. Context validity, destination - mapping and fetch/alignment are exonerated; the recovered-method-stream - form is the investigation. + the ladder channel's GPFIFO and rings the real doorbell. `[live]` — + exercised on every verified guest launch (default-off while the seams are + validated). +5. **Engine.** The front-end fetches the entry and executes methods; the + inline launch descriptor installs and the engine reads it, the grid + dispatches to the SMs, and the engine authors the release. `[live]` — + context validity, destination mapping and fetch/alignment were exonerated + on the way. 6. **Completion.** The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X - wakes the guest. CE work is live in this shape `[gated]`; for GR the - completion-counter publication is the current second face of the wall. + wakes the guest. `[live]` for CE and GR: engine-authored completions relay + to the guest and its synchronisation waits resolve. ## 4. How work is found: discovery, truth, completion @@ -129,27 +130,34 @@ guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) ## 5. Challenges, honestly -* **GR dispatch wall.** The forward path runs front-end methods, the context - is valid and the destination window is accepted — and the descriptor - install is still not observed. The mode matrix carries dozens of - single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory - block, lane controls, shadow windows, pointer relocation). -* **Completion publication.** A boot where GR rings were discovered and - releases applied still saw the guest re-queue the same push and give up. A - later `sema_dbg=Y` run settled it: the release was written to the exact - slot, the notifier cleared and the event posted, while the completion - counter publication never wrote because the fixed-total autoreg mapping - makes it a structural no-op. The leading reading is the counter-slot - publication; the GP_GET write-back is the secondary, still-open candidate. - Both are debug-gated today. -* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The - first fix wave landed and the earlier CE freeze is gone; the stall moved to - GR ring discovery, which the latest discovery work addresses. Acceptance - is 10 cold boots green, pending. -* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest - UVM because host per-client sema/GR state is not reset when the guest - re-runs its per-client boot handshake. Per-client state containers and - reset are the designed path. +* **The staged-transfer class (compute).** Some large multi-chunk CUDA clients + still mis-verify; the failure modes are proven (a release applied before + the op it owes, a ledger-vs-writer page divergence, a readback racing a + writeback) and the fixes are in flight. +* **Every-boot reproducibility.** The boot-poisoning defect (wrong + status-queue consumption cursor under the driver's swap mode) and the + post-reboot ring-block ownership defect were root-caused and fixed; green + clients reproduce across fresh boots, and the formal every-boot acceptance + campaign is the remaining work. +* **Multiple clients and per-guest isolation.** Two guests with different + emulated VRAM sizes coexist at the device level; per-guest execution + isolation for simultaneous CUDA guests is in progress. +* **The graphics pixel frontier.** Graphics work rides this plane in three + shapes (verbatim 3D method-stream replay, synthesized launches, and the + direct GL client's own GR/CE channels, which now submit and retire). The + display/KMS path is served by the emulated display engine (modeset push + buffers plus the notifier/`UPDATE` completion, class-aware arm), so NvKms + registers, X settles and GL 4.6 is live, the X DDX waits are fixed, and + the guest's real GL draws now forward and execute on the physical GPU + (`backend_ok=5-7`, `backend_timeout=0`); the DDX 3D replay executes, the 2D + path is green, the first non-black GPU-rendered frames are proven (500/500 + off-screen frames plus a 45.4 s video), and the direct-GL **GR ring + `0xbeef0100` resolves with advancing releases**. No non-black **window/ + screen** frame has been produced through the NVIDIA path yet: the direct-GL + **CE ring `0xbeef0101` never resolves** (the guest wedges ~90 s after + `V_Init`), glxgears' window is black, and X window creation wedges on some + boots, so those are the named blockers. The lanes and the ladder are on + [`Graphics-Execution.md`](Graphics-Execution.md). ## 6. Portability @@ -164,15 +172,17 @@ human approval gate. ## 7. Future -The planned next steps are reproducibility acceptance, the descriptor-install -and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger -where possible), the arbitrary-kernel acceptance suite, then the near-native -performance program. The unsolved future classes — multi-tenant fairness, -fault replay/cancel, context switching between tenants, hostile-guest +The planned next steps are the staged-transfer fixes, the every-boot +acceptance campaign, per-guest isolation, and the graphics pixel frontier +(offline-first with the RISC-V GSP-RM debugger where possible), then the +near-native performance program. The unsolved future classes — multi-tenant +fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction in the -canonical document. 3D graphics reuses this pipeline once compute dispatch is -proven; it adds engine breadth (video/image), more channels and synchronisation -volume, and a much larger method vocabulary, but not a new plane. +canonical document. 3D graphics reuses this pipeline and now executes across +three shapes, including the direct GL client's own GR/CE channels +([`Graphics-Execution.md`](Graphics-Execution.md)); it adds engine +breadth (video/image), more channels and synchronisation volume, and a much +larger method vocabulary, but not a new plane. --- diff --git a/wiki/FAQ.md b/wiki/FAQ.md index 9472740..d3e42db 100644 --- a/wiki/FAQ.md +++ b/wiki/FAQ.md @@ -5,7 +5,7 @@ > [`../docs/09-reference/glossary-and-faq.md`](../docs/09-reference/glossary-and-faq.md); > the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. +Status: 2026-11-11, tree `master @ 4acac093`. ## Project and positioning @@ -27,26 +27,44 @@ not for the QEMU/emulated path ([`Install-and-First-Run.md`](Install-and-First-Run.md) §2). **Why is `nvidia-smi` working but my CUDA kernel not running?** -Because execution is the frontier, not the setup. Today, a stock guest driver -probes, `nvidia-smi` works, device enumeration works, and the full CUDA -**driver-API** path (context creation, allocation, copies, free, destroy) -completes; a guest **kernel launch** is forwarded to the physical engine queue -but the engine does not yet execute the forwarded launch descriptor -(`[wall]`). This is the project's known status, not a misconfiguration — see +It probably is running: guest CUDA kernels execute on the physical GPU and +verify — the vector-add, matrix-multiply and transformer clients are green on +fresh boots. If a specific workload fails, the known classes are: multi-megabyte +staged transfers between guest and device memory (under repair, failure modes +proven), and simultaneous CUDA guests (per-guest execution isolation is in +progress). This is the project's known status, not a misconfiguration — see [`Project-Status.md`](Project-Status.md) and [`Execution-Frontier.md`](Execution-Frontier.md). +**Can I run OpenGL or Vulkan applications in the guest?** +Partially, and it is an active frontier. The display/KMS stack and the X +server run on the emulated vGPU with the stock driver: nvidia-modeset/NvKms +registers (`/dev/dri`), `glxinfo` reports OpenGL 4.6.0 with direct rendering, +the 2D path is green and the DDX 3D replay executes. The **first non-black +GPU-rendered output** is proven: a pbuffer probe returns `pixel=ff0000ff`, the +off-screen path produced 500/500 non-black animated frames, and a 45.4 s video +was captured. The direct-GL GR channel now submits and its releases advance; +GZDoom reaches `V_Init: allocate screen` with its GL 4.6.0 context. Window and +screen pixels are still the wall: glxgears' window is black, the direct-GL CE +channel does not yet resolve (GZDoom wedges ~90 s after `V_Init`), X window +creation wedges on some boots, and the DDX's 4 GB sysmem allocation forces a +12 GB guest. The guest Vulkan pipeline runs end to end (device → command +buffer → fence → readback) but frames are black. See +[`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and +[`Graphics-Execution.md`](Graphics-Execution.md). + **How fast is it?** -No published numbers yet. Copy-engine-class forwarding and its completion and -fault relay are implemented (default off) and GR execution is the open -frontier; even the initialization path has not yet been measured under the -project's methodology. Any performance claim will ship together with its -methodology and measurements. +No published numbers yet. Kernel launches execute on the physical engine; +what remains unmeasured is the project's own methodology — nothing ships as a +performance claim without its measurements. The first scheduled measurement +is the graphics target (60 fps sustained in one VM, no forcing). ## Compatibility **Will it support Windows guests?** -Yes, it is a target: same architecture, measured bring-up. +Yes, it is a target: same architecture, measured bring-up. No Windows guest +has been booted on the emulated vGPU yet; the Windows/DirectX path (WDDM, +`nvlddmkm`, Direct3D) is described on [`DirectX.md`](DirectX.md). [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) lists the support ladder. diff --git a/wiki/GSP-and-GSP-RM.md b/wiki/GSP-and-GSP-RM.md index c714612..651259b 100644 --- a/wiki/GSP-and-GSP-RM.md +++ b/wiki/GSP-and-GSP-RM.md @@ -7,7 +7,7 @@ > [`../docs/03-architecture/03-gsp-and-gsp-rm.md`](../docs/03-architecture/03-gsp-and-gsp-rm.md); > the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` designed end state. @@ -40,6 +40,26 @@ it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an **RM client**: it issues requests, receives callbacks, and observes state through a well-defined protocol. +### What the GSP does on bare metal + +* **It is the device's manager processor.** Since Turing, the driver does not + program most of the GPU. The GSP runs NVIDIA's signed firmware — a small + RTOS plus the RM server — and owns GPU-global state: initialisation, the + memory-management unit's deeper page-table levels, engine configuration, + context scheduling, interrupts and power/clock management (PMU). The + driver's kernel RM boots it, then becomes its client. +* **It boots from driver-provided memory.** At power-on the driver carves a + firmware-protected region of video memory (WPR2), copies the firmware image + there, has the device verify signatures (FWSEC), hands control to the boot + ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and + RM initialisation and signals readiness. Everything the driver does + afterwards is a request to that server. +* **Why this matters for virtualization.** Because the management path is a + *firmware interface* — shared-memory queues plus a small RPC protocol, + not a pile of register lore — it is small, stable, and observable. That is + the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is + practical at all. + ## 2. The boot handshake we emulate The guest driver boots the GSP exactly as it would on bare metal: @@ -130,7 +150,32 @@ vector. The emulation's MSI-X path is emulation thread → `eventfd` → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost. -## 7. What we emulate today +## 7. How Open-IOV deals with the GSP — two modes + +The project handles the GSP in two ways, and they complement each other: + +1. **Guest-facing GSP emulation** (`vgsp.ko` *is* the virtual GSP). For each + mediated device, the module presents a device whose GSP behaviour is + software: it implements the boot state machine the driver polls (stage + completion, not silicon), answers the WPR/BAR0 register protocol, serves + the message-queue transport and every RPC verb the driver uses during + probe, initialisation and runtime, and delivers events via emulated MSI-X. + The guest's RM believes it is talking to firmware; it is talking to the + emulation. +2. **The real GSP as the backend server.** When the module owns the physical + GPU, it loads the *matching real firmware image* into the device, runs the + actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM + client itself — "be the host driver's RM" becomes "be the RM client". The + same protocol the guest uses one layer up is used here one layer down, and + it is what the work path submits through + ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). + +Keeping the two in sync is a project discipline: what the guest's RM is +promised (objects, controls, page tables, completions) must be answerable by +the emulation, and what executes on hardware must round-trip through the real +firmware session. + +### What is emulated today | Layer | State | Notes | |---|---|---| @@ -140,9 +185,10 @@ interrupt is not lost. | Static info tables | `[wip]` | device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list | | Controls during init | `[wip]` | ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving | | Object creation | `[wip]` | handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet | -| Page tables for server-owned ranges | `[wip]` | contract received and recorded; leaf population is the execution work | -| Event/notification delivery | `[wip]` | MSI-X works; richer event semantics land with forwarding | -| Firmware behaviour (routing, scheduling, real execution) | `[wall]` | the execution frontier — see [`Execution-Frontier.md`](Execution-Frontier.md) | +| Page tables for server-owned ranges | `[wip]` | contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work | +| Event/notification delivery | `[live]` | MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest | +| Real GSP as the backend server | `[live]` | the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it | +| Firmware behaviour (routing, scheduling, real execution) | `[live]` | forwarded guest CUDA launches and graphics method streams execute on the physical engines — see [`Execution-Frontier.md`](Execution-Frontier.md) | ## 8. Where the firmware knowledge lives diff --git a/wiki/Graphics-Execution.md b/wiki/Graphics-Execution.md new file mode 100644 index 0000000..eea0574 --- /dev/null +++ b/wiki/Graphics-Execution.md @@ -0,0 +1,245 @@ +# Graphics Execution + +> **Scope.** How the emulated vGPU runs **3D graphics** work — the OpenGL and +> Vulkan command streams a stock guest submits — on the physical GPU, and what +> is proven about it. This is the shared machinery behind +> [`OpenGL.md`](OpenGL.md), [`Vulkan.md`](Vulkan.md) and the Windows/DirectX +> target ([`DirectX.md`](DirectX.md)); CUDA compute uses the same submission and +> completion path for a different engine workload +> ([`Execution-Frontier.md`](Execution-Frontier.md)). Canonical sources: +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) +> (the plan and milestone ladder) and +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> (the measured walls); the docs tree wins on disagreement. + +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` observed end to +end; `[wip]` partially built; `[wall]` current blocker; `[target]` defined +success state; `[planned]` designed, not built. + +## 1. What "real pixels" means + +The acceptance target is strict: a **stock** guest GL or Vulkan application's +draw commands are executed by the physical GPU's fixed-function 3D pipeline +(vertex fetch, primitive assembly, rasterisation, fragment work, ROP), the +render target is readable by the guest, and that image reaches the capture +path through the emulated mdev. The pixels must not come from another GPU, a +software rasteriser, or a host display round-trip. + +## 2. The path a graphics command takes + +```mermaid +graph TD + APP["Guest applications<br/>OpenGL · Vulkan · CUDA"] --> LIB["Guest user space<br/>GL: GLX client + the driver's GL core<br/>Vulkan: the NVIDIA ICD<br/>CUDA: libcuda"] + APP --> X["X server (Linux GL)<br/>GLX module + the NVIDIA DDX"] + LIB --> KMD["Guest kernel: nvidia.ko<br/>RM core, GSP client, channels, modeset/drm"] + X --> KMD + KMD -- "GPFIFO rings + doorbell;<br/>method streams in push buffers" --> VG["Host module vgsp.ko<br/>device model + emulation core"] + VG --> L1["Lane 1: verbatim 3D<br/>method-stream replay"] + VG --> L2["Lane 2: synthesized launch<br/>from a captured section"] + VG --> L3["Lane 3: forwarded CUDA<br/>launch (compute)"] + L1 --> FE["Physical GPU<br/>front-end fetch -> GR engine"] + L2 --> FE + L3 --> FE + FE -- "engine-authored releases<br/>(semaphores in guest memory)" --> VG + VG -- "completion relay: MSI-X +<br/>poller/event wake the guest" --> KMD +``` + +```text +guest apps (GL / Vulkan / CUDA) + -> guest user space (GLX client + GL core, Vulkan ICD, libcuda) + -> guest X server (GLX module + NVIDIA DDX) [OpenGL] + -> nvidia.ko: channels, GPFIFO rings, doorbells [all APIs] + -> vgsp.ko: capture sections, decode pushes, forward + -> physical GPU front-end -> GR engine (3D + compute) / copy engines + <- engine-authored releases written into guest memory + <- completion relay (MSI-X + wakes) back to the guest +ASCII fallback above; the diagram shows the three execution lanes. +``` + +## 3. The three execution lanes + +Graphics work reaches the engines today in one of three shapes, in increasing +fidelity: + +1. **Verbatim 3D method-stream replay** `[live]`. The emulation captures the + X server's graphics-class (`0xC597`, `TURING_A`) section from its channel, + relocates the operand addresses into host-built shadow pages the engine can + reach, and submits the stream unchanged. The engine retires it: the uploads + (`LINE_LENGTH_IN`/`LAUNCH_DMA`), the state, program and texture binds, and + the release all execute with no MMU fault. This proves the *mechanics* of + 3D execution — object bind, window placement, operand relocation, + completion — on real hardware. +2. **Synthesized launch from a captured section** `[live]`. A compute-shaped + launch built from the captured section (operand list, program, a 64-word + launch descriptor) reaches the engine and retires real `sm_75` SASS; the + Turing shader-program-header form (`V3`) is accepted with zero header + faults across repeated launches. This proved engine-side dispatch before + the verbatim lane existed. The program itself is still a development + stand-in at this rung — execution proof, not pixels. +3. **Forwarding the client's own pushes** `[wip]`. The X DDX channel + `0x1000014` executes its 3D replay (12 launches per cycle), the 2D path is + green, and a guest pbuffer `glClear(red)+glFinish+glReadPixels` probe + returns a real pixel (`pixel=ff0000ff`): the replay and readback machinery + works. The off-screen frame path then produced **500/500 non-black 640x480 + animated frames** and a captured 45.4 s video. The direct-GL **GR ring + `0xbeef0100` now resolves** (`4acac093`: beef-family always-probe, + per-channel promote backoff, rank-collision fallback, parked-candidate + eviction); its releases advance frame by frame and GZDoom reaches + `V_Init: allocate screen` with its GL 4.6.0 context. The direct-GL **CE + ring `0xbeef0101` never resolves**, so the client's CE work does not + execute and the guest wedges ~90 s after `V_Init`; window/screen present + is still the frontier (§6). + +## 4. The channel and object picture + +* **Every submission lives on a channel**: a GPFIFO ring plus engine objects, + with a USERD the driver advances its put pointer in and a doorbell the host + module traps. Channels are typed by engine: a graphics channel carries 3D + state and draws; copy-class channels carry memory movements. +* **The graphics object class** the guest's 3D stream binds is the Turing + graphics class, `0xC597`. Streams now come from both the **X server's + channel** and the **direct GL client's own channels** (`0xbeef0100` GR plus + the `0xbeef0101`/`0xbeef0102` CE siblings); the 2D present/readback work + rides lighter 2D-class and copy-engine shapes. +* **Display and modeset** `[live]`. The guest's modeset path is served by an + emulated display engine: the display push-buffer channels (core `0xc57d`, + window `0xc57e`, immediate/window `0xc57b`) are registered, their push + buffers parsed, and the notifier/`UPDATE` completion authored. The arm is + class-aware (the window class has no NOTIFY bit; a bound notifier is the + arm), and the X DDX's `0x8000 -> 0` notification completion is authored. + This is what lets nvidia-modeset/NvKms register (`/dev/dri`), X settle and + GLX clients obtain a real GL context. The display-capability seam is + parameterised in the campaign configuration. +* **Cells of state**: 3D streams are long—hundreds of method words per draw — + and mix state binds (viewport, render target, ZS, blend/ROP), program and + constant binds, vertex/index streams and draw-issuing methods. The capture + machinery recognises the init-shaped operands today; the per-draw state + vocabulary is being extended ([G-B in the plan]( + ../docs/04-implementation/05-graphics-execution-plan.md)). +* **Synchronisation**: tracking semaphores, notifications and fence values + written by the engine; the guest's swap/present waits sit on top of those. + +## 5. Completion, fences and presents + +The acceptance signal is the **engine-authored release**: a semaphore the +engine writes when the work completes. For GL, completion flows through the +channel's semaphores/notifications to the X server and the app's swap wait; +for Vulkan, the client's fence value is released; for CUDA, the launch's +release group wakes the client. The emulation relays engine completions into +guest-observable memory (MSI-X plus a poller), and the guest resumes from +there. + +For the display path, completion is the notifier write and `UPDATE -> +FINISHED`; a completion event deferred by the reply path is force-retried +until it posts, so the X server's mode set finishes. For the X DDX's own 2D +push buffers, completion also requires the GPU-written engine get word to +advance and the notifier to be recycled to `0xffff` after a kick; both are +host-side levers now, and a diagnostic write of the get word released the +DDX's reserve loop. For a windowed GL client the present is a kick; the GSP +event-IRQ fix now sets the RISC-V SWGEN0 IRQSTAT/IRQMASK/IRQDEST and latches +the CPU-tree leaf, so the guest services the event and glxgears completes +swaps, and the GR backend no longer times out (the pair-close/pair-safe +replay fixes are default ON; draws retire in 3-2400 ms). The DDX 3D replay +renders and reads back a pbuffer pixel (`pixel=ff0000ff`), the off-screen +frame path produced non-black animated frames and a 45.4 s video, the +RM write-lock holder (Xorg in the DDX vidheap scrub wait) is fixed +(`df6e71bb`), and the direct-GL GR ring now resolves with advancing releases. +The next step is the direct-GL CE ring `0xbeef0101` and the window/screen +present path. + +During bring-up the module also carries a **diagnostics-only mirror** for apps +that spin on a shared progress word; it exists so a run can proceed while the +natural completion path is being finished. The 60-fps milestone explicitly +requires it switched off, along with any external poke. + +## 6. Where the frontier is (ranked walls) + +1. **Direct-GL CE ring resolution** `[wall]`. The GR ring `0xbeef0100` + resolves and its releases advance, but the CE ring `0xbeef0101` + (`gpfifo=0x4100000`) never maps: `VAS-PROMOTE` walks a parked candidate and + returns `rc=-2`, then `NOT mapped by any registered VAS` every ~7 s. The + client's CE work never executes and GZDoom wedges ~90 s after `V_Init`. + Next: walk all parked candidates regardless of owner VAS and/or keep at + least two fresh same-VAS parks. +2. **Window/screen present and the GLX lottery** `[wall]`. The pbuffer and + off-screen frame paths are proven (`pixel=ff0000ff`, 500/500 non-black + frames, 45.4 s video), but glxgears' window is black and the GLX + probe/frames path is a per-window lottery (hang/black/BadValue). A + synchronous window draw wedges X; screen capture must use the ivshmem + frame path and the present must be made to land. +3. **X startup / stability** `[wall]`. X startup is a per-boot lottery: on + 3/6 boots window creation wedges Xorg in an unkillable kernel RM spin + (`_nv052600rm+0x7b`); `Failed to initialize DMA`, push-buffer `0x51`, + `Wait for channel idle timed out` and occasional vmalloc failures also + appear. A fresh cycle clears it. The shared nvClip `modeNum` must be even + for the GL client's state init to succeed; the odd value is the guest + kernel's nvKms deferred close, not module-induced. +4. **The X DDX footprint** `[wall]`. The DDX allocates 4 GB of system pages at + startup; an 8 GB guest has no headroom once GZDoom's own footprint lands + and both are OOM-killed. Current runs use a 12 GB guest; the allocation is + a suspected emulation gap (a 512 MiB emulated VRAM falling back to a 4 GB + sysmem allocation) to fix. +5. **Non-black window frames and sustained rate** `[wall]`/`[target]`. Once + the CE ring, the window/screen present and X stability land, the capture + must show motion in a window, then 60 fps in one VM with no forcing, then + two VMs on two mediated devices with non-black frames in both. + +## 7. What is proven today + +| Capability | State | +|---|---| +| X server initialises on the emulated vGPU (driver DMA sanity push, ScreenInit) | `[live]` | +| Display engine serves the modeset push buffers; notifier plus `UPDATE -> FINISHED` | `[live]` (campaign configuration) | +| nvidia-modeset/NvKms registers (`/dev/dri/card0`, `renderD128`, `fb0`); `nvidia-smi -L` enumerates | `[live]` | +| `glxinfo` reports the emulated GPU with the NVIDIA renderer (OpenGL 4.6.0, direct rendering) | `[live]` | +| X green on attempt 1 in three consecutive cycles (`ebf3130f`) | `[live]` | +| GSP event-IRQ fix wakes the guest poll; glxgears completes swaps | `[live]` | +| 2D path end-to-end: accelerated X fills read back via `xwd` | `[live]` | +| Guest's real GL draws forward and execute on the physical GPU (`work_gr backend_ok=5-7`) | `[live]` | +| GR backend timeouts fixed (`backend_timeout=0`; draws retire in 3-2400 ms) | `[live]` | +| DDX 3D replay executes (`section captured` + `GR LAUNCH` on `0x1000014`, 12 launches/cycle) | `[live]` | +| First real GPU readback: pbuffer `glClear(red)+glReadPixels` -> `pixel=ff0000ff` | `[live]` | +| First non-black GPU-rendered frames: 500/500 off-screen frames; 45.4 s video captured (`r1769`) | `[live]` | +| Direct-GL GR ring `0xbeef0100` resolves; releases advance (`4acac093`) | `[live]` | +| GZDoom passes `V_Init: allocate screen` with GL 4.6.0 when shared nvClip `modeNum` is even | `[live]` | +| RM write-lock holder identified (Xorg in the DDX vidheap scrub wait) and fixed (`df6e71bb`) | `[live]` | +| GL section capture: the X channel's `0xC597` stream decoded | `[live]` | +| Verbatim 3D method-stream replay retired by the engine (no MMU fault) | `[live]` | +| Synthesized launch executes real `sm_75` SASS; Turing `V3` program header accepted | `[live]` | +| X server survives the run on a 12 GB guest (the DDX's 4 GB sysmem allocation forces the size) | `[live]` | +| Direct-GL CE ring `0xbeef0101` resolves and the client's CE work executes | `[wall]` | +| Window/screen present produces non-black window pixels | `[wall]` | +| X startup/stability (`_nv052600rm` soft-lock, `0x51`, vmalloc) | `[wall]` | +| Non-black screen pixels from the emulated vGPU's 3D pipeline | `[wall]` | +| Windows/DirectX guests | `[planned]` | + +## 8. What is not claimed + +* No **window/screen** pixel has been produced by the emulated vGPU's 3D + pipeline yet; every screen capture so far is black. The proven non-black + frames and the 45.4 s video come from the off-screen pbuffer path + (pbuffer -> sysmem -> kvmfr ivshmem -> host reader), not from a windowed X + screen capture. +* Display/KMS bring-up (NvKms, `/dev/dri`, X, a real GL context) is not a + rendered-frame claim. GZDoom reaching `V_Init: allocate screen` proves the + client's GL init and the direct-GL GR lane, not the game loop or a + presented frame; the direct-GL CE ring still does not resolve. +* No frame-rate or latency numbers: the scheduled measurement is defined + (60 fps over 60 s, no forcing) but not yet met. + +## 9. Related pages + +* [`OpenGL.md`](OpenGL.md) — the guest OpenGL path and its status. +* [`Vulkan.md`](Vulkan.md) — the guest Vulkan path and its status. +* [`DirectX.md`](DirectX.md) — Windows guests and the DirectX target. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the CUDA compute chain, + proven end to end. +* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work + is forwarded to. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — channels, GPFIFO, + engines, and page tables. +* [`Host-Module.md`](Host-Module.md) — where capture, decode and forwarding + live. +* [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) — + the canonical plan: gap list, three approaches, and the M-GL1..M-GL6 + milestone ladder with gates. diff --git a/wiki/Guest-Stack-and-CUDASW.md b/wiki/Guest-Stack-and-CUDASW.md index 1548f63..eca21a3 100644 --- a/wiki/Guest-Stack-and-CUDASW.md +++ b/wiki/Guest-Stack-and-CUDASW.md @@ -6,7 +6,7 @@ > [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md); > the docs tree wins on disagreement with this page. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with > the stock guest driver; `[gated]` implemented default-off behind supervised > gates; `[wall]` known blocking limit; `[target]` designed, not built; > `[planned]` roadmap. @@ -31,9 +31,12 @@ nvidia.ko (RM + UVM + modeset) Adjacent members of the family: **libnvidia-ptxjitcompiler** (PTX→SASS JIT — usually "just works" once the driver initializes), **libnvidia-fatbinaryloader / nvrtc**, **UVM** (`nvidia_uvm.ko` — [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md)), -the **graphics stack** (`libGLX_nvidia`, `libnvidia-glcore`, Vulkan ICD, -`nvidia_modeset`/`nvidia_drm`) and the **video stack** (NVENC/NVDEC) — both -roadmapped. +the **graphics stack** (`libGLX_nvidia`, the driver's GL core, the Vulkan ICD, +`nvidia_modeset`/`nvidia_drm` — partially live on the emulated vGPU: +Xorg/GLX run, the engine executes GL method streams, and the Vulkan pipeline +runs end to end; rendered pixels are the frontier — +[`Graphics-Execution.md`](Graphics-Execution.md)) and the **video stack** +(NVENC/NVDEC) — roadmap. ## 2. Layering rules the emulation must respect @@ -91,14 +94,18 @@ open /dev/nvidia* → cuInit → UVM registration (per process) → faults, completions, events ``` -Context creation and the full CUDA driver-API smoke now pass (`ALL OK`, -copies matched) [live]. The current focus is execution: the backend is a -**module-owned GPU running its real GSP firmware as the resource-manager -server** ([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)), with -CE-class copies forwarded to the physical copy engine default-off [gated]. -For GR, a forwarded guest launch reaches the front-end but the inline -launch-descriptor install has not been observed — the open execution wall -([`Execution-Frontier.md`](Execution-Frontier.md)) [wall]. +Context creation and the full CUDA driver-API smoke pass (`ALL OK`, copies +matched) [live], and guest kernel launches now **execute on the physical +engine** with engine-authored completions relayed to the guest [live]. The +backend is a **module-owned GPU running its real GSP firmware as the +resource-manager server** +([`Physical-GPU-Backend.md`](Physical-GPU-Backend.md)). The leading edge is +the staged-transfer class, the every-boot acceptance campaign and per-guest +isolation ([`Execution-Frontier.md`](Execution-Frontier.md)). Graphics work +reuses the same submission machinery on its own frontier: the X server and +GLX run, the engine executes GL method streams, and the Vulkan pipeline runs +end to end; rendered pixels are the wall +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 5. What the guest expects from the device, per phase @@ -135,7 +142,9 @@ underlying GSP device/firmware model but different host interfaces (WDDM, MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the firmware interface, forward the work. Windows support is a roadmap item ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) [planned], and -the design goal is that no Windows-specific code lives in the guest. +the design goal is that no Windows-specific code lives in the guest. The +Windows/DirectX path — WDDM, the Direct3D user-/kernel-mode drivers, and what +a bring-up requires — is described on [`DirectX.md`](DirectX.md). ## 8. Compatibility notes for versions diff --git a/wiki/HAAEEP-and-Collaboration.md b/wiki/HAAEEP-and-Collaboration.md index 201d430..0e12636 100644 --- a/wiki/HAAEEP-and-Collaboration.md +++ b/wiki/HAAEEP-and-Collaboration.md @@ -8,7 +8,7 @@ > not a shipped product. When this page and the docs tree disagree, the docs > tree wins. Front door: [`../docs/README.md`](../docs/README.md). -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[proposal]` architecture and roadmap; `[target]` milestone to reach; `[goal]` end state; `[design]` integration point designed, not built. +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[proposal]` architecture and roadmap; `[target]` milestone to reach; `[goal]` end state; `[design]` integration point designed, not built. ## 1. What HAAEEP is @@ -97,7 +97,7 @@ isolation model is described in | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | rung 2 (runtime start) is green; CE-class work is forwarded with release and fault relay in place (default off); for GR the guest launch reaches the front-end but no grid has dispatched yet — the frontier is tracked in [`Execution-Frontier.md`](Execution-Frontier.md) | +| CUDA execution in a guest (rung 3) | green: verified guest CUDA clients execute on the physical engine with engine-authored completions relayed to the guest; the staged-transfer class, every-boot acceptance, per-guest isolation and the graphics pixel frontier remain — tracked in [`Execution-Frontier.md`](Execution-Frontier.md) and [`Graphics-Execution.md`](Graphics-Execution.md) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | diff --git a/wiki/Hardware-Software-Matrix.md b/wiki/Hardware-Software-Matrix.md index 0363636..697d92e 100644 --- a/wiki/Hardware-Software-Matrix.md +++ b/wiki/Hardware-Software-Matrix.md @@ -9,7 +9,7 @@ > When this page and the docs tree disagree, the docs tree wins. Front door: > [`../docs/README.md`](../docs/README.md). -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[reference]` continuously tested, bugs block releases; `[supported]` tested per release; `[experimental]` partial or bring-up; `[target]` designed for, bring-up in progress; `[design]` design direction, not the current build. +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[reference]` continuously tested, bugs block releases; `[supported]` tested per release; `[experimental]` partial or bring-up; `[target]` designed for, bring-up in progress; `[design]` design direction, not the current build. ## 1. Support ladder @@ -43,9 +43,9 @@ selection is compile-time today and the module is built for the TU104 profile. | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** — probe, `nvidia-smi` and the driver-API smoke pass; CE-class copies are forwarded default-off; for GR the guest launch reaches the front-end but no grid has dispatched yet (see [`Execution-Frontier.md`](Execution-Frontier.md)); no continuous guest testing yet | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** — probe, `nvidia-smi`, the driver-API smoke and CUDA kernel execution on the physical engine pass (verified clients green on fresh boots); the staged-transfer class and per-guest isolation are in progress; graphics partially: the X server and GLX run, the engine executes GL method streams, rendered pixels are the frontier (see [`Graphics-Execution.md`](Graphics-Execution.md)); no continuous guest testing yet | | Linux aarch64 | **Target** | -| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | +| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** — see [`DirectX.md`](DirectX.md) | | Other Unix (FreeBSD, ...) | Out of scope until demand | **Guest rule:** the guest never contains Open-IOV code — no guest agent, no diff --git a/wiki/Host-Module.md b/wiki/Host-Module.md index acf98ed..6ffbed1 100644 --- a/wiki/Host-Module.md +++ b/wiki/Host-Module.md @@ -6,7 +6,7 @@ > source: [`../docs/03-architecture/06-host-module-and-hybrid-model.md`](../docs/03-architecture/06-host-module-and-hybrid-model.md); > the docs tree wins on disagreement with this page. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with +> Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` observed with > the stock guest driver; `[wip]` demonstrated live, being productized; > `[gated]` implemented default-off behind supervised gates; `[wall]` known > blocking limit; `[target]` designed, not built; `[planned]` roadmap. @@ -106,9 +106,33 @@ Proxying through the distribution host driver has **no callable entry point from kernel context**, so the module **owns the device and runs its real GSP firmware as the resource-manager server**, with emulation as the control plane. CE copies, release and fault relay are implemented default-off and -verified live [gated]; GR execution is the open wall [wall] — a forwarded -launch reaches the front-end; the inline launch-descriptor install is unobserved -([`Execution-Frontier.md`](Execution-Frontier.md)). +verified live [gated]; forwarded guest CUDA launches execute on the engine +with engine-authored completions relayed to the guest; the guest's real GL +draws now forward and execute on the physical GR engine (`work_gr +backend_ok=5-7`, `backend_timeout=0`), the DDX 3D replay reads back a real +pixel, and graphics method streams execute in three shapes, +including the direct GL client's own GR/CE channels +([`Execution-Frontier.md`](Execution-Frontier.md), +[`Graphics-Execution.md`](Graphics-Execution.md)). + +The same plane carries the display/modeset path [live in the campaign +configuration]: the module serves the guest display engine's push-buffer +channels (core `0xc57d`, window `0xc57e`, immediate/window `0xc57b`), authors +the notifier/`UPDATE` completion with a class-aware arm (the window class has +no NOTIFY bit; a bound notifier is the arm) and a force-retry for a deferred +event, completes the X DDX's `0x8000 -> 0` notifications, and lets +nvidia-modeset/NvKms register (`/dev/dri`), X settle and GLX clients obtain a +real GL context. The display-capability seam is parameterised. For the DDX's +own 2D push buffers the remaining host levers are publishing/advancing the +GPU-written engine get word and recycling the notifier to `0xffff` after +completion. The DDX 3D replay renders and reads back a pbuffer pixel +(`pixel=ff0000ff`), the off-screen frame path produced 500/500 non-black +animated frames and a 45.4 s video, the RM write-lock holder (Xorg in the +DDX vidheap scrub wait) is fixed (`df6e71bb`), and the direct-GL GR ring +`0xbeef0100` now resolves with advancing releases (`4acac093`). The remaining +GL gap is the direct-GL CE ring `0xbeef0101` (it never resolves, so the +client's CE work does not execute) and the window/screen present path, which +currently wedges X or stays black. ## 6. Debug parameters diff --git a/wiki/Interfaces-and-Stability.md b/wiki/Interfaces-and-Stability.md new file mode 100644 index 0000000..44cade2 --- /dev/null +++ b/wiki/Interfaces-and-Stability.md @@ -0,0 +1,117 @@ +# Interfaces and Stability + +> **Scope.** The interfaces Open-IOV exposes or consumes, which of them are +> **contracts** (stable, versioned, safe to build on), which are **not** +> (bring-up and debug surfaces, free to change), and *why* the stable set is +> stable. This is the compatibility policy for the project's own software — the +> interfaces of *other people's* software are fixed by the project's rules +> (nothing is patched). Canonical sources: +> [`../docs/eng-ops.md`](../docs/eng-ops.md) (engineering rules), +> [`../docs/06-portability/hardware-software-matrix.md`](../docs/06-portability/hardware-software-matrix.md) +> (version/device profiles) and the headers/schemas named below; the docs tree +> wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. + +## 1. Why interfaces are the product + +Open-IOV's durability comes from a single idea: **own the interface, not a +snapshot of it**, and **never patch anyone**. The interfaces the project +*consumes* — the stock NVIDIA driver's device expectations, the kernel's VFIO/ +mdev UAPI, the hypervisor's DMA and interrupt interfaces — are therefore +immutable by construction: if the project changed what the guest driver sees, +the stock driver would break, and the project would have failed its own rule. + +The interfaces the project *exposes* are split deliberately into three +classes, and everything below is one of them: + +| Class | Promise | Examples | +|---|---|---| +| **Consumer contracts** | stable; versioned; changes are additive or bump a version | the guest-visible device interface, the mdev lifecycle surface, the userspace management library, the configuration schemas, tooling evidence formats | +| **Bring-up and debug surfaces** | *no* promise — names and semantics may change between builds | module parameters, debug filesystem files, internal state shapes | +| **Internal surfaces** | no promise, no consumers outside the module | kernel functions, structs, heuristics | + +## 2. Stable interfaces (contracts) + +| Interface | Consumers | Why it is stable | Change policy | +|---|---|---|---| +| **The guest-visible device interface**: PCI configuration space, BAR0 register interface, BAR1/BAR3 apertures, MSI-X; the GSP boot protocol; the GSP-RM message-queue transport and RPC functions; RM object/control semantics; events and completions | stock NVIDIA guest drivers (Linux and Windows) | *It is NVIDIA's interface*, fixed by the driver version the profile targets. The no-patch rule means changing it is not an option: any update must keep the unmodified driver working | Never changed by us. New driver/firmware versions enter as **profiles with measured deltas** ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)); the emulation answers the version the driver negotiates | +| **The host/platform surface of `vgsp.ko`**: the VFIO **mdev lifecycle** (mdev type names such as `nvidia-vgpu-turing`, `-1g`, `-2g`; sysfs create/remove; the VFIO region, DMA and IRQ UAPI) | QEMU/libvirt and management tooling; any conforming VFIO consumer | It is the kernel's mdev/VFIO contract plus a small, documented set of type names that tooling (and user scripts) reference by name | Additive by default; renaming or removing a type requires a migration note and a supported overlap window | +| **The userspace management library** (`libvgpu` / `libiov`; header `userspace/include/libvgpu.h`): error codes, memory/type enums, VF, migration and WPR operations; versioned soname | project tooling and integrators writing management workflows | A small C API with a versioned shared-object name; its consumers link rather than copy | Additive changes keep the soname; breaking changes bump it. The surface is small on purpose — it is a management API, not the data path | +| **Configuration profiles** (`config/versions/*.yaml`, `config/hardware/*.yaml`): per-driver and per-device adaptation data | the module's version adaptation and the porting workflows | Data contract: the porting promise is "a new version is a profile plus a measured delta", which only holds if the schema is stable | Additive fields; semantic changes are versioned with the profile | +| **Developer-tooling evidence formats**: the recorded-session format (`greeneye-session/1`), the symbol database, the injection ABI (v1) | recorded evidence, replays and future sessions of the firmware workbench | Evidence must remain interpretable years later, so the formats are declared back-compatibility items even though the tooling itself is development-only | Version bumps with migration notes; additive fields preferred; a breaking change needs a shim or a format-version bump | +| **The documentation canon** | users, integrators, future contributors | The docs tree is the declared source of truth: where a wiki page or summary disagrees with the repository documents, the repository wins | Corrections update the docs tree first, then the derived pages | + +## 3. Explicitly not stable (may change) + +Nothing in this list is safe to build on; treat it as subject to change +between builds, without notice: + +* **Module parameters** (`/sys/module/vgsp/parameters/*`) — bring-up and debug + knobs (targeted probes, forced behaviours, diagnostic mirrors). They exist to + run experiments; names and semantics change as the canonical paths replace + them. +* **Debug filesystem surfaces** (the module's `gsp-phys-*` controls and + counters) — observability only; no format or lifetime promise. +* **`vgsp.ko` internals** — functions, structures, state layouts; there are no + external consumers and no stability promise. +* **Emulation internals** — ring/USERD discovery heuristics, shadow windows, + scratch mappings, scan cadences, and the parameter-gated forwarding and + graphics seams while they are validated. When a seam graduates, it becomes a + documented configuration surface and leaves this list. +* **Scripts, harnesses and test clients** (`scripts/`, `tools/`, + `tools/guest-diag/`) — working material, not products. +* **The firmware workbench's CLI, plugins and probe files** + ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) — development-only + by declaration; interfaces and file formats may change without notice. + (Exception: the *session format* above is versioned precisely so recorded + evidence survives tool change.) +* **This wiki and the site** — a curated, dated documentation view, refreshed + with status changes; it is not an API. + +## 4. Why the stable set is stable + +Three kinds of reason, and each stable interface has at least one of them: + +1. **It is externally specified.** The guest-visible device interface is + NVIDIA's; the mdev lifecycle is the kernel's. Stability is inherited, and + the project's rules forbid "fixing" either side. +2. **It has declared consumers.** The user library, the mdev type names, the + configuration schemas and the evidence formats have real consumers + (tooling, scripts, evidence archives). A contract exists because someone + else depends on it — not because it was convenient to freeze. +3. **The project's own rules make it so.** *No patches* + *profiles, not + rewrites* means the device-facing behaviour must evolve the way NVIDIA's + driver evolves — by version, never by surprise. + +**Change discipline, in one sentence:** prefer additive; version what +consumers link against or parse; never break evidence or an unmodified +driver; document the migration when something must move. + +## 5. Open items (stability questions we expect to close) + +* **Userspace library freeze point.** When the management workflows stabilise, + declare an API freeze (a 1.0 surface) so integrators can rely on it. +* **Graduating debug seams.** The parameter-gated forwarding and graphics + seams will migrate from debug knobs to documented configuration as they are + validated; the graduating list is tracked on + [`Project-Status.md`](Project-Status.md). +* **Schema formality.** The configuration profiles are documented data today; + a formal schema would make the "profile + measured delta" promise checkable + by tooling. +* **Evidence-format externalisation.** If parties outside the project archive + workbench sessions, the session and symbol formats need the same public + versioning treatment as the user library. + +## 6. Related pages + +* [`Project-Goals.md`](Project-Goals.md) — the goals this compatibility + policy serves. +* [`Host-Module.md`](Host-Module.md) — where the mdev surface and the debug + parameters live. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — how + profiles, versions and devices are handled. +* [`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md) — the development-only + tooling and its versioned session format. +* [`Performance.md`](Performance.md) — why debug surfaces are explicitly not + configuration. diff --git a/wiki/OpenGL.md b/wiki/OpenGL.md new file mode 100644 index 0000000..9540720 --- /dev/null +++ b/wiki/OpenGL.md @@ -0,0 +1,147 @@ +# OpenGL + +> **Scope.** How a stock guest OpenGL application's commands travel through the +> unmodified NVIDIA stack on the emulated vGPU, and the honest state of running +> OpenGL on it. The host-side execution machinery shared by all graphics APIs +> is on [`Graphics-Execution.md`](Graphics-Execution.md); CUDA compute is on +> [`Execution-Frontier.md`](Execution-Frontier.md). Canonical sources: +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) +> (the plan and milestones), +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> (the measured walls) and +> [`../docs/03-architecture/02-guest-stack-and-cudasw.md`](../docs/03-architecture/02-guest-stack-and-cudasw.md) +> (the guest stack). The docs tree wins on disagreement. + +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` observed with the +stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` +defined success state. + +## 1. The guest OpenGL stack + +On Linux, an OpenGL application does not talk to the kernel driver directly: +it talks to the GL implementation, and — when a display is involved — through +the X server's GLX extension: + +```text +application (toolkit / SDL / GLFW / raw GLX) + -> libGL (dispatch) -> libGLX_nvidia (NVIDIA GLX client) + -> libnvidia-glcore (the vendor GL engine: state, programs, shaders, draws) + -> ioctls + mappings to nvidia.ko -> channels, buffers, synchronization +X server (Xorg) + -> GLX extension module (glxserver_nvidia) and the NVIDIA DDX (X driver) + -> owns the screen, drawables and the present path +``` + +Two forms of operation exist, and both are visible in the guest: + +* **Direct rendering** — the normal desktop case. The GL client builds GPU + commands in its own process and the X server coordinates buffers and + presents. The client's channel is the source of truth for per-draw state. +* **Indirect GLX** — the client's GL protocol is serialised to the X server, + which executes it against its own GL context. Here the X server's channel + carries the graphics stream. + +On the emulated vGPU the readable command sources today are the **direct GL +client's own channels** — the GR channel `0xbeef0100` and its CE siblings +`0xbeef0101`/`0xbeef0102` — and the **X server's channels**, whose +graphics-class (`0xC597`, `TURING_A`) sections the emulation captures and +whose DDX kick channel (`0xbfef0002`) is decoded from its ring head +([`Graphics-Execution.md`](Graphics-Execution.md) §3). + +## 2. What an NVIDIA GL frame involves (observed shapes) + +* **Context and state setup** — shader programs (SASS) uploaded to device + memory, constant/descriptor banks, render targets and depth surfaces + allocated and mapped. +* **Per-draw work** — a graphics-class method stream: state binds (viewport, + render target, ZS, blend/ROP), program and constant binds, vertex/index + data, and draw-issuing methods that drive the fixed-function pipeline + (vertex fetch, primitive assembly, rasterisation, fragment work, ROP). +* **Program/descriptor delivery** — the driver uploads shader code and + descriptors through its copy/utility-channel inline-data path. The GL draw + path decodes these uploads and the engine executes the draw; Vulkan's + inline-to-memory arm is the remaining mirror-image gap + ([`Vulkan.md`](Vulkan.md)). +* **Present** — the X server's present/readback path runs as light 2D-class + and copy-engine operations; the emulator decodes and executes those shapes. + +## 3. Status on the emulated vGPU + +Rated honestly, against the milestone ladder in the plan: + +| Item | State | +|---|---| +| X server initialises on the emulated vGPU (driver push-DMA sanity test, ScreenInit) | `[live]` | +| nvidia-modeset/NvKms registers (`/dev/dri/card0`, `renderD128`, `fb0`); `nvidia-smi -L` enumerates | `[live]` (campaign configuration) | +| `glxinfo -B` reports the NVIDIA renderer (OpenGL 4.6.0, driver 570.144, direct rendering) | `[live]` | +| The X driver's RC-watchdog channel registers and its notifier is authored | `[live]` | +| X green on attempt 1 in three consecutive cycles (`ebf3130f`) | `[live]` | +| GSP event-IRQ fix wakes the guest poll; glxgears completes swaps | `[live]` | +| 2D path end-to-end: accelerated X fills read back via `xwd` | `[live]` | +| Guest's real GL draws forward and execute on the physical GPU (`work_gr backend_ok=5-7`) | `[live]` | +| GR backend timeouts fixed (`backend_timeout=0`; draws retire in 3-2400 ms) | `[live]` | +| DDX 3D replay executes (`section captured` + `GR LAUNCH` on `0x1000014`, 12 launches/cycle) | `[live]` | +| First real GPU readback: pbuffer `glClear(red)+glReadPixels` -> `pixel=ff0000ff` | `[live]` | +| First non-black GPU-rendered frames: 500/500 off-screen 640x480 frames, 45.4 s video captured (`r1769`) | `[live]` | +| Direct-GL GR ring `0xbeef0100` resolves; releases advance frame by frame (`4acac093`) | `[live]` | +| GZDoom passes `V_Init: allocate screen` with GL 4.6.0 when shared nvClip `modeNum` is even | `[live]` | +| RM write-lock holder identified (Xorg in the DDX vidheap scrub wait) and fixed (`df6e71bb`) | `[live]` | +| Engine executes a launch built from the captured section (real `sm_75` SASS retired) | `[live]` | +| Verbatim 3D method-stream replay retired by the engine, no MMU fault | `[live]` | +| X server survives the run on a 12 GB guest (the DDX's 4 GB sysmem allocation forces the size) | `[live]` | +| Direct-GL CE ring `0xbeef0101` resolves (GZDoom CE work executes) | `[wall]` | +| Window/screen present produces non-black screen pixels | `[wall]` | +| X startup/stability (`_nv052600rm` soft-lock, `0x51`, vmalloc) | `[wall]` | +| 60 fps sustained, no forcing (M-GL4) | `[target]` | + +## 4. How OpenGL runs are verified + +Instruments that matter, in order of usefulness: the X server log and +`xdpyinfo` (did the server initialise?), `glxinfo -B` (which renderer is +real), the application's own progress output, and frame capture at the GL +source (a swap-hook capture is the ground truth for pixels — the X screen +itself is not, on some stacks). Captures are decoded offline and scored: +fraction of pixels above a luminance threshold, colour count, and +frame-to-frame motion. A run counts only when the capture shows motion with +the NVIDIA renderer in use. + +## 5. Milestones on this path + +The graphics plan defines M-GL1 (client context-init decodes and executes; the +client leaves its retry loop), M-GL2 (`glxgears` non-black), M-GL3 (GZDoom GL +non-black through the mdev), M-GL4 (60 fps, no forcing), M-GL5 (two VMs, two +mdevs), M-GL6 (split-screen video). The project sits **between M-GL1 and +M-GL2, with M-GL2's off-screen half proven**: the display/KMS stack registers, +X settles, the client submits and retires its GR/CE channels, the direct-GL GR +ring resolves and its releases advance, and the GPU-rendered pbuffer path +produced non-black animated frames and a captured 45.4 s video. The remaining +gates are the direct-GL CE ring `0xbeef0101`, the window/screen present path +(a window draw currently wedges X and glxgears' window is black), and X +stability across boots — then the first rendered screen frame. + +## 6. What is not claimed + +* No **window/screen** frame has rendered real pixels through the emulated + vGPU yet; every screen capture so far is black. The non-black frames and the + video come from the off-screen pbuffer path (pbuffer -> sysmem -> kvmfr + ivshmem -> host reader), not from a windowed X screen capture. The direct-GL + CE ring `0xbeef0101` does not resolve, a synchronous window draw wedges X, + and X window creation wedges on some boots. Those are the current walls. +* GZDoom reaching `V_Init: allocate screen` is not a rendered-frame claim; it + proves the client's GL init and the direct-GL GR lane, not the game loop or + a presented frame. +* No frame-rate or latency numbers. + +## 7. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the shared execution + machinery and the ranked walls. +* [`Vulkan.md`](Vulkan.md) — the sibling API path, further along in the + pipeline lifecycle. +* [`DirectX.md`](DirectX.md) — the Windows/DirectX target. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — how the NVIDIA + userspace stack is analysed. +* [`Install-and-First-Run.md`](Install-and-First-Run.md) — bringing the stack + up. +* [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md) — + the canonical plan, gap list and gates. diff --git a/wiki/Performance.md b/wiki/Performance.md new file mode 100644 index 0000000..b62d6a7 --- /dev/null +++ b/wiki/Performance.md @@ -0,0 +1,192 @@ +# Performance + +> **Scope.** How Open-IOV's performance is modelled, traced, and planned: the +> cost structure of the system, the metric set, what is observed today +> (honestly — there are no published numbers yet), the known gaps with their +> improvement plans, and the architecture questions the performance work will +> have to answer. Canonical sources: +> [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) +> (the measurement contract), [`../STATE-CURRENT.md`](../STATE-CURRENT.md) (the +> dated snapshot) and [`RUNBOOK.md`](../docs/RUNBOOK.md); the docs tree wins on +> disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` measured with +working code; `[wip]` measurement framework partial; `[planned]` designed, +not built. + +## 1. The performance model + +The cost structure of a vGPU session splits into four layers, and each is +measured separately: + +| Layer | What it costs | Representative metric | +|---|---|---| +| **Initialization** | VM boot → driver probe → `cuInit` | `T_init` | +| **Control plane** | Resource-manager request/reply round trips (objects, controls, memory) | RPC latency p50/p99 | +| **Steady-state data path** | Submission → decode → forward → engine → completion → guest wake-up | push overhead; completion latency; kernel throughput | +| **Data movement** | Copies between guest memory and device memory | copy bandwidth (H2D/D2H/D2D) | + +Presentation (3D frames) sits on top of the data path, with its own budget +(frame pacing, not just frame rate). + +Three project rules govern every number: + +1. **Correctness before speed.** A change that makes the vGPU faster but + breaks the acceptance corpus does not count. +2. **Proxy first, optimise by measurement.** The architecture deliberately + prototyped the safe path (emulate the interface, forward the work through + the module's own GSP client) before considering a hot path; whether to move + the hot path closer to direct submission is a *measurement decision*, not a + design preference. +3. **No number without methodology.** Numbers are only quoted with their + harness, versions, clocks and warm/cold state, and ideally as a ratio + against a same-guest, same-hardware baseline (the A/B method below). + +## 2. The metric set + +| Metric | Definition | Why it matters | +|---|---|---| +| `T_init` | VM boot → `cuInit` returns | user-visible startup | +| RPC latency p50/p99 | request write → reply visible in the status queue | init and management overhead | +| Push overhead | guest submission → engine start (forwarded) | steady-state latency | +| Semaphore completion latency | release written → tracker observes | per-operation sync cost | +| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality | +| Kernel throughput | kernels/second for a fixed workload | launch overhead | +| Multi-tenant fairness | per-tenant progress under contention | isolation quality | +| Frame pacing | inter-frame deltas under load | graphics smoothness (p95 is the budget) | + +## 3. How performance is traced today + +* **Host-side telemetry.** The module timestamps its own work: per-run + counters in the kernel log (accepted / failed / retried / refused work, + decode outcomes, forward outcomes), `trace_printk`/tracepoints at dispatch + entry/exit for distributions, and counter surfaces readable through the + kernel's tracing and debug filesystems. Module parameters exist for targeted + probes during bring-up — they are debug surfaces, not configuration + ([`Interfaces-and-Stability.md`](Interfaces-and-Stability.md)). +* **Guest-side instruments.** The guest test clients print their own + acceptance *and* timing lines (they are the ground truth for what the + application observed); guest-side kernel probes and ptrace tools attribute + driver-side waits when a client stalls. +* **The A/B harness.** Every measurement runs twice on the same machine and + preferably the same boot: **A** = the workload on the physical GPU with the + stock driver; **B** = the identical workload in the guest on the emulated + vGPU. Results are reported as **B/A ratios**; warm and cold, with variance + and run counts. One variable at a time; the harness is published so results + are comparable. +* **Offline tools — with a hard boundary.** The risc-v-gsp-rm-debugger + workbench models *instruction semantics and memory behaviour only*; it makes + **no timing claims** (its fidelity levels explicitly exclude time and + concurrency). Timing questions are answered on hardware, with + instrumentation. +* **The evaluation ladder.** Performance is only meaningful from the + *execution* rung upwards: initialization → runtime start → execution → + sustained → multi-tenant. Claims are made per rung, never across it. + +## 4. Where it stands today (honest) + +**There are no published performance numbers yet.** The framework above is +the contract; the first formally recorded measurements are scheduled, not +done. What *is* observed today, clearly labelled as internal observation +rather than benchmark: + +* **Forwarded CUDA launch latency is on the order of hundreds of + milliseconds** end-to-end on the development setup (one ten-client arc + measured p50 ≈ 650 ms). This is the dominant known cost, dominated by the + submission trap → decode → forward chain and by polled completion waits — + not by the engine itself. +* **Data movement is verified correct, not yet measured for throughput**: + copies through the forwarded path round-trip byte-exact; the A/B bandwidth + runs (the `nvbandwidth`-class suite) are pending. +* **Graphics frame delivery is far below interactive rates** in the recorded + pipeline runs — the defined target is 60 fps *sustained with no forcing* + ([`Graphics-Execution.md`](Graphics-Execution.md)); today's frame numbers + come from development instrumentation and are explicitly not a benchmark. +* **Initialization is functional**: the backend's GSP boot reaches init-done + in about a second on the reference hardware; `T_init` as a guest-visible + measurement has not been taken. +* **The software path is already cheap where it is work-shaped**: decoded + memory operations execute at host memory speed; control-plane replies are + produced inline in the emulation thread; the expensive parts are the + *control-heavy* structures (per-push passes, scanning, polling). + +## 5. Observed gaps and the plan for each + +| Gap | Observation | Plan | +|---|---|---| +| Forwarded-launch latency | hundreds of ms (internal); dominated by the multi-step forward chain and completion polling | instrument the distribution end to end; reduce per-push passes; move completion to event-driven paths; re-measure per step | +| Completion latency | releases are detected by polling the engine/guest memory | prefer engine-authored release paths already used by verified launches; doorbell/event wake-ups instead of periodic scans where possible | +| RPC dispatch cost | all management funnels through one emulation thread | build the RPC round-trip microbenchmark first; batch and prioritise; reduce wake-ups | +| Scanning overhead | per-pass discovery/scan work (guest RAM, shadow apertures) exists for correctness while registrations are incomplete | convert discovery to registered/learned state so per-pass cost scales with *work*, not with memory size | +| BAR0 shadow reads | uncached to the guest, as MMIO must be; bulk reads would be slow | if a consumer bulk-reads, introduce cache-safe sub-regions rather than a slower guest | +| Memory-model gap | guest VRAM is backed by host RAM today | real VRAM backing is a target; DMA-heavy workloads are not representative until it exists | +| Graphics frame path | far below 60 fps; forcing/diagnostic assistance excluded by the target definition | the graphics milestone ladder: natural completions with no mirror, then measure pacing | +| Multi-tenant fairness | two guests coexist; fairness under load is untested | define the fairness metric, run two-VM contention, then add scheduling policy | +| Initialization | functional but unmeasured | take `T_init` and the reply-fidelity tail as the next control-plane measurement | + +## 6. The improvement programme + +1. **Build the microbenchmarks first** — the four the measurement contract + names: RPC round-trip, MSI-X delivery latency, BAR0 shadow read latency, + and doorbell-polling cost. These attribute overhead to components before + any optimisation. +2. **Run and publish the A/B corpus** — the canonical suite (vector-add, + SGEMM, copy bandwidth, a GL frame-rate app, a framework op loop) with the + contract's pass criteria (e.g. ≥95% B/A for compute, ≥90% for GL and + framework loops) and full environment capture. +3. **Optimise the observed hot structures** — fewer per-push passes, batched + decode, event-driven completion, shadow-window reuse — each as an A/B + change with its numbers kept. +4. **Evaluate the hot-path placement decision** — keep the module as a GSP + client proxy, or move submissions closer to direct channels — using the + measured budgets as the criterion. +5. **Publish the first graphics measurement** — 60 fps sustained, no forcing, + p95 inter-frame pacing — as the first end-to-end performance claim. +6. **Then multi-tenant**: define fairness, measure it, and only then claim it. + +## 7. Architecture areas that may need to improve — open questions + +These are the structural questions the performance programme exists to close: + +* **One ordered workqueue, one in-flight request.** The current design + serialises forwarded work for ordering safety. Is that a throughput ceiling + for multi-channel/multi-launch workloads, and how much pipelining is + possible without weakening the ordering guarantees the guest observes? +* **Per-pass scanning vs registered state.** Ring/USERD discovery, content + scans and shadow diffs are correctness mechanisms; can they be replaced (or + narrowed) by registrations learned on first sight, so steady-state cost does + not scale with aperture size? +* **Completion publication machinery.** Which of poll, engine-authored + release, notifier and event paths should carry the steady state — and what + is the p99, not just the median? +* **MMIO trap frequency.** Doorbells and shadow writes pay a VM-exit cost; + which guest patterns produce the most traps, and can they be coalesced or + served with less work? +* **The transport itself.** Guest-system-memory message queues plus doorbells + are deliberately conservative; would a shared-ring fast path change the + budget materially? +* **Hot-path placement.** Proxy through the module's own GSP session vs + moving hot submissions closer to the hardware — decided by the A/B numbers, + not by preference. +* **Multi-tenant scheduling.** Time-slicing exists only as a default; the + fairness policy (and MIG-backed partitioning where available) is a design + area, with fairness measurements defined before the claim. +* **Fidelity boundary.** The offline simulator can never answer timing or + concurrency questions; the performance programme therefore owns all timing + claims, and no timing claim may cite a simulator session. + +## 8. Related pages + +* [`Interfaces-and-Stability.md`](Interfaces-and-Stability.md) — which + surfaces are contracts and which are free to change. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute path whose + latency this page budgets. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the 60 fps target and its + milestone ladder. +* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work + is forwarded to. +* [`Host-Module.md`](Host-Module.md) — the module whose threads, queues and + scans carry the costs above. +* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — + the measurement contract: ladder, metrics, A/B method, microbenchmarks, + targets. diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md index bbca084..ee014ad 100644 --- a/wiki/Physical-GPU-Backend.md +++ b/wiki/Physical-GPU-Backend.md @@ -5,7 +5,7 @@ > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. -Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. @@ -107,20 +107,31 @@ FAIL a failed mutating operation parks the device in a locked fail-safe; | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | -| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the staged-transfer class is under repair `[live]` | | CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest | +| Graphics method-stream execution (verbatim 3D replay, synthesized launches, direct-GL client channels) | `[wip]` — the engine retires real graphics streams, the display/KMS stack registers and the direct GL client's GR/CE channels submit; rendered pixels are the frontier ([`Graphics-Execution.md`](Graphics-Execution.md)) | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a -compute-class object). Forwarded guest CUDA launches now execute on the +compute-class object). Forwarded guest CUDA launches execute on the engine: the inline launch descriptor installs and is read, the grid dispatches, the engine authors the completion, the host relays it, and the guest's synchronisation wait resolves — verified on fresh boots with the -vector-add, matrix-multiply and transformer clients. The leading edge is the -large-buffer staged-transfer class (under repair), reproducibility across -fresh boots, and per-guest execution isolation — see -[`Execution-Frontier.md`](Execution-Frontier.md). +vector-add, matrix-multiply and transformer clients. Graphics work rides the +same machinery in three shapes (a verbatim 3D method-stream replay, +synthesized launches, and the direct GL client's own GR/CE channels); the +captured and synthesized streams execute on the engine, and the direct GL +client's channels submit. The display/KMS stack registers, the X DDX waits are +fixed, X is reliable on good boots, the DDX 3D replay executes, and the first +non-black GPU-rendered frames are proven (500/500 off-screen frames plus a +45.4 s video); the direct-GL GR ring resolves with advancing releases, so the +remaining graphics gates are the direct-GL CE ring `0xbeef0101`, the window/ +screen present, and X stability before non-black window pixels. The remaining +compute work is the staged-transfer class (under repair), the every-boot +acceptance campaign, and per-guest execution isolation — see +[`Execution-Frontier.md`](Execution-Frontier.md) and +[`Graphics-Execution.md`](Graphics-Execution.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. @@ -154,8 +165,8 @@ fail-safe that is never auto-retried; recovery is a supervised step. interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). * Multiple mediated devices with different emulated VRAM sizes and multiple - guests already coexist at the device level; per-guest execution isolation - for simultaneous CUDA guests is in progress. + guests already coexist at the device level (two guests demonstrated); + per-guest execution isolation for simultaneous CUDA guests is in progress. ## 10. Open questions diff --git a/wiki/Project-Goals.md b/wiki/Project-Goals.md new file mode 100644 index 0000000..884dbb6 --- /dev/null +++ b/wiki/Project-Goals.md @@ -0,0 +1,106 @@ +# Project Goals + +> **Scope.** What Open-IOV is for, what it is not for, and the dated status of +> each goal. Condensed from +> [`../docs/01-orientation/01-purpose-and-scope.md`](../docs/01-orientation/01-purpose-and-scope.md) +> (mission, scope and success criteria) and +> [`../docs/README.md`](../docs/README.md) (the project front door); the docs +> tree wins on disagreement. The live milestone ladder is on +> [`Project-Status.md`](Project-Status.md), and the current execution detail on +> [`Execution-Frontier.md`](Execution-Frontier.md) and +> [`Graphics-Execution.md`](Graphics-Execution.md). + +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` working with the +stock guest driver; `[wip]` in progress; `[planned]` designed, not built; +`[target]` bring-up ahead; `[non-goal]` explicitly out of scope. + +## 1. Mission + +**Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without +modifying NVIDIA software, firmware or hardware.** + +Concretely, Open-IOV builds an independent, maintained IP block that: + +* presents a virtual NVIDIA GPU to a guest operating system; +* lets the guest run the **stock NVIDIA driver** (Linux and Windows); +* supports **compute (CUDA)** first, then **3D graphics** and **video**; +* runs on ordinary NVIDIA GPUs already deployed in the field; +* isolates guests from each other and from the host; +* and requires no license key, no proprietary host service, no patched guest + driver, and no specific hypervisor. + +The guiding sentence is: ***Emulate the interface. Pass through the work. +Never patch anyone.*** + +## 2. The gap it fills + +| Approach | Limitation | +|---|---| +| Discrete device assignment (VFIO passthrough) | One GPU per VM; no sharing; fragile on consumer/laptop systems. | +| NVIDIA proprietary vGPU | Requires a license and a vendor host stack; driver-version locked. | +| Software GPU emulation | No real CUDA; no real 3D performance; not usable for AI. | +| Cloud-only inference | Requires an external service; unacceptable for secure edge, offline, or private data. | + +Open-IOV occupies the gap: **real GPU acceleration, multiple VMs, stock +drivers, no license.** + +## 3. Goals and their status + +| Goal | What success looks like | Status (2026-11-11) | +|---|---|---| +| Stock guest driver initialisation | The unmodified driver probes, initialises and enumerates as on bare metal (`nvidia-smi`) | `[live]` — TU104 reference platform | +| Full CUDA driver-API surface | Context creation, allocation, copies, free, destroy complete with correct results | `[live]` — the driver-API smoke prints `ALL OK`; the 64-byte payload round-trips | +| CUDA kernel execution on real hardware | Verified guest kernels execute on the physical GPU, with engine-authored completions waking the guest | `[live]` — vector-add, matrix-multiply and transformer clients green on fresh boots; the staged-transfer class, the every-boot acceptance campaign and per-guest isolation are in progress ([`Execution-Frontier.md`](Execution-Frontier.md)) | +| OpenGL (3D graphics) | Stock guest GL applications render real pixels through the emulated vGPU | `[wip]` — the display/KMS stack and X run (NvKms registers, `glxinfo` OpenGL 4.6.0 direct), the 2D path is green, the DDX 3D replay executes, and the first non-black GPU-rendered frames plus a 45.4 s video are captured off-screen; the direct-GL GR ring resolves and GZDoom reaches `V_Init`. Window/screen pixels remain the frontier: the direct-GL CE ring `0xbeef0101` does not resolve, glxgears' window is black, and X window creation wedges on some boots ([`OpenGL.md`](OpenGL.md)) | +| Vulkan | Stock guest Vulkan applications render real pixels | `[wip]` — the pipeline runs end to end (device → command buffer → fence → readback); frames are black — program delivery and program-VA mapping are the frontier ([`Vulkan.md`](Vulkan.md)) | +| Windows guests (DirectX, CUDA) | A stock WDDM driver and stock D3D/CUDA applications run in Windows guests | `[target]` — not started; same emulation target ([`DirectX.md`](DirectX.md)) | +| Video (NVENC/NVDEC) | Encode/decode through the emulated device | `[planned]` | +| Multi-tenant isolation | Multiple VMs on one GPU, isolated by the IOMMU and the device model, with explicit scheduling | `[wip]` — two mediated devices (different emulated VRAM sizes) and two guests coexist; per-guest execution isolation is in progress ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)) | +| Hypervisor portability | The same IP block attaches to KVM, Xen, Argo/OpenXT, Qubes OS Air, ... | `[planned]` — the emulation core is hypervisor-independent by construction; KVM/QEMU with VFIO mdev is the reference transport ([`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)) | +| Version and hardware portability | New driver, firmware or GPU generations enter by configuration plus measurement | `[wip]` — 570.144/TU104 is the anchor; per-version evidence workflows exist; newer versions are gated work | +| Live migration | A VM migrates with its GPU state | `[planned]` — the protocol analysis is kept for the record ([`Migration-Protocol.md`](Migration-Protocol.md)) | +| Performance | Within an agreed factor of native for the supported cases, with published methodology | `[planned]` — no published numbers yet; the first scheduled measurement is the 60 fps graphics target ([`Graphics-Execution.md`](Graphics-Execution.md)) | +| Security posture | Isolation survives adversarial review; the device fails closed | `[wip]` — fail-closed design and audit practice; per-tenant execution isolation is in progress ([`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md)) | + +## 4. How success is judged + +The project evaluates itself on five axes, each with a methodology document +in [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md): + +1. **Initialization** — the driver reaches bare-metal parity through boot, + probe and initialisation. +2. **Runtime** — the guest runs real work: CUDA kernels with correct results, + OpenGL/Vulkan rendering, encode/decode; repeated, sustained, resilient. +3. **Interoperability** — no changes anywhere: guest driver, host driver, + firmware, hypervisor. +4. **Stability** — crash-free under stress: start/stop cycles, VM kill, driver + reload, suspend/resume, migration. +5. **Performance and security** — within an agreed factor of native, with + isolation that survives adversarial review. + +## 5. Non-goals + +* **Replacing bare-metal inference serving.** Continuous batching on + bare-metal hosts remains the right answer for centralised datacenter + inference; Open-IOV is for compute that must live *inside* guests. +* **Forking or patching** the NVIDIA driver (guest or host), the GPU firmware, + or hypervisors to make the device work. +* **Anti-cheat / DRM circumvention.** The project interoperates with + interfaces and implements **its own** device; it does not counterfeit + vendor licensing. +* **Emulating GPU instruction execution in software.** Software emulation of + CUDA kernels is orders of magnitude too slow; work must run on real silicon. + +## 6. Related pages + +* [`Project-Status.md`](Project-Status.md) — the dated milestone ladder. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute execution + frontier. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the 3D execution + frontier. +* [`HAAEEP-and-Collaboration.md`](HAAEEP-and-Collaboration.md) — the + agent-edge use case and collaboration models. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the support + ladder. +* [`../docs/01-orientation/01-purpose-and-scope.md`](../docs/01-orientation/01-purpose-and-scope.md) — + the canonical purpose-and-scope document. diff --git a/wiki/Project-Plan.md b/wiki/Project-Plan.md new file mode 100644 index 0000000..dd8de18 --- /dev/null +++ b/wiki/Project-Plan.md @@ -0,0 +1,150 @@ +# Project Plan + +> **Scope.** The sequence of work: the phases, the gates that close them, what +> comes next, and how the plan itself is maintained. This is the *roadmap* +> view — [`Project-Goals.md`](Project-Goals.md) says what the project is for, +> [`Project-Status.md`](Project-Status.md) says where it stands today, and this +> page says in what order the remaining work happens and what must be true +> before each step. Canonical sources: the compute milestone ladder in +> [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md), +> the graphics ladder in +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md), +> the five-rung evaluation ladder in +> [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md), +> and the porting sequence in +> [`../docs/06-portability/03-device-validation-and-porting-sequence.md`](../docs/06-portability/03-device-validation-and-porting-sequence.md); +> the docs tree wins on disagreement. + +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[done]`, `[active]`, +`[ahead]` — the plan's own state vocabulary, distinct from the goal tags. + +## 1. How the plan works + +Five rules shape every phase below, and they are the project's contract with +itself: + +1. **Every milestone has a gate and an acceptance signal, named before the + work starts.** A wall is closed only by its acceptance signal — not by an + intention, a decompiler reading, or a lucky run. +2. **The evaluation ladder is the spine.** A platform advances + initialization → runtime start → execution → sustained → multi-tenant, and + nothing claims a rung it has not reached and reproduced. +3. **Correctness before speed; no number without methodology.** Performance + claims carry their harness, versions and A/B baseline. +4. **Default-off until validated.** New mechanisms land behind parameters + (the forwarding and graphics seams today) and graduate only after their + gate passes; graduation replaces the knob with documented configuration. +5. **No patches anywhere, and profiles over rewrites.** Guest drivers, host + drivers, firmware and hypervisors stay stock; a new driver, firmware or GPU + enters the plan as a *profile plus a measured delta*. + +## 2. The phases + +| Phase | What it delivers | Exit gate | State (2026-11-11) | +|---|---|---|---| +| **P0 — Platform** | Emulated vGPU device, GSP boot and GSP-RM emulation, stock guest driver initialisation | Driver-API smoke green (`ALL OK`) = evaluation rung 2 | `[done]` | +| **P1 — First execution (compute)** | Guest CUDA kernels executing on the physical GPU, with engine-authored completions; then hardening | Verified clients + engine-authored completion = rung 3 core; hardening: staged-transfer class, every-boot acceptance, per-guest isolation | `[active]` — execution green; hardening in progress | +| **P2 — Graphics execution** | OpenGL/Vulkan draw commands executed by the physical 3D pipeline and delivered as readable pixels | The graphics ladder: M-GL1 client context-init executes; M-GL2 `glxgears` non-black; M-GL3 GZDoom GL non-black through the mdev | `[active]` — between M-GL1 and M-GL2 with M-GL2's off-screen half proven: the display/KMS and DDX replay machinery work, the first non-black GPU-rendered frames and a 45.4 s video are captured, and the direct-GL GR ring resolves; the direct-GL CE ring, the window/screen present and X stability are the walls | +| **P3 — Sustained and multi-tenant** | Frame rates without assistance; multiple VMs sharing the GPU with measured fairness; soak-stable operation | Graphics M-GL4 (60 fps, no forcing) and M-GL5 (two VMs, two mdevs); CUDA battery + soak; per-guest execution isolation = rungs 4/5 | `[ahead]` | +| **P4 — Breadth** | Video (NVENC/NVDEC), Windows/DirectX guests, live migration, non-KVM hypervisor transports | Per-case bring-up gates plus the matrix's evidence expectations | `[ahead]` | +| **P5 — Scale and portability** | New GPU generations and driver families carried forward; published performance programme | A profile + measured delta per version, validated on hardware; A/B numbers published per the measurement contract | `[ahead]` | + +The phases overlap on purpose: the graphics workstream is what unblocks P3's +first exit gate, and portability work starts from whichever hardware becomes +available (P5 runs across the whole plan). + +## 3. Where the plan stands, and what is next + +Ranked, in dependency order: + +1. **Graphics pixels (P2).** The client and display stacks execute, the + DDX 3D replay runs, the first non-black GPU-rendered frames and a 45.4 s + video are captured off-screen, and the direct-GL GR ring `0xbeef0100` + resolves with advancing releases. The named walls are the direct-GL CE + ring `0xbeef0101` (it never resolves, so GZDoom wedges ~90 s after + `V_Init`), the window/screen present path (glxgears' window is black and a + window draw wedges X), X startup/stability (`_nv052600rm` soft-lock, + `0x51`, vmalloc), and the DDX's 4 GB sysmem allocation (12 GB guest + workaround), then non-black window frames. Full detail: + [`Graphics-Execution.md`](Graphics-Execution.md) §6. +2. **Compute hardening (P1).** The staged-transfer class (failure modes proven, + fixes in flight); the every-boot acceptance campaign; per-guest execution + isolation. Detail: + [`Execution-Frontier.md`](Execution-Frontier.md). +3. **The visible sequence after these**: first non-black frame → 60 fps in one + VM with no forcing → two VMs on two mdevs → the CUDA battery 15/15 plus a + soak → video → Windows. +4. **Then P4/P5 row by row**: each item ships with its own gate, profile and + evidence, and only then moves in the matrix. + +Recently closed, as evidence the gates work: the boot-poisoning defect and the +post-reboot client-stall defect (compute reproducibility); the display/KMS +bring-up (NvKms registers, X settles, GL 4.6); the UVM channel-pool tracker; +the direct-GL GR release and CE ring discovery; the X DDX kick-channel decode +and its three wait fixes (class-aware display arm, `0x8000 -> 0` notification +completion, decoded GL release applied); the GSP event-IRQ fix that completes +swaps; the X DMA-init fix (`ebf3130f`); the GR backend timeout fix +(`backend_timeout=0`); the 2D path end-to-end; the DDX 3D replay and the +first non-black GPU-rendered frames with a captured video (the ring-ledger +scrub release, `f6184ac3`); the direct-GL GR ring resolution (`4acac093`); +the RM write-lock holder fix (`df6e71bb`); and the guest Vulkan pipeline +running end to end. + +## 4. The gates + +Each gate's full text lives in its canonical document; this is the map: + +| Gate | What must be true | Where it is defined | +|---|---|---| +| **Rung gate** | The previous rung is green and reproducible for this platform before the next is claimed | Evaluation ladder | +| **Execution gate** | Engine-authored release plus verified guest results, reproduced across fresh boots | Execution frontier | +| **Graphics gate** | Milestone definitions: capture shows motion, no forcing, no diagnostic mirror, renderer is the NVIDIA driver | Graphics plan (§C) | +| **Performance gate** | A/B ratios from the published harness; no absolute numbers; variance and run counts reported | Measurement contract | +| **Portability gate** | Profile fields recorded, firmware/device pairing hashed, first-touch differences explained, supervised physical validation done | Porting sequence + matrix | +| **Isolation gate** | Fail-closed enforcement; fairness measured before any multi-tenant claim | Security document | +| **Security gate** | Adversarial review before multi-tenant or production claims | Security document | + +## 5. Workstreams (always on) + +| Workstream | What it does | Entry point | +|---|---|---| +| Correctness and acceptance | The regression battery, battery-first ordering, regression gates | [`Execution-Frontier.md`](Execution-Frontier.md) | +| Graphics execution | The execution lanes, capture/decode, the M-GL ladder | [`Graphics-Execution.md`](Graphics-Execution.md) | +| Performance measurement | Microbenchmarks first, then the A/B corpus, then structural optimisation | [`Performance.md`](Performance.md) | +| Portability and versions | Profiles, measured deltas, the validation sequence; new driver/firmware families behind a review gate | [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) | +| Developer tooling | The firmware workbench: recorded sessions as evidence, offline-first investigation of walls | [`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md) | +| Security and isolation | Threat model, audits, fail-closed posture | [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) | +| Documentation and evidence | Docs tree canonical; dated status; replayable evidence for every claim | [`../docs/README.md`](../docs/README.md) | + +## 6. Contributing to the plan + +* **The unit of contribution is a profile plus evidence**, evaluated through + the five-rung ladder — see "Adding a new combination" in + [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) and the porting + sequence document. +* **Nasty bugs** (non-reproducible, cross-subsystem, blocking a gate) get a + tracked issue in the repository; ordinary work stays in the live handoffs. +* **Support claims** are made only by recording hardware + driver + firmware + + rung + date in the matrix with the evidence bundle behind them. + +## 7. How this plan is maintained + +The plan is a living sequence, not a dated roadmap: each milestone's gate is +owned by its canonical document, the current state is tracked on +[`Project-Status.md`](Project-Status.md), and the plan changes only when a gate +is named and its evidence rules are written down. When a wall closes, the +canonical doc is updated first and the site pages follow on the next refresh. + +## 8. Related pages + +* [`Project-Goals.md`](Project-Goals.md) — the goals and their status. +* [`Project-Status.md`](Project-Status.md) — the dated milestone ladder. +* [`Execution-Frontier.md`](Execution-Frontier.md) — the compute frontier and + its gates. +* [`Graphics-Execution.md`](Graphics-Execution.md) — the graphics frontier and + the M-GL ladder. +* [`Performance.md`](Performance.md) — the measurement programme. +* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) — the + portability plan and support ladder. +* [`../docs/README.md`](../docs/README.md) — the documentation front door and + the project's own status table. diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md index e63a198..137b7d9 100644 --- a/wiki/Project-Status.md +++ b/wiki/Project-Status.md @@ -10,7 +10,7 @@ > (the execution frontier). When this page and the docs disagree, the docs tree > wins. -Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or behind supervised gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]` named success state; `[planned]` designed, not built. @@ -24,11 +24,14 @@ named success state; `[planned]` designed, not built. | GSP-RM message queues and RPC transport | `[live]` working | | Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working | | CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly | -| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. Remaining: the large-buffer staged-transfer class is under repair (some large multi-chunk clients still mis-verify); fresh-boot re-measurement is in progress after the boot-poisoning fix; per-guest execution isolation for simultaneous CUDA guests is in progress | -| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend | -| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | -| OpenGL / Vulkan (3D), video | `[planned]` | -| Windows guests | `[planned]` (target) | +| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. A ring-block ownership defect that could stall the first client after a reboot was root-caused and fixed. Remaining: the staged-transfer class (some large multi-chunk clients still mis-verify; the failure modes are proven and under repair) and per-guest execution isolation for simultaneous CUDA guests | +| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend, and graphics method streams execute on the engine ([`Graphics-Execution.md`](Graphics-Execution.md)) | +| Display and KMS on the emulated vGPU (nvidia-modeset/NvKms, `/dev/dri`, X) | `[live]` in the campaign configuration — the emulated display engine serves the guest's display push-buffer channels (notifier plus `UPDATE -> FINISHED`), nvidia-modeset/NvKms registers (`/dev/dri/card0`, `renderD128`, `fb0`), `nvidia-smi -L` enumerates and X settles; the display-capability seam is parameterised ([`Graphics-Execution.md`](Graphics-Execution.md)) | +| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress | +| OpenGL (3D graphics) | `[wip]` — the display/KMS stack registers, X settles and `glxinfo` reports OpenGL 4.6.0 with direct rendering; the 2D path is green, the DDX 3D replay executes on `0x1000014`, and the **first non-black GPU-rendered output** is proven: a pbuffer probe returns `pixel=ff0000ff`, the off-screen frame path produced 500/500 non-black animated frames, and a 45.4 s video was captured. The direct-GL **GR ring `0xbeef0100` now resolves** (its releases advance frame by frame), so GZDoom reaches `V_Init: allocate screen` with its GL 4.6.0 context whenever the shared nvClip `modeNum` is even. **Window/screen pixels remain the wall:** glxgears' window is still black, the direct-GL **CE ring `0xbeef0101` never resolves** and the guest wedges ~90 s after `V_Init`, X window creation wedges on some boots (`_nv052600rm+0x7b`), and the DDX's 4 GB sysmem allocation forces a 12 GB guest ([`OpenGL.md`](OpenGL.md)) | +| Vulkan (3D/compute) | `[wip]` — the guest Vulkan pipeline runs end to end on the emulated vGPU (device, allocation, pipeline, command buffer, fence, queue submit, readback) and the engine executes translated launches; frames are still black — program delivery and program-VA mapping are the frontier ([`Vulkan.md`](Vulkan.md)) | +| DirectX / Windows guests | `[planned]` (target) — no Windows guest booted yet; the architecture is the same emulation target ([`DirectX.md`](DirectX.md)) | +| Video (NVENC/NVDEC) | `[planned]` | | Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) | | Live migration | `[planned]` | | Performance numbers | `[planned]` — no published numbers yet; any claim will ship with its measurement methodology | @@ -50,17 +53,20 @@ The project's definition of done for CUDA requires all of the following 5. Correct completion, semaphore and fault semantics back to the guest — the GR completion path now relays (engine-authored releases wake the guest's synchronisation waits); fault semantics remain partial. -6. Reproducibility: every boot, not a lucky one — root-caused and fixed (the - boot-poisoning defect: the wrong status-queue consumption cursor could - desynchronise the reply ring under load and poison the boot); - re-measurement across fresh boots is in progress. +6. Reproducibility: every boot, not a lucky one — two root causes landed: the + boot-poisoning defect (the wrong status-queue consumption cursor could + desynchronise the reply ring under load) and the ring-block ownership + defect that could stall the first client after a reboot. Green clients are + reproduced across fresh boots; a formal every-boot acceptance campaign is + the remaining work. 7. No ad-hoc default-off parameters required for the canonical path — still - open. + open (the forwarding and graphics seams remain parameterised while they + are validated). Until every gate holds, the honest formulation is: **initialization, the driver-API path and kernel execution on the physical GPU work and verify; -hardening (large-buffer transfers, boot-to-boot reproducibility) and -per-guest isolation remain.** +the staged-transfer class, per-guest isolation and the graphics +direct-GL/present/first-frame frontier remain.** ## 3. How to see it yourself @@ -68,7 +74,14 @@ The reference setup takes about half an hour from a fresh host ([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full driver-API smoke, and guest CUDA kernels executing and verifying on the -physical GPU. The +physical GPU. On the graphics side the same setup brings up the display/KMS +stack and the X server on the emulated vGPU (`glxinfo` reports OpenGL 4.6.0 +with direct rendering); the first non-black GPU-rendered frames are proven +off-screen (the pbuffer path produced a 500-frame, 45.4 s video on the +emulated vGPU) and the direct-GL GR channel now submits with advancing +releases, but windowed/screen pixels remain the frontier (the direct-GL CE +channel does not yet resolve and X window creation can wedge) — see +[`OpenGL.md`](OpenGL.md). The risc-v-gsp-rm-debugger workbench ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side tooling used to study the firmware side of that frontier. diff --git a/wiki/Riscv-GSP-RM-Debugger.md b/wiki/Riscv-GSP-RM-Debugger.md index 645f4a8..8155364 100644 --- a/wiki/Riscv-GSP-RM-Debugger.md +++ b/wiki/Riscv-GSP-RM-Debugger.md @@ -4,7 +4,7 @@ > > **Development-only.** Host-side tooling that is never shipped, never a driver dependency and never present in a guest; interfaces and file formats may change without notice. > -> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` recorded QEMU sessions; `[design]` device model and custom simulator are roadmap. +> Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` recorded QEMU sessions; `[wip]` partial device model; `[design]` full device model and custom simulator are roadmap. ## 1. What it is @@ -21,7 +21,15 @@ Its purpose is **capturing how the firmware server behaves, repeatably and safel ## 2. What it is not -* **Not a GSP boot environment.** QEMU `virt` has no falcon windows, WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. `probe` reports exactly where the real image first diverges (for the TU104 570.144 image: a load-access fault at firmware vaddr `0x4009014` reading `0x2000000000000A00`) — expected data, not a boot. A device model is a roadmap item, not a claim. +* **Not a complete GSP boot environment.** QEMU `virt` has no falcon windows, + WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. `probe` reports exactly where the + real image first diverges (for the TU104 570.144 image: a load-access fault + at firmware vaddr `0x4009014` reading `0x2000000000000A00`) — expected data, + not a boot. A **partial device model** ("S1") exists and boots the real + image natively through its relocation copy on the simulator's boot ladder + ([`../debug/risc-v-gsp-rm-vm/docs/36-GSP-DEVICE-MODEL.md`](../debug/risc-v-gsp-rm-vm/docs/36-GSP-DEVICE-MODEL.md)); + the message-loop rungs and the remaining device environment are roadmap, + not claims. * **Not a timing model.** Do not infer latency, jitter or concurrency behaviour from it; its in-scope conclusions are execution flow and memory side effects. * **Not a replacement for Ghidra or the guest diagnostics.** Ghidra answers *what is the code*; risc-v-gsp-rm-debugger answers *what did it do*; the guest driver remains the oracle for guest-side behaviour. * **Not a hardware or security tool.** It never attaches to a real GPU, a guest, a kernel module or an mdev, and it is never a security or confidentiality boundary. @@ -112,7 +120,15 @@ Rules that make a session usable as evidence: * **Bounded everywhere.** Runs have durations and stop caps; reads and snapshots have size caps; a wedged target is left stopped and restarted clean rather than poked. * **Local only.** The Ghidra bridge is a localhost transport, unauthenticated by design; never expose it on a network interface. * **No confidentiality claims.** The tool is a study environment; never use injection to defeat a hardware security boundary. -* **Status.** The audited v0.1 surfaces are the CLI and the session format. Working and live-verified: hooks/watchpoints, memory tools, sessions, declarative probes, plugins, remote calls, run-to-cursor, the injection SDK, the Ghidra bridges, the first-fault probe and the instruction/access trace. The device model, the custom simulator and serving full host message traffic remain roadmap; porting and packaging scaffolds are pilot-status. Treat interface changes as reviewed events. +* **Status.** The audited surfaces are the CLI and the session format. + Working and live-verified: hooks/watchpoints, memory tools, sessions, + declarative probes, plugins, remote calls, run-to-cursor, the injection + SDK, the Ghidra bridges, the first-fault probe and the instruction/access + trace. A partial device model boots the real GSP image through its + relocation-copy rung deterministically; the message loop, the custom + simulator and serving full host message traffic remain roadmap; porting and + packaging scaffolds are pilot-status. Treat interface changes as reviewed + events. ## 9. Related documentation diff --git a/wiki/System-Overview.md b/wiki/System-Overview.md index 3f2d3e2..6ad4d3e 100644 --- a/wiki/System-Overview.md +++ b/wiki/System-Overview.md @@ -7,7 +7,7 @@ > and [`../docs/03-architecture/10-interaction-and-scheduling.md`](../docs/03-architecture/10-interaction-and-scheduling.md) > for the full picture); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-11-11, tree `master @ 4acac093`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partial; `[wall]` current blocker; `[target]` designed end state. @@ -32,8 +32,10 @@ EMULATE (software; the guest expects virtual hardware) PASS THROUGH (real hardware; correctness and performance) GPU memory: guest VRAM objects backed by real VRAM [target] page tables / GMMU state used by the GPU [wip] - channel execution: guest push buffers on real channels [gated/wall] - compute (CUDA) and, later, 3D/video work [wip/planned] + channel execution: guest push buffers on real channels [wip] + compute (CUDA) kernel launches on the real engine [live] + 3D graphics method streams on the real engine [wip] + rendered 3D pixels / video work [wall/planned] DMA, completion and fault interrupts relayed to the guest ``` @@ -130,9 +132,25 @@ RPC from the guest to "firmware": payload round-trips). 5. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored semaphore releases and a fault relay. For GR - (graphics/compute), a forwarded guest launch reaches the front-end but the - engine has not executed it yet — the current frontier + (graphics/compute), forwarded guest CUDA launches execute on the engine + with engine-authored completions relayed to the guest; the direct GL + client's GR and CE channels submit and retire, and graphics method streams + execute in three shapes ([`Graphics-Execution.md`](Graphics-Execution.md)); + rendered 3D pixels are the graphics frontier ([`Execution-Frontier.md`](Execution-Frontier.md)). +6. On the graphics side, the emulated display engine serves the modeset push + buffers and the display notifier/completion (class-aware arm), so + nvidia-modeset/NvKms registers (`/dev/dri`), X settles and `glxinfo` + reports OpenGL 4.6.0 with direct rendering; the direct GL client's + channels submit, the X DDX waits are fixed, the DDX 3D replay executes, + the 2D path is green, the first non-black GPU-rendered frames and a + 45.4 s video are captured off-screen, the direct-GL GR ring resolves with + advancing releases, and the guest Vulkan pipeline runs end to end. The + direct-GL CE ring `0xbeef0101` still does not resolve, a window draw + wedges X (glxgears' window is black), and X window creation wedges on some + boots, so the remaining gate is the window/screen present before the + first non-black window frame ([`OpenGL.md`](OpenGL.md), + [`Vulkan.md`](Vulkan.md)). ## 6. Where the detail lives diff --git a/wiki/UVM-in-the-Guest.md b/wiki/UVM-in-the-Guest.md index 53ca755..af1238f 100644 --- a/wiki/UVM-in-the-Guest.md +++ b/wiki/UVM-in-the-Guest.md @@ -5,7 +5,7 @@ > Canonical source: [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md); > where this page and the docs tree disagree, the docs tree wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the stock guest driver; `[wip]` partially working, software bridge in charge; `[gated]` implemented default-off; `[target]` planned end state. @@ -60,13 +60,13 @@ Each stage has a distinct failure signature — see | Completion semantics | tracker values must advance exactly | `[wip]` | **Current frontier.** Context creation and the full CUDA driver-API smoke pass -(`ALL OK`, copies matched) run end to end. CE-class copies are forwarded to the -physical copy engine (default off) with engine-authored releases and a fault -relay. For GR, a forwarded guest launch reaches the front-end but the inline -launch-descriptor install has not been observed; context validity at dispatch -is ruled out ([`Execution-Frontier.md`](Execution-Frontier.md)). The remaining -UVM work is completion translation and the guest-side fault replay/cancel -contract. +(`ALL OK`, copies matched) run end to end, and guest CUDA kernel launches now +execute on the physical engine with engine-authored completions relayed to the +guest. The remaining UVM work is completion translation for the staged-transfer +class, the guest-side fault replay/cancel contract, and per-guest execution +isolation ([`Execution-Frontier.md`](Execution-Frontier.md)). Graphics work +rides the same submission machinery on its own frontier +([`Graphics-Execution.md`](Graphics-Execution.md)). ## 4. The discovery that defines the work: how UVM waits @@ -164,5 +164,5 @@ behaving like a real GPU. * [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md) — canonical chapter (this page's source). * [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the guest stack and the RM objects UVM consumes. -* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current wall. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current leading edge. * [`../docs/04-implementation/01-known-knowns-and-unknowns.md`](../docs/04-implementation/01-known-knowns-and-unknowns.md) — server-owned page tables and the fault contract. diff --git a/wiki/Vulkan.md b/wiki/Vulkan.md new file mode 100644 index 0000000..d213136 --- /dev/null +++ b/wiki/Vulkan.md @@ -0,0 +1,127 @@ +# Vulkan + +> **Scope.** How a stock guest Vulkan application runs on the emulated vGPU: +> the guest-side stack, the submit/completion path, and the honest state of +> the work — including the one thing every Vulkan run on the emulated vGPU +> currently has in common: **the pipeline completes, and the frames are +> black**. The shared host machinery is on +> [`Graphics-Execution.md`](Graphics-Execution.md); the CUDA compute chain +> (further along) is on [`Execution-Frontier.md`](Execution-Frontier.md). +> Canonical sources: +> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) +> and +> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md); +> the docs tree wins on disagreement. + +Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the +stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]` +defined success state. + +## 1. The guest Vulkan stack + +A Vulkan application loads the **loader** (`libvulkan.so`), which dispatches to +an **installable client driver (ICD)** — for NVIDIA GPUs, the NVIDIA Vulkan +driver. On Linux, Vulkan compute does not require an X server; the display +path does (through `nvidia-modeset`/DRM for presentation). The kernel-side +requirements are the same as CUDA's: `nvidia.ko`, the GSP client, channels, +memory objects and fences. + +At device creation the driver also walks the **display/modeset device** +allocation path (the `ALLOC_DEVICE` class family) — on a vGPU-style device the +driver expects the host to provide those display classes, which the emulation +must answer for `vkCreateDevice` to succeed. Past that gate, the driver +behaves much like the compute stack: memory types through its VidHeap classes, +queues as channels, and work as method streams. + +## 2. The submit and completion path (observed) + +```mermaid +sequenceDiagram + participant App as Guest app + participant ICD as NVIDIA Vulkan driver + participant KMD as nvidia.ko + GSP client + participant VG as vgsp.ko emulation + participant GPU as Physical GPU (GR/CE) + App->>ICD: record command buffer, then vkQueueSubmit(fence) + ICD->>KMD: push buffer + GPFIFO entry + doorbell + KMD-->>VG: method stream (compute C5C0 shape + copy shapes) + VG->>VG: capture the section, translate to a vendor launch,<br/>relocate operands into reachable shadow pages + VG->>GPU: submit on a real channel (launch descriptor install) + GPU-->>VG: engine-authored release + VG-->>KMD: completion relay (MSI-X + poller) + KMD-->>App: fence released, vkWaitForFences returns + App->>App: reuse buffers, read back the frame +``` + +```text +app: vkQueueSubmit(fence) + -> NVIDIA driver: push buffer + GPFIFO entry + doorbell + -> vgsp.ko: capture section, translate to a vendor launch, + relocate operands into shadow pages, submit on a real channel + -> physical GPU: front-end fetch, engine executes, authors release + -> vgsp.ko: completion relay (MSI-X + poller) back to the guest + -> app: fence releases; frame readback proceeds +ASCII fallback above; the sequence shows one queue submit end to end. +``` + +Two mechanics make this work today: the emulation decodes the driver's +compute-dispatch shape — a **direct method stream** (object bind, window runs, +code runs, a subchannel data-port stream) with **no inline launch descriptor** +— and *translates* it into a launch the real engine accepts, and it derives +the submit's release groups from the driver's own push words so the fence +value is released when the engine completes (the "fence fold": the pre-built +GPFIFO continuation beyond the put pointer is swept and the submit's release +groups are applied forward-only). + +## 3. Status on the emulated vGPU + +| Item | State | +|---|---| +| `vkCreateDevice` on the emulated vGPU (display/modeset allocation path answered) | `[live]` | +| Full pipeline: instance, device, allocation, image/view/buffer/shader/descriptor, pipeline, command buffer, fence | `[live]` | +| Queue submit, engine-side execution of the translated launch, fence release, frame readback | `[live]` | +| The client saves frames end to end (`VKFRAME` evidence) | `[live]` | +| Non-black frames | `[wall]` | +| The driver's program (SASS) upload materialised by the emulation | `[wall]` | +| The engine's page tables mapping the program VA | `[wall]` | + +## 4. Where the wall is, precisely + +The client *believes* it uploaded its shader and dispatches with a valid +program address, but the bytes never appear in any source the emulation can +see: the driver delivers programmed SASS and descriptors through an **MME +program + inline-to-memory stream** on its copy/utility channel, and the +inline-data writes are not yet materialised for the graphics clients. Two +coupled facts define the remaining work: + +1. **Delivery** — decode and execute the driver's inline upload so the + program page becomes non-zero (the same machinery already works for CUDA's + uploads; the graphics clients' shapes are the extension). +2. **Mapping** — the engine's page tables must map the program VA. With the + data-set gate bypassed, the engine fetches the launch and raises a real + MMU fault (`FAULT_PDE`) at the program VA — proof the channel, context and + fetch path are alive, and that mapping plus content is the missing pair. + +When both land, the emulated vGPU should carry its first non-black Vulkan +frame — the milestone the whole graphics effort is converging on +([`Graphics-Execution.md`](Graphics-Execution.md) §6). + +## 5. What this page is not claiming + +* No rendered pixels yet — every frame saved so far is black. +* No performance numbers. +* `vkCreateDevice` success does not imply rendering; it means the device + model answers the driver's device-view correctly. + +## 6. Related pages + +* [`Graphics-Execution.md`](Graphics-Execution.md) — the shared lanes, the + `0xC597` graphics stream and the ranked walls. +* [`OpenGL.md`](OpenGL.md) — the GL path through the X server. +* [`DirectX.md`](DirectX.md) — the Windows/DirectX target. +* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace + driver model the Vulkan ICD mirrors in structure. +* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — translation and the + engines. +* [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) — + the measured wall ledger, with every experiment and its verdict. |